JP7702892B2 - Inspection device and inspection method - Google Patents

Description

The present invention relates to an inspection device and an inspection method, and in particular to a food presentation inspection technology that uses image recognition.

Appearance inspections, such as inspecting food presentation and checking for missing ingredients, are known as quality control methods for food produced on production lines. Traditionally, these inspections have been carried out visually. In recent years, with the development of image recognition technology, technology has been developed in the food production field that uses image recognition for quality control of produced foods.

For example, Patent Document 1 discloses a technology for capturing images of a food to be inspected at two different wavelengths, performing spectral analysis of each pixel to generate an inspection image, and inspecting the condition of the food to be inspected.

However, in actual food manufacturing sites, quality inspections such as food presentation inspections are still often performed visually. The reason for this is that the ingredients arranged on food products often have irregular shapes, and even the same ingredients can have different shapes and ingredients whose colors are also not always consistent. As a result, it is often difficult to always use the same standards to inspect food presentation, including checking that the specified ingredients are arranged on manufactured foods and the number, amount, and presentation position of each ingredient. For this reason, creating the detection logic for food presentation inspections using image recognition can be difficult.

Furthermore, in recent years, diversifying consumer needs have led to the trend toward high-mix, low-volume production and shorter product lifecycles. Even if detection logic is created, the scope of its use may be limited, making it difficult to use image recognition technology to inspect food presentation as a practical method.

Shorten, C., Khoshgoftaar, T.M. “A survey on Image Data Augmentation for Deep Learning.” J Big Data 6, 60 (2019). https://doi.org/10.1186/s40537-019-0197-0 Khalifa, Nour Eldeen & Loey, Mohamed & Mirjalili, Seyedali. (2021). “A comprehensive survey of recent trends in deep learning for digital images augmentation.” Artificial Intelligence Review. 10.1007/s10462-021-10066-4. Naveed, Humza. (2021). “Survey: Image Mixing and Deleting for Data Augmentation.”

JP 2019-74324 A Patent No. 6955734 Patent No. 6929322

With conventional technology, it has been difficult to easily inspect food presentation using image recognition.

The present invention has been made to solve the above-mentioned problems, and aims to make it easier to inspect food presentation using image recognition.

To solve the above-mentioned problems, the inspection device of the present invention includes an acquisition unit that acquires an image containing arranged ingredients, a first estimation unit that estimates the area and type of the ingredients in the image using a trained first model and outputs a first estimation result including an output image in which the estimated area and type of the ingredients are associated with the image, a first judgment unit that judges, based on the first estimation result, whether the corresponding area for each type of ingredient satisfies a first predetermined criterion regarding the presence or absence and excess or deficiency of the ingredient, and an inspection unit that inspects the arrangement of the ingredients based on the judgment result by the first judgment unit.

In addition, in the inspection device according to the present invention, the first judgment unit may identify at least one of the number and size of the regions for each type of food material and judge whether or not at least one of the number and size of the regions satisfies the first criterion.

The inspection device according to the present invention may further include a second estimation unit that estimates the relative positional relationship between the ingredients in the output image using a trained second model that uses the output image included in the first estimation result as an input, and outputs a second estimation result that associates the estimated positional relationship with the output image, and a second judgment unit that judges whether the positional relationship satisfies a second criterion set in advance regarding the validity of the positional relationship based on the second estimation result, and the inspection unit may inspect the positional relationship of the ingredients based on the judgment result by the second judgment unit.

The inspection device according to the present invention may further include a presentation unit that presents at least the inspection results by the inspection unit, and the presentation unit may present at least one of the image, the first estimation result, the judgment result by the first judgment unit, the second estimation result, the judgment result by the second judgment unit, and information including the inspection results by the inspection unit.

The inspection device according to the present invention may further include a learning device that performs a learning process to construct a preset model, and the learning device may include a first learning unit that uses an image containing the arranged ingredients as a first learning image, learns a first model of a neural network, extracts a first feature amount that identifies the area and type of the ingredients, and acquires the trained first model using the extracted first feature amount, and a first storage unit that stores the trained first model acquired by the first learning unit, and the first estimation unit may read the trained first model from the first storage unit and estimate the area and type of the ingredients in the image using the trained first model.

In addition, in the inspection device according to the present invention, the first model may be an image recognition model based on instance segmentation.

In the inspection device according to the present invention, the learning device may further include a second learning unit that uses the output image included in the first estimation result output by the first estimation unit as a second learning image to train a second model of a neural network, extracts second features that identify the relative positional relationship between the multiple ingredients in the second learning image, and acquires the trained second model using the extracted second features, and a second storage unit that stores the trained second model acquired by the second learning unit.

In addition, the inspection device according to the present invention may further include a conversion unit that converts the output image included in the first estimation result to generate a simplified converted image, and the second learning unit may use the converted image as the second learning image to train the second model and acquire the trained second model.

In addition, in the inspection device according to the present invention, the output image included in the first estimation result may be a color pattern image in which the corresponding areas for each type of food ingredient are painted with an arbitrary color, and the converted image may be a color pattern image in which each area of the color pattern image is replaced with an object of the same shape.

In addition, in the inspection device according to the present invention, the output image included in the first estimation result may be a color pattern image in which the corresponding area is painted with an arbitrary color for each type of food ingredient, and the converted image may be an image in which each area of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the replaced object has a different color for each shape.

In addition, in the inspection device according to the present invention, the output image included in the first estimation result may be a color pattern image in which the corresponding area is painted with an arbitrary color for each type of food ingredient, and the converted image may be an image in which each area of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the colors of the color patterns of the replaced objects are all the same color.

In addition, the inspection device according to the present invention may further include a processing unit that generates a processed image by processing the output image by assuming that the inside of the longest estimated outline of the area of the output image included in the first estimation result is the area in question, and the second learning unit may use the processed image as the second learning image to train the second model and acquire the trained second model.

In addition, in the inspection device according to the present invention, the learning device further includes a data expansion unit that performs data expansion of a target image in which the region and type of the ingredient are associated with the image to generate the second learning image, and the data expansion unit may generate the second learning image based on label information that is assigned to the target image and that is related to the validity of the relative positional relationship between the multiple ingredients in the target image.

In addition, in the inspection device according to the present invention, the data expansion unit may perform data expansion of a target image to which a first label indicating that the relative positional relationship between the multiple ingredients in the target image is valid is assigned, and generate the second learning image to which the first label is assigned.

In addition, in the inspection device according to the present invention, the data expansion unit may perform data expansion of a target image to which a first label indicating that the relative positional relationship between the multiple ingredients in the target image is valid is assigned, and generate the second training image to which a second label indicating that the relative positional relationship between the multiple ingredients is invalid is assigned.

In addition, in the inspection device according to the present invention, the data expansion unit may perform data expansion of a target image to which a second label indicating that the relative positional relationship between the multiple ingredients is not valid is assigned, and generate the second learning image to which the second label is assigned.

The inspection device according to the present invention may further include an extraction unit that extracts the regions for each type of food ingredient from the target image, and the data expansion unit may process and rearrange the regions extracted by the extraction unit while maintaining the relative positional relationship between the multiple food ingredients in the target image to generate the second learning image.

The inspection device according to the present invention may further include an extraction unit that extracts the regions for each type of food ingredient from the target image, and the data expansion unit may generate the second learning image by mutually replacing positions of regions corresponding to two or more different types of food ingredients among the regions extracted by the extraction unit.

The inspection device according to the present invention may further include an extraction unit that extracts the regions for each type of food material from the target image, and the data expansion unit may generate the second learning image by deleting one or more of the regions extracted by the extraction unit from the target image.

The inspection device according to the present invention may further include an extraction unit that extracts the regions for each type of food material from the target image, and the data expansion unit may perform data expansion of the target image by performing at least one of inversion, rotation, expansion, and contraction of each of the regions extracted by the extraction unit.

In order to solve the above-mentioned problems, the inspection device according to the present invention includes an acquisition unit that acquires an image containing arranged ingredients, a first estimation unit that uses a trained first model to detect the area of the ingredients in the image using a bounding box, estimates the relative arrangement between the ingredients and the type of the ingredients, and outputs a first estimation result including an output image in which the estimated arrangement is associated with the image, a first judgment unit that judges whether the arrangement satisfies a preset criterion based on the first estimation result, and an inspection unit that inspects the arrangement of the ingredients based on the judgment result by the first judgment unit.

Furthermore, in order to solve the above-mentioned problems, foods that can be inspected using the inspection device according to the present invention include processed noodles, frozen foods, boxed lunches, and side dishes that are manufactured on a food manufacturing line and in which the ingredients are arranged and served.

To solve the above-mentioned problems, the inspection method of the present invention includes a first step of acquiring an image containing arranged ingredients, a second step of estimating the area and type of the ingredients in the image using a trained first model and outputting a first estimation result including an output image in which the estimated area and type of the ingredients are associated with the image, a third step of determining, based on the first estimation result, whether the corresponding area for each type of ingredient satisfies a first predetermined criterion regarding the presence or absence and excess or deficiency of the ingredient, and a fourth step of inspecting the arrangement of the ingredients based on the determination result in the third step.

In addition, in the inspection method according to the present invention, the third step may involve identifying at least one of the number and size of the regions for each type of food ingredient and determining whether or not at least one of the number and size of the regions meets the first criterion, and the fourth step may involve inspecting the arrangement of the food ingredients based on the determination result.

The inspection method according to the present invention further includes a fifth step of estimating the relative positional relationship between the ingredients in the output image using a trained second model that uses the output image included in the first estimation result as an input, and outputting a second estimation result in which the estimated positional relationship is associated with the output image, and a sixth step of determining whether the positional relationship satisfies a second predetermined criterion regarding the validity of the positional relationship based on the second estimation result, and the fourth step may inspect the positional relationship of the ingredients based on the determination result in the sixth step.

The inspection method according to the present invention further includes a learning step of performing a learning process to construct a preset model, the learning step including a seventh step of using an image including the arranged ingredients as a first learning image to train a first model of a neural network, extracting a first feature that identifies the area and type of the ingredients, and acquiring the trained first model using the extracted first feature, and an eighth step of storing the trained first model acquired in the seventh step in a first storage unit, and the third step may include reading the trained first model from the first storage unit and estimating the area and type of the ingredients in the image using the trained first model.

In the inspection method according to the present invention, the learning step may further include a ninth step of using the output image included in the first estimation result output in the third step as a second learning image to train a second model of a neural network, extracting a second feature that identifies the relative positional relationship between the multiple ingredients in the second learning image, and acquiring the trained second model using the extracted second feature, and a tenth step of storing the trained second model acquired in the ninth step in a second storage unit.

In addition, the inspection method according to the present invention may further include an eleventh step of generating a processed image by processing the output image, assuming that the inside of the longest estimated outline of the region of the output image included in the first estimation result is the region in question, and the ninth step may use the processed image as the second learning image to train the second model, thereby acquiring the trained second model.

In addition, in the inspection method according to the present invention, the learning step further includes a twelfth step of performing data expansion of a target image in which the area and type of the food ingredient are associated with the image to generate the second learning image, and the twelfth step may generate the second learning image based on label information attached to the target image regarding the validity of the relative positional relationship between the multiple food ingredients in the target image.

In addition, in the inspection method according to the present invention, the 12th step may involve performing data expansion on a target image to which a first label indicating that the relative positional relationship between the multiple ingredients in the target image is valid is assigned, to generate the second learning image to which the first label is assigned.

In addition, in the inspection method according to the present invention, the 12th step may involve performing data expansion of a target image to which a first label has been assigned indicating that the relative positional relationship between the multiple ingredients in the target image is appropriate, and generating the second learning image to which a second label has been assigned indicating that the relative positional relationship between the multiple ingredients is not appropriate.

In addition, in the inspection method according to the present invention, the 12th step may involve performing data expansion of a target image to which a second label indicating that the relative positional relationship between the multiple ingredients is not valid is assigned, and generating the second learning image to which the second label is assigned.

The inspection method according to the present invention may further include a thirteenth step of extracting the areas for each type of food ingredient from the target image, and the twelfth step may process and rearrange the areas extracted in the thirteenth step while maintaining the relative positional relationship between the multiple food ingredients in the target image to generate the second learning image.

The inspection method according to the present invention may further include a thirteenth step of extracting the regions for each type of food ingredient from the target image, and the twelfth step may generate the second learning image by mutually replacing the positions of the regions corresponding to two or more different types of food ingredients among the regions extracted in the thirteenth step.

The inspection method according to the present invention may further include a thirteenth step of extracting the regions for each type of food material from the target image, and the twelfth step may generate the second learning image by deleting one or more of the regions extracted in the thirteenth step from the target image.

The inspection method according to the present invention may further include a thirteenth step of extracting the regions for each type of food material from the target image, and the twelfth step may perform at least one of inversion, rotation, expansion, and contraction of each of the regions extracted in the thirteenth step to expand the data of the target image.

The inspection method according to the present invention may further include a fourteenth step of presenting at least the inspection results from the fourth step, and the fourteenth step may present at least any of information including the image, the first estimation result, the judgment result from the third step, and the inspection results from the fourth step.

According to the present invention, the area and type of ingredients in an image containing arranged ingredients are estimated using a trained first model, and for each type of ingredient, it is determined whether the corresponding area meets a pre-set first criterion regarding the presence or absence of ingredients and whether there is an excess or deficiency based on the first estimation result. This makes it easier to inspect food presentation using image recognition.

FIG. 1 is a block diagram showing the configuration of an inspection device according to a first embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a hardware configuration of the inspection apparatus according to the first embodiment. FIG. 3 is a flowchart showing an example of the operation of the inspection apparatus according to the first embodiment. FIG. 4 is an explanatory diagram showing an outline of the operation of the inspection device according to the first embodiment. FIG. 5 is a diagram showing an example of a display on a display device provided in the inspection device according to the first embodiment. FIG. 6 is a block diagram showing the configuration of an inspection device according to the second embodiment. FIG. 7 is a diagram for explaining a second estimating unit included in the inspection device according to the second embodiment. FIG. 8 is a flowchart showing an example of the operation of the inspection apparatus according to the second embodiment. FIG. 9 is an explanatory diagram showing an outline of the operation of the inspection device according to the second embodiment. FIG. 10 is a diagram showing an example of a display on a display device provided in the inspection device according to the second embodiment. FIG. 11 is a diagram for explaining the effect of the inspection device according to the second embodiment. FIG. 12 is a block diagram showing the configuration of an inspection device according to the third embodiment. FIG. 13 is a flowchart showing an example of the operation of the inspection apparatus according to the third embodiment. FIG. 14 is a block diagram showing the configuration of an inspection device according to the fourth embodiment. FIG. 15 is an explanatory diagram of a conversion unit included in an inspection device according to the fourth embodiment. FIG. 16 is an explanatory diagram of a conversion unit included in an inspection device according to the fourth embodiment. FIG. 17 is a block diagram showing the configuration of an inspection device according to the fifth embodiment. FIG. 18 is an explanatory diagram of a processing unit provided in an inspection device according to the fifth embodiment. FIG. 19 is a block diagram showing the configuration of an inspection device according to the sixth embodiment. FIG. 20 is a diagram for explaining an overview of an inspection device according to the sixth embodiment. FIG. 21 is a diagram for explaining a data expansion unit included in the inspection device according to the sixth embodiment. FIG. 22 is a flowchart showing an example of the operation of the inspection apparatus according to the sixth embodiment. FIG. 23 is a diagram for explaining an example of use of the inspection device according to the present embodiment. FIG. 24 is a diagram for explaining an example of use of the inspection device according to the present embodiment. FIG. 25 is a diagram for explaining an example of use of the inspection device according to the present embodiment.

A preferred embodiment of the present invention will be described in detail below with reference to Figures 1 to 25. In the following description, the inspection device according to the present invention inspects "arranged ingredients", which constitute "food" produced, for example, on a food production line, and includes both one and multiple ingredients. "Inspection of the arrangement of ingredients" may include one or more inspection items of the type, presence, number, and amount of each of multiple ingredients produced, for example, on a food production line and arranged in a container, and includes inspection of the location and arrangement of ingredients, which is the arrangement of the ingredients, and these are sometimes collectively referred to as "food presentation inspection". The arrangement of ingredients includes the way the ingredients are spread out.

[First embodiment]
FIG. 1 is a block diagram showing the configuration of an inspection device 1 according to a first embodiment of the present invention. The inspection device 1 according to this embodiment estimates the areas and types of multiple ingredients in an image containing multiple ingredients placed in a food container or the like, using a trained first model NN1 that has been trained and prepared in advance on an external server or the like. Furthermore, the inspection device 1 judges whether or not the corresponding area for each type of multiple ingredients meets a first criterion set in advance regarding the presence or absence and excess or deficiency of each of the multiple ingredients based on the estimated result (first estimation result). Furthermore, the inspection device 1 inspects the arrangement of ingredients placed in a container or the like based on the judgment result. In this embodiment, an example will be described in which instance segmentation is used as the trained first model NN1, which distinguishes ingredients from each other and identifies areas in an image in which ingredients are present on a pixel-by-pixel basis.

[Functional block of the inspection device]
The inspection device 1 according to this embodiment includes an acquisition unit 10 , a first estimation unit 11 , a first judgment unit 12 , an inspection unit 13 , a presentation unit 14 , and a memory 15 .

The acquisition unit 10 acquires an image including the arranged ingredients. For example, the acquisition unit 10 can acquire an image including multiple ingredients arranged in a food container or the like. The acquisition unit 10 acquires, for example, still or moving images of food captured by an externally installed camera 105 via wired or wireless connection. In this embodiment, the camera 105 is provided in a shooting environment that is constructed in advance in the food production line, and captures images including ingredients arranged in a container or the like (hereinafter, sometimes referred to as "food images"). In particular, the camera 105 is fixedly installed in the shooting environment in the production line, and captures images with a fixed angle of view. The acquisition unit 10 can perform known pre-processing such as adjusting the brightness of the food image acquired from the camera 105 and removing noise.

The first estimation unit 11 uses the trained first model NN1 to estimate the areas and types of multiple ingredients in the image acquired by the acquisition unit 10, and outputs a first estimation result including an output image by associating the estimated areas and types of multiple ingredients with the image acquired by the acquisition unit 10.

More specifically, the first estimation unit 11 provides the image acquired by the acquisition unit 10 to the trained first model NN1, and performs calculations of the trained first model NN1 to extract the type of ingredient, which is the ingredient class, for each pixel in the input image, and the area of the ingredient instance, which is the actual entity or individual of each ingredient that actually exists in the image.

The first estimation unit 11 reads the learned first model NN1 stored in the first memory 15A of the memory 15 described below and executes the estimation process.

Here, the estimation process by the first estimation unit 11 will be described in more detail with reference to Figs. 4 and 5. For example, image i shown in Figs. 4 and 5 shows an image of multiple ingredients arranged in a food container acquired by the acquisition unit 10, and four types of ingredients "a", "b", "c", and "d" are arranged as toppings for the food "hiyashi chuka". More specifically, the food "hiyashi chuka" has predetermined topping ingredients such as "ham" as ingredient "a", "egg" as ingredient "b", "cucumber" as ingredient "c", and "tomato" as ingredient "d" arranged and served on top of the noodles of the hiyashi chuka in the container. In addition, the number and amount of each of these ingredients are determined, and it is preferable that the number and amount of ingredients for each product manufactured on the production line are constant.

The output image p shown in Figures 4 and 5 is an image associated with image i, in which the regions of each ingredient instance contained in image i are extracted, and the regions "a1", "b1", "c1", and "d1" and the ingredient classes "a", "b", "c", and "d" are estimated.

As shown in FIG. 4, the first estimation unit 11 performs estimation processing using image i as input, and outputs image p in which the regions "a1", "b1", "c1", and "d1" for each ingredient instance are shown in a color pattern. For example, ingredient types (classes) are distinguished by color patterns of any color. When there are multiple regions of the same type of ingredient instance, these multiple regions are shown, for example, in a color pattern of the same color. The first estimation unit 11 can detect the number of each type of ingredient based on the number of ingredient regions shown in a color pattern of the same color or design. If the number of regions for a particular type of ingredient is zero, this indicates that the particular ingredient does not exist.

Returning to FIG. 1, the first judgment unit 12 judges, based on the first estimation result by the first estimation unit 11, whether or not the corresponding area for each of the multiple types of ingredients satisfies a first criterion set in advance regarding the presence or absence of ingredients and the surplus or deficiency of ingredients.

For example, the first judgment unit 12 can identify at least one of the number and size of regions for each of multiple types of ingredients and judge whether or not at least one of the number and size of regions satisfies a first criterion.

The first criterion is set as a criterion for determining whether a specific ingredient is present, whether the ingredient is arranged in a specific number, and whether the ingredient is present in a specific amount.

The first criterion is set based on the characteristics of each type of ingredient, such as the shape and size, so as to make it possible to more appropriately determine whether an ingredient is present or absent, and whether there is an excess or deficiency. Thus, if it is preferable to determine whether there is an excess or deficiency based only on the number of ingredients, a reference value for the number can be used. Alternatively, if it is more appropriate to determine whether there is an excess or deficiency based on the size of the area of the ingredient, a reference value related to the size of the area can be used. Similarly, a determination can also be made based on a combination of these. In this way, the first criterion may include multiple criteria.

For example, the first judgment unit 12 can use the area value as the first criterion for the size of the area for each type of ingredient, assuming that the absolute area of the area of the ingredient extracted in the image corresponds to the amount of the ingredient. The area value can be expressed as the number of pixels of the area in the image. Alternatively, as described below, the first criterion can be set to the sum of the areas of the estimated areas for each type of ingredient, representing the amount of the ingredient.

In addition, when one type of ingredient is composed of multiple ingredients, the first judgment unit 12 can use as a first criterion whether the total area of one type of ingredient, including each region, exceeds a certain area to judge whether there is an excess or deficiency of the same type of ingredient. An example is shredded cheese, which is placed in a certain amount in a certain area as a pizza topping. For example, when the area surrounding each region of shredded cheese exceeds a certain area and the number of shredded cheese pieces exceeds a certain number, the first judgment unit 12 can judge that the cheese ingredient meets the first criterion and there is no excess or deficiency of the cheese ingredient.

For example, as shown in area B in FIG. 5, the first criterion may be a threshold value for the number of ingredients that is preset for each type of ingredient, and a threshold value for the area of the ingredient's area in the image. For example, in the example of ingredient a in FIG. 5, the "estimated number" estimated by the first estimation unit 11 is "2", and the "estimated area", which is the absolute area of area a1 in ingredient a, is "100" pixels. In addition, the first criterion set for the number, the "number threshold", is "3", and the first criterion set for the area, the "area threshold", is in the range of "110-200" pixels.

The first judgment unit 12 judges whether the number of each estimated ingredient meets a first criterion, and whether the area of each estimated ingredient's region meets a first criterion. In the example of FIG. 5, the first judgment unit 12 judges that the number of estimated ingredient a does not meet the "number threshold" of "3" ("NG") as shown in the item "number judgment". Furthermore, the first judgment unit 12 judges that the area of region a1 in estimated ingredient a does not meet the "area threshold" of "110-200" pixels ("NG") as shown in the item "area judgment".

The first judgment unit 12 judges whether the number of ingredients a meets the first criterion and the area of area a1 meets the first criterion, for example, as shown in the "inspection items" shown in area B of FIG. 5. Note that, for example, for ingredients b and c, only the areas of areas b1 and c1 are taken into consideration, and for ingredient d, only the number is taken into consideration in judging the presence or absence of the ingredient and whether there is an excess or shortage.

The inspection unit 13 inspects the arrangement of multiple ingredients placed in a container or the like based on the judgment result by the first judgment unit 12. For example, if the judgment by the first judgment unit 12 regarding the presence or absence and excess or deficiency of all types of ingredients "a", "b", "c", and "d" placed in a food container or the like is positive, the inspection unit 13 can output an inspection result indicating that the inspection of the ingredient arrangement has passed. On the other hand, if a negative judgment result is obtained for even one of the multiple types of ingredients, the inspection unit 13 can output an inspection result indicating that the inspection of the ingredient arrangement has failed.

For example, in the item "Inspection Result" in area B of FIG. 5, the judgment results regarding the presence or absence and excess or deficiency of ingredients a and c among ingredients a to d do not meet the first criterion, so the final inspection result regarding the placement of ingredients, i.e., the food presentation inspection result, is a failure "NG."

The presentation unit 14 presents at least the inspection results obtained by the inspection unit 13. For example, the presentation unit 14 can include a display device 108 that displays the inspection results. As shown in FIG. 5, for example, the presentation unit 14 can display, in area A of the display screen of the display device 108, an image i of a plurality of ingredients photographed by the camera 105, an output image p showing the first estimation result obtained by the first estimation unit 11, and an image s obtained by combining the image i and the output image p in which the estimation result is distinguished by a color pattern.

In addition, the presentation unit 14 can display in area B on the display screen of the display device 108 values corresponding to each of the following items: "Type: chilled Chinese noodles," "Ingredient" type of food, "Estimated number," "Estimated area," "Number threshold," "Area threshold," "Number judgment," "Area judgment," "Test item," "Judgment result," and "Test result."

In addition, if the presentation unit 14 can present the test results obtained by the test unit 13 to the user of the test device 1, it can present the information not only by displaying the screen, but also by voice or other forms of information.

The memory 15 stores images acquired by the acquisition unit 10 from the camera 105, the images including ingredients placed in food containers, etc. The memory 15 also stores the first estimation result by the first estimation unit 11, the first criterion set in advance to be used by the first judgment unit 12, etc.

The memory 15 includes a first memory 15A. The first memory 15A stores a trained first model NN1, which is an image recognition model based on instance segmentation that has been previously trained and constructed on an external server or the like, and a first criterion such as threshold information used for making judgments using the results. For example, the first memory 15A can store a trained first model NN1 for each food variety in advance. In this way, by storing trained first models NN1 and first criteria corresponding to each of a plurality of foods in advance, even when different foods are manufactured on the same production line, it is possible to handle the plating inspection of different foods by switching the trained first model NN1 used by the inspection device 1.

[Hardware configuration of inspection device]
Next, an example of a hardware configuration for realizing the inspection device 1 according to the present embodiment will be described with reference to the block diagram of FIG.

As shown in FIG. 2, the inspection device 1 can be realized by, for example, a computer including a processor 102, a main memory device 103, a communication interface (I/F) 104, an auxiliary memory device 106, and an input/output I/O 107 connected via a bus 101, and a program that controls these hardware resources. The inspection device 1 also connects to an external camera 105 via the bus 101. The inspection device 1 can also include a display device 108 connected via the bus 101.

The main memory device 103 stores in advance programs that allow the processor 102 to perform various controls and calculations. The processor 102 and the main memory device 103 realize the various functions of the inspection device 1, such as the acquisition unit 10, the first estimation unit 11, the first judgment unit 12, and the inspection unit 13 shown in FIG. 1.

The communication I/F 104 is an interface circuit for network connection between the inspection device 1 and various external electronic devices. The acquisition unit 10 and the presentation unit 14 shown in FIG. 1 are realized by the communication I/F 104. The communication I/F 104 may receive the learned first model NN1 used by the first estimation unit 11 described in FIG. 1 and threshold information used for judgment using the result from the learned first model NN1 from an external terminal via the network NW or may be acquired using a storage medium such as a memory card. The communication I/F 104 may also receive the preset criteria used by the first judgment unit 12 and the inspection unit 13 described in FIG. 1 from an external terminal via the network NW or may be acquired using a storage medium such as a memory card.

The camera 105 can convert the optical signal into an image signal to generate video and still images. Video and still images captured by the camera 105, which include ingredients placed in a food container or the like, are used as input images for the trained first model NN1, and the processor 102 detects the type of ingredient for each instance at the pixel level in the image by instance segmentation.

The auxiliary storage device 106 is composed of a readable/writable storage medium and a drive for reading and writing various information such as programs and data from the storage medium. The auxiliary storage device 106 can use semiconductor memory such as a hard disk or flash memory as the storage medium.

The auxiliary storage device 106 has a program storage area for storing the inspection program executed by the inspection device 1. The auxiliary storage device 106 also has an area for storing programs for performing estimation processes such as detecting ingredients by instance segmentation, judgment processes based on the estimation results, and inspection processes related to food presentation. The auxiliary storage device 106 realizes the memory 15 and first memory 15A described in FIG. 1. The auxiliary storage device 106 also stores the first criterion, such as the learned first model NN1 and threshold information used for judgments using the results of the learned model. Furthermore, the auxiliary storage device 106 may have, for example, a backup area for backing up the above-mentioned data, programs, etc.

The input/output I/O 107 is composed of I/O terminals that input signals from external devices and output signals to external devices.

The display device 108 is configured with a liquid crystal display or the like. The display device 108 can also realize the presentation unit 14 described in FIG. 1.

The program stored in the program storage area of the auxiliary storage device 106 may be a program that is processed chronologically in accordance with the sequence of the inspection method described in this specification, or may be a program that is processed in parallel or at the required timing, such as when called. In addition, the program may be processed by one computer, or may be processed in a distributed manner by multiple computers.

[Operation of the inspection device]
Next, the operation of the inspection device 1 having the above-mentioned configuration will be described in detail with reference to the flowchart in Fig. 3 and the explanatory diagram in Fig. 4. The first memory 15A stores a trained first model NN1 obtained by a learning process performed in advance on an external server (not shown) and a first criterion such as threshold information used for judgment using the result of the learning process, and the following process is executed.

First, when the inspection is started, the first estimation unit 11 loads the trained first model NN1 from the first memory 15A (step S1). For example, the trained first model NN1 loaded in step S1 has an instance segmentation architecture represented by Mask R-CNN, DeepMask, FCIS, and the like. For example, as shown in FIG. 4, the trained first model NN1 is a model that has been trained to identify ingredients a, b, c, and d arranged in a food container, etc., for each pixel, estimate the ingredient instance contained in the input image i, extract the areas a1, b1, c1, and d1 of the ingredient instance, and estimate the ingredients a, b, c, and d for each area (first feature amount).

Next, the acquisition unit 10 acquires an image including multiple ingredients captured by the camera 105 (step S2). As shown in FIG. 4, the image i acquired in step S2 is, for example, an image captured from above, which makes it easier to grasp the state of ingredients a, b, c, and d arranged or plated in food containers or the like produced on a production line.

Next, the first estimation unit 11 performs calculations on the trained first model NN1 loaded in step S1 using the image acquired in the second step as input (step S3). Next, the first estimation unit 11 outputs the first estimation result, which is the calculation result of the trained first model NN1 (step S4).

Specifically, the first estimation unit 11 uses the trained first model NN1 to estimate the regions a1, b1, c1, d1 of multiple ingredients and the types a, b, c, d of ingredients in the image acquired in step S2, and outputs a first estimation result including an output image p in which the estimated regions and types of multiple ingredients are associated with the input image i. The output image p is a color pattern image in which the corresponding regions for each type of multiple ingredients are painted in an arbitrary color.

In other words, the output image included in the first estimation result includes pixel information and position information within the image. In this embodiment, because this type of instance segmentation is adopted as the first model NN1, it is possible to distinguish and detect ingredients by instance even if the image contains multiple ingredients of the same type.

Next, the first judgment unit 12 judges, for each of the multiple types of ingredients, whether the corresponding area satisfies a first preset criterion regarding the presence or absence of ingredients and the excess or deficiency of ingredients, based on the first estimation result output in step S4 (step S5). For example, the first judgment unit 12 can perform threshold processing on the estimated number of each ingredient, that is, the number of areas or color patterns of the same type of ingredient. The first judgment unit 12 can also perform threshold processing on the area of the ingredient area within the image, estimated for each type of ingredient.

Next, the inspection unit 13 inspects the arrangement of multiple ingredients placed in a food container or the like based on the judgment results regarding the presence or absence and surplus/deficiency of each ingredient obtained in step S5 (step S6). For example, if in step S5 the inspection unit 13 finds that all types of ingredients meet the first criteria regarding the number and area, it can output an inspection result indicating that the inspection of the arrangement of multiple ingredients placed in a food container or the like has passed. As described above, in this embodiment, the inspection of the arrangement of ingredients placed in a container or the like includes an inspection of the food presentation regarding the type of each ingredient, its presence or absence, number, amount, etc.

Next, the presentation unit 14 presents the inspection results obtained in step S6 (step S7). For example, as shown in FIG. 5, the presentation unit 14 can display the estimated number and estimated area of each of ingredients a, b, c, and d calculated in step S4 in a table format in area B of the display screen of the display device 108. The presentation unit 14 can also display values indicating the judgment results in step S5 regarding the number and area in a table provided in area B. As described above, the presentation unit 14 displays the original image i, the output image p in which the areas of the ingredient instances are distinguished by a color pattern, and the image s obtained by combining the original image i and the output image p side by side in area A. In this way, by displaying the input image, the output image, and the image obtained by combining these side by side together with the inspection results, the user can more intuitively grasp the results of the food presentation inspection.

As described above, according to the first embodiment, based on an image containing ingredients arranged in a food container or the like, the type of ingredient is estimated for each pixel in the image, and an area is extracted for each ingredient instance, making it easier to inspect food presentation using image recognition.

In addition, according to the first embodiment, an object detection method related to instance segmentation is adopted among image recognition techniques, so that the type, number, and amount of ingredients can be easily estimated for each instance in the image even if the presence or shape of each ingredient is not consistent, such as differences in color, shade, brightness, etc.

In addition, according to the first embodiment, when taking images of ingredients placed in food containers or the like, there is less need to create separate shooting environments for each type of food, and the inspection device 1 and camera 105 have a simpler configuration, making them easy to incorporate into food production lines. Furthermore, even when different foods are produced on the same production line, inspection is possible by switching between the trained first model NN1 prepared for each food. Furthermore, the inspection device 1 can contribute to labor savings by performing food presentation inspection, which was previously performed visually.

In the embodiment described above, the camera 105 is installed externally, but the acquisition unit 10 may also be configured to include a camera.

In the embodiment described above, all the functional units are provided in one inspection device 1. However, the functional units of the inspection device 1 can be realized as a single computer or can be distributed over a network.

[Second embodiment]
Next, a second embodiment of the present invention will be described. In the following description, the same components as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

In the first embodiment, a trained first model NN1 is used to determine whether a predetermined first criterion is met regarding the presence or absence and surplus/deficiency of each type of multiple ingredients placed in a food container or the like. In addition, in the second embodiment, a trained second model NN2 is further used to estimate the relative positional relationship between multiple types of ingredients.

[Functional block of the inspection device]
6 is a functional block diagram showing the configuration of an inspection device 1A according to the present embodiment. The inspection device 1A according to the present embodiment includes an acquisition unit 10, a first estimation unit 11, a first judgment unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, and a second judgment unit 17. The inspection device 1A according to the present embodiment differs from the inspection device 1 according to the first embodiment in that the inspection device 1A further includes a second memory 15B, a second estimation unit 16, and a second judgment unit 17. The following description will focus on the configuration different from the first embodiment.

The second memory 15B stores the trained second model NN2 that has been constructed by a prior learning process on an external server or the like. More specifically, the second memory 15B stores the trained second model NN2 and a second criterion (described below) that is a judgment criterion for the results of the calculations of the trained second model NN2.

The trained second model NN2 is a model in which features (second features) that identify the relative positional relationships between the multiple types of ingredients being inspected have been trained, but it may also be a neural network model such as machine learning, particularly deep learning. By using such a trained second model NN2, it is possible to classify the presentation positions of ingredients in an image of an unknown food. In this embodiment, two learning models, the trained first model NN1 stored in the first memory 15A and the trained second model NN2 stored in the second memory 15B, are used as a pair to inspect the presentation of the same food item.

Figure 7 is a diagram for explaining an overview of the estimation process for the placement of ingredients using the trained second model NN2, which is performed by the second estimation unit 16 and the second judgment unit 17 according to this embodiment. The images i shown on the left side of Figure 7(a) and Figure 7(b) each show an example of a food image acquired by the acquisition unit 10. For example, assume that the positions of the ingredients in Figure 7(a) (b is placed next to a, and c is placed next to b) are the correct placement relationship. In contrast, in Figure 7(b), the positions of ingredients a and b are reversed. This embodiment detects errors in the placement relationship between ingredients such as that shown in Figure 7(b).

As can be seen from image p shown in the center of Figures 7(a) and 7(b), the output image included in the first estimation result obtained by the first estimation unit 11 is a color pattern image in which the type (class) of ingredient is labeled on a pixel-by-pixel basis and the area for each ingredient instance is distinguished by a color pattern. The pre-constructed trained second model NN2 is a classifier that learns the features of the color pattern image for each ingredient instance obtained by the first estimation unit 11, i.e., the relative positional relationship between ingredients, and when an unknown color pattern image is input, extracts the features of the image and classifies the accuracy of the relative positional relationship between ingredients.

Examples of the second model NN2 include two-class classification, which learns and classifies good and bad products, and one-class classification, which learns and classifies only good products like an autoencoder.

Returning to FIG. 6, the second estimation unit 16 uses a trained second model NN2 that receives as input the output image included in the first estimation result by the first estimation unit 11 to estimate the relative positional relationship between multiple types of ingredients in the output image, and outputs a second estimation result that associates the estimated positional relationship with the output image from the first estimation unit 11.

The second estimation unit 16 uses as input the output image in which a color pattern is displayed for each region of the ingredient instance included in the first estimation result. The second estimation unit 16 also extracts features that indicate the arrangement relationship, i.e., the relative positions between multiple types of ingredients, from the output image of the input color pattern, and probabilistically classifies the accuracy of the arrangement relationship between the multiple types of ingredients from the features. Note that the relative arrangement relationship between ingredients estimated by the second estimation unit 16 may not only be the arrangement relationship between multiple ingredients of different types, but may also be the relative arrangement relationship between individual ingredients when one type of ingredient is composed of multiple ingredients, depending on the settings.

The second judgment unit 17 judges whether or not a preset second criterion is satisfied regarding the validity of the relative arrangement of multiple types of ingredients placed in a food container or the like, based on the second estimation result by the second estimation unit 16. For example, the second judgment unit 17 can set a preset threshold value for the classification result shown as a probability and judge the accuracy of the estimated arrangement.

In addition, the second criterion for determining the appropriateness of the relative arrangement when one type of ingredient is composed of multiple ingredients can be set as a criterion for determining whether the multiple ingredients that make up the same type of ingredient are arranged in a certain way. By using such a second criterion, the second judgment unit 17 can determine whether the relative arrangement of the multiple ingredients that make up the same type of ingredient is appropriate.

For example, the following describes shredded cheese placed as a pizza topping. For example, it is preferable that multiple pieces of shredded cheese are evenly spaced apart from one another on the pizza dough. In this case, the second criterion is set to determine whether the relative arrangement of the multiple pieces of shredded cheese is appropriate, in other words, whether the arrangement of the shredded cheese satisfies a criterion, or whether the degree of scattering of the shredded cheese placed as a topping is uniform. For example, based on such a second criterion, the second judgment unit 17 can detect whether each piece of shredded cheese is placed in a predetermined position, and further whether the shredded cheese is unevenly placed in a particular position.

The inspection unit 13 performs an inspection of the arrangement of multiple ingredients placed in a food container or the like based on the judgment results from the first judgment unit 12 and the judgment results from the second judgment unit 17. In this embodiment, the inspection unit 13 performs a final food presentation inspection based on the judgment results from the first judgment unit 12 regarding the presence or absence of ingredients and the excess or deficiency of ingredients, as well as the judgment results from the second judgment unit 17 regarding the arrangement of ingredients.

The presentation unit 14 presents the inspection results from the inspection unit 13. More specifically, as shown in FIG. 10, the presentation unit 14 displays the "estimated number" and "estimated area" for each ingredient by the first estimation unit 11, and the "number threshold" and "area threshold" used by the first judgment unit 12 in area B displayed on the display screen of the display device 108. Furthermore, the presentation unit 14 displays in area B the values of the judgment results for each threshold, "number judgment" and "area judgment", and the value of the inspection result item "judgment result" based on the "inspection item".

In addition, the presentation unit 14 displays the second estimation result "Probability (0.999)" by the second estimation unit 16 and the judgment result "NG" regarding the validity of the relative positional relationship between the ingredients by the second judgment unit 17 in the item "Location inspection result" of the table provided in area B.

Furthermore, the presentation unit 14 displays in area B the final inspection result ("Inspection result") regarding the arrangement of multiple ingredients based on the judgment result by the first judgment unit 12 regarding the presence or absence of ingredients and the surplus or deficiency (item "Judgment result") and the judgment result by the second judgment unit 17 regarding the appropriateness of the arrangement relationship between the ingredients ("Location inspection result").

The presentation unit 14 can also display, side by side in the area A, the image i of the inspection object acquired by the acquisition unit 10, the output image p of the color pattern for each type of food ingredient including the first estimation result by the first estimation unit 11, and an image s obtained by combining the image i of the inspection object and the output image p.

[Operation of the inspection device]
Next, the operation of the inspection device 1A having the above-mentioned configuration will be described with reference to the flowchart in Fig. 8 and the explanatory diagram in Fig. 9. It is assumed that the learning process of the first learning model NN1 and the second learning model NN2 is performed in advance by an external server or the like. In addition, the first memory 15A stores a first standard including the learned first learning model NN1 constructed in advance and threshold value information used in the judgment process for the calculation result thereof. Similarly, it is assumed that the second memory 15B stores a second standard including the learned second learning model NN2 and threshold value information used in the judgment process for the calculation result thereof.

First, the first estimation unit 11 loads the trained first model NN1 from the first memory 15A (step S10). Next, the second estimation unit 16 loads the trained second model NN2 from the second memory 15B (step S11).

Next, the acquisition unit 10 acquires an image captured by the camera 105 that includes multiple ingredients arranged in a food container or the like (step S12). Next, the first estimation unit 11 performs calculations using the trained first model NN1 that was loaded in step S10 (step S13).

Specifically, as shown in FIG. 9, the first estimation unit 11 uses the trained first model NN1 to estimate the regions a1, b1, c1, d1 of multiple ingredients and the types a, b, c, d of ingredients in the image i acquired in step S10, and outputs a first estimation result including an output image p in which the estimated regions and types of multiple ingredients are associated with the input image i. The output image p is a color pattern image in which the corresponding regions for each type of multiple ingredients are painted in an arbitrary color.

In other words, the output image included in the first estimation result includes pixel information and position information within the image. In this embodiment, because this type of instance segmentation is adopted as the first model NN1, it is possible to distinguish and detect ingredients by instance even if the image contains multiple ingredients of the same type.

Next, the first estimation unit 11 inputs the first estimation result, which is the calculation result of the trained first model NN1, to the second estimation unit 16 (step S14). More specifically, as shown in step S13 of FIG. 9, the first estimation unit 11 inputs the output image p, in which a color pattern is assigned to each region of the food ingredient instance, to the second estimation unit 16. The first estimation result by the first estimation unit 11 is also passed to the first judgment unit 12.

Next, the second estimation unit 16 performs calculations using the trained second model NN2 loaded in step S11, and passes the calculation results to the second judgment unit 17 (step S15). More specifically, the second estimation unit 16 uses the trained second model NN2, which uses as input the output image included in the first estimation result, to estimate the positional relationship between multiple types of ingredients in the output image, and outputs a second estimation result that associates the estimated positional relationship with the output image.

Next, the first judgment unit 12 judges whether the first criteria set in advance for the presence or absence and surplus/deficiency of each type of ingredient are met based on the first estimation result obtained by the calculation of the trained first model NN1 in step S13 (step S16). More specifically, the first judgment unit 12 reads from the first memory 15A the first criteria, such as judgment items and thresholds for each type of ingredient that are stored in the first memory 15A in association with the trained first model NN1, and performs the process of step S16.

For example, the first judgment unit 12 can perform threshold processing on the estimated number of each ingredient, that is, the number of areas or color patterns of the same type of ingredient. The first judgment unit 12 can also perform threshold processing on the area that the ingredient area occupies within the image, estimated for each type of ingredient.

Next, the second judgment unit 17 judges whether the mutual arrangement of the multiple types of ingredients satisfies a second criterion set in advance regarding the validity of the arrangement based on the second estimation result obtained in step S15 (step S17). More specifically, the second judgment unit 17 reads from the second memory 15B the second criterion including a threshold value etc. stored in the second memory 15B in association with the trained second model NN2, and performs the process of step S17.

For example, in area B of FIG. 10, as indicated by the value of the item "Location inspection result," the position of the ingredients is determined to be incorrect ("NG") based on the probability indicating whether the relative placement relationship between ingredients estimated by the second estimation unit 16 is correct.

Next, the inspection unit 13 inspects the arrangement of multiple ingredients placed in a food container, etc., based on the judgment results obtained in step S16 regarding the presence or absence and surplus/deficiency of each ingredient, and the judgment results obtained in step S17 regarding the appropriateness of the arrangement relationship between multiple types of ingredients (step S18).

For example, if in step S16 all types of ingredients meet the first criteria related to the number and area, and in step S17 the arrangement of the multiple types of ingredients meets the second criterion, the inspection unit 13 can output an inspection result indicating that the inspection of the arrangement of multiple ingredients arranged in a food container or the like has passed. In this embodiment, the inspection of the arrangement of ingredients arranged in a container or the like includes an inspection of the food presentation regarding the type, presence, number, amount, and arrangement position of each ingredient.

Next, the presentation unit 14 presents the inspection results regarding the arrangement of ingredients obtained in step S18 (step S19). For example, as shown in FIG. 10, the presentation unit 14 can display the estimated numbers and estimated areas of each of ingredients a, b, c, and d in a table format in area B of the display screen of the display device 108. The presentation unit 14 can also display the judgment results based on the comparison of the numbers and areas with threshold values in the table provided in area B.

Furthermore, the presentation unit 14 can display, in the table in area B, the second estimation result by the second estimation unit 16 and the judgment result by the second judgment unit 17 regarding the validity of the arrangement relationship between multiple types of ingredients. The presentation unit 14 can also display, in the table in area B, the results of the comprehensive food presentation inspection performed by the inspection unit 13.

As described above, the presentation unit 14 displays the image i captured by the camera 105, the output image p of the color pattern, and the image s obtained by combining the image i and the output image p of the color pattern side by side in the area A. The presentation unit 14 also displays an image of the food in which ingredients are placed in the correct positions as a reference image side by side in the area A, allowing the user to more intuitively understand the inspection results regarding the placement of ingredients.

Now, referring to FIG. 11, we will explain the processing time when inspecting food presentation using the inspection device 1A according to this embodiment. The horizontal axis of FIG. 11 indicates the data interval [msec], and the vertical axis indicates the frequency, which is the number of foods to be inspected. The processing time is recorded as the time from when an image of the manufactured food is taken by the camera 105, to the estimation process using instance segmentation by the first estimation unit 11, classification by the second estimation unit 16, and further, to the saving of the image and each numerical value. The accuracy of the instance segmentation and the classifier has been designed in advance to be sufficient for practical use.

The inspection time for the first meal, shown at the far right of the horizontal axis in Figure 11, is approximately 700 msec. It can also be seen that the processing time for the second meal onwards is 400 msec. This shows that there is ample processing time available when the takt time in a food production line is, for example, 1.0 sec. In this way, it can be seen that the inspection device 1A of this embodiment has a simpler configuration and can provide sufficient processing time while maintaining sufficient inspection accuracy.

As described above, according to the second embodiment, the first estimation result by the first estimation unit 11 is used as an input to the classifier of the second estimation unit 16, making it possible to perform inspection of the arrangement of ingredients using image recognition with a simpler configuration.

In addition, according to the second embodiment, a single inspection device 1A can inspect not only the presence, number, and amount of ingredients placed in a food container, etc., but also the relative positions of multiple types of ingredients placed in the container, allowing for more efficient and effective inspection of the placement of ingredients.

Furthermore, according to the second embodiment, it is possible to inspect whether the food ingredients are placed correctly, regardless of the shape or material of the container in which the food ingredients are placed.

[Third embodiment]
Next, a third embodiment of the present invention will be described. In the following description, the same components as those in the first and second embodiments are given the same reference numerals, and the description thereof will be omitted.

In the second embodiment, a case was described in which estimation processing is performed using a trained first model NN1 and a trained second model NN2 that have been constructed in advance through learning processing on an external server or the like. In contrast, in the third embodiment, a learning device is further provided that has a first learning unit 18 that learns the first model NN1 and a second learning unit 19 that learns the second model NN2, and each learning processing is performed within the device itself.

[Functional block of the inspection device]
12 is a block diagram showing the configuration of an inspection device 1B according to the present embodiment. The inspection device 1B includes an acquisition unit 10, a first estimation unit 11, a first judgment unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second judgment unit 17, a first learning unit 18, and a second learning unit 19. The inspection device 1B differs from the configuration of the second embodiment in that it further includes the first learning unit 18 and the second learning unit 19. The following description will focus on the configuration that differs from the first and second embodiments.

The first learning unit 18 uses an image containing ingredients placed in a food container or the like as a learning image (first learning image) to train a first model of the neural network, extracts a first feature that identifies the area and type of the ingredient, and obtains a trained first model NN1 from the extracted first feature. The trained first model NN1 obtained by the first learning unit 18 is stored in the first memory 15A.

More specifically, the first learning unit 18 uses a neural network model having an instance segmentation architecture such as Mask R-CNN as the first model NN1. The first learning unit 18 identifies multiple ingredients a, b, c, and d included in the learning image for each pixel, and learns features that allow the types of ingredients a, b, c, and d to be estimated in the areas of the ingredient instances included in the learning image.

For example, as shown in FIG. 9, learning images for each of ingredients a, b, c, and d are given as input to the first model NN1. The first learning unit 18 performs learning calculations for the first model NN1 and learns the characteristics of ingredients a, b, c, and d. The number of learning images used in the learning process is a number that ensures sufficient accuracy in the subsequent inspection process, and for example, 10 images each of ingredients a, b, c, and d photographed by the camera 105 may be cut out and used as learning images. Note that images acquired in advance by the acquisition unit 10 may also be used as learning images.

The second learning unit 19 uses the output image included in the estimation result by the first estimation unit 11 as a learning image (second learning image) to learn a second model using a neural network, extracts second features that identify the relative positional relationship between ingredients in the learning image, and obtains a learned second model NN2 from the extracted second features. The learned second model NN2 obtained by the second learning unit 19 is stored in the second memory 15B.

More specifically, the second learning unit 19 learns the output image of the color pattern by instance segmentation and constructs a trained second model NN2 that outputs the correctness of the positional relationship between ingredients as a classification result. Examples of the second model NN2 include two-class classification that learns and classifies good and bad products, and one-class classification that learns and classifies only good products like an autoencoder.

As will be described in detail in the sixth embodiment below, for example, the second learning unit 19 can secure the required number of learning images by duplicating or processing a certain number of output images output by the first estimation unit 11 performing calculations of the learned first model NN1. As the learning images, images in which color patterns representing multiple types of ingredients a, b, c, and d are correctly positioned relative to each other and images in which they are incorrectly positioned, as shown in FIG. 9, are used. Images in which ingredients are incorrectly positioned can be created by processing the output images included in the first estimation results.

The first learning unit 18 and the second learning unit 19 can construct a learned first model NN1 and a learned second model NN2 for each of a number of different food varieties and store them in the memory 15 (first memory 15A, second memory 15B). The first memory 15A also stores a first criterion, such as a threshold value, associated with the learned first model NN1 and used for the calculation results of the learned first model NN1. Similarly, the second memory 15B stores a second criterion, such as a threshold value, associated with the learned second model NN2 and used for the calculation results of the learned second model NN2.

In this embodiment, the first learning unit 18, the second learning unit 19, the first memory 15A, and the second memory 15B constitute a learning device that performs a learning process to construct a preset model.

[Operation of the inspection device]
Next, the operation of the inspection device 1B having the above-mentioned configuration will be described with reference to the flowchart of FIG.

First, the acquisition unit 10 acquires an image captured by the camera 105, which includes multiple ingredients arranged in a food container or the like, as a learning image (step S101). Next, the first learning unit 18 uses the learning image acquired in step S101 to learn a first model of a neural network that realizes a preset instance segmentation, extracts a first feature that identifies the area and type of ingredients for each pixel in the learning image, and acquires a learned first model NN1 from the extracted first feature (step S102). Next, the learned first model NN1 obtained in step S102 is stored in the first memory 15A (step S103). More specifically, in step S103, a first criterion used in a judgment process (step S111 described later) for the calculation result of the learned first model NN1 is also stored in the first memory 15A.

Next, the second learning unit 19 acquires an output image from the trained first model NN1 as a learning image (step S104). For example, the second learning unit 19 can acquire an output image output by an estimation process using the trained first model NN1 constructed in advance.

The second learning unit 19 uses the acquired learning images to train a second model NN2 based on a neural network, extracts second features from the learning images that identify the relative positional relationships between multiple types of ingredients, and acquires a trained second model NN2 based on the extracted second features (step S105).

Then, the trained second model NN2 obtained in step S105 is stored in the second memory 15B (step S106). More specifically, in step S106, a second criterion to be used in the judgment process for the calculation result of the trained second model NN2 (step S112 described below) is also stored in the second memory 15B.

The acquisition unit 10 acquires an image for inspection captured by the camera 105 (step S107). Next, the first estimation unit 11 reads the trained first model NN1 stored in the first memory 15A in step S103, and performs calculations of the trained first model NN1 using the image acquired in step S107 as input (step S108).

In addition, in step S108, the first estimation unit 11 uses the trained first model NN1 to estimate the areas and types of ingredients in the image acquired in step S107, and outputs a first estimation result including an output image in which the estimated areas and types of ingredients are associated with the input image. The output image is a color pattern image in which the areas corresponding to each type of ingredient are painted in an arbitrary color.

The output image contained in the first estimation result includes pixel information and position information. In this embodiment, instance segmentation is adopted as the first model NN1, so even if multiple ingredients of the same type are served in the image, it is possible to distinguish and detect each instance of the ingredient.

Next, the first estimation unit 11 inputs the first estimation result to the second estimation unit 16 (step S109). The first estimation unit 11 outputs the first estimation result including an output image in which a color pattern is assigned to each region of the ingredient. This first estimation result is used as an input image for the second estimation unit 16. The first estimation result is also passed to the first judgment unit 12.

Next, the second estimation unit 16 reads the trained second model NN2 stored in the second memory 15B in step S106, provides the output image included in the first estimation result to the trained second model NN2, performs calculations on the trained second model NN2, and passes the calculation results to the second judgment unit 17 (step S110).

More specifically, the second estimation unit 16 uses a trained second model NN2 that uses the output image included in the first estimation result as input to estimate the relative positions of multiple types of ingredients in the output image, and outputs a second estimation result that associates the estimated relative positions with the output image.

Next, the first judgment unit 12 judges whether the first criteria set in advance for the presence or absence and surplus/deficiency of each type of ingredient are satisfied based on the first estimation result obtained by the calculation of the trained first model NN1 in step S13 (step S111). More specifically, the first judgment unit 12 reads from the first memory 15A the first criteria including thresholds and the like stored in the first memory 15A in association with the trained first model NN1, and performs the judgment process of step S111.

For example, the first judgment unit 12 can perform threshold processing on the estimated number of each ingredient, that is, the number of areas or color patterns of the same type of ingredient. The first judgment unit 12 can also perform threshold processing on the area that the ingredient area occupies within the image, estimated for each type of ingredient.

Next, the second judgment unit 17 judges whether the arrangement relationship between the multiple types of ingredients satisfies a second criterion set in advance regarding the validity of the arrangement relationship based on the second estimation result obtained in step S110 (step S112). More specifically, the second judgment unit 17 reads from the second memory 15B the second criterion including a threshold value etc. stored in the second memory 15B in association with the trained second model NN2, and performs the judgment process of step S112.

Next, the inspection unit 13 inspects the arrangement of multiple ingredients placed in a food container or the like based on the judgment results obtained in step S111 regarding the presence or absence and surplus/deficiency of each ingredient, and the judgment results obtained in step S112 regarding the appropriateness of the arrangement relationship between the ingredients (step S113).

For example, in step S111, if all types of ingredients satisfy the first criteria related to the number and area, and in step S112, the positional relationship between the ingredients satisfies the second criterion, the inspection unit 13 can output an inspection result indicating that the inspection of the arrangement of multiple ingredients arranged in a food container or the like has passed. In this embodiment, the inspection of the arrangement of ingredients arranged in a container or the like includes an inspection of the food presentation regarding the type, presence, number, amount, and arrangement position of each ingredient.

Next, the presentation unit 14 presents the inspection results regarding the placement of ingredients obtained in step S113 (step S114).

As described above, according to the third embodiment, the inspection device 1B is equipped with a learning device, and the first learning unit 18 learns a first model NN1 that realizes instance segmentation, and the second learning unit 19 learns a second model NN2 that uses output images from the trained first model NN1 as training images to classify whether the positional relationship between ingredients is correct or incorrect. Therefore, with a simpler configuration, the processes from learning to estimation and even inspection can be performed by a single inspection device 1B.

Furthermore, according to the third embodiment, the second model NN2 uses the output image of the trained first model NN1 as an input image. Therefore, it is possible to easily prepare training images for the second model NN2, and further simplify the training process of the second model NN2.

In addition, according to the third embodiment, learning images can be easily acquired, so if the first model NN1 and the second model NN2 are constructed for each food, food presentation inspection for small quantities of many different products can be easily performed even when the food produced is small quantities of many different types or when products are frequently revised or discontinued.

Also, in the third embodiment, a simply constructed image capture environment is used to acquire learning images, and the acquired images themselves can be processed simply, making it easy to inspect food presentation using image processing.

[Fourth embodiment]
Next, a fourth embodiment of the present invention will be described. In the following description, the same components as those in the first to third embodiments are given the same reference numerals, and the description thereof will be omitted.

In the second and third embodiments, the second learning unit 19 and the second estimation unit 16 use the output image obtained by the calculation of the trained first model NN1 as the learning image and the input image for the estimation process as is. In contrast, in the fourth embodiment, the learning image and the input image used in the second learning unit 19 and the second estimation unit 16 are converted, and the simplified image is used as the learning image for the second learning unit 19.

[Functional block of the inspection device]
14 is a block diagram showing the configuration of an inspection device 1C according to the present embodiment. The inspection device 1C includes an acquisition unit 10, a first estimation unit 11, a first judgment unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second judgment unit 17, a first learning unit 18, a second learning unit 19, and a conversion unit 20. The inspection device 1C differs from the configuration of the third embodiment in that it further includes the conversion unit 20. The following description will focus on the configurations that differ from the first to third embodiments.

The conversion unit 20 converts the output image included in the first estimation result by the first estimation unit 11 to generate a more simplified converted image. For example, as shown in the example of FIG. 15, the output image from the first estimation unit 11 has a corresponding area for each type of ingredient that has the actual shape of the ingredient. Also, it is a color pattern image in which each area is painted with an arbitrary color. The conversion unit 20 converts the areas of the color pattern image into a color pattern image in which each area is replaced with an object of the same shape. For example, the conversion unit 20 can replace the shapes of areas a1, b1, c1, and d1 with objects such as circles. In this case, the image converted by the conversion unit 20 has color patterns of different colors for each area, but the same shape.

As shown in Figures 15 and 16, the color pattern image has a complex shape in which each region has the outline of an actual food ingredient instance. Therefore, the conversion unit 20 replaces each region with a simplified object that removes minute changes that do not affect boundary extraction, thereby reducing the computational load in the learning process and estimation process using the second model NN2.

As another example, as shown in FIG. 16, the conversion unit 20 can convert each region of the color pattern image of the output image from the first estimation unit 11 into an image in which the regions are replaced with objects of different shapes for each color of the color pattern, and the color patterns of the replaced objects are all the same color. In other words, the conversion unit 20 replaces regions with complex shapes with objects of simple shapes that differ for each type of ingredient. For example, a converted image can be generated in which the shape of the region is replaced with a simplified geometric shape, such as a triangle, square, circle, star, or any other shape. In one example, the converted image can be a monochrome binary image, which can reduce the data size of the converted image.

It goes without saying that the conversion unit 20 can replace each area of the color pattern image with an object of a different shape for each color of the color pattern, and can also give the replaced object a different color for each object shape.

The second learning unit 19 uses the images converted by the conversion unit 20 as learning images to learn the second model NN2 and acquire a trained second model NN2.

The second estimation unit 16 uses a trained second model NN2, which receives as input the converted image generated by the conversion unit 20, to estimate the relative positional relationship between multiple ingredients in the converted image.

As described above, according to the fourth embodiment, the conversion unit 20 generates a converted image that is a simplified version of the output image from the first estimation unit 11, and the converted image is used as a learning image to learn the second model NN2. This makes it possible to reduce the computational load and memory size in the learning process of the second model NN2 in the second learning unit 19. In addition, the computational load when calculating the learned second model NN2 in the second estimation unit 16 can be reduced.

[Fifth embodiment]
Next, a fifth embodiment of the present invention will be described. In the following description, the same components as those in the first to fourth embodiments are given the same reference numerals, and the description thereof will be omitted.

In the fourth embodiment, a converted image is generated in which the color pattern of each region of the food ingredient included in the output image from the first estimation unit 11 is converted and replaced with an object of a simpler shape. In contrast, in the fifth embodiment, an image is generated in which discontinuous parts in the color pattern included in the output image from the first estimation unit 11 are complemented to be continuous, and the image is used for training the second model NN2.

[Functional block of the inspection device]
17 is a block diagram showing the configuration of an inspection device 1D according to the present embodiment. The inspection device 1C includes an acquisition unit 10, a first estimation unit 11, a first judgment unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second judgment unit 17, a first learning unit 18, a second learning unit 19, and a processing unit 21. The inspection device 1D differs from the configuration of the fourth embodiment including the conversion unit 20 in that it includes the processing unit 21. The following description will focus on the configurations different from the first to fourth embodiments.

The processing unit 21 determines that the area of the ingredient contained in the output image of the estimation result by the first estimation unit 11 is the area of the ingredient, and generates a processed image that complements the discontinuity in the color pattern to make it continuous.

Specifically, as shown in FIG. 18, assume that the output image from the first estimation unit 11 includes discontinuous parts in the region b1 of the ingredient. For example, there may be a cavity in the shape of the ingredient itself, or gaps may form depending on how the ingredient is processed. In such cases, from the perspective of reducing the learning load on the neural network in the second learning unit 19, it is effective to clarify in advance that all areas within the region represent the estimated ingredient.

Therefore, the processing unit 21 regards the hole contained within the region b1 as a region representing the ingredient b, and generates a processed image in which the discontinuous parts within the region b1 are complemented to make them continuous by using the color pattern assigned to the region b1.

The second learning unit 19 uses the processed image, which is obtained by complementing discontinuous parts of the color pattern processed by the processing unit 21 to make it continuous, as a learning image to learn the second model NN2 and acquire the learned second model NN2.

The second estimation unit 16 performs calculations of the learned second model NN2 using as input a processed image of the output image in which the discontinuity of the color pattern generated by the processing unit 21 is complemented to make it continuous, estimates the relative positional relationship between the ingredients, and outputs a second estimation result that associates the estimated positional relationship with the output image.

As described above, according to the fifth embodiment, the processing unit 21 generates a processed image in which discontinuous parts of the color pattern in the area of the food material included in the output image from the first estimation unit 11 are complemented to make them continuous, and the second model NN2 is trained. This reduces the computational load in training the second model NN2 in the second learning unit 19, making it possible to perform training more efficiently.

Sixth embodiment
Next, a sixth embodiment of the present invention will be described. In the following description, the same components as those in the above-mentioned embodiment are given the same reference numerals, and the description thereof will be omitted.

In the above-mentioned third embodiment, a configuration was described that further includes a learning device having a first learning unit 18 that learns the first model NN1, and a second learning unit 19 that learns the second model NN2. Also, in the third embodiment, a case was described in which when the second learning unit 19 learns the second model NN2, only the output image obtained by the calculation of the trained first model NN1 is used as the learning image. In contrast, in the sixth embodiment, the learning device performs data expansion to pad the learning images used in learning the second model NN2.

Figure 20 is a diagram for explaining an overview of the data expansion process performed by the inspection device 1E according to this embodiment. The image data shown in the dashed frame DA in Figure 20 is a training data set for training the second model NN2, which classifies whether the positional relationship between ingredients is correct or incorrect. Each training image t is assigned label information ("good" or "bad") indicating whether the positional relationship between ingredients is correct or incorrect. When improving the classification accuracy of the second model NN2, it is considered effective to increase the number of training images associated with both labels indicating correct positional relationships between ingredients (hereinafter sometimes referred to as "good" or "good") and incorrect positional relationships (hereinafter sometimes referred to as "defective" or "bad") and to perform training.

However, when the inspection device 1E is used to inspect food production lines, it is relatively easy to obtain a large number of learning images of good products, but it is difficult to obtain learning images of defective products. In particular, the second model NN2 according to this embodiment is a classifier that classifies two classes, good products and defective products, and in order to ensure and improve accuracy, it is desirable to prepare a sufficient number of learning images of defective products and perform learning.

For example, Non-Patent Documents 1 to 3 disclose techniques for generating learning images by data augmentation processing. Patent Document 2 discloses a technique for identifying position and morphological information of an augmented object, performing data augmentation processing, and generating learning images in order to prevent unnatural boundaries of characteristic parts of image data. Patent Document 3 discloses a technique for processing characteristic parts of an image, performing data augmentation based on the processed data, and efficiently generating learning images.

However, the conventional technologies disclosed in Non-Patent Documents 1 to 3 and Patent Documents 2 and 3 all perform so-called data expansion of good product images to generate a large number of good product images with the same label, and it is difficult to perform data expansion based on good product images to generate a large number of defective product images with different labels.

In contrast, in this embodiment, by utilizing the unique properties of the training images of the second model NN2, it is possible to generate training images of defective products that are relatively difficult to obtain on a production line. More specifically, in this embodiment, attention is paid to the fact that the training images of the second model NN2 are images of color patterns obtained by instance segmentation, which distinguishes ingredients from one another and identifies, on a pixel-by-pixel basis, areas in the image where the ingredients are present, and that the label information of the training images is the correctness or incorrectness of the relative positional relationships between the ingredients.

Based on these points of focus, the inspection device 1E according to this embodiment performs data expansion of target images of relatively easily obtainable non-defective products, making it possible to generate a large number of learning images of defective products that are relatively difficult to obtain on an actual production line.

[Functional block of the inspection device]
19 is a block diagram showing the configuration of an inspection device 1E according to the present embodiment. The inspection device 1E includes an acquisition unit 10, a first estimation unit 11, a first judgment unit 12, an inspection unit 13, a presentation unit 14, a memory 15, a second estimation unit 16, a second judgment unit 17, a first learning unit 18, a second learning unit 19, an extraction unit 22, and a data extension unit 23. The inspection device 1E is different from the configurations of the third to fifth embodiments in that the learning device further includes the extraction unit 22 and the data extension unit 23. The following mainly describes the configurations different from the third to fifth embodiments.

The extraction unit 22 extracts regions for each type of ingredient from an image (target image) for which data expansion is to be performed. The target image is an image for which data expansion is to be performed by the data expansion unit 23 described below, and is an image to which label information relating to the accuracy of the relative positional relationship between ingredients has been added in advance. For example, an output image in which the estimated areas and types of ingredients are associated with the image, which is included in the first estimation result output by the first estimation unit 11, can be used as the target image. Alternatively, an image to which label information has been added can be obtained in advance from outside and used as the target image, such as when an image created in advance by a user, engineer, etc. is used as the target image. In this embodiment, in particular, an image to which label information has been added is used as the target image from an output image in which the areas for each ingredient are represented as a color pattern. The extraction unit 22 can also extract only the areas of ingredients related to a previously specified color pattern from the target image according to the settings.

For example, of the three images included in the folder of target image data DA0 in FIG. 21, the extraction unit 22 extracts areas a1, b1, c1, and d1 of ingredients a, b, c, and d included in image p (hereinafter sometimes referred to as "target image p") surrounded by a thick frame. Target image p has been assigned a first label "good" indicating that the relative positional relationship between the ingredients is correct. Furthermore, the information on areas a1, b1, c1, and d1 of ingredients a, b, c, and d extracted by the extraction unit 22 includes color information and coordinate information for each area.

The target image is not limited to the output image of a color pattern. For example, as described in the fourth embodiment, a converted image in which the output image of a color pattern is replaced with an object of a further simplified shape can be used. Similarly, the target image can be a processed image in which discontinuous parts in a color pattern are complemented to make them continuous, as described in the fifth embodiment.

The data expansion unit 23 performs data expansion of a target image in which the area and type of ingredient are associated with the image to generate a learning image (second learning image). More specifically, the data expansion unit 23 generates the learning image based on label information regarding the validity of the relative positional relationship between multiple ingredients that is assigned to the target image. The label information is information corresponding to the second criterion that the second judgment unit 17 uses to judge the validity of the relative positional relationship between multiple types of ingredients arranged in a food container or the like. In other words, an image that satisfies the second criterion is assigned the first label "good" in advance, and an image that does not satisfy the second criterion is assigned the second label "poor".

The data expansion unit 23 can perform data expansion of a target image to which a first label "good" has been assigned, indicating that the relative positional relationship between multiple ingredients in the target image is appropriate, to generate a learning image to which a second label "bad" has been assigned, indicating that the relative positional relationship between multiple ingredients is not appropriate. Furthermore, the data expansion unit 23 can also perform data expansion of a target image to which a first label "good" has been assigned, to generate a learning image to which the same label information, the first label "good", has been assigned. Similarly, the data expansion unit 23 can perform data expansion of a target image to which a second label "bad" has been assigned, to further generate a learning image to which a second label "bad" has been assigned.

For example, the data expansion unit 23 can generate learning images by processing the areas extracted by the extraction unit 22 while maintaining the relative positional relationship between the multiple ingredients in the target image. Specifically, as shown in data expansion DA1 in FIG. 21, while maintaining the relative positions of areas a1, b1, c1, and d1 included in the target image p to which the first label "good" surrounded by a thick frame is assigned, each of the areas a1, b1, c1, and d1 can be rotated to generate learning images with the first label "good" in consecutive numbers. In the example of data expansion DA1 in FIG. 21, the learning image shown on the left rotates area a1 of the target image p by a random number angle from the center coordinates, the learning image shown in the middle further rotates area b1, and the learning image shown on the right further rotates area c1.

In this way, the data expansion unit 23 rotates each of the regions a1, b1, c1, and d1 from one target image p to which the first label "good" has been assigned, and rearranges each region to the same position, thereby generating a large number of consecutive learning images to which the first label "good" associated with the same label has been assigned. In the data expansion DA1 shown in FIG. 21, an example is given of rotating each region, but other processing that can be performed while maintaining the positional relationship between the regions includes changing the size or shape of the regions and rearranging them in the same position.

Next, an example will be described in which data expansion is performed on a target image to which a first label "good" has been assigned, to generate a learning image to which a second label "bad" has been assigned. In this case, the data expansion unit 23 can generate a learning image by mutually replacing the positions of areas corresponding to two or more different types of ingredients among the areas extracted by the extraction unit 22.

For example, in the data extension DA2 shown in FIG. 21, the positions of regions b1 and c1 of the thick-framed target image p, which have been assigned the first label "good" in the target image data DA0, are swapped and rearranged to generate a learning image with the second label "bad" shown on the left. Similarly, the image shown in the middle is a learning image with the second label "bad" generated by swapping and rearranging regions a1 and d1 of the target image p. The image shown on the right is a learning image with the second label "bad" generated by swapping and rearranging regions b1 and d1 of the target image p.

The data expansion unit 23 calculates the number of combinations for replacing any two colors for a target image that has been assigned the first label "good" and generates images of all of these combinations, making it possible to generate a large number of learning images that have been assigned the second label "bad."

Next, another example will be described in which data expansion is performed on a target image to which a first label "good" has been assigned, to generate a learning image to which a second label "bad" has been assigned. In this case, for example, the data expansion unit 23 can generate a learning image by deleting one or more of the regions extracted by the extraction unit 22 from the target image.

Specifically, as shown in data expansion DA3 in FIG. 21, the data expansion unit 23 extracts only one of the regions a1, b1, c1, and d1 contained in the thick-framed target image p to which the first label "good" has been assigned, and deletes and rearranges the other regions to generate a learning image to which the second label "bad" has been assigned.

Furthermore, we will give another example in which data expansion is performed on a target image to which a first label "good" has been assigned, to generate a learning image to which a second label "bad" has been assigned. In this case, the data expansion unit 23 can perform data expansion of the target image by performing at least one of inversion, rotation, expansion, and contraction of each of the regions extracted by the extraction unit 22.

For example, as shown in data expansion DA4 in FIG. 21, the data expansion unit 23 can generate a learning image with the second label "bad" by vertically flipping and rearranging the bold-framed target image p with the first label "good" assigned.

In the data expansion DA5 shown in FIG. 21, the data expansion unit 23 horizontally flips and rearranges the bold-framed target image p to which the first label "good" has been assigned, thereby generating a learning image to which the second label "bad" has been assigned.

The data expansion unit 23 may further perform data expansion of the learning image to which the first label "good" or the second label "bad" has been assigned, obtained by performing data expansion of the target image, to generate a learning image to which the first label "good" or the second label "bad" has been assigned. For example, as shown in data expansion DA4 in FIG. 21, the data expansion unit 23 can generate a learning image to which the second label "bad" has been assigned by rotating the learning image to which the second label "bad" has been assigned, the learning image being obtained by mirroring the target image to which the first label "good" has been mirrored on the vertical axis, by a generated random number angle, starting from the central coordinates of the area of each color pattern.

In this way, the data expansion unit 23 can generate a desired number of learning images to which the second label "bad" has been assigned, based on a target image to which the first label "good" has been assigned. Note that when the data expansion unit 23 repeats the data expansion process to generate learning images related to defective products, learning images related to good products may be generated. However, learning images related to good products can be removed from the generated learning images by using a pre-set setting.

[Operation of the inspection device]
Next, the operation of the inspection device 1E having the above-mentioned configuration will be described with reference to Fig. 22. Fig. 22 is a flowchart showing each step from the generation of the learning image for the second model NN2 to the learning process of the second model NN2 in the inspection device 1E.

First, the learning device acquires a target image that is an image to be subjected to data augmentation (step S201). For example, the learning device can use, as the target image, an output image p of a color pattern in which the regions and types of multiple ingredients obtained as a result of the calculation of the trained first model NN1 by the first estimation unit 11 are associated with the input image i.

Next, the learning device acquires label information of the target image acquired in step S201 (step S202). For example, the learning device can acquire label information of the target image by input from an external source.

Next, the learning device performs pre-settings when generating learning images (step S203). Specifically, a specific target image can be selected from among the target images for which label information was obtained in step S202. Also, in step S203, the number of colors in the areas for each ingredient included in the target image and the colors of the areas of the color pattern extracted by the extraction unit 22 can be specified. Furthermore, in step S203, the number of learning images to be generated, the rotation angle range of the areas, etc. are set in advance.

Next, the extraction unit 22 extracts a color pattern related to the area of the ingredient from the target image based on the setting information in step S203 (step S204). Next, the data expansion unit 23 processes the area extracted by the extraction unit 22, expands the data of the target image, and generates a learning image (step S205).

For example, in step S205, the data expansion unit 23 can generate a learning image by processing the area extracted by the extraction unit 22 in step S202 while maintaining the relative positional relationship between multiple ingredients in the target image to which the first label "good" has been assigned. In this case, the data expansion unit 23 assigns the second label "bad" to the generated learning image (step S206).

Next, the second learning unit 19 learns the second model NN2 using the learning images obtained in step S206, and acquires a learned second model NN2 (step S207). The learning images used in step S207 can include a sufficient number of good and defective images from the perspective of the classification accuracy of the second model NN2.

Next, the learning device stores the trained second model NN2 obtained in step S207 in the second memory 15B (step S208). After that, the process returns to, for example, step S10 in FIG. 8, and calculations are performed on the trained first model NN1 and the trained second model NN2 trained with the training image generated by data augmentation, and an inspection of the arrangement of ingredients can be performed (steps S10 to S19 in FIG. 8).

As described above, according to the sixth embodiment, the inspection device 1E performs data expansion of the target image obtained by the calculation of the trained first model NN1 using instance segmentation, and pads the training images used for training the second model NN2. Therefore, based on images of good products that are relatively easy to obtain on a food production line, it is possible to generate a large number of images of defective products, which are relatively difficult to obtain. As a result, it is possible to prepare a sufficient number of training images of defective products, and the classification accuracy of the trained second model NN2 can be further improved.

[Seventh embodiment]
Next, a seventh embodiment of the present invention will be described. In the following description, the same components as those in the first to sixth embodiments are given the same reference numerals, and the description thereof will be omitted.

In the first to sixth embodiments, a trained first model NN1 that performs instance segmentation is used to estimate the regions and types of multiple ingredients, a first estimation result including an output image in which the estimated regions and types of ingredients are associated with the image is output, and an inspection of the arrangement of ingredients is performed. In addition, the arrangement relationship between ingredients is classified using a trained second learning model NN2.

In contrast, in the seventh embodiment, the first model NN1 can be any deep learning model, such as R-CNN, YOLO, Mask R-CNN, SSD, etc., which detects objects by enclosing the type (class) of ingredients and their position in an image with a bounding box. Note that a bounding box is a boundary line or boundary area that surrounds an object in an image, and it will be understood by those skilled in the art that it is usually rectangular.

In this embodiment, the first model NN1 uses food images acquired by the acquisition unit 10 as input or learning images, and the output includes an anchor box, each ingredient which is a label for each grid, and an area indicating a bounding box. More specifically, the output obtained by the calculation of the trained first model NN1 includes, for example, the probability (score) of including an ingredient center within a grid regardless of the label indicating the ingredient, the ingredient class (class) which is the classification label of the ingredient center included within the grid, and position information indicating the relative position (length, width) and size (width, height) within the grid of the bounding box in which the ingredient center is located within the grid.

The first learning unit 18 uses the input images acquired by the acquisition unit 10 as learning images to learn features that predict the class of an ingredient and a bounding box that contains that ingredient from the learning images.

The first estimation unit 11 is capable of estimating the type of each ingredient included in the input image and the bounding box surrounding the ingredient by calculation of the trained first model NN1. The first estimation unit 11 estimates the relative positional relationship between multiple ingredients and the type of ingredient, and outputs a first estimation result including an output image that associates the estimated positional relationship with the image.

The first judgment unit 12 judges whether the layout relationship satisfies a preset criterion based on the first estimation result. The first judgment unit 12 can also judge the presence or absence of each ingredient from the estimation result by the first estimation unit 11.

Here, when the same type of food is composed of multiple ingredients, the first judgment unit 12 can judge whether the relative placement relationship between the ingredients satisfies a set criterion based on the four vertices or center point of the bounding box of the estimated multiple ingredients. With this configuration, the first judgment unit 12 can judge the appropriateness of the arrangement or degree of dispersion of ingredients of the same type. For example, it becomes possible to judge whether shredded cheese, which is placed as a pizza topping, is placed in a certain position.

The inspection unit 13 inspects the arrangement of multiple ingredients placed in a food container, etc., based on the judgment result by the first judgment unit 12.

As described above, according to the seventh embodiment, even if an object detection model that identifies the position of ingredients in an image and encloses them in a bounding box is used as the first model NN1, it is possible to perform food presentation inspection, including the presence or absence of ingredients placed in food containers, etc., and the appropriateness of the relative placement of ingredients.

[Examples of using inspection equipment]
Here, a usage example of the inspection device 1 according to the embodiment of the present invention will be described with reference to FIGS.

Figure 23 shows an example of an inspection screen displayed on the display device 108 of the inspection device 1. The image i on the left side of the screen shows food photographed by the camera 105. The image s on the right side of the screen is an image obtained by combining the original image i with a color pattern image showing the presentation position of the food ingredient estimated using instance segmentation. In this example, the food variety is "Product B," and the ingredients to be inspected are four types: "Wasabi," "Chopped seaweed," "Cucumber," and "Green onion."

The inspection screen displays the first criteria ("Inspection item") set for each ingredient and the judgment results ("Overall result", "Number recognized", "Total recognized area"). The inspection screen also shows the classification result of the placement position between ingredients ("Location inspection (Class/Score) "0.96076"). The inspection result of the food being inspected is also displayed in the top right of the screen as "OK", meaning it has passed. The second criterion for the placement position between ingredients is displayed in the "Score threshold" item.

As described above, in the inspection device 1, a user can switch between the trained first model NN1 and trained second model NN2, as well as the judgment criteria such as thresholds for the calculation results, for each type of food, thereby enabling a single inspection device 1 to inspect the presentation of multiple different foods. For example, the user can click on the icon mc ("change inspection settings") displayed on the inspection screen, or touch the screen if it is a touch panel, to change settings such as switching the type of food being inspected.

When the user touches or clicks the icon mc displayed on the inspection screen shown in FIG. 23, the screen switches to, for example, a password request screen before transitioning to a setting change screen. When changing the food to be inspected to a different type of food, the user performs input operations using a keyboard or the like to input a user name and password. By requesting a password in this way, it is possible to restrict the setting items that can be changed according to the authority given to the user.

Figure 24 shows an example of a setting change screen displayed on the display device 108. When changing the type of food to be inspected, the user can input the desired food type from the type pull-down list of menu f ("File Selection") displayed on the "Inspection Setting File" setting screen. In the example of Figure 20, the type is selected from three types of food types: "Product A", "Product B", and "Product C".

For example, when the user selects "Product B," the trained first model NN1 ("Segmentation model") and trained second model NN2 ("Classification model") corresponding to the food "Product B" are subsequently called up from memory 15. In addition, thresholds for each ingredient ("Test thresholds," "Test items") used as criteria for the calculation results of the trained first model NN1 and the trained second model NN2 are also called up from memory 15.

As shown in the example of the setting change screen in Figure 25, the user selects "Product B" from the file selection menu f, and then confirms the call of the variety with the confirmation icon e. Areas dl, ct, etc. of the setting change screen display the currently set trained first model NN1, trained second model NN2, threshold values for each ingredient used as judgment criteria, etc. In addition, in the example of Figure 25, in addition to the trained second model NN2, other prepared position correction models can be selected as models used to inspect the placement positions of ingredients. The user can select either the position correction model displayed in the directory in area pc, or the trained second model NN2 displayed in area dl.

The user can also change the first criteria ("test item"), which includes a threshold value for each ingredient used as a judgment criterion for the calculation results of the trained first model NN1, for each ingredient by selecting from a pull-down menu. For example, for the "test item" for the ingredient "cucumber," the user can select "quantity," "single area," "total area," or a combination of these, as well as "not tested," from a pull-down menu. Similarly, the user can change the threshold value set for the test item for each ingredient by selecting from a pull-down menu.

When making changes to settings such as changing the "test items" for each ingredient or changing the thresholds, the user can click icon w to overwrite and save the settings, or click icon n to create a new file. In addition, the user can delete files that are no longer needed by clicking icon d.

The above describes the embodiments of the inspection device and inspection method of the present invention, but the present invention is not limited to the described embodiments, and various modifications that a person skilled in the art can imagine are possible within the scope of the invention described in the claims.

For example, in the embodiment described above, when the inspection process by the inspection device is started, the acquisition unit 10 acquires an image of the inspection target captured by the camera 105 in accordance with the flow of food production on the production line. However, the inspection device can acquire images of manufactured foods captured in the past and perform inspection process not only when food is being manufactured on the food production line, but also, for example, even when the production line is not in operation. For example, this can be done when performing maintenance on the production line.

Furthermore, in the described embodiment, the foods to be inspected include processed noodles, frozen foods, boxed lunches, and prepared foods that are produced on a production line and served with ingredients arranged inside. Note that the foods to be inspected include not only foods in which ingredients are arranged inside a container, but also foods in which ingredients are arranged without using a container.

In addition, in the embodiment described above, a case where there are multiple types of ingredients to be inspected is exemplified, but the number of types of ingredients may be single.

The learning device that performs the learning process for the first model NN1 and the second model NN2 can be configured separately and independently from the inspection device.

It should be noted that the various functional blocks, modules, and circuits described in connection with the embodiments disclosed herein may be implemented using a general purpose processor, a GPU, a digital signal processor (DSP), an application specific integrated circuit (ASIC), an FPGA or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of the above designed to achieve the functionality described above.

A general purpose processor may be a microprocessor, but may alternatively be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors connected to a DSP core, or any such combination of computing devices.

1...inspection device, 10...acquisition unit, 11...first estimation unit, 12...first judgment unit, 13...inspection unit, 14...presentation unit, 15...memory, 15A...first memory, 101...bus, 102...processor, 103...main memory device, 104...communication I/F, 105...camera, 106...auxiliary memory device, 107...input/output I/O, 108...display device, NW...network.

Claims (33)

An acquisition unit that acquires an image including the arranged ingredients;
a first estimation unit that estimates a region and a type of the ingredient in the image using a trained first model and outputs a first estimation result including an output image in which the estimated region and type of the ingredient are associated with the image;
a first determination unit that determines, for each type of ingredient, whether the corresponding area satisfies a first criterion set in advance regarding the presence or absence and the excess or deficiency of the ingredient based on the first estimation result;
An inspection unit that inspects the arrangement of the ingredients based on the judgment result by the first judgment unit ;
a second estimation unit that estimates a relative positional relationship between the ingredients in the output image by using a trained second model in which the output image included in the first estimation result is input, and outputs a second estimation result in which the estimated positional relationship is associated with the output image;
a second determination unit that determines whether or not the layout relationship satisfies a second criterion that is set in advance regarding the validity of the layout relationship based on the second estimation result;
Equipped with
The inspection unit inspects the arrangement of the ingredients based on the judgment result by the second judgment unit.
An inspection device characterized by: 2. The inspection device according to claim 1,
The first judgment unit identifies at least one of the number and size of the regions for each type of food material and judges whether or not at least one of the number and size of the regions satisfies the first criterion. In the inspection device according to claim 1 or 2 ,
Further, a presentation unit is provided that presents at least the test result by the test unit,
the presentation unit presents at least one of information including the image, the first estimation result, the judgment result by the first judgment unit, the second estimation result, the judgment result by the second judgment unit, and the inspection result by the inspection unit. 4. The inspection device according to claim 1,
Further, a learning device is provided for performing a learning process for constructing a preset model,
The learning device includes:
a first learning unit that uses an image including the arranged ingredients as a first learning image, learns a first model of a neural network, extracts a first feature amount that identifies a region and a type of the ingredients, and acquires the learned first model based on the extracted first feature amount;
a first storage unit that stores the trained first model acquired by the first learning unit,
The inspection device, characterized in that the first estimation unit reads the trained first model from the first storage unit and estimates the area and type of the food ingredient in the image using the trained first model. 5. The inspection device according to claim 4 ,
The inspection apparatus according to claim 1, wherein the first model is an image recognition model based on instance segmentation. In the inspection device according to claim 4 or 5 ,
The learning device includes:
a second learning unit that uses the output image included in the first estimation result output by the first estimation unit as a second learning image to learn a second model of a neural network, extracts second feature amounts that identify a relative positional relationship between the plurality of ingredients in the second learning image, and acquires the trained second model using the extracted second feature amounts;
and a second memory unit that stores the trained second model acquired by the second learning unit. 7. The inspection device according to claim 6 ,
a conversion unit that converts the output image included in the first estimation result to generate a more simplified converted image,
the second learning unit uses the converted image as the second learning image to learn the second model, and acquires the trained second model. 8. The inspection device according to claim 7 ,
the output image included in the first estimation result is a color pattern image in which the corresponding area is painted with an arbitrary color for each type of ingredient,
The inspection device according to claim 1, wherein the converted image is a color pattern image in which each area of the color pattern image is replaced with an object of the same shape. 8. The inspection device according to claim 7 ,
the output image included in the first estimation result is a color pattern image in which the corresponding area is painted with an arbitrary color for each type of ingredient,
an inspection device characterized in that the converted image is an image in which each area of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the colors of the replaced objects are made different for each shape. 8. The inspection device according to claim 7 ,
the output image included in the first estimation result is a color pattern image in which the corresponding area is painted with an arbitrary color for each type of ingredient,
an inspection device characterized in that the converted image is an image in which each area of the color pattern image is replaced with an object having a different shape for each color of the color pattern, and the colors of the replaced objects are all the same color. 7. The inspection device according to claim 6 ,
a processing unit that processes the output image by determining that an inside of a longest estimated outline of the region of the output image included in the first estimation result is the region, and generates a processed image by processing the output image,
The second learning unit uses the processed image as the second learning image to learn the second model and acquires the trained second model. In the inspection device according to any one of claims 6 to 11 ,
The learning device includes:
Further, a data extension unit performs data extension on a target image in which the region and type of the ingredient are associated with the image to generate the second learning image,
The data augmentation unit generates the second learning image based on label information regarding the validity of a relative positional relationship between the plurality of ingredients in the target image, the label information being assigned to the target image. The inspection device according to claim 12 ,
The data expansion unit performs data expansion on a target image to which a first label indicating that a relative positional relationship between the multiple ingredients in the target image is valid is assigned, and generates the second learning image to which the first label is assigned. In the inspection device according to claim 1 2 or claim 1 3 ,
The data expansion unit performs data expansion on a target image to which a first label indicating that the relative positional relationship between the multiple ingredients in the target image is appropriate is assigned, and generates the second learning image to which a second label indicating that the relative positional relationship between the multiple ingredients is not appropriate is assigned. In the inspection device according to claim 1 2 or claim 1 4 ,
The data augmentation unit performs data augmentation on a target image to which a second label indicating that the relative positional relationship between the multiple ingredients is not valid is assigned, and generates the second learning image to which the second label is assigned. The inspection device according to any one of claims 12 to 15 ,
Further, an extraction unit is provided that extracts the area for each type of food material from the target image,
The data expansion unit processes and rearranges the area extracted by the extraction unit while maintaining the relative positional relationship between the multiple ingredients in the target image to generate the second learning image. The inspection device according to any one of claims 12 to 15 ,
Further, an extraction unit is provided that extracts the area for each type of food material from the target image,
The data expansion unit replaces positions of areas corresponding to two or more different types of ingredients among the areas extracted by the extraction unit to generate the second learning image. The inspection device according to any one of claims 12 to 15 ,
Further, an extraction unit is provided that extracts the area for each type of food material from the target image,
The data extension unit generates the second learning image by deleting one or more of the regions extracted by the extraction unit from the target image. The inspection device according to any one of claims 12 to 15 ,
Further, an extraction unit is provided that extracts the area for each type of food material from the target image,
The data expansion unit performs data expansion of the target image by performing at least one of inversion, rotation, expansion, and contraction of each of the regions extracted by the extraction unit. A first step of acquiring an image including the arranged ingredients;
a second step of estimating a region and a type of the ingredient in the image using the trained first model, and outputting a first estimation result including an output image in which the estimated region and type of the ingredient are associated with the image;
a third step of determining whether or not the corresponding area for each type of ingredient satisfies a first criterion set in advance regarding the presence or absence and the excess or deficiency of the ingredient based on the first estimation result;
A fourth step of inspecting the arrangement of the ingredients based on the judgment result in the third step ;
a fifth step of estimating a relative arrangement relationship between the ingredients in the output image using a trained second model that uses the output image included in the first estimation result as an input, and outputting a second estimation result in which the estimated arrangement relationship is associated with the output image;
a sixth step of determining whether or not the layout relationship satisfies a second criterion set in advance regarding the validity of the layout relationship based on the second estimation result;
Equipped with
The fourth step is to inspect the arrangement of the ingredients based on the result of the judgment in the sixth step.
13. An inspection method comprising : The inspection method according to claim 20 ,
The third step includes identifying at least one of the number and size of the regions for each type of food ingredient and determining whether or not at least one of the number and size of the regions satisfies the first criterion;
The fourth step comprises inspecting an arrangement of the ingredients based on the result of the judgment. In the inspection method according to claim 20 or claim 21 ,
Further, a learning step is provided for performing a learning process for constructing a preset model,
The learning step includes:
a seventh step of using an image including the arranged ingredients as a first learning image to train a first model of a neural network, extracting a first feature amount that identifies a region and a type of the ingredients, and acquiring the trained first model based on the extracted first feature amount;
and an eighth step of storing the trained first model acquired in the seventh step in a first storage unit,
The inspection method, characterized in that the third step reads the trained first model from the first storage unit and estimates the area and type of the food ingredient in the image using the trained first model. The inspection method according to claim 22 ,
The learning step includes:
a ninth step of training a second model of a neural network using the output image included in the first estimation result output in the third step as a second training image, extracting a second feature amount that identifies a relative positional relationship between the plurality of ingredients in the second training image, and acquiring the trained second model using the extracted second feature amount;
A tenth step of storing the trained second model acquired in the ninth step in a second memory unit. In the inspection method according to claim 2 or 3 ,
and generating a processed image by processing the output image, the processed image being determined to be an area inside a longest estimated outline of the area of the output image included in the first estimation result,
The inspection method, wherein the ninth step includes using the processed image as the second learning image to learn the second model and obtain the learned second model. In the inspection method according to claim 2, 3 or 2 ,
The learning step includes:
Further, a twelfth step of performing data expansion of a target image in which the region and type of the ingredient are associated with the image to generate the second learning image,
The twelfth step comprises generating the second learning image based on label information regarding the validity of a relative positional relationship between the plurality of ingredients in the target image, the label information being assigned to the target image. 26. The inspection method according to claim 25 ,
The twelfth step comprises performing data augmentation on a target image to which a first label indicating that a relative positional relationship between the multiple ingredients in the target image is valid is assigned, thereby generating the second learning image to which the first label is assigned. The inspection method according to claim 25 or claim 26 ,
The twelfth step comprises performing data expansion on a target image to which a first label indicating that the relative positional relationship between the multiple ingredients in the target image is valid is assigned, and generating the second learning image to which a second label indicating that the relative positional relationship between the multiple ingredients is invalid is assigned. The inspection method according to claim 25 or claim 27 ,
The twelfth step comprises performing data augmentation on a target image to which a second label indicating that the relative positional relationship between the multiple ingredients is not valid is assigned, and generating the second learning image to which the second label is assigned. The inspection method according to any one of claims 25 to 26 ,
Further, a thirteenth step of extracting the area for each type of food material from the target image,
The twelfth step comprises processing and rearranging the area extracted in the thirteenth step while maintaining the relative positional relationship between the plurality of ingredients in the target image, to generate the second learning image. The inspection method according to any one of claims 5 to 28 ,
Further, a thirteenth step of extracting the area for each type of food material from the target image,
The twelfth step is to mutually replace positions of areas corresponding to two or more different types of ingredients among the areas extracted in the thirteenth step to generate the second learning image. The inspection method according to any one of claims 5 to 28 ,
Further, a thirteenth step of extracting the area for each type of food material from the target image,
The inspection method, wherein the twelfth step generates the second learning image by deleting one or more of the regions extracted in the thirteenth step from the target image. The inspection method according to any one of claims 5 to 28 ,
Further, a thirteenth step of extracting the area for each type of food material from the target image,
The inspection method according to claim 1, wherein the twelfth step performs at least one of inversion, rotation, expansion, and contraction of each of the regions extracted in the thirteenth step to perform data expansion of the target image. The inspection method according to any one of claims 20 to 32 ,
Further, a fourteenth step of presenting at least the test result in the fourth step is provided,
an inspection method comprising: presenting at least one of information including the image, the first estimation result, the judgment result in the third step, and the inspection result in the fourth step;