JP7637519B2 - Image processing method and image processing device - Google Patents

Description

The present invention relates to an image processing method and an image processing device.

There are known defect inspection systems, anomaly inspection systems, and peculiar part detection systems that inspect for the presence or absence of an abnormal state or a part with characteristics different from others. In recent years, a method has been proposed that uses machine learning to evaluate the presence or absence of a part with characteristics different from most other areas, including defects (hereinafter referred to as a "peculiar part")

Patent document 1 describes a peculiar part detection system that extracts images of peculiar parts that have arbitrary characteristics from an image of an object captured by an imaging unit, and identifies the type of peculiar part through machine learning using the images of the peculiar parts as input.

International Publication No. 2019/003813

The unique part to be detected can have a variety of shapes. If the unique part has a large shape or is formed in an elongated shape, the image of the unique part extracted from the captured image will also be large.

However, if an attempt is made to detect such singular parts using machine learning, the learning time required to generate a trained model will be long. In addition, the calculation time required to detect singular parts using the trained model will increase. Although these issues can be expected to be resolved by using information processing devices with high computing power, this will result in increased costs, which will be a hindrance when implementing such devices on a manufacturing line for applications such as in-line inspection.

In addition, if the size of the unique part image is larger than the image size set as the image size to which machine learning is applied, the unique part image must be extracted without including part of the unique part, resulting in a decrease in the accuracy of detection of the unique part.

In order to solve the above problems, the present invention aims to provide an image processing method and image processing device that achieve both detection accuracy and processing speed in an image processing method that applies machine learning.

The image processing method according to the present disclosure includes obtaining a plurality of images of an object, each of which captures a different portion of a predetermined range of the object; applying a predetermined image processing method obtained using machine learning to each of the plurality of images to perform an individual evaluation of whether or not each of the plurality of images contains a feature of the object to be detected; and performing a comprehensive evaluation of whether or not the object contains a peculiar part to be detected based on whether or not the feature is contained in two or more of the plurality of images. Here, the comprehensive evaluation evaluates that the object contains a peculiar part to be detected when the ratio X1/Y1 of the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to the number X1 of the Y1 images that are evaluated in the individual evaluation as containing the feature is equal to or greater than a first predetermined value.

According to this image processing method, a predetermined image processing method acquired using machine learning is applied to perform individual evaluation of whether each of the multiple images contains the feature of the detection target, and a comprehensive evaluation of whether the object contains the peculiar part of the detection target is performed based on whether the feature is included in two or more of the multiple images. Specifically, the presence or absence of the peculiar part is evaluated based on the number Y1 of at least one of the multiple images corresponding to a predetermined range or the multiple images constituting a partial range that is a part of the predetermined range, and the number X1 of images evaluated as containing the feature. In other words, the object is evaluated as to whether it contains the peculiar part of the detection target based on the number of images that have an estimated score indicating that the feature is included in the individual evaluation that is equal to or greater than a predetermined value among the multiple images constituting the predetermined range or the partial range of the object. Therefore, even if the size of the peculiar part image is larger than the image size set as the image size to which machine learning is applied, individual evaluation is performed at that image size, and a comprehensive evaluation is performed based on the number of multiple images containing the feature in the predetermined range or partial range, thereby making it possible to shorten the learning time and calculation time while detecting the peculiar part larger than the image size.

In addition, in this disclosure, "evaluation" includes determining whether or not there is a unique part, or determining whether or not there is a unique part along with probability information, and outputting probability information (estimated score) regarding the presence or absence of a unique part. Furthermore, in the present disclosure, "evaluation" includes outputting information regarding the presence or absence of the unique part and/or information regarding the type of unique part when there are multiple types of unique parts.

Note that different or the same evaluations may be performed before or after the individual evaluations and overall evaluations described above. For example, after performing an overall evaluation for a specified range, an additional overall evaluation may be performed once or multiple times for a partial range that is part of the specified range.

An aspect of the present disclosure further includes dividing an original image that captures a predetermined range of the object to generate the multiple images, each of which has the same image size.

In one aspect of the present disclosure, the overall evaluation evaluates that the object contains a peculiar portion to be detected when the ratio X1/Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value that is smaller than the first predetermined value, and the ratio X2/Y2 between the number Y2 (<Y1) of the images constituting the predetermined range or a second partial range that is part of the partial range consisting of the Y1 images and the number X2 of the images among the Y2 images that are evaluated as containing the feature in the individual evaluation is equal to or greater than a third predetermined value.

In one aspect of the present disclosure, the comprehensive evaluation can be such that the object contains a peculiar portion to be detected if image A is present that is evaluated in the individual evaluation as containing the characteristic, and if, in the individual evaluation, two or more of the multiple images located around image A are evaluated as containing the characteristic.

In one aspect of the present disclosure, the comprehensive evaluation can be such that the object includes a peculiar portion to be detected when image A is present that is evaluated to include the feature in the individual evaluation, image B is present that is evaluated to include the feature in the individual evaluation, image C is present that is not located around image A but is located around image B, image D is present that is evaluated to include the feature in the individual evaluation, image E is present that is not located around image A, image B, or image C but is located around image D, and image E is present that is evaluated to include the feature in the individual evaluation.

The present disclosure discloses an image processing device. The image processing device includes an image acquisition means for acquiring a plurality of images capturing different parts of a predetermined range of an object, an individual evaluation means for applying a predetermined image processing method acquired using machine learning to each of the plurality of images to perform an individual evaluation of whether or not each of the plurality of images contains a feature of the detection target, and a comprehensive evaluation means for performing a comprehensive evaluation of whether or not the object contains a peculiar part of the detection target based on whether or not the feature is contained in two or more images of the plurality of images, and the comprehensive evaluation means is configured to evaluate that the object contains a peculiar part of the detection target when the ratio X1/Y1 of the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to the number X1 of the images evaluated by the individual evaluation means as containing the feature is equal to or greater than a first predetermined value.

According to such an image processing device, it is possible to apply a predetermined image processing method acquired using machine learning to perform individual evaluation of whether each of the multiple images contains the feature to be detected, and to perform a comprehensive evaluation of whether the target object contains the peculiar part to be detected based on whether the feature is included in two or more of the multiple images. Specifically, it is possible to evaluate the presence or absence of a peculiar part based on the number Y1 of multiple images constituting a predetermined range or a partial range that is a part of the predetermined range, and the number X1 of images evaluated as containing the feature. Therefore, even if the size of the peculiar part image is larger than the image size set as the image size to which machine learning is applied, it is possible to shorten the learning time and calculation time while detecting a peculiar part larger than the image size by performing individual evaluation at that image size and performing a comprehensive evaluation based on the number of multiple images containing the feature within the predetermined range or partial range.

In addition, the image processing device of the present disclosure may further include a means for dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the multiple images, each of which has the same image size.

In one aspect of the present disclosure, the comprehensive evaluation means is configured to evaluate that the object includes a peculiar portion to be detected when the ratio X1/Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value smaller than the first predetermined value, and the ratio X2/Y2 between the number Y2 (<Y1) of a plurality of images constituting the predetermined range or a second partial range that is a part of the partial range consisting of the Y1 images and the number X2 of the Y2 images evaluated by the individual evaluation means as including the feature is equal to or greater than a third predetermined value.

In one aspect of the present disclosure, the comprehensive evaluation means may be configured to evaluate the object as including the peculiar portion to be detected when image A is present that has been evaluated by the individual evaluation means as including the characteristic, and when the individual evaluation means has evaluated that two or more of the multiple images located around image A include the characteristic.

In one aspect of the present disclosure, the comprehensive evaluation means may be configured to evaluate the object as including the peculiar portion of the detection target when image A evaluated by the individual evaluation means as including the feature exists, image B evaluated by the individual evaluation means as including the feature exists, image C not located around image A but around image B exists, image D not located around image A and image B but around image C exists, image D evaluated by the individual evaluation means as including the feature exists, image E not located around image A, image B, and image C exists, and image E evaluated by the individual evaluation means as including the feature exists.

Note that an image being "located around" another image refers to a positional relationship in which no other image exists between the one image and the other image, typically meaning that the vertices or edges of the one image and the other image face each other or are shared, and that an image not being "located around" the other image refers to a positional relationship in which there is yet another image between the one image and the other image.

In one aspect of the present disclosure, the peculiar portion is a defect in the object.

The computer program according to the present disclosure includes instructions to cause a computer to execute the following: acquiring a plurality of images of an object, each of which is an image of a different portion of a predetermined range of the object; applying a predetermined image processing method acquired using machine learning to each of the plurality of images to perform an individual evaluation of whether or not each of the plurality of images includes a feature of the object to be detected; and performing a comprehensive evaluation of whether or not the object includes a peculiar portion of the object to be detected, based on whether or not the feature is included in two or more of the plurality of images. Here, the comprehensive evaluation evaluates that the object includes a peculiar portion of the object to be detected when the ratio X1/Y1 of the number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to the number X1 of the Y1 images that are evaluated in the individual evaluation as including the feature is equal to or greater than a first predetermined value.

The computer program may be stored on a non-transitory storage medium. The non-transitory storage medium includes a non-volatile semiconductor memory element.

By executing such a computer program on a computer, it is possible to achieve the same effect as the image processing method described above.

The image processing system according to the present disclosure includes the image processing device described above and a server. The server stores a first image data group including different types of features and a second image data group including no features as training data for machine learning. The first image data may be assigned the same label. The second image data group is assigned a label different from the label assigned to the first image data. The image processing system according to the present disclosure also includes a mode in which the execution instructions included in the computer program according to the present disclosure described above are executed in cooperation with a plurality of arithmetic devices including a processor and a memory and connected to each other wirelessly or via a wired connection so as to be able to communicate with each other.

According to such an image processing system, it is possible to achieve the same effects as the image processing device described above.

FIG. 1 is a functional block diagram of an image processing system according to this embodiment. FIG. 2 is a block diagram showing the hardware configuration of the image processing system according to this embodiment. FIG. 3 is a flowchart of the image processing method according to this embodiment. FIG. 4 is a diagram showing a schematic diagram of the results of individual evaluation of divided images of partial ranges. FIG. 5 shows an example of a pattern of positional relationships between a plurality of divided images used in the overall evaluation of the image processing method according to the modified example of this embodiment. FIG. 6 shows an example of a pattern of positional relationships between a plurality of divided images used in the overall evaluation of the image processing method according to the modified example of this embodiment. FIG. 7 is a diagram showing the relationship between the size of a unique portion and a divided image.

The following describes an embodiment of the present invention with reference to the drawings. The following embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only that embodiment.

Figure 1 is a functional block diagram of an image processing system 10 according to this embodiment. This image processing system 10 is an image processing system for determining whether or not a foreign object is present inside a plastic bottle (an example of an "object") that contains liquid. However, as will be described later, the present invention can be applied to various objects and various unique parts.

This image processing system 10 includes an image processing device 20 having an imaging unit 12, an image division unit 14, an individual evaluation unit 16, and a comprehensive evaluation unit 18. Furthermore, the image processing system 10 further includes a server device connected to the image processing device 20 via a network N. The server device includes a database DB1 for storing images of normal samples and a database DB2 for storing images of abnormal samples. In this embodiment, a normal sample is a sample that does not include a unique part, in other words, a sample in which the object has no abnormality and corresponds to a good product. An abnormal sample is a sample that includes a unique part, in other words, a sample in which the object has an abnormality and corresponds to a defective product.

The imaging unit 12 captures a part or the whole of the object and acquires an image (an example of a "base image"). The imaging unit 12 can be composed of an image sensor such as a line sensor or an area sensor. For example, if the object has a shape that is rotationally symmetrical with respect to a central axis (such as a container having an internal space for accommodating contents), an image corresponding to the entire side of the object (an example of a "predetermined range") can be acquired by capturing an image of the side of the object with a line sensor while rotating the object around the central axis. Note that the object is not limited to a cylindrical shape. In this embodiment, the imaging unit 12 is composed of a line sensor, and acquires grayscale data having a luminance value of 8 bits (256 gradations) for each pixel of 5000 pixels x 5000 pixels as the image of the object. Note that the imaging unit 12 may acquire color data such as RGB data as the image of the object. Also, instead of acquiring an image by capturing an image of the object with the imaging unit 12, an image of the object may be acquired from a database or the like that stores images of the object.

The image division unit 14 (an example of an "image acquisition means") divides the image acquired by the imaging unit 12 into multiple images of the same size, and acquires multiple images capturing different parts of a predetermined range of the object (hereinafter, each divided image may be referred to as a divided image). The divided images are set to have a size suitable for an image processing method using machine learning. Here, the size of the divided images does not need to be set to include the size that the unique part to be detected may have, and may be set to be smaller than the size that the unique part may have. For example, when the size of the predetermined range is 5000 pixels x 5000 pixels and the size of the divided images is 75 pixels x 75 pixels, the image division unit 14 acquires approximately 4400 divided images. By setting the size of the divided images small so as to be suitable for machine learning in this way, it is possible to shorten the calculation time for machine learning compared to the case where the image size is set to include the unique part. Furthermore, it is possible to reduce the number of images required for learning, which has the great advantage of shortening the learning time and making it easier to collect images of abnormal samples.

The individual evaluation unit 16 applies a predetermined image processing method acquired using machine learning to each of the multiple segmented images, and performs an individual evaluation of whether each of the multiple segmented images contains the feature to be detected.

Here, a unique portion refers to a portion that has characteristics different from the surrounding area in terms of at least one of shape, brightness, and color, and the characteristic is the unique portion itself or a part of the unique portion. Therefore, a characteristic refers to a portion that has characteristics different from the surrounding area in terms of at least one of shape, brightness, and color.

Unique parts and features may be defects, or may be artificially set marks or patterns. Unique parts and features related to shape refer to parts that have shape characteristics (e.g., volume, area, height, particle size, aspect ratio, circularity, contour shape, etc.) that differ from the surrounding areas. Similarly, unique parts and features related to luminance refer to parts that have characteristics that differ from the surrounding areas in luminance-related characteristics such as brightness and histogram, and in color refer to parts that have color-related characteristics such as spectrum, central wavelength, maximum wavelength, etc.

The unique parts and features of this embodiment are dust, human hair, air bubbles, and dirt on the container that may be contained in the liquid inside the plastic bottle.

The individual evaluation unit 16 of this embodiment includes a trained model composed of a convolutional neural network. The convolutional neural network includes an input layer for inputting data of a divided image, an output layer for outputting the presence or absence of features, and an intermediate layer for connecting the two layers. The intermediate layer includes a combination of a convolutional layer that performs a convolution operation to calculate the sum of products of overlapping parts as features by sliding a filter to generate a map having features, and a pooling layer that extracts the maximum value of the two-dimensional array data output from the convolutional layer, and a combination layer that combines internal parameters as weighting coefficients to provide an evaluation result to the output layer. In addition to the convolutional neural network, the neural network can be composed of a recurrent neural network (RNN), an Elman network, a Jordan network, an echo state network (ESN), a long short term memory network (LSTM), a bi-directional RNN, etc. The output layer also outputs probability information (sometimes called "classification probability") that indicates the likelihood that the target segmented image has a feature. However, the output from the output layer may be a binary result of the presence or absence of a feature, or may be an output for each type of feature.

The server device includes a database DB1 that stores sample images of normal objects, and a database DB2 that stores sample images of abnormal objects. Database DB1 stores data of segmented images that do not include peculiar parts. In this embodiment, database DB1 stores data of segmented images that do not include peculiar parts or features, which is 75 pixels x 75 pixels, acquired by capturing an image of the side of a plastic bottle, which is the object.

Database DB2 stores a second image data group including unique parts or features. In this embodiment, database DB2 stores data of divided images including at least one of unique parts or features of 75 pixels x 75 pixels acquired by imaging the side of the target PET bottle. In this embodiment, the unique parts and features are, for example, human hair, dust, stains on the container, or parts thereof. Therefore, database DB2 stores data of multiple divided images including human hair, data of multiple divided images including dust, data of multiple divided images including stains on the container, etc.

The unique parts and features differ depending on the detection target. For example, if the detection target is air bubbles, data of the divided image including the air bubbles is stored in database DB2 as second image data including the unique parts and features. However, if the detection target is a plastic bottle containing a carbonated drink, air bubbles are not the detection target, so data of the divided image including the air bubbles is stored in database DB1 as first image data including the unique parts or features.

The individual evaluation unit 16 acquires a trained model including internal parameters for evaluating the presence or absence of features by learning from the data of the segmented images stored in databases DB1 and DB2 as training data. The internal parameters are, for example, bias values in the convolution layer and weighting coefficients in the connection layer. The internal parameters change when training data is added.

The trained model in this embodiment is a trained model configured to output information regarding the presence or absence of a feature included in an input image of a predetermined size. The trained model in this embodiment is generated by machine learning using training data in which each of the data of a plurality of segmented images is labeled with the presence or absence of a feature. The trained model may be configured to output information regarding the presence or absence of a feature included in the input image and/or information regarding the type of feature. Such a trained model is generated by machine learning using training data in which each of the data of a plurality of segmented images is labeled with the presence or absence of a feature and/or the type of feature.

The individual evaluation unit 16 applies the trained model obtained using machine learning as described above to each of the divided images, performs an individual evaluation of whether or not each of the 1,000 or more divided images contains a feature, and obtains probability information indicating the possibility of the presence or absence of a feature.

The overall evaluation unit 18 evaluates the presence or absence of a unique part based on two or more of the divided images that have been evaluated by the individual evaluation unit 16 as possibly including a feature among the multiple divided images. More specifically, the overall evaluation unit 18 acquires the number Y1 of divided images within a predetermined range and the number X1 of divided images that have been evaluated by the individual evaluation unit 16 as possibly including a feature, and evaluates that the object includes a unique part if the ratio X1/Y1 is equal to or greater than a predetermined threshold value.

2 is a block diagram showing a hardware configuration for realizing the image processing device 20. As described above, the imaging unit 12 captures an image of an object to obtain an image of a predetermined range of the object, and is composed of, for example, a line camera. The processor 22 executes a computer program (including a trained model for evaluating the presence or absence of a singular part) stored in the memory unit 24 to perform each arithmetic process shown in this embodiment. Therefore, the processor 22 and the memory unit 24 cooperate to function as the image segmentation unit 14, the individual evaluation unit 16, and the overall evaluation unit 18. The processor 22 is composed of, for example, an ASIC (Application Specific Integrated Circuit) having multiple arithmetic cores, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), or a quantum computer. The memory unit 24 stores each command and other information including a computer program (including a trained model for evaluating the presence or absence of a feature) for executing each arithmetic process shown in this embodiment. The storage unit 24 is composed of a non-volatile semiconductor storage element (non-transient storage element) such as a NAND flash memory, FeRAM, or MRAM that can electrically record and read information, or a magnetic storage element such as a HDD (Hard Disk Drive). The RAM 26 is composed of a volatile semiconductor storage element such as an SRAM (Static Random Access Memory) and a DRAM (Dynamic Random Access Memory) for temporarily storing data and other information used for each arithmetic processing and the like shown in this embodiment. The display unit 28 has a display for displaying the results of the arithmetic operations performed by the processor 22. The arithmetic operations include the evaluation results performed by the individual evaluation unit 16 and the overall evaluation unit 18. The communication I/F unit 30 is connected to the databases DB1 and DB2 via the network N, and can receive information from the databases DB1 and DB2. The input unit 32 has an input device such as a keyboard for an inspector to input information to the image processing system 10. These components are connected to each other via a bus so that they can transmit and receive data. However, some components may be provided in a remote location via the network N, or may be integrated with other components. In addition, some of the functions executed by the processor 22 may be executed by other components, such as the imaging unit 12. Furthermore, the server device may also have the hardware configuration shown in FIG. 2. Furthermore, the databases DB1 and DB2 may be configured as information processing devices integrated with the image processing device 20. Furthermore, the information processing devices that execute the image segmentation unit 14, the individual evaluation unit 16, and the overall evaluation unit 18 of the image processing device 20 may be provided separately from the image processing device 20, and these may be electrically connected via the network N to configure the image processing system 10.

An image processing method using the image processing system 10 described above will now be described. Figure 3 is a flowchart of the image processing method shown in this embodiment.

First, the imaging unit 12 captures an image of the object to obtain a base image (step S1). Specifically, while rotating the object, which is a plastic bottle, around its central axis, the imaging unit 12 captures an image of the side of the plastic bottle using the line camera to obtain a base image corresponding to a predetermined range. In this embodiment, the imaging unit 12 captures an image of 5000 pixels x 5000 pixels as the base image corresponding to the predetermined range. Also, in this embodiment, the imaging unit 12 obtains grayscale data having a luminance value of 8 bits (256 gradations) for each pixel of the 5000 pixels x 5000 pixels as the base image. Here, the plastic bottle container is translucent. Therefore, the imaging unit 12 obtains an image of the plastic bottle container and the liquid contained therein.

Next, the image division unit 14 divides the base image corresponding to the side of the PET bottle acquired by the imaging unit 12 into divided images of 75 pixels x 75 pixels, and obtains more than 1,000 divided images (step S2).

Then, the individual evaluation unit 16 individually evaluates each divided image based on the trained model generated using machine learning, and evaluates whether each divided image contains features (step S3).

Figure 4 is a diagram showing the results of individual evaluation of divided images of a partial range that is a part of a predetermined range. The partial range in the figure includes 80 divided images arranged in 8 rows in the vertical direction of the paper and 10 columns in the horizontal direction of the paper. The numerical value shown for each divided image indicates probability information obtained by the individual evaluation unit 16 indicating the possibility that each divided image has a feature. In this embodiment, the closer the numerical value is to 0, the higher the possibility that the divided image has a feature, and the closer the numerical value is to 1, the lower the possibility that the divided image has a feature. For example, the probability information of divided image DI1 is 0.041, so the probability that divided image DI1 does not have a feature is 4.1% (the probability that the divided image has a feature is 95.9%). The probability information of divided image DI2 is 0.444, so the probability that divided image DI2 does not have a feature is 44.4% (the probability that the divided image has a feature is 55.6%).

The image processing device 20 may be configured to display probability information for each divided image on the display unit 28. For example, the probability information may be classified using three thresholds, and each divided image may be configured to be displayed in a different color, thereby making it easy to see how divided images evaluated to have a feature are distributed within a specified range or partial range. The threshold value that is the standard for evaluating whether an image has a feature can be set appropriately depending on the detection target. For example, the individual evaluation unit 16 may be configured to evaluate an image as not having a feature when the probability of not having a feature is 90% or higher.

There are various reasons why an image may be evaluated as having a feature. For example, an image may be evaluated as having a feature due to noise. An image may also be evaluated as having a feature due to dust, human hair, etc. For example, if a human hair H (Figure 4) is mixed in, an image may be evaluated as having a feature across multiple adjacent divided images, such as divided images DI3 to DI7.

Then, the overall evaluation unit 18 obtains the number Y1 of divided images within the predetermined range and the number X1 of divided images evaluated by the individual evaluation unit 16 as possibly containing features, and determines whether the ratio X1/Y1 is equal to or greater than a predetermined threshold value (an example of a "first predetermined value") (step S4).

For example, if Y1 is 5000 and X1 is 10, X1/Y1 is 0.002. If this value is equal to or greater than a predetermined threshold (YES), the object to be detected is evaluated as including a unique portion (step S5). If it is less than the threshold (NO), the object to be detected is evaluated as not including a unique portion (step S6).

If the detection object is evaluated to include a unique part, the display unit 28 outputs information to that effect, and prompts the inspector to perform a visual inspection (step S7). At this time, the display unit 28 is configured to display the position within a predetermined range of the divided image evaluated to include a unique part or feature. With this configuration, the inspector can easily identify the area evaluated to include a unique part or feature. This makes it possible to reduce the visual inspection time. Note that if the detection object is evaluated to include a unique part, the image processing device 20 may output information to that effect to a controller (e.g., a programmable logic controller (PLC), etc.) that controls the controlled object of the production line, and the controller that receives the information may control the production line so that the object evaluated to include a unique part is discharged from the production line as a defective product or a candidate for a defective product.

Even if the detection target is evaluated as not containing a unique portion, visual inspection may be further performed.

Furthermore, if the object to be detected is large, the image processing method according to this embodiment including steps S1 to S7 may be repeatedly executed for different ranges. On the other hand, if the singular portion is a defect and the locations where the defect may occur are concentrated in a certain area, even if the object to be detected is large, the image processing method according to this embodiment including steps S1 to S7 may be executed for that area, and the object to be detected may be inspected for another area using a different method (e.g., visual inspection).

According to the image processing system 10 and image processing method described above, it is possible to set the size of the divided images small so as to be suitable for machine learning, and therefore it is possible to shorten the learning time and calculation time of machine learning compared to the case where the image size is set large so as to include the unique part. Furthermore, the inventors of the present application have focused on the fact that noise and the like that should not be evaluated as a unique part or feature rarely spans multiple divided images, while dust, human hair, etc. that should be evaluated as a unique part or feature are relatively often spread across multiple divided images, and have devised an image processing system 10 and an image processing method that evaluates a unique part based on the fact that multiple divided images have characteristics, rather than evaluating that a single divided image has a characteristic. As a result, it is possible to reduce the possibility of overdetection due to a mistaken recognition that a unique part exists due to noise, etc., and to suppress overlooking dust, human hair, etc. that should be evaluated as a unique part or feature, thereby improving detection accuracy.

In other words, by setting the size of the divided images so that the unique parts span multiple divided images, it can be said that an image processing system 10 and an image processing method that achieve both detection accuracy and processing speed are provided.

When objects are mass-produced, even the presence of a unique part in just one of them can cause a major problem. However, by using the image processing system 10 and image processing method according to this embodiment and deliberately generating segmented images that are smaller in size than the unique part, it is possible to reduce the rate at which the unique part is overlooked.

In addition, the image processing method in this embodiment employs a method that uses a model obtained by machine learning that can autonomously generate estimation rules for input data based on teacher data (training data), and by using a segmented image with a small number of pixels as the teacher data, it is possible to shorten the learning time.

In particular, assuming that the evaluation method in this embodiment is implemented in a manufacturing line and used for product inspection, it is preferable that the time required to generate a trained model is short in order to flexibly respond to lot changes and model changes in the products to be manufactured. As described above, by suppressing increases in learning time, it is possible to realize an evaluation method suitable for implementation in a manufacturing line.

In order to further enhance the above-mentioned effects, it is preferable to use a rule-based image processing method as the first method, which has fewer elements that need to be changed in response to changes in the lot or model of the product being manufactured and is also applicable to image processing of large amounts of data.

It is also possible to simultaneously image multiple objects by imaging the multiple objects while moving them so that they pass through the area imaged by the imaging unit 12. This allows multiple objects to be evaluated efficiently. In this case, the image processing system 10 may be configured to be able to evaluate the entire side by moving the object while rotating it around the central axis. Also, while imaging a first object using the imaging unit 12, an evaluation of the presence or absence of features or peculiar parts of a second object may be performed using the individual evaluation unit 16 or the overall evaluation unit 18 at the same time.

[First Modification]
The following describes a first modified example of the first embodiment. Note that, in the modified example described below, explanations of points that a person skilled in the art can reasonably understand as having the same or similar configuration as the first embodiment or other modified examples will be omitted or simplified, and the description will focus on the differences.

The image processing device and image processing method according to the first embodiment performed a comprehensive evaluation of a given range based on the ratio of the number Y1 of pixels constituting the given range to the number X1 of images that were evaluated as containing features in the individual evaluation of the Y1 images.

The image processing method according to the first modification performs an overall evaluation not on a predetermined range, but on a partial range that is a part of the predetermined range, based on the ratio of the number Y1 of pixels constituting the partial range to the number X1 of images that are evaluated as containing features in the individual evaluation of the Y1 images.

The size of the partial range can be set appropriately. For example, the partial range is set so as to include the size of the peculiar part expected to be detected. For example, when the size of a typical peculiar part is often included in 16 divided images (a partial range consisting of 16 divided images of 4 rows and 4 columns), the partial range may be set as a part consisting of the same 16 divided images of 4 rows and 4 columns, and when the number of pixels evaluated as containing a feature in the individual evaluation is 4 or more, that is, when Y1 is 16, X1 is 4, and X1/Y1 is 0.25 or more, the overall evaluation unit 18 may be configured to evaluate that a peculiar part is included.

[Second Modification]
A second modification of the first embodiment will now be described.
The image processing method according to the second modification performs an evaluation by combining the overall evaluation described in the first embodiment and the overall evaluation described in the first modification.

Specifically, in step S4, even if X1/Y1 is less than the threshold (NO), if it is equal to or greater than a second threshold (an example of a "second predetermined value") that is smaller than this threshold, the overall evaluation unit 18 may be configured to evaluate the image as including a singular part if the ratio X2/Y2 of the number Y2 (<Y1) of the divided images constituting a second partial range that is part of a predetermined range composed of Y1 divided images and the number X2 of the Y2 images that were evaluated as including a feature in the individual evaluation is equal to or greater than a third threshold (an example of a "third predetermined value").

By adopting such a configuration, even if X1 within the specified range is small, it is possible to capture cases where divided images that are evaluated to contain features in some areas are densely packed, making it possible to reduce the rate at which singular parts are overlooked.

[Third Modification]
A third modified example of the first embodiment will now be described.
The image processing method according to the third modified example performs a comprehensive evaluation based on the pattern of positional relationships of a plurality of segmented images that have been evaluated as including features in the individual evaluations.

Figure 5 shows an example of a pattern of positional relationships between multiple segmented images. In the figure, black cells indicate segmented images that were evaluated as containing features in the individual evaluation of the segmented images (step S3), and white cells indicate segmented images that were evaluated as not containing features.

As shown in the figure, Fig. 5(A) and Fig. 5(B) show a pattern in which, out of the eight divided images located around a central divided image (an example of "Image A") among the divided images evaluated to contain features in a partial range consisting of nine divided images in three rows and three columns, two divided images are evaluated to contain features. Fig. 5(A) and Fig. 5(B) both show patterns in which there is one divided image evaluated to contain features in each column (vertical direction) and which include two divided images adjacent in a diagonal direction, Fig. 5(A) shows a pattern in which three divided images are connected diagonally, and Fig. 5(B) shows a pattern in which one divided image is connected diagonally with respect to the central divided image, and another divided image is connected vertically or horizontally.

The pattern shown in the figure may be rotated 90 degrees, or may be symmetrical up and down or left and right on the page.

When it is known in advance that there is a high possibility that the peculiar portion extends diagonally for divided images arranged in a matrix, as shown in the figure, the overall evaluation unit 18 may be configured to evaluate a divided image as including a peculiar portion if, in the individual evaluation, two or more of the eight divided images positioned around a central divided image are evaluated as including a feature, and at least one divided image is positioned diagonally relative to the divided image evaluated as including a feature.

The overall evaluation unit 18 may be configured to evaluate the other six divided images as including a unique portion even if the other six divided images are individually evaluated as including no features, and may be configured to evaluate the other six divided images as including a unique portion even if the other six divided images are individually evaluated as including features.

When the unique portion is linear and has a width narrower than the size of the divided image, it is preferable to perform the comprehensive evaluation as in the former. On the other hand, even if the unique portion is linear, when the width is wider than the size of the divided image, it is also possible to perform the comprehensive evaluation as in the latter. In other words, it is preferable to set a pattern of positional relationships of multiple divided images that have been evaluated as containing features in the individual evaluations, depending on the expected shape of the unique portion.

Such a comprehensive evaluation based on the pattern of the positional relationships of multiple divided images that have been evaluated as containing features in the individual evaluations may be performed in step S4 in the first embodiment, and when X1/Y1 is equal to or greater than a predetermined threshold value and a pattern such as that shown in FIG. 5(A) or FIG. 5(B) is detected, the detected object may be evaluated as containing a singular part.

Alternatively, instead of performing step S4 of checking whether X1/Y1 is equal to or greater than a predetermined threshold, after the individual evaluations, a comprehensive evaluation may be performed in which the detected object is evaluated as including a unique portion based on the positional relationship pattern of the multiple divided images. The image processing method in such a case will be described later.

[Fourth Modification]
A fourth modified example of the first embodiment will now be described.
The image processing method according to the fourth modified example is common to the third modified example in that a comprehensive evaluation is performed based on the pattern of positional relationships of multiple segment images that have been evaluated as containing features in the individual evaluations.

The inventors of the present application have noted that even in cases other than the case described in the third modified example (when, among the partial range consisting of nine divided images in three rows and three columns, two divided images are evaluated as including features out of the eight divided images located around the central divided image (an example of "Image A") among the divided images evaluated as including features out of the eight divided images that are evaluated as including features), there may be a singular part. Such a case is a pattern in which two divided images (an example of "Image A" and "Image B") evaluated as including features in the individual evaluation are adjacent, two other divided images (an example of "Image D" and "Image E") evaluated as including features in the individual evaluation are adjacent, and images A and B and images C and D are not adjacent to each other, and are separated by one divided image (an example of "Image C") evaluated as not including features in the individual evaluation.

In other words, when image A exists that is evaluated to include a feature in the individual evaluation, and is located around image A, image B exists that is evaluated to include the feature in the individual evaluation, and image C exists that is not located around image A but is located around image B, and image D exists that is evaluated to include a feature in the individual evaluation, and image E exists that is not located around image A and image B but is located around image C, and image D exists that is evaluated to include a feature in the individual evaluation, and image E exists that is not located around image A, image B, and image C but is located around image D, and image E exists that is evaluated to include a feature in the individual evaluation, and image E exists that is evaluated to include a feature in the individual evaluation, and image E exists that is not located around image A, image B, and image C but is located around image D. The overall evaluation unit 18 is configured to evaluate that the object includes a singular portion to be detected.

Figure 6 shows an example of such a pattern. Note that patterns obtained by rotating the pattern shown in the figure by 90 degrees and patterns obtained by making this pattern symmetrical in the vertical or horizontal directions on the page are not shown. In these figures, black cells indicate segmented images that were evaluated as containing features in the individual evaluation of the segmented images (step S3), and white cells indicate segmented images that were evaluated as not containing features.

Figure 6 (A) shows a pattern that includes a pattern in which images B, C, and D are arranged in a straight line in the row or column direction. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E3.

Figure 6 (B) shows a pattern that includes a pattern in which, of images B, C, and D, images B and C are arranged in a straight line in the row or column direction, and image D is arranged diagonally relative to image C. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E5.

Figure 6 (C) shows a pattern including, among images B, C, and D, a pattern in which images B and D are arranged in the same position in the row or column direction, and image C is arranged diagonally with respect to images B and D. Image A may be any one of images A1 to A3. Image E may be any one of images E1 to E4.

Figure 6 (D) shows a pattern that includes a pattern in which images B, C, and D are arranged in a straight line in a diagonal direction. Image A may be any one of images A1 to A5. Image E may be any one of images E1 to E5.

The pattern shown in FIG. 6(E) corresponds to a pattern that is rotated 90 degrees from FIG. 6(B) and then inverted symmetrically, and is therefore substantially identical to the pattern in FIG. 6(B), so a description thereof will be omitted.

Such an overall evaluation based on the pattern of the positional relationships of multiple divided images that have been evaluated as containing features in the individual evaluations may be performed together with or independently of the evaluation that X1/Y1 is equal to or greater than a predetermined threshold, which is performed in step S4 in the first embodiment.

With the image processing method and image processing device described above, even if a feature that should be included in the divided pixel corresponding to image C is overlooked, it is possible to detect a peculiar part based on the surrounding divided pixels, thereby preventing the peculiar part from being overlooked and improving detection accuracy.

[How to generate training data]
A method for generating the training data to be stored in the databases DB1 and DB2 will be described below.

Conventionally, when performing sensory testing on objects with complex structures that generate noise, images were taken while masking areas that generate noise, such as the boundaries between parts, in order to prevent overdetection.

However, if a unique portion is divided by a mask, the performance of detecting the unique portion decreases. For this reason, in this embodiment, when an object made up of multiple parts is imaged to obtain a base image in order to generate training data, a predetermined range of the object is imaged so as to include the multiple parts and their boundaries without using a mask. Here, multiple parts include one part made up of a flat surface and one part made up of a curved surface connected to it, or one part made up of a certain material and another part connected to this part and made of a different material.

Then, the base image is divided into multiple split images to generate training data. The split images that do not include singular parts are stored in database DB1. Here, these split images constitute a predetermined range that includes multiple parts and their boundaries, and the split images that become training data are generated without performing processing to remove noise, such as filtering. For this reason, these split images may contain noise.

Then, when capturing an image of the object to obtain a base image in step S1, the imaging unit 12 captures an image of a predetermined range of the object to include multiple parts and their boundaries, and a divided image is obtained in step S2 based on this image.

In step S3, each divided image is individually evaluated based on a trained model generated using machine learning to evaluate whether or not each divided image contains features. Since the trained model uses trained data consisting of divided images that constitute a specified range including multiple parts and their boundaries, it is possible to accurately individually evaluate whether or not each divided image that constitutes this specified range contains features.

Database DB2 stores segmented images that include unique parts or features. These segmented images also form a predetermined range that includes multiple parts and their boundaries. Figure 7 shows an example of unique parts H2 to H4 and an example of a segmented image generated based on them. As shown in this figure, the unique part may be larger than the size of the segmented image (in other words, the size of the segmented image is set so that it is smaller than some of the unique parts). Furthermore, even if the unique part is smaller than the size of the segmented image, the unique part may span multiple segmented images. The overall evaluation unit 18 shown in this embodiment performs an overall evaluation based on two or more segmented images, making it possible to improve the accuracy of detecting such unique parts.

[Scope of application]
The image processing system and image processing device according to the present disclosure can be applied to evaluate various objects. For example, the image processing system and image processing device can be used to detect foreign matter, scratches, wrinkles, etc. that may be contained in a transparent film as a specific part in a manufacturing process of a transparent film made of polyvinylidene chloride (PVDC).

It can also be used to detect specific areas such as foreign objects and scratches that may be contained in polymer films during the manufacturing process of polymer films for flat panel displays, etc.

It can also be used to detect cracks, scratches, and other defects that may be present in glass during the manufacturing process of glass products, etc., as unique parts. In this case, the glass product is illuminated and an image is taken, and the unique parts can be detected by utilizing the fact that the brightness of the unique parts is higher than that of other parts.

It can also be used to detect deformations or air bubbles on the membrane surface as unique areas during the manufacturing process of separation membranes for chromatography. In this case, the separation membrane is illuminated and an image is taken, and the unique areas can be detected by utilizing the fact that the brightness of the unique areas is lower or higher than that of other areas.

In addition, the present disclosure may be modified in various ways without departing from the spirit of the disclosure. For example, within the scope of ordinary creative ability of a person skilled in the art, some components in one embodiment may be added to another embodiment. Also, some components in one embodiment may be replaced with corresponding components in another embodiment.

For example, the image processing method according to the present disclosure includes:
Obtaining a plurality of images of an object, each image capturing a different portion of a predetermined range of the object;
applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected;
performing a comprehensive evaluation as to whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images;
Including,
The comprehensive evaluation evaluates whether or not the object includes a singular portion to be detected based on a positional relationship of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range in the individual evaluation.
It may be an image processing method.

For example, the image processing device according to the present disclosure includes:
A means for acquiring a plurality of images of an object, each of the images capturing a different portion of a predetermined range of the object;
a means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected;
a means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected based on whether or not the feature is included in two or more images among the plurality of images;
Including,
The means for performing the comprehensive evaluation is configured to evaluate, in the individual evaluation, whether or not the object includes a unique portion to be detected based on a positional relationship of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range.
It may be an image processing device.

In such an image processing method and image processing device,
Each of the plurality of images is an image of a rectangular area having a long side and a short side,
The overall evaluation may be configured to evaluate the object as including a peculiar portion to be detected when at least two of the images evaluated as including the feature in the individual evaluation are adjacent in a diagonal direction.

REFERENCE SIGNS LIST 10 Image processing system 12 Imaging unit 14 Image division unit 16 Individual evaluation unit 18 Overall evaluation unit 18 Overall evaluation unit 20 Image processing device 20 Image processing device 22 Processor 24 Memory unit 28 Display unit 32 Input unit DB1 Database DB2 Database

Claims (10)

acquiring a plurality of images of an object, each image including a different portion of a predetermined range of the object;
applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected;
performing a comprehensive evaluation as to whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images;
Including,
the comprehensive evaluation is performed by evaluating that the object includes a unique portion to be detected when a ratio X1/Y1 of a number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to a number X1 of the Y1 images that are evaluated in the individual evaluation as including the feature is equal to or greater than a first predetermined value ;
Furthermore, the comprehensive evaluation evaluates that the object includes a unique portion to be detected when the ratio X1/Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value smaller than the first predetermined value, and when a ratio X2/Y2 of a number Y2 (<Y1) of a plurality of images constituting the predetermined range or a second partial range that is a part of the partial range constituted by the Y1 images and a number X2 of images evaluated as including the feature in the individual evaluation among the Y2 images is equal to or greater than a third predetermined value.
Image processing methods. Dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the plurality of images each having the same image size;
The image processing method of claim 1 further comprising: The overall evaluation is as follows:
There is an image A that is evaluated as including the feature in the individual evaluation, and
In the individual evaluation, when it is evaluated that two or more of the plurality of images located around the image A include the feature,
Evaluating that the object includes a specific portion to be detected;
3. The image processing method according to claim 1 . acquiring a plurality of images of an object, each image including a different portion of a predetermined range of the object;
applying a predetermined image processing method acquired using machine learning to each of the plurality of images to individually evaluate whether each of the plurality of images includes a feature to be detected;
performing a comprehensive evaluation as to whether the object includes a specific portion to be detected based on whether the feature is included in two or more images of the plurality of images;
Including,
The comprehensive evaluation is performed in the following case: an image A evaluated to include the feature in the individual evaluation exists, and is located around the image A; an image B evaluated to include the feature in the individual evaluation exists, and is not located around the image A but around the image B; an image C evaluated to not include the feature in the individual evaluation exists, and is not located around the images A and B but around the image C; an image D evaluated to include the feature in the individual evaluation exists, and an image E evaluated to include the feature in the individual evaluation exists, and is not located around the images A, B, and C but is located around the image D ;
Image processing methods. The image processing method according to claim 1 , wherein the unique portion is a defect of the object. an image acquisition means for acquiring a plurality of images of different parts of a predetermined range of an object;
an individual evaluation means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to perform an individual evaluation of whether each of the plurality of images includes a feature to be detected;
a comprehensive evaluation means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected, based on whether or not the feature is included in two or more images among the plurality of images;
Equipped with
the comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when a ratio X1/Y1 of a number Y1 of the plurality of images constituting the predetermined range or a partial range that is a part of the predetermined range to a number X1 of the Y1 images evaluated by the individual evaluation means as including the feature is equal to or greater than a first predetermined value ;
Furthermore, the comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when the ratio X1/Y1 is less than the first predetermined value and is equal to or greater than a second predetermined value smaller than the first predetermined value, and when a ratio X2/Y2 between a number Y2 (<Y1) of a plurality of images constituting the predetermined range or a second partial range that is a part of the partial range constituted by the Y1 images and a number X2 of images evaluated by the individual evaluation means to include the feature among the Y2 images is equal to or greater than a third predetermined value.
Image processing device. a means for dividing an original image obtained by capturing an image of a predetermined range of the object, and generating the plurality of images each having the same image size;
The image processing apparatus according to claim 6 , further comprising: The comprehensive evaluation means includes:
There is an image A that has been evaluated by the individual evaluation means as including the feature, and
When the individual evaluation means evaluates that two or more of the plurality of images located around the image A include the feature,
The object is configured to evaluate that it includes a specific portion to be detected;
8. The image processing device according to claim 6 or 7. an image acquisition means for acquiring a plurality of images of different parts of a predetermined range of an object;
an individual evaluation means for applying a predetermined image processing method acquired by machine learning to each of the plurality of images to perform an individual evaluation of whether each of the plurality of images includes a feature to be detected;
a comprehensive evaluation means for performing a comprehensive evaluation as to whether or not the object includes a specific portion to be detected, based on whether or not the feature is included in two or more images among the plurality of images;
Equipped with
The comprehensive evaluation means is configured to evaluate that the object includes a unique portion to be detected when an image A evaluated by the individual evaluation means to include the feature exists, an image B evaluated by the individual evaluation means to include the feature exists, an image C not located around the image A but located around the image B, an image D evaluated by the individual evaluation means to include the feature exists, an image D not located around the images A and B but located around the image C, an image D evaluated by the individual evaluation means to include the feature exists, and an image E evaluated by the individual evaluation means to include the feature exists, an image E not located around the images A, B, and C but located around the image D ,
Image processing device. The image processing device according to claim 6 , wherein the unique portion is a defect of the object.

Images