JP7639744B2 - Inspection device and inspection method - Google Patents

Description

This disclosure relates to an inspection device and an inspection method.

A pass/fail judgment device is known that calculates feature values from each sample image based on the results of associating pass/fail judgment results with the sample image, and generates pass/fail judgment criteria by performing learning on the calculated feature values using a neural network (for example, Patent Document 1).

JP 2007-114843 A

Improved inspection accuracy is desirable for inspection devices that use learning models.

This disclosure can be realized in the following forms:

(1) According to one embodiment of the present disclosure, an inspection device is provided. The inspection device includes an image acquisition unit that acquires an image of an inspection target, a storage device that stores a learning model that has been trained using a normal image that shows a normal inspection target and is used to generate a comparison image that approximates the normal image, and an inspection unit that determines whether or not the inspection target has an abnormality by using the captured image and the comparison image. The learning model includes a generator that generates the comparison image by inputting a first noise and a second noise, the generator including a mapping network layer that performs nonlinear conversion on the first noise to generate a third noise, and a synthesis network layer that generates an image by using the second noise and the generated third noise, and generates the comparison image by repeating upsampling of the generated image multiple times.
According to the inspection device of this embodiment, the presence or absence of an abnormality can be determined using high-resolution image data, and the inspection accuracy of the inspection device can be improved.
(2) In the inspection device of the above aspect, the learning model does not need to include a classifier for discriminating between the captured image and the comparison image.
According to the inspection device of this aspect, the captured image and the comparison image can be compared with each other using a simple configuration, and the productivity of the inspection device can be increased.
(3) In the inspection device of the above aspect, the learning model may be a generative adversarial network further including a classifier that outputs a numerical value indicating the degree of difference between the comparison image and the captured image.
According to the inspection device of this aspect, it is possible to determine the presence or absence of an abnormality by using a captured image and a comparison image, utilizing a known configuration.
(4) In the inspection device of the above form, when the inspection unit determines that the captured image and the comparison image are not similar, the inspection unit may repeat a predetermined number of re-evaluations to determine whether or not there is an abnormality in the inspection object using a new comparison image generated by changing at least one of the first noise and the second noise.
According to the inspection device of this aspect, by changing the characteristics of the comparison image multiple times, it is possible to prevent abnormalities from being overlooked, and it is possible to improve the inspection accuracy.
(5) In the inspection device of the above form, the re-evaluation may include a first re-evaluation in which the second noise is fixed and the re-evaluation is performed a predetermined number of times using the comparison image generated by varying the first noise, and a second re-evaluation in which, after the first re-evaluation, the first noise is fixed and the re-evaluation is performed a predetermined number of times using the comparison image generated by varying the second noise.
According to this form of inspection device, by first performing the first re-evaluation, which changes the first noise that significantly changes the features in the image, the abnormality judgment can be completed earlier than when the second re-evaluation is performed first.
(6) In the inspection device of the above aspect, the generator may further include a weight adjustment layer that changes the first noise input to the mapping network layer so that the difference between the captured image and the comparison image is reduced.
According to the inspection device of this aspect, a comparison image that is similar to the captured image can be generated by a simple method of changing the first noise, without training the generator during inspection.
(7) In the inspection device of the above form, the inspection unit may determine that there is an abnormality in the inspection object when an accumulated value of the difference between the gradation value of a first pixel included in the captured image and the gradation value of a second pixel at a position corresponding to the first pixel in the comparison image is greater than a predetermined threshold value.
According to the inspection device of this aspect, it is possible to prevent the difference values of the RGB gradation values, which are easily recognized by humans, from becoming excessively large, compared to the case where the likelihood is calculated using the mean square error.
The present disclosure can also be realized in various forms other than an inspection device and an inspection method, such as an inspection program, a trained model, a learning method for a learning model, a manufacturing method for a part to be inspected, a manufacturing method for an inspection device, a control method for an inspection device, a learning program for a machine learning model, a non-transitory recording medium on which these computer programs are recorded, and the like.

FIG. 1 is an explanatory diagram illustrating a schematic overall configuration of an inspection system. FIG. 2 is a block diagram showing the internal functional configuration of the inspection apparatus. An explanatory diagram conceptually showing the configuration of a machine learning model. FIG. 1 is an explanatory diagram showing a detailed configuration of a machine learning model. 4 is a flowchart showing an inspection method executed by the inspection device. FIG. 2 is an explanatory diagram showing a connector as an example of an inspection target. FIG. 13 is an explanatory diagram showing an example of a captured image as a normal image showing a normal connector. FIG. 13 is an explanatory diagram showing an example of a captured image as an abnormality image showing a connector having an abnormality. FIG. 11 is an explanatory diagram showing a comparison image generated using a captured image as a normal image. FIG. 11 is an explanatory diagram showing a comparison image generated using a captured image as an abnormal image. 11 is an explanatory diagram showing a difference image generated using the difference in RGB gradation values between a captured image and a comparison image; FIG. FIG. 11 is an explanatory diagram illustrating a method for calculating a difference. FIG. 11 is an explanatory diagram showing a schematic configuration of a machine learning model provided in an inspection device according to a second embodiment.

A. First embodiment:
FIG. 1 is an explanatory diagram showing a schematic overall configuration of an inspection system 100. The inspection system 100 includes an inspection device 60 and a camera 70 as a first embodiment of the present disclosure. The inspection device 60 cooperates with the camera 70 to inspect the presence or absence of an abnormality of the inspection object using a captured image of the inspection object processed in the manufacturing line Ln. The inspection device 60 uses a trained machine learning model. The inspection object is, for example, an automobile part WK manufactured in the front-end process PR1. In this embodiment, a connector described later will be used as an example of the automobile part WK. The connector is a member used to connect pipes that circulate fuel, refrigerant, etc. in an automobile. The abnormality to be detected may include, for example, a poor connection of a connector, damage to a connector, poor fitting of a locking portion of a connector, damage to a pipe, and the like, and may include many types of abnormalities. In this embodiment, a machine learning model for unsupervised learning is used because it is difficult to prepare images of the inspection object having an abnormality.

The camera 70 captures an image of the inspection target and transmits the captured image of the inspection target to the inspection device 60. The camera 70 includes an imaging unit 72 and a communication unit 74. The imaging unit 72 captures an image of the part WK during transport from the front-end process PR1 to the back-end process PR2. In this embodiment, the imaging unit 72 captures an image of the part WK while it is mounted on the automobile. The captured image captured by the imaging unit 72 is output to the communication unit 74. The camera 70 is not limited to being separate from the inspection device 60, and may be integrated with the inspection device 60.

In this embodiment, the image captured by the imaging unit 72 is composed of an RGB input image signal composed of image signal components represented by R (red), G (green), and B (blue). Each image signal is assigned, for example, 8 bits each, totaling 24 bits of information. In this embodiment, the imaging unit 72 acquires a so-called 12 million pixel image of the entire part WK, which is 4000 pixels wide by 3000 pixels long. It is preferable that the resolution of the image acquired by the imaging unit 72 is determined taking into consideration the inspection range and defect detection accuracy required for the inspection device 60, the size of the defect to be detected, the image size that can be processed by the learning model, etc. In addition, in order to improve the inspection accuracy and expand the inspection range of the inspection device 60, it is preferable that the image of the inspection target has a high resolution. For example, in order to inspect the part WK in a state where it is mounted on a car, for example, the inspection target may be imaged from a distant position in order to move the camera 70 away from the car to avoid contact between the camera 70 and the car. In addition, there are cases where the imaging range is small or where small defects are detected in a large imaging range. In these cases, it is also preferable that the image of the inspection target has a high resolution.

The communication unit 74 transmits captured images to the inspection device 60 via wireless communication according to an arbitrary communication protocol. The communication unit 74 may also receive execution commands from the inspection device 60.

In this embodiment, the inspection device 60 and the camera 70 can transmit and receive data to each other by wireless communication, and can also transmit and receive data to the information processing device PC and the terminal devices PD1 and PD2 shown in FIG. 1. For example, the inspection device 60 can transmit inspection results to the information processing device PC and the terminal devices PD1 and PD2 in response to requests from the information processing device PC and the terminal devices PD1 and PD2. The inspection device 60 may be located in an area near the production line Ln as a local server, or may be located away from the production line Ln as a remote server. When the inspection device 60 is installed as a remote server, the captured image from the camera 70 and the inspection results to the terminal devices PD1 and PD2 and the information processing device PC are transmitted and received by the inspection device 60 via a wireless access point in the factory and a network such as an intranet or the Internet.

Figure 2 is a block diagram showing the internal functional configuration of the inspection device 60. The inspection device 60 includes a CPU 62 as a central processing unit, a storage device 64, and a communication unit 66. The CPU 62, the storage device 64, and the communication unit 66 are connected to each other via a bus 61 and are capable of bidirectional communication. The inspection device 60 may further include a display unit for displaying the inspection results, such as a liquid crystal display or a touch panel.

The communication unit 66 is an interface that controls communication between the camera 70 and the inspection device 66 to receive captured images to be used in the inspection via a network. In this embodiment, the communication unit 66 functions as an image acquisition unit that acquires captured images of the inspection target via the camera 70. In this embodiment, the communication unit 66 also transmits inspection results and data for displaying the inspection results to the terminal devices PD1 and PD2 and the information processing device PC. The communication unit 66 may also receive command signals requesting the execution of various processes from the terminal devices PD1 and PD2 and the information processing device PC. Note that in the case where the inspection device 60 includes an imaging unit 72, such as when the inspection device 60 is integrated with the camera 70, the imaging unit 72 functions as the image acquisition unit.

The storage device 64 is, for example, a RAM, a ROM, or a hard disk drive (HDD). Various programs for implementing the functions provided in this embodiment are stored in the HDD or ROM. Various programs read from the HDD or ROM are deployed on the RAM and executed by the CPU 62. The storage device 64 includes a learning model storage unit 642 for storing machine learning models. The storage device 64 temporarily stores image data acquired by the captured image acquisition unit, various images and calculation results generated by the inspection unit 622, and the like. The storage device 64 may be an optical disk, a solid state drive (SSD), a flash memory, or the like.

The CPU 62 functions as an inspection unit 622 by executing a program stored in the storage device 64. The inspection unit 622 uses a machine learning model stored in the learning model storage unit 642 to inspect whether or not there is an abnormality in the inspection target.

Figure 3 is an explanatory diagram conceptually illustrating the configuration of the machine learning model 80 stored in the learning model storage unit 642. In this embodiment, the learning model storage unit 642 stores a machine learning model 80 that has been trained using a normal image that indicates a normal inspection target. The machine learning model 80 includes a weight adjustment layer 82 and a generator 84.

The generator 84 is in a state where it has been sufficiently trained using a captured image of a normal test subject (hereinafter also referred to as a "normal image"). The generator 84 generates image data (hereinafter also referred to as a "comparison image") that is similar to the captured image by inputting a first noise Z1 and a second noise Z2 as latent variables. In this embodiment, a machine learning model also called StyleGAN is applied to the generator 84, as described below.

The first noise Z1 and the second noise Z2 are latent variables, for example, with about 100 dimensions, each element of which is a variable ranging from 0 to 1. The first noise Z1 and the second noise Z2 are determined by the inspection unit 622, for example, by random sampling from a uniform distribution or a normal distribution, and input to the generator 84.

When the inspection unit 622 acquires the comparison image PR and the captured image PT acquired by the imaging unit 72, it compares the captured image PT with the comparison image PR to determine whether or not there is an abnormality in the inspection object. If the captured image PT and the comparison image PR are similar, the inspection unit 622 determines that there is no abnormality in the inspection object. If the inspection unit 622 determines that the captured image PT and the comparison image PR are not similar, it changes at least one of the first noise Z1 and the second noise Z2 to generate a new comparison image PR, and performs a re-evaluation to determine again whether or not there is an abnormality in the inspection object.

The weight adjustment layer 82 adjusts the weight of the first noise Z1 input to the generator 84. In this embodiment, the weight adjustment layer 82 applies a fully connected layer that uses an activation function. The weight adjustment layer 82 changes the first noise Z1 in re-determination so that the difference between the captured image PT and the comparison image PR is reduced. As a result, the generator 84 is able to generate a comparison image PR that more closely resembles the captured image PT each time re-determination is performed without new learning during the inspection.

Figure 4 is an explanatory diagram showing the detailed configuration of the machine learning model 80. The configuration shown in Figure 4 corresponds to the configuration of range AR in Figure 3. As shown in Figure 4, a StyleGAN model is used in the generator 84. This makes it possible to generate a comparison image PR with a higher resolution than DCGAN, which generates image data with a resolution of, for example, about 64 pixels x 64 pixels.

The generator 84 includes a mapping network section 842 and multiple synthesis network sections 844. The mapping network layer 842 includes multiple fully connected layers (FC layers), for example, eight layers. The mapping network layer 842 generates a third noise W1 as a latent space by nonlinearly transforming the first noise Z1 as a latent variable using these eight fully connected layers. The generated third noise W1 is input to the synthesis network layer 844.

The synthesis network layer 844 generates a comparison image PR from a fixed value CV as a 4x4x512 tensor. The synthesis network layer 844 generates a high-resolution comparison image PR by performing upsampling on the image multiple times and gradually increasing the resolution. The synthesis network layer 844 is provided with a convolution processing layer and a normalization layer (AdaIN: Adaptive Instance Normalization) AL for each resolution to be upsampled. The synthesis network layer 844 incorporates style information and noise information each time upsampling is performed. Specifically, the second noise Z2 is input after the convolution processing layer to incorporate it as noise information that imparts probabilistic fluctuations, and the third noise W1 as a latent space generated by the mapping network layer 842 is incorporated as multidimensional style information via the normalization layer AL. The third noise W1 as a latent space and the first noise Z1 for generating the third noise W1 are factors that can affect major features in an image, and the second noise Z2 as noise information is a factor that can affect detailed features.

Figure 5 is a flowchart showing the inspection method executed by the inspection device 60. This flow starts when the part WK delivered from the previous process PR1 reaches the inspection process. Whether the part WK has reached the inspection process can be determined, for example, by image recognition of the part WK that has reached the inspection process using a camera 70 or the like.

In step S10, the inspection device 60 acquires the captured image PT. In this embodiment, the imaging unit 72 of the camera 70 acquires the captured image PT by capturing an image of the part WK to be inspected. The captured image PT acquired by the imaging unit 72 is transmitted to the inspection device 60 via the communication unit 74. The inspection device 60 acquires the captured image PT via the communication unit 66.

Figure 6 is an explanatory diagram showing a connector 30 as an example of an inspection target. As shown in Figure 6, the connector 30 has a main body 36. The main body 36 has a fitting portion 38. With the first pipeline 32 and the second pipeline 34 inserted into the main body 36, the fitting portion 38 is pushed toward the connection portion of the first pipeline 32 and the second pipeline 34, whereby the first pipeline 32 and the second pipeline 34 are fixed in a connected state.

Figure 7 is an explanatory diagram showing an example of a captured image PTg as a normal image showing a normal connector 30. Figure 8 is an explanatory diagram showing an example of a captured image PTn as an abnormal image showing a connector 30 having an abnormality. As shown in Figure 7, the captured image PTg shows the connector 30 in a normal state in which the fitting portion 38 is fully pushed toward the connection portion between the first pipeline 32 and the second pipeline 34, and the first pipeline 32 and the second pipeline 34 are fixed. As shown in Figure 8, the captured image PTn shows the connector 30 in a state in which the fitting portion 38 is not fully pushed in and protrudes from the main body 36. In the captured image PTn, the protruding portion P1 of the fitting portion 38 protruding from the main body 36 and the cutout portion P2 at a position different from that of the connector 30 in a normal state are captured. In the following explanation, an example will be explained in which the inspection device 60 judges that the fitting portion 38 of the connector 30 is not fully pushed in as an abnormality.

In step S20, the inspection unit 622 inputs the randomly sampled first noise Z1 and second noise Z2 to the generator 84 of the machine learning model 80 to generate a comparison image PR. The first noise Z1 is input to the generator 84 via the weight adjustment layer 82 and nonlinearly converted to the third noise W1 as a latent space via the mapping network layer 842 shown in FIG. 4. The synthesis network layer 844 generates an image from a fixed value CV, and while incorporating the third noise W1 as style information and the second noise Z2 as noise information, repeatedly upsamples the image to generate a comparison image PR of 1024 pixels x 1024 pixels.

Figure 9 is an explanatory diagram showing a comparison image PRg generated using a captured image PTg as a normal image. Figure 10 is an explanatory diagram showing a comparison image PRn generated using a captured image PTn as an abnormal image. As shown in Figure 9, it can be seen that the comparison image PRg is generated in a state that is similar to the normal image. Note that the comparison image PRn shown in Figure 10 is an example of a comparison image generated when re-evaluation is performed until the upper limit number of times is reached. As shown in Figure 10, it can be seen that the comparison image PRn has a generated portion P3 generated so that the protruding portion P1 in the captured image PTn shown in Figure 8 is similar to the normal image, and a generated portion P4 generated so that the cutout portion P2 is similar to the normal image.

In step S30, the inspection unit 622 compares the captured image PT with the generated comparison image PR. In this embodiment, the inspection unit 622 performs the comparison using the integrated value of the difference between the sum of the RGB gradation values of the pixels of the captured image PT and the sum of the RGB gradation values of the pixels of the comparison image PR at positions corresponding to the pixels of the captured image PT. The pixels of the captured image PT are also called "first pixels," and the pixels of the comparison image PR are also called "second pixels."

Figure 11 is an explanatory diagram showing a difference image CP generated using the difference in RGB gradation values between the captured image PTn and the comparison image PRn. From Figure 11, it can be seen that the difference image CP has areas with large differences in RGB gradation values, such as difference area P5, which is the difference between protruding portion P1 and generated portion P3, and difference area P6, which is the difference between cutout portion P2 and generated portion P4.

FIG. 12 is an explanatory diagram showing a method of calculating the difference between the captured image PTn and the comparison image PRn. In FIG. 12, the number of pixels is shown to be less than the number of pixels in the actual image in order to facilitate understanding of the technology. Table TB1 shown in FIG. 12 conceptually shows the RGB gradation values of each pixel included in the captured image PTn. Table TB2 conceptually shows the RGB gradation values of each pixel included in the comparison image PRn, and Table TB3 conceptually shows the RGB gradation values of each pixel included in the difference image CP. Each value shown in Table TB1 indicates the total value of the RGB gradation values of each pixel of the captured image PTn. Similarly, each value shown in Table TB2 indicates the total value of the RGB gradation values of each pixel of the comparison image PRn. Each value shown in Table TB3 indicates the difference value between the total value of the RGB gradation values for each pixel between Table TB1 and Table TB2.

As shown in Table TB3, the inspection unit 622 calculates the difference between the sum of the RGB gradation values of the captured image PTn and the sum of the RGB gradation values of the generated comparison image PRn for each pixel. If the captured image PT is an abnormal image, the comparison image is generated by the generator 84, and then re-evaluation is repeated to regenerate the comparison image so as to approximate the captured image PTn as an abnormal image. Therefore, as shown in Table TB3, for example, the difference between the captured image and the comparison image remains as a difference in RGB gradation values. The inspection unit 622 adds up the calculated RGB gradation values for each pixel for all pixels. In the example of Table TB3 in FIG. 12, the added up value of the RGB gradation values is 11.

Returning to FIG. 5, in step S40, the inspection unit 622 uses the calculated integrated value of the RGB gradation values of all pixels to determine whether the inspection object has an abnormality. In this embodiment, the inspection unit 622 determines whether the inspection object has an abnormality by comparing the calculated integrated value of the RGB gradation values of all pixels with a predetermined threshold value stored in the storage device 64. If the integrated value is lower than the threshold value (S40: YES), the process proceeds to step S42, where the inspection unit 622 determines that the inspection object has no abnormality and ends the process. If the integrated value is equal to or greater than the threshold value (S40: NO), the process proceeds to step S50.

In step S50, the inspection unit 622 judges whether the number of inspections N has reached a predetermined upper limit. The number of inspections corresponds to the number of re-judgments plus 1. The upper limit of the number of inspections N can be set arbitrarily based on the takt time required for the inspection device 60, and is set to 100 in this embodiment. If the number of inspections N reaches 100 (S50: YES), the process proceeds to step S52, where the inspection unit 622 judges that there is an abnormality in the inspection object and ends the process. The case where the number of inspections N reaches 100 corresponds to the case where it is judged that none of the 100 comparison images PR generated by changing the first noise Z1 and the second noise Z2 is similar to the captured image PT. If the number of inspections N is less than 100 (S50: NO), the process proceeds to step S60.

In step S60, the inspection unit 622 determines whether the number of inspections N has reached a predetermined number of switchings. The number of switchings is a threshold value used to determine whether to switch the combination method of the first noise Z1 and the second noise Z2. The number of switchings can be set arbitrarily, and in this embodiment, the number of switchings is set to 50, which is half the upper limit. If the number of inspections N is less than 50 (S60: YES), the process proceeds to step S62, where the inspection unit 622 fixes the second noise Z2 to the value input in step S20, and newly determines the first noise Z1 by random sampling. If the number of inspections N is 50 or more (S60: NO), the process proceeds to step S64, where the inspection unit 622 fixes the first noise Z1 to the value at the time when the number of inspections N is 50, and newly determines the second noise Z2 by random sampling. Among the re-judgments, the re-judgment using a comparison image PR newly generated by fixing the second noise Z2 and varying the first noise Z1 is also referred to as a "first re-judgment," and the re-judgment using a comparison image PR newly generated by fixing the first noise Z1 and varying the second noise Z2 is also referred to as a "second re-judgment." When the first noise Z1 and the second noise Z2 are newly determined in step S62 or step S64, the inspection unit 622 increments the inspection count N by 1 and returns to step S20.

As described above, the inspection device 60 of this embodiment includes an image acquisition unit that acquires an image PT of an inspection target, a memory device 64 that stores a machine learning model 80 that has been trained using a normal image showing a normal inspection target and is used to generate a comparison image PR that approximates the normal image, and an inspection unit 622 that determines whether or not an abnormality exists in the inspection target using the image PT and the comparison image PR. The machine learning model 80 includes a generator 84 that generates a comparison image PR by inputting a first noise Z1 and a second noise Z2. The generator 84 includes a mapping network layer 842 that performs nonlinear conversion on the first noise Z1 to generate a third noise W1, and a synthesis network layer 844 that generates an image using the second noise Z2 and the generated third noise W1, and generates a comparison image PR by repeating upsampling of the generated image multiple times. The generator 84 of the machine learning model 80 includes a mapping network layer 842 that uses StyleGAN to perform nonlinear conversion on the input first noise Z1 to generate a third noise W1, and a synthesis network layer 844 that generates an image using the second noise Z2 and the third noise W1, and generates a comparison image PR by repeating upsampling multiple times to increase the resolution of the generated image. Therefore, in the inspection device 60 that uses the machine learning model 80, by adopting a StyleGAN network structure for the generator 84, it is possible to determine the presence or absence of anomalies using high-resolution image data.

According to the inspection device 60 of this embodiment, the machine learning model 80 does not include a classifier 86 for distinguishing between the captured image PT and the comparison image PR. Therefore, the captured image PT and the comparison image PR can be compared with a simple configuration, and the productivity of the inspection device 60 can be increased.

According to the inspection device 60 of this embodiment, when the inspection unit 622 determines that there is no abnormality in the inspection object, it repeats a predetermined number of times a re-determination of the presence or absence of an abnormality in the inspection object using a new comparison image PR generated by changing at least one of the first noise Z1 and the second noise Z2. By changing the characteristics of the comparison image PR multiple times, it is possible to prevent abnormalities from being overlooked, and the inspection accuracy can be improved.

According to the inspection device 60 of this embodiment, the re-determination performed by the inspection unit 622 includes a first re-determination in which the first noise Z1 is fixed and the second noise Z2 is changed to generate a comparison image PR, and the first re-determination is performed a predetermined number of times, and a second re-determination in which the second noise Z2 is fixed and the first noise Z1 is changed to generate a comparison image PR, and the first re-determination is performed up to a predetermined upper limit number of times after the first re-determination. By changing only one of the first noise Z1 and the second noise Z2 to generate a new comparison image PR and perform the re-determination, it is possible to suppress a large deviation between the features of the captured image PT and the features of the comparison image PR at the time of the re-determination. In addition, by performing the first re-determination in which the first noise Z1, which changes the large features in the image, is changed first, and then performing the second re-determination in which the second noise Z2, which changes the detailed features, is changed, the possibility of determining that there is no abnormality within the number of switching times is increased, and the abnormality determination can be terminated early, thereby increasing the productivity of the inspection process.

According to the inspection device 60 of this embodiment, the generator 84 further includes a weight adjustment layer 82 that changes the first noise Z1 input to the mapping network layer 842 so that the difference between the captured image PT and the comparison image PR is reduced. Therefore, during the inspection process by the inspection device 60, by using the simple method of changing the first noise Z1, it is possible to generate a comparison image PR that is close to the captured image PT without training the generator 84 during the inspection.

According to the inspection device 60 of this embodiment, the inspection unit 622 calculates the difference between the RGB gradation value of a first pixel included in the captured image PT and the RGB gradation value of a second pixel at a position corresponding to the first pixel in the comparison image PR, and determines that there is an abnormality in the inspection object when the accumulated value of the calculated RGB gradation value differences is greater than a predetermined threshold value. Therefore, compared to the case where the difference value is calculated using the mean squared error (MSE) of the RGB gradation values at each pixel between the captured image PT and the comparison image PR, for example, it is possible to prevent the difference value of the RGB gradation values that are easily recognized by humans from becoming excessively large, and to determine an abnormality while properly considering the change in the color of the RGB gradation values, and to obtain inspection results that are approximately equivalent to those of manual inspection.

B. Second embodiment:
FIG. 13 is an explanatory diagram showing a schematic configuration of a machine learning model 80b provided in the inspection device 60 as the second embodiment. As shown in FIG. 13, the machine learning model 80b is a model that can be used as an anomaly detection method using Generative Adversarial Networks (GAN). The machine learning model 80b may also be called Anomaly Detection with Generative Adversarial Networks (AnoGAN). The machine learning model 80b differs from the machine learning model 80 shown in the first embodiment in that it further includes a discriminator 86 and a determiner 88. In this embodiment, the inspection unit 622 determines the presence or absence of an anomaly using the output result from the determiner 88.

The classifier 86 is in a state of being sufficiently trained to be able to distinguish between a normal image showing a normal test object and the generated comparison image PR. The classifier 86 outputs a likelihood indicating the degree of difference between the comparison image PR and the captured image PT. When the classifier 86 acquires the comparison image PR and the captured image PT acquired by the imaging unit 72, it compares the captured image PT with the comparison image PR and outputs the likelihood, which is a numerical value indicating the degree of difference between the comparison image PR and the captured image PT, to the determiner 88. In this embodiment, the likelihood is a value indicating the probability that the captured image PT has an abnormality, and for example, when the likelihood is a high value, it can be determined that the test object has an abnormality. In this embodiment, the classifier 86 calculates, for example, the sum of a first difference value, which is the difference between the captured image PT and the comparison image PR, and a second difference value, which is the difference between the output value of the intermediate layer when the comparison image PR is input to the classifier 86 and the output value of the intermediate layer when the captured image PT is input to the classifier 86, and outputs it as the likelihood. However, if the likelihood indicates the probability that the product is normal, a low value may be detected as an abnormality.

The determiner 88 uses the likelihood output from the discriminator 86 to determine whether or not the test object has an abnormality. In step S40, if the likelihood is lower than a predetermined threshold, the determiner 88 may determine that the captured image PT and the comparison image PR are similar and proceed to step S42, and if the likelihood is equal to or greater than the threshold, the determiner 88 may determine that the captured image PT and the comparison image PR are not similar and proceed to step S50. The result of the determination of the abnormality by the determiner 88 of the machine learning model 80b is output to the inspection unit 622, which determines whether or not the test object has an abnormality based on the result of the determiner 88. Note that in the machine learning model 80b, the generator 84 and the discriminator 86 do not need to learn while the test is being performed.

According to the inspection device 60 of this embodiment, the machine learning model 80b is a generative adversarial network that further includes a classifier 86 that outputs a numerical value indicating the degree of difference between the captured image PT and the comparison image PR. Therefore, by utilizing a known configuration, it is possible to determine the presence or absence of an abnormality using the captured image PT and the comparison image PR.

C. Other embodiments:
(C1) In the above first embodiment, the inspection object is described as an example of a connector as a part WK for an automobile. In contrast, the inspection object is not limited to a connector used in an automobile, but may be various parts included in an automobile, such as the body or parts of the automobile. The inspection object is not limited to only automobile parts, but may be parts used in various moving bodies such as railroad cars, aircraft, and ships, or may be various manufactured products other than moving bodies.

(C2) In the above first embodiment, the inspection unit 622 may calculate the difference between the sum of the RGB gradation values of the captured image PTn and the sum of the RGB gradation values of the generated comparison image PRn for each pixel. In contrast, the inspection unit 622 may compare the difference between the sum of the RGB gradation values of the captured image PTn and the sum of the RGB gradation values of the generated comparison image PRn with a predetermined threshold, accumulate the number of pixels where the difference is greater than or equal to the threshold, and determine that there is an abnormality if the number of pixels is greater than or equal to a predetermined value, and determine that there is no abnormality if it is less than the predetermined value. In addition, the inspection unit 622 may calculate the difference between each of the R, G, and B gradation values of the RGB gradation values for each pixel between the captured image PT and the comparison image PR, and then use the value obtained by accumulating the differences between the calculated R, G, and B gradation values. The inspection unit 622 may calculate the likelihood using the mean squared error (MSE) of the RGB gradation values for each pixel between the captured image PT and the comparison image PR.

The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the Summary of the Invention column can be replaced or combined as appropriate to solve some or all of the above-described problems or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

30...connector, 32...first pipeline, 34...second pipeline, 36...main body, 38...fitting section, 60...inspection device, 61...bus, 62...CPU, 64...storage device, 66...communication section, 70...camera, 72...imaging section, 74...communication section, 80, 80b...machine learning model, 82...weight adjustment layer, 84...generator, 86...classifier, 88...determinator, 100...inspection system, 622...inspection section, 642...learning model memory section, 842...mapping network layer, 844...synthesis network Network layer, AL... normalization layer, AR... range, CP... difference image, CV... fixed value, Ln... production line, P1... protruding part, P2... cutout part, P3, P4... production part, P5, P6... difference part, PC... information processing device, PD1, PD2... terminal device, PR, PRg, PRn... comparison image, PR1... previous process, PR2... subsequent process, PT, PTg, PTn... captured image, TB1-TB3... table, W1... third noise, WK... part, Z1... first noise, Z2... second noise

Claims (7)

An inspection device comprising:
an image acquisition unit that acquires an image of an object to be inspected;
A storage device that stores a learning model that has been trained using a normal image that shows a normal inspection target, and is used to generate a comparison image that is similar to the normal image;
an inspection unit that determines whether or not an abnormality exists in the inspection object by using the captured image and the comparison image,
The learning model is
A generator for generating the comparison image by inputting a first noise and a second noise,
a mapping network layer that nonlinearly transforms the first noise to generate a third noise;
a synthesis network layer that generates an image using the second noise and the generated third noise, and generates the comparison image by repeating upsampling of the generated image a number of times;
When it is determined that the captured image and the comparison image are not similar to each other, the inspection unit repeats a re-determination of the presence or absence of an abnormality in the inspection object a predetermined number of times using a new comparison image generated by changing at least one of the first noise and the second noise.
Inspection equipment. 2. The inspection device according to claim 1,
The learning model does not include a classifier for discriminating between the captured image and the comparison image.
Inspection equipment. 2. The inspection device according to claim 1,
The re-determination is
a first re-determination in which the second noise is fixed and the re-determination using the comparison image generated by changing the first noise is performed a predetermined number of times;
and after performing the first re-determination, performing the re-determination a predetermined number of times using the comparison image generated by fixing the first noise and varying the second noise.
Inspection equipment. 2. The inspection device according to claim 1,
the inspection unit determines that the inspection object is abnormal when an integrated value of a difference between a gradation value of a first pixel included in the captured image and a gradation value of a second pixel at a position corresponding to the first pixel in the comparison image is greater than a predetermined threshold value.
Inspection equipment. 2. The inspection device according to claim 1 ,
The learning model is
The generator ; and
a classifier that outputs a likelihood indicating a degree of difference between the captured image and the comparison image;
a judger that judges the presence or absence of an anomaly in the inspection target by using the likelihood output from the classifier ,
The inspection unit determines whether or not the inspection target has an abnormality based on the result of the determiner.
Inspection equipment. 2. The inspection device according to claim 1 ,
The generator further includes a weight adjustment layer that changes the first noise input to the mapping network layer so that a difference between the captured image and the comparison image is reduced.
Inspection equipment. 1. A testing method comprising:
Acquire an image of the inspection object;
generating a comparison image that is similar to the normal image using a learning model that has been trained using a normal image that shows a normal object;
determining whether or not an abnormality exists in the inspection target using the captured image and the comparison image;
The learning model is
A generator for generating the comparison image by inputting a first noise and a second noise,
a mapping network layer that nonlinearly transforms the first noise to generate a third noise;
a synthesis network layer that generates an image using the second noise and the generated third noise, and generates the comparison image by repeating upsampling of the generated image a number of times;
when it is determined that the captured image and the comparison image are not similar to each other, a re-determination is performed a predetermined number of times to determine the presence or absence of an abnormality in the inspection object using a new comparison image generated by changing at least one of the first noise and the second noise.
Testing method.

Images