JP7675504B2 - Inspection system, inspection device, inspection program, and inspection method - Google Patents

Description

The present disclosure relates to technology for inspecting an inspection object, and in particular to an inspection system for inspecting an inspection object, a learning device, a learning program, a learning method that can be used in the inspection system, an inspection device, an inspection program, and an inspection method that can be used in the inspection system.

When manufacturing sanitary ceramics such as toilet tanks, defects such as peeling of the glaze and cracks can occur during processes such as glazing and firing. Traditionally, skilled inspectors have visually inspected the appearance, but securing sufficient personnel is difficult and the training burden is high, making it a major challenge to eliminate dependency on individual personnel.

To automatically detect such defects, technology has been proposed to determine the surface condition from captured images of the object being inspected (see, for example, Patent Document 1).

JP 2006-047098 A

If defects cannot be detected and products are shipped with defects, it will require the time and effort of recalling the product, shipping a replacement, and reprocessing it, and it will also reduce the reliability of the product. Therefore, there is a need to further improve the accuracy of automatically detecting defects.

This disclosure has been made in light of these issues, and its purpose is to improve the accuracy of inspecting test objects.

In order to solve the above problem, an inspection system according to an embodiment of the present disclosure includes a learning device that learns a detector for detecting a detection target portion of an inspection target, and an inspection device that detects the detection target portion of an inspection target using a detector learned by the learning device. The learning device includes a learning image acquisition unit that acquires an image of the inspection target including the detection target portion, a learning data generation unit that generates learning data from an image of the detection target portion included in the image of the inspection target acquired by the learning image acquisition unit, and a learning unit that trains the detector using the learning data generated by the learning data generation unit. The learning data generation unit divides at least a portion of the images of the detection target portions into multiple parts and generates learning data from each of the divided images of the detection target portions, and the inspection device includes an inspection image acquisition unit that acquires an image of the inspection target, and a detection unit that uses a detector to detect the detection target portion from the image acquired by the inspection image acquisition unit.

Another aspect of the present disclosure is a learning device. This device includes a learning image acquisition unit that acquires captured images of an inspection target including a detection target portion, a learning data generation unit that generates learning data from images of the detection target portion included in the captured images of the inspection target acquired by the learning image acquisition unit, and a learning unit that uses the learning data generated by the learning data generation unit to train a detector for detecting the detection target portion of the inspection target. The learning data generation unit divides at least a portion of the images of the detection target portions into multiple parts, and generates learning data from each of the divided images of the detection target portions.

Yet another aspect of the present disclosure is a learning program. This program causes a computer to function as a learning image acquisition unit that acquires captured images of an inspection target including a detection target portion, a learning data generation unit that generates learning data from images of the detection target portion included in the captured images of the inspection target acquired by the learning image acquisition unit, and a learning unit that uses the learning data generated by the learning data generation unit to train a detector for detecting the detection target portion of the inspection target, and the learning data generation unit divides at least a portion of the images of the detection target portions into multiple parts and generates learning data from each of the divided images of the detection target portions.

Yet another aspect of the present disclosure is a learning method. This method causes a computer to execute the steps of acquiring an image of an inspection target including a detection target portion, generating learning data from an image of the detection target portion included in the acquired image of the inspection target, and training a detector for detecting the detection target portion of the inspection target using the generated learning data. In the learning step, at least a portion of the images of the detection target portions are divided into a plurality of portions, and learning data is generated from each of the divided images of the detection target portions.

Yet another aspect of the present disclosure is an inspection device. This device includes an inspection image acquisition unit that acquires an image of an object to be inspected, and a detection unit that detects the detection target part from the image acquired by the inspection image acquisition unit using a detector trained using learning data generated by dividing an image of the detection target part included in the image of the object to be inspected into multiple parts.

Yet another aspect of the present disclosure is an inspection program. This program causes a computer to function as an inspection image acquisition unit that acquires an image of an inspection target, and a detection unit that detects a detection target portion from an image acquired by the inspection image acquisition unit using a detector trained using training data generated by dividing an image of a detection target portion included in the image of the inspection target into multiple parts.

Yet another aspect of the present disclosure is an inspection method. This method causes a computer to execute the steps of acquiring an image of an object to be inspected, and detecting the object to be detected from the acquired image using a detector trained using training data generated by dividing an image of the object to be detected, which is included in the image of the object to be inspected, into multiple parts.

In addition, any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention.

FIG. 1 is a diagram showing a configuration of an inspection system according to an embodiment. FIG. 13 is a diagram showing an example of a captured image of the surface of a tank. FIG. 13 is a diagram showing an example of a captured image of the surface of a tank. 4 is a flowchart showing a procedure of a learning method according to an embodiment. 4 is a flowchart showing a procedure of an inspection method according to an embodiment. FIG. 1 is a diagram illustrating a configuration of a learning device according to an embodiment. FIG. 1 is a diagram showing a configuration of an inspection device according to an embodiment.

As an embodiment of the present disclosure, a technique for inspecting the appearance of a tank, which is an example of sanitary ware, will be described. A learning device according to the embodiment learns a defect detector for detecting defects such as cracks that occur on the surface of the tank. An inspection device according to the embodiment detects defects from captured images of the tank using the defect detector learned by the learning device. This improves the efficiency and accuracy of the inspection, significantly reducing the burden on the inspector. In addition, the rate at which passing products are judged to be failing can be reduced, reducing the effort required to review failing products. In addition, accurate inspections can be performed regardless of the experience or skill of the inspector, eliminating the dependency on individual inspections, reducing the shortage of inspectors, and stabilizing product quality.

Figure 1 shows the configuration of an inspection system according to an embodiment. The inspection system 1 includes a tank 2 to be inspected, an imaging device 3 for imaging the exterior of the tank 2, a learning device 100 for learning a defect detector for detecting defects on the surface of the tank 2, an inspection device 200 for inspecting the surface of the tank 2 using the defect detector, and the Internet 4, which is an example of a communication network connecting these devices.

Figure 2 shows an example of a captured image of the surface of the tank 2. Figure 2(a) shows an image of a crack that has occurred on the surface of the tank 2. The captured image 10 contains images of four cracks 11a, 11b, 11c, and 11d. The learning device 100 assigns annotations (teacher labels) indicating that the cracks contained in the captured image 10 are cracks, and trains the defect detector by supervised learning.

In order to improve the generalization performance (detection performance for unknown data) of the defect detector, it is necessary to train the defect detector using images of a wide variety of defects as training data. However, since the rate at which defects occur on the surface of the tank 2 during the manufacture of the tank 2 is very low, it is difficult to collect a large number of images of defects actually occurring on the surface of the tank 2 to generate training data. In order to solve this problem, in this embodiment, an image of one defect is divided into multiple parts, and training data is generated from each of the images of the divided defects. This makes it possible to efficiently generate a large amount of training data and train the defect detector, thereby improving the generalization performance and accuracy of the defect detector. When detecting defects using a defect detector trained in this way, even if there is one defect, there is a possibility that each part of the subdivided defect will be detected as multiple pieces, but after detection, adjacent defects can be grouped together. This method is particularly effective when training a detector for defects such as cracks, where the shape and other characteristics of a single defect may differ depending on the position.

As shown in FIG. 2(a), the learning device 100 may add annotations 12a, 12b, 12c, and 12d to the images of the four cracks 11a, 11b, 11c, and 11d, respectively. In this embodiment, however, as shown in FIG. 2(b), among the images of the four cracks 11a, 11b, 11c, and 11d, three images of 11a, 11b, and 11c are divided into a plurality of images, and annotations are added to each of the divided images of the plurality of cracks to generate separate learning data. For example, the image of the crack 11a is divided into three, and annotations 12a1, 12a2, and 12a3 are added to each of the images to generate three learning data. This makes it possible to increase the number of learning data to which annotations are added, thereby improving the generalization performance and accuracy of the defect detector.

The learning device 100 may receive annotations from the person in charge, or may automatically or semi-automatically assign annotations. The learning device 100 may receive the position, size, and type of defects included in the captured image 10 from the person in charge, and may determine whether to generate learning data after dividing the image of the defects or to generate learning data without dividing it, depending on the size, type, number, etc. of the specified defects, and if dividing it, the number of divisions and the size and shape of each divided area. For example, the crack 11d shown in FIG. 2(a) has a monotonous shape regardless of the position, so the annotation 12d may be assigned to the entire crack without dividing it. The cracks 11a, 11b, and 11c have a variety of shapes, so they may be divided into a number of parts according to the size and shape changes of the crack, and annotations may be assigned to each part.

Figure 3 shows an example of a captured image of the surface of the tank 2. Figure 3(a) shows an image of a crack that has occurred on the surface of the tank 2. In addition to the brightness contrast caused by defects such as cracks, the captured image of the surface of the tank 2 may also include brightness contrast caused by unevenness on the surface of the tank 2 and ambient light at the time of capture. If the defect detector erroneously learns such features of the background image that should not be detected, the detection accuracy may decrease. In order to solve such problems, in this embodiment, image processing is performed on the captured image before generating learning data from the captured image of the inspection target. This preprocessing may be image processing that can smooth the brightness by reducing the contrast of the image of a portion that is not a detection target portion such as a defect. For example, a blur filter may be applied to the entire image. The blur filter may be, for example, a box filter that takes the average of pixel values within the range of the kernel, a Gaussian filter that changes the weight depending on the distance from the pixel of interest, a median filter that adopts the median value of all pixels within the range of the kernel, a bilateral filter that is a Gaussian filter weighted by normal distribution, or the like. The preprocessing may be image processing that reduces the resolution of the image. This allows for simple image processing to improve learning efficiency, making it possible to generate highly accurate defect detectors in a short period of time.

Figure 3(b) shows the image shown in Figure 3(a) after applying a blur filter. The background is blurred to reduce contrast, which reduces the risk of erroneous learning of background features. The contrast of the defects is also slightly reduced, but their features are preserved, allowing the defect detector to learn the features of the defects.

Image processing may be performed under conditions such that the contrast characteristics of the detection target area are not lost and the contrast of areas that are not the detection target area is sufficiently smoothed. The learning device 100 may obtain the luminance distribution of the detection target area and the luminance distribution of the background that is not the detection target area, and determine conditions such as the type of image processing and kernel size based on the respective luminance distributions. When inspecting multiple captured images of the same type of inspection target captured in a similar environment, the learning device 100 may apply image processing under the same conditions.

When the imaging device 3 captures an image of the surface of the tank 2 to be inspected, the imaging may be performed in an imaging environment in which the contrast of non-defective areas is sufficiently low. For example, the tank 2 may be irradiated with light at an illuminance according to the color and reflectance of the surface of the tank 2. The focal length, resolution, aperture, etc. of the imaging device 3 may also be adjusted.

FIG. 4 is a flowchart showing the procedure of the learning method according to the embodiment. The learning device 100 acquires an image of an inspection target having a defect (S10), and performs preprocessing such as blurring on the acquired image (S12). The learning device 100 divides at least a part of the image of the defect included in the captured image into a plurality of images (S14), and generates learning data by adding annotations to each of the divided plurality of defect images (S16). The learning device 100 may add different annotations for each type of defect. This allows the defect detector to learn features for each type of defect, so that the defect detector can be used to detect defects from the image of the inspection target and simultaneously determine the type of defect. The learning device 100 uses the generated learning data to train the defect detector through supervised learning. The learning device 100 may apply any known learning algorithm when training the defect detector.

Figure 5 is a flowchart showing the steps of the inspection method according to the embodiment. The inspection device 200 acquires a captured image of the inspection object (S30), and performs image processing on the captured image similar to that performed by the learning device 100 when generating the learning data (S32). The inspection device 200 detects defects in the captured image using a trained defect detector (S34), and outputs the detection results (S36).

The inventors used the above method to train a defect detector using images of 212 defective and rejected products of tank 2, and then used the trained defect detector to inspect 500 passing products and 23 failing products of tank 2. There were zero cases of failing products being erroneously determined to be passing products. This shows that the technology of this embodiment can minimize the possibility of erroneously shipping a failing product. There were also seven cases of passing products being erroneously determined to be failing products, for an accuracy rate of 98.6%. This shows that it is possible to significantly reduce the effort required for inspectors to check failing products, and also to significantly reduce the proportion of passing products that are lost due to failure to ship.

Figure 6 shows the configuration of a learning device 100 according to an embodiment. The learning device 100 includes a display device 112, an input device 113, a communication device 114, a processing device 120, and a storage device 130. The learning device 100 may be a server device, a device such as a personal computer, or a mobile terminal such as a mobile phone terminal, a smartphone, or a tablet terminal.

The display device 112 displays a screen generated by the processing device 120. The display device 112 may be a liquid crystal display device, an organic EL display device, or the like. The input device 113 transmits instructions input by a user of the learning device 100 to the processing device 120. The input device 113 may be a mouse, a keyboard, a touch pad, or the like. The display device 112 and the input device 113 may be implemented as a touch panel.

The communication device 114 controls communication with other devices. The communication device 114 may communicate using any wired or wireless communication method. The communication device 114 communicates with the imaging device 3 and the inspection device 200 via the Internet 4.

The storage device 130 stores programs, data, etc. used by the processing device 120. The storage device 130 may be a semiconductor memory, a hard disk, etc. The storage device 130 stores a learning image storage unit 131, a learning data storage unit 132, and a defect detector 133.

The processing device 120 comprises a learning image acquisition unit 121, an image processing unit 122, a learning data generation unit 123, a learning unit 124, and a defect detector provision unit 125. In terms of hardware, these components are realized by the CPU, memory, and other LSIs of any computer, and in terms of software, by programs loaded into memory, but here we depict functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways, such as hardware alone or a combination of hardware and software.

The learning image acquisition unit 121 acquires captured images of an inspection target having defects from the imaging device 3 and stores them in the learning image storage unit 131. The image processing unit 122 performs image processing such as blurring on the learning images stored in the learning image storage unit 131. The learning data generation unit 123 annotates defects included in the learning images. The learning data generation unit 123 divides at least a portion of the defect images into multiple images and annotates each of the divided defect images. The learning data generation unit 123 may display the learning images on the display device 112 and accept annotations from a person in charge via the input device 113. The learning data generation unit 123 may automatically annotate defects. The learning data generation unit 123 generates learning data and stores it in the learning data storage unit 132.

The learning unit 124 uses the learning data stored in the learning data storage unit 132 to train the defect detector 133. The defect detector providing unit 125 provides the trained defect detector 133 to the inspection device 200.

Figure 7 shows the configuration of an inspection device 200 according to an embodiment. The inspection device 200 includes a display device 212, an input device 213, a communication device 214, a processing device 220, and a storage device 230. The inspection device 200 may be a server device, a device such as a personal computer, or a mobile terminal such as a mobile phone terminal, a smartphone, or a tablet terminal.

The display device 212 displays a screen generated by the processing device 220. The display device 212 may be a liquid crystal display device, an organic EL display device, or the like. The input device 213 transmits instructions input by a user of the inspection device 200 to the processing device 220. The input device 213 may be a mouse, a keyboard, a touch pad, or the like. The display device 212 and the input device 213 may be implemented as a touch panel.

The communication device 214 controls communication with other devices. The communication device 214 may communicate using any wired or wireless communication method. The communication device 214 communicates with the imaging device 3 and the learning device 100 via the Internet 4.

The storage device 230 stores programs, data, etc. used by the processing device 220. The storage device 230 may be a semiconductor memory, a hard disk, etc. The storage device 230 stores an inspection image storage unit 231, a detection result storage unit 232, and a defect detector 233.

The processing device 220 includes an inspection image acquisition unit 221, an image processing unit 222, a defect detection unit 223, a detection result generation unit 224, and a detection result output unit 225. These configurations can also be realized in various forms, such as hardware only or a combination of hardware and software.

The test image acquisition unit 221 acquires an image of the test object captured by the imaging device 3 from the imaging device 3 and stores it in the test image storage unit 231. The image processing unit 222 performs the same image processing on the test image as the learning device 100 performed on the learning image.

The defect detection unit 223 detects defects from the inspection images stored in the inspection image storage unit 231 using the trained defect detector 233 acquired from the learning device 100, and stores the detection results in the detection result storage unit 232. The detection result generation unit 224 generates information about the detected defects. When multiple defects detected by the defect detection unit 223 are close to each other, the detection result generation unit 224 groups the defects into a single defect. The detection result generation unit 224 generates information about the position, size, number, type, etc. of the detected defects. The detection result output unit 225 outputs the detection results generated by the detection result generation unit 224 to the display device 212, etc.

The present invention has been described above based on the embodiments, but the embodiments merely show the principles and applications of the present invention. Furthermore, many modifications and changes in arrangement are possible to the embodiments without departing from the spirit of the present invention as defined in the claims.

In the above embodiment, the technology for detecting cracks on the surface of sanitary ware has been mainly described, but the technology of this embodiment can also be applied to the detection of other types of defects on the surface of sanitary ware, such as adhesion of iron, copper, base material, foreign matter, etc., abnormalities in glaze or color, and the occurrence of air bubbles, cracks, chips, etc. Furthermore, any product other than sanitary ware may be inspected, and any part other than defects may be detected.

1 Inspection system, 2 Tank, 3 Imaging device, 4 Internet, 10 Captured image, 11 Crack, 12 Annotation, 100 Learning device, 121 Learning image acquisition unit, 122 Image processing unit, 123 Learning data generation unit, 124 Learning unit, 125 Defect detector provision unit, 131 Learning image storage unit, 132 Learning data storage unit, 133 Defect detector, 200 Inspection device, 221 Inspection image acquisition unit, 222 Image processing unit, 223 Defect detection unit, 224 Detection result generation unit, 225 Detection result output unit, 231 Inspection image storage unit, 232 Detection result storage unit, 233 Defect detector.

Claims (13)

a learning device that learns a detector for detecting defects occurring on the surface of the sanitary ware and determining the type of the defect;
an inspection device that detects defects occurring on a surface of the sanitary ware using the detector trained by the learning device;
Equipped with
The learning device includes:
a learning image acquisition unit that acquires an image of the sanitary ware including a defect;
a learning data generating unit that generates learning data from images of defects included in the captured images of the sanitary ware acquired by the learning image acquiring unit;
a learning unit that learns the detector using the learning data generated by the learning data generation unit;
Equipped with
the learning data generation unit divides at least a portion of the images of the plurality of defects into a plurality of images, and generates learning data from each of the divided images of the defects;
The inspection device includes:
an inspection image acquisition unit for acquiring an image of the sanitary ware;
a detection unit that detects defects from the captured image acquired by the inspection image acquisition unit using the detector and determines a type of the defect;
Equipped with
When the detected defects are adjacent to each other, the detection unit detects the defects as one defect.
Inspection system. 2. The inspection system according to claim 1, wherein the training data generation unit generates the training data by attaching a teacher label to an image of a defect included in the captured image of the sanitary ware, and when the image of the defect is divided into a plurality of parts, generates the training data by attaching a teacher label to each of the plurality of divided images of the defect. 3. The inspection system according to claim 1 , further comprising an image processing unit that performs image processing on the captured image of the sanitary ware acquired by the learning image acquisition unit before generating learning data from the captured image. The inspection system according to claim 3 , wherein the image processing is a process for reducing the resolution or contrast of pixel values in the portion that is not the defect. The inspection system according to claim 4 , wherein the image processing is a process of applying a blurring filter to the entire captured image. The inspection system according to claim 1 , wherein the defect is a crack . an inspection image acquisition unit for acquiring an image of the sanitary ware;
a detection unit that detects defects from the captured images acquired by the inspection image acquisition unit using a detector that has been trained using learning data generated by dividing an image of a defect included in a captured image of the sanitary ware into a plurality of parts, and determines a type of the defect;
Equipped with
When the detected defects are adjacent to each other, the detection unit detects the defects as one defect.
Inspection equipment. 8. The inspection device according to claim 7 , further comprising an image processing unit that performs image processing on the captured image of the sanitary ware acquired by the inspection image acquisition unit before detecting defects in the captured image. 9. The inspection apparatus according to claim 8 , wherein the image processing is a process for reducing the resolution or contrast of pixel values in the portion that is not the defect. 10. The inspection device according to claim 9 , wherein the image processing is a process of applying a blurring filter to the entire captured image. 11. The inspection device according to claim 7 , wherein the defect is a crack. Computer,
an inspection image acquisition unit for acquiring an image of the sanitary ware;
a detection unit that detects defects from the captured images acquired by the inspection image acquisition unit using a detector that has been trained using learning data generated by dividing an image of a defect included in a captured image of the sanitary ware into a plurality of parts, and determines a type of the defect;
Function as a
When the detected defects are adjacent to each other, the detection unit detects the defects as one defect.
Inspection program. On the computer,
acquiring an image of the sanitary ware;
a step of detecting defects from the captured image of the sanitary ware using a detector trained using training data generated after dividing an image of a defect included in the captured image of the sanitary ware into a plurality of parts, and determining the type of the defect;
Run the command ,
In the detecting step, when the detected defects are adjacent to each other, the defects are collectively detected as one defect.
Testing method.

Images