JP7745429B2 - Inspection device, inspection method, and inspection program - Google Patents

Description

The present invention relates to an inspection device, an inspection method, and an inspection program for performing image inspection of objects.

In the manufacturing process of an article, similar defects can occur in multiple articles manufactured in succession. For example, in the manufacturing process of LD (Laser Diode) chips, multiple LD chips are transferred sequentially using a suction head (see Patent Document 1). Therefore, if foreign matter adheres to the suction head, similar transfer scratches can occur on these multiple LD chips. Alternatively, in the manufacturing process of castings, multiple castings are cast using a mold. Therefore, if foreign matter adheres to the mold, similar foreign matter marks can occur on these multiple castings.

JP 2004-55721

Incidentally, AI (Artificial Intelligence) is rapidly being used in the inspection of various objects. In such inspections, a trained model (an algorithm constructed through machine learning) is used as input to images containing the object as a subject, and the object's class is determined. However, minor defects that can occur in multiple objects, such as transfer scratches on LD chips or foreign matter marks on castings, may be detected by visual inspection by an inspector but not by image inspection using a trained model. This is because visual inspection by an inspector takes into account the collective characteristics of multiple objects (such as the commonality of defects that occur in multiple objects), making it difficult to overlook individual defects even if they are minor. In contrast, image inspection using a trained model does not take into account the collective characteristics of multiple objects, making it easy to overlook individual defects as if they are minor.

One aspect of the present invention was made in consideration of the above problems, and its purpose is to realize technology that can inspect for minor defects that may occur commonly across multiple items.

The inspection device according to aspect 1 of the present invention comprises a single or multiple processors that execute an inspection process for performing image inspection of multiple items, and in the inspection process, the processor takes as input multiple images containing each of the multiple items as a subject, or a merged image obtained by merging the multiple images, and performs image inspection of the multiple items using a trained model that outputs one of multiple classes indicating the result of a group judgment for the multiple items, or the probability that the result of the group judgment for the multiple items belongs to each of the multiple classes.

This makes it possible to realize an inspection device that can inspect for minor defects that may occur commonly across multiple items.

In the inspection device according to Aspect 2 of the present invention, in addition to the configuration of Aspect 1, the processor further performs the following steps: (1) an identification process to identify multiple regions containing images of the multiple items in a single image containing the multiple items as subjects, or in a low-resolution image obtained by reducing the resolution of the single image; (2) a cropping process to generate the multiple images by cropping each region identified by the identification process from the single image; and (3) a merging process to generate the merged image by merging the multiple images generated by the cropping process; and the input of the trained model is the merged image.

This makes it possible to realize an inspection device that can inspect for minor defects that may occur commonly across multiple items based on a single image that includes multiple items as subjects.

In the inspection device according to Aspect 3 of the present invention, in addition to the configuration of Aspect 1 or 2, the items are semiconductor chips, and the multiple classes include a normal class indicating that the items have no transfer damage, and an abnormal class indicating that the items have transfer damage.

This makes it possible to realize an inspection device that can inspect multiple semiconductor chips for transfer scratches that may occur in common.

In the inspection device according to Aspect 4 of the present invention, in addition to any of Aspects 1 to 3, the processor further executes a construction process to construct the trained model by machine learning using a combination of multiple images containing each of the multiple items as a subject, or a merged image obtained by merging the multiple images, and a class selected by the inspector from the multiple classes as the class to which the set of the multiple items belongs, as training data.

This makes it possible to realize an inspection device that can inspect for minor defects that may occur commonly across multiple items without having to rely on trained models from other devices.

An inspection method according to aspect 5 of the present invention includes an inspection process in which a single or multiple processors perform image inspection of multiple items, and in the inspection process, the processors take as input multiple images containing each of the multiple items as a subject, or a merged image obtained by merging the multiple images, and perform image inspection of the multiple items using a trained model that outputs one of multiple classes indicating the result of a group judgment for the multiple items, or the probability that the result of the group judgment for the multiple items belongs to each of the multiple classes.

This makes it possible to realize an inspection method that can check for minor defects that may occur commonly across multiple items.

The inspection program according to aspect 6 of the present invention is an inspection program for causing a computer to operate as an inspection device according to any one of aspects 1 to 4, and is configured to cause a processor included in the computer to execute each of the above processes.

This makes it possible to create an inspection program that can check for minor defects that may occur commonly across multiple items.

One aspect of the present invention makes it possible to realize technology that can inspect for minor defects that may occur commonly across multiple items.

1 is a block diagram showing a configuration of an inspection device according to an embodiment of the present invention; FIG. 1 is a flowchart showing the flow of an inspection method according to an embodiment of the present invention. 3A and 3B are diagrams showing a specific example of the inspection method shown in Fig. 2. (a) illustrates a single image ImgX, (b) illustrates multiple areas A1, A2, ..., A5 in the single image ImgX, (c) illustrates multiple images Img1, Img2, ..., Img5, and (d) illustrates a merged image ImgY.

(Single judgment and collective judgment)
When inspecting multiple items, individual judgment may be performed or collective judgment may be performed. Here, individual judgment refers to class judgment performed on individual items. For example, when individual judgment is performed on 10 items, 10 judgment results are obtained that indicate the pass/fail status of the individual items. On the other hand, collective judgment refers to class judgment performed on a group of items. For example, when collective judgment is performed on 10 items, one judgment result is obtained that indicates the pass/fail status of the group of items.

(Configuration of inspection device)
The configuration of an inspection device 1 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the inspection device 1.

The inspection device 1 is a device that performs a collective judgment of multiple items O1, O2, ..., On (n is a natural number greater than or equal to 2) using images Img1, Img2, ..., Imgn that include each of these items as a subject. In this embodiment, LD (Laser Diode) chips are assumed as items O1, O2, ..., On. However, this embodiment can be applied to semiconductor chips in general. This is because scratches can occur during transfer on semiconductor chips other than LD chips for the same reasons as on LD chips.

As shown in FIG. 1, the inspection device 1 includes a memory 11, a processor 12, and a storage 13. The memory 11, the processor 12, and the storage 13 are connected to one another via a bus (not shown). An input/output interface (not shown) and a communication interface (not shown) may also be connected to this bus. This input/output interface is used, for example, to input an input image from an external device (e.g., a camera) to the inspection device 1, or to output inspection results from the inspection device 1 to an external device (e.g., a display). The communication interface is also used, for example, to allow the inspection device 1 to receive an input image provided from an external device (e.g., another computer connected to a camera), or to allow the inspection device 1 to transmit inspection results to be provided to an external device (e.g., another computer connected to a display).

Memory 11 is configured to record an inspection program P for implementing inspection method S1, which will be described later, and a trained model M used in inspection method S1, which will be described later. For example, semiconductor random access memory (RAM) can be used as memory 11. For example, a convolutional neural network (CNN) can be used as trained model M. A logistic regression model, a support vector machine, a random forest, or the like can also be used as trained model M.

The processor 12 is configured to execute the inspection method S1, described below, in accordance with the inspection program P stored in the memory 11. The processor 12 may be, for example, a CPU (Central Processing Unit), GPU (Graphic Processing Unit), TPU (Tensor Processing Unit), digital signal processor, microprocessor, microcontroller, or a combination of these.

Storage 13 is configured to store (non-volatilely store) the above-mentioned inspection program P and the above-mentioned trained model M. When executing inspection method S1 described below, processor 12 expands and references the inspection program P and trained model M stored in storage 13 on memory 11. Note that storage 13 may be, for example, a flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these.

In this specification, the term "image" refers to a two-dimensional array of pixel values. A pixel value in a monochrome image is, for example, a single numerical value corresponding to brightness, while a pixel value in a color image is, for example, three numerical values corresponding to RGB, CMY, and YUV. Furthermore, an area in an input image is also a two-dimensional array of pixel values and can be considered an image.

The inspection performed by the inspection device 1 may be a binary judgment that does not distinguish between normal (e.g., no transfer damage) and abnormal (e.g., transfer damage), or a multi-value judgment that distinguishes between normal and/or abnormal types. In the case of a binary judgment, the inspection result is one of two classes (normal, abnormal). In this case, the judgment result may be a numerical value or a combination thereof representing one or both of the normal and abnormal probabilities. In the case of a multi-value judgment, assuming that p types of normal and q types of abnormal can be distinguished, the inspection result is one of p + q classes (normal 1, normal 2, ..., normal p, abnormal 1, abnormal 2, ..., abnormal q). In this case, the judgment result may be a numerical value or a combination thereof representing some or all of the probability of normal 1, the probability of normal 2, ..., the probability of normal p, the probability of abnormal 1, the probability of abnormal 2, ..., and the probability of abnormal q. Here, one of p and q is a natural number greater than or equal to 1, and the other is a natural number greater than or equal to 2. Furthermore, test results may include results that are neither normal nor abnormal (or their probabilities), such as "pending."

Note that while the configuration described here is one in which inspection method S1 is executed by a single processor 12 provided in a single computer, this is not limiting. In other words, it is also possible to adopt a configuration in which inspection method S1 is executed jointly by multiple processors provided in a single computer or distributed across multiple computers.

Furthermore, while the configuration described here is one in which the trained model M is stored in a single memory 11 provided in a single computer, this is not limiting. In other words, it is also possible to adopt a configuration in which the trained model M is stored in a single computer or in multiple memories distributed across multiple computers.

The inspection program P for causing the processor 12 to execute the inspection method S1 may be recorded on a computer-readable, non-transitory, tangible recording medium. This recording medium may be memory 11, storage 13, or another recording medium. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit may be used as the other recording medium.

(Testing method flow)
The flow of the inspection method S1 performed by the inspection device 1 will be described with reference to Fig. 2. Fig. 2 is a flow chart showing the flow of the inspection method S1.

As shown in FIG. 2, the inspection method S1 includes a specification process S11, a cropping process S12, a merging process S13, and an inspection process S14. In the following description, it is assumed that a single image ImgX containing multiple items O1, O2, ..., On as subjects, obtained by simultaneously capturing images of the multiple items O1, O2, ..., On, is stored in the memory 11 of the inspection device 1.

The identification process S11 is a process for identifying multiple areas A1, A2, ..., An in the image ImgX, each of which contains the image of multiple objects O1, O2, ..., On. The area Ai containing the image of each object Oi (i = 1, 2, ..., n) is preferably set so that (1) the image of that object Oi does not share any common area with the outside of the area Ai, and (2) the images of objects Oj (j ≠ i) other than the object Oi do not share any common area with the inside of the area Ai. In this embodiment, the identification process S11 is executed by the processor 12 provided in the inspection device 1. The method for identifying the areas is not particularly limited, but template matching may be used, for example.

Note that, in this embodiment, a configuration is adopted in which an area Ai containing the image of each item Oi is identified in image ImgX, but the present invention is not limited to this. For example, a configuration may be adopted in which an area Ai containing the image of each item Oi is identified in a low-resolution image ImgX' obtained by reducing the resolution of image ImgX. When the former configuration is adopted, the identification process S11 can be executed with higher accuracy. On the other hand, when the latter configuration is adopted, the identification process S11 can be executed at higher speed.

The cropping process S12 is a process for generating multiple images Img1, Img2, ..., Imgn by cropping (cutting out) the areas A1, A2, ..., An identified in the identification process S11 from the image ImgX. The image Imgi is an image that includes the corresponding item Oi as a subject. In this embodiment, the cropping process S12 is executed by the processor 12 provided in the inspection device 1.

The merge process S13 is a process for generating a merged image ImgY by merging (combining) the multiple images Img1, Img2, ..., Imgn generated in the crop process S12. The merged image ImgY is an image that includes multiple objects O1, O2, ..., On as its subjects. In this embodiment, the merge process S13 is executed by the processor 12 provided in the inspection device 1. For example, if each image Imgi is a color image with H pixels vertically, W pixels horizontally, and C channels, the merged image ImgY will be (1) a color image with H pixels vertically, n x W pixels horizontally, and C channels (when merged horizontally), (2) a color image with n x H pixels vertically, W pixels horizontally, and C channels (when merged vertically), or (3) a color image with H pixels vertically, W pixels horizontally, and n x C channels (when merged channel-wise).

In this embodiment, the multiple images Img1, Img2, ..., Imgn obtained by the cropping process S12 are the targets of the merging process S13, but the present invention is not limited to this. For example, the multiple images Img1, Img2, ..., Imgn containing multiple items O1, O2, ..., On as subjects, obtained by individually capturing images of each of the multiple items O1, O2, ..., On, may be the targets of the merging process S13. In this case, the identification process S11 and the cropping process S12 can be omitted.

Inspection process S14 is a process for performing image inspection of multiple items O1, O2, ..., On using trained model M. Here, trained model M is a trained model that takes the merged image ImgY generated in merging process S13 as input and outputs a class representing the result of group judgment for multiple items O1, O2, ..., On. In this embodiment, inspection process S14 is executed by processor 12 provided in inspection device 1.

When performing a binary judgment, the output of the trained model M is, for example, either the normal class CN or the abnormal class CA. The normal class CN is a class indicating that multiple items O1, O2, ..., On as a set are normal (e.g., no damage caused by transfer). The abnormal class CA is a class indicating that multiple items O1, O2, ..., On as a set are abnormal (e.g., damage caused by transfer). The output of the trained model M may be the probability that the set of multiple items O1, O2, ..., On belongs to the normal class CN and the probability that the set of multiple items O1, O2, ..., On belongs to the abnormal class CA.

Furthermore, when performing multi-value judgment, the output of the trained model M is one of p normal classes CN1, CN2, ..., CNp and q abnormal classes CA1, CA2, ..., CAq. The normal classes CN1, CN2, ..., CNp are classes indicating that multiple items O1, O2, ..., On are normal as a set (e.g., no damage during transportation). The abnormal classes CA1, CA2, ..., CAq are classes indicating that multiple items O1, O2, ..., On are abnormal as a set (e.g., damage during transportation). The output of the trained model M may be the probability that a set of multiple items O1, O2, ..., On belongs to normal class CN1, the probability that a set of multiple items O1, O2, ..., On belongs to normal class CN2, ..., the probability that a set of multiple items O1, O2, ..., On belongs to normal class CNp, the probability that a set of multiple items O1, O2, ..., On belongs to abnormal class CA1, the probability that a set of multiple items O1, O2, ..., On belongs to abnormal class CA2, ..., and the probability that a set of multiple items O1, O2, ..., On belongs to abnormal class CAq.

In this embodiment, the merged image ImgY obtained by the merging process S13 is used as the input to the trained model M, but the present invention is not limited to this. For example, multiple images Img1, Img2, ..., Imgn obtained by the cropping process S12 may be used as the input to the trained model M. Alternatively, multiple images Img1, Img2, ..., Imgn containing multiple items O1, O2, ..., On as subjects, obtained by individually capturing images of each of the multiple items O1, O2, ..., On, may be used as the input to the trained model M.

In addition, prior to carrying out the above-described inspection method S1, the inspection device 1 may execute a machine learning process to construct a trained model M through machine learning. When a merged image ImgY is used as input to the trained model M, the training data is, for example, a combination of the merged image and a class selected by the inspector from the above-described multiple classes as a class to which a set of multiple items included as subjects in the merged image belongs. When multiple images Img1, Img2, ..., Imgn are used as input to the trained model M, the training data is, for example, a combination of multiple images and a class selected by the inspector from the above-described multiple classes as a class to which a set of multiple items included as subjects in each of these multiple images belongs.

The inspection device 1 may also be configured to perform alignment processing between the cropping process S12 and the merging process S13. This alignment processing is a process for aligning the positions of the images of the objects O1, O2, ..., On in the images Img1, Img2, ..., Imgn, and is realized using, for example, pattern matching technology.

(Specific examples of inspection methods)
A specific example of the inspection method S1 will be described with reference to Fig. 3. In this example, five LD chips are inspected to determine whether or not these five LD chips contain any damage caused by transportation.

Figure 3(a) shows an example of image ImgX. This image ImgX contains five LD chips as its subject.

Figure 3(b) shows an example of five areas A1 to A5 identified in the identification process S11. Each of these five areas A1 to A5 includes one LD chip as the subject.

Figure 3(c) shows five examples of images Img1 to Img5 generated in the cropping process S12. These five images Img1 to Img5 are obtained by cropping the five areas A1 to A5 described above from image ImgX, and each contains one LD chip as its subject.

Figure 3(d) shows an example of a merged image ImgY generated in the merging process S13. This merged image ImgY is an image obtained by merging the five images Img1 to Img5 described above horizontally, and includes five LD chips as its subjects. As mentioned above, the five images Img1 to Img5 may also be merged vertically or in the channel direction.

In this specific example, the merged image ImgY shown in Figure 3 (d) is input to the trained model M. The trained model M outputs either class CN, which indicates the absence of transfer damage, or class CA, which indicates the presence of transfer damage, as a class indicating the result of the collective judgment for the five LD chips. In the example shown in Figure 3, five LD chips contain transfer damage. Therefore, the trained model M outputs class CA, which indicates the presence of transfer damage.

(Additional Notes)
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiments are also included in the technical scope of the present invention. Furthermore, the application of the present invention is not limited to semiconductor chips such as LD chips, but can be applied to general products in which common defects may occur during the manufacturing process.

REFERENCE SIGNS LIST 1 Inspection device 11 Memory 12 Processor 13 Storage P Inspection program M Trained model S1 Inspection method S11 Identification process S12 Cropping process S13 Merge process S14 Inspection process

Claims (6)

a single or multiple processors that execute an inspection process for inspecting images of a plurality of articles;
In the inspection process, the processor receives as input a plurality of images including each of the plurality of items as a subject, or a merged image obtained by merging the plurality of images, and outputs one of a plurality of classes indicating the result of a group determination for the plurality of items, or the probability that the result of the group determination for the plurality of items belongs to each of the plurality of classes , and performs image inspection of the plurality of items using a trained model constructed by machine learning using as training data a combination of a plurality of images including each of the plurality of items as a subject, or a merged image obtained by merging the plurality of images, and a class selected by an inspector from the plurality of classes as the class to which the group of the plurality of items belongs.
An inspection device characterized by: The processor further executes: (1) an identification process for identifying a plurality of regions including images of the plurality of articles in a single image including the plurality of articles as subjects, or in a low-resolution image obtained by reducing the resolution of the single image; (2) a cropping process for generating the plurality of images by cropping each region identified by the identification process from the single image; and (3) a merging process for generating the merged image by merging the plurality of images generated by the cropping process.
The input of the trained model is the merged image.
2. The inspection device according to claim 1. the article is a semiconductor chip;
The plurality of classes include a normal class indicating that the plurality of articles have no transfer damage, and an abnormal class indicating that the plurality of articles have transfer damage.
3. The inspection device according to claim 1 or 2. The processor further executes a construction process of constructing the trained model by the machine learning.
4. The inspection device according to claim 1, wherein the inspection device is a semiconductor laser. an inspection process in which a single or multiple processors perform image inspection of multiple articles;
In the inspection process, the processor receives as input a plurality of images including each of the plurality of items as a subject, or a merged image obtained by merging the plurality of images, and outputs one of a plurality of classes indicating the result of a group determination for the plurality of items, or the probability that the result of the group determination for the plurality of items belongs to each of the plurality of classes , and performs image inspection of the plurality of items using a trained model constructed by machine learning using as training data a combination of a plurality of images including each of the plurality of items as a subject, or a merged image obtained by merging the plurality of images, and a class selected by an inspector from the plurality of classes as the class to which the group of the plurality of items belongs.
An inspection method characterized by: An inspection program for causing a computer to operate as the inspection device according to any one of claims 1 to 4, the inspection program causing a processor included in the computer to execute each of the processes.
An inspection program characterized by: