JP6597972B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus that executes image classification and the like, and an image processing method.

  For example, in the technical field of manufacturing semiconductor substrates, glass substrates, printed boards, and the like, it is possible to detect, analyze, or evaluate defects such as foreign matters, scratches, and etching defects contained in products. In order to analyze and evaluate the defect, an appearance inspection is performed on the object to be evaluated using a microscope such as an optical microscope or a scanning electron microscope. In the appearance inspection, it has been studied that an evaluation target is imaged with a microscope or the like, and automatic classification is performed on the obtained image by calculating a plurality of types of feature amounts. In this type of technique, a classification technique using a learning algorithm based on a teacher image given in advance as a reference image including a typical defect has been proposed (see Patent Documents 1 and 2). In the conventional technique, for each of the classification categories corresponding to the defect types, the operator designates a plurality of typical images serving as a criterion for selecting a teacher image, and processing based on the designated typical images is executed. Thus, the classification accuracy is improved. However, in this technique, it is necessary for the operator to visually check a large number of defect images individually and to select a typical image from the large number of defect images.

  In order to deal with such a problem, when a plurality of defect images are plotted in a multi-dimensional feature amount space, an image group forming a lump in close proximity to each other is regarded as one group, and belongs to the group in order from a larger group. A method of presenting an image to a user as a candidate for a typical image has been proposed. In this method, a large number of collected defect images are presented in a state of being sorted into several groups in advance according to their feature amounts, so that the load on the user in selecting a typical image is reduced.

JP 2010-071826 A JP2013-254286A JP 2003-317082 A JP2013-167545A JP 2014-006613 A

  In the conventional technique described above, the operator may adjust the condition of “close” for each coordinate axis in order to obtain a group of images that form a mass close to each other when plotted in a multidimensional feature amount space. As a result, the operator performs work by trial and error. The present invention has been made in view of such problems.

  An object of the present invention is to effectively support an operator in a task of selecting a teacher image used in a learning algorithm for classifying an image or a task called a typical image that serves as a criterion for selecting a teacher image. It is to be.

One aspect of the present invention is exemplified by the following image processing apparatus. That is, the image processing apparatus includes a feature amount acquisition unit, a coding unit, a grouping unit, and a category setting unit. The feature amount acquisition unit acquires a plurality of types of feature amounts for each of the plurality of target images. The encoding unit divides each of the acquired plural types of feature quantities into two or more stages, assigns a numerical value according to the classification, and generates a code value that combines the numerical values in a predetermined order. Therefore, a code value is generated regardless of work involving trial and error of the operator.
The grouping unit forms an image group by combining images having the same code value among a plurality of target images. Therefore, an image group is formed regardless of work involving trial and error of the operator.
The category setting unit displays at least one target image from the formed image group and accepts setting of information indicating the category of the image group including the displayed target image. Therefore, the image processing apparatus forms an image group from the target image and displays it to the operator, so that the operator can support setting of information indicating the category of the image group. This category can be called a classification category when a plurality of types of feature quantities are calculated and automatically classified. In addition, this image group can be used as a teacher image used in a learning algorithm as a reference image including a typical defect or a typical image for selecting a teacher image.

  According to the present invention, an operator is effectively supported in a task of selecting a teacher image used in a learning algorithm for classifying an image or a task called a typical image serving as a reference for selecting a teacher image. can do.

It is a figure which shows schematic structure of an image classification device. It is a figure which shows the flow of a defect image classification | category by an image classification device. It is a figure which shows the structure of a host computer. It is a block diagram which shows the relationship between the structure of the host computer for learning a classifier, and a structure test | inspection / classification apparatus. It is a block diagram which shows the function structure of a teacher data creation part. It is a figure which shows a part of structure of the information memorize | stored in the memory | storage part. It is a flowchart which illustrates a teacher data creation process. It is an example of a sample of a defect image. It is a figure which shows the example of the characteristic code produced | generated by the image classification device. It is a figure which illustrates a defect image. It is a figure which illustrates the feature code of the defect image of the largest cell. It is a figure which illustrates the defect image of the largest cell. It is a figure which illustrates the characteristic code of the 4th cell. It is a figure which illustrates the defect image contained in the 4th cell. It is a figure which illustrates the same defect image as what belongs to the 4th cell. It is a figure which illustrates the characteristic code of the 7th cell. It is a figure which illustrates the defect image contained in the 7th cell. It is a figure which illustrates the feature code of the cell of the integration object searched in the vicinity of the 7th cell. It is a figure which illustrates the defect image contained in the cell of the integration object searched in the vicinity of the 7th cell. It is a figure which illustrates the defect image of 39th cell. It is a figure which illustrates the defect image of the 100th cell. It is a figure which displays the feature code of the 39th cell and the feature code of the 100th cell side by side.

FIG. 1 is a diagram showing a schematic configuration of an image classification apparatus 1 according to an embodiment of the present invention. As an example of an image processing apparatus, the image classification apparatus 1 classifies, for example, a defect image indicating a defect on a semiconductor substrate 9 (hereinafter simply referred to as “substrate 9”). In the present embodiment, the processing executed by the image classification device 1 can be called an image processing method. The image classification device 1 includes an imaging device 2, an inspection / classification device 4, and a host computer 5. The imaging device 2 images the inspection target area on the substrate 9. The inspection / classification device 4 inspects the defect based on the image data from the imaging device 2 and automatically classifies the defect into a category to which the defect should belong when the defect is detected. The host computer 5 controls the overall operation of the image classification device 1. In FIG. 1, the inspection / classification apparatus 4 and the host computer 5 are illustrated as separate bodies, but the inspection / classification apparatus 4 and the host computer 5 may be provided integrally in the same casing. Good. Further, the host computer 5 may execute part or all of the processing of the inspection / classification device 4 according to the computer program. The types of defects present on the substrate 9 are, for example, chipping, protrusion, disconnection, short circuit, and foreign matter, and these are defined as defect categories. In addition, the imaging device 2 is incorporated in the production line of the substrate 9, and the image classification device 1 is a so-called inline system.

  The imaging device 2 captures an inspection target region on the substrate 9 to acquire image data, a stage 22 that holds the substrate 9, and a stage that moves the stage 22 relative to the imaging unit 21. A drive unit 23 is included. The imaging unit 21 electrically outputs an illumination unit 211 that emits illumination light, an optical system 212 that guides the illumination light to the substrate 9 and receives light from the substrate 9, and an image of the substrate 9 formed by the optical system 212. An imaging device 213 that converts the signal into a signal is included. The stage drive unit 23 includes a ball screw, a guide rail, a motor, and the like. The host computer 5 controls the stage driving unit 23 and the imaging unit 21 so that the inspection target area on the substrate 9 is imaged.

  The inspection / classification device 4 includes a defect detection unit 41 that detects a defect while processing image data of an inspection target region, and a defect automatic classification unit 42 that classifies a defect image. The defect detection unit 41 has a dedicated electric circuit for processing image data of the inspection target area at high speed, and performs defect inspection of the inspection target area by comparing the captured image with a reference image having no defect or by image processing. Do. The automatic defect classification unit 42 includes a CPU that performs various arithmetic processes, a memory that stores various information, and the like. The defect automatic classification unit 42 performs defect classification (that is, defect image classification) using a classifier 421 that uses a neural network, a decision tree, discriminant analysis, and the like.

In the present embodiment, the CPU is also referred to as an MPU (Microprocessor) or a processor. The CPU is not limited to a single processor, and may have a multiprocessor configuration. A single CPU connected by a single socket may have a multi-core configuration. The processing of at least a part of each part of the inspection / classification apparatus 4 and the host computer 5 is performed by a processor other than a CPU, such as a digital signal processor (DSP), a graphics processing unit (GPU), a numerical processor, a vector processor, It may be performed by a dedicated processor such as an image processor. Further, at least part of the processing of the inspection / classification apparatus 4 and the host computer 5 may be executed by an integrated circuit (IC) or other digital circuits. In addition, an analog circuit may be included in at least a part of each of the above parts. The integrated circuit includes an LSI, an application specific integrated circuit (ASIC), and a programmable logic device (PLD). PLD is, for example, Field-Programmable Gate Array (FPG).
A). That is, each unit of the inspection / classification apparatus 4 and the host computer 5 may be a combination of a processor and an integrated circuit. The combination is, for example, a microcontroller (MCU), SoC (System-on-a-chip), system LS
I, called chipset.

  FIG. 2 is a diagram showing a flow of defect image classification by the image classification apparatus 1. In the present embodiment, each procedure in the flowchart is called a step or a process, and a code starting with S is given. First, when the imaging device 2 shown in FIG. 1 images the substrate 9, the defect detection unit 41 of the inspection / classification device 4 acquires image data (S11). Next, when the defect detection unit 41 performs a defect inspection on the inspection target region and a defect is detected (S12), data of an image of the defective portion (that is, a defect image) is transmitted to the defect automatic classification unit 42. . The automatic defect classification unit 42 calculates a plurality of types of feature values of the defect image (S13), the feature values of the defect image are input to the classifier 421 of the automatic defect classification unit 42, and the classification result is output. In other words, the defect image is classified into one of a plurality of categories by the classifier 421 (S14). In the image classification device 1, the feature amount is calculated in real time every time a defect is detected by the defect detection unit 41, and automatic classification of a large number of defect images is performed at high speed. However, the processing of the image classification device 1 is not limited to real-time processing. For example, the processes of S13 and S14 may be executed by an offline process separated from the processes of S11 and S12.

  Next, creation of teacher data by the host computer 5 will be described. FIG. 3 is a diagram showing the configuration of the host computer 5. The host computer 5 has a general computer system configuration in which a CPU 51 that performs various arithmetic processes, a ROM 52 that stores basic programs, and a RAM 53 that stores various information are connected to a bus line. Further, an external storage device that stores information is connected to the bus line. Examples of the external storage device include a reading device 57 that reads information from a computer-readable recording medium 8 such as a fixed disk 54, an optical disk, a magnetic disk, and a magneto-optical disk. A solid state drive (SSD) may be provided together with the fixed disk 54 or instead of the fixed disk 54. In the figure, the reading device 57 is illustrated, but a writable medium may be used as the recording medium 8. In this case, a medium input / output device is used instead of the reading device 57.

  A device that forms a user interface with the operator is connected to the bus line. For example, a display 55 that displays various types of information such as images, a keyboard 56 a that accepts input from an operator, a mouse 56 b (hereinafter collectively referred to as “input unit 56”), and the like are connected to the bus line. However, the input unit 56 is not limited to the keyboard 56a and the mouse 56b. For example, other pointing devices such as an electrostatic pad may be used. A touch panel that detects a screen operation on the display 55 by the operator may be used. Furthermore, a communication unit 58 that transmits and receives signals to and from other components of the image classification device 1 is connected to the bus line through an interface (I / F) as appropriate. The communication unit 58 may communicate with other computers or other devices that cooperate with the image classification device 1 via a local area network (LAN) or the like.

  In the host computer 5, the program 80 is read in advance from the recording medium 8 or other computer connected via the communication unit 58 via the reader 57 and stored in the fixed disk 54. Further, the program 80 is copied to the RAM 53 and the CPU 51 executes arithmetic processing according to the program in the RAM 53.

  FIG. 4 is a block diagram showing a configuration for learning a classifier realized by the CPU 51, ROM 52, RAM 53, fixed disk 54, etc. of the host computer 5, and also shows the inspection / classification device 4. The host computer 5 includes a teacher data creation unit 61 that creates teacher data used for learning the classifier and a learning unit 62 that learns the classifier using the teacher data. In addition, the structure of each part illustrated in FIG. 4 may be constructed | assembled by a dedicated electrical circuit, and a partially dedicated electrical circuit may be utilized.

Based on the defect image generated by the defect detection unit 41, the teacher data creation unit 61 creates teacher data that serves as a reference for classifying the defect image and sets it in the classifier 421 of the automatic defect classification unit 42. Based on the set teacher data, the classifier 421 classifies other defect images that are not employed in the teacher data and defect images included in images acquired by the imaging device 2 after the teacher data setting. The teacher data includes defect image data (referred to as a teacher image), a feature amount of the defect image, and a teaching signal which is information indicating a defect category. As the feature amount of the defect image, for example, the area of the defect, the brightness average, the peripheral length, the flatness (also referred to as extension), the inclination of the major axis when the defect is approximated to an ellipse, and the like are employed. In the learning unit 62, the feature amount of the defect image read from the teacher data is input to a classifier (not shown) in the host computer 5, and the output of the classifier is the same as the teaching signal indicating the defect category. Learning is done. The learning result, that is, the classifier 421 after learning (more precisely, information indicating the structure of the classifier 421 and the value of the variable) is transferred to the automatic defect classification unit 42.

  FIG. 5 is a block diagram showing a functional configuration of the teacher data creation unit 61 of the host computer 5, and also shows a display 55, an input unit 56, and a learning unit 62. The CPU 51 executes processing as the teacher data creation unit 61 in accordance with a computer program developed in the RAM 53 so as to be executable. However, as already described, part of the processing of the teacher data creation unit 61 may be executed by another processor, a digital circuit, an analog circuit, or the like.

  The teacher data creation unit 61 includes a storage unit 611, a feature amount acquisition unit 612, and a category determination unit 613. The category determination unit 613 also assigns a category to the defect image based on the feature amount of the defect image acquired by the feature amount acquisition unit 612. More specifically, the category determination unit 613 includes a coding unit 614, a grouping unit 615, an integration unit 616, and a category setting unit 617. The storage unit 611 stores image data of a defect image (hereinafter simply referred to as an image) and various types of information related to the defect image. The storage unit 611 is constructed, for example, on the RAM 53 or the fixed disk 54 in FIG. The feature amount acquisition unit 612 acquires the feature amount of the defect image. The feature amount acquisition unit 612 calculates, for the defect image to be processed, the above-described defect area, brightness average, peripheral length, flatness, inclination of the long axis when the defect is approximated to an ellipse, and the like.

  The encoding unit 614 classifies each of a plurality of types of feature amounts acquired by the feature amount acquisition unit 612 for one defect image into two or more stages, assigns a numerical value according to the classification, A code value that is a feature code combined in a predetermined order is generated. The grouping unit 615 forms an image group by combining images having the same code value among a plurality of images to be processed. The integration unit 616 searches for one or more neighboring images having a code value whose difference from the code value of the target image of one image group is 1 bit, and integrates the searched neighboring images into one image group. Execute the process. An image group generated by integrating such neighboring images into one image group is referred to as an integrated image group. A plurality of neighborhood images is also referred to as a neighborhood image group.

  The category setting unit 617 displays at least one target image from the formed image group on the display 55. Then, the category setting unit 617 receives information indicating the category of the image group or the integrated image group including the displayed image via the input unit 56. The category setting unit 617 sets information indicating the accepted category in the image group. The defect image may be acquired using the imaging device 2 and the defect detection unit 41 illustrated in FIG. 1, may be transmitted from another device via the communication unit 58, or the recording medium 8. Alternatively, it may be separately prepared offline.

FIG. 6 is a diagram showing a part of the structure of information stored in the storage unit 611. In the storage unit 611, the feature amount of the defect image calculated by the feature amount acquisition unit 612 shown in FIG. 5 is stored in association with the defect image data together with its type. When a category is determined for a defect image, a category number indicating the category is stored in a categorical variable associated with the defect image data. Further, when the defect image is designated as the typical image, 1 is stored in the typical image identification variable associated with the defect image data. In the initial state, 0 is stored in the categorical variable and the typical image identification variable.

  However, if it is not necessary to identify a typical image in the process of creating teacher data, the typical image identification variable is omitted. Further, the information structure of FIG. 6 is an exemplification, and the implementation of the present invention is not limited to the information structure of FIG. For example, a table may be provided in the storage unit 611 and the feature amount may be stored in the table entry. The row of the table may be identified by the image number for identifying the defect image, and the column of the table may be identified by the feature number indicating the type of feature amount. The teacher data creation unit 61 can hold the feature amount for each defect image and for each type of feature amount using such a table. The categorical variable and the typical image identification variable may be stored in a table corresponding to the image number for identifying the row in the table (one column table).

  FIG. 7 is a flowchart illustrating teacher data creation processing according to the present embodiment. By the processing shown in the drawing, a teacher image used for a learning algorithm as a reference image including a typical defect or a typical image serving as a reference for selecting a teacher image is selected from the defect image. In this sense, in the present embodiment, the defect image to be processed by the teacher data creation unit 61 is also referred to as teacher data.

  As shown in the figure, the teacher data creation unit 61 acquires the feature amount of the defect image by the feature amount acquisition unit 612 (S21). The teacher data creation unit 61 performs the process of S21 as an example of acquiring a plurality of types of feature amounts for each of a plurality of target images. That is, in the present embodiment, the defect image can be called a target image in the sense of a processing target image.

  FIG. 8 is an example of a defect image (image A, image B) stored in the storage unit 611. In FIG. 8, images A and B are illustrated as samples corresponding to two defects. The defect detection unit 41 of the image classification device 1 generates a plurality of images (captured image, reference image, mask image, difference image) as shown in FIG. 8 based on the defect image at one location obtained from the imaging device 2. To do. The captured image is a defect image captured by the imaging device 2 or the like at the defect location. The reference image is an image at a position corresponding to a defective portion in another die (for example, another integrated circuit chip) corresponding to the target image. The difference image is a difference image between the captured image and the reference image. The mask image is an image obtained as a result of binarizing the difference image.

  In the following processing, it is assumed that a set of four images (captured image, reference image, mask image, difference image) illustrated in FIG. 8 is obtained for defects at a plurality of locations (hereinafter referred to as K locations). Is explained. In the present embodiment, a plurality of types of feature amounts are calculated from one defect image. That is, in the present embodiment, the teacher data creation unit 61 calculates N (dimensional) feature amounts from the K defect images, respectively, through the process of S21. As a result, the feature amount becomes N-dimensional space data (hereinafter referred to as an N-dimensional feature vector). However, the teacher data creation unit 61 (feature amount acquisition unit 612) may acquire the feature amount of the defect image calculated by another image classification device or the like connected via the communication unit 58.

  As specific examples of the feature amount, in addition to the area of the defect area obtained from the mask image, the perimeter, flatness, and moment, a density histogram obtained from the defect image or the difference image, an average density value, an autocorrelation coefficient value, Examples include spatial frequency, difference statistics, concentration co-occurrence matrix, higher-order local autocorrelation, run-length matrix, and fractal feature.

Next, the teacher data creation unit 61 uses the encoding unit 614 to encode each feature amount with a threshold value appropriately determined (S22). The coded feature quantity is called a feature code. As an example of the teacher data creation unit 61, each of the acquired plural types of feature quantities is divided into two or more stages, a numerical value corresponding to the classification is given, and a code value is generated by combining the numerical values in a predetermined order. The process of S22 is executed. For example, the encoding can be obtained by binarizing into 0 and 1 with one threshold set for each type of feature amount. A feature code obtained by binarization, that is, by dividing into two stages, is called binarized data.

  As a threshold value for each feature amount in the case of binarization, an average value of corresponding feature amounts obtained from all defect images to be processed, a median value of a distribution range, or a value at which the cumulative relative frequency is approximately 50% Values automatically calculated by the teacher data creation unit 61 can be used. Further, when coding in a large number of stages, for example, by using a value with a cumulative relative frequency of approximately 33 percent or a value of 67 percent, the feature amount is set to “large”, “medium”, “small”, or “ It can be coded as “high”, “medium”, “low”, “long”, “medium”, “short”, and “dark”, “medium”, “light”. The same applies to the case of encoding in four or more stages.

  However, encoding in three or more stages with two or more threshold values can be converted into a substantially equivalent binary encoding. For example, instead of coding the feature amount A into three stages of “large”, “medium”, and “small”, whether or not the feature amount A can be determined as “large” (is a value greater than or equal to the threshold L). The second dimension, whether the feature amount A can be determined as “medium” in the first dimension (whether it falls within the range from the threshold value S to the value L), and the feature amount A can be determined as “small”. It can be binarized and coded by the third dimension of whether or not (a value less than a threshold value S). When binarization is performed in S22, the process of S22 is an example of dividing each of a plurality of types of feature amounts into two stages and adding binarized numerical values.

  Furthermore, for example, when there is an operational rule such as “an area with an area of a pixel or less is erroneously detected unconditionally”, the threshold value for encoding the area of “a pixel” may be used. In any case, the operator's judgment and trial and error work are not required in the process of S22. By the process of S22, the N-dimensional feature vector is converted into a bit string of a predetermined length. For example, when the feature quantity is coded by binarization, an N-dimensional feature vector is converted into an N-bit numerical value. Therefore, by the process of S22, a feature code having a predetermined number of bits (N bits in the case of binarization) is assigned to all defect images.

Next, the teacher data creation unit 61 uses the grouping unit 615 to group defect images having the same feature code (S23). As an example of forming an image group by combining images having the same code value among a plurality of target images, the teacher data creation unit 61 executes the process of S23. Now, assuming that a sufficient number of independent feature quantity types are prepared, the number of defect types that can be expressed by the feature code obtained in step S22 is enormous. For example, if the are 32 types of features, the feature code for the binarization, different defect type of 2 ³² (approximately 42 595 million units) it is can be expressed. Compared to this, the number of defect images for creating a teacher image actually prepared is as small as tens of thousands at most. Therefore, even if different feature codes are assigned to all images, it can be understood that there is no wonder. Despite this understanding, if there are a plurality of defect images having the same feature code, the plurality of defect images having the same feature code are regarded as images classified into the same defect type. Good. Based on this concept, defect images having the same feature code are grouped. This grouping is called “temporary teaching”. The teacher data creation unit 61 automatically generates a label (defect type number) for temporary teaching.

This method uses the fact that the volume of the feature space increases as the number of feature dimensions increases, and therefore the density of data that is assumed to be uniformly distributed in the feature space becomes sparse. Can be interpreted as clustering. If only the feature quantity dimension (the number of coordinate axes) can be secured, whether or not all defect images can be divided into two by the threshold value related to the feature quantity when viewing one feature quantity axis (whether it can be divided into multiple stages) is This is not a problem in the embodiment.

  Next, the teacher data creation unit 61 arranges in descending order from the group with the largest number of defects (S24). The teacher data creation unit 61 executes the process of S24 as an example of displaying a list of a plurality of image groups in the order of the number of target images included in each of the plurality of image groups. When a plurality of defect images having the same feature code exist to form a group, the group is likely to be a typical defect type. However, the specific name of the defect type may remain unknown at the time of the processing of S24. Then, there is a possibility that defect images of similar defect types are distributed near the group in the feature amount space. Therefore, the teacher data creation unit 61 first displays a list of defect images (or feature amounts of defect images) in order from the group with the largest number of defect images, and a group is arbitrarily selected from the list displayed by the operator. The operation is received (S25). The teacher data creation unit 61 executes the process of S25 as an example of setting an image group designated by the operator as one image group among a plurality of image groups displayed in a list.

  Next, the teacher data creation unit 61 searches the defect image of the feature code located near the feature code of the defect image belonging to the selected group (that is, the one image group described above) by the integration unit 616 (S26). ). For example, the teacher data creation unit 61 may search for a defect image of a feature code or a group of defect images whose difference from the feature code of the selected group is 1 bit. The teacher data creation unit 61 executes the process of S26 as an example of searching for a neighboring image having a code value whose difference from the code value of the target image of one image group is 1 bit.

  In the present embodiment, a small region in a high-dimensional feature space represented by a feature code is called a “cell”. The cell dimensions are not particularly limited. For example, for each dimension, an area delimited by a 1-bit difference value can be called a cell. A cell corresponding to the group selected by the operation of S25 is referred to as a “target cell”. Further, a “cell adjacent in the N direction” of a certain target cell is defined as a “neighboring cell”. For example, in the process of S21, when the feature value is coded by binarization, in this method, the feature value axis is divided into two by the threshold value. Therefore, all the cells corresponding to the feature code formed by inverting only one arbitrary bit of the feature code corresponding to a certain target cell correspond to the neighboring cells. In the process of S21, when feature quantities are coded in multiple stages by a plurality of threshold values, 1 is added to or subtracted from a bit string corresponding to an arbitrary axis of a feature code included in a certain target cell. A cell corresponding to a possible feature code corresponds to a neighboring cell.

  Therefore, the teacher data creation unit 61 displays a list of defect images (or defect image groups) in neighboring cells located in the vicinity of the group selected by the operator in the process of S24 (S27). The teacher data creation unit 61 executes the process of S27 as an example of displaying the searched neighborhood image. By the process of S27, candidate images that can be integrated when a group of defect images existing in the target cell are centered are presented. When there are a plurality of images in neighboring cells, a plurality of existing images, that is, groups of candidate images are presented.

Then, the operator determines whether or not to actually integrate while looking at the candidate images (or group of candidate images) that can be targeted for integration displayed in the list in S27. Then, the teacher data creation unit 61 accepts whether or not integration is possible from the operator (S28). When the operator instructs integration of one or more neighboring candidate images (or candidate image groups) (YES in S29), the teacher data creation unit 61 executes integration processing by the integration unit 616 (S30). . That is, the teacher data creation unit 61 integrates a group of defect images existing in the target cell and a neighboring candidate image (or group of candidate images) for which integration is instructed, and reorganizes the group. By reorganization, the target cell and the cell in which the candidate image (or group of candidate images) exists are integrated. The teacher data creation unit 61 performs the processes of S28 and S29 as an example of accepting designation of whether or not to display the displayed neighboring image into the one image group. The teacher data creation unit 61 performs the process of S30 as an example of integrating the searched neighboring images into one image group.

  Then, the teacher data creation unit 61 receives designation from the operator as to whether or not to continue the integration (S31). When an instruction to continue the integration is given (YES in S31), the teacher data creation unit 61 returns the process to S24 and updates the display according to the reorganized group. In this manner, the teacher data creation unit 61 repeats the processing from S25 to S31 in accordance with an instruction from the operator. For example, the operator continues the integration process until the desired number of defect types is reached. The teacher data creation unit 61 deletes the dimension corresponding to the bit of the feature code that is different between the vicinity image and the one image group by integrating the vicinity image into the one image group, and the dimension is deleted. As an example of repeating the searching process and the integrating process to a predetermined limit according to the feature code, the processes of S30 and S31 are executed.

  When the feature code is coded by binarization, the integration of neighboring cells is the same as reducing the axis (dimension) of a certain feature amount. That is, the defect type is determined regardless of the value of the feature amount to be reduced. Therefore, if the feature amount does not play an important role in classifying the image, the operator often does not feel inconvenience visually even if integration for reducing the feature amount is performed. Conversely, when the operator can recognize that it is inconvenient visually, it is estimated that the feature amount reduced by the integration plays an important role in classifying the defect images. As described above, this method can support teaching work and can also be expected to have an effect of finding feature quantities effective for classification.

  Further, the teacher data creation unit 61 repeats the processing from S24 to S31, so that the integration of groups proceeds while the feature quantity dimension is gradually reduced. For example, two groups that originally had a two-bit difference may become two groups that exist in the vicinity of which there is only a one-bit difference as a result of integration. Similarly, two groups that originally had n-bit differences are gradually moved closer to n-1 bit differences, n-2 bit differences,... There can be two existing groups.

  If it is determined in S31 that an instruction not to continue integration has been given, the teacher data creation unit 61 selects, for at least one group, at least one defect image included in the group as a teacher image or a teacher image selection. It is displayed as a typical image candidate as a reference (S32). Then, when the operator sets the displayed defect image as a teacher image, or when setting it as a typical image serving as a reference for selecting a teacher image, the name indicating the feature of the image indicated by the displayed defect image A defect type name (also referred to as a label) is input. In this case, it can be said that the group is a group used as teacher data (or a group including a typical image serving as a reference for selecting a teacher image). Then, the teacher data creation unit 61 accepts the defect type name by the category setting unit 617 according to the operation of the operator, and assigns the defect type name to the group to be the teacher data (S33).

  When the number of images in the group reaches a certain number by the above procedure, a defect type name is assigned to the group. As described above, an image of a group to which a defect type name is assigned is called a taught image. The teacher data creation unit 61 displays at least one target image from the formed image group, and accepts the setting of information indicating the category of the image group including the displayed target image as an example of S32 and S33. Execute the process.

  In the process of FIG. 7, the group integration is executed by the processes of S24 to S31 after the grouping process (S23). However, the implementation of the present invention is not limited to the processing of FIG. For example, after the grouping process (S23), the defect types may be assigned to the grouped defect images according to the input of the operator by the processes of S32 and S33. That is, the group integration by the processing from S24 to S31 may be performed separately according to a request from the operator. For example, when the number of images in the group reaches a certain number by the grouping process (S23), a defect type may be assigned to such a group.

  By the way, it is expected that a large number of groups that cannot be integrated will remain if the teacher data creation unit 61 simply integrates following the vicinity through the processing from S24 to S31. This is because it is assumed that a considerable number of defect images are “isolated points” in a so-called feature amount space in which only one defect image exists in one cell. Therefore, after the processing according to FIG. 7 is completed, the teacher data creation unit 61 may execute automatic assignment of teaching information by a method exemplified in Japanese Patent No. 5075070, for example. In the case of the method exemplified in Japanese Patent No. 5075070, the group to which the defect type is assigned in the process of FIG. 7 is a typical image, and the typical image identification variable of FIG. The operator can complete the teaching for a large number of defect images in a short time by making a slight manual correction on the result of the automatic assignment.

Here, the method exemplified in Japanese Patent No. 5075070 is as follows. First, as a premise, at least one typical image indicating a typical defect is designated in advance in each of the groups corresponding to a plurality of defect types (categories). Here, a group corresponding to a plurality of defect types (categories) for which a typical image is designated corresponds to, for example, a group to which the defect type name is assigned. And the teacher data creation part 61 performs the following processes.
(A) One type of feature amount is acquired from a target defect image that is an unclassified defect image.
(B) The difference between the one type of feature amount of all the typical images and the one type of feature amount of the target defect image is obtained. As a result of obtaining differences from all typical images, a plurality of feature amount differences corresponding to the number of typical images are obtained.
(C) Vote for the category to which the typical image corresponding to the smallest value among the plurality of feature quantity differences belongs.
(D) The steps (a) to (c) are performed for each type of feature amount of the target defect image.
(E) The category having the largest number of votes among the plurality of categories is determined as the category to which the target defect image belongs. By repeating such processing, the taught images can be increased.

  When the above processing is completed, the image data of the typical image and the image data of the taught image that are stored in the storage unit 611 of the teacher data creation unit 61, and information on these feature amounts and categories are used as the learning data. 62. The learning unit 62 learns the classifier using the teacher data. That is, the values of variables constituting the classifier (see FIG. 4) are determined, the structure of the classifier is determined, and the classifier is generated.

  An embodiment of the present invention will be described with reference to FIGS. 9 to 22. A teaching operation with the assistance of the image classification apparatus 1 was performed on the result of inspection by the wafer appearance inspection apparatus with a pattern. Here, the teaching work is a work for classifying images into groups and assigning defect types. The total number of defect images is 5140 points. For each of these defect images, 174 feature amounts were calculated, and feature codes were generated by binarization.

  FIG. 9 shows an example of the feature code generated by the image classification device 1. In FIG. 9, the bit pattern is displayed in hexadecimal. In FIG. 9, 232 bits are exemplified for the feature code. Of these, 174 bits are effective. Further, FIG. 10 illustrates a part of the defect images included therein (eight defect images). In FIG. 9, “file name” is the file name of the defect image. “Train” is a label, that is, a defect type name. The defect type name is originally given by the processing of the teacher data creation unit 61 according to FIG. 7, but in this embodiment, the defect type name is obtained by a visual teaching operation in advance for reference. Has been granted. “Code” is a feature code generated by the teacher data creation unit 61.

  As a result of applying the processing of the teacher data creation unit 61 to 5140 defect images, 420 cells including two or more defect images were generated. The number of defect images belonging to these 420 cells was 1594 points. Hereinafter, these 420 cells are referred to as all cells. The remaining 3546 defect images were isolated points belonging to only one point per cell. Among all the cells, the cell with the largest number of defect images (hereinafter referred to as the maximum cell) contained 31 defect images.

  FIG. 11 illustrates 14 feature codes of defect images included in the largest cell with the largest number of defect images. FIG. 12 illustrates eight defect images included in the largest cell. As shown in FIG. 11, two types (20 and 30) of defect types are mixed in the defect type information given by the prior visual teaching work. Even when the operator visually classifies the defect image of the largest cell, there are many cases where the determination is lost. Therefore, in the processing by the teacher data creation unit 61, information such as “A collection of confusing things” is given to the cells as shown in FIGS. 11 and 12, and the teaching work is postponed.

  Next, the cell with the fourth largest number of images (hereinafter referred to as the fourth cell) contains 19 defect images, and was easily determined to be a typical image group. FIG. 13 illustrates 14 feature codes of the fourth cell. FIG. 14 illustrates eight defect images included in the fourth cell.

  Furthermore, when the defect images included in other cells are checked, a defect image of 7 points similar to those belonging to the fourth cell is found in the cell with the 49th largest number of images (hereinafter referred to as the 49th cell). I was able to. FIG. 15 illustrates four defect images similar to those belonging to the fourth cell. Note that most of the seven defect images similar to those belonging to the fourth cell were given the same defect type name as that of the fourth cell by prior visual teaching work.

  In addition, the cell having the seventh largest number of images (hereinafter referred to as the seventh cell) contained 17 defect images. In FIG. 16, two types (10 and 20) of defect types are mixed in the defect type information given by the prior visual teaching work. Therefore, when the defect image classified into the seventh cell by the process of the teacher data creation unit 61 is examined again, the defect image of the seventh cell is a typical image group that can be said to be an intermediate type of both defect types (10 and 20). It can be determined that the defect types can be separated from the defect types 10 and 20 assigned by the prior visual teaching work and integrated as a new defect type (for example, 15).

FIG. 18 exemplifies the feature code of the integration target cell searched in the vicinity of the seventh cell. FIG. 19 illustrates four defect images included in the integration target cell searched in the vicinity of the seventh cell. Comparing FIG. 16 and FIG. 18, the thirteenth code from the right (4 bits at the arrow) is “9” in FIG. 16, and “B” in FIG. Moreover, the defect image of FIG. 17 and the defect image of FIG. 19 are hardly distinguishable. Therefore, the 183rd bit which is the difference between the 13th code from the right (4 bits at the arrow), that is, the dimension of the 183th feature amount, was searched for in the defect image of the 7th cell and its vicinity. It can be seen that it does not contribute to the separation of the defect type of the cell defect image. In this example, the 183rd feature amount is an absolute value of a difference between the defect maximum pixel value and the background average pixel value.

  Similarly, the cell with the 39th largest number of images (hereinafter referred to as the 39th cell) includes nine defect images. The 39th cell was also easily determined as the third typical image group. FIG. 20 illustrates four defect images of the 39th cell. Similar defect images were found in the cell having the 100th largest number of images (hereinafter referred to as the 100th cell) with respect to the 39th cell. FIG. 21 illustrates a defect image of the 100th cell. However, the defect image of the 100th cell is different from the defect image of the 100th cell in the defect diameter. In FIG. 22, the feature code of the 39th cell and the feature code of the 100th cell are displayed side by side. The image of the 39th cell is Defect — 0000746. It is indicated by the file name of tif. The image of the 100th cell is Defect_0001716. It is indicated by the file name of tif. As shown in FIG. 22, the defect type 30 is assigned to both the 39th cell and the 100th cell by prior visual teaching work. However, there is a difference of 76 bits between the feature code of the defect image belonging to the 39th cell and the feature code of the defect image belonging to the 100th cell. Therefore, the 39th cell and the 100th cell cannot be integrated. In such a case, the vicinity of the 100th cell is searched again and integration is performed.

  As described above, according to the image classification device 1 of the present embodiment, the feature amount of the defect image is encoded into the feature code by the threshold value. Each defect image is arranged in the feature amount space by the feature code. Then, defect images having the same feature code are grouped. Therefore, a group of defect images can be formed without depending on whether or not the positions of the defect images are close to each other in the feature amount space. That is, according to the image classification device 1, it is not necessary to set a condition as to whether or not the operator is close by trial and error. In other words, by dividing the feature quantity into multiple stages according to the threshold and assigning values corresponding to the stages, it is possible to code a multi-dimensional feature quantity as a feature code. Then, the image classification apparatus 1 can create teacher data from the defect image without grouping the defect image by the feature code and assigning the defect type name to the operator without the work by the trial and error of the operator. Also, the image classification device 1 can select a typical image serving as a reference for teacher image selection, which is one element of teacher data, by this processing.

  And the defect image of each cell is shown to a user in order with many defect images contained in the cell specified by the characteristic code. Furthermore, when the feature code is obtained by binarization, a cell whose feature code differs by 1 bit is searched as a neighboring cell with respect to the cell arbitrarily selected by the operator, and becomes a target of integration. . Therefore, according to the image classification device 1, the cells are integrated without depending on the operator's work. As a result, the defect images are classified into groups corresponding to the integrated cells. As described above, when the feature values are integrated by binarizing, the relationship between bits and dimensions becomes clear and the processing becomes simple.

  Also, the image classification apparatus 1 according to the present embodiment presents to the user in descending order of the number of defect images included in a cell in group integration, and determines a target cell to be integrated. A cell having more defect images included in the cell is considered to be a group of images having a typical feature amount. Therefore, the image classification device 1 can cause the operator to effectively select an integration target. Then, a neighboring cell for one cell (image group) which is the attention cell is searched and presented to the operator. Therefore, according to the image classification device 1, the defect images are classified into an integrated group without adjusting the condition of whether or not the defect images are close to each other. That is, according to the image classification device 1, it is not necessary to set a condition as to whether or not the operator is close by trial and error. Further, when a neighboring cell for one cell (image group) that is a target cell is searched, the image classification device 1 receives designation of whether or not to integrate the searched neighboring cell into the target cell. Therefore, the image classification device 1 enables the operator to make a final confirmation in the integration.

  Further, according to the image classification apparatus 1 of the present embodiment, after a cell whose feature code is different by 1 bit is searched as a neighbor cell and grouping is performed, further grouping is performed according to an instruction from the operator. The process is repeated. In addition, by performing grouping, the number of bits of the feature code is reduced by 1 bit (the feature dimension is one dimension). That is, by repeating the grouping, it is possible to gradually bring the groups of defect images that are separated by a plurality of bits into a single group. In particular, when the feature code is coded by binarization, each bit of the feature code corresponds to the dimension of the feature amount. Therefore, dimension reduction accompanying integration and dimension management are simplified. Note that the relationship apart from a plurality of bits means that bit contents at a plurality of bit positions are different.

[Modification]
In the embodiment described above, the image classification device 1 that classifies defect images indicating defects on the substrate 9 has been exemplified. However, the implementation of the present invention is not limited to the image classification device 1. For example, the image processing apparatus of the present invention can be applied to classification of images obtained by photographing cells, medical images, and other image classifications in various applications.

  In the above embodiment, the processing for an image acquired from the optical system has been exemplified. However, the implementation of the present invention is not limited to images acquired from an optical system. For example, the present invention can be applied to classification of images generated by various apparatuses such as an electron microscope, a laser terahertz emission microscope (LTEM), an X-ray imaging apparatus, and a diagnostic apparatus using nuclear magnetic resonance imaging (MRI). For example, the inspection / classification device 4, the host computer 5, and the image capturing device may be connected via a LAN or the like, and the processing of the above embodiment may be executed.

2 imaging device 4 inspection / classification device 5 host computer 55 display 56 input unit 61 teacher data creation unit 62 learning unit 611 storage unit 612 feature amount acquisition unit 614 encoding unit 615 grouping unit 615 integration unit 617 category setting unit

Claims (8)

A feature amount acquisition unit that acquires a plurality of types of feature amounts for each of a plurality of target images;
A coding unit that classifies each of the acquired plural types of feature values into two or more stages, assigns a numerical value according to the classification, and generates a code value that combines the numerical values in a predetermined order;
A grouping unit that forms an image group by combining images having the same code value among the plurality of target images;
An image processing apparatus comprising: a category setting unit that displays at least one target image from the formed image group and receives setting of information indicating a category of an image group including the displayed target image.   The image processing apparatus according to claim 1, wherein the encoding unit divides each of the plurality of types of feature amounts into two stages and assigns a binarized numerical value.   And further comprising an integration unit that searches for a neighboring image having a code value whose difference from a code value of a target image of one image group is 1 bit, and integrates the searched neighboring image into the one image group. Item 3. The image processing apparatus according to Item 1 or 2.   The image processing apparatus according to claim 3, wherein the integration unit displays the searched neighborhood image and receives designation of whether or not to integrate the displayed neighborhood image into the one image group.   The integration unit displays a list of information on the plurality of image groups in the order of the number of target images included in each of the plurality of image groups, and is designated by an operator among the plurality of image groups displayed on the list. The image processing apparatus according to claim 3, wherein the received image group is the one image group. The feature code includes binary data obtained by dividing the plurality of feature amounts into two stages.
The integration unit deletes a dimension corresponding to a bit of a feature code that is different between the vicinity image and the one image group by integrating the vicinity image into the one image group, and the dimension is The image processing apparatus according to claim 3, wherein the searching process and the integrating process are repeated up to a predetermined limit according to the deleted feature code. Acquiring a plurality of types of feature amounts for each of a plurality of target images;
Classifying each of the acquired plural types of feature quantities into two or more stages, giving a numerical value according to the classification, and generating a code value combining the numerical values in a predetermined order; and
Combining the images having the same code value among the plurality of target images to form an image group;
An image processing method for displaying at least one target image from the formed image group and receiving a setting of information indicating a category of an image group including the displayed target image. Searching for a neighboring image having a code value whose difference from the code value of the target image of one image group is 1 bit;
The image processing method according to claim 7, further comprising: integrating the searched neighboring images into the one image group.

Images