JP7649646B2 - Machine learning device, machine learning method, and machine learning program - Google Patents

Description

The present invention relates to a machine learning device, a machine learning method, and a machine learning program that use image data as input and improve the performance of a model constructed by machine learning through additional learning.

A method of inspecting an object using a model constructed by machine learning (hereinafter also referred to as a trained model) is known. For example, Patent Literature 1 discloses an inspection device that inspects an object using a trained model that receives image data including an object as a subject and outputs the class of the object.

Patent Publication No. 2020-042754

The trained model outputs one of multiple classes as the class to which the input image data belongs. However, some of these classes may have an extremely low frequency of occurrence. In this case, learning for image data belonging to such a class may be insufficient, and as a result, the accuracy of determining image data belonging to such a class may decrease.

To address this issue, it is effective to perform additional learning using a training dataset that contains sufficient image data belonging to classes with low occurrence frequencies. However, it is impossible or extremely difficult for an inspector to visually pick out image data that belongs to classes with low occurrence frequencies from a dataset that contains a huge number of image data. Another possible method is to use a trained model to pick out the image data, but trained models have low accuracy in determining image data that belongs to classes with low occurrence frequencies, making this method ineffective.

The present invention was made in consideration of the above problems, and its purpose is to realize a machine learning technique that can improve the performance of a model constructed by machine learning, in particular the judgment accuracy for image data belonging to a class that appears infrequently, through additional learning.

The machine learning device according to aspect 1 of the present invention includes at least one processor, and the processor executes the following steps: a class estimation process that estimates a class to which each image data included in a dataset belongs using a model constructed by machine learning; a data selection process that selects from the dataset image data that is estimated to belong to a specific class in the class estimation process, image data whose likelihood of estimation is equal to or greater than a threshold; a labeling process that assigns a label indicating the specific class to image data that an inspector has determined to belong to the specific class from the image data selected in the data selection process; and an additional learning process that performs additional learning of the model using a learning dataset that includes image data that has been assigned a label in the labeling process.

According to the above configuration, by performing each of the above processes with a class that occurs less frequently as a specific class, it is possible to improve the accuracy of judgment for image data that belongs to a class that occurs less frequently. Moreover, according to the above configuration, the image data for which the inspector needs to visually judge the class is limited to the image data selected in the data selection process. Therefore, the burden on the inspector can be reduced.

In the machine learning device according to aspect 2 of the present invention, in addition to the configuration of the machine learning device according to aspect 1, the processor is configured to repeatedly execute the class estimation process, the data selection process, the label assignment process, and the additional learning process.

The above configuration can further improve the accuracy of judgment for image data that belongs to a class with a low occurrence frequency.

The machine learning device according to aspect 3 of the present invention has a configuration that, in addition to the configuration of the machine learning device according to aspect 1 or 2, further includes a threshold setting process that sets the threshold.

The above configuration allows the amount of image data selected in the data selection process, i.e., the amount of image data for which the inspector needs to visually determine the class, to be appropriately adjusted. Therefore, the burden on the inspector can be appropriately adjusted.

The machine learning method according to aspect 4 of the present invention is characterized in that it includes a class estimation process in which at least one processor estimates a class to which each image data included in a dataset belongs using a model constructed by machine learning; a data selection process in which the processor selects from the dataset image data estimated to belong to a specific class in the class estimation process, image data whose likelihood of estimation is equal to or greater than a threshold; a labeling process in which the processor assigns a label indicating the specific class to image data determined by an inspector to belong to the specific class from the image data selected in the data selection process; and an additional learning process in which the processor performs additional learning of the model using a learning dataset including image data to which a label has been assigned in the labeling process.

According to the above configuration, by performing each of the above processes with a class that occurs less frequently as a specific class, it is possible to improve the accuracy of judgment for image data that belongs to a class that occurs less frequently. Moreover, according to the above configuration, the image data for which the inspector needs to visually judge the class is limited to the image data selected in the data selection process. Therefore, the burden on the inspector can be reduced.

The machine learning program according to aspect 5 of the present invention is a program for operating a computer as a machine learning device according to any one of aspects 1 to 3, and is a program for causing a processor included in the computer to execute each of the processes.

With the above configuration, the above machine learning device can be realized using a computer.

According to one aspect of the present invention, it is possible to realize a machine learning technique that can improve the performance of a model constructed by machine learning, in particular the judgment accuracy for image data belonging to a class that appears infrequently, through additional learning.

1 is a block diagram showing a configuration of a machine learning device according to an embodiment of the present invention. 2 is a flow diagram showing the flow of a machine learning method implemented by the machine learning device of FIG. 1 . 1A is a diagram for explaining what kind of image data is selected in the data selection process when the threshold (the lower limit of the likelihood) is large, and FIG. 1B is a diagram for explaining what kind of image data is selected in the data selection process when the threshold (the lower limit of the likelihood) is small. 1A is a diagram showing an example of image data belonging to class C1 (a class other than a specific class) in an application example (visual inspection of an LD chip) of the present invention, and FIG. 1B is a diagram showing an example of image data belonging to class C4 (a specific class) in the same application example.

(Configuration of machine learning device)
The configuration of a machine learning device 1 according to an embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the machine learning device 1.

The machine learning device 1 is a device for implementing a machine learning method S for improving the performance of a model M constructed by machine learning through additional learning. As shown in FIG. 1, the machine learning device 1 includes a memory 11, a processor 12, a storage 13, and a bus 14. The memory 11, the processor 12, and the storage 13 are connected to each other via the bus 14.

The memory 11 is configured to record a machine learning program P for implementing the machine learning method S, and a model M that is constructed and evaluated in the machine learning method S. Note that, for example, a semiconductor RAM (Random Access Memory) can be used as the memory 11.

The processor 12 is configured to execute the machine learning method S according to the machine learning program P stored in the memory 11. The processor 12 may be, for example, a CPU (Central Processing Unit), a GPU (Graphic Processing Unit), a microprocessor, a digital signal processor, a microcontroller, an ASIC (Application Specific Integrated Circuit) such as a TPU (Tensor Processing Unit), or a combination of these.

Storage 13 is configured to store dataset D referenced in machine learning method S. For example, flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or a combination of these can be used as storage 13.

Note that, although the configuration in which the machine learning method S is executed by a single processor 12 provided in a single computer has been described here, the present invention is not limited to this. In other words, it is also possible to adopt a configuration in which the machine learning method S is executed jointly by multiple processors provided in a single computer, or distributed across multiple computers.

Although the configuration in which the dataset D is stored in a single storage 13 provided in a single computer has been described here, the present invention is not limited to this. In other words, it is also possible to adopt a configuration in which the dataset D is stored in a single computer or in multiple storages provided in a distributed manner across multiple computers.

The machine learning program P for causing the processor 12 to execute the machine learning method S may be recorded on a computer-readable, non-transitory, tangible recording medium. This recording medium may be the memory 11, the storage 13, or another recording medium. For example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit may be used as the other recording medium.

(Machine learning methods)
The flow of the machine learning method S performed by the machine learning device 1 will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the machine learning method S.

As described above, the machine learning method S is a method for improving the performance of the model M constructed by machine learning through additional learning. More specifically, it is a method for improving the performance of the model M so that the class can be correctly estimated even for image data that belongs to a specific class with a low occurrence frequency. As shown in FIG. 2, the machine learning method S includes a class estimation process S1, a data selection process S2, a label assignment process S3, and an additional learning process S4.

The dataset DS is a collection of image data. The model M is an algorithm that takes image data as input and outputs one of multiple classes, and is constructed by machine learning. The model M outputs the class to which it estimates that the input image data belongs, along with the likelihood of the estimate. In this embodiment, a convolutional neural network (CNN) is used as the model M, but neural networks other than CNN, logistic regression models, support vector machines, random forests, etc. may also be used as the model M.

The class estimation process S1 is a process that estimates the class to which each image data included in the dataset D belongs using the model M. In this embodiment, the class estimation process S1 is executed by the processor 12, and the estimated class is stored in the memory 11 together with the likelihood of the estimation.

The data selection process S2 is a process for selecting, from the data set DS, image data that is estimated to belong to a specific class in the class estimation process S1 and whose likelihood of estimation is equal to or greater than a predetermined threshold value α. In this embodiment, the data selection process S2 is executed by the processor 12, and the selected image data is stored in the memory 11.

The image data selected in the data selection process S2 is provided to an inspector. The inspector visually determines the class to which each piece of image data selected in the data selection process S2 belongs.

The labeling process S3 is a process of assigning a label indicating a specific class to image data that the inspector has determined to belong to a specific class from among the image data selected in the data selection process S2. In this embodiment, the labeling process S3 is executed by the processor 12, and the image data to which the label has been assigned is stored in the memory 11.

The additional learning process S4 is a process for additional learning of the model M using a learning dataset DL that includes image data to which labels have been assigned in the labeling process S3. The learning dataset DL may include only image data to which labels have been assigned in the labeling process S3, or may include other image data. An example of the latter is using a collection of image data selected in the data selection process S2 as the learning dataset DL. In this embodiment, the additional learning process S4 is executed by the processor 12, and the model M after the additional learning is stored in the memory 11.

The processor 12 may repeatedly execute the class estimation process S1, the data selection process S2, the label assignment process S3, and the additional learning process S4. This allows the performance of the model for image data belonging to a specific class to be gradually improved while gradually increasing the amount of image data belonging to a specific class contained in the learning dataset DL. The threshold value α described above may be kept constant or gradually increased when these processes are repeated.

These processes may be terminated as desired. Examples include (1) when the determination accuracy of model M for image data belonging to a specific class reaches a predetermined determination accuracy, (2) when the determination accuracy of model M for any image data reaches a predetermined determination accuracy, and (3) when the number of repetitions reaches a predetermined number.

(Adjustment of threshold value α)
The adjustment of the threshold value α will be described with reference to FIG. 3. In FIG. 3, image data is shown projected onto a two-dimensional plane. Triangle marks in FIG. 3 correspond to image data belonging to a specific class, and square marks in FIG. 3 correspond to image data belonging to a class other than the specific class. Curve γ in FIG. 3 represents the watershed between whether or not data is selected in the data selection process S2. Image data located to the lower left of curve γ is selected as image data belonging to the specific class in the data selection process S2. Image data located to the upper right of curve γ is not selected as image data belonging to the specific class in the data selection process S2.

When the threshold value α is increased, as shown in FIG. 3(a), the amount of image data selected in the data selection process S2 is reduced. As a result, the burden on the inspector to visually determine the class to which each piece of image data selected in the data selection process S2 belongs is reduced. However, the number of image data belonging to a specific class that can be incorporated into the learning dataset DL is reduced, and the effect of additional learning is reduced.

Conversely, if the above-mentioned threshold value α is made smaller, more image data will be selected in the data selection process S2, as shown in FIG. 3B. As a result, the number of image data belonging to a specific class that can be incorporated into the learning dataset DL increases, and the effect of additional learning is improved. However, the burden on the inspector to visually determine the class to which each image data selected in the data selection process S2 belongs increases.

For this reason, it is preferable to set the above-mentioned threshold value α to an optimal value taking into consideration both the effect of additional learning and the burden on the examiner. For this reason, it is preferable that the machine learning method S further includes a threshold setting process for setting the above-mentioned threshold value α. The threshold value α set in this threshold setting process may be a value specified by the user, or may be a value derived by the machine learning device 1.

(Application example)
An application example of the machine learning method S will be described with reference to FIG. 4. The model M to be applied in this application example is a trained model used for visual inspection of LD chips. The input of the model M is image data including an LD chip as a subject, and the output of the model M is one of five classes C1, C2, C3, C4, and C5 that represent the state of the LD chip. An example of image data belonging to class C1 is shown in FIG. 4(a). As shown in FIG. 4(a), class C1 is a class corresponding to a normal LD chip. Also, an example of image data belonging to class C5 is shown in FIG. 4(b). As shown in FIG. 4(b), class C4 is a class corresponding to a cracked LD chip. Class C5 is a class with an extremely low occurrence frequency that corresponds to the above-mentioned specific class.

The given model M had low judgment accuracy for image data belonging to class C5, and additional learning was necessary. For this reason, a dataset D consisting of 500,000 pieces of image data including LD chips as subjects was prepared. It is impossible or extremely difficult for an inspector to visually pick out image data belonging to class C5 from 500,000 pieces of image data. Therefore, additional learning of model M was performed using machine learning method S.

In the first class estimation process S1, a given model M was used. In the first data selection process S2, the threshold value α was set to 10%. As a result, 146 pieces of image data were selected in the first data selection process S2. In the first labeling process S3, of these 146 pieces of image data, 20 pieces of image data that the inspector visually confirmed to belong to class C5 were assigned a label indicating class C5. In the first additional learning process S4, additional learning of model M was performed using a learning dataset DL including these 20 pieces of image data. As a result, the judgment accuracy of model M for image data belonging to class C5 was 46%.

In the second class estimation process S1, the model M constructed in the first additional learning process S4 was used. In the second data selection process S2, the threshold α was set to 10% again. As a result, 72 pieces of image data were selected in the second data selection process S2. In the second labeling process S3, of these 72 pieces of image data, 41 pieces of image data that the inspector visually confirmed to belong to class C5 were assigned a label indicating class C5. In the second additional learning process S4, additional learning of the model M was performed using a learning dataset DL including these 41 pieces of image data. As a result, the judgment accuracy of the model M for image data belonging to class C5 was 85%.

From the above, it was confirmed that the machine learning method S can be used to improve the judgment accuracy of the model M for image data belonging to a class with a low occurrence frequency. If such processing is further repeated, it is expected that the judgment accuracy of the model M for image data belonging to a class with a low occurrence frequency will be further improved. In addition, the number of image data sheets on which the inspector performed visual class discrimination for the first labeling process S3 was 146, and the number of image data sheets on which the inspector performed visual class discrimination for the second labeling process S3 was 72. Therefore, it was confirmed that the performance improvement of the model M described above can be achieved without placing an excessive burden on the inspector.

(Additional Notes)
The present invention is not limited to the above-described embodiments, and various modifications are possible within the scope of the claims. Embodiments obtained by appropriately combining the technical means disclosed in the different embodiments are also included in the technical scope of the present invention.

REFERENCE SIGNS LIST 1 Machine learning device 11 Memory 12 Processor 13 Storage 14 Bus S Machine learning method S1 Class estimation process S2 Data selection process S3 Labeling process S4 Additional learning process D Data set DL Learning data set

Claims (5)

At least one processor;
The processor executes a class estimation process for estimating a class to which each image data included in a data set belongs using a model constructed by machine learning; a data selection process for selecting, from the data set, image data estimated to belong to a specific class in the class estimation process, image data having a likelihood of the estimation equal to or greater than a threshold; a labeling process for assigning a label indicating the specific class to image data determined by an inspector to belong to the specific class from the image data selected in the data selection process; and an additional learning process for performing additional learning of the model using a learning data set including image data to which a label has been assigned in the labeling process ,
The specific class is a class that is preselected as a class that occurs less frequently.
A machine learning device characterized by: The processor repeatedly executes the class estimation process, the data selection process, the label assignment process, and the additional learning process.
The machine learning device according to claim 1 . The method further includes a threshold setting process for setting the threshold.
The machine learning device according to claim 1 or 2. A class estimation process in which at least one processor estimates a class to which each image data included in the dataset belongs using a model constructed by machine learning;
a data selection process in which the processor selects, from the data set, image data that is estimated to belong to a specific class in the class estimation process and whose likelihood of the estimation is equal to or greater than a threshold;
a labeling process in which the processor assigns a label indicating the specific class to image data that an inspector has determined to belong to the specific class, from among the image data selected in the data selection process;
The processor includes an additional learning process for additionally learning the model using a learning dataset including image data to which labels have been added in the labeling process,
The specific class is a class that is preselected as a class that occurs less frequently.
A machine learning method comprising: A machine learning program for operating a computer as the machine learning device described in any one of claims 1 to 3, the machine learning program causing a processor provided in the computer to execute each of the processes.

Images