JP7630930B2 - IMAGE PROCESSING APPARATUS, APPEARANCE INSPECTION APPARATUS, IMAGE PROCESSING METHOD, AND PROGRAM - Google Patents

Description

The present invention relates to a technology for inspecting an object.

Traditionally, many methods have been proposed for detecting defects using machine learning in the visual inspection process of industrial products. For example, an algorithm is known that uses supervised machine learning to determine product defects by preparing images of products without defects (good product images) and images of products with defects (defective product images) as learning data. In general, the more correctly image data is labeled as good or defective, the higher the accuracy of defect detection.

However, when creating training data, there can be inconsistencies in label selection due to factors such as the ability or fatigue of the training worker (the person who selects the labels to be assigned to image data). For this reason, it is not always easy to gather the data necessary to achieve a certain level of accuracy, and this can be very costly.

In response to this, for example, Patent Document 1 proposes a method for automatically labeling without human intervention. Patent Document 2 proposes a method for correcting erroneous data labels using the worker's reliability and certainty as class attribute information. Non-Patent Document 1 also proposes a method for improving accuracy and efficiency when working with humans by increasing the number of classes, such as using not only "Yes/No" as labels but also "Yes/Part/Container/Mixed/Missed Verification."

JP 2016-062544 A JP 2018-106662 A

Papadopoulos, Dim P., et al. "We don't need no bounding-boxes: Training object class detectors using only human verification." Proceedings of the IEEE conference on computer vision and pattern recognition. 2016

However, when labeling is performed automatically as in Patent Document 1, there is a problem in that the accuracy of the labeling is not fully guaranteed. In addition, in the case of Patent Document 2, it is necessary to collect the reliability and confidence of the operators in advance. In the case of Non-Cited Document 1, although erroneous teaching can be mitigated by increasing the number of classes, the problem of fluctuation in class definitions remains.

The objective of the present invention is to provide a technology that easily reduces the impact of fluctuations in label selection by operators on learning.

In order to solve the above problem, a first aspect is an image processing device comprising: a label assignment unit that assigns a label to an image that serves as teacher data, based on a worker's input including a selection input that selects one from a plurality of types of labels; a work status recording unit that records a work status of the worker based on the worker's input; a confidence calculation unit that calculates a confidence level for the worker's input based on work information recorded by the work status recording unit; and a learning unit that performs learning according to the confidence level calculated for the image by the confidence calculation unit, using the image to which the label has been assigned by the label assignment unit as teacher data, and the learning unit changes a learning rate for updating weights in the learning, based on the confidence level .

A second aspect is the image processing device of the first aspect, in which the work information includes a label selection time, which is the time required for the worker to select the label.

A third aspect is the image processing device of the second aspect, in which the work information includes an average value of the label selection time for each of the multiple images, and the confidence level calculation unit calculates the confidence level based on the difference between the label selection time and the average value.

A fourth aspect is an image processing device according to the second or third aspect, in which the work information includes an overall average value that is an average of the label selection times for each of the first number of images, and a partial average value that is an average of the label selection times for each of a portion of the first number of images, and the confidence level calculation unit calculates the confidence level based on the difference between the overall average value and the partial average value.

A fifth aspect is an image processing device according to any one of the first to fourth aspects, in which the worker's input includes a magnification input for enlarging the image, and the work information includes information related to the magnification input.

A sixth aspect is an image processing device according to any one of the first to fifth aspects, wherein the learning unit restricts the use of the image as the training data if the confidence level for the image to which the label has been assigned does not satisfy a predetermined criterion.

A seventh aspect is an image processing device according to any one of the first to sixth aspects, wherein the label assignment unit accepts a correction to the label when the confidence level for the image to which the label has been assigned does not satisfy a predetermined criterion.

An eighth aspect is an image processing device according to any one of the first to seventh aspects, further comprising an inference unit that performs inference on an image to be inspected using learned parameters generated by the learning unit.

A ninth aspect is a visual inspection apparatus comprising an imaging section that images an object, and the image processing apparatus of the eighth aspect.

A tenth aspect is a program executable by a computer, wherein the learning unit of the image processing device of the first aspect uses the image to which the label has been assigned by the label assignment unit of the image processing device as teacher data, performs learning on the image according to the degree of confidence calculated by the confidence degree calculation unit of the image processing device, and includes learned parameters generated by changing a learning rate for updating weights in the learning based on the degree of confidence , and causes the computer to perform inference on an image to be inspected using the learned parameters.

An eleventh aspect is an image processing method including: (a) a computer assigning a label to an image that serves as training data based on an input from a worker, the label including a selection input for selecting one of a plurality of labels; (b) the computer recording a work status of the worker based on the input from the worker; (c) the computer calculating a confidence level for the input from the worker based on the work information recorded by the step (b ); and (d) the computer learning the image to which the label has been assigned by the step (a) as training data in accordance with the confidence level calculated by the step (c), wherein the step (d) includes a step of changing a learning rate when updating weights in the learning, based on the confidence level .

A fourteenth aspect is the image processing method of the thirteenth aspect, further comprising (d) a step of learning the image to which the label has been assigned in the step (a) as training data according to the degree of confidence calculated in the step (c) for the image.

A twelfth aspect is the image processing method of the eleventh aspect, further comprising: (e) a step in which the computer performs inference on the image to be inspected using the learned parameters generated by the step (d).

A thirteenth aspect is a computer-readable program, the computer reading the program causing the computer to execute the image processing method of the eleventh or twelfth aspect.

According to the image processing device of the first aspect, the confidence level of the worker is calculated based on the work information recorded during the label selection work. Therefore, the confidence level of each labeled image can be easily calculated. Furthermore, by performing learning according to the confidence level calculated from the work information, the impact of fluctuations in the worker's label selection on learning can be easily reduced.
Also, learning is done according to confidence level, which allows you to adjust the impact on learning according to confidence level.
In addition, the learning rate is changed based on the confidence level, which allows the degree of influence on learning to be adjusted by adjusting the progress of learning according to the confidence level.

The image processing device of the second aspect can calculate the confidence level based on the time required to select a label.

According to the image processing device of the third aspect, the difference between the label selection time and the average value may depend on the worker's hesitation. Therefore, by using the difference between the label selection time and the average value as a feature related to hesitation, the confidence level can be appropriately calculated.

According to the image processing device of the fourth aspect, the difference between the overall average value and the partial average value may depend on the fatigue of the worker. Therefore, by using the difference between the overall average value and the partial average value as a feature value related to fatigue, the confidence level can be appropriately calculated.

According to the image processing device of the fifth aspect, when a worker performs a magnification operation input, the magnification input is recorded as work information. This makes it possible to calculate the confidence level based on the magnification operation input.

According to the image processing device of the sixth aspect, it is possible to restrict images whose confidence level does not satisfy a predetermined standard from being used as training data.

According to the image processing device of the seventh aspect, it is possible to accept correction of the label for an image whose confidence level does not satisfy a predetermined standard.

1 is a schematic diagram showing a visual inspection apparatus 100 according to a first embodiment. FIG. 2 is a diagram illustrating an example of the hardware configuration of an information processing device 20. 2 is a block diagram showing an example of a functional configuration of an information processing device 20. FIG. FIG. 13 is a diagram showing an example of a labeling window. FIG. 13 is a diagram conceptually illustrating how a label selection task is divided into a plurality of sections from start to finish. FIG. 11 is a block diagram showing a functional configuration of an information processing apparatus 20A according to a second embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the attached drawings. Note that the components described in this embodiment are merely examples, and are not intended to limit the scope of the present invention to those components alone. In the drawings, the dimensions and numbers of each part may be exaggerated or simplified as necessary for ease of understanding.

<1. First embodiment>

Figure 1 is a schematic diagram showing an appearance inspection device 100 of the first embodiment. The appearance inspection device 100 is a device that inspects the appearance of a tablet 9, which is an object. Note that the object is not limited to a tablet 9.

The appearance inspection device 100 includes a camera 10 (imaging unit) and an information processing device 20. The camera 10 is electrically connected to the information processing device 20. The camera 10 includes an image sensor. The camera 10 outputs an image signal to the information processing device 20 according to the intensity of light incident on the image sensor. The camera 10 may capture an image of a tablet 9 that is stopped at a predetermined position, or may capture an image of a tablet 9 that is moving in a predetermined direction by a conveying mechanism such as a belt conveyor. The appearance inspection device 100 may include one camera 10 or multiple cameras 10.

The information processing device 20 inspects the tablet 9 based on an image of the tablet 9 captured by the camera 10 (hereinafter referred to as a "tablet image"). The information processing device 20 can also generate training data from the object image and perform machine learning using the generated training data. The information processing device 20 uses the learned parameters (weights) generated by machine learning to inspect the tablet 9. The information processing device 20 is an example of an image processing device.

FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 20. The information processing device 20 has a configuration as a computer. Specifically, the information processing device 20 has a processor 21, a RAM 23, a storage unit 25, a communication I/F 27, a device I/F 28, and a bus 29. The processor 21, the RAM 23, the storage unit 25, the communication I/F 27, and the device I/F 28 are electrically connected via the bus 29.

The processor 21 includes, for example, a CPU or a GPU. The RAM 23 and the storage unit 25 are storage media capable of reading and writing information. The RAM 23 includes, for example, a DRAM, an SRAM, or an SDRAM. The storage unit 25 includes, for example, a fixed drive such as a hard disk drive or a solid state drive. The storage unit 25 may include a portable removable medium (such as an optical disk, a magnetic disk, or a semiconductor memory) and a media drive that reads and writes from and to the removable medium. The storage unit 25 stores a program P. The processor 21 executes the program P using the RAM 23 as a working area to realize various functions.

The communication I/F 27 is an interface for connecting the information processing device 20 to a network such as the Internet. The program P may be provided or distributed to the information processing device 20 via the network.

The device I/F 28 is an interface for electrically connecting the camera 10, the input unit 31, and the display unit 33 to the information processing device 20. The input unit 31 includes an input device such as a keyboard or a mouse. The display unit 33 displays display information such as characters and figures. The display unit 33 includes, for example, a liquid crystal display. The display unit 33 may include a touch panel. In this case, the display unit 33 can function as an input unit.

FIG. 3 is a block diagram showing an example of the functional configuration of the information processing device 20. The information processing device 20 includes a label assignment unit 41, a work status recording unit 43, a confidence level calculation unit 45, a learning unit 47, and an inference unit 49. The label assignment unit 41, the work status recording unit 43, the confidence level calculation unit 45, the learning unit 47, and the inference unit 49 are functions that are realized in software by the processor 21 executing the program P.

The labeling unit 41 assigns a label to the tablet image serving as training data based on the operator's input. The operator's input includes a selection input for selecting one of multiple types of labels. The labeling unit 41 stores the tablet image 90 with the label (labeled image) in the storage unit 25. The operator's input also includes a magnification input for enlarging the image.

Here, the training data is data that includes a pair of "input data" and "label." The input data used by the information processing device 20 is a tablet image 90. The labels include "OK," "NG," "Maybe OK," and "Maybe NG." "OK" is a label indicating that the tablet image 90 has no defects, i.e., is a good product. "NG" is a label indicating that the tablet image 90 has defects, i.e., is a defective product. "Maybe OK" is a label indicating that there is a relatively high possibility that the tablet image 90 is a good product. "Maybe NG" is a label indicating that there is a relatively high possibility that the tablet image 90 is a defective product. Note that the labels "Maybe OK" and "Maybe NG" are not required.

The work status recording unit 43 records the work status of the worker based on the worker's input. That is, the work status recording unit 43 records information related to the above selection input and information related to the above enlarged input as work information. The work status recording unit 43 saves the work information in the memory unit 25.

The confidence level calculation unit 45 calculates the confidence level of the worker's input based on the work information recorded by the work status recording unit 43. The confidence level is numerical information indicating the worker's degree of confidence in the act of selecting a label for a tablet image 90. The confidence level calculation unit 45 calculates the confidence level for each tablet image 90 to which a label has been applied.

The learning unit 47 performs supervised machine learning using an inference model (inference program) with the tablet image 90 (labeled image) to which the label is assigned by the label assignment unit 41 as training data. The learning unit 47 performs learning according to the confidence level calculated by the confidence level calculation unit 45. The learning according to the confidence level will be described in detail later. The learning unit 47 stores the learned parameters (weights) obtained by learning in the storage unit 25.

The inference unit 49 performs inference on the tablet image 90 to be inspected, using the learned parameters (weights) generated by the learning unit 47. That is, the inference unit 49 performs inference using a trained model (an inference program incorporating trained parameters). In this example, the inference unit 49 determines whether the tablet image 90 to be inspected is "OK" or "NG."

<Label selection work>
4 is a diagram showing an example of a labeling window. The label selection window W1 shown in FIG. 4 is display information displayed on the screen of the display unit 33 by the labeling unit 41. The label selection window W1 is a GUI prepared for an operator to select a label to be applied to each tablet image 90 that is to be used as teacher data (hereinafter referred to as a "label selection operation"). The label selection window W1 has an image display unit 51, a label selection unit 53, an enlargement operation unit 55, and an image selection unit 57.

The image display unit 51 is an area for displaying a tablet image 90 to which a label is to be applied. In the example shown in FIG. 4, the image display unit 51 can simultaneously display three tablet images 90. Of the three tablet images 90, the image displayed in the center is the target for processing. In other words, when the worker operates the label selection unit 53 or the enlargement operation unit 55, the label application unit 41 performs processing on the tablet image 90 to be processed according to the operation content. As shown in FIG. 4, the tablet image 90 to be processed may be displayed larger than the other tablet images 90.

The label selection unit 53 is an area that displays buttons for the worker to select a label. The label selection unit 53 includes label selection buttons 531, 532, 533, and 534. The label selection buttons 531 to 534 correspond to "OK", "NG", "Maybe OK", and "Maybe NG", respectively. That is, when the worker operates (e.g., clicks) any of the label selection buttons 531 to 534, the label assignment unit 41 assigns a label corresponding to the operated selection button to the tablet image 90 to be processed. The input of the worker operating the label selection buttons 531 to 534 is an example of an input for selecting a label. The label assignment unit 41 saves information on the labels assigned to each tablet image 90 in the memory unit 25.

The enlargement operation section 55 is an area that displays a button for performing an operation to enlarge the display magnification of the tablet image 90 displayed on the image display section 51. The enlargement operation section 55 includes an enlargement button 551 and a reset button 552. When the worker operates the enlargement button 551, the labeling section 41 increases the display magnification of the tablet image 90 displayed on the image display section 51. When the worker operates the reset button 552, the labeling section 41 sets the display magnification of the tablet image 90 to a predetermined size.

With the tablet image 90 enlarged, the worker performs a specific operation input to change the portion to be displayed. The labeling unit 41 then changes the displayed portion of the tablet image 90 in response to the operation input. This allows the worker to enlarge and view any portion of the tablet image 90. The input in which the worker operates the enlarge button 551 and the reset button 552 is an example of an enlargement input.

The image selection unit 57 is an area that displays buttons for selecting an image. The image selection unit 57 includes a previous image selection button 571 and a next image selection button 572. When the worker operates the previous image selection button 571, the labeling unit 41 changes the tablet image 90 to be processed that is currently displayed on the image display unit 51 to the previous tablet image 90. When the worker operates the next image selection button 572, the labeling unit 41 changes the tablet image 90 to be processed that is currently displayed on the image display unit 51 to the next tablet image 90. In this way, the image display unit 51 can display a group of tablet images 90 that are prepared in advance in order. The order in which the group of tablet images 90 are displayed can be appropriately set according to, for example, the acquisition date and time or the file name of each tablet image 90. The image selection unit 57 may have an operation unit that allows the worker to arbitrarily select a specific tablet image 90 from among the multiple tablet images 90.

The work status recording unit 43 records the operations performed by the worker using the label selection window W1. For example, the work status recording unit 43 records the time when the worker operates each button of the label selection unit 53, the enlargement operation unit 55, or the image selection unit 57 as work information.

The confidence level calculation unit 45 acquires a feature amount from the work information recorded by the work status recording unit 43 in order to calculate the confidence level. The confidence level calculation unit 45 acquires, for example, the "label selection time" as a feature amount. The "label selection time" is the time required for the worker to select a label for one tablet image 90. For example, in the label selection window W1 shown in FIG. 4, the time from the moment one tablet image 90 is selected as a processing target via the image selection unit 57 to the moment the label of the tablet image 90 is specified via the label selection unit 53 may be the "label selection time".

When selecting a label, an operator cannot always select the label with 100% confidence every time. For example, if a defect in an image is not clear and it is difficult to tell whether or not it contains a defect, the operator may enlarge the image or check it carefully. In this way, the operator may become hesitant and it may take a long time to select a label. In addition, if the labeling task takes a long time or the operator's physical condition, etc., the operator may become tired. Indecision and fatigue may affect the operation input during the selection task. The label selection time may be a value that reflects the operator's indecision and fatigue.

Figure 5 is a conceptual diagram showing how the label selection task is divided into multiple intervals from start to finish. In Figure 5, the horizontal axis indicates time. In the label selection task shown in Figure 5, the total work time from the start to the end of the task is "T". In addition, in the label selection task, it is assumed that the worker has assigned labels to N tablet images 90. As shown in Figure 5, the confidence calculation unit 45 calculates the "overall average value" and the "interval average value" as the average values of the label selection time.

The "overall average value" is the average of all label selection times in the entire label selection task performed by the worker. For example, in the case of the label selection task shown in FIG. 5, the confidence calculation unit 45 calculates the average value of each label selection time for N tablet images 90 as the "overall average value." Note that the confidence calculation unit 45 may calculate the value obtained by dividing the total task time "T" by the total number N of tablet images 90 (= T/N) as the overall average value.

The "interval average value" is the average (partial average value) of the label selection time in each interval when the total work time T for N tablet images 90 is divided into multiple intervals. The example shown in FIG. 5 shows each interval (first interval, second interval, third interval, ...) in which the total work time T is divided into a certain time unit (for example, one hour unit). The confidence calculation unit 45 may divide the total work time T into units of a number of tablet images 90 (for example, 100 sheets). For each interval, the confidence calculation unit 45 calculates the average of the label selection time measured in each interval as the "interval average value". In addition, if the number of tablet images 90 to which a worker has applied a label in a certain interval is n and the work time in that interval is t, the value obtained by dividing the work time t by the number n (= t/n) may be used as the "interval average value".

The confidence calculation unit 45 calculates a feature amount related to "confusion" (hereinafter referred to as "confusion feature amount") and a feature amount related to "fatigue" (hereinafter referred to as "fatigue feature amount") for each tablet image 90 based on the "label selection time", "overall average value", and "section average value". Specifically, the confidence calculation unit 45 calculates, as the confusion feature amount, the difference between the label selection time and the section average value of the section to which the label selection time belongs. In addition, the confidence calculation unit 45 calculates, as the fatigue feature amount, the difference between the overall average value and the section average value.

As the task time increases, the worker becomes tired, and the section average value tends to become longer. In other words, the section average value in the latter half of the task tends to be longer than the section average value in the first half of the task. For this reason, by calculating the difference between the section average value and the overall average value, the fatigue of the worker in each section can be quantified. Furthermore, by setting the hesitation feature to the difference between the label selection time and the section average value, the component of the worker's "fatigue" that depends on time can be removed, and the component that depends on "hesitation" can be extracted. Note that the confidence calculation unit 45 may calculate the difference between the label selection time and the overall average value as the hesitation feature.

The confidence level calculation unit 45 calculates the confidence level for each labeled image based on the hesitation feature amount and the fatigue feature amount. The confidence level calculation unit 45 may calculate the confidence level based on a function that uses the hesitation feature amount and the fatigue feature amount as variables. As an example, the confidence level calculation unit 45 may take the value obtained by adding the hesitation feature amount and the fatigue feature amount as the confidence level. In this case, the confidence level calculation unit 45 may weight the hesitation feature amount and the fatigue feature amount and add them together.

The confidence level calculation unit 45 may calculate the confidence level of an image labeled with "Maybe OK" or "Maybe NG" so that it is relatively lower than that of an image labeled with "OK" or "NG". This makes it possible to relatively lower the confidence level of an image labeled with "Maybe OK" or "Maybe NG" by the worker.

When the operator performs an operation to enlarge the tablet image 90 via the enlargement operation unit 55, the confidence level calculation unit 45 may calculate the confidence level according to the content of the enlargement operation (e.g., the enlargement ratio). When it is difficult to visually recognize defects in the tablet image 90, it is expected that the operator will enlarge the image. Therefore, the greater the enlargement ratio of the tablet image 90, the smaller the confidence level calculated by the confidence level calculation unit 45, thereby making it possible to obtain an appropriate confidence level.

<Supervised machine learning>

The learning unit 47 uses the confidence level calculated by the confidence level calculation unit 45 to learn the inference model while adjusting the degree of influence on learning. The learning of the inference model is described below, but the learning method is not limited to the following content.

The learning unit 47 performs learning using a predetermined inference model with the labeled images as training data. As the inference model, for example, a logistic regression model can be used. As described above, the learning unit 47 performs learning according to the confidence level. Specifically, the learning unit 47 changes the learning rate when updating the weights according to the confidence level.

We will use formulas to explain learning when using a logistic regression model as an inference model. If the input feature vector is x, the model output is y, and the model weight matrix is w, the logistic regression model is expressed by the following formula.

The learning unit 47 normalizes the confidence calculated by the confidence calculation unit 45 so that it falls within the value range from 0 to 1. Then, the learning unit 47 multiplies the learning rate coefficient at the time of weight update by the normalized confidence. The teaching class is u, the learning rate coefficient is η, and the normalized confidence is α. Then, the weight update formula is expressed by the following formula.

By optimizing based on the above formula (2), the learning unit 47 can perform learning according to the confidence level of each labeled image. For example, in the case of a labeled image with a normalized confidence level of "0.1", the weight update amount is 1/10 compared to the case of a labeled image with a normalized confidence level of "1". In this way, the influence of a labeled image with a low confidence level on learning is smaller than that of a labeled image with a high confidence level. By changing the learning rate according to the confidence level, the progress of learning is adjusted, and thus the influence on learning can be adjusted. The learning unit 47 stores the weights updated by learning in the memory unit 25 as learned parameters (see FIG. 3).

If the confidence level of a labeled image does not meet a predetermined standard, the learning unit 47 may restrict the use of the labeled image as training data. Specifically, "restricting use as training data" is not limited to "not using as training data" and may include, for example, requesting confirmation or permission for use as training data.

The criterion for restricting use as training data may be that the confidence level is less than a predetermined threshold, or that the confidence level is an outlier. For example, the learning unit 47 may determine a value that deviates from the 1σ point in the distribution of confidence levels (a value that is outside the range of the average confidence level ±σ) as an outlier. The learning unit 47 may also determine the outlier based on various tests such as a Q test or a Smirnoff test. By restricting the use of a labeled image as training data according to the confidence level of the labeled image, the learning unit 47 can use a labeled image with an appropriate confidence level for learning.

If the confidence level of a labeled image does not meet a predetermined standard, the label assignment unit 41 may accept a change to the label assigned to the labeled image. For example, the label assignment unit 41 causes the display unit 33 to display the label selection window W1 shown in FIG. 4, and causes the image display unit 51 to display the labeled image whose confidence level does not meet the standard. The label assignment unit 41 may then correct the label based on the operator's operation of the label selection unit 53. This allows the operator to be given an opportunity to correct the label, making it possible to generate training data to which accurate labels have been assigned.

As described above, according to the information processing device 20, when a worker performs label selection work, the work situation is simultaneously recorded as work information. Therefore, along with the labeled images that serve as training data, a confidence level that indicates the worker's confidence can be easily obtained based on the work information. Using the confidence level, a trained inference model can be created in which the degree of influence on learning for labeled images with large fluctuations in label selection is adjusted. In other words, by increasing the influence on learning of labeled images with high confidence levels and conversely reducing the influence on learning of data with low confidence levels, the influence on learning of fluctuations in label selection by the worker can be easily reduced. This makes it possible to create a trained inference model that is capable of detecting defects with high accuracy.

In addition, by using the confidence level, it is possible to appropriately restrict the use of labeled images as training data and extract labeled images that require label correction.

<2. Second embodiment>
Next, a second embodiment will be described. In the following description, elements having the same functions as elements already described will be given the same reference numerals or reference numerals with an additional alphabetical character, and detailed description thereof may be omitted.

In the first embodiment, the confidence level is calculated by calculation. However, the confidence level may be calculated using an inference model that estimates the confidence level (hereinafter, referred to as a "confidence level inference model"). As the confidence level inference model, for example, a logistic regression model may be used. Note that as the confidence level inference model, a model that outputs a probability value for each class, such as a support vector machine (SVM), may also be used.

FIG. 6 is a block diagram showing the functional configuration of an information processing device 20A according to the second embodiment. The confidence level learning unit 61 shown in FIG. 6 uses the confidence level learning data to train a confidence level inference model, thereby generating learned confidence levels. The confidence level inference model takes as input data work information that records the work status of label selection, and outputs the confidence level for the input data. For this reason, the confidence level learning data may have a pair of work information as input data and a confidence level as a correct label. The work information recorded by the work status recording unit 43 may be used as the work information of the input data.

The confidence level learning unit 61 also assigns a label of "confident", indicating a high level of confidence, to tablet images 90 to which the label assignment unit 41 has assigned "OK" or "NG". The confidence level learning unit 61 also assigns a label of "not confident", indicating a low level of confidence, to tablet images 90 to which the label assignment unit 41 has assigned "Maybe OK" or "Maybe NG". In this way, confidence level learning data is prepared for each tablet image 90, which includes a pair of work information and a label related to the level of confidence ("confident" or "not confident").

The confidence learning unit 61 may obtain feature amounts from the work information as input data. The feature amounts include, for example, "label selection time," "overall average value," or "interval average value." The feature amounts may also include "overall median," "overall standard deviation," "interval median value," or "interval standard deviation." The "overall median" is the median of all label selection times obtained by the label selection work of the workers. The "overall standard deviation" is the standard deviation of all label selection times. The "interval median" is the median of multiple label selection times included in each interval when the total work time is divided into multiple intervals. The "interval standard deviation" is the standard deviation of multiple label selection times included in each interval. The confidence learning unit 61 may also obtain, as feature amounts, "difference between label selection time and interval average value," and "difference between interval average value and total recovery average value." The feature amounts may also include "magnification rate."

The confidence level learning unit 61 learns the confidence level inference model using confidence level learning data that has pairs of task information (feature values) and labels related to confidence levels. In detail, the weights are updated so that the output when the feature values obtained from the task information are input to the confidence level inference model matches the label ("confident" or "unconfident") indicating the confidence level corresponding to the input. The confidence level learning unit 61 stores the weights updated by learning in the memory unit 25 as confidence level learned parameters.

The confidence level calculation unit 45A calculates the confidence level using the confidence level inference model used for learning and the confidence level learned parameters generated by the confidence level learning unit 61. The confidence level calculation unit 45A may take the output value of "confident" output by the confidence level inference model as the confidence level.

The learning unit 47 (see FIG. 3) performs learning based on the confidence level calculated by the confidence level calculation unit 45A. As in the first embodiment, the learning unit 47 may appropriately restrict the use of labeled images as training data according to the confidence level calculated by the confidence level calculation unit 45A. Furthermore, the label assignment unit 41 may extract labeled images for which label correction is to be performed based on the confidence level calculated by the confidence level calculation unit 45A.

Although this invention has been described in detail, the above description is illustrative in all respects and does not limit the invention. It is understood that countless variations not illustrated can be imagined without departing from the scope of this invention. The configurations described in the above embodiments and variations can be combined or omitted as appropriate as long as they are not mutually contradictory.

100 Appearance inspection device 10 Camera 20 Information processing device (image processing device)
20A Information processing device 21 Processor 31 Input section 41 Labeling section 43 Work status recording section 45, 45A Confidence level calculation section 47 Learning section 49 Inference section 51 Image display section 53 Label selection section 55 Enlargement operation section 57 Image selection section 61 Confidence level learning section 9 Tablet 90 Tablet image P Program W1 Label selection window

Claims (13)

An image processing device,
a label assignment unit that assigns a label to an image serving as training data based on an input by an operator, the input including a selection input for selecting one of a plurality of types of labels;
a work status recording unit that records the work status of the worker based on the input of the worker;
a confidence level calculation unit that calculates a confidence level for the input of the worker based on the work information recorded by the work status recording unit;
a learning unit that performs learning according to the degree of confidence calculated by the degree of confidence calculation unit for the image, using the image to which the label is assigned by the label assignment unit as training data;
Equipped with
The learning unit changes a learning rate used when updating weights in the learning, based on the degree of confidence . 2. The image processing device according to claim 1,
The work information includes a label selection time, which is a time required for the worker to select the label. 3. The image processing device according to claim 2,
The task information includes an average value of the label selection time for each of the plurality of images,
The confidence level calculation unit calculates the confidence level based on a difference between the label selection time and the average value. 4. The image processing device according to claim 2,
The work information includes:
an overall average value that is an average of the label selection times for the first number of images;
a partial average value that is an average of the label selection times for a portion of the first number of the images;
Including,
The confidence level calculation unit calculates the confidence level based on a difference between the overall average value and the partial average value. 5. The image processing device according to claim 1,
the input by the worker includes a magnification input for magnifying the image;
The image processing device, wherein the work information includes information regarding the enlarged input. 6. The image processing device according to claim 1 ,
The image processing device, wherein the learning unit restricts use of the image as the training data if the confidence level for the image to which the label has been assigned does not satisfy a predetermined standard. 7. The image processing device according to claim 1 ,
The image processing device, wherein the label assignment unit accepts a correction of the label when the degree of confidence for the image to which the label has been assigned does not satisfy a predetermined criterion. 8. The image processing device according to claim 1 ,
an inference unit that performs inference on an image to be inspected by using the trained parameters generated by the training unit;
The image processing device further comprises: A visual inspection apparatus comprising:
An imaging unit that captures an image of an object;
The image processing device according to claim 8 ;
An appearance inspection device comprising: A computer-executable program,
2. A program comprising: the learning unit of the image processing device according to claim 1 , using the image to which the label has been assigned by the label assignment unit of the image processing device as teacher data, performing learning on the image according to the degree of confidence calculated by the confidence degree calculation unit of the image processing device, and including learned parameters generated by changing a learning rate for updating weights in the learning based on the degree of confidence , and causing the computer to perform inference on the image to be inspected using the learned parameters. 1. An image processing method, comprising:
(a) a step in which a computer assigns a label to an image serving as training data based on an input from an operator, the input including a selection input for selecting one of a plurality of labels;
(b) the computer recording the work status of the worker based on the input of the worker;
(c) calculating a degree of confidence in the input by the computer based on the work information recorded in the step (b);
(d) performing learning by the computer using the image to which the label is assigned in the step (a) as training data in accordance with the degree of confidence calculated in the step (c) for the image;
Including,
The image processing method , wherein the step (d) includes a step of changing a learning rate used when updating weights in the learning, based on the degree of confidence, by the computer . 12. The image processing method according to claim 11 ,
(e) performing inference by the computer on the image to be inspected using the learned parameters generated by (d);
The image processing method further comprises: A computer-readable program, the computer being capable of reading the program and
A program for causing an image processing method according to claim 11 to be executed.

Images