JP7619215B2 - Inspection system, method, and program - Google Patents

Description

This disclosure relates to an inspection system, method, and program.

In recent years, machine learning models have been used for quality control in the manufacturing field from the perspective of labor saving. For example, a computer on which a machine learning model has been built predicts whether a product meets predetermined inspection standards. Patent Document 1 also describes that a computer on which a machine learning model has been built predicts, from input equipment data for a manufacturing process, whether the quality of a product that has gone through that manufacturing process will meet predetermined conditions.

It is known that machine learning models deteriorate with continued use. When a machine learning model deteriorates, its prediction accuracy decreases. For this reason, Patent Document 1 describes a method for determining that a machine learning model has deteriorated when the prediction accuracy of the machine learning model changes, and generating a new machine learning model.

JP 2019-87101 A

However, even if the machine learning model has not actually deteriorated, the prediction accuracy of the machine learning model may decline due to other conditions. For this reason, as described in Patent Document 1, if a change in the prediction accuracy of the machine learning model is uniformly regarded as deterioration of the machine learning model, there is a problem in that the machine learning model may be erroneously determined to have deteriorated even when it has not deteriorated.

This disclosure can be realized in the following forms:

(1) According to an embodiment of the present disclosure, there is provided an inspection system that performs inspection using a machine learning model. The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event for an inspection target. The inspection system includes an inspection unit that repeatedly performs a prediction for the inspection target using the machine learning model with data for the inspection target as input data, and outputs prediction result data indicating the predicted result, a storage unit that stores the prediction result data output by the inspection unit, and an evaluation unit that evaluates whether the input data is within a range assumed at the time of learning the machine learning model based on a trend of time-series change in the predicted result for the inspection target indicated by the prediction result data stored in the storage unit, and outputs an evaluation result.
According to this aspect, when determining whether or not the machine learning model has deteriorated, not only the prediction accuracy of the machine learning model but also other conditions, such as whether or not the input data is within an expected range, can be taken into consideration. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent a mistaken determination that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions.
(2) In the inspection system of the above embodiment, the machine learning model may be one that has undergone machine learning to predict which of a plurality of preset classifications the inspection object will be classified into. The inspection unit predicts a predicted probability indicating the possibility that the inspection object will be classified into each of a plurality of classifications. The inspection unit determines that the inspection object will be classified into the classification with the highest predicted probability value. The inspection unit outputs prediction result data including a predicted predicted probability for the classification determined for the inspection object. The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection object. The evaluation unit may evaluate whether or not the input data is within a range assumed at the time of learning the machine learning model, based on a trend of a time series change in the predicted predicted probability for the classification determined for the inspection object, which is indicated by the prediction result data stored in the storage unit.
According to this aspect, it is possible to evaluate whether the input data is within the range assumed when training the machine learning model based on the trend of time-series changes in the predicted prediction probability for the classification determined for the test subject.
(3) In the inspection system of the above embodiment, the machine learning model may be one that has undergone machine learning to predict whether or not the inspection object is within a normal range. The inspection unit predicts whether or not the inspection object is within a normal range. The inspection unit outputs prediction result data including a value representing a ratio of the number of inspection objects predicted to be within the normal range to the total number of inspection objects inspected in a set period. The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection object. The evaluation unit may evaluate whether or not the input data is within a range assumed at the time of learning the machine learning model, based on a trend of time-series change in the ratio indicated by the prediction result data stored in the storage unit.
According to this aspect, it is possible to evaluate whether the input data is within the range assumed when training the machine learning model from the trend of time-series changes in the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested.
(4) In the inspection system of the above aspect, the input data may be image data representing the appearance of the inspection target acquired by a sensor.
According to this aspect, the inspection system can perform visual inspection of the inspection target using the image data, and can evaluate whether the input data is within the range assumed at the time of learning the machine learning model based on the trend of time-series changes in the prediction result predicted by the machine learning model.
(5) The input data may be voice data representing the operating state of the inspection target obtained by a sensor.
According to this aspect, the inspection system can inspect the inspection target for abnormal sounds, and can evaluate whether the input data is within the range assumed at the time of learning the machine learning model, based on the tendency of time-series changes in the prediction results predicted by the machine learning model.
The present disclosure may be realized in various forms other than an inspection system, for example, a method executed by a computer that performs inspection using a machine learning model, or a computer program that realizes the method.

FIG. 1 is a diagram illustrating a functional configuration of an inspection system according to a first embodiment. 1 is a diagram illustrating a hardware configuration of an inspection apparatus according to a first embodiment. FIG. 2 is a diagram illustrating a hardware configuration of an evaluation device according to the first embodiment. 4 is a diagram showing an example of data stored in a storage unit according to the first embodiment; FIG. 1 is a flowchart of a classification process according to the first embodiment. 4 is a flowchart of an evaluation process according to the first embodiment. FIG. 11 is a diagram showing a functional configuration of an inspection system according to a second embodiment. FIG. 11 is a diagram showing an example of data stored in a storage unit according to the second embodiment; 10 is a flowchart of an inspection process according to a second embodiment. 10 is a flowchart of an evaluation process according to a second embodiment.

A. First Embodiment Fig. 1 is a diagram showing the functional configuration of an inspection system 1000 according to a first embodiment. The inspection system 1000 includes an inspection device 100 and an evaluation device 200. The inspection device 100 and the evaluation device 200 are different computers installed in a server room in a factory.

A sensor 50 is connected to the inspection device 100 via a network. In this embodiment, an example will be described in which the sensor 50 is a camera. The sensor 50 photographs a part to be inspected by the inspection device 100 at a predetermined position on the production line. The inspection object is, for example, an automobile part such as a body panel, an engine cylinder block, or a gear incorporated in a transmission. In this embodiment, the inspection objects are two types of parts, part A1 and part A2. Therefore, the object photographed by the sensor 50 is either part A1 or A2. The sensor 50 transmits image data of either part A1 or A2 to the inspection device 100.

The inspection device 100 uses image data supplied from the sensor 50 as input data and uses a machine learning model to classify the inspection object and determine its pass/fail quality. In this embodiment, the machine learning model is a trained model that has undergone machine learning to predict predetermined events regarding the inspection object. The evaluation device 200 evaluates the results of the inspection device 100's determination of the inspection object.

Figure 2 is a diagram showing the hardware configuration of the inspection device 100. The inspection device 100 includes a processor 11, a main memory unit 12, an auxiliary memory unit 13, an interface 14, and a communication unit 15. The main memory unit 12, the auxiliary memory unit 13, the interface 14, and the communication unit 15 are connected to the processor 11 via an internal bus 19.

The processor 11 is, for example, a CPU (Central Processing Unit). The processor 11 loads a program stored in the auxiliary memory unit 13 into the main memory unit 12 and executes it, thereby realizing each function of the inspection device 100.

The main memory unit 12 is, for example, a main storage device including a RAM (Random Access Memory). Programs are loaded into the main memory unit 12 from the auxiliary memory unit 13. The main memory unit 12 is also used as a working area when the processor 11 executes a program.

The auxiliary storage unit 13 is, for example, a hard disk drive (HDD), a solid state drive (SSD), or a solid state hybrid drive (SSHD). The auxiliary storage unit 13 stores programs executed by the processor 11 and various data used to execute the programs. For example, the auxiliary storage unit 13 stores programs that the inspection device 100 uses to inspect an inspection target. Under the control of the processor 11, the auxiliary storage unit 13 supplies the processor 11 with data used by the processor 11 and stores the data supplied from the processor 11.

The interface 14 is, for example, an I/O (Input/Output) port such as a serial port or a USB (Universal Serial Bus) port. Input devices such as a keyboard, a mouse, and a touch panel are connected to the interface 14. The interface 14 outputs information input by a user via the input device to the processor 11. A display device 61 is also connected to the interface 14. The display device 61 is, for example, a liquid crystal display or an organic EL (Electro Luminescence) display. The interface 14 outputs an image supplied from the processor 11 to the display device 61.

The communication unit 15 includes a network interface circuit for communicating with an external device. The communication unit 15 receives a signal from the outside and outputs data indicated by this signal to the processor 11. The communication unit 15 also transmits a signal indicating the data output from the processor 11 to the external device.

Figure 3 is a diagram showing the hardware configuration of the evaluation device 200. The evaluation device 200 includes a processor 21, a main memory unit 22, an auxiliary memory unit 23, an interface 24, and a communication unit 25. The main memory unit 22, the auxiliary memory unit 23, the interface 24, and the communication unit 25 are connected to the processor 21 via an internal bus 29.

The processor 21 has the same functions as the processor 11 of the inspection device 100. The main memory unit 22 has the same functions as the main memory unit 12 of the inspection device 100. The auxiliary memory unit 23 has the same functions as the auxiliary memory unit 13 of the inspection device 100. The interface 24 has the same functions as the interface 14 of the inspection device 100. A display device 62 is connected to the interface 24. The display device 62 is, for example, a liquid crystal display or an organic EL display. The interface 24 outputs an image supplied from the processor 21 to the display device 62. The communication unit 25 has the same functions as the communication unit 15 of the inspection device 100.

Next, the functions of the inspection device 100 will be described with reference to FIG. 1. The inspection device 100 includes a data acquisition unit 110, a classification unit 120, and an inspection unit 130.

When the data acquisition unit 110 receives image data of an object to be inspected from the sensor 50, it stores the image data in a predetermined area. The data acquisition unit 110 transmits the path of the storage destination of the image data to the classification unit 120. The function of the data acquisition unit 110 is realized by the processor 11 executing a program for processing related to data acquisition.

The classification unit 120 uses a trained machine learning model to determine which of the two types of parts the inspection target corresponds to. The machine learning model is generated by any machine learning method. Examples of machine learning techniques include random forest and SVM (support vector machine). The machine learning model used by the classification unit 120 is generated by machine learning to predict whether the inspection target corresponds to part A1 or part A2. This machine learning is performed using a large amount of image data of parts A1 and A2 as training data.

The reason why a large number of image data are used as training data is as follows. In the manufacturing line, the sensor 50 captures the parts transported by the conveyor. The position where the sensor 50 is installed is fixed. On the other hand, the position where the parts are placed on the conveyor is not always the same. For example, when comparing multiple images showing the same type of parts, the position and inclination of the parts in each image may differ. Therefore, it is preferable that the image data used as training data is image data with various variations. For example, it may include images taken while shifting the position where the parts are placed little by little, and images taken while rotating the parts little by little. By using image data with various variations as training data, the machine learning model can correctly determine the type of the part to be inspected.

In this embodiment, the prediction accuracy of the machine learning model used in the classification unit 120 is assumed to meet a predetermined standard. For example, it is a machine learning model that has been used for a short period of time since its generation, and whose index value for evaluating the prediction accuracy meets a predetermined standard.

The classification unit 120 uses the image data of the inspection target as input data, and utilizes a trained machine learning model to predict the probability that the inspection target corresponds to each of parts A1 and A2. Hereinafter, this probability may be referred to as the predicted probability.

Furthermore, the classification unit 120 determines whether the inspection target is part A1 or part A2 from the predicted probability. For example, the classification unit 120 predicts that the probability that the inspection target corresponds to part A1 is 70 percent, and the probability that the inspection target corresponds to part A2 is 30 percent. In this case, the classification unit 120 determines that the inspection target is part A1. Also, for example, the classification unit 120 predicts that the probability that the inspection target corresponds to part A1 is 15 percent, and the probability that the inspection target corresponds to part A2 is 85 percent. In this case, the classification unit 120 determines that the inspection target is part A2. The classification unit 120 notifies the evaluation device 200 of the results of the prediction and determination.

The classification unit 120 also outputs to the inspection unit 130 the determination result as to which part the inspection target corresponds to, and the path to the storage destination of the image data. The function of the classification unit 120 is realized by the processor 11, for example, by executing application software in which a trained machine learning model is implemented. The classification unit 120 is also referred to as an inspection unit.

The inspection unit 130 uses the trained machine learning model to perform a visual inspection of the inspection target. A visual inspection is an inspection to check for defects in appearance, such as foreign matter attached to the surface of the part or deformation of the part.

The machine learning model used by the inspection unit 130 is generated by machine learning to predict whether the inspection target is a good or defective product. This machine learning is performed using image data of the parts to which a judgment value of good or defective is associated as training data. The function of the inspection unit 130 is realized by the processor 11, for example, by executing application software in which a learned machine learning model is implemented. Note that the machine learning model used by the inspection unit 130 is different from the machine learning model used by the classification unit 120.

The inspection unit 130 uses as input data the image data and the result of the determination by the classification unit 120 as to which part the inspection target corresponds to, and predicts the probability that the part shown in the image data corresponds to a good product and a defective product. In this embodiment, the inspection unit 130 determines that the inspection target is a good product if the probability that it is a good product is higher than the probability that it is a defective product. The inspection unit 130 determines that the inspection target is a defective product if the probability that it is a defective product is higher than the probability that it is a good product.

For example, suppose that the inspection unit 130 predicts that the part A1 being inspected has an 80 percent probability of being a good product and a 20 percent probability of being a defective product. In this case, the inspection unit 130 determines that the part being inspected is a good product. The inspection unit 130 outputs to the display device 61 an image that displays the probability that the part being inspected is a good product and a defective product, respectively, and the determination result that the part is a good product or a defective product.

In this embodiment, the prediction accuracy of the machine learning model used by the inspection unit 130 is assumed to meet a predetermined level. In this embodiment, the evaluation device 200 does not evaluate the judgment result of the inspection unit 130. Therefore, the inspection unit 130 does not output the judgment result to the evaluation device 200.

Next, the functions of the evaluation device 200 will be described. The evaluation device 200 includes an accumulation unit 210 and an evaluation unit 220. In this embodiment, an example will be described in which the evaluation device 200 evaluates the prediction results obtained by the classification unit 120 of the inspection device 100.

Figure 4 is a diagram showing an example of data stored in the storage unit 210. The storage unit 210 stores data supplied from the classification unit 120. The data supplied from the classification unit 120 to the storage unit 210 includes the date and time, the determination result as to which part the inspection target corresponds to, the predicted probability that the inspection target corresponds to each part, and the path to the storage destination of the image data. The function of the storage unit 210 is realized by the processor 11 executing software that saves and reads data on request, such as a database management system.

The evaluation unit 220 shown in FIG. 1 evaluates whether the data input to the machine learning model used by the classification unit 120 during the specified evaluation period is within the range expected when the machine learning model was trained.

In this embodiment, the evaluation unit 220 detects anomalies by detecting whether the time series data of the predicted probabilities contains outliers. For example, the evaluation unit 220 detects outliers contained in the time series data of the predicted probabilities using an existing method for outlier detection.

When the evaluation unit 220 detects that the time series data of the prediction probability contains an outlier, it determines that the data input to the machine learning model is not within the range expected when the machine learning model was trained. As described above, in this embodiment, the prediction accuracy of the machine learning model meets a predetermined level. In other words, the machine learning model has not deteriorated. In addition, when input data is input, the machine learning model outputs result data regarding the prediction. Therefore, when an abnormality is detected in the trend of time series changes in the output result data, it is considered that the input data was data that was not within the expected range. The function of the evaluation unit 220 is realized by the processor 21 executing a program for processing related to the evaluation.

According to the above aspect, when the test subject is determined to be one of the categories, the evaluation device 200 can evaluate whether or not the input data is within the range assumed when the machine learning model was trained, based on the trend of time-series changes in the predicted probability predicted for that category.

Figure 5 is a flowchart of the classification process in which the inspection device 100 classifies parts from image data. Each time the inspection device 100 receives image data from the sensor 50, it executes the following classification process.

In step S101, the data acquisition unit 110 receives image data from the sensor 50. The data acquisition unit 110 performs preprocessing such as noise removal and binarization on the received image data. The data acquisition unit 110 stores the preprocessed image data in a specified area in the auxiliary memory unit 13 of the inspection device 100. The data acquisition unit 110 transmits to the classification unit 120 the path to the storage destination of the preprocessed image data.

In step S102, the classification unit 120 first reads out the image data stored in the path notified by the data acquisition unit 110. The classification unit 120 uses the image data as input data and predicts the predicted probability that the part to be inspected shown in the image corresponds to each of the two types of parts. In step S103, the classification unit 120 determines which part the inspection target is based on the predicted probability. Specifically, the classification unit 120 determines that the inspection target corresponds to the part with the higher predicted probability value.

In step S104, the classification unit 120 transmits data to the evaluation device 200, including the predicted probability that the inspection target corresponds to each part, the determination result as to which part the inspection target corresponds to, the path to the storage destination of the image data, and date and time information. This data is also called prediction result data. The date and time indicate when the classification unit 120 made the prediction. The date and time are also called time information. This completes the series of steps in the classification process.

Figure 6 is a flowchart of the evaluation process by the evaluation device 200. The evaluation device 200 executes the following evaluation process each time it receives prediction result data from the inspection device 100.

In step S111, the storage unit 210 receives data from the classification unit 120, including the predicted probability that the inspection target corresponds to each part, the determination result as to which part the inspection target corresponds to, the path to the storage destination of the image data, and the date and time. In step S112, the storage unit 210 stores the data received from the classification unit 120.

In step S113, when it is time to evaluate the input data (step S113; YES), the evaluation unit 220 executes the process of step S114. On the other hand, in step S113, when it is not time to evaluate the input data (step S113; YES), the storage unit 210 executes the process of step S211 again.

In step S114, the evaluation unit 220 detects outliers in the time series data of the predicted probability. First, the evaluation unit 220 reads out from the storage unit 210 the predicted probability values of the inspection objects predicted to correspond to the part A1 in the past week, sorted by date and time. Next, the evaluation unit 220 reads out from the storage unit 210 the predicted probability values of the inspection objects predicted to correspond to the part A2 in the past week, sorted by date and time. The evaluation unit 220 determines whether or not the time series data of the predicted probability of the inspection objects predicted to correspond to the part A1 in the past week includes an outlier. Furthermore, the evaluation unit 220 determines whether or not the time series data of the predicted probability of the inspection objects predicted to correspond to the part A2 in the past week includes an outlier. If the evaluation unit 220 detects that the time series data of the predicted probability contains an outlier, it determines that the input data to the machine learning model is not within the range assumed when the machine learning model was trained.

In step S115, the evaluation unit 220 presents the evaluation result for the input data based on the outlier detection result. For example, the evaluation unit 220 outputs to the display device 62 an image indicating that there was input data that was not within the expected range, the predicted probability value detected as the outlier, the date and time when the predicted probability value was predicted, and the path indicating the storage location of the input data used. This allows the user to know that data that was not within the expected range when the machine learning model was trained was input to the machine learning model. This completes the flow of the evaluation process.

As described above, in the first embodiment, the test object is evaluated based on the time-series change tendency of the predicted probability that it is predicted to fall into a classification, to determine whether the input data is within the range assumed when training the machine learning model.

In this way, whether or not the input data is within the range expected when the machine learning model was trained is evaluated based on the trend of time-series changes in the prediction results predicted by the machine learning model. This makes it possible to consider not only the prediction accuracy of the machine learning model, but also other conditions, such as whether or not the input data is within the expected range, when determining whether or not the machine learning model has deteriorated. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent erroneous judgment that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions.

B. Second embodiment Fig. 7 is a diagram showing the functional configuration of an inspection system 1001 according to a second embodiment. The following mainly describes the configuration different from the first embodiment. The inspection system 1001 includes an inspection device 100 and an evaluation device 200.

The inspection device 100 performs an abnormal sound inspection. The object of inspection by the inspection device 100 is, for example, an automobile engine. In this embodiment, the sensor 50 is a microphone. The sensor 50 is connected to the inspection device 100 via a network. The sensor 50 collects the sound of the automobile engine before shipment at a predetermined position on the production line. In this embodiment, audio data that indicates the operating state of the engine that is the object of inspection becomes input data for the abnormal sound inspection. The sensor 50 transmits the collected audio data to the inspection device 100.

The inspection device 100 includes a data acquisition unit 110 and an inspection unit 130. In the second embodiment, the inspection device 100 does not include a classification unit 120.

When the data acquisition unit 110 receives engine sound audio data from the sensor 50, it stores the audio data in a specified area. The data acquisition unit 110 transmits the path to the storage destination of the audio data to the inspection unit 130.

The inspection unit 130 uses a trained machine learning model to predict whether the inspection target is normal or not. The machine learning model used by the inspection unit 130 is generated by machine learning to predict whether the inspection target is normal or not. This machine learning uses a large amount of voice data that collects normal engine sounds as training data. Many machines are in operation at the manufacturing site, and sounds emitted from these machines are also included in the voice data that is the inspection target. Therefore, it is preferable that the voice data used as training data is voice data with various variations. For example, it may include voice data that collects normal engine sounds in an environment where surrounding noise is blocked, and voice data that collects normal engine sounds in an environment with surrounding noise. By using voice data with various variations as training data, the machine learning model can correctly determine whether the inspection target is good or bad.

The inspection unit 130 uses the audio data supplied from the data acquisition unit 110 as input data to predict the degree of abnormality, which is an index showing how far the engine sound being inspected is from a normal engine sound. In this embodiment, the degree of abnormality is an index in which the higher the numerical value, the closer the inspection object is to an abnormal state. Furthermore, the inspection unit 130 judges whether the inspection object is normal or not. If the calculated degree of abnormality is equal to or less than a preset threshold, the inspection unit 130 judges that the inspection object is normal. On the other hand, if the calculated degree of abnormality exceeds the preset threshold, the inspection unit 130 judges that the inspection object is abnormal.

Furthermore, the inspection unit 130 counts the number of inspections performed during the set period and the number of inspection subjects determined to be normal during that period. The set period is, for example, one day.

The inspection unit 130 calculates, at a set timing, the proportion of the number of test subjects that were determined to be normal relative to the number of tests performed during a set period. For example, after all tests for a day are completed, the inspection unit 130 calculates the proportion of the number of test subjects that were determined to be normal relative to the total number of test subjects that were tested on that day. Hereinafter, this proportion may be referred to as the proportion of normal determinations. The classification unit 120 notifies the evaluation device 200 of the calculated proportion. The function of the inspection unit 130 is realized by the processor 11, for example, by executing application software in which a trained machine learning model is implemented.

The evaluation device 200 includes an accumulation unit 210 and an evaluation unit 220.

Figure 8 is a diagram showing an example of data stored in the storage unit 210. The storage unit 210 stores data supplied from the inspection unit 130. The data supplied from the inspection unit 130 to the storage unit 210 includes a date and the percentage of data determined to be normal.

As shown in FIG. 7, the evaluation unit 220 evaluates whether the data input to the machine learning model used by the inspection unit 130 is within the range expected when the machine learning model was trained. The evaluation unit 220 evaluates the data input to the machine learning model used by the inspection unit 130 during a specified evaluation period. The evaluation period is, for example, the past month.

First, the evaluation unit 220 reads out from the storage unit 210 the values of the percentage of normal judgments for the period to be evaluated, sorted by date. Therefore, the data read out from the storage unit 210 is time-series data of the percentage of normal judgments.

Next, the evaluation unit 220 detects anomalies in the trend of time-series changes in the proportion of normal judgments based on the time-series data of the proportion of normal judgments.

In this embodiment, the evaluation unit 220 detects anomalies based on the presence or absence of outliers. For example, the evaluation unit 220 uses an existing method for outlier detection to determine whether or not the time series data for the proportion of data determined to be normal over the past month contains outliers.

When the evaluation unit 220 detects that the time series data of the proportion of normal judgments contains an outlier, it determines that the input data of the machine learning model is not within the range assumed when training the machine learning model.

According to the above aspect, the evaluation device 200 can evaluate whether the input data is within the range assumed when training the machine learning model from the trend of time-series change in the ratio of the number of test subjects predicted to be within the normal range to the total number of test subjects tested.

Figure 9 is a flowchart of the inspection process in which the inspection device 100 inspects the inspection object from image data. Each time the inspection device 100 receives audio data from the sensor 50, it executes the following inspection process.

In step S201, the data acquisition unit 110 of the inspection device 100 receives voice data from the sensor 50. The data acquisition unit 110 performs preprocessing on the acquired voice data. The data acquisition unit 110 saves the preprocessed image data in a specified area in the auxiliary storage unit 13 of the inspection device 100. The data acquisition unit 110 transmits to the inspection unit 130 the path to the storage destination of the preprocessed image data.

In step S202, the inspection unit 130 determines whether the test object is normal. Specifically, first, the inspection unit 130 predicts the degree of abnormality for the test object using the voice data supplied from the data acquisition unit 110 as input data. Next, the inspection unit 130 determines whether the test object is normal or abnormal based on the value of the degree of abnormality. The inspection unit 130 outputs an image displaying the determination result of whether the test object is normal or abnormal and the value indicating the degree of abnormality to the display device 61. Furthermore, the inspection unit 130 counts the number of tests performed during a set period and the number of test objects determined to be normal during that period.

In step S203, when it is time to calculate the proportion of test objects determined to be normal (step S203; YES), the inspection unit 130 executes the process of step S204. On the other hand, in step S203, when it is not time to calculate the proportion of test objects determined to be normal (step S203; NO), the inspection unit 130 executes the process of step S201 again.

In step S204, the inspection unit 130 calculates the ratio of the number of test subjects that were determined to be normal to the total number of test subjects that were tested during the set period. In step S205, the inspection unit 130 transmits data to the evaluation device 200 that includes the calculated ratio of test subjects that were determined to be normal and a date. This data is also called prediction result data. The date indicates when the inspection unit 130 made the prediction of whether or not the test subjects were normal. The date is also called time information.

Figure 10 is a flowchart of the evaluation process by the evaluation device 200. The evaluation device 200 executes the following evaluation process each time it receives prediction result data from the inspection device 100.

In step S211, the storage unit 210 receives data from the inspection unit 130, including the percentage of the inspection objects that were determined to be normal and the date. In step S212, the storage unit 210 stores the received data.

In step S213, when it is time to evaluate the input data (step S213; YES), the evaluation unit 220 executes the process of step S214. On the other hand, in step S213, when it is not time to evaluate the input data (step S213; NO), the storage unit 210 executes the process of step S211 again.

In step S214, the evaluation unit 220 detects outliers in the time series data of the proportion of normal judgments. First, the evaluation unit 220 reads out the value of the proportion of normal judgments in the past month from the accumulation unit 210. The evaluation unit 220 determines whether the read time series data contains any outliers.

In step S215, the evaluation unit 220 presents the evaluation result for the input data based on the outlier detection result. When the evaluation unit 220 detects that an outlier is included in the time series data of the normal judgment rate, it outputs to the display device 62 an image indicating that there was input data that was not within the expected range, the normal judgment rate value detected as the outlier, and the examination date. This allows the user to know that data that was not within the range expected when the machine learning model was trained was input to the machine learning model. This completes the flow of the evaluation process.

In the second embodiment, when inspecting for abnormal sounds using audio data, it is possible to evaluate whether the input data is within the range assumed when the machine learning model was trained, based on the tendency of time-series changes in the prediction results predicted by the machine learning model.

In this way, whether or not the input data is within the range expected when the machine learning model was trained is evaluated based on the trend of time-series changes in the prediction results predicted by the machine learning model. This makes it possible to consider not only the prediction accuracy of the machine learning model, but also other conditions, such as whether or not the input data is within the expected range, when determining whether or not the machine learning model has deteriorated. Therefore, even if the machine learning model has not actually deteriorated, it is possible to prevent erroneous judgment that the machine learning model has deteriorated when the prediction accuracy of the machine learning model has decreased due to other conditions.

C. Other Embodiments (C1) In the first embodiment, an example has been described in which the evaluation device 200 evaluates the result of the classification process by the classification unit 120 of the inspection device 100. However, the evaluation device 200 may not only evaluate the result of the classification process by the classification unit 120, but also evaluate the result of the inspection process by the inspection unit 130 in a similar manner.

As described above, the inspection unit 130 performs an appearance inspection of the inspection object using the trained machine learning model. For example, the inspection unit 130 predicts the probability that the inspection object is judged to be a good product and the probability that the inspection object is judged to be a defective product. The inspection unit 130 judges the inspection object to be a good product when the value of the probability that the inspection object is judged to be a good product exceeds a threshold. The inspection unit 130 judges the inspection object to be a defective product when the value of the probability that the inspection object is judged to be a defective product exceeds a threshold. The inspection unit 130 transmits the value of the probability that the inspection object judged to be a good product to the evaluation device 200. Furthermore, the inspection unit 130 transmits the value of the probability that the inspection object judged to be a defective product to the evaluation device 200. The evaluation device 200 may determine whether the input data is within a range expected when learning the machine learning model from the trend of change in the time series data of the value of the probability that the inspection object is judged to be a good product and the trend of change in the time series data of the value of the probability that the inspection object is judged to be a defective product.

According to this aspect, when performing visual inspection using image data, it is possible to evaluate whether the input data is within the range assumed when the machine learning model was trained, based on the trend of time-series changes in the prediction results predicted by the machine learning model.

(C2) In the first embodiment, the prediction result data transmitted by the classification unit 120 to the evaluation device 200 included the date and time when the classification unit 120 made the prediction. In the second embodiment, the prediction result data transmitted by the inspection unit 130 to the evaluation device 200 included the date when the inspection unit 130 made the prediction. In this way, in the first and second embodiments, the party transmitting the prediction result data acquired the time information. However, the party receiving the prediction result data may also acquire the time information.

For example, the classification unit 120 in the inspection system 1000 does not need to include the date and time in the prediction result data sent to the evaluation device 200. In this case, the classification unit 120 sends the prediction result data to the evaluation device 200 immediately after determining which part the inspection target corresponds to. When the storage unit 210 receives the prediction result data from the classification unit 120, it acquires the date and time the data was received. The storage unit 210 stores the data received from the classification unit 120 together with information indicating the date and time.

Since the classification unit 120 transmits the prediction result data to the evaluation device 200 immediately after the judgment, the date and time stored in the storage unit 210 can be regarded as time information indicating when the prediction was made for the test object. Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time series data of the prediction probability.

The inspection unit 130 in the inspection system 1001 does not need to include the date and time in the prediction result data sent to the evaluation device 200. In this case, the inspection unit 130 calculates the ratio of the number of test objects determined to be normal to the number of tests performed during a set period, and then immediately sends the prediction result data to the evaluation device 200. When the storage unit 210 receives the prediction result data from the inspection unit 130, it obtains the date the data was received, and stores the received data together with information indicating the date.

After calculating the ratio, the inspection unit 130 immediately transmits the prediction result data to the evaluation device 200, so the date stored in the storage unit 210 can be regarded as time information indicating when the prediction was made for the inspection target. Therefore, the data read from the storage unit 210 by the evaluation unit 220 is time-series data of the ratio of normal judgments.

Alternatively, neither the party sending the prediction result data nor the party receiving the prediction result data need to acquire time information. The storage unit 210 stores the prediction result data in the order in which it is received. The data read from the storage unit 210 by the evaluation unit 220 can be said to be time-series data of the proportion of normal judgments.

(C3) In the first embodiment, an example was described in which the classification unit 120 determines whether the inspection object is one of two types of parts. The classification unit 120 may also determine whether the inspection object is one of three or more types of parts. In this case, the machine learning model used by the classification unit 120 is a model that has undergone machine learning to predict which of multiple preset classifications the inspection object will be classified into.

(C4) In the first and second embodiments, the inspection systems 1000 and 1001 are described as including the inspection device 100 and the evaluation device 200. However, the functions of the inspection device 100 and the evaluation device 200 may be realized by a single computer.

(C5) In the first embodiment, an example has been described in which the data acquisition unit 110 transmits to the classification unit 120 the path to the storage destination of the image data on which preprocessing has been performed. Alternatively, the data acquisition unit 110 may not perform preprocessing. In this case, the data acquisition unit 110 may store the image data received from the sensor 50 in a specified area in the auxiliary memory unit 13 of the inspection device 100. Alternatively, the data acquisition unit 110 may transmit to the classification unit 120 the image data on which preprocessing has been performed or the image data on which preprocessing has not been performed.

In the second embodiment, an example has been described in which the data acquisition unit 110 transmits to the inspection unit 130 the path to the storage destination of the preprocessed voice data. Alternatively, the data acquisition unit 110 may not perform preprocessing. In this case, the data acquisition unit 110 may store the voice data received from the sensor 50 in a specified area in the auxiliary memory unit 13 of the inspection device 100. Alternatively, the data acquisition unit 110 may transmit to the inspection unit 130 voice data that has been preprocessed or voice data that has not been preprocessed.

A computer-readable recording medium including a magnetic disk, optical disk, magneto-optical disk, flash memory, semiconductor memory, and magnetic tape can be used as a recording medium for recording a program that realizes the functions of the above-mentioned inspection device 100 and evaluation device 200.

In addition, the means for realizing the functions of the inspection device 100 and the evaluation device 200 are not limited to software, and some or all of them may be realized by dedicated hardware. For example, circuits such as FPGAs (Field Programmable Gate Arrays) or ASICs (Application Specific Integrated Circuits) may be used as the dedicated hardware.

The present disclosure is not limited to the above-described embodiments, and can be realized in various configurations without departing from the spirit of the present disclosure. For example, the technical features in the embodiments corresponding to the technical features in each form described in the Summary of the Invention column can be replaced or combined as appropriate to solve some or all of the above-described problems or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

11, 21...processor, 12, 22...main memory unit, 13, 23...auxiliary memory unit, 14, 24...interface, 15, 25...communication unit, 19, 29...internal bus, 50...sensor, 61, 62...display unit, 100...inspection device, 110...data acquisition unit, 120...classification unit, 130...inspection unit, 200...evaluation device, 210...storage unit, 220...evaluation unit, 1000, 1001...inspection system, A1, A2...components

Claims (7)

An inspection system that performs inspection using a machine learning model,
The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected,
An inspection unit that repeatedly executes the steps of: predicting the inspection target using the machine learning model with data about the inspection target as input data; and outputting prediction result data indicating the predicted result;
a storage unit that stores the prediction result data output by the inspection unit;
an evaluation unit that evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model based on a trend of a time series change of the result predicted for the test object indicated by the prediction result data stored in the storage unit, and outputs an evaluation result;
An inspection system comprising: 2. The inspection system of claim 1,
The machine learning model is a result of performing the machine learning for predicting which of a plurality of preset classifications the test object will be classified into,
The inspection unit includes:
predicting a predicted probability indicating the likelihood that the test subject will be classified into each of the plurality of classes;
determining that the test object is classified into the classification having the highest predicted probability value;
outputting the prediction result data including the predicted probability for the classification determined for the test object;
The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection target,
The evaluation unit evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model, based on a trend of a time-series change in the predicted prediction probability for the classification determined for the test subject, which is indicated by the prediction result data stored in the storage unit.
Inspection system. 2. The inspection system of claim 1,
The machine learning model is a model obtained by performing machine learning to predict whether the test subject is within a normal range,
The inspection unit includes:
predicting whether the test subject is within the normal range;
outputting the prediction result data including a value representing a ratio of the number of the test subjects predicted to be within the normal range to the total number of the test subjects tested during a set period;
The storage unit stores the prediction result data output by the inspection unit together with time information indicating when the prediction was made for the inspection target,
The evaluation unit evaluates whether or not the input data is within a range assumed at the time of learning of the machine learning model, based on a trend of a time-series change in the ratio indicated by the prediction result data stored in the storage unit.
Inspection system. 4. The inspection system according to claim 1, further comprising:
The input data is image data representing an appearance of the inspection object acquired by a sensor.
Inspection system. 4. The inspection system according to claim 1, further comprising:
The input data is voice data representing an operating state of the inspection target acquired by a sensor.
Inspection system. 1. A computer-implemented method for performing testing using a machine learning model, comprising:
The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected,
The computer,
A step of predicting the test object using the machine learning model with data about the test object as input data, and outputting prediction result data indicating the predicted result;
accumulating the prediction result data;
A step of evaluating whether or not the input data is within a range assumed at the time of learning of the machine learning model based on a trend of time-series change of the result predicted for the test subject indicated by the accumulated prediction result data, and outputting an evaluation result;
How to do it. A program executed by a computer,
The computer performs the inspection using a machine learning model;
The machine learning model is a trained machine learning model that has undergone machine learning to predict a predetermined event regarding an object to be inspected,
The computer includes:
A function of predicting the test subject using the machine learning model with data about the test subject as input data, and outputting prediction result data indicating the predicted result;
A function of accumulating the prediction result data;
A function of evaluating whether the input data is within a range assumed at the time of learning of the machine learning model based on a trend of time-series changes in the results predicted for the test subject indicated by the accumulated prediction result data, and outputting an evaluation result;
A program to achieve this.

Images