JP7625550B2 - Accuracy monitoring system, accuracy monitoring method, and accuracy monitoring program - Google Patents

Description

The present invention relates to an accuracy monitoring system, an accuracy monitoring method, and an accuracy monitoring program.

Predictive models that output predicted values for measurement data from control systems and the like are machine-learned from past measurement data. The accuracy of a predictive model (prediction accuracy) indicates the degree of match between the predicted value of the predictive model and the actual measured value that will be measured in the future; the closer the predicted value is to the actual measured value, the higher the accuracy. In general, prediction accuracy is highest immediately after learning, and gradually deteriorates during operation.

Patent Document 1 proposes maintaining prediction accuracy in a prediction model, which is a type of machine learning model, by re-learning the prediction model using operational data when certain conditions are met. In order to maintain prediction accuracy during operation with as little calculation cost as possible, Patent Document 1 calculates the prediction accuracy when re-learning using data acquired during operation (operational data). Re-learning is then performed if the prediction accuracy is expected to improve after re-learning, or if the prediction accuracy during operation is below a threshold.

The condition for re-learning the predictive model is that a threshold is set for the prediction accuracy during operation, and the prediction accuracy exceeds the threshold (when a deterioration in the model accuracy is detected).
However, this condition only captures momentary deterioration in accuracy, and cannot capture long-term trends in accuracy deterioration, and it also reacts to momentary deterioration in accuracy due to noise or outliers. Even if the machine learning model is retrained in response to momentary deterioration in accuracy, the long-term prediction accuracy will not change.

On the other hand, Non-Patent Document 1 describes a method for evaluating the long-term accuracy of a classification model, which is a type of model different from the prediction model of a machine learning model, using concept drift detection technology. A classification model is a model that classifies input data into certain attributes. Concept drift is a change over time in the data input/output relationship of a system that has been learned by a machine learning model. Non-Patent Document 1 detects a data change that is a decrease in the model accuracy of a machine learning model, which is one type of concept drift.

JP 2021-184139 A

Joao Gama et al. “Learning with Drift Detection”, Brazilian Symposium on Artificial Intelligence (SBIA 2004), Pages 286-295

As mentioned above, concept drift can lead to a deterioration in model accuracy, so it is important to properly detect the occurrence of concept drift.

FIG. 17 is an explanatory diagram of the prediction model 95 during learning.
The production system 92 outputs measurement data such as production volume 93 according to input data 91 and an operation volume input by an operator 94 .
The prediction model 95 is the result of learning the relationship between the input data 91, the operation amount input by the operator 94, and the production amount 93.

FIG. 18 is an explanatory diagram of the prediction model 95 during operation.
The prediction model 95 inputs an operation amount to the production system 92 instead of an operator 94, and outputs a production amount prediction value 96 that predicts the production amount 93 of the production system 92 for that operation amount.
Therefore, the accuracy 97 of the prediction model 95 is calculated to be higher as the difference between the production volume 93 and the predicted value 96 of the production volume is smaller.
Here, even if the prediction model 95 itself does not change from the time of learning to the time of operation, the accuracy 97 may deteriorate as a result of changes in the surroundings of the prediction model 95. Examples of changes in the surroundings of the prediction model 95 include deterioration of the equipment of the production system 92, the occurrence of new input data 91 that does not tend to exist during learning, and the like.

In this way, the changes that occur from the time of learning to the time of operation are due to the occurrence of concept drift. In order to deal with the deterioration of the accuracy of the prediction model 95 due to the occurrence of concept drift, a mechanism for maintaining the accuracy of the prediction model 95 during operation is required.
However, Non-Patent Document 1 targets a classification model, which is a type of machine learning model, and cannot be directly applied to the prediction model 95. Therefore, a designer must construct logic for converting the prediction result of the prediction model 95 into a format similar to the output result of the classification model according to the case, which is dependent on the know-how of the designer.

The main objective of this invention is to stably detect concept drift that occurs when a predictive model is used, without relying on the know-how of the designer.

In order to solve the above problems, the accuracy monitoring device of the present invention has the following features.
The present invention relates to a data collection unit that collects learning data acquired from each device of an operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system;
and a drift detection logic constructing unit that constructs a drift detection unit that detects a concept drift between the prediction model and the operational data,
The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit;
the anomaly detection model calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model;
the concept drift detection unit is a processing unit that inputs an anomaly rate of the operational data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operational system during operation;
The drift detection unit is characterized in that when the abnormality rate during operation continues to be higher than the abnormality rate during learning, it outputs a signal that a concept drift has occurred in the operational system during operation.
Other means will be described later.

According to the present invention, it is possible to stably detect concept drift that occurs when a predictive model is operated, without relying on the know-how of the designer.

FIG. 2 is an explanatory diagram of each piece of data used when learning a prediction model according to the present embodiment. FIG. 2 is an explanatory diagram of each piece of data used when operating a prediction model according to this embodiment. 1 is an overall configuration diagram of an accuracy monitoring system according to an embodiment of the present invention. FIG. 2 is a hardware configuration diagram of the prediction model monitoring system according to the present embodiment. 5 is a flowchart showing a process of a data collection unit according to the present embodiment. 13 is a flowchart showing a process of a drift detection logic construction unit according to the present embodiment. 13 is a flowchart showing details of the processing of a learning-time accuracy index calculation unit according to the present embodiment. FIG. 13 is a diagram showing the relationship between absolute error and frequency during learning in this embodiment. FIG. 13 is a diagram showing the relationship between absolute error and frequency when the accuracy of a prediction model deteriorates during operation of this embodiment. 10 is a flowchart showing details of a process of an anomaly detection model learning unit according to the present embodiment. FIG. 2 is an explanatory diagram of a drift index and a drift determination criterion according to the embodiment. 10 is a flowchart showing details of a process of a drift judgment criterion creation unit according to the present embodiment. 10 is a flowchart showing a process of an accuracy monitoring unit according to the present embodiment. 13 is a flowchart showing the process of an operational accuracy index calculation unit according to the present embodiment. 5 is a flowchart showing a process of a drift detection unit according to the present embodiment. FIG. 2 is an explanatory diagram of a system data input/output unit according to the embodiment. FIG. 13 is an explanatory diagram of a prediction model during learning. FIG. 13 is an explanatory diagram of a prediction model in operation.

The embodiment of the present invention will be described in detail with reference to the drawings.

First, an overview of main data used in this embodiment will be described with reference to FIGS.
FIG. 1 is an explanatory diagram of each piece of data used when learning the prediction model 31A.
The data collection unit 3 collects measurement data from each device of the operation system 1 in FIG. 3 during learning of the prediction model 31A as learning data 32A, and also collects the prediction model 31A learned from the learning data 32A.
The accuracy index 23A is an index for measuring the accuracy between the prediction model 31A and the learning data 32A. Any accuracy index 23A may be used as long as it can quantitatively evaluate the difference between the objective variable and the prediction result regardless of the positive or negative sign, such as the absolute error, squared value, L1 norm, or L2 norm shown in FIG.
As an example, the present embodiment assumes an absolute error between a predicted value of the prediction model 31A and an actual value of the training data 32A. The larger the absolute error value, the lower the accuracy index 23A is evaluated to be (the more likely it is to deteriorate).

The anomaly detection model 24A is a model that classifies the learning data 32A into two categories, normal values and abnormal values, in order to treat the output of the prediction model 31A as the output of a classification model (as a classification problem). By implementing the classification logic of the anomaly detection model 24A in this way, it is no longer necessary to use individual classification logic that has been used by designers according to projects and that depends on the designers' know-how.
The anomaly detection model 24A may be any model, such as Isolation Forest, One-Class SVM (Support Vector Machine), or Local Outlier Factor, as long as it can classify the learning data 32A into two categories, normal values and abnormal values. In this embodiment, the anomaly detection model 24A is assumed to be Isolation Forest. In Isolation Forest, the proportion of abnormal values contained in the learning data 32A is specified as an "anomaly rate" as a learning parameter.
The anomaly detection model 24A calculates classification logic for classifying each of the learning data 32A and the operation data 33B into a normal value or an abnormal value according to the frequency of the accuracy index 23 defined between the learning data 32A and the predicted value output by the prediction model 31A. As a result, the anomaly detection model 24A is a model that calculates and outputs an anomaly rate at the time of learning indicating the proportion of abnormal values in the learning data 32A and an anomaly rate at the time of operation indicating the proportion of abnormal values in the operation data 33B ( FIG. 2 ).

The error rate 25A is the probability of an incorrect classification, for example, the anomaly rate output by the anomaly detection model 24A. The error rate 25A is in a data format that can be input to the CD detection unit 28B in Fig. 2 as data of the classification model. In Non-Patent Document 1, the drift index 26A is statistical data indicating a combination of the average value of the error rate 25A and the standard deviation of the error rate 25A, and is in a data format that can be input to the CD detection unit 28B instead of the error rate 25A.
The drift judgment criterion 27A is the numerical value of the drift index 26A during learning, and indicates the prediction accuracy of the prediction model 31A.

FIG. 2 is an explanatory diagram of each piece of data used when the prediction model 31A is operated.
2, the prediction model 31A, the anomaly detection model 24A, and the drift determination criterion 27A, which are data with the suffix "A," use data calculated during learning in Fig. 1. On the other hand, the operation data 33B, the accuracy index 23B, the error rate 25B, the drift index 26B, and the CD detection unit 28B, which are data with the suffix "B," use data prepared during operation.

The data collection unit 3 collects measurement data from each device of the operation system 1 in FIG. 3 during operation of the prediction model 31A as operation data 33B.
The accuracy index 23B is an index for measuring accuracy between the prediction model 31A and the operational data 33B. The error rate 25B is an anomaly rate of the operational data 33B output by the anomaly detection model 24A. The drift index 26B is statistical data of the error rate 25B, similar to the drift index 26A.

The drift detection unit 62 has an anomaly detection model 24A and a CD detection unit 28B, and detects a concept drift between the prediction model 31A and the operation data 33B. That is, the drift detection unit 62 receives an input of the accuracy index 23B during operation, and outputs a drift detection result (output result of the CD detection unit 28B) indicating whether or not a concept drift has been detected during operation. Furthermore, when the prediction accuracy of the prediction model 31A during operation is higher than the drift judgment criterion 27A, the drift detection unit 62 may correct the output indicating that a concept drift has occurred by the CD detection unit 28B to an output indicating that a concept drift has not occurred. Furthermore, the drift detection unit 62 may also output the output result of the anomaly detection model 24A.
The drift detection unit 62 outputs a signal indicating that a concept drift has occurred in the operation system 1 during operation, for example, when a state in which the abnormality rate during operation remains higher than the abnormality rate during learning continues.

The CD detection unit 28B is configured as a means for detecting concept drift accompanied by deterioration in accuracy from the operation data 33B targeting the classification model as in Non-Patent Document 1. The CD detection unit 28B is a processing unit that inputs the anomaly rate of the operation data 33B output by the anomaly detection model 24A as an error rate 25 in which incorrect classification is performed, and outputs whether or not a concept drift has occurred in the operation system 1 during operation.
The CD detection unit 28B outputs a drift detection result indicating whether or not a concept drift has been detected during operation by calculating the error rate 25B or the drift index 26B. The CD detection unit 28B may also appropriately correct the drift detection result according to a comparison result between the drift judgment criterion 27A and the drift index 26B during operation.
In this embodiment, the CD detection unit 28B is assumed to be the Drift Detection Method described in Non-Patent Document 1. Any technique that detects concept drift accompanied by accuracy degradation, such as the Early Drift Detection Method and HDDMs (drift diffusion models), can be adopted for the CD detection unit 28B.

Hereinafter, the reason why the model for inputting data to the CD detection unit 28B can be the anomaly detection model 24A that receives the output of the prediction model 31A, similar to a conventional classification model, will be described.
The CD detection unit 28B receives data such as a binomial distribution indicating two classification results (correct or failed) as output values of a conventional classification model. The CD detection unit 28B detects concept drift based on a drift index calculated from the failure probability (error rate) of the received classification result.

On the other hand, the output of the prediction model 31A is a continuous value that may include a positive or negative sign or a decimal point in the form of a floating point number, etc. And the error rate and drift index that can be obtained with the binomial distribution do not exist in the case of continuous values.
Therefore, the anomaly detection model 24A converts the continuous values output by the prediction model 31A into data indicating a classification result such as a binomial distribution, and outputs the error rate of the converted data as an error rate 25B. Note that the more the prediction accuracy deteriorates, the higher the value of the error rate 25B increases.

The CD detection unit 28B can detect concept drift from the error rate 25B or drift index 26B reflecting the classification result, in the same way as with conventional classification models. Note that the drift index 26B is, for example, a binomial distribution that outputs two results, normal and abnormal. In other words, the anomaly detection model 24A can generate data that the CD detection unit 28B can handle by classifying the output results of the prediction model 31A by the drift index 26B.

3 is a diagram showing the overall configuration of the accuracy monitoring system. The accuracy monitoring system is configured by connecting an operation system 1 that is to be monitored and a prediction model monitoring system 2 that is to monitor the accuracy monitoring system via a network.
The prediction model monitoring system 2 may be realized either in a local environment such as a general PC (accuracy monitoring device) or via a network such as a cloud service. Furthermore, the connection method between the prediction model monitoring system 2 and the operation system 1, and the connection method between the prediction model monitoring system 2 and the interface unit 7 may be established in a local environment or via a network.
The prediction model monitoring system 2 includes a data collection unit 3, a storage unit 4, a drift detection logic construction unit 5, and an accuracy monitoring unit 6.
The drift detection logic constructing unit 5 has a learning-time accuracy index calculating unit 51, an anomaly detection model learning unit 52, and a drift determination criterion creating unit 53. The accuracy monitoring unit 6 has an operation-time accuracy index calculating unit 61 and a drift detecting unit 62.

The operational system 1 is a system that utilizes a prediction model 31 A. The data collection unit 3 acquires the following necessary data from the operational system 1 and stores them in the storage unit 4.
・Prediction model 31A in operation
Learning data 32A used to learn the prediction model 31A
Operation data 33B acquired during operation of the operation system 1

The memory unit 4 stores data required by each component in the prediction model monitoring system 2 and data required for output to the interface unit 7. The drift detection logic construction unit 5 constructs a drift detection unit 62 using the data stored in the memory unit 4.
The accuracy monitoring unit 6 detects whether or not the accuracy of the prediction model 31A during operation is degraded, using the data stored in the storage unit 4. That is, the accuracy monitoring unit 6 detects degradation of the prediction accuracy of the prediction model 31A during operation of the operation system 1, using the drift detection unit 62 constructed by the drift detection logic construction unit 5. Specifically, the accuracy monitoring unit 6 calculates the accuracy index 23 during operation based on the operation data 33B and the prediction result of the prediction model 31A, and inputs the calculated accuracy index 23 to the anomaly detection model 24A, thereby causing the CD detection unit 28B to output whether or not a concept drift has occurred in the operation system 1 during operation.
The interface unit 7 is an input/output means capable of inputting parameters for the drift detection logic construction unit 5 and displaying data from the prediction model monitoring system 2 .

FIG. 4 is a hardware configuration diagram of the prediction model monitoring system 2.
The prediction model monitoring system 2 is configured as a computer 900 having a CPU 901 , a RAM 902 , a ROM 903 , a HDD 904 , a communication I/F 905 , an input/output I/F 906 , and a media I/F 907 .
The communication I/F 905 is connected to an external communication device 915. The input/output I/F 906 is connected to an input/output device 916. The media I/F 907 reads and writes data from a recording medium 917. Furthermore, the CPU 901 controls each processing unit by executing a program (also called an application or its abbreviation, "app") loaded into the RAM 902. This program can be distributed via a communication line, or can be recorded on a recording medium 917 such as a CD-ROM and distributed.

FIG. 5 is a flowchart showing the process of the data collection unit 3.
The data collection unit 3 checks whether the prediction model 31A and the learning data 32A obtained when the prediction model 31A is learned are present in the storage unit 4 (S101). If the data are not present, the data collection unit 3 acquires the prediction model 31A and the learning data 32A from the operation system 1 (S102). Next, the data collection unit 3 always acquires the latest operation data 33B from the operation system 1 (S103). Finally, the data collection unit 3 stores the data acquired in S102 and S103 in the storage unit 4 (S104).

6 is a flowchart showing the processing of the drift detection logic construction unit 5. This flowchart is started by an input from the interface unit 7. The interface unit 7 may cooperate with another system to obtain a processing start signal for the drift detection logic construction unit 5.
First, the training-time accuracy index calculation unit 51 calculates the accuracy index 23A of the prediction model 31A in the training data 32A (S201).
Next, the anomaly detection model learning unit 52 uses the calculated accuracy index 23A to learn the anomaly detection model 24A (S202).
Finally, the drift determination criterion creation unit 53 calculates the drift index 26A during learning using the learning data 32A and the anomaly detection model 24A, and creates the drift determination criterion 27A using the drift index 26A (S203).

FIG. 7 is a flowchart showing details of the process (S201) of the learning-time accuracy index calculation unit 51.
First, the learning-time accuracy index calculation unit 51 acquires the prediction model 31A and the learning data 32A from the storage unit 4 (S301). Next, the learning-time accuracy index calculation unit 51 extracts explanatory variables at the time of learning from the learning data 32A and inputs the explanatory variables to the prediction model 31A to acquire a prediction result of the prediction model 31A at the time of learning (S302).
Thereafter, the learning-time accuracy index calculation unit 51 extracts the learning-time objective variable from the learning data 32A, and calculates the accuracy index 23A using the objective variable and the prediction result of S302 (S303).

The role that the anomaly detection model learning unit 52 plays in constructing the drift detection unit 62 will be described with reference to FIGS. 8 and 9. FIG.
FIG. 8 is a diagram showing the relationship between absolute error and frequency during learning.
The anomaly detection model 24A learns how to classify the learning data 32A into an abnormal value DA5 that occurs less frequently and the other normal values DA1-DA4. During learning in Fig. 8, the anomaly detection model 24A generally outputs a lower anomaly rate than during operation in Fig. 9. The anomaly rate is the ratio of the frequency of the abnormal value DA5 to the frequency of all data (normal values DA1-DA4, abnormal value DA5).
Moreover, the prediction model 31A is trained so as to minimize the absolute error between the predicted value output by the prediction model 31A and the actual measured value of the training data 32A. Therefore, in FIG. 8, the abnormal value DA5 is data with a larger absolute error than the normal values DA1-DA4.

FIG. 9 is a diagram showing the relationship between the absolute error and the frequency when the accuracy of the prediction model deteriorates during operation.
9, the anomaly detection model 24A classifies the operational data 33B into an abnormal value DB5 that occurs at a low frequency and the other normal values DB1 to DB4. In other words, the anomaly detection model 24A uses the same logic for classifying the learning data 32A and the operational data 33B.
However, since the frequency of abnormal value DB5 is higher than the frequency of abnormal value DA5 in Fig. 8, during operation in Fig. 9, anomaly detection model 24A outputs an anomaly rate that is about twice as high as the anomaly rate in Fig. 8. A high anomaly rate indicates that prediction model 31A is incompatible with operation data 33B and that the accuracy of prediction model 31A has deteriorated compared to when it was learned.

In this way, the anomaly detection model 24A that evaluates the prediction model 31A that has been trained to fit the learning data 32A outputs a high anomaly rate for the operational data 33B that has a different data tendency (concept drift) from that of the learning data 32A. By using this anomaly rate as the error rate 25B, it is possible to apply a CD detection unit 28B such as that described in Non-Patent Document 1 to the prediction model 31A.
In order to make the prediction model 31A applicable to the CD detection unit 28B as a classification model, the prediction model monitoring system 2 constructs the anomaly detection model 24A as a preprocessing unit of the CD detection unit 28B. This eliminates the need for customization of the classification logic by the designer, making it possible to detect deterioration in accuracy without relying on the know-how of the designer, and achieving uniform prediction model monitoring performance.

FIG. 10 is a flowchart showing details of the process (S202) of the anomaly detection model learning unit 52.
First, the anomaly detection model learning unit 52 acquires learning parameters of the anomaly detection model 24A to be used from the storage unit 4 (S401).
Next, the anomaly detection model learning unit 52 learns the anomaly detection model 24A using the accuracy index 23A calculated by the learning-time accuracy index calculation unit 51 (S402). After that, the anomaly detection model learning unit 52 stores the anomaly detection model 24A in the storage unit 4 (S403).

The role of the drift judgment criterion creation unit 53 in constructing the drift detection unit 62 will be described below. The CD detection unit 28B detects concept drift accompanied by deterioration in accuracy from the operation data 33B. Therefore, even if the prediction model 31A running in the operation system 1 shows higher accuracy than at the time of learning, if the accuracy is deteriorated in the operation data 33B, it is detected as concept drift. However, generally, when the accuracy at the time of operation is higher than the accuracy at the time of learning, it is often not appropriate to detect concept drift accompanied by deterioration in accuracy.
Therefore, by creating drift judgment criterion 27A using drift judgment criterion creation section 53, it is possible to prevent drift detection section 62 from detecting concept drift when the accuracy is higher than that during learning.

In the CD detection unit 28B as in Non-Patent Document 1, the drift index 26B is calculated by processing the operation data 33B, and the transition of the drift index 26B is statistically processed to detect concept drift. For example, in Non-Patent Document 1, the average value and standard deviation of the error rate 25B are used as the drift index 26B. The prediction model monitoring system 2 calculates the drift index 26A during learning and uses this as the drift judgment criterion 27A in the processing of the drift detection unit 62, so that it is possible not to detect concept drift when the accuracy during operation is higher than that during learning.

FIG. 11 is an explanatory diagram of the drift index 26A and the drift determination criterion 27A.
In FIG. 11, the average value of the drift index 26A is set as the drift judgment criterion 27A, but the maximum value, median value, etc. may also be used.

FIG. 12 is a flowchart showing details of the process (S203) of the drift judgment criterion generating unit 53.
First, the drift determination criterion creation unit 53 calculates an error rate 25A using the calculation result of the accuracy index 23A and the learned anomaly detection model 24A (S501). Next, the drift determination criterion creation unit 53 calculates a drift index 26A that is matched to the CD detection unit 28B used in the drift detection unit 62 using the calculated error rate 25A (S502). Finally, the drift determination criterion creation unit 53 calculates a numerical value (average value, maximum value, etc.) that becomes the drift determination criterion 27A from the drift index 26A, and stores the calculation result in the storage unit 4 (S503).

FIG. 13 is a flowchart showing the process of the accuracy monitoring unit 6.
First, the accuracy monitoring unit 6 calculates the accuracy index 23B of the prediction model 31A during operation (S601), and the drift detection unit 62 detects drift from the calculated accuracy index 23B and the data acquired from the memory unit 4 (S602).
Next, the accuracy monitoring unit 6 stores the following example data processed by the accuracy monitoring unit 6 in the storage unit 4 (S603).
The accuracy index 23B during operation calculated by the accuracy index calculation unit 61 during operation
Concept drift detection result calculated by the drift detection unit 62

Next, the accuracy monitoring unit 6 checks whether or not a signal to end accuracy monitoring has been input (S604). The signal to end accuracy monitoring is input when the operation system 1 is stopped or by the interface unit 7. If a signal to end accuracy monitoring has been input, the accuracy monitoring unit 6 ends its processing.
If there is no signal input for terminating the accuracy monitoring, the accuracy monitoring unit 6 checks whether the operation data 33B stored in the storage unit 4 has been updated (S605). If the operation data 33B has not been updated, the process returns to S604, and if the operation data 33B has been updated, the process returns to S601.

FIG. 14 is a flowchart showing the process (S601) of the operational accuracy index calculation unit 61.
14, the processing of the learning data 32A (S301 to S303) described in the learning-time accuracy index calculation unit 51 shown in FIG. 7 is replaced with processing of the operational data 33B (S701 to S703).
The operation-time accuracy index calculation unit 61 acquires the prediction model 31A and the operation data 33B from the storage unit 4 (S701). The operation-time accuracy index calculation unit 61 inputs the explanatory variables during operation to the prediction model 31A and acquires the prediction result (S702). The operation-time accuracy index calculation unit 61 calculates the accuracy index 23B from the objective variables during operation and the prediction result (S703).

FIG. 15 is a flowchart showing the process of the drift detection unit 62.
First, the drift detection unit 62 acquires information necessary for drift detection from the storage unit 4 (S801). The necessary information in S801 is the anomaly detection model 24A, the operation data 33B, and the drift determination criterion 27A.
Next, the drift detection unit 62 inputs the calculation result of the accuracy index 23B calculated by the operational accuracy index calculation unit 61 to the anomaly detection model 24A, and calculates the error rate 25B (S802).
Next, the drift detection unit 62 uses the error rate 25B to calculate the drift index 26B to be used in the CD detection unit 28B (S803).

Next, the drift detection unit 62 inputs the drift indicator 26B to the CD detection unit 28B (S804). Next, the drift detection unit 62 checks whether or not a concept drift has been detected by the CD detection unit 28B (S805). If it has been detected, the process proceeds to S806, and if it has not been detected, the process proceeds to S807.

If the process proceeds to S806, a comparison is made to see whether the drift index 26B is greater than the drift judgment criterion 27A (S806). If the drift index 26B is smaller than the drift judgment criterion 27AB, the prediction model accuracy 27B during operation is higher than the prediction model accuracy 27A at the time of learning, so the process proceeds to S807 and the detection result of the accuracy deterioration is set to "no detection". This allows the result of "detection" of a false positive detected by the concept drift technology to be correctly corrected to "no detection".
On the other hand, if the drift index 26B is greater than the drift judgment criterion 27A, the accuracy of the prediction model during operation is lower than that during learning, so the process proceeds to S808, and the detection result of the accuracy degradation is set to "detected."
The drift detection unit 62 outputs the detection result of the accuracy degradation from the system data input/output unit 15, and ends the process.

FIG. 16 is an explanatory diagram of the system data input/output unit 15.
The drift detection unit 62 displays the results of calculating the accuracy index 23 during operation via the system data input/output unit 15 on the screen as a time series graph as shown in Figure 16, and also displays information indicating the time when the concept drift occurred (time t3 below) on the time series graph.
The system data input/output unit 15 can output data stored in the storage unit 4 and input parameters of the prediction model monitoring system 2 to the storage unit 4. As shown in the display tab, the data stored in the storage unit 4 includes an accuracy index 23B of the prediction model 31A, learning data 32A, operation data 33B, parameters such as an error rate 25B, a drift index 26B, an error rate 25B, etc. In the configuration example of Fig. 16, the accuracy index 23B at the time of detection of accuracy deterioration is displayed.
16 shows the data as a graph, but the display method may be freely determined, such as a time-series table format. Also, the data of the prediction model monitoring system 2 may be processed and the results may be displayed. In the configuration example of FIG. 16, the start point of accuracy degradation is calculated using the slope of the accuracy index 23 and displayed on the graph. An example of parameter input is the learning parameter (anomaly rate) of the Isolation Forest assumed as the anomaly detection model 24A.

Below, characteristic points on the graph of FIG. 16 will be described.
The time t1 indicates an outlier in the accuracy index. The drift detection unit 62 does not detect such outliers or momentary accuracy degradation such as noise, and therefore does not detect accuracy degradation at the time t1.
Time t2 indicates the time when the accuracy of the prediction model starts to deteriorate. As described above, the method of obtaining time t2 may be a method of using the slope of the accuracy index 23B.
Time t3 indicates that the drift detection unit 62 has detected deterioration in accuracy of the prediction model. The drift detection unit 62 is capable of detecting long-term deterioration in accuracy. When deterioration in accuracy has occurred continuously since time t2, the drift detection unit 62 detects deterioration in accuracy at time t3 and outputs the deterioration to the system data input/output unit 15.

In the prediction model monitoring system 2 of the present embodiment described above, the drift detection unit 62 for the prediction model 31A is constructed by the drift detection logic construction unit 5. The anomaly detection model 24A of the drift detection unit 62 enables application of the CD detection unit 28B for classification models that does not depend on the know-how of a designer.
This makes it possible to detect long-term deterioration in accuracy and efficiently perform re-learning to maintain the accuracy of the predictive model. In addition, by clarifying the conditions for whether or not a predictive model needs to be re-learned, it is possible to improve the explainability and efficiency of maintenance work for predictive models.

The present invention is not limited to the above-described embodiment, and various modifications are possible. For example, it is conceivable to link the predictive model monitoring system 2 of the present invention with other systems. The above-described embodiment has been described in detail to clearly explain the present invention, and is not necessarily limited to having all of the configurations described. In addition, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

Furthermore, the above-mentioned configurations, functions, processing units, etc. may be realized in part or in whole by hardware, for example, by designing them as an integrated circuit. As the hardware, a broad processor device such as an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) may be used.
Furthermore, each of the components of the above-described predictive model monitoring system 2 may be implemented in any hardware as long as the respective hardware can transmit and receive information to each other via a network. Furthermore, the processing executed by a certain processing unit may be realized by a single piece of hardware, or may be realized by distributed processing using multiple pieces of hardware.

1 Operational system 2 Predictive model monitoring system (accuracy monitoring device)
3 Data collection unit 4 Memory unit 5 Drift detection logic construction unit 6 Accuracy monitoring unit 7 Interface unit 15 System data input/output unit 23 Accuracy index 24A Anomaly detection model 25 Error rate 26 Drift index 27A Drift judgment criterion 28B CD detection unit (concept drift detection unit)
31A Prediction model 32A Learning data 33B Operation data 51 Learning time accuracy index calculation unit 52 Anomaly detection model learning unit 53 Drift judgment criterion creation unit 61 Operation time accuracy index calculation unit 62 Drift detection unit

Claims (7)

A data collection unit that collects learning data acquired from each device of an operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system;
and a drift detection logic constructing unit that constructs a drift detection unit that detects a concept drift between the prediction model and the operational data,
The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit;
the anomaly detection model is a model that calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model;
the concept drift detection unit is a processing unit that inputs an anomaly rate of the operation data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operation system during operation;
The accuracy monitoring device is characterized in that the drift detection unit outputs a signal indicating that a concept drift has occurred in the operational system during operation when a state in which the abnormality rate during operation is higher than the abnormality rate during learning continues. The accuracy monitoring device further includes an accuracy monitoring unit that detects deterioration of prediction accuracy of the prediction model during operation of the operation system by using the drift detection unit constructed by the drift detection logic construction unit,
The accuracy monitoring device according to claim 1, characterized in that the accuracy monitoring unit calculates the accuracy index during operation based on the operational data and the prediction results of the prediction model, and inputs the calculated accuracy index into the anomaly detection model, thereby outputting to the concept drift detection unit whether or not concept drift has occurred in the operational system during operation. the drift detection logic construction unit further creates a drift judgment criterion indicating a prediction accuracy of the prediction model during learning, and includes the drift judgment criterion in the drift detection unit;
3. The accuracy monitoring device according to claim 2, wherein the drift detection unit corrects an output indicating that a concept drift has occurred by the concept drift detection unit to an output indicating that a concept drift has not occurred when the prediction accuracy of the prediction model during operation is higher than the drift judgment criterion. 3. The accuracy monitoring device according to claim 2, wherein the drift detection unit displays the results of calculating the accuracy index during operation as a time series graph on a screen, and also displays information indicating the time point when concept drift occurred on the time series graph. An accuracy monitoring system comprising: the accuracy monitoring device according to any one of claims 1 to 4; and each device of the operation system. The accuracy monitoring device includes a data collection unit and a drift detection logic construction unit.
The data collection unit collects learning data acquired from each device of the operation system, a prediction model trained with the learning data, and operation data acquired during operation of the operation system;
The drift detection logic construction unit constructs a drift detection unit that detects concept drift between the predictive model and the operational data;
The drift detection logic construction unit constructs the drift detection unit including an anomaly detection model and a concept drift detection unit;
the anomaly detection model is a model that calculates and outputs an anomaly rate at learning time indicating a proportion of abnormal values in the learning data and an anomaly rate at operation time indicating a proportion of abnormal values in the operational data, in accordance with a classification logic that classifies each of the learning data and the operational data into a normal value or an abnormal value depending on a frequency of an accuracy index defined between the learning data and the operational data output by the prediction model;
the concept drift detection unit is a processing unit that inputs an anomaly rate of the operation data output by the anomaly detection model as an error rate of an incorrect classification, and outputs whether or not a concept drift has occurred in the operation system during operation;
The accuracy monitoring method, characterized in that the drift detection unit outputs a signal indicating that a concept drift has occurred in the operational system during operation when a state in which the anomaly rate during operation is higher than the anomaly rate during learning continues. An accuracy monitoring program for causing a computer to function as an accuracy monitoring device according to any one of claims 1 to 4.

Images