JP7674173B2 - Information processing device, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Traditionally, machine learning involves learning input-output relationships from training data and making inferences (classification, regression, prediction, etc.) for new data. When considering the application of a trained model to a real-world situation, the estimation accuracy of the model becomes an important concern in operation. In other words, rather than simply making an estimation, users need to determine the validity of the inference results (guarantee of model operation).

In addition, when a trained model is applied to a real field, the estimation accuracy of the trained model may be reduced. For example, the estimation accuracy of a trained model may be reduced when the input-output relationship itself of the object to be estimated changes, or when data that is not included in the dataset at the time of training is input to the model.

JP 2017-142654 A

However, conventional technology had the problem that it was not easy for users to judge the validity of the model's estimation results. In other words, in the past, simply applying a trained model to the field did not allow users to judge the validity of the model's estimation results, and users did not know to what extent they could trust the estimated results.

The present invention has been made in consideration of the above, and aims to provide an information processing device, an information processing method, and an information processing program that enable a user to easily determine the validity of the estimation results of a model.

To solve the above-mentioned problems and achieve the object, the information processing device of the present invention includes an acquisition unit that inputs input data into a trained model trained with training data and acquires an output result from the trained model, a calculation unit that calculates the degree of deviation between the distribution of the input data and the distribution of the training data, and displays the output result acquired by the acquisition unit and the degree of deviation calculated by the calculation unit.

The present invention allows users to easily determine the validity of the model's estimation results.

FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device according to an embodiment. FIG. 2 is a diagram illustrating a process of calculating the degree of deviation between the distribution of input data and the distribution of training data while inputting input data into a model. FIG. 3 is a diagram for explaining a process for displaying the output result of a model and the distribution deviation degree in a case where the input data and the learning data are time-series data of a sensor or the like. FIG. 4 is a diagram for explaining a process for displaying the output result of a model and the distribution deviation degree in a case where the input data and the learning data are image data. FIG. 5 is a diagram for explaining a process for displaying the output result of a model and the distribution deviation degree in a case where the input data and the learning data are image data. FIG. 6 is a flowchart showing a display process in the information processing device according to the embodiment. FIG. 7 is a flowchart showing the re-learning process in the information processing device according to the embodiment. FIG. 8 is a diagram illustrating a computer that executes a program.

Below, an embodiment of an information processing device, an information processing method, and an information processing program according to the present application will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments.

In the following embodiments, the process flow of the information processing device, information processing method, and information processing program according to the embodiments will be described in order, and finally, the effects of the embodiments will be described.

[Embodiment]
In the following embodiment, the configuration of an information processing device 10 according to the embodiment and the flow of processing by the information processing device 10 will be described in order, and finally, the effects of the embodiment will be described.

[Information processing device]
First, the configuration of an information processing device 10 will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of the configuration of an information processing device according to an embodiment. As shown in Fig. 1, the information processing device 10 includes a communication processing unit 11, a control unit 12, and a storage unit 13.

The communication processing unit 11 communicates with other devices wirelessly or via wires. The communication processing unit 11 is a communication interface that transmits and receives various information with other devices connected via a network or the like. The communication processing unit 11 is realized by a NIC (Network Interface Card) or the like, and communicates between other devices and the control unit 12 (described later) via electrical communication lines such as a LAN (Local Area Network) or the Internet.

The storage unit 13 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. The storage unit 13 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM). The storage unit 13 stores an operating system (OS) and various programs executed by the information processing device 10. Furthermore, the storage unit 13 stores various information used in the execution of the programs. The storage unit 13 has a learning data storage unit 13a and an input data storage unit 13b.

The learning data storage unit 13a stores the learning data used when learning the trained model. For example, the learning data storage unit 13a stores process data acquired from multiple sensors as the learning data. Furthermore, the learning data storage unit 13a is not limited to storing process data, and may store images, signals, etc. Furthermore, the learning data storage unit 13a may store not only raw data but also processed data as the learning data.

The input data storage unit 13b stores input data collected by the collection unit 12a (described later) and input to the trained model. For example, the input data storage unit 13b stores process data acquired from multiple sensors as input data. The process data is, for example, data acquired by various sensors installed in reactors and various devices in factories and plants. The process data may also be, for example, data such as various signals for operating reactors and various devices. The process data is time-series data acquired by each sensor at a predetermined time interval, for example, every second. The input data storage unit 13b is not limited to storing process data, and may also store images, signals, and the like. The input data storage unit 13b may also store processed data as well as raw data as input data.

The control unit 12 controls the entire information processing device 10. The control unit 12 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 12 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 12 also functions as various processing units by running various programs. The control unit 12 has a collection unit 12a, an acquisition unit 12b, a calculation unit 12c, a display unit 12d, a judgment unit 12e, a relearning unit 12f, and a deletion unit 12g.

The collection unit 12a collects input data. For example, the collection unit 12a collects process data representing time series data at a predetermined time interval from multiple sensors installed in monitored equipment such as a factory or plant, and stores the collected data in the input data storage unit 13b. Furthermore, when a predetermined condition is satisfied, for example, when one hour's worth of process data is stored in the input data storage unit 13b, the collection unit 12a notifies the acquisition unit 12b of an instruction to input the input data into the trained model. The collection unit 12a may also acquire input data manually entered by a user.

The acquisition unit 12b inputs the input data into a trained model trained with the training data, and acquires an output result from the trained model. Note that the input data and trained model may be of any type and are not limited thereto. For example, as an output result from the trained model, a predicted value of process data may be acquired, the degree of anomaly of the target equipment may be acquired, or the recognition result of image data may be acquired.

To explain more specifically, for example, when the input data is process data and the output result is an anomaly degree, the acquisition unit 12b, upon receiving an instruction from the collection unit 12a, reads the process data from the input data storage unit 13b and cuts out the process data of a predetermined width using a sliding window of a predetermined width. The acquisition unit 12b then inputs the cut-out process data of the predetermined width into the trained model and acquires the output result from the trained model. Here, the predetermined width is the width of the sliding window that is set in advance, and a value such as one minute can be used. In other words, the calculation unit 15 moves the sliding window of the predetermined width by a predetermined shift width in the time axis direction to cut out the process data of the predetermined width. The processing by the acquisition unit 12b may be performed offline or online using streaming data.

The calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data. Specifically, the calculation unit 12c reads the learning data from the learning data storage unit 13a and reads the process data from the input data storage unit 13b, and calculates the degree of deviation between the distribution of the input data and the distribution of the learning data. Furthermore, when there are multiple types of input data, the calculation unit 12c may calculate the degree of deviation between the distribution of each input data and the distribution of the same type of learning data. For example, when there is data (input data) from multiple sensors, the calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data for each sensor.

Here, the calculation unit 12c may use any method that can calculate the degree of deviation of the distribution between data. For example, the density ratio may be calculated by using a density ratio estimation method such as uLSIF (Unconstrained Least-Squares Importance Fitting) or KLEP, or by using Generative Adversarial Networks. The deviation may be a measure such as Kullback-Leibler divergence or Pearson distance that can be calculated naturally from density ratio estimation. The deviation may be not only the density ratio but also the Mahalanobis distance that measures the distance between distributions or the Euclidean distance used in the nearest neighbor method. The density ratio is output as a continuous value, but may be a discrete value (such as large, medium, and small) by setting an appropriate threshold value. The calculation unit 12c may compress the input data and learning data in advance using a dimension compression method such as PCA, and calculate the density ratio after compression. The algorithm of the density ratio estimation method may be automatically or manually switched.

Now, with reference to FIG. 2, a process of inputting input data into a model and calculating the degree of deviation between the distribution of input data and the distribution of training data will be described. FIG. 2 is a diagram for explaining a process of inputting input data into a model and calculating the degree of deviation between the distribution of input data and the distribution of training data. The example in FIG. 2 illustrates a case where there are more than 100,000 pieces of process data as training data, and about 300 pieces of process data as input data.

The acquisition unit 12b shifts a sliding window of a predetermined width in the time axis direction by a predetermined shift width to extract process data of a predetermined width, and inputs the extracted process data to the trained model to obtain an output result of the trained model. The calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the trained data.

The display unit 12d displays the output results acquired by the acquisition unit 12b and the deviation calculated by the calculation unit 12c. For example, the display unit 12d displays, by a GUI (Graphical User Interface) application, a graph showing the time series change in the output results of the trained model and a graph showing the time series change in the deviation between the distribution of the input data and the distribution of the trained data, side by side so that the time series are the same.

Here, using FIG. 3, a process for displaying the model output results and distribution deviation when the input data and training data are time-series data will be described. FIG. 3 is a diagram for explaining a process for displaying the model output results and distribution deviation when the input data and training data are time-series data. As illustrated in FIG. 3, the display unit 12d displays a graph showing the time-series change in the output results obtained by inputting new input data into the trained model.

The display unit 12d also displays a graph showing the time series change in the distribution deviation obtained by the distribution deviation estimation process for new input data and learning data. The display unit 12d displays these two graphs vertically arranged so that the time series are the same. This allows the user to determine that the output result for section A, where the distribution deviation between the input data and learning data is large, is low in reliability, for example. Furthermore, when there are multiple types of input data, the display unit 12d may display the deviation for each type of input data. For example, when there is data (input data) from multiple sensors, the display unit 12d can display the deviation for each sensor, allowing the user to understand which data deviates from the learning data.

Note that the display method is not limited to this. In addition, it is not limited to the case where the input data and the learning data are time-series data, and for example, the input data and the learning data may be image data. Figures 4 and 5 are diagrams that explain the process of displaying the model output results and the distribution deviation when the input data and the learning data are image data.

Figures 4 and 5 show an example in which the trained model is trained using image data of the numbers "3", "5", "0", "7", etc. as training data. In the examples of Figures 4 and 5, the trained model outputs an estimation result of the number contained in the input image data. In such a case, as shown in Figure 4, the information processing device 10 inputs image data containing the number "9" as input data into the trained model, thereby obtaining and displaying the output result "9". The information processing device 10 also displays the deviation "0.03" between the distribution of the input data and the distribution of the training data. A user who sees this display can determine that the reliability of the output result "9" is high because the deviation value is low.

On the other hand, as illustrated in FIG. 5, the information processing device 10 inputs image data containing "horse" as input data into the trained model, thereby acquiring and displaying the output result "9". The information processing device 10 also displays the degree of discrepancy between the distribution of the input data and the distribution of the trained data, "0.98". A user who sees this display can determine that the reliability of the output result "9" is low because the degree of discrepancy is high. Also, for example, the display unit 12d of the information processing device 10 may sort and display the multiple input image data in order of the degree of discrepancy, allowing the input images to be compared.

The display unit 12d may also display a message to the effect that the trained model will be retrained if the determination unit 12e, which will be described later, determines that the degree of deviation is equal to or greater than a predetermined threshold. In other words, if the degree of deviation between the distributions of the input data and the training data is large, data that is not included in the training data of the trained model has been input, so the accuracy of the output results of the trained model may be low, and the display unit 12d prompts the trained model to be retrained so that it can adapt to the new input data.

The determination unit 12e determines whether the deviation calculated by the calculation unit 12c is equal to or greater than a predetermined threshold. For example, when the input data and the learning data are time-series data, the determination unit 12e may determine whether the deviation continues to be equal to or greater than the predetermined threshold for a predetermined period of time (e.g., 5 minutes) or more, or whether the deviation is equal to or greater than the predetermined threshold at a certain rate or more during the predetermined period of time (e.g., 50%). In this manner, the determination unit 12e does not determine that the deviation is equal to or greater than the threshold when the deviation only increases momentarily, but determines that the deviation is equal to or greater than the threshold only when the deviation continues for a certain period of time, thereby preventing re-learning, which will be described later, from being performed more frequently than necessary.

The determination unit 12e may change the predetermined threshold value according to the accuracy of the output result acquired by the acquisition unit 12b. For example, when the accuracy of the output result of the trained model can be acquired, the determination unit 12e changes the threshold value to a higher value if the accuracy is equal to or greater than the predetermined threshold value, since it is determined that re-learning or the like is not required urgently, and changes the threshold value to a lower value if the accuracy is less than the predetermined threshold value, since it is determined that re-learning or the like is required. Note that, when multiple deviation degrees are calculated by the calculation unit 12c, the determination unit 12e determines whether each deviation degree is equal to or greater than the predetermined threshold value.

When the determination unit 12e determines that the deviation is equal to or greater than a predetermined threshold, the re-learning unit 12f performs re-learning of the trained model. Any re-learning method may be performed by the re-learning unit 12f. For example, the re-learning unit 12f performs re-learning by using the input data stored in the input data storage unit 13b as training data.

The deletion unit 12g identifies the type of input data for which the determination unit 12e has determined that the deviation is equal to or greater than a predetermined threshold, and deletes the identified type of input data from the input data to be input to the trained model. For example, when the input data is a plurality of sensor data, the deletion unit 12g identifies the sensor data for which the determination unit 12e has determined that the deviation is equal to or greater than a predetermined threshold, and deletes the identified sensor data from the input data to be input to the trained model. In this way, when only some of the multiple input data have a high deviation, the deletion unit 12g can prevent a decrease in prediction accuracy by deleting only the relevant data.

[Processing procedure of information processing device]
Next, an example of a processing procedure by the information processing device 10 according to the embodiment will be described with reference to Fig. 6 and Fig. 7. Fig. 6 is a flowchart showing a display process in the information processing device according to the embodiment. Fig. 7 is a flowchart showing a re-learning process in the information processing device according to the embodiment.

As illustrated in FIG. 6, when the collection unit 12a of the information processing device 10 collects input data (Yes in step S101), the collection unit 12a stores the input data in the input data storage unit 13b (step S102). Then, the acquisition unit 12b inputs the input data into a trained model trained with the training data (step S103) and acquires an output result from the trained model (step S104).

Then, the calculation unit 12c calculates the degree of deviation between the distribution of the input data and the distribution of the learning data (step S105). Specifically, the calculation unit 12c reads the learning data from the learning data storage unit 13a and reads the process data from the input data storage unit 13b, and calculates the degree of deviation between the distribution of the input data and the distribution of the learning data.

Then, the display unit 12d displays the output result acquired by the acquisition unit 12b and the deviation calculated by the calculation unit 12c (step S106). For example, the display unit 12d displays a graph showing the time series change in the output result of the trained model and a graph showing the time series change in the deviation between the distribution of the input data and the distribution of the trained data, side by side so that the time series are the same.

7, when the calculation unit 12c of the information processing device 10 calculates the deviation (step S201: Yes), the determination unit 12e determines whether the calculated deviation is equal to or greater than a predetermined threshold (step S202). For example, the determination unit 12e may determine whether the deviation is equal to or greater than the predetermined threshold continuously for a predetermined period of time (e.g., 5 minutes) or more, or whether the deviation is equal to or greater than the predetermined threshold at a certain percentage or more during the predetermined period of time (e.g., 50%).

As a result, if the determination unit 12e determines that the calculated deviation is equal to or greater than the predetermined threshold (Yes in step S202), the re-learning unit 12f executes re-learning of the trained model (step S203) if the determination unit 12e determines that the deviation is equal to or greater than the predetermined threshold. On the other hand, if the determination unit 12e determines that the calculated deviation is not equal to or greater than the predetermined threshold (No in step S202), the process of this flow ends as is.

[Effects of the embodiment]
In this manner, the information processing device 10 according to the present embodiment inputs input data to a trained model trained with training data, and acquires an output result from the trained model. Next, the information processing device 10 calculates the degree of deviation between the distribution of the input data and the distribution of the training data. Then, the information processing device 10 displays the acquired output result and the calculated degree of deviation. This enables the information processing device 10 to allow the user to easily determine the validity of the estimation result of the model.

In other words, the information processing device 10 can clearly show the validity of the estimation results of the model to the user by displaying the degree of deviation between the distribution of the training data set and the data newly input at the operational site along with the estimation results of the trained model. For example, the user can determine that the greater the degree of deviation between the distributions of the training data and the new data, the closer the input data is to unknown data, and therefore the lower the validity of the estimation results of the model.

[System configuration, etc.]
In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in part by a CPU or GPU and a program analyzed and executed by the CPU or GPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the information including the processing procedures, control procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

[program]
It is also possible to create a program in which the process executed by the information processing device 10 described in the above embodiment is written in a language executable by a computer. For example, it is also possible to create a program in which the process executed by the information processing device 10 in the embodiment is written in a language executable by a computer. In this case, the same effect as in the above embodiment can be obtained by the computer executing the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to realize the same process as in the above embodiment.

Figure 8 is a diagram showing a computer that executes a program. As shown in Figure 8, the computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and these components are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as exemplified in FIG. 8. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090, as exemplified in FIG. 8. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

As shown in FIG. 8, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored, for example, in the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

The various data described in the above embodiment are stored as program data, for example, in memory 1010 or hard disk drive 1090. Then, the CPU 1020 reads the program module 1093 and program data 1094 stored in the memory 1010 or hard disk drive 1090 into the RAM 1012 as necessary, and executes various processing procedures.

The program module 1093 and program data 1094 relating to the program are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 relating to the program may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)), and read by the CPU 1020 via the network interface 1070.

The above embodiments and their variations are included in the technology disclosed in this application, and are also included in the scope of the invention and its equivalents described in the claims.

REFERENCE SIGNS LIST 10 Information processing device 11 Communication processing unit 12 Control unit 12a Collection unit 12b Acquisition unit 12c Calculation unit 12d Display unit 12e Determination unit 12f Re-learning unit 12g Deletion unit 13 Storage unit 13a Learning data storage unit 13b Input data storage unit

Claims (7)

An acquisition unit that inputs input data into a trained model trained with training data and acquires an output result from the trained model;
a calculation unit that calculates a degree of deviation between the distribution of the input data and the distribution of the training data;
a display unit that displays the output result acquired by the acquisition unit and the deviation degree calculated by the calculation unit ,
the calculation unit calculates, when there are multiple types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type;
a determination unit that determines whether each deviation calculated by the calculation unit is equal to or greater than a predetermined threshold;
and a deletion unit that identifies a type of input data for which the determination unit has determined that the deviation is equal to or greater than a predetermined threshold, and deletes the identified type of input data from the input data to be input to the trained model . The method further includes a determination unit that determines whether the deviation calculated by the calculation unit is equal to or greater than a predetermined threshold value,
The information processing device according to claim 1 , wherein the display unit displays a message indicating that the trained model will be retrained when the determination unit determines that the degree of deviation is equal to or greater than the predetermined threshold. a determination unit that determines whether the deviation calculated by the calculation unit is equal to or greater than a predetermined threshold;
The information processing device according to claim 1 , further comprising a re-learning unit that executes re-learning of the trained model when the determination unit determines that the deviation is equal to or greater than the predetermined threshold. The information processing device according to any one of claims 1 to 3, characterized in that the judgment unit judges whether the deviation degree is equal to or greater than the predetermined threshold value continuously for a predetermined period of time or more, or whether the deviation degree is equal to or greater than the predetermined threshold value at a certain rate or more during the predetermined period of time. 4. The information processing apparatus according to claim 1 , wherein the determination unit changes the predetermined threshold value depending on the accuracy of the output result acquired by the acquisition unit. An information processing method executed by an information processing device,
An acquisition step of inputting input data into a trained model trained with training data and acquiring an output result from the trained model;
a calculation step of calculating a deviation between the distribution of the input data and the distribution of the training data;
a display step of displaying the output result acquired by the acquisition step and the deviation degree calculated by the calculation step ,
In the calculation step, when there are a plurality of types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type is calculated,
a determination step of determining whether each deviation calculated by the calculation step is equal to or greater than a predetermined threshold value;
an information processing method further comprising: identifying a type of input data for which the deviation is determined to be equal to or greater than a predetermined threshold by the determination step; and deleting the identified type of input data from the input data to be input to the trained model . An acquisition step of inputting input data into a trained model trained with training data and acquiring an output result from the trained model;
a calculation step of calculating a degree of deviation between the distribution of the input data and the distribution of the training data;
a display step of displaying the output result acquired in the acquisition step and the deviation degree calculated in the calculation step ;
In the calculation step, when there are a plurality of types of input data, a degree of deviation between a distribution of each input data and a distribution of training data of the same type is calculated,
a determination step of determining whether each deviation calculated by the calculation step is equal to or greater than a predetermined threshold;
an information processing program further comprising: identifying a type of input data for which the deviation is determined to be equal to or greater than a predetermined threshold by the determination step; and deleting the identified type of input data from the input data to be input to the trained model .

Images