JP7540339B2 - Information processing device, information processing method, and program - Google Patents

Description

The present disclosure relates to an information processing device, an information processing method, and a program.

In recent years, various applications of a technology known as machine learning have been considered. For example, Patent Literature 1 discloses, as an example of a technology using machine learning, a technology that can determine an abnormal signal based on learning of normal signals even if no abnormal signal is given from a device.

In particular, in recent years, a technology known as predictive analysis, which makes it possible to utilize knowledge accumulated from past cases, has been attracting attention and is increasingly being used in business applications. One specific example is the use of predictive analysis to optimize back-end operations, such as demand forecasting, inventory forecasting, and call center incoming call forecasting.

International Publication No. 2013/195164

On the other hand, in some fields, it may take a relatively long time to determine whether a prediction result is correct, and as a result, it may be difficult to evaluate the validity of the prediction result for a relatively long period of time after the prediction is made. In such a situation, it may be an issue how to guarantee the validity of the prediction result during the period until it becomes clear whether the prediction result is correct.

Therefore, this disclosure proposes a technology that enables the validity of the predictive model used for the prediction to be evaluated in a more suitable manner even in situations where it is difficult to evaluate the legitimacy of the prediction results.

According to the present disclosure, there is provided an information processing device comprising: a data acquisition unit that acquires first output data and second output data corresponding to a result of application of a predictive model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples; and a verification processing unit that verifies the validity of the predictive model based on a comparison between the first output data and the second output data.

The present disclosure also provides an information processing method including a computer obtaining first output data and second output data corresponding to a result of applying a predictive model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples, and verifying the validity of the predictive model based on a comparison between the first output data and the second output data.

Furthermore, according to the present disclosure, a program is provided that causes a computer to obtain first output data and second output data corresponding to a result of applying a predictive model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples, and to verify the validity of the predictive model based on a comparison between the first output data and the second output data.

FIG. 1 is an explanatory diagram for describing an example of a schematic configuration of a system according to an embodiment of the present disclosure. FIG. 11 is an explanatory diagram for outlining the basic concept of a technology relating to verification of the validity of a prediction model in the system according to the embodiment. FIG. 2 is a block diagram showing an example of a functional configuration of the system according to the embodiment. FIG. 11 is an explanatory diagram for explaining a process related to verification of a prediction model in the system according to the embodiment. 2 is a functional block diagram showing an example of a hardware configuration of an information processing device constituting the system according to the embodiment. FIG.

A preferred embodiment of the present disclosure will be described in detail below with reference to the accompanying drawings. Note that in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals to avoid redundant description.

The explanation will be given in the following order.
1. Overview 2. Consideration of analysis using predictive models 3. Technical features 3.1. Basic concept 3.2. Functional configuration 3.3. Processing 3.4. Example 4. Hardware configuration 5. Conclusion

<<1. Outline configuration >>
First, an example of a schematic configuration of a system according to an embodiment of the present disclosure will be described with reference to FIG. 1. FIG. 1 is a diagram for explaining an example of a schematic configuration of a system according to the present embodiment. FIG.

As shown in FIG. 1, the system 1 according to the present embodiment includes an information processing device 10 and a terminal device 30. The information processing device 10 and the terminal device 30 are configured to be able to transmit and receive information to each other via a specific network N1. The type of the network N1 is not particularly limited. As a specific example, the network N1 may be configured as a so-called wireless network, such as a network based on the Wi-Fi (registered trademark) standard. As another example, the network N1 may be configured as the Internet, a dedicated line, a LAN (Local Area Network), or a WAN (Wide Area Network). The network N1 may include multiple networks, and at least a part of the network may be configured as a wired network. It is sufficient that communication between the information processing device 10 and the terminal device 30 is logically established, and the information processing device 10 and the terminal device 30 may be physically relayed by another communication device or the like.

The information processing device 10 applies a prediction model constructed based on machine learning to input data, thereby making predictions for a given problem, and outputs the results of the prediction. In the system according to this embodiment, the prediction model is constructed based on so-called supervised learning, and corresponds to logic that classifies, recognizes, discriminates, etc., the input data based on a given algorithm. That is, the above-mentioned prediction model may correspond to a "classifier," "recognizer," "discriminator," etc. in the field of machine learning. Examples of algorithms for realizing the above-mentioned prediction model include linear regression, logistic regression, neural networks, decision trees, SVM (Support Vector Machine), etc. Of course, these are merely examples, and the algorithm for realizing the prediction model is not particularly limited as long as it is possible to make a given prediction (in other words, classification, recognition, discrimination, etc.) based on the results of machine learning.

Based on this configuration, for example, the information processing device 10 may make a predetermined prediction using data transmitted from the terminal device 30 as input data, and return the result of the prediction to the terminal device 30.

The information processing device 10 also has a function of verifying the validity of the prediction model. That is, the information processing device 10 verifies, at a desired opportunity, whether the prediction model is capable of making a correct prediction (i.e., whether a discrepancy may occur between the prediction result and the actual situation). In this case, for example, the information processing device 10 may notify the terminal device 30 of predetermined notification information (e.g., an alert) depending on the result of the verification. The method of the verification will be described in detail later.

The terminal device 30 is configured to input various data to the information processing device 10 and present information notified from the information processing device 10. As a specific example, the terminal device 30 may transmit input data to the information processing device 10 in response to an instruction from a user, and obtain a prediction result related to the input data (i.e., the result of applying the above-mentioned prediction model to the input data) from the information processing device 10. The terminal device 30 may then present information corresponding to the prediction result to the user via a predetermined output unit (e.g., a display, etc.).

Note that the configuration shown in FIG. 1 is merely an example and does not necessarily limit the configuration of the system according to this embodiment. As a specific example, the information processing device 10 and the terminal device 30 may be configured as an integrated device. That is, a configuration equivalent to the above-mentioned system may be realized as a so-called standalone device. Also, at least some of the functions of the information processing device 10 may be realized by multiple devices working together. As a specific example, a function for making predictions based on the above-mentioned prediction model and a function for verifying the prediction model may be realized by different devices. As another example, when a function equivalent to the information processing device 10 is executed, the load may be distributed to multiple devices. Also, the information processing device 10 may be configured to be accessible by multiple terminal devices 30.

Above, with reference to Figure 1, an example of the general configuration of a system relating to one embodiment of the present disclosure has been described.

<<2. Consideration of analysis using predictive models>>
Next, the analysis and prediction using the predictive model will be examined, and then the technical issues of the system according to an embodiment of the present disclosure will be summarized.

In recent years, predictive analysis technology that uses machine learning to utilize knowledge accumulated from past cases has been attracting attention, and is increasingly being used in business applications. As a specific example, predictive analysis technology is used in areas where even if an error occurs in a prediction, there is little loss and the impact on customers is unlikely, such as the optimization of back-end operations such as demand forecasting, inventory forecasting, and call center call prediction. On the other hand, predictive analysis can utilize knowledge accumulated based on past cases, and is therefore expected to be highly effective if it can be used correctly in areas that are likely to have a direct impact on customers, such as hiring prediction, health status prediction, and loan default prediction.

In contrast, in some fields, it may take a relatively long time to determine whether the prediction results were correct or not, and in such cases, it can become an issue how to guarantee the validity of the prediction results during that period.

As a specific example, in fields such as demand forecasting and inventory forecasting, it is possible to obtain a judgment result regarding whether the prediction result was correct or not in a relatively short period of time, such as within a few days. Therefore, in this case, by comparing the prediction result with the judgment result, it is possible to ensure that the prediction model used for the prediction is operating as expected.

On the other hand, in fields such as hiring prediction, health status prediction, and loan default prediction, it tends to take years (e.g., one to three years after the prediction is made) to determine whether the prediction results were correct. In other words, in such cases, it is difficult to evaluate whether the prediction results were correct over a relatively long period of time, such as years after the prediction is made. Therefore, in such cases, it can be an issue how to guarantee the validity of the prediction results during that period.

One example of a method for ensuring that the predictive model used in the prediction is operating as expected during the period until it becomes possible to determine whether the prediction result is correct is to compare the input data during learning with the input data during actual operation.

Specifically, in a situation where the future is predicted based on past data, the greater the divergence between the distribution of the data used during learning and the distribution of the data used during prediction, the more difficult it tends to be to make accurate predictions. By utilizing this characteristic, for example, by statistically comparing the degree of divergence between the distributions of samples contained in the input data used during learning and the input data based on the results of collecting information in an actual operating environment, it is possible to use the results of this comparison as one element in evaluating the validity of the predictive model (i.e., whether the predictive model is operating as expected).

On the other hand, each sample included in the input data often contains various information as features. As a specific example, when assuming a health status prediction, features such as age, sex, height, weight, blood pressure, and blood component values may be set for the sample patient. In this way, in a situation where the number of feature items included in each sample is large, the amount of data and processing required to statistically compare the deviation of the distribution of the samples included in each input data becomes enormous. More specifically, when the number of feature items included in each sample increases, the variation related to the comparison of the deviation of the distribution of the samples also increases further according to the number of combinations of the items. Therefore, in such a situation, it may be practically difficult to realize the above comparison between the input data.

In light of the above situation, this disclosure proposes a technology that makes it possible to evaluate the validity of the predictive model used for the prediction in a more suitable manner, even in situations where it is difficult to evaluate the legitimacy of the prediction result.

<<3. Technical Features>>
Next, technical features of a system according to an embodiment of the present disclosure will be described.

<3.1. Basic philosophy＞
First, among the technical features of the system according to an embodiment of the present disclosure, in particular, an overview of the basic concept of the technology for realizing verification of the validity of a predictive model will be provided.

In the system according to the present embodiment, the input data used during learning and the input data based on the results of collecting information in an actual operating environment are input into a prediction model, and the validity of the prediction model is verified by comparing the similarity of the distribution of the prediction results (classification results) output for each. At this time, the input data may contain various information as features for each sample. Therefore, for example, when comparing input data, the number of dimensions related to the comparison (in other words, the number of variations related to the comparison) increases according to the number of combinations of the items of the features of each sample, and may become so large that it is practically difficult to realize. In contrast, the data output as a prediction result (hereinafter also referred to as "output data") is one-dimensional prediction probability data (prediction value in the case of a regression problem). Therefore, when comparing output data, the number of dimensions related to the comparison is smaller than when comparing input data, and the amount of data and processing related to the comparison is also reduced.

Here, referring to FIG. 2, a procedure for verifying the validity of a predictive model in the system according to this embodiment will be described with a specific example. FIG. 2 is an explanatory diagram for outlining the basic concept of the technology for verifying the validity of a predictive model in the system according to this embodiment. Specifically, FIG. 2 shows an example of a case where the validity of a predictive model that outputs the probability that an applicant will achieve excellent results (i.e., the probability that the applicant is an excellent applicant) is verified, which is constructed based on supervised machine learning using information of each applicant as input data, assuming a case where employees are hired. That is, each of the input data at the time of learning and the input data at the time of operation in the actual operation environment includes information for each applicant as a sample, and the sample may include at least one or more feature amounts of items. In addition, the input data at the time of learning is data generated in advance to perform machine learning. As the input data at the time of learning, for example, data for confirming the performance of a predictive model that is generally used at the time of learning may be applied. In addition, the input data at the time of operation in the actual operation environment is data generated using information for each applicant collected in the actual operation environment as a sample. Moreover, the prediction model shown in FIG. 2 outputs, for each sample included in the input data, the probability that the applicant corresponding to that sample is excellent as a prediction result.

Based on such a premise, as shown in the upper diagram of FIG. 2, input data at the time of learning (for example, data for confirming the performance of the prediction model) is input to the prediction model, and output values for each sample contained in the input data are obtained, and the distribution of the output values is calculated. Here, the output value corresponds to the prediction result by the prediction model, and may be a probability value from 0 to 100% in the case of a classification problem, or may be a predicted value in the case of a regression problem. In addition, the obtained output value and the calculation result of the distribution of the output value are stored (for example, stored in a predetermined storage area). Note that the output value and the distribution of the output value stored at this time correspond to an example of output data corresponding to the input data at the time of learning in the example shown in FIG. 2 (hereinafter, also referred to as "output data at the time of learning"). That is, the input data corresponds to an example of "first input data", and the output data corresponds to an example of "first output data". In addition, the timing at which the predicted value based on the input data at the time of learning is obtained and the distribution of the predicted value is calculated is not particularly limited. As a specific example, the above-mentioned various information may be obtained and stored when a prediction model based on machine learning is constructed. As another example, when verifying the validity of a prediction model, the various pieces of information described above may be acquired and stored.

Next, as shown in the lower diagram of Figure 2, a distribution of output values is calculated based on the output values obtained during prediction using the prediction model in the actual operating environment. The obtained output values and the calculation results of the distribution of the output values are retained. The output values and the distribution of the output values retained at this time correspond to an example of output data corresponding to input data during operation in the actual operating environment in the example shown in Figure 2 (hereinafter also referred to as "output data during actual operation"). In other words, the input data corresponds to an example of "second input data", and the output data corresponds to an example of "second output data".

Then, after a predetermined number of samples (e.g., tens to hundreds of samples) of prediction results in the actual operation environment have been accumulated, a comparison is made between the output data during learning (first output data) and the output data during actual operation (second output data), and the validity of the prediction model is determined based on the comparison result. Specifically, the higher the similarity between the output data during learning and the output data during actual operation, the more valid the prediction model is determined to be (in other words, the more valid the prediction results by the prediction model are).

As a more specific example, the average value or median value of the output value for each sample contained in the output data during learning and the output data during actual operation may be compared. In this case, for example, if the error in the average value or the median value between the output data during learning and the output data during actual operation is equal to or less than a threshold value, the output data may be determined to be similar, and if the error exceeds the threshold value, the output data may be determined to be dissimilar. In addition, the significance of the difference in the output value between the output data during learning and the output data during actual operation may be calculated by a t-test. Of course, these are merely examples, and the method is not particularly limited as long as it is possible to compare the similarity between the output data during learning and the output data during actual operation based on the output value.

As another example, the distribution of output values calculated for each of the output data during learning and the output data during actual operation may be compared. In this case, for example, the range between the minimum and maximum output values may be divided into multiple ranges for each of the output data during learning and the output data during actual operation, and each output value may be plotted in the corresponding range to generate a histogram, which may then be used as the subject of comparison. Note that, as a method for comparing the similarity of histograms (in other words, a method for determining the difference between histograms), for example, a method for determining the difference between the range of the most frequent value and the range of the minimum and maximum values may be used. Of course, these are merely examples, and as long as it is possible to compare the similarity of the distribution of output values between the output data during learning and the output data during actual operation, the method is not particularly limited.

As a result of the above comparison, it is considered that the greater the similarity between the output data during learning and the output data during actual operation (in other words, if it is determined that the output data are similar), the higher the likelihood that the predictive model is operating as expected (i.e., the more valid the predictions made by the predictive model are).

On the other hand, the lower the similarity between the output data during learning and the output data during actual operation (in other words, when it is determined that the output data are not similar), the higher the possibility that the predictive model is not operating as expected (i.e., the prediction by the predictive model is less valid). In this case, for example, an alert can be issued to the analyst (user), which can provide an opportunity for the analyst to review the target predictive model. In other words, upon receiving the alert, the analyst can begin to search for the cause based on the input data during learning, the predictive model, the input data during actual operation, the output data during actual operation, etc. In particular, a comparison of output values that often appear in an actual operation environment with the corresponding input data can be useful in identifying the cause.

There are various possible reasons why a predictive model may not behave as expected, including factors other than the data, such as mistakes when introducing the predictive model into the actual operating environment, or factors due to the nature of the data, such as when there is a discrepancy between the data at the time of training and the data at the time of actual operation.

As described above, the results obtained based on the above-described configuration can be used as one element of the evaluation of the validity of the prediction model (i.e., whether or not the prediction model is operating as expected). Therefore, for example, even when it is difficult to evaluate the validity of a prediction result using a prediction model, it is possible to use the above-described evaluation result of the validity of the prediction model as one element for guaranteeing the validity of the prediction result. As a result, in a situation where it takes a relatively long time to evaluate the validity of a prediction result using a prediction model, it becomes possible to determine whether or not the prediction model is operating as expected before the period has elapsed, and it is also possible to review the prediction model if necessary.

Above, with reference to Figure 2, we have outlined the technical features of the system according to one embodiment of the present disclosure, in particular the basic concept of the technology for verifying the validity of a predictive model.

<3.2. Functional configuration>
Next, an example of a functional configuration of a system according to an embodiment of the present disclosure will be described with reference to Fig. 3. Fig. 3 is a block diagram showing an example of a functional configuration of a system according to the present embodiment, and shows an example of a functional configuration of the system 1 shown in Fig. 1.

As shown in Fig. 3, the system 1 according to this embodiment includes an information processing device 10, a terminal device 30, and a storage unit 111. Note that the information processing device 10 and the terminal device 30 shown in Fig. 3 correspond to the information processing device 10 and the terminal device 30 shown in Fig. 1.

(Terminal device 30)
First, a description will be given of the configuration of the terminal device 30. As shown in Fig. 3, the terminal device 30 includes a communication unit 301 and a control unit 303. The terminal device 30 may also include at least one of an input unit 307 and an output unit 309.

The communication unit 301 is a configuration for the configuration within the terminal device 30 to transmit and receive information with other devices (e.g., information processing device 10) via a specified network N1. The configuration of the communication unit 301 may be changed as appropriate depending on the type of communication method and communication path. As a specific example, when the network N1 is configured with a wireless communication path, the communication unit 301 may include various configurations for realizing wireless communication such as an RF circuit. In addition, in the following explanation, when the configuration within the terminal device 30 transmits and receives information with other devices (e.g., information processing device 10), it is assumed that the information is transmitted and received via the communication unit 301 unless otherwise specified.

The input unit 307 is an input interface for the user to input information to the terminal device 30. The input unit 307 may include, for example, input devices such as a button, a lever, a keyboard, and a touch panel. Note that the configuration of the input unit 307 is not particularly limited as long as the user can input information to the terminal device 30. As a specific example, the input unit 307 may include a sound collection unit for collecting the user's voice, and the user's voice input to the sound collection unit may be acquired as information input from the user.

The output unit 309 is an output interface for the terminal device 30 to present various information to the user. The output unit 309 may include a display device that outputs images such as still images and moving images, such as a so-called display. As long as the terminal device 30 is capable of presenting various information to the user, the configuration of the output unit 309 is not particularly limited. As a specific example, the output unit 309 may include an acoustic device that outputs sound, such as a speaker. The output unit 309 may also include a vibration device that presents the information to be presented to the user by vibrating in a pattern corresponding to the information to be presented, such as a so-called vibrator.

The control unit 303 provides various functions of the terminal device 30. The control unit 303 includes an input/output control unit 305. Note that the control unit 303 may further include other components in addition to the above-mentioned components. In other words, the control unit 303 may also perform operations other than those of the above-mentioned components.

The input/output control unit 305 executes various processes according to the contents of the instructions from the user via the input unit 307. The input/output control unit 305 may also transmit information according to the contents of the instructions from the user to another device (e.g., the information processing device 10) via the network N1. As a more specific example, the input/output control unit 305 may transmit data according to the instructions from the user to the information processing device 10 as input data for the information processing device 10 to make a prediction based on a predetermined prediction model.

The input/output control unit 305 also presents desired information to the user via the output unit 309. For example, the input/output control unit 305 may present to the user information corresponding to data received from another device (e.g., information processing device 10) via the network N1. As a more specific example, the input/output control unit 305 may obtain output data corresponding to a prediction result based on a predetermined prediction model from the information processing device 10, and present to the user information corresponding to the output data (e.g., information corresponding to at least a portion of the prediction result).

(Information processing device 10)
Next, a description will be given of the configuration of the information processing device 10. As shown in FIG.

The communication unit 101 is substantially similar to the communication unit 301 in the terminal device 30. Therefore, detailed description will be omitted. Furthermore, in the following description, when the configuration within the information processing device 10 transmits and receives information to and from another device (e.g., the terminal device 30), it is assumed that the information is transmitted and received via the communication unit 101 unless otherwise specified.

The control unit 103 provides various functions of the information processing device 10. The control unit 103 includes an analysis processing unit 105, a verification processing unit 107, and an output control unit 109. Note that the control unit 103 may further include other components in addition to the above-mentioned components. In other words, the control unit 103 may also perform operations other than those of the above-mentioned components.

The analysis processing unit 105 executes processing related to various analyses. For example, the analysis processing unit 105 performs a prediction for a specific problem by applying a prediction model constructed based on machine learning to desired input data, and outputs output data corresponding to the result of the prediction to a specific output destination. As a specific example, the analysis processing unit 105 may apply a prediction model to input data at the time of learning as described with reference to the upper diagram of FIG. 2, and generate output data corresponding to the input data according to the prediction result (i.e., output data at the time of learning). Also, the analysis processing unit 105 may apply a prediction model to input data at the time of actual operation as described with reference to the lower diagram of FIG. 2, and generate output data corresponding to the input data according to the prediction result (i.e., output data at the time of actual operation). Note that the source of the input data is not particularly limited. As a specific example, the analysis processing unit 105 may acquire input data from the terminal device 30 via the network N1. Also, as another example, the analysis processing unit 105 may read out input data stored in a specific storage area (e.g., the storage unit 111). The generated output data may be temporarily or permanently stored in a predetermined storage area (for example, the storage unit 111).

The verification processing unit 107 verifies the validity of the processing related to various analyses by the analysis processing unit 105. As a specific example, the verification processing unit 107 may verify the validity of the prediction model used by the analysis processing unit 105 for prediction. Specifically, as described with reference to FIG. 2, the verification processing unit 107 may verify the validity of the prediction model used to generate the output data by comparing the output data generated by the analysis processing unit 105 during learning with the output data during actual operation. Note that an example of the verification method is as described above with reference to FIG. 2, so a detailed description will be omitted. In addition, each output data may be acquired from the analysis processing unit 105, or the data stored in a specified memory area may be read out. In addition, the part of the verification processing unit 107 that acquires the output data corresponds to an example of a "data acquisition unit". That is, in this example, the part that acquires the output data from the analysis processing unit 105 or the part that reads the output data stored in a specified memory area may correspond to an example of a "data acquisition unit".

The output control unit 109 controls the output of various information. For example, the output control unit 109 may generate output information according to various prediction results by the analysis processing unit 105, and transmit the output information to the terminal device 30 via the network N1, so that the output information (in other words, the prediction results) is presented to the user via the output unit 309 of the terminal device 30. As a specific example, the output control unit 109 may generate a histogram showing the distribution of output values corresponding to each sample based on the output data as the output information, and transmit the histogram to the terminal device 30. In addition, the output control unit 109 may generate output information according to the verification results by the verification processing unit 107, and transmit the output information to the terminal device 30 via the network N1, so that the output information (in other words, the verification results) is presented to the user via the output unit 309 of the terminal device 30.

The storage unit 111 temporarily or permanently stores programs and data for the information processing device 10 to realize the various functions described above. For example, the storage unit 111 may hold the input data used by the analysis processing unit 105 for various predictions. The storage unit 111 may also hold the output data generated in response to the results of predictions made by the analysis processing unit 105 based on the prediction model.

Note that the above-mentioned configuration is merely an example, and the functional configuration of the system 1 according to this embodiment is not necessarily limited to the example shown in FIG. 3. As a specific example, the part corresponding to the terminal device 30 and the part corresponding to the information processing device 10 may be integrally configured. As another example, some of the components of the information processing device 10 may be provided outside the information processing device 10. As a specific example, the part corresponding to the analysis processing unit 105 may be provided in another device different from the information processing device 10. In this case, among the components of the information processing device 10, the part that acquires the output data generated by the analysis processing unit 105 from the other device corresponds to an example of a "data acquisition unit". As another example, at least some of the functions realized by the information processing device 10 may be realized by multiple devices operating in cooperation with each other. That is, when the function is executed, the processing load may be distributed to multiple devices operating in cooperation with each other.

Above, with reference to Figure 3, an example of the functional configuration of a system relating to one embodiment of the present disclosure has been described.

3.3. Processing
Next, a process flow of the system according to an embodiment of the present disclosure will be described with particular focus on the process related to validation of a prediction model with reference to Fig. 4. Fig. 4 is an explanatory diagram for describing the process related to validation of a prediction model in the system according to the embodiment.

4, the information processing device 10 (analysis processing unit 105) first inputs input data D111 (i.e., input data during actual operation) based on the results of collecting information in the actual operation environment and input data D121 (i.e., input data during learning) used during learning to the target prediction model, and obtains the corresponding prediction results (S101). Then, the information processing device 10 calculates a prediction probability distribution D113 (i.e., distribution of output values for each sample) during operation in the actual operation environment based on the prediction result corresponding to the input data D111 (output value for each sample included in the input data D111). Similarly, the information processing device 10 calculates a prediction probability distribution D123 during learning based on the prediction result corresponding to the input data D121 (output value for each sample included in the input data D121). Note that the prediction probability distribution D123 during learning may be generated in advance.

Next, the information processing device 10 (verification processing unit 107) compares the predicted probability distribution D113 during operation in the actual operating environment with the predicted probability distribution D123 during learning (S103). As a specific example, the information processing device 10 may calculate the average value and variance value of the output values contained in each of the predicted probability distributions D113 and D123, and calculate the statistical distance between the distributions based on the average value and variance value for each of these distributions, thereby statistically comparing the degree of deviation between these distributions (in other words, the similarity between the distributions).

Then, the information processing device 10 (verification processing unit 107) judges the validity of the prediction model M101 based on the comparison result between the predicted probability distribution D113 during operation in the actual operation environment and the predicted probability distribution D123 during learning (S105). Specifically, when the predicted probability distributions D113 and D123 are similar, the information processing device 10 judges that the prediction model M101 is valid, that is, the prediction model M101 is operating as expected. On the other hand, when there is a deviation between the predicted probability distributions D113 and D123, the information processing device 10 judges that the prediction model M101 is not valid, that is, the prediction model M101 may not be operating as expected. In this case, the information processing device 10 (output control unit 109) may issue an alert to the analyst (user) by notifying the analyst of information via a predetermined output unit. This makes it possible to create an opportunity for the analyst to review the prediction model M101.

Above, with reference to Figure 4, the processing flow of the system according to one embodiment of the present disclosure has been explained, focusing in particular on the processing related to verifying the predictive model.

3.4. Example
Next, as an example of a system according to an embodiment of the present disclosure, an example of the operation of the system assuming a specific use case will be described.

(Example 1: Predicting the probability that an applicant will leave within two years at the time of hiring)
First, as Example 1, an example of predicting the probability that an applicant will leave within two years at the time of hiring using the system according to this embodiment will be described. That is, in this Example, an example of predicting how likely an applicant wishing to join a company mid-career will be to leave a company will be described based on past cases of mid-career employees.

In this case, a predictive model is first constructed based on machine learning using past cases of mid-career hires. At this time, the system calculates the distribution of output values (in other words, the distribution of predicted probabilities) based on the output values for each sample (i.e., the output values for each applicant) output in response to the results of applying the predictive model to the validation data (i.e., the input data during learning). The obtained output values and the distribution of the output values (i.e., the output data during learning) are then stored in a specified memory area.

Next, when operating in the actual operating environment, the system applies the prediction model constructed to input data in which information about applicants in the actual operating environment is used as samples, and obtains output values for each sample (i.e., output values for each applicant) according to the prediction results. In addition, based on the output values obtained for each sample, a distribution of the output values (in other words, the distribution of predicted probabilities) is calculated. In this way, output values and the distribution of the output values according to the prediction results for each sample when operating in the actual operating environment (i.e., output data during actual operation) are obtained.

The system then compares the distribution of predicted probabilities during learning with the distribution of predicted probabilities during actual operation, and calculates the degree of discrepancy (in other words, the degree of similarity) between the distributions. In this example, for the probability that an applicant will leave their job within two years, the most frequent predicted probability in the distribution of predicted probabilities during learning is 30-40%, whereas the most frequent predicted probability in the distribution of predicted probabilities during actual operation is 0-10%. In this case, there is a possibility that the input data during operation in the actual operating environment is different from that expected.

In this case, the investigation revealed that factors such as the difficulty of the web test for mid-career employees in the target year, making it difficult to obtain high scores overall, were influencing the results. In this case, for example, the results of the web test used as input data during learning can be modified so that pre-processing that scales each year is applied to the results that were previously input as absolute values, and then re-learned. The prediction model reconstructed based on the re-learning can then be used to calculate the distribution of prediction probabilities during learning and the distribution of prediction probabilities during actual operation, and it can be confirmed that the prediction model has been corrected based on a comparison between these distributions.

Above, as Example 1, we have explained an example of using the system of this embodiment to predict the probability that an applicant will leave their job within two years at the time of hiring.

(Example 2: Prediction of a customer's probability of developing diabetes using the results of monitoring of the biological cycle)
Next, as a second embodiment, an example of predicting the probability that a customer will develop diabetes based on the results of monitoring a life cycle using a smart watch or the like will be described.

In this embodiment, first, as in the first embodiment, a prediction model is constructed based on past cases and machine learning, and the output values for each sample output according to the application result of the prediction model to the validation data and the distribution of the output values (i.e., the output data during learning) are stored in a predetermined storage area. Also, as in the first embodiment, the output values for each sample output in response to input data in which customer information in the actual operation environment is used as a sample, and the distribution of the output values (i.e., the output data during actual operation) are obtained.

The system then compares the distribution of predicted probabilities during learning with the distribution of predicted probabilities during actual operation, and calculates the degree of deviation between the distributions. In this example, the probability that a customer will develop diabetes peaks at 5-10% in the distribution of predicted probabilities during learning, whereas the distribution of predicted probabilities during actual operation peaks at 30-40%. In this case, there is a possibility that the input data during operation in the actual operating environment is different from that expected.

In this case, it is assumed that the investigation revealed that customers in the country that will be the subject of predictions during actual operation have different life cycles and heart rate sensor values from the samples (customers) included in the data used during learning, resulting in a high prediction probability overall. In this case, for example, for customers in the above country that will be the subject of predictions during actual operation, it is advisable to apply a prediction model constructed based on machine learning using only customers in that country as samples. Then, using this prediction model, the distribution of prediction probabilities during learning and the distribution of prediction probabilities during actual operation can be calculated again, and it can be confirmed that the prediction model has been corrected based on a comparison between these distributions.

Above, as Example 2, we have explained an example of predicting the probability that a customer will develop diabetes based on the results of monitoring their life cycle using a smart watch or the like.

(Example 3: Prediction of the upper limit of the amount that can be lent to a customer at the time of a loan contract)
Next, as a third embodiment, an example of predicting the upper limit of the amount of money that can be lent to a customer at the time of a loan contract will be described.

In this embodiment, first, as in the first embodiment, a prediction model is constructed based on past cases and machine learning, and the output values for each sample output according to the application result of the prediction model to the validation data and the distribution of the output values (i.e., the output data during learning) are stored in a predetermined storage area. Also, as in the first embodiment, the output values for each sample output in response to input data in which customer information in the actual operation environment is used as a sample, and the distribution of the output values (i.e., the output data during actual operation) are obtained.

The system then compares the distribution of predicted probabilities during learning with the distribution of predicted probabilities during actual operation, and calculates the degree of deviation between the distributions. In this example, the upper limit of the amount that can be lent to a customer was the same in that the peak was 3 million for both the distribution of predicted probabilities during learning and the distribution of predicted probabilities during actual operation, but in the actual operation environment, the proportion of customers who could lent more than 10 million was 10% higher.

In this case, it was found that overlearning had occurred because loans of more than 10 million yen had been made to all customers in some regions, and that the prediction model had been constructed to output that customers living in those regions could be loaned more than 10 million yen. In this case, it is advisable to perform re-learning after making corrections to prevent overlearning so that pre-processing is applied to aggregate each region by region, such as treating customers in those regions as customers in the region that includes that region. Then, using the prediction model reconstructed based on the re-learning, the distribution of prediction probabilities at the time of learning and the distribution of prediction probabilities during actual operation can be calculated again, and it is possible to confirm that the prediction model has been corrected based on a comparison between these distributions.

Above, as Example 3, we have explained an example of predicting the upper limit of the amount that can be lent to a customer at the time of a loan contract.

<<4. Hardware Configuration>>
Next, an example of a hardware configuration of an information processing device (for example, the information processing device 10 and the terminal device 30 shown in FIG. 3) constituting the system according to the present embodiment will be described with reference to FIG. 5. FIG. 5 is a functional block diagram showing an example of a hardware configuration of an information processing device constituting the system according to an embodiment of the present disclosure.

The information processing device 900 constituting the imaging system according to this embodiment mainly includes a CPU 901, a ROM 903, and a RAM 905. The information processing device 900 further includes a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925.

The CPU 901 functions as an arithmetic processing device and control device, and controls all or part of the operations within the information processing device 900 in accordance with various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927. The ROM 903 stores programs and arithmetic parameters used by the CPU 901. The RAM 905 temporarily stores programs used by the CPU 901 and parameters that change appropriately during program execution. These are interconnected by a host bus 907 consisting of an internal bus such as a CPU bus. Note that the control unit 103 of the information processing device 10 and the control unit 303 of the terminal device 30 described above with reference to FIG. 3 can be realized, for example, by the CPU 901.

The host bus 907 is connected to an external bus 911, such as a PCI (Peripheral Component Interconnect/Interface) bus, via a bridge 909. In addition, an input device 915, an output device 917, a storage device 919, a drive 921, a connection port 923, and a communication device 925 are connected to the external bus 911 via an interface 913.

The input device 915 is an operation means operated by the user, such as a mouse, keyboard, touch panel, button, switch, lever, and pedal. The input device 915 may be, for example, a remote control means (so-called remote control) using infrared rays or other radio waves, or an external connection device 929 such as a mobile phone or PDA corresponding to the operation of the information processing device 900. The input device 915 is composed of, for example, an input control circuit that generates an input signal based on information input by the user using the above-mentioned operation means and outputs it to the CPU 901. The user of the information processing device 900 can input various data to the information processing device 900 or instruct the information processing device 900 to perform processing operations by operating the input device 915. The input unit 307 of the terminal device 30 described above with reference to FIG. 3 can be realized by, for example, the input device 915.

The output device 917 is composed of a device capable of visually or audibly notifying the user of acquired information. Such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, audio output devices such as speakers and headphones, and printer devices. The output device 917 outputs, for example, the results obtained by various processes performed by the information processing device 900. Specifically, the display device displays the results obtained by various processes performed by the information processing device 900 in text or image form. On the other hand, the audio output device converts audio signals consisting of reproduced voice data, acoustic data, etc. into analog signals and outputs them. The output unit 309 of the terminal device 30 described above with reference to FIG. 3 can be realized, for example, by the output device 917.

The storage device 919 is a data storage device configured as an example of a memory unit of the information processing device 900. The storage device 919 is configured, for example, by a magnetic memory device such as a hard disk drive (HDD), a semiconductor memory device, an optical memory device, or a magneto-optical memory device. This storage device 919 stores programs executed by the CPU 901 and various data. The memory unit 111 described above with reference to FIG. 3 can be realized, for example, by the storage device 919.

The drive 921 is a reader/writer for recording media, and is built into the information processing device 900 or attached externally. The drive 921 reads out information recorded on a removable recording medium 927, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 905. The drive 921 can also write information to a removable recording medium 927, such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, that is mounted on the device. The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, or a Blu-ray (registered trademark) medium. The removable recording medium 927 may also be a Compact Flash (registered trademark) (CF: CompactFlash), a flash memory, or an SD memory card (Secure Digital memory card), etc. Furthermore, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) equipped with a non-contact type IC chip, or an electronic device.

The connection port 923 is a port for directly connecting to the information processing device 900. Examples of the connection port 923 include a USB (Universal Serial Bus) port, an IEEE 1394 port, and a SCSI (Small Computer System Interface) port. Other examples of the connection port 923 include an RS-232C port, an optical audio terminal, and an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external connection device 929 to this connection port 923, the information processing device 900 can directly obtain various data from the external connection device 929 or provide various data to the external connection device 929.

The communication device 925 is, for example, a communication interface configured with a communication device for connecting to a communication network (network) 931. The communication device 925 is, for example, a wired or wireless LAN (Local Area Network), a communication card for Bluetooth (registered trademark) or WUSB (Wireless USB), etc. The communication device 925 may also be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communications. This communication device 925 can transmit and receive signals, etc., in accordance with a predetermined protocol such as TCP/IP, for example, between the Internet and other communication devices. The communication network 931 connected to the communication device 925 is configured with a network connected by wire or wirelessly, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, or satellite communication. The communication unit 101 of the information processing device 10 and the communication unit 301 of the terminal device 30 described above with reference to FIG. 3 can be realized by, for example, the communication device 925.

The above shows an example of a hardware configuration capable of realizing the functions of the information processing device 900 constituting the imaging system according to an embodiment of the present disclosure. Each of the above components may be configured using general-purpose parts, or may be configured using hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration to be used as appropriate depending on the technical level at the time of implementing this embodiment. Note that, although not shown in FIG. 13, it naturally has various components corresponding to the information processing device 900 constituting the imaging system according to this embodiment.

It is possible to create a computer program for implementing each function of the information processing device 900 constituting the imaging system according to this embodiment as described above, and to implement it in a personal computer or the like. It is also possible to provide a computer-readable recording medium in which such a computer program is stored. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, etc. The above computer program may also be distributed, for example, via a network, without using a recording medium. The number of computers that execute the computer program is not particularly limited. For example, the computer program may be executed by multiple computers (for example, multiple servers, etc.) in cooperation with each other. A single computer, or multiple computers working together, is also referred to as a "computer system."

<<5. Conclusion>>
As described above, in the system according to an embodiment of the present disclosure, the information processing device 10 includes a data acquisition unit and a verification processing unit. The data processing unit acquires first output data and second output data according to the application result of a prediction model based on supervised machine learning to first input data and second input data each including a plurality of samples. The verification processing unit verifies the validity of the prediction model based on a comparison between the first output data and the second output data. With the above configuration, for example, even when it is difficult to evaluate the validity of a prediction result, it is possible to evaluate the validity of the prediction model. That is, it is possible to use the evaluation result of the validity of such a prediction model as one element for guaranteeing the validity of a prediction result using the prediction model. As a result, in a situation where it takes a relatively long time to evaluate the validity of a prediction result using a prediction model, it is possible to determine whether the prediction model is operating as expected before the period has elapsed, and it is also possible to review the prediction model as necessary.

Although the preferred embodiment of the present disclosure has been described in detail above with reference to the attached drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field of the present disclosure can conceive of various modified or revised examples within the scope of the technical ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

In addition, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology disclosed herein may provide other effects that are apparent to a person skilled in the art from the description herein, in addition to or in place of the above effects.

Note that the following configurations also fall within the technical scope of the present disclosure.
(1)
a data acquisition unit that acquires first output data and second output data according to a result of application of a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
a verification processing unit that verifies the validity of the prediction model based on a comparison between the first output data and the second output data;
An information processing device comprising:
(2)
The data acquisition unit is
obtaining, as the first output data, a classification result of a plurality of samples included in the first input data;
obtaining, as the second output data, a classification result of a plurality of samples included in the second input data;
the verification processing unit verifies the validity of the prediction model based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data.
The information processing device according to (1).
(3)
The information processing device according to (2), wherein the verification processing unit determines that the higher the similarity is, the more valid the prediction based on the prediction model is.
(4)
The information processing device described in (2) or (3), wherein the number of dimensions related to the similarity comparison between the first output data and the second output data is less than the number of dimensions related to the similarity comparison between the first input data and the second input data.
(5)
The information processing device according to any one of (2) to (4), further comprising an output control unit configured to control information corresponding to at least one of a distribution corresponding to the classification result acquired as the first output data and a distribution corresponding to the classification result acquired as the second output data to be presented via an output unit.
(6)
The information processing device described in (5), wherein the output control unit controls so that a histogram corresponding to at least one of a distribution corresponding to the classification result acquired as the first output data and a distribution corresponding to the classification result acquired as the second output data is presented.
(7)
the first input data is data based on a plurality of pre-generated samples;
The second input data is based on a plurality of samples collected in a real operational environment.
The information processing device according to any one of (1) to (6).
(8)
an analysis processing unit that generates the first output data and the second output data by applying the prediction model to the first input data and the second input data;
The data acquisition unit acquires the first output data and the second output data generated by the analysis processing unit.
The information processing device according to any one of (1) to (7).
(9)
The information processing device according to any one of (1) to (7), wherein the data acquisition unit acquires the first output data and the second output data from an external device.
(10)
The computer
obtaining first output data and second output data corresponding to a result of applying a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
verifying the validity of the prediction model based on a comparison between the first output data and the second output data;
An information processing method comprising:
(11)
A classification result of a plurality of samples included in the first input data is obtained as the first output data;
A classification result of a plurality of samples included in the second input data is obtained as the second output data;
The validity of the prediction model is verified based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data.
The information processing method according to (10) above.
(12)
The information processing method according to (11), wherein the higher the similarity, the more valid the prediction based on the prediction model is determined to be.
(13)
the first input data is data based on a plurality of pre-generated samples;
The second input data is based on a plurality of samples collected in a real operational environment.
The information processing method according to any one of (10) to (12).
(14)
On the computer,
obtaining first output data and second output data corresponding to a result of applying a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
verifying the validity of the prediction model based on a comparison between the first output data and the second output data;
A program to execute.
(15)
A classification result of a plurality of samples included in the first input data is obtained as the first output data;
A classification result of a plurality of samples included in the second input data is obtained as the second output data;
The validity of the prediction model is verified based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data.
The program described in (14).
(16)
The program according to (15), wherein the higher the similarity, the more valid the prediction based on the prediction model is determined to be.
(17)
the first input data is data based on a plurality of pre-generated samples;
The second input data is based on a plurality of samples collected in a real operational environment.
The program according to any one of (14) to (16).

REFERENCE SIGNS LIST 1 System 10 Information processing device 101 Communication unit 103 Control unit 105 Analysis processing unit 107 Verification processing unit 109 Output control unit 111 Storage unit 30 Terminal device 301 Communication unit 303 Control unit 305 Input/output control unit 307 Input unit 309 Output unit

Claims (13)

a data acquisition unit that acquires first output data and second output data according to a result of application of a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
a verification processing unit that verifies the validity of the prediction model based on a comparison between the first output data and the second output data;
a pre-processing unit that scales and corrects the first input data;
Equipped with
the first input data is data based on a plurality of pre-generated samples;
the second input data is based on a plurality of samples collected in a real operational environment;
The data acquisition unit is
obtaining, as the first output data, a classification result of a plurality of samples included in the first input data;
obtaining, as the second output data, a classification result of a plurality of samples included in the second input data;
The verification processing unit:
verifying the validity of the prediction model based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data;
When the prediction based on the prediction model is determined to be invalid by the verification, the prediction model is re-learned and constructed from the first input data corrected by the pre-processing unit;
The verification processing unit verifies the validity of the prediction model constructed by the re-learning.
Information processing device. The information processing device according to claim 1, wherein the verification processing unit determines that the higher the similarity, the more valid the prediction based on the prediction model is. The information processing device according to claim 1, wherein the number of dimensions involved in the comparison of the similarity between the first output data and the second output data is less than the number of dimensions involved in the comparison of the similarity between the first input data and the second input data. The information processing device according to claim 1, further comprising an output control unit that controls so that information corresponding to at least one of a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data is presented via an output unit. The information processing device according to claim 4, wherein the output control unit controls so that a histogram corresponding to at least one of a distribution corresponding to the classification result acquired as the first output data and a distribution corresponding to the classification result acquired as the second output data is presented. an analysis processing unit that generates the first output data and the second output data by applying the prediction model to the first input data and the second input data;
The data acquisition unit acquires the first output data and the second output data generated by the analysis processing unit.
The information processing device according to claim 1 . The information processing device according to claim 1, wherein the data acquisition unit acquires the first output data and the second output data from an external device. The computer
obtaining first output data and second output data corresponding to a result of applying a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
verifying the validity of the prediction model based on a comparison between the first output data and the second output data;
correcting the second input data;
Scaling and correcting the first input data;
Including,
the first input data is data based on a plurality of pre-generated samples;
the second input data is based on a plurality of samples collected in a real operational environment;
Obtaining the first output data and the second output data includes:
Obtaining, as the first output data, a classification result of a plurality of samples included in the first input data;
obtaining, as the second output data, a classification result of a plurality of samples included in the second input data;
Including,
The step of verifying includes:
verifying the validity of the prediction model based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data;
When the prediction based on the prediction model is determined to be invalid by the verification, the prediction model is re-learned and constructed from the corrected first input data;
The verifying step includes verifying the validity of the re-trained and constructed predictive model.
Information processing methods. The information processing method according to claim 8 , wherein the prediction based on the prediction model is determined to be more valid as the degree of similarity increases. the first input data is data based on a plurality of pre-generated samples;
The second input data is based on a plurality of samples collected in a real operational environment.
The information processing method according to claim 8 . On the computer,
obtaining first output data and second output data corresponding to a result of applying a prediction model based on supervised machine learning to first input data and second input data, each of which includes a plurality of samples;
verifying the validity of the prediction model based on a comparison between the first output data and the second output data;
Scaling and correcting the first input data;
Run the command,
the first input data is data based on a plurality of pre-generated samples;
the second input data is based on a plurality of samples collected in a real operational environment;
Obtaining the first output data and the second output data includes:
Obtaining, as the first output data, a classification result of a plurality of samples included in the first input data;
obtaining, as the second output data, a classification result of a plurality of samples included in the second input data;
Including,
The step of verifying includes:
verifying the validity of the prediction model based on a similarity between a distribution according to the classification result acquired as the first output data and a distribution according to the classification result acquired as the second output data;
When the prediction based on the prediction model is determined to be invalid by the verification, the prediction model is re-learned and constructed from the corrected first input data;
The verifying step includes verifying the validity of the re-trained and constructed predictive model.
program. The program according to claim 11 , wherein the higher the degree of similarity, the more valid the prediction based on the prediction model is determined to be. the first input data is data based on a plurality of pre-generated samples;
The second input data is based on a plurality of samples collected in a real operational environment.
The program according to claim 11 .

Images