JP7849027B2 - Method and program for determining the reliability of an estimation model, and measurement system. - Google Patents

Description

This invention relates to a method and program for determining the reliability of an estimation model that estimates a target variable according to explanatory variables, as well as a measurement system.

In recent years, research and development in multivariate analysis, machine learning, and deep learning has been rapidly advancing. These methods involve generating estimation models that predict the target variable from explanatory variables. These estimation models are generated based on existing data sets (training data) and can also be used for new data sets.

In multivariate analysis, machine learning, and other related fields, it is crucial to construct reliable estimation models that can be broadly applied to new datasets. The degree to which an estimation model can be applied to new datasets is also referred to as its "robustness." The wider the range of new datasets to which it can be applied, the more robust the estimation model is considered to be.

Numerous algorithms have been proposed to provide highly reliable estimation models. Furthermore, many methods have been proposed for determining the reliability of estimation models. Examples of methods for determining the reliability of estimation models include cross-validation and test set validation.

In determining the reliability of an estimation model, existing data is split into training data (for model construction) and test data (for reliability testing). The estimation model constructed using the former is then applied to the latter, and the error between the observed and estimated values of the target variable is measured. The reliability of the estimation model is determined according to this error. A smaller error indicates a higher reliability of the estimation model, increasing its applicability to new data sets. Furthermore, methods have been devised to enhance the reliability of estimation models by combining the selection of training data used for model construction with these reliability determination methods.

However, there is a problem in that it is not easy to determine whether an estimation model generated based on existing data sets is applicable to a new data set. That is, while the applicability of the generated estimation model to a new data set and the selection of optimal variables are usually performed based on simulations using existing data, such as cross-validation and test set validation, whether the generated estimation model is applicable to a new data set can only be determined by actually applying the new data set to the estimation model and verifying the error between the observed and estimated values of the target variable.

Japanese Patent Publication No. 2017-51162

Campomanes et al. (2014) Origin of the spectral shifts among the early intermediates of the rhodopsin photocycle. Journal of the American chemical society. 136:3842-3851. Trivittayasil et al. (2018) Classification of 1-methylcyclopropene treated apples by fluorescence fingerprint using partial least squares discriminant analysis with stepwise selectivity ratio variable selection method. Chemometrics and Intelligent Laboratory Systems. 175:30-36. Kurihara et al. (2021) Examination of acceleration of computational kernels for causal search processing of large numbers of variables using LiNGAM, Research Report High-Performance Computing, 26:1-8. Oyama et al. (2022) Causal exploration of specific health checkup data in Osaka Prefecture, Information Processing, 63(2) Esbensen and Geladi (2010) Principles of Proper Validation: use and abuse of re-sampling for validation. Chemometrics and Intelligent Laboratory Systems. 168-187. Andersen and Bro (2010) Variable selection in regression - a tutorial. Chemometrics and Intelligent Laboratory Systems. 728-737 A. Hyvarinen and S. M. Smith. Pairwise Likelihood Ratios for Estimation of Non-Gaussian Structural Equation Models. J. of Machine Learning Research 14:111-152, 2013.

This invention provides a method, program, and measurement system for determining the reliability of an estimation model, enabling the determination of the model's reliability without inputting new data sets, thereby improving the model's reliability.

To solve the above problems, the method for determining the reliability of an estimation model according to the present invention is a method for determining the reliability of an estimation model that estimates a dependent variable from explanatory variables, and is characterized by comprising the steps of: applying statistical causal inference to the explanatory variables and the dependent variable to estimate the direction of causality between the explanatory variables and the dependent variable; determining whether the dependent variable is the direct and sole cause of the explanatory variables; and determining, according to the result of the determination, whether the estimation model is applicable to new data other than the data used to construct the estimation model.

Furthermore, the computer program for determining the reliability of the estimation model according to the present invention may be configured to allow a computer to execute the above method. Also, the measurement system according to the present invention is a measurement system comprising a measurement device that measures a target object and acquires measurement data as explanatory variables, and a computer that applies the explanatory variables to an estimation model to output a target variable and calculates various characteristics of the target object, wherein the computer may be equipped with the above-mentioned computer program and configured to execute the above method.

In the step of estimating the direction of causality in the determination method according to the present invention, if it is determined that the dependent variable is unlikely to be the cause of the explanatory variable, a further step of deleting that explanatory variable from the estimation model can be performed. In this case, the step of deleting the explanatory variable can delete all explanatory variables related to the determination that the dependent variable is unlikely to be the cause of the explanatory variable at once. Alternatively, the step of deleting the dependent variable can delete the explanatory variables related to the determination that the dependent variable is unlikely to be the cause of the explanatory variable sequentially, calculate the estimation error of the estimation model after the reduction, and stop the deletion step if the estimation error satisfies predetermined conditions.

Furthermore, this determination method may include the steps of appropriately selecting and combining multiple explanatory variables to generate multiple composite variables, and selecting one composite variable from the multiple composite variables to use as the explanatory variable. The step of generating multiple composite variables may include principal component analysis, factor analysis, t-distributed stochastic neighbor embedding (t-SNE), cluster analysis, non-negative matrix factorization (NMF), multivariate curve resolution (MCR), parallel factor analysis (PARAFAC), partial least squares regression analysis, or ensemble learning. Additionally, the step of estimating the direction of causality can utilize LiNGAM (Linear Non-Gaussian Acyclic Model).

According to the present invention, it is possible to provide a method for determining the reliability of an estimation model that can enhance its applicability to new data sets and improve the reliability of the estimation model.

This is a schematic diagram illustrating an example of the configuration of the measurement system 1 according to the first embodiment. Figure 1 is a block diagram illustrating an example of the configuration of the computer 10 shown in Figure 1. This is a flowchart illustrating the procedure for determining the reliability of the estimation model in the measurement system 1 of the first embodiment. This graph illustrates an example of the determination in step S16 of the flowchart in Figure 3. This graph shows the relationship between measured values and estimated values from the estimation model for dependent variable y1, which has a direct causal relationship, and dependent variable y2, which does not have a direct causal relationship. This graph illustrates another example of the decision in step S16 of the flowchart in Figure 3. This graph shows the relationship between measured values and estimated values from the estimation model for the dependent variable y1, which has a direct causal relationship, and dependent variable y3, which does not have a direct causal relationship. This is a diagram illustrating the configuration of the computer 10 in the second embodiment. This is a flowchart illustrating the operation of the second embodiment. This graph shows the relationship between the measured value and the estimated value from the estimation model for the dependent variable y3, for which no direct causal relationship has been found, before and after the removal of the explanatory variable. This is a configuration diagram illustrating the configuration of the computer 10 in the third embodiment. This is a flowchart illustrating the operation of the third embodiment. This graph shows an example of the measurement results related to the example. This graph shows an example of the measurement results related to the example. This is a schematic diagram illustrating the causal relationships between explanatory variables, dependent variables, and unobserved variables. This is an example of a heatmap resulting from calculating the likelihood between the composite variable of the explanatory variables and the dependent variable using the Pairwise LiNGAM algorithm (Non-Patent Document 7). This is a schematic diagram illustrating a method for determining the direction of causality between the dependent variable and the composite variable. This is a schematic diagram illustrating a method for determining the direction of causality between the dependent variable and the composite variable.

This embodiment will be described below with reference to the attached drawings. In the attached drawings, functionally identical elements may be indicated by the same number. While the attached drawings show embodiments and implementation examples in accordance with the principles of this disclosure, they are for the purpose of understanding this disclosure and are not intended to be used to restrictively interpret this disclosure. The descriptions in this specification are merely typical examples and do not limit the claims or applications of this disclosure in any way.

While this embodiment is described in sufficient detail for those skilled in the art to implement the disclosure, other implementations and forms are possible, and it is important to understand that the configuration and structure can be modified and various elements replaced without departing from the scope and spirit of the technical idea of this disclosure. Therefore, the following description should not be interpreted as limiting the scope to this embodiment.

[First Embodiment]
Referring to the schematic diagram in Figure 1, an example of the configuration of the measurement system 1 according to the first embodiment will be described. This measurement system 1 is a measurement system that measures various characteristics of agricultural products such as fruits (e.g., sugar content, moisture content, color, saturation, etc.), and is equipped with a computer 10 and a spectroscopic measuring device 20, and is connected to a server 40 and a database 50 via a network NW.

To measure various characteristics of agricultural products, the computer 10 generates and utilizes estimation models based on multivariate analysis, machine learning, or deep learning. The following description explains a configuration in which estimation models are generated using spectral data obtained from the spectroscopic measurement device 20 as input. However, this is merely an example; instead of, or in addition to, the spectroscopic measurement device 20, data from various measuring devices, cameras, sensors, etc., may be input to the computer and used for measuring various characteristics.

Computer 10 is configured to receive spectral data as input data (explanatory variables) from the spectroscopic measuring device 20, for example, by measuring agricultural products. It applies this spectral data to an estimation model and outputs various characteristics of the agricultural product, such as characteristics y1, y2, y3, etc. (dependent variables). Characteristics y1, y2, and y3 are, for example, the sugar content, moisture content, color, and saturation of the agricultural product (e.g., fruit).

In generating the estimation model, the computer 10 is supplied with spectral data (xt1, xt2, ...) as training data obtained from the spectroscopic measurement device 20, and generates the estimation model. After the estimation model is generated, if new spectral data (x1, x2, ...) is obtained from the spectroscopic measurement device 20 as a new data set (data set not used to generate the estimation model), this spectral data is input into the estimation model, and various characteristics y1, y2, y3, ... are output.

Computer 10 uses statistical causal inference techniques to determine the reliability of the generated estimation model. Based on the obtained reliability level, the applicability of the estimation model to new data sets x1, x2, ... that were not used in the model's generation is determined. Furthermore, the estimation model is updated as appropriate based on the reliability evaluation results, thereby potentially improving its applicability to the new data sets.

Computer 10 is connected to Server 40 and Database 50 via a network (NW). During the generation and updating of the estimation model, various data is supplied from Server 40 and Database 50 via the network (NW). Computer 10 also transmits data related to the generated estimation model and the training data used for its generation to Server 40 via the network (NW). Furthermore, during the determination of the reliability of the estimation model, Computer 10 receives various data for this determination from Server 40 and Database 50. In addition, Computer 10 can supply data related to the reliability determination results of the estimation model to Server 40 via the network (NW).

Referring to the block diagram in Figure 2, an example of the configuration of the computer 10 will be described. As shown in Figure 2, the computer 10 is configured, for example, with a CPU (Central Processing Unit) 101, ROM 102, RAM 103, flash memory 104, input/output control unit 105, and communication control unit 106.

The CPU 101 is a central control unit responsible for various calculations, including calculations for generating and updating estimation models, and calculations for output data using the estimation models. In addition to the CPU 101, a GPU (Graphics Processing Unit) may be included as a control unit for performing image processing and image recognition.

ROM 102 is a storage device that stores programs for various calculations and various data necessary for executing these programs. RAM 103 is a storage device that temporarily stores the calculation results of the programs. Flash memory 104 is a storage medium for storing programs read from ROM 102 and for storing various update programs provided by server 40.

The input/output control unit 105 controls data input from the server 40 and data output to the server 40. The communication control unit 106 controls the transmission and reception of data exchanged between the server 40 and the system.

Computer 10 virtually implements an estimation model generation/update unit 112, a composite variable generation unit 113, a causal inference unit 114, and a new data set applicability determination unit 115 internally using a computer program stored in ROM 102 or the like. This computer program generates/updates the estimation model within the computer and executes a method for determining its reliability.

The estimation model generation/update unit 112 generates an estimation model according to the training data and updates the estimation model according to the results of calculations performed by the causal inference unit 114. For example, the generation and updating of the estimation model can be performed using linear regression analysis. The composite variable generation unit 113 has the function of generating composite variables Sy1, Sy2, etc. by combining the explanatory variables xt1, xt2, ... from the training data used as input data. As described later, if there are multiple explanatory variables xt1, xt2, ..., the composite variable generation unit 113 appropriately selects and combines these multiple explanatory variables to generate multiple composite variables.

The causal inference unit 114 applies statistical causal inference to the explanatory variables (including composite variables) and the dependent variables y1, y2, y3, ... to estimate the direction of causality between the explanatory variables and the dependent variables y1, y2, y3, ... (which variable causes which change). If there are multiple explanatory variables, these multiple explanatory variables are combined to generate a composite variable in the composite variable generation unit 113. This composite variable is then selected as an explanatory variable and can be used for statistical causal inference in the causal inference unit 114.

The causal inference unit 114 performs statistical causal inference calculations according to the dependent variables y1, y2, y3, ... and explanatory variables used in the estimation model, and estimates whether the dependent variables y1, y2, y3, ... are the direct and sole causes of the explanatory variables.

When explanatory variables are acquired by passive sensors, the explanatory variables x1, x2, x3, ... cannot be the cause of the dependent variable or other unobserved variables (z1, z2, z3, ...). Furthermore, individual explanatory variables cannot cause each other. Therefore, the causal relationships between explanatory variables, dependent variables, and unobserved variables are limited to one of the eight patterns shown in Figure 15.

Here, "direct and sole cause" means that a dependent variable changes an independent variable without the intervention of other variables, and that no other variables change the independent variable. In Figure 15, patterns 1-3 fall into this category. If, in a given estimation model, the dependent variables y1, y2, y3, ... are determined to be the direct and sole cause of the independent variables, then that estimation model is likely to be applicable to new data sets separate from the training data, as long as the relationship between the dependent variables y1, y2, y3, ... and the independent variables remains unchanged.

On the other hand, if, in a particular estimation model, the dependent variables y1, y2, y3, ... are not the direct and sole cause of the explanatory variables, or if it is determined that there are other unobserved variables that cause either the dependent or explanatory variables, then it can be concluded that the estimation model is unlikely to be applicable to a new set of data separate from the training data.

Furthermore, if an estimation model determines that the explanatory variables are due to other causes, it can be concluded that the estimation model is unlikely to be applicable to a new data set separate from the training data. In such cases, the estimation model cannot be considered to accurately reflect the real-world situation, and therefore, even if the estimation model is applied to the new data set, it can be concluded that the dependent variable will not reflect the actual numerical values.

The results of the calculations and estimations performed by the causal inference unit 114 are provided to the estimation model generation/update unit 112 and used to update the generated estimation model. The causal inference unit 114 can be constructed based on known models such as LiNGAM (Linear Non-Gaussian Acyclic Model).

The new data set applicability determination unit 115 determines, based on the results of the calculations performed by the causal inference unit 114, whether the estimation model generated using the training data is applicable to a new data set different from the training data. If the new data set is determined to be applicable, it is input into the estimation model as explanatory variables x1, x2, ..., the target variables y1, y2, y3, ... are calculated and output, and these are then output as measured values by the measurement system 1.

Next, referring to the flowchart in Figure 3, the procedure for determining the reliability of the estimation model in the measurement system 1 of the first embodiment will be explained. Once the estimation model is generated based on the training data xt1, xt2, ..., the reliability of the estimation model is determined. First, in step S11, it is determined whether or not there are multiple explanatory variables xt1, xt2, ... as training data (step S11).

If there are no multiple explanatory variables (i.e., only one explanatory variable exists), in step S15, statistical causal inference is applied to that explanatory variable and the dependent variables y1, y2, y3… in the causal inference unit 114, and the direction of causality between the explanatory variable and the dependent variables y1, y2… is estimated (step S15).

On the other hand, if there are multiple explanatory variables xt1, xt2, etc., multiple composite variables Sy1, Sy2, etc. are generated by appropriately combining these multiple explanatory variables (step S12). The generation of composite functions can be performed using methods such as principal component analysis, factor analysis, t-distributed stochastic neighbor embedding (t-SNE), cluster analysis, non-negative matrix factorization (NMF), multivariate curve resolution (MCR), parallel factor analysis (PARAFAC), partial least squares regression analysis, and ensemble learning.

Then, from among these multiple composite variables Sy1, Sy2, etc., the composite variable Syx that can best contribute to the estimation of the dependent variables y1 and y2 is selected (Step S13). The selection of composite variables can be performed using a PLS regression model as an example. The selected composite variable Syx is then used as an explanatory variable, and statistical causal inference is applied to this explanatory variable and the dependent variables y1 and y2 to estimate the direction of causality (Step S14).

If, as a result of causal inference in step S14 or S15, it is determined that the dependent variables y1, y2, ... are the direct and sole causes of the explanatory variables (Y in step S16), then the estimation model is considered highly likely to be applicable to the new data set (step S17). On the other hand, if it is determined that the dependent variables y1, y2, ... are not the direct and sole causes of the explanatory variables (N in step S16), then the estimation model is considered less likely to be applicable to the new data set (step S18).

An example of the determination in step S16 will be explained with reference to Figure 4. For example, as shown in Figure 4(a), if data is generated in which the dependent variable y1 is the direct and sole cause of a certain explanatory variable (e.g., absorbance around 700 nm), and further partial least square (PLS) regression analysis is performed to generate a composite variable of the explanatory variables, then calculating the likelihood between the composite variable of the explanatory variables and the dependent variable using the Pairwise LiNGAM algorithm (Non-Patent Literature 7) will yield a heatmap like the one shown in Figure 16. If the value in row i and column j on the heatmap is positive, the direction of causality is determined to be from i to j, and if it is negative, it is determined to be from j to i. Therefore, in the case of Figure 16, the direction of causality is from the dependent variable y1 to the composite variable 1, and it can be determined that y1 is the direct and sole cause of the composite variable 1, and the estimation model for the dependent variable y1 can be determined to have a certain level of reliability. On the other hand, as shown in Figure 4(b), if the unobserved variable z is the direct cause of a certain explanatory variable (e.g., absorbance around 520 nm), and the dependent variable y2 changes due to the unobserved variable z, but not as the cause of the explanatory variable, then Pairwise LiNGAM yields the likelihood heatmap shown in Figure 17, and it is determined that the dependent variable y2 is not the direct and sole cause of the composite variable. Therefore, the reliability of the estimation model can be judged as low with respect to the dependent variable y2.

Figure 5 is a graph showing the relationship between measured values and estimated values from the estimation model for the dependent variable y1, for which a direct causal relationship was found, and for the dependent variable y2, for which a direct causal relationship was not found. For the dependent variable y1, for which a direct causal relationship was found, a certain relationship was observed between the measured and estimated values. The root mean square error (RMSEnew) of the dependent variable y1 obtained by inputting a new data set into the estimation model did not show a significant difference compared to the root mean square error (RMSEtrain) of the dependent variable y1 obtained by inputting the training data.

On the other hand, for the dependent variable y2, where no direct causal relationship is found and it is considered unlikely that the dependent variable is the cause of the explanatory variable, the root mean square error (RMSEnew) of the dependent variable y2 obtained by inputting a new data set into the estimation model is significantly larger than the root mean square error (RMSEtrain) of the dependent variable y2 obtained by inputting the training data. This means that the estimation model for the dependent variable y2 cannot be applied to the new data set.

Another example of the determination in step S16 will be explained with reference to Figure 6. For example, as shown in Figure 6(a), this is the case when it is determined that the dependent variable y1 is the direct and sole cause of a certain explanatory variable (in Figure 6(a), the absorbance around 700 nm), which is similar to Figure 4(a).

On the other hand, in Figure 6(b), it is determined that the unobserved variable z is the cause of a certain explanatory variable (absorbance around 520 nm in Figure 4(b)), while the dependent variable y3 changes due to the unobserved variable z, and furthermore, the dependent variable y3 causes a change in a certain explanatory variable. In such cases, the Pairwise LiNGAM obtains the likelihood heatmap shown in Figure 18, and it is determined that the dependent variable y3 is not the direct and sole cause of the explanatory variable. Therefore, the estimation model for the dependent variable y3 is also considered unreliable and unlikely to be applicable to new data sets. Thus, according to this system 1, the reliability of the estimation model can be determined based on the inference of the direct causal relationship between the dependent variables y1, y2, y3… and the explanatory variables.

Figure 7 is a graph showing the relationship between measured values and estimated values from the estimation model for the dependent variable y1, for which a direct causal relationship was found, and for the dependent variable y3, for which no direct causal relationship was found. For the dependent variable y1, for which a direct causal relationship was found, a certain relationship was observed between the measured and estimated values. The root mean square error (RMSEnew) of the dependent variable y1 obtained by inputting a new data set into the estimation model did not show a significant difference compared to the root mean square error (RMSEtrain) of the dependent variable y1 obtained by inputting the training data.

On the other hand, for the dependent variable y3, where no direct causal relationship is found and the explanatory variable is considered to be the cause of the dependent variable, the root mean square error (RMSEnew) of the dependent variable y3 obtained by inputting a new data set into the estimation model is significantly larger than the root mean square error (RMSEtrain) of the dependent variable y3 obtained by inputting the training data. This means that the estimation model for the dependent variable y3 cannot be applied to the new data set.

As explained above, according to the measurement system 1, the method for determining the reliability of the estimation model, and the program of the first embodiment, the reliability of the estimation model generated using training data (the possibility that the estimation model can be applied to new data sets different from the training data) is determined according to the direction of causality based on statistical causal inference. Therefore, it becomes possible to determine the reliability without inputting the new data set into the estimation model for verification.

[Second Embodiment]
Next, the measurement system 1 of the second embodiment will be described with reference to Figures 8 and 9. The overall configuration of the measurement system 1 of this second embodiment may be the same as that of the first embodiment, so redundant explanations will be omitted. Figure 8 is a configuration diagram illustrating the configuration of the computer 10 of the second embodiment. Figure 9 is a flowchart illustrating the operation of the second embodiment. In addition to the configuration of the first embodiment, the computer 10 of the second embodiment is equipped with a variable batch deletion processing unit 116 that batch deletes explanatory variables that are to be deleted. The other configurations are the same as those of the first embodiment (Figure 2), so redundant explanations will be omitted.

The variable batch deletion processing unit 116 has the function of batch deleting predetermined explanatory variables from the estimation model according to the results of the inference performed by the causal inference unit 114. That is, as shown in Figure 9, sequential statistical causal inference is performed on each explanatory variable and multiple dependent variables y1, y2, y3… to estimate the direction of causality (step S21). After obtaining reliability data, explanatory variables determined to be caused by the dependent variable are retained, while explanatory variables determined not to be caused by the dependent variable are deleted from the estimation model (step S22). The explanatory variables remaining after deletion are adopted as the final explanatory variables for constructing the estimation model (step S23).

Figures 10(a) and (b) are graphs showing the relationship between the measured value and the estimated value from the estimation model for the dependent variable y3, for which no direct causal relationship was found, before and after the removal of the explanatory variable.

As shown in Figure 10(a), before removing the explanatory variables, the root mean square error (RMSEnew) of the dependent variable y3 obtained by inputting the new data set into the estimation model is significantly larger than the root mean square error (RMSEtrain) obtained by inputting the training data. This means that the estimation model for the dependent variable y3 before removing the explanatory functions cannot be applied to the new data set.

On the other hand, as shown in Figure 10(b), after removing the explanatory variables, the root mean square error (RMSEnew) of the dependent variable y3 obtained by inputting the new data set into the estimation model does not show a significant difference compared to the root mean square error (RMSEtrain) of the dependent variable y3 obtained by inputting the training data. This means that the reliability of the estimation model improved by removing explanatory variables that were not caused by the dependent variable, and the applicability of the estimation model increased even for new data sets that were judged to have a low probability of being applicable before removal.

As explained above, this second embodiment achieves the same effects as the first embodiment, and furthermore, the reliability of the estimation model can be improved by deleting explanatory variables according to the information on the direction of causality.

[Third Embodiment]
Next, the measurement system 1 of the third embodiment will be described with reference to Figures 11 and 12. The overall configuration of the measurement system 1 of this third embodiment may be the same as that of the first embodiment, so redundant explanations will be omitted. Figure 11 is a configuration diagram illustrating the configuration of the computer 10 of the third embodiment. Figure 12 is a flowchart illustrating the operation of the third embodiment. In addition to the configuration of the first embodiment, the computer 10 of the third embodiment is equipped with a variable sequential deletion processing unit 117 that sequentially deletes explanatory variables that are to be deleted. The other configurations are the same as those of the first embodiment (Figure 2), so redundant explanations will be omitted.

The variable sequential deletion processing unit 117 has the function of sequentially deleting predetermined explanatory variables from the estimated model according to the inference results of the causal inference unit 114. That is, as shown in Figure 12, the variable sequential deletion processing unit 117 first determines the final number (k) of explanatory variables to be reduced (step S31), and then, similar to the embodiment described above, generates an estimated model according to the training data and records its cross-validation error and the explanatory variables remaining at that time (step S32).

Next, similar to the previously described embodiment, statistical causal inference is applied to each explanatory variable and the dependent variable to estimate the direction of causality between the explanatory and dependent variables (step S33), and the explanatory variable with the lowest probability of being caused by the dependent variable is removed from the estimation model (step S34). Steps S32 to S34 are repeated until the number of removed explanatory variables reaches k (step S35). Once the number of removed explanatory variables reaches the target of k, the explanatory variables remaining at the point where the cross-validation error was minimized are adopted as the final explanatory variables for constructing the estimation model. It should be noted that whether or not the number of removed explanatory variables reaches k is just one example of a termination condition, and other conditions can be adopted.

As explained above, this third embodiment achieves the same effects as the first embodiment, and furthermore, the reliability of the estimation model can be improved by sequentially deleting explanatory variables according to the information on the direction of causality.

(Example 1)
Figure 13 shows an example of measurement results when a tablet is used as the measurement target in the measurement system 1 of Figure 1, its near-infrared spectral data is acquired, and the active ingredient content of the tablet is measured by applying an estimation model. Figure 13(a) shows the distribution of the measured value of the active ingredient content and the estimated value by the estimation model when a method of sequentially deleting explanatory variables is adopted, as in the third embodiment. On the other hand, Figure 13(b) shows the distribution of measured and estimated values according to the conventional technology. According to this embodiment, even with a new data set, the variability between the measured and estimated values is small, and it can be seen that the reliability of the estimation model has improved.

(Example 2)
Figure 14 shows an example of measurement results when this embodiment is applied to measurement data from a gas sensor in a gas chamber. Figure 14(a) shows the distribution of measured and estimated values according to the conventional technology. On the other hand, Figure 14(b) shows the distribution of measured ethylene concentration in the gas and estimated values from the estimation model when a method of sequentially deleting explanatory variables is adopted, as in the third embodiment. According to this embodiment, even with a new data set, the variability between measured and estimated values is small, and the reliability of the estimation model is improved.

Furthermore, the present invention is not limited to the embodiments described above, and various modifications are possible. For example, the embodiments described above are detailed explanations provided to clearly illustrate the present invention, and the present invention is not necessarily limited to embodiments comprising all the described configurations. Also, it is possible to replace parts of the configuration of one embodiment with those of another embodiment. It is also possible to add configurations from other embodiments to the configuration of one embodiment. Furthermore, it is possible to delete parts of the configuration of each embodiment, or to add or replace other configurations.

10…Computer, 20…Spectroscopic Measurement Device, NW…Network, 40…Server, 50…Database, 101…CPU, 102…ROM, 103…RAM, 104…Flash Memory, 105…Input/Output Control Unit, 106…Communication Control Unit, 112…Estimation Model Generation/Update Unit, 113…Composite Variable Generation Unit, 114…Causal Inference Unit, 115…New Data Set Applicability Determination Unit, 116…Batch Variable Deletion Processing Unit, 117…Sequential Variable Deletion Processing Unit.

Claims (9)

In a determination method for determining the reliability of an estimation model that estimates a dependent variable from explanatory variables using a computer , the computer performs the following:
The steps include: applying statistical causal inference to the explanatory variable and the dependent variable to estimate the direction of causality between the explanatory variable and the dependent variable;
The steps include determining whether the dependent variable is the direct and sole cause of the independent variable,
The step of determining, in accordance with the result of the above determination, whether or not the estimation model is applicable to new data other than the data used to construct the estimation model ,
In the step of determining whether or not there is a direct and sole cause, the determination of whether or not there is a direct and sole cause is made by using an algorithm that calculates the likelihood of the causal relationship between the dependent variable and the independent variable.
A method for determining the reliability of an estimation model, characterized by the features described above. A method for determining the reliability of an estimation model according to claim 1, further comprising the step of removing an explanatory variable from the estimation model if, in the step of estimating the direction of causality, it is determined that the dependent variable is unlikely to be the cause of the explanatory variable. The method for determining the reliability of an estimation model according to claim 2, wherein the step of deleting the explanatory variables involves deleting all explanatory variables that are deemed unlikely to be the cause of the dependent variable. The method for determining the reliability of an estimation model according to claim 2, wherein the step of deleting the explanatory variables involves successively deleting explanatory variables that are deemed unlikely to be the cause of the dependent variable, and stopping the deletion step when a predetermined condition is met. The steps include: appropriately selecting and combining multiple explanatory variables to generate multiple composite variables;
A method for determining the reliability of an estimation model according to claim 1, comprising the step of selecting one composite variable from among the plurality of composite variables and using it as the explanatory variable. The method for determining the reliability of an estimation model according to claim 5, wherein the step of generating the plurality of composite variables includes principal component analysis, factor analysis, t-distributed stochastic neighbor embedding (t-SNE), cluster analysis, non-negative matrix factorization (NMF), multivariate curve resolution (MCR), parallel factor analysis (PARAFAC), partial least squares regression analysis, or ensemble learning. The step of estimating the direction of causality is to use LiNGAM (Linear Non-Gaussian Acyclic Model) to determine the reliability of the estimation model according to claim 1. In a computer program for determining the reliability of an estimation model that estimates the dependent variable from the explanatory variables,
The steps include: applying statistical causal inference to the explanatory variable and the dependent variable to estimate the direction of causality between the explanatory variable and the dependent variable;
The steps include determining whether the dependent variable is the direct and sole cause of the independent variable,
The system is configured to cause the computer to perform the following steps : determining whether the estimation model is applicable to new data other than the data used to construct the estimation model, in accordance with the result of the above determination;
In the step of determining whether or not there is a direct and sole cause, the determination of whether or not there is a direct and sole cause is made by using an algorithm that calculates the likelihood of the causal relationship between the dependent variable and the independent variable.
A computer program for determining the reliability of an estimation model. A measuring device that measures the object to be measured and acquires measurement data as explanatory variables,
A measurement system comprising a computer that applies the explanatory variables to an estimation model to output a target variable and calculates various characteristics of the object to be measured,
The computer includes a computer program for determining the reliability of the estimation model.
The aforementioned computer program,
The steps include: applying statistical causal inference to the explanatory variable and the dependent variable to estimate the direction of causality between the explanatory variable and the dependent variable;
The steps include determining whether the dependent variable is the direct and sole cause of the independent variable,
The system is configured to cause the computer to perform the following steps : determining whether the estimation model is applicable to new data other than the data used to construct the estimation model, in accordance with the result of the above determination;
In the step of determining whether or not there is a direct and sole cause, the determination of whether or not there is a direct and sole cause is made by using an algorithm that calculates the likelihood of the causal relationship between the dependent variable and the independent variable.
Measurement system.