JP7647774B2 - Generation device, method, and program - Google Patents

Description

The disclosed technology relates to a generation device, a generation method, and a generation program.

Machine learning technology is becoming widespread, which automatically learns patterns of input data and creates learning models that can predict things like video and audio classification and trends in time-series data. Furthermore, learning models can be created in a variety of patterns by setting items related to the behavior of the learning model, such as the base structure, parameters, and input data shape. For this reason, when using machine learning technology, it is necessary to select an appropriate pattern of learning model so that the prediction accuracy meets a specified standard.

As a technology for selecting a learning model, a technology has been proposed in which various learning models that predict the same target variable but have different structures or features that they use are generated, and the generated learning models are listed in descending order of their prediction accuracy (Non-Patent Document 1).

Satoshi Hara, Masakazu Ishibata, Takanori Maehara, "Enumeration of Machine Learning Models", Artificial Intelligence Basic Issues Research Group, 105: P. 1-5, 2018.

In machine learning technology, not only the predictive accuracy of the learning model but also the basis for the prediction can be important. For example, in tasks that support human decision-making, such as diagnosing a disease based on the prediction results of a learning model, the reliability of the prediction result is judged not only by the prediction result of the learning model but also by looking at the basis for that prediction. For this reason, it is necessary to select an appropriate learning model so that not only the prediction accuracy but also the basis for the prediction meets specified standards.

Conventional technology lists learning models based only on the accuracy of prediction results estimated based on predetermined indicators, so when one wants to place importance on the basis of the prediction as well, as described above, there is a problem in that the learning models are not necessarily listed in the most optimal order.

The disclosed technology has been developed in consideration of the above points, and aims to generate an indicator for appropriately selecting a learning model that provides the desired prediction accuracy and basis for prediction.

A first aspect of the present disclosure is a generation device that generates an index for selecting a predetermined learning model from among multiple learning models that have been machine-learned using multiple types of features, and is configured to include an acquisition unit that acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and a generation unit that generates the index from the accuracy and the contribution.

A second aspect of the present disclosure is a method for generating an index for selecting a predetermined learning model from among multiple learning models that have been machine-learned using multiple types of features, in which an acquisition unit acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and a generation unit generates the index from the accuracy and the contribution.

A third aspect of the present disclosure is a generation program for generating an index for selecting a predetermined learning model from among multiple learning models machine-learned using multiple types of features, the program causing a computer to function as an acquisition unit that acquires, for each of the multiple learning models, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature specified by a user to the prediction result, and a generation unit that generates the index from the accuracy and the contribution.

The disclosed technology makes it possible to generate indicators for appropriately selecting a learning model that provides the desired prediction accuracy and basis for prediction.

FIG. 1 is a diagram for explaining an overview of a first embodiment. FIG. 11 is a schematic diagram for explaining an example of a SHAP value. FIG. 2 is a block diagram showing a hardware configuration of the learning model selection device. 1 is a block diagram showing an example of a functional configuration of a learning model selection device according to a first embodiment. FIG. FIG. 11 is a sequence diagram showing the flow of a learning model selection process. A block diagram showing an example of the functional configuration of a learning model selection device according to a second embodiment. FIG. 11 is a diagram showing an example of material data. FIG. 13 is a diagram illustrating an example of a learning pattern set. FIG. 13 is a diagram illustrating an example of a column created as a response variable of learning data. FIG. 13 is a diagram illustrating an example of a sequence column created as an explanatory variable of learning data. 11 is a diagram for explaining calculation of a contribution degree for each feature identifier; FIG. FIG. 13 is a diagram illustrating an example of a contribution weight vector and an overall contribution weight. FIG. 11 is a diagram illustrating an example of a calculation result by an acquisition unit. 11A and 11B are diagrams illustrating an example of an acquisition result by an acquisition unit and a generation result by a generation unit. 13 is a flowchart showing an example of an evaluation dataset creation process. 13 is a flowchart illustrating an example of a learning model evaluation process. FIG. 13 is a diagram for explaining differences in model evaluation functions due to differences in overall contribution weights. FIG. 13 is a diagram for explaining adjustment of overall contribution weights. FIG. 13 is a diagram showing an example of a prediction result when a learning pattern that maximizes the model evaluation function is selected when W2=1.0. FIG. 13 is a diagram showing an example of a prediction result when a learning pattern that maximizes the model evaluation function is selected when W2=1.5.

An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

First Embodiment
Fig. 1 is a diagram for explaining an overview of the first embodiment. As shown in Fig. 1, input data is input to each of a plurality of learning models (learning models A and B in the example of Fig. 1), a prediction process is performed in each learning model, and output data that is a prediction result is output. The prediction accuracy of the learning model is obtained by comparing the output data that is the prediction result with the correct answer of the output data.

Conventionally, the learning model is evaluated using this prediction accuracy. On the other hand, as described above, if it is desired to evaluate the learning model by placing emphasis not only on the prediction accuracy but also on the basis of the prediction, it is possible to use the contribution of the desired feature to obtaining the prediction result by the learning model. The desired feature may be selected based on prior knowledge. For example, when trying to predict the probability of developing lung cancer using a learning model, since it is known that the probability of developing lung cancer is related to the amount of cigarettes consumed, the amount of cigarettes consumed or a value extracted from the amount of cigarettes consumed may be used as the desired feature.

In light of the above, in the first embodiment, in addition to the prediction accuracy, the contribution of the feature is also used to generate an index for selecting a learning model. The contribution of the feature is a value indicating the contribution of each type of feature used in the learning model to the prediction result. In other words, the greater the influence of a certain feature value on the prediction result, the greater the contribution of that feature. Here, the type of feature is information that distinguishes each column present in the learning data used for machine learning of the learning model. In other words, the types of feature are different between different columns. However, in the second embodiment described later, time series data is used as the learning data, and the columns of the learning data also include columns in which the time series of a certain column is shifted. In this case, the types of feature are the same in the original column and the column in which the time series is shifted. In addition, even in the case of a column in which the time series is shifted, if the feature of the original column is the objective variable and the feature of the column in which the time series is shifted is the explanatory variable, the original column and the column in which the time series is shifted are treated as having different types of feature.

An example of the contribution level is the SHAP (SHAPley Additive exPlanations, Reference 1) value.

Reference 1: Scott M. Lundberg and Su-In Lee, "A Unified Approach to Interpreting Model Predictions", Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.

Figure 2 shows an example of a schematic diagram of the SHAP value. In the example of Figure 2, the horizontal axis represents the magnitude of the SHAP value, with the contribution increasing in the positive direction toward the right and the contribution increasing in the negative direction toward the left. In addition, in the example of Figure 2, the SHAP value for each feature data (each point in Figure 2) is shown in histogram format for each type of feature. Note that in the example of Figure 2, even if the type of feature is the same, features whose time series are shifted are shown separately. Also, the darker the color of each point, the greater the value of the feature indicated by that point.

In the first embodiment, an index for selecting a learning model is generated so that a learning model having a high contribution of a desired type of feature among the contributions of each type of feature, such as the above-mentioned SHAP value, is selected. The type of desired feature is specified by the user. Then, using the generated index, a learning model is selected so that the prediction accuracy and the contribution of the specified type of feature satisfy a predetermined standard. Below, a learning model selection device including a function for generating an index for selecting a predetermined learning model from multiple learning models machine-learned using multiple types of feature, as described above, is described.

Figure 3 is a block diagram showing the hardware configuration of the learning model selection device 10 according to the first embodiment. As shown in Figure 3, the learning model selection device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication I/F (Interface) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or the storage 14. In the first embodiment, the ROM 12 or the storage 14 stores a learning model selection program for executing the learning model selection process described below.

ROM 12 stores various programs and various data. RAM 13 temporarily stores programs or data as a working area. Storage 14 is composed of storage devices such as HDD (Hard Disk Drive) and SSD (Solid State Drive), and stores various programs and various data including the operating system.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may be a touch panel type and function as the input unit 15.

The communication I/F 17 is an interface for communicating with other devices external to the learning model selection device 10, and uses, for example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark).

Next, the functional configuration of the learning model selection device 10 will be described. FIG. 4 is a block diagram showing an example of the functional configuration of the learning model selection device 10. As shown in FIG. 4, the learning model selection device 10 includes, as functional configurations, a material data collection unit 21, a material data storage unit 22, a learning pattern transmission unit 23, an evaluation data set creation unit 24, and an evaluation data set storage unit 27. The evaluation data set creation unit 24 further includes a verification data creation unit 25 and a learning model creation unit 26. The learning model selection device 10 also includes a learning model evaluation unit 28 and a selected learning model storage unit 32. The learning model evaluation unit 28 further includes an acquisition unit 29, a generation unit 30, and a selection unit 31. The acquisition unit 29 and the generation unit 30 are examples of a generation device of the disclosed technology. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

The material data collection unit 21 collects material data that is used to construct a learning dataset to be used to create a learning model. Material data is data that can be used as a target variable and an explanatory variable, and has a unique data identifier for distinguishing each piece of data. The material data collection unit 21 accepts input of a data identifier specified by a user, and collects material data that has the accepted data identifier. The material data collection unit 21 may collect, for example, sensor values output from a sensor, data stored in an external or internal storage device, etc. as material data. The material data collection unit 21 stores the collected material data in the material data storage unit 22.

The learning pattern transmission unit 23 accepts input of a learning pattern set specified by the user and transmits the accepted learning pattern set to the evaluation dataset creation unit 24.

The behavior of output data from a learning model changes depending on differences in the base structure, parameters, input data shape, etc. Here, a "learning pattern set" refers to a collection of multiple learning patterns, which are settings of information that affect the behavior of the output data of a learning model, that are created in advance by a user. Each learning pattern is assigned an index for identifying it. Furthermore, the components of each learning pattern include, for example, a base model identifier, hyperparameters, and a learning data construction method.

The base model identifier is an identifier that identifies the model structure that is the base of the learning model, and is, for example, the API (Application Programming Interface) name of the learning model. The hyperparameter is a parameter related to the machine learning method that corresponds to the base model identifier. The value of each hyperparameter may be uniquely determined, or there may be multiple candidates. The learning data construction method is a method of processing material data and constructing learning data including explanatory variables and objective variables to be input to the learning model. As the learning data construction method, multiple items are set, such as feature identifiers that are identifiers indicating the type of feature for each of the multiple feature amounts that make up the learning data, and a calculation method for the material data to obtain the feature amount for each feature identifier. As described above, when time series data is used as the learning data and a column of the learning data includes a column in which the time series of a certain column is shifted, the type of feature amount in the original column is the same as that in the column in which the time series is shifted. Therefore, the same feature identifier is used for the feature amount of the original column and the feature amount of the column in which the time series is shifted. However, when the feature quantities of the original column are the objective variables and the feature quantities of the time-shifted column are the explanatory variables, different feature identifiers are used for the feature quantities of the original column and the feature quantities of the time-shifted column.

The evaluation dataset creation unit 24 uses the validation data created by the validation data creation unit 25 and the learning model created by the learning model creation unit 26 as evaluation data. The evaluation dataset creation unit 24 collects the evaluation data for the number of learning patterns, i.e., the number of indexes included in the learning pattern set, as an evaluation dataset and stores it in the evaluation dataset storage unit 27.

The verification data creation unit 25 acquires material data from the material data storage unit 22, and creates learning data from the material data according to the learning data construction method included in the learning pattern. The verification data creation unit 25 also extracts a portion of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 26. The verification data is data used to verify the created learning model, and is data that is not used for machine learning of the learning model.

The learning model creation unit 26 creates a learning model using the learning data output from the validation data creation unit 25 according to the base model identifier and hyperparameters contained in the learning pattern.

The learning model evaluation unit 28 evaluates each of the multiple learning models stored in the evaluation dataset storage unit 27 based on the indexes generated by the acquisition unit 29 and the generation unit 30, and selects a desired learning model by the selection unit 31. Then, the learning model evaluation unit 28 stores the learning model selected by the selection unit 31 in the selected learning model storage unit 32.

The acquisition unit 29 acquires the prediction accuracy of each learning model and the contribution of at least one type of feature specified by the user to the prediction result by the learning model. Specifically, the acquisition unit 29 acquires the prediction accuracy based on the error between the output data of the learning model when the explanatory variables included in the validation data are input to the learning model and the objective variables included in the validation data. The prediction accuracy may be, for example, a value based on RMSE (Root Mean Square Error). In addition, the acquisition unit 29 acquires the contribution of each type of feature constituting the learning data using the validation data. The contribution may be, for example, a value based on the SHAP value. The acquisition unit 29 outputs the prediction accuracy and contribution of each acquired learning model to the generation unit 30.

The generating unit 30 generates an index for selecting a learning model for each learning model using the prediction accuracy and contribution output from the acquiring unit 29. The generating unit 30 may generate an index using only the contribution of the type of feature specified by the user. The generating unit 30 may also generate an index using the contribution of each type of feature to which a weight specified by the user is added. This makes it possible to generate an index that takes into account the prediction accuracy and the contribution of a desired feature, i.e., the basis for a desired prediction. When weighting the contribution, it is further possible to adjust which type of feature is given more importance in generating an index.

The generating unit 30 may generate an index after matching the scale of the contribution degree for each type of feature. The scale here refers to a scale of the magnitude of the value. This allows the generation of an index that can fairly evaluate the relative contribution degree for each type of feature. The generating unit 30 may also match the scale of the prediction accuracy and the contribution degree to generate an index. This allows the generation of an index that can fairly evaluate the magnitude of the prediction accuracy and the contribution degree. The generating unit 30 may also generate an index by adding a weight to at least one of the prediction accuracy and the contribution degree. This allows the adjustment of which of the prediction accuracy and the contribution degree is emphasized when selecting a learning model. Note that, as described above, by matching the scale of the prediction accuracy and the contribution degree and adding a weight to at least one of the prediction accuracy and the contribution degree, it is possible to more appropriately adjust which of the prediction accuracy and the contribution degree is emphasized. The generating unit 30 outputs the index generated for each learning model to the selecting unit 31.

The selection unit 31 selects the learning model with the highest index from the multiple learning models stored in the evaluation dataset storage unit 27 based on the index of each learning model output from the generation unit 30. The selection unit 31 may select one or more learning models with an index equal to or greater than a predetermined value, or may select a predetermined number of learning models with the highest indexes. When multiple learning models are selected, the selection unit 31 may present the selected learning models together with the indexes to the user and accept the user's selection of the learning model to be adopted.

Next, the operation of the learning model selection device 10 will be described. Figure 5 is a sequence diagram showing the flow of the learning model selection process by the learning model selection device 10. The learning model selection process is performed by the CPU 11 reading out the learning model selection program including the generation program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it.

In step S11, the CPU 11, as the material data collection unit 21, accepts input of a data identifier specified by the user, collects material data having the accepted data identifier, and stores it in the material data storage unit 22. Next, in step S12, the CPU 11, as the learning pattern transmission unit 23, accepts input of a learning pattern set specified by the user, and transmits the accepted learning pattern set to the evaluation dataset creation unit 24.

Next, in step S13, the CPU 11, as the evaluation dataset creation unit 24, acquires material data from the material data storage unit 22. Next, in step S14, the CPU 11, as the evaluation dataset creation unit 24, executes an evaluation dataset creation process. Specifically, the CPU 11, as the verification data creation unit 25 of the evaluation dataset creation unit 24, creates learning data from the material data for each learning pattern included in the learning pattern set, according to the learning data construction method included in the learning pattern. Also, the CPU 11, as the verification data creation unit 25, extracts a part of the created learning data as verification data, and outputs the remaining learning data to the learning model creation unit 26. Furthermore, the CPU 11, as the learning model creation unit 26 of the evaluation dataset creation unit 24, creates a learning model using the learning data output from the verification data creation unit 25 according to the base model identifier and hyperparameters included in the learning pattern.

Next, in step S15, the CPU 11, as the evaluation dataset creation unit 24, sets the validation data created by the validation data creation unit 25 and the learning model created by the learning model creation unit 26 as validation data. Then, the CPU 11, as the evaluation dataset creation unit 24, compiles the validation data for the number of learning patterns, i.e., the number of indexes included in the learning pattern set, as an evaluation dataset, and stores it in the evaluation dataset storage unit 27.

Next, in step S16, the CPU 11, as the learning model evaluation unit 28, acquires the evaluation dataset stored in the evaluation dataset storage unit 27. Also, in step S17, the CPU 11, as the learning model evaluation unit 28, accepts input of the type of feature specified by the user ("designation of feature" in FIG. 5). Note that the CPU 11, as the learning model evaluation unit 28, may accept weights for each type of feature with respect to the contribution degree ("contribution weight" in FIG. 5) instead of designating the feature. Furthermore, the CPU 11, as the learning model evaluation unit 28, may accept a weight to be assigned to at least one of the prediction accuracy and the contribution degree when generating an index ("overall weight" in FIG. 5).

Next, in step S18, the CPU 11 executes a learning model evaluation process as the learning model evaluation unit 28. Specifically, the CPU 11, as the acquisition unit 29 of the learning model evaluation unit 28, acquires prediction accuracy for each learning model included in the evaluation dataset using the verification data. Also, as the acquisition unit 29, the CPU 11 acquires the contribution rate for each type of feature using the verification data for each learning model. Then, as the generation unit 30 of the learning model evaluation unit 28, the CPU 11 generates an index for selecting a learning model for each learning model using the prediction accuracy acquired by the acquisition unit 29 and the contribution rate for the type of feature specified by the user. Note that, if a contribution rate weight is accepted instead of the specification of the feature, the CPU 11, as the generation unit 30, generates an index by adding the contribution rate weight to the contribution rate for each type of feature. Furthermore, if an overall weight is accepted, the CPU 11, as the generation unit 30, generates an index by adding an overall weight to at least one of the prediction accuracy and the contribution rate. Then, the CPU 11, as the selection unit 31 of the learning model evaluation unit 28, selects the learning model with the highest index from the multiple learning models stored in the evaluation dataset storage unit 27 based on the index of each learning model generated by the generation unit 30.

Next, in step S19, the CPU 11, as the learning model evaluation unit 28, stores the learning model selected by the selection unit 31 ("selected learning model" in Figure 5) in the selected learning model storage unit 32, and the learning model selection process is terminated.

In step S18, the process executed by CPU 11 as acquisition unit 29 and generation unit 30 is an example of a generation process performed by CPU 11 executing a generation program included in the learning model selection program. The generation process is also an example of a generation method of the disclosed technology.

Conventionally, the evaluation of learning models does not take into account the magnitude or positive/negative of the contribution of each type of feature to the prediction result. Therefore, when it is specified that the learning model is evaluated using two indices, "prediction accuracy" and "contribution of each type of feature," it may not be possible to select an appropriate learning model. For example, as in the technology described in Non-Patent Document 1, even if multiple learning models with different structures and features used are listed in descending order of prediction accuracy, they are not necessarily listed in the optimal order that satisfies the desired prediction accuracy and prediction basis. This is because in the conventional technology, model candidates are listed based only on the prediction accuracy estimated based on a predetermined index. The contribution of each type of feature changes depending on the nature of the learning data and the learning model, the cost function designed after selecting the learning model, etc. This point is not taken into consideration in the conventional technology, making it difficult to select a learning model that satisfies both the high prediction accuracy of the learning model and the distribution of the contribution of the model's features that is close to the distribution of the contribution of the desired features. Therefore, in the conventional technology, it is necessary to manually search for the optimal learning model from the listed learning models. In particular, as the number of types of features increases, the number of candidate learning models also becomes enormous, making the task of selecting a learning model itself difficult.

On the other hand, the learning model selection device according to the first embodiment generates an index for each of a plurality of learning models from the prediction accuracy of the learning model and the contribution of at least one type of feature specified by the user. In other words, this index takes into account both the contribution of the type of feature specified by the user and the prediction accuracy. The learning model selection device then uses this index to select a predetermined learning model from among a plurality of learning models that have been machine-learned using a plurality of types of feature. This allows the learning model selection device according to the first embodiment to appropriately select a learning model that provides the desired prediction accuracy and basis for prediction.

Second Embodiment
Next, a second embodiment will be described. In the second embodiment, an embodiment that is a more specific version of the first embodiment will be described. Specifically, in an air conditioning utilization section that exists inside a building and is separated from the outside by a door or the like, the optimum control method for the temperature setting of an air conditioning controller is calculated by reinforcement learning with the goal of optimizing thermal comfort. The learning model to be selected is an environmental model that predicts temperature changes according to the temperature setting of the air conditioner, which is the action of the reinforcement learning agent. Note that the hardware configuration of the learning model selection device according to the second embodiment is the same as the hardware configuration of the learning model selection device 10 according to the first embodiment shown in FIG. 3, and therefore a description thereof will be omitted.

The functional configuration of the learning model selection device 110 according to the second embodiment will be described. FIG. 6 is a block diagram showing an example of the functional configuration of the learning model selection device 110. As shown in FIG. 6, the learning model selection device 110 includes, as its functional configuration, a material data collection unit 121, a material data storage unit 122, a learning pattern transmission unit 123, an evaluation data set creation unit 124, and an evaluation data set storage unit 127. The evaluation data set creation unit 124 further includes a verification data creation unit 125 and a learning model creation unit 126. The learning model selection device 110 also includes a learning model evaluation unit 128 and a selected learning model storage unit 132. The learning model evaluation unit 128 further includes an acquisition unit 129, a generation unit 130, and a selection unit 131. The acquisition unit 129 and the generation unit 130 are examples of a generation device of the disclosed technology. Each functional configuration is realized by the CPU 11 reading out a learning model selection program including a generation program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

Note that detailed explanations of the common contents between the functional configuration of the learning model selection device 110 in the second embodiment and the functional configuration of the learning model selection device 10 in the first embodiment, where the last two digits of the symbols are the same, will be omitted.

The material data collection unit 121 collects material data related to air conditioning control. For example, the material data collection unit 121 collects each of the material data having a data identifier represented by the room temperature, the outside temperature, the flow of people, the air conditioning setting value, and the open flag. The room temperature is the temperature measured in the air conditioning utilization unit. The outside temperature is the temperature measured outdoors. The flow of people is the number of unique people present in the air conditioning utilization unit. The unique number of people is the number of people present in the air conditioning utilization unit per unit time, and is the number of people that are not counted twice for the same person even if they enter or exit the air conditioning utilization unit within the unit time (i.e., it is not the total number of people). The unit time may be, for example, the sampling interval of the data. The air conditioning setting value is the temperature setting value of the air conditioner present in the air conditioning utilization unit. The open flag is a flag indicating whether people can enter or exit the building including the air conditioning utilization unit. For example, if it is possible to enter or exit the building, it may be set to "1", and if it is not possible to enter or exit, it may be set to "0".

Of the material data, room temperature is material data used as the objective variable, while outside temperature, foot traffic, air conditioning setting value, and open flag are material data used as explanatory variables. If the learning model is intended to be used as an environmental model for reinforcement learning, as in the second embodiment, it is necessary to include data that corresponds to the action of the reinforcement learning agent, or data that has been processed from that, among the explanatory variables. This is because the output of the environmental model needs to change depending on the action of the reinforcement learning agent. Here, an explanatory variable that uses the air conditioning setting value is essential.

The material data collection unit 121 may collect, for example, room temperature, outside temperature, air conditioning settings, and open flags from the BEMS (Building and Energy Management System) 201. The material data collection unit 121 may also collect people flow from a people flow detection sensor 202 installed in the air conditioning utilization unit. All of this material data is time-series data. Specifically, each material data is time-series data in which the date and time of the data sampling point is used as an index, and the index corresponds to the data value at the date and time indicated by the index. The material data collection unit 121 stores the collected material data in the material data storage unit 122. Figure 7 shows an example of material data stored in the material data storage unit 122.

The learning pattern transmission unit 123 accepts input of a learning pattern set specified by the user and transmits the accepted learning pattern set to the evaluation dataset creation unit 124. FIG. 8 shows an example of a learning pattern set. In the example of FIG. 8, learning pattern (p) is an index of the learning pattern. Here, p = 1, 2, 3. In addition, hereinafter, a learning pattern with index p (p = 1, 2, 3) is written as "learning pattern p". In addition, in the example of FIG. 8, the base model identifier of learning pattern 1 is Light GBM (Gradient Boosting Machine, Reference 2). In addition, the base model identifier of learning patterns 2 and 3 is XGBoost (eXtreme Gradient Boosting, Reference 3). There is a partial difference in the learning data construction method (details will be described later) between learning pattern 2 and learning pattern 3.

Reference 2: Ke et al., "LightGBM: A Highly Efficient Gradient Boosting Decision Tree", 2017.
Reference 3: Tianqi Chen, Carlos Guestrin, "XGBoost: A Scalable Tree Boosting System", 2016.

Here, in the second embodiment, the material data is time-series data, and can be expressed as table data including multiple columns as shown in FIG. 7 above. Therefore, it is possible to create new columns that become features constituting the learning data by performing calculations between different columns of table data, obtaining the time difference of the same column, etc. Furthermore, in the case of explanatory variables, it is possible to create a new column by shifting the index of the new column created as above, i.e., a column by shifting the time series. Hereinafter, a column with a shifted time series in this way is referred to as a "series column." The method of creating this new column is defined in the learning data construction method for learning patterns.

In the example of FIG. 8, a feature identifier F _i ^p , a calculation formula E _i ^p , and sequence parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p are defined as the learning data construction method. The feature identifier F _i ^p is an identifier indicating the type of feature. i is the index of the feature identifier, and here, i=0, 1, 2, 3, 4, 5. Therefore, the feature identifier F _i ^p indicates the i-th type of feature of the learning pattern p. In addition, in the case of a sequence column, the feature of the material data column that is the source of the shift or the newly created column and the sequence column are essentially the same type of feature, except that the time series is shifted, so the same feature identifier F _i ^p is used. For example, a certain column A and a newly created column A' obtained by shifting the data of column A 30 minutes backward both have the same feature identifier A. In the following, in order to distinguish each of the sequence columns, a notation indicating the shifted time, such as F _i ^p -30 min, is added to the feature identifier. The feature quantity indicated by the feature identifier F _i ^p for i=0 is the objective variable, and the feature quantity indicated by the feature identifier F _i ^p for i≧1 is the explanatory variable. For example, the room temperature difference (F ₀ ¹ ) in the first row of Fig. 8 indicates that the type of feature quantity indicated by the feature identifier F ₀ ¹ in learning pattern 1 is the objective variable "room temperature difference". Also, for example, the room temperature difference (F ₁ ¹ ) in the second row of Fig. 8 indicates that the type of feature quantity indicated by the feature identifier F ₁ ¹ in learning pattern 1 is the explanatory variable "room temperature difference".

The formula E _i ^p is a formula for calculating a new column that is a feature quantity from the material data, and is specified using the data identifier of the material data. For example, the formula in the first line of Fig. 8 is a formula E ₀ ¹ for calculating the room temperature difference (F ₀ ¹ ) at time t, and E ₀ ¹ specifies that the room temperature 60 minutes before time t is subtracted from the room temperature at time t. Hereinafter, the <data identifier> or <feature identifier> at time t will be expressed as "<data identifier> or <feature identifier>(t)", for example, "room temperature (t)" or "room temperature difference (t)".

The sequence parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p are parameters for creating a sequence column based on the new column X created by the formula E _i ^p . S1 _i ^p is a parameter that specifies the number of sequences, S2 _i ^p the start time point, and S3 _i ^p the end time point. Specifically, this indicates that the interval from S2 _i ^p to S3 _i ^p is divided into equal intervals the number of S1 _i ^p , and a sequence column is created by shifting X to each time point. In the case of feature identifier F ₀ ^p , i.e., in the case of the objective variable, there is no need to create a sequence column, so no sequence parameters are specified.

The verification data creation unit 125 acquires material data from the material data storage unit 122, and creates learning data from the material data according to the learning data construction method included in the learning pattern. A specific example of learning data creation will be described using the example of the learning pattern set shown in FIG. 8. The verification data creation unit 125 calculates the room temperature difference (t) indicated by the feature identifier F ₀ ¹ by E ₀ ¹ = room temperature (t) - room temperature (t-60 min). For example, the verification data creation unit 125 calculates as follows: [F ₀ ¹ at 2020-01-01 09:00:00] = [room temperature at 2020-01-01 09:00:00] - [room temperature at 2020-01-01 08:00:00]. FIG. 9 shows an example of the room temperature difference (t) indicated by the feature identifier F ₀ ¹ created as the objective variable of the learning data.

Furthermore, for example, the verification data creation unit 125 uses the calculation formula E ₁ ¹ and the series parameters S1 _{1 1} ¹ /S2 ₁ ^{1 1} /S3 ₁ ¹ =6/-60 min/-360 min to calculate the series column of the room temperature difference (t) indicated by the feature identifier F ₁ ^1. Specifically, the verification data creation unit 125 shifts F ₁ ¹ calculated by the calculation formula E ₁ ¹ =room temperature (t)-room temperature (t-60 min) by -60 min to calculate F ₁ ¹ -60 min as room temperature (t-60 min)-room temperature (t-120 min). For example, the verification data creation unit 125 calculates [F ₁ ¹ -60 min at 09:00:00 on 2020-01-01] = [Room temperature at 08:00:00 on 2020-01-01] - [Room temperature at 07:00:00 on 2020-01-01]. FIG. 10 shows an example of a series column of feature identifiers F ₁ ¹ to F ₅ ¹ created as explanatory variables of the learning data. Note that, following the example of FIG. 8, F ₁ ¹ in FIG. 10 is a room temperature difference, F ₂ ¹ is an outside temperature, F ₃ ¹ is a people flow difference, F ₄ ¹ is an air conditioning setting difference, and F ₅ ¹ is an open flag.

Furthermore, the verification data creation unit 125 extracts a portion of the learning data as verification data, and outputs the remaining learning data to the learning model creation unit 126. For example, when the verification data creation unit 125 acquires one month's worth of material data, it creates one month's worth of learning data. Then, the verification data creation unit 125 extracts one week's worth of learning data, which is specified in advance, from the one month as verification data.

The learning model creation unit 126 sets hyperparameters in the model structure indicated by the base model identifier included in the learning pattern, inputs explanatory variables of the learning data into the learning model in which the parameters to be adjusted have been set to initial values, and obtains output data. The learning model creation unit 126 then executes machine learning of the learning model by updating the parameters so that the output data and the objective variable are closer to each other. Note that if the hyperparameters are not uniquely determined but are specified as ranges, the learning model creation unit 126 may create the learning model while searching for hyperparameters that will maximize the performance of the learning model.

The acquiring unit 129 uses the validation data for each learning pattern to calculate SHAP i p, which is obtained by converting the _SHAP value of the same number of elements as the explanatory variables of the validation data (the number of columns of the explanatory variables × the number of indexes of the validation data) into an absolute value. Then, as shown in FIG _. 11, the acquiring unit _{129 calculates the average (SHAP i} ^{p )} _̂ ⁽ in FIGS. 11, ¹³ , and in the formula, "̂ (overline)" above "SHAP _i ^p ") of each element in the validation data period of all columns belonging to the feature identifier F i p). The acquiring unit 129 calculates c _i ^p scaled so that all (SHAP _i ^p ) ̂ falls within the range of 0 to 100, using the following formula (1). The acquiring unit 129 also creates a contribution evaluation vector c ^p (written in bold in the formula) in which c _i ^p are arranged, as shown in the following formula (2). That is, c _i ^p corresponds to the contribution degree for the type of feature amount indicated by the feature identifier F _i ^p .

The acquisition unit 129 also acquires a contribution weight vector W1 (indicated in bold in the formula) designated by the user. The contribution weight vector W1 is a vector in which each element represents a weight corresponding to each element of the contribution evaluation vector c ^p , such as [w ₁ ^p , w ₂ ^p , w ₃ ^p , w ₄ ^p , w ₅ ^p ] ^T (T represents transposition), as shown in FIG. 12, for example. The contribution weight vector W1 is created in advance by the user, and the weight of each element may be set to a value according to the degree of importance to be placed on when selecting a learning model. For example, as in the second embodiment, in the context of "construction of an environmental model for reinforcement learning", the weight of the "feature amount related to the action of the reinforcement learning agent" may be set large. Here, the "air conditioning setting value" corresponds to this. Note that the designation of the feature amount as described in the first embodiment can be realized by using two values, 0 and 1, as the weight. The acquisition unit 129 multiplies the contribution evaluation vector c ^p by the contribution weight vector W1 as shown in the following formula (3) to acquire the contribution evaluation α ^p .

The acquiring unit 129 also calculates the root mean square error RMSE ^p using the validation data, and acquires the inverse of RMSE ^p as the accuracy evaluation β ^p , for example, as shown in the following formula (4). The acquiring unit 129 outputs the acquired contribution evaluation α ^p and accuracy evaluation β ^p to the generating unit 130.

The generation unit 130 uses the contribution evaluation ^αp and accuracy evaluation ^βp output from the acquisition unit 129 to calculate a contribution scaling constant K for matching the scale of the contribution evaluation ^αp and the accuracy evaluation ^βp , for example, by the following equation (5).

Furthermore, the generating unit 130 acquires an overall contribution weight W2, for example, as shown in Fig. 12. The overall contribution weight W2 is a value set in advance by the user, and may be set to a value according to how much importance is attached to the contribution when the prediction accuracy is set to 1. The generating unit 130 uses the contribution evaluation ^αp , the accuracy evaluation ^βp , the contribution scaling constant K, and the overall contribution weight W2 to generate a model evaluation function ^Lp , for example, by the following formula (6).

Note that instead of the contribution scaling constant K, a prediction accuracy scaling constant K' obtained by reciprocating equation (5) may be used. In this case, in equation (6), the term β ^p is multiplied by K'. Also, instead of the overall contribution weight W2, an overall prediction accuracy weight W2', which is a weight according to how much importance is attached to prediction accuracy when the contribution is set to 1, may be used. In this case, in equation (6), the term β ^p is multiplied by W2'.

Fig. 13 shows an example of (SHAP _i ^p ), contribution evaluation vector c ^p , and RMSE ^p calculated for each learning pattern by the acquisition unit 129. Fig. 14 shows an example of the contribution evaluation α ^p and accuracy evaluation β ^p acquired by the acquisition unit 129, the contribution scaling constant K calculated by the generation unit 130, the contribution evaluation after scale conversion α ^p ×K, and the model evaluation function L ^p generated by the generation unit 130 for each learning pattern.

The selection unit 131 selects, from the multiple learning models stored in the evaluation dataset storage unit 127, a learning model that maximizes the model evaluation function L ^p generated by the generation unit 130.

Next, the operation of the learning model selection device 110 will be described. In the second embodiment, as in the first embodiment, the CPU 11 reads out a learning model selection program including a generation program from the ROM 12 or storage 14, expands it into the RAM 13, and executes it to perform the learning model selection process shown in FIG. 5. Here, more detailed processing of the evaluation data creation process in step S14 in FIG. 5 and the learning model evaluation process in step S18 will be described using the flowcharts shown in FIG. 15 and FIG. 16.

First, we will explain the evaluation dataset creation process shown in Figure 15.

In step S101, the CPU 11, functioning as the evaluation dataset creation unit 124, initializes a learning model set and a validation dataset. Specifically, the CPU 11, functioning as the evaluation dataset creation unit 124, prepares an empty set to which the created learning model is to be added, and an empty set to which the created validation data is to be added. Furthermore, the CPU 11, functioning as the evaluation dataset creation unit 124, sets a variable p indicating the index of the learning pattern to 1.

Next, in step S102, the CPU 11, functioning as the evaluation dataset creation unit 124, determines whether p is equal to or less than the maximum index value pmax (here, pmax = 3). If p ≤ pmax, the process proceeds to step S103.

In step S103, the CPU 11, functioning as the evaluation dataset creation unit 124, initializes the learning dataset. Specifically, the CPU 11, functioning as the evaluation dataset creation unit 124, prepares an empty set to which the created learning data is to be added. Then, the CPU 11, functioning as the verification data creation unit 125, sets a variable i indicating an index of the feature identifier to 0.

Next, in step S104, the CPU 11, functioning as the verification data creation unit 125, determines whether i is equal to or less than the maximum index value imax (here, imax = 5). If i ≤ imax, the process proceeds to step S105.

In step S105, the CPU 11, functioning as the verification data generating unit 125, obtains the material data used in the calculation formula _E ^ip from the material data storage unit 122, and applies the calculation formula _E ^ip to the obtained material data to generate a new column X.

Next, in step S106, the CPU 11, functioning as the verification data creation unit 125, determines whether i is 0 or not, i.e., whether the feature quantity indicated by the feature identifier F _i ^p corresponding to column X is the objective variable or not. If i=0, the process proceeds to step S107, and the CPU 11, functioning as the verification data creation unit 125, adds column X to the learning data set. On the other hand, if i≧1, the process proceeds to step S108. In step S108, the CPU 11, functioning as the verification data creation unit 125, reads sequence parameters S1 _i ^p , S2 _i ^p , and S3 _i ^p , divides the interval from S2 _i ^p to S3 _i ^p at equal intervals by the number of S1 _i ^p , and creates sequence columns in which X is shifted to each time point. Then, the CPU 11, functioning as the verification data creation unit 125, adds the created sequence columns to the learning data set.

Next, in step S109, the CPU 11, functioning as the verification data creation unit 125, increments i by 1, and the process returns to step S104. If it is determined in step S104 that i>imax, the process proceeds to step S110.

In step S110, the CPU 11, functioning as the validation data creation unit 125, extracts a portion of the learning data included in the learning dataset as validation data and adds it to the validation dataset. The CPU 11, functioning as the validation data creation unit 125, also outputs the remaining learning data to the learning model creation unit 126.

Next, in step S111, the CPU 11, as the learning model creation unit 126, obtains the model structure specified by the base model identifier of the learning pattern p by calling an API from the base model identifier, etc., and sets the hyperparameters of the learning pattern p. Then, the CPU 11, as the learning model creation unit 126, uses the remaining learning data output from the validation data creation unit 125 to perform machine learning of the learning model in which the hyperparameters are set to the model structure indicated by the base model identifier. The CPU 11, as the learning model creation unit 126, performs machine learning while evaluating the learning model by, for example, grid search cross validation.

Next, in step S112, the CPU 11, functioning as the learning model creation unit 126, adds the completed learning model to the learning model set. Next, in step S113, the CPU 11, functioning as the evaluation dataset creation unit 124, increments p by 1, and the process returns to step S102. If it is determined in step S102 that p>pmax, the evaluation dataset creation process ends.

The validation dataset and learning model set created by the evaluation dataset creation process are stored as evaluation datasets in the evaluation dataset storage unit 127 (S17 in Figure 5).

Next, we will explain the learning model evaluation process shown in Figure 16.

In step S121, the CPU 11, functioning as the learning model evaluation unit 128, sets a variable p indicating the index of the learning pattern to 1. Next, in step S122, the CPU 11, functioning as the learning model evaluation unit 128, determines whether p is equal to or less than the maximum index value pmax (here, pmax = 3). If p ≤ pmax, the process proceeds to step S123.

In step S123, the CPU 11, functioning as the acquisition unit 129, calculates SHAP _i ^p by converting the SHAP value having the same number of elements as the explanatory variables of the validation data into an absolute value, using the validation data of the learning pattern p. Next, in step S124, the CPU 11, functioning as the acquisition unit 129, sets 1 to a variable i indicating an index of the feature identifier.

Next, in step S125, the CPU 11, functioning as the acquisition unit 129, determines whether i is equal to or less than the maximum index value imax (here, imax = 5). If i ≤ imax, the process proceeds to step S126.

In step S126, the CPU 11, as the acquisition unit 129, calculates the average (SHAP _i ^p ) of the SHAP _i ^p of each element during the period of the verification data of all columns belonging to the feature identifier F _i ^p . Next, in step S127, the CPU 11, as the acquisition unit 129, increments i by 1, and the process returns to step S125. If it is determined in step S125 that i>imax, the process proceeds to step S128.

In step S128, the CPU 11, as the acquisition unit 129, calculates c _i ^p scaled using, for example, equation (1) so that all (SHAP _i ^p ) falls within the range of 0 to 100. Next, in step S129, the CPU 11, as the acquisition unit 129, creates a contribution evaluation vector c ^p in which c _i ^p are arranged as shown in equation (2). Furthermore, the CPU 11, as the acquisition unit 129, acquires a contribution weight vector W1 designated by the user, and multiplies the contribution evaluation vector c ^p and the contribution weight vector W1 together as shown in equation (3) to acquire a contribution evaluation α ^p .

Next, in step S130, the CPU 11, as the acquisition unit 129, calculates the mean square error RMSE ^p using the verification data of the learning pattern p, and acquires the inverse of RMSE ^p as the accuracy evaluation β ^p , for example, as shown in equation (4). Next, in step S131, the CPU 11, as the learning model evaluation unit 128, increments p by 1, and the process returns to step S122. If it is determined in step S122 that p>pmax, the process proceeds to step S132.

In step S132, the CPU 11, as the generation unit 130, uses the contribution evaluation α ^p and the accuracy evaluation β ^p acquired by the acquisition unit 129 to calculate a contribution scaling constant K for matching the scales of the contribution evaluation α ^p and the accuracy evaluation β ^p , for example, by equation (5).

Next, in step S133, the CPU 11, as the generation unit 130, obtains the overall contribution weight W2 specified by the user. Then, the CPU 11, as the generation unit 130, generates a model evaluation function ^Lp for each learning pattern using the contribution evaluation ^αp , the accuracy evaluation ^βp , the contribution scaling constant K, and the overall contribution weight W2, for example, by equation (6). Then, the CPU 11, as the selection unit 131, selects a learning model that maximizes the model evaluation function ^Lp generated by the generation unit 130 from the multiple learning models stored in the evaluation dataset storage unit 127, and the learning model evaluation process ends.

The learning model selected by the learning model evaluation process is stored in the selected learning model storage unit 132 (S19 in Figure 5).

As described above, the learning model selection device of the second embodiment can also appropriately select a learning model that provides the desired prediction accuracy and basis for prediction, just like the learning model selection device of the first embodiment.

In addition, the scale of the contribution degree for each type of feature calculated for each learning model does not necessarily match, and therefore, if used as is, it is not possible to perform a fair comparison between learning models. For example, the sum of (SHAP _i ^p ) for each learning pattern does not match, and a learning model with a large sum of (SHAP _i ^p ) is likely to be evaluated as having an unduly high contribution degree. In the second embodiment, the contribution degree for each type of feature is scaled from 0 to 100, for example, as shown in formula (1), so that the sum of c ^p for each learning pattern becomes 100, and the relative magnitude of the contribution degree for each type of feature can be evaluated fairly.

In the second embodiment, by using a "contribution weight vector" that indicates the individual weight for each type of feature, the contribution of a desired feature can be emphasized in the evaluation. For example, by designing a contribution weight vector W1 as shown in Fig. 12, it is possible to emphasize the air conditioning setting value or air conditioning room temperature difference indicated by the fourth feature identifier F ₄ ^p , which has the greatest weight. As a result, as shown in Figs. 13 and 14, the contribution evaluation α ^p of learning pattern 1, in which c ₄ ^p corresponding to feature identifier F ₄ ^p is the largest among c _i ^p , is maximized.

In addition, since the scales of the contribution and accuracy do not necessarily match, either the contribution evaluation or the accuracy evaluation may be too large or too small. For example, in the example of FIG. 14, the contribution evaluation α ^p is generally larger than the accuracy evaluation β ^p . Therefore, if the model evaluation function L ^p is generated with K=1, the contribution evaluation becomes too large, and the difference in the accuracy evaluation has almost no effect on the model evaluation function L ^p regardless of the user's intention. In the second embodiment, for example, as shown in formula (5), a contribution scaling constant K for matching the scales of the contribution evaluation and the accuracy evaluation is calculated and multiplied by the contribution evaluation α ^p , so that the scales of the contribution evaluation α ^p and the accuracy evaluation β ^p become almost the same, as shown in FIG. 14, and the evaluation can be performed fairly.

Moreover, by designing K as in the second embodiment, the user can intuitively design the overall contribution weight W2. For example, when the contribution evaluation and the accuracy evaluation are equally important, W2=1, when the contribution evaluation is important, W2>1, and when the accuracy evaluation is important, W2<1. For example, in the second embodiment, an example is shown in which W2=1.5, but when W2=0.5, the accuracy evaluation is emphasized, and as shown in FIG. 17, the model evaluation function ^Lp of the learning pattern 3 with the largest accuracy evaluation ^βp is the largest.

Here, a method for adjusting W2 will be described using a specific example similar to that of the second embodiment. As a premise, it is desirable that the room temperature predicted by the learning model fluctuates according to the air conditioning setting value. In addition, after a certain time has passed since the air conditioner was started, the room temperature becomes steady due to thermal equilibrium. We would like to reproduce that behavior with the prediction of the learning model. Specifically, if the heating is started one hour earlier, the predicted room temperature also starts one hour earlier, and after four hours have passed since the air conditioner was started, the predicted room temperature converges to the room temperature when the air conditioner was not started one hour earlier, that is, the true room temperature value. After the air conditioner is started, the speed at which the predicted room temperature converges to the true room temperature value depends on the contribution of the air conditioning setting value, and in the example of the second embodiment, it depends on the contribution evaluation. In order to confirm that the air conditioning setting value contributes to the prediction of the room temperature as expected, the behavior of the predicted room temperature is observed when the air conditioning setting value is shifted one hour earlier from the original data.

The procedure for adjusting W2 under the above assumptions is as follows. First, as shown in FIG. 18, when W2 = 1.0, FIG. 19 shows the prediction results when learning pattern 3 that maximizes the model evaluation function is selected. Note that in FIG. 19 and FIG. 20 described below, temperature is the true room temperature, and shifted_predict is the predicted room temperature when the air conditioning setting value is shifted one hour ago. Also, air control temperature is the air conditioning setting value or air conditioning setting value difference, and Shifted air control temperature is the air conditioning setting value or air conditioning setting value difference shifted to one hour ago.

The learning model for learning pattern 3 has the highest accuracy, but even if the air conditioner is started at around 7:00, the predicted room temperature converges to the true room temperature by around 10:00 (the area circled in dashed lines in Figure 19), which deviates from the ideal behavior. In this case, it is thought that the contribution of the air conditioning setting to the learning model's predictions is insufficient, so W2 is increased to 1.5 and the learning model is selected again.

Figure 20 shows the prediction results when learning pattern 1, which maximizes the model evaluation function, is selected when W2 = 1.5, as shown in Figure 18. The accuracy of this learning model is slightly lower than that of the learning model of learning pattern 3, but after the air conditioner is started at around 7:00, the predicted room temperature converges to the true room temperature value at around 11:00 (the area surrounded by the dashed circle in Figure 20), which is close to ideal behavior. Therefore, the contribution of the air conditioning setting value is considered to be sufficient, and the learning model selected with W2 = 1.5 is adopted. Note that if the time at which the predicted room temperature converges to the true room temperature value is late, such as 12:00, the contribution is considered to be too high, so W2 is reduced and the learning model is selected again.

<Modification>
In the second embodiment, a case has been described in which time-series data such as room temperature and air conditioning setting values and a regression problem using a decision tree model such as LightGBM are handled, but the disclosed technology is also applicable to other types of data and problems. Below, the differences between each modification and the second embodiment will be mainly described.

As a first variant, a regression problem of stock price prediction using a deep learning model and time series data of stock transactions can be given. In this problem, based on data on stock transactions of a certain stock, a future "current price" is regressed as the objective variable. As data identifiers for material data, "current price", "highest purchase price", "lowest sale price", "highest number of sale prices", "highest number of purchase prices", "total number of purchase orders", "total number of sell orders", etc. are used. The base model identifier may be selected from two types, for example, LSTM (Long Short-Term Memory, Reference 4) and QRNN (Quasi-Recurrent Neural Networks, Reference 5). The hyperparameters are the number of hidden layer nodes, number of steps, batch size, dropout rate, and number of layers for the layer of the base model identifier, the number and total number of nodes for the fully connected layer, the number of nodes and activation function for the activation function layer, and the optimization function. In addition, since the size of data related to transactions varies greatly depending on the stock, a normalization process may be added to all feature identifiers in the calculation formula of the learning data construction method. In addition, in the evaluation dataset creation unit, the explanatory variables of the learning data are a two-dimensional matrix of [number of data × (number of features × number of steps)], but a process is added to convert this into a three-dimensional matrix of [number of data × number of steps × number of features].

As a second modified example, a classification problem of the subscription status of a paid membership service using a machine learning model and time series data of membership information can be given. In this problem, for a certain paid membership service, the future "subscription status of the service (subscribed or unsubscribed)" is predicted based on time series data such as the customer's usage history. As data identifiers for the material data, "customer ID", "gender", "service subscription date", "service unsubscription date (those who have not unsubscribed are Nan values)", "service usage time on that day", "service subscription status", etc. are used. When the customer IDs are different, the same index may be duplicated for the index of the time series data. In addition, in the learning data construction method, "number of days of service usage in the current month" may be created as a new feature identifier. The calculation formula for this feature identifier may be to divide the data by "customer ID", group the data by the year and month of the index of the time series data, and count the number of data whose "service usage time on that day" is 0 or more. In addition, the calculation formula for the "service subscription date" may include a process of converting the year and month to days. In addition, a process of converting the Nan value of the "service unsubscription date" to -1 may be included. Since "customer ID" and "gender" are categorical variables, it may be specified that label encoding is performed in these calculation formulas. When creating a series column using the series parameters of the evaluation dataset creation unit, the data is divided by "customer ID" before processing.

A third variant is a regression problem of house prices using a machine learning model and house feature data. In this problem, the "price" of a house is regressed from various information about the house. The data identifiers of the material data include "house type (apartment, house, etc.)", "prefecture", "city/town", "walking time to the nearest station", "age", "floor plan (1K, 2LDK, etc.)", "exclusive area", "whether renovated", and "price". In addition, in the formula of the learning data construction method, categorical variables containing characters such as "house type", "prefecture", "city/town", "floor plan", and "whether renovated" may be specified to perform label encoding. Since the material data in this problem is not time series data, there is no need to specify series parameters in the learning data construction method, and no process is performed to create a series column.

As a fourth variant, there is a problem of classifying iris varieties using a machine learning model and feature data of iris flowers. In this problem, the "variety" of iris flowers is classified based on various feature data of the flowers. As data identifiers of the material data, "sepal length", "sepal width", "petal length", "petal width", "variety", etc. are used. The base model identifier may be selected from two types, support vector machine (Reference 6) and logistic regression (Reference 7). In the case of a support vector machine, the hyperparameters are the kernel type, regularization method, evaluation function, whether or not to solve the dual problem, algorithm termination condition, strictness of the soft margin, etc. In the case of logistic regression, the hyperparameters are the regularization method, strength of regularization, etc. In addition, in the learning data construction method, "sepal information" calculated by the formula ("sepal length" x "sepal width") and "petal information" calculated by the formula ("petal length" x "petal width") may be added as new feature identifiers. In the formula of the training data construction method, since "variety" is a categorical variable, label encoding may be specified in the formula. Since the material data in this problem is not time-series data, there is no need to specify series parameters in the training data construction method, and the process of creating a series column is not performed.

Reference 4: S. Hochreiter, J. Schmidhuber, "Long short-term memory", Neural Computation 9 (8), pp. 1735-1780, 1997.
Reference 5: J. Bradbury, et al, "Quasi-Recurrent Neural Networks", ICLP, 2016.
Reference 6: Ioannis Tsochantaridis, Thorsten Joachims, Thomas Hofmann, Yasemin Altun, "Large Margin Methods for Structured and Interdependent Output Variables", The Journal of Machine Learning Research 6 (9), pp. 1453-1484, 2005.
Reference 7: DR Cox, "The regression analysis of binary sequences (with discussion)", Journal of the Royal Statistical Society, Series B (Methodological), Vol. 20, No. 2, pp. 215-242, 1958.

In addition, the learning model selection process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. In this case, examples of the processor include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture such as an FPGA (Field-Programmable Gate Array), and a dedicated electric circuit that is a processor having a circuit configuration designed exclusively for executing a specific process such as an ASIC (Application Specific Integrated Circuit). In addition, the learning model selection process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, and a combination of a CPU and an FPGA, etc.). In addition, the hardware structure of these various processors is, more specifically, an electric circuit that combines circuit elements such as semiconductor elements.

In addition, in each of the above embodiments, a learning model selection program including a generation processing program is pre-stored (installed) in ROM 12 or storage 14, but this is not limited to the above. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

The following notes are further disclosed with respect to the above embodiments.

(Additional Note 1)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature quantity designated by a user to the prediction result are obtained;
and generating an index for selecting a predetermined learning model from among the plurality of learning models, based on the accuracy and the contribution degree.

(Additional Note 2)
A non-transitory recording medium storing a program executable by a computer to execute a generation process,
The generation process includes:
For each of a plurality of learning models machine-learned using a plurality of types of feature quantities, the accuracy of a prediction result by the learning model and the contribution of at least one type of feature quantity designated by a user to the prediction result are obtained;
generating an index for selecting a predetermined learning model from the plurality of learning models based on the accuracy and the contribution degree.

10, 110 Learning model selection device 11 CPU
12 ROM
13 RAM
14 Storage 15 Input unit 16 Display unit 17 Communication I/F
19 Bus 21, 121 Material data collection unit 22, 122 Material data storage unit 23, 123 Learning pattern transmission unit 24, 124 Evaluation data set creation unit 25, 125 Verification data creation unit 26, 126 Learning model creation unit 27, 127 Evaluation data set storage unit 28, 128 Learning model evaluation unit 29, 129 Acquisition unit 30, 130 Generation unit 31, 131 Selection unit 32, 132 Selected learning model storage unit 201 BEMS
202 People flow detection sensor

Claims (8)

A generation device for generating an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of feature amounts, the generation device comprising:
an acquisition unit that acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree to the prediction result of at least one type of feature value designated by a user;
A generation unit that generates the index from the accuracy and the contribution degree;
A generating device comprising: The generating device according to claim 1, wherein the generating unit generates the index using the contribution of each type of the feature to which a weight specified by a user is added. The generating device according to claim 2, wherein the generating unit generates the index using the contribution of each type of the feature quantity with a consistent size scale. The generating device according to any one of claims 1 to 3, wherein the generating unit generates the index by matching the scale of the accuracy and the scale of the contribution. The generating device according to any one of claims 1 to 4, wherein the generating unit generates the index by adding a weight to at least one of the accuracy and the contribution degree. The learning model is an environment model that predicts a change in temperature according to the temperature setting of an air conditioner, which is an action of a reinforcement learning agent, when an optimal control method for the temperature setting of an air conditioning controller is calculated by reinforcement learning with the goal of optimizing the temperature of an air conditioning utilization unit,
The generating device according to any one of claims 1 to 5, wherein the feature amount is a feature amount related to air conditioning control including a temperature of the air conditioning utilization unit and a temperature setting value of the air conditioner. A generation method executed by a generation device that includes an acquisition unit and a generation unit and generates an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of feature amounts, the method comprising:
the acquiring unit acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree of at least one type of feature value designated by a user to the prediction result;
The generating unit generates the index from the accuracy and the contribution degree. A generation program for generating an index for selecting a predetermined learning model from a plurality of learning models machine-learned using a plurality of types of features,
Computer,
an acquisition unit that acquires, for each of the plurality of learning models, an accuracy of a prediction result by the learning model and a contribution degree to the prediction result of at least one type of feature value designated by a user; and
A generating program for causing the index to function as a generating unit that generates the index from the accuracy and the contribution degree.