JP7806883B2 - Model search device, model search method, and program - Google Patents

Description

This disclosure relates to the exploration of machine learning models.

In recent years, predictive models obtained through machine learning have been used in a variety of fields. To make useful predictions in a given environment, it is necessary to select an appropriate model depending on the target task and the data being used. Patent Document 1 describes an example in which a user sets the details of a newly created model.

International Publication WO2020/085114

In Patent Document 1, the user must enter information for each setting item independently, which may make it difficult for inexperienced users to select and enter detailed information.

One objective of the present disclosure is to provide a model search device that allows users to easily search for models to achieve a desired task.

In one aspect of the present disclosure, a model exploration device includes:
a task list output means for outputting a task list showing candidate tasks;
a data list output means for outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, by referring to correspondence information indicating correspondence between tasks, data, and trained models ;
a model list output means for outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, by referring to the correspondence information ;
a model output means for outputting the trained model selected by the user from the output model list ;
Equipped with.

In another aspect of the present disclosure, a computer-implemented model exploration method includes:
Output a task list showing candidate tasks,
outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models;
outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information ;
The trained model selected by the user from the output model list is output.

In yet another aspect of the invention, a program includes:
Output a task list showing candidate tasks,
outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models;
outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information ;
The computer is caused to execute a process of outputting the trained model selected by the user from the output model list .

This disclosure enables users to easily search for models to achieve their desired tasks.

1 is a block diagram showing the overall configuration of a model management system according to a first embodiment. FIG. FIG. 2 is a block diagram showing the hardware configuration of the model search device according to the first embodiment. FIG. 2 is a block diagram showing the functional configuration of the model search device according to the first embodiment. An example of the correspondence information is shown below. An example of a model search screen is shown. 10 shows another example of the model search screen. 10 is a flowchart of a model search process performed by the model search device. FIG. 1 is a block diagram showing a schematic configuration of a model management system using a server and a terminal device. FIG. 10 is a block diagram showing the functional configuration of a model search device according to a second embodiment. 10 is a flowchart of a process performed by a model search device according to a second embodiment.

Hereinafter, preferred embodiments of the present disclosure will be described with reference to the drawings.
First Embodiment
[Overall configuration]
1 is a block diagram showing the overall configuration of a model management system according to a first embodiment. The model management system 1 includes a model exploration device 100, a display device 2, and an input device 3. The model exploration device 100 is configured by a computer such as a personal computer (PC). The display device 2 is, for example, a liquid crystal display device, and displays a model search screen generated by the model exploration device 100. The input device 3 is, for example, a mouse, a keyboard, etc., and is used by a user to make inputs and selections required when searching for a model.

First, an overview of the operation of the model search device 100 will be provided. The model search device 100 stores various machine learning models. As an example, these machine learning models include models that have already been trained. As another example, these machine learning models may include models that have not been trained. Furthermore, these machine learning models may include both trained and untrained models. Note that untrained models include, for example, untrained models for which hyperparameters have been set, and models for which parameters have been trained to a certain extent.

The model search device 100 also stores various data (hereinafter referred to as "actual data") acquired in a real environment. A user uses the model search device 100 to search for a model suitable for a target task. Specifically, the model search device 100 first presents the user with a list of candidate tasks (hereinafter referred to as a "task list"). When the user selects a target task from the task list, the model search device 100 presents the user with a list of candidate data to be used for the task (hereinafter referred to as a "data list"). When the user selects data from the data list, the model search device 100 presents the user with a list of candidate models suitable for the selected data (hereinafter referred to as a "model list"). The user can determine a model suitable for the target task by selecting an appropriate model from the presented model list. The user can then obtain data related to the determined model from the model management system and use it for actual operation.

A machine learning model is information that represents the relationship between explanatory variables and a response variable. A machine learning model is, for example, a component that estimates the results of an estimation target by calculating a response variable based on explanatory variables. A machine learning model is generated by executing a learning algorithm using training data, for which the response variable value has already been obtained, and arbitrary parameters as input. A machine learning model may be represented, for example, by a function c that maps an input x to a correct answer y. A machine learning model may estimate the numerical value of an estimation target, or may estimate the label of an estimation target. A machine learning model may output a variable that describes the probability distribution of the response variable. A machine learning model may also be referred to as a "learning model," "analysis model," "AI (Artificial Intelligence) model," "trained model," "inference model," or "prediction formula." Here, an explanatory variable is a variable used as input in a machine learning model. An explanatory variable may also be referred to as a "feature" or "characteristic."

Furthermore, a task refers to the type of prediction or inference that a machine learning model makes based on a problem or question and available data. For example, a classification task refers to assigning data to categories, and a clustering task refers to grouping data according to similarity. Specific examples of tasks in this embodiment are described below.

[Hardware configuration]
2 is a block diagram showing the hardware configuration of the model search device 100. As shown in the figure, the model search device 100 includes an interface (I/F) 111, a processor 112, a memory 113, a recording medium 114, and a database (DB) 115.

I/F 111 inputs and outputs data to and from external devices. Specifically, instructions and selections entered by the user using the input device 3 are input to the model exploration device 100 via I/F 111. In addition, data for the model selection screen, including a list of tasks, data, and models generated by the model exploration device 100, is output to the display device 2 via I/F 111.

The processor 112 is a computer such as a CPU (Central Processing Unit) that controls the entire model search device 100 by executing a pre-prepared program. The processor 112 may also be a GPU (Graphics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, or an FPGA (Field-Programmable Gate Array). The processor 112 executes the model search process described below.

Memory 113 is composed of ROM (Read Only Memory), RAM (Random Access Memory), etc. Memory 113 is also used as working memory while processor 112 is executing various processes.

The recording medium 114 is a non-volatile, non-transitory recording medium such as a disk-shaped recording medium or semiconductor memory, and is configured to be detachable from the model search device 100. The recording medium 114 records various programs executed by the processor 112. When the model search device 100 executes various processes, the programs recorded on the recording medium 114 are loaded into the memory 113 and executed by the processor 112.

DB115 stores pre-prepared models, various real-world data obtained in real-world environments, etc. DB115 may also store training data used to train each learned model, evaluation data used for evaluation, etc.

(Functional configuration)
3 is a block diagram showing the functional configuration of the model search device 100 according to the first embodiment. The model search device 100 functionally includes a data storage unit 121, a model storage unit 122, and a model search unit 123.

The data storage unit 121 stores various types of real data obtained in the real environment. Specifically, real data includes various types of data such as image data, natural language data, table data, and time-series data. The data storage unit 121 may also store so-called open data, as well as training models and evaluation models used to learn trained models. The data storage unit 121 also stores attribute information for each piece of stored data. The attribute information includes, for example, information such as the type and structure of the data, the explanatory variables that the data has, and the date and time the data was acquired.

The model storage unit 122 stores machine learning models that have been trained using various algorithms. For example, the model storage unit 122 stores machine learning models such as decision tree models, piecewise linear models using FAB inference (Factorized Asymptotic Bayesian Inference), and neural networks. Note that techniques for piecewise linear models using FAB inference are disclosed, for example, in U.S. Patent Publication US 2014/0222741 A1.

The model storage unit 122 also stores correspondence information that indicates the correspondence between tasks, data, and models. The correspondence information indicates the correspondence between tasks, data, and models, such as, for example, that when executing task A, data B is used, and model C is the model for processing data B.

Figure 4 shows an example of correspondence information. In this example, multiple tasks, data, and models are prepared, and some of the correspondences between them are indicated by dotted lines. For example, it is shown that image data can be used for a face recognition task, which belongs to the classification task, and that in this case the image data can be processed using a neural network model. It is also shown that time series data can be used for a regression task, and that in this case the time series data can be processed using a decision tree model. In this way, the model storage unit 122 pre-stores information on the data used by each model and the tasks that each model can execute for the stored models.

The model storage unit 122 also stores attribute information for each model. Attribute information includes, for example, information on related models related to each model, and information on the creation date and time or update date and time of each model. Information on related models includes, for example, information on whether the model is the original model or an updated model, and version information for the same model.

The model search unit 123 searches for models stored in the model storage unit 122 based on user input. By inputting the target task and usage data, the user searches for an appropriate model for achieving the target task using the usage data. Specifically, the model search unit 123 generates a model search screen and displays it on the display device 2. Then, based on the input and selection made by the user while viewing the model search screen, the model search unit 123 refers to the data storage unit 121 and the model storage unit 122 and presents candidate models appropriate for the user's input. The model search unit 123 refers to the correspondence information exemplified in Figure 4, extracts models appropriate for the task and data selected by the user, and displays them as a model list on the model search screen.

Figure 5 shows an example of a model search screen. The model search screen 40 has a task column 41, a data column 42, and a model column 43. The model search screen 40 is designed to accept user selections in the order of task, data, and model. Note that in the initial state, a default list is displayed in the task column 41, but the data column 42 and model column 43 are blank.

Note that the display order of the model search screen 40 is not limited to the example shown in Figure 5. For example, the model search screen 40 may be designed to first display the data column 42 or the model column 43 and then accept the user's selection.

The task column 41 displays a list of multiple tasks that can be achieved using the machine learning model. Each displayed task corresponds to a candidate task. In the example of Figure 5, the major categories of tasks are "classification," "regression," "sign detection," and "object recognition," and each category has its own subcategory of tasks.

In addition, a sort button 41a is displayed in the task column 41. The sort button 41a is a button for sorting the display order of the tasks displayed in the task column 41. The user can change the display order of the tasks in the task column 41 by operating the sort button 41a and selecting a display order such as alphabetical order or hiragana order.

In addition, a narrowing down button 41b is displayed in the task column 41. The narrowing down button 41b is a button for narrowing down the tasks displayed in the task column 41 to those in a specific category. The user can narrow down the tasks displayed in the task column 41 to those in a specific major category by operating the narrowing down button 41b and selecting one of the tasks in the major categories. For example, if the user operates the narrowing down button 41b to select "regression," only "regression" tasks will be displayed in the task column 41.

An OK button 41c is also displayed in the task column 41. The OK button 41c is a button for confirming the user's selection in the task column 41. The user selects the desired task from the displayed list of tasks by, for example, moving the cursor C. In the example of Figure 5, the task "Demand Forecast" currently selected by the user is underlined. The user confirms the task selection by pressing the OK button 41c.

When the user selects the desired task from the task column 41 and presses the OK button 41c to confirm the selection, a list of data is displayed in the data column 42. The list of data displayed here is a list of data corresponding to the task selected by the user in the task column 41. Specifically, the model search unit 123 references the correspondence information described above to identify the type of data corresponding to the task selected by the user, extracts data that falls into that type and is actually stored in the data storage unit 121, and creates a data list. This allows the user to search for a model that uses data that is actually registered in the data storage unit 121. In the example of Figure 5, the user selected the task "demand forecasting" in the task column 41, so a list of data used in demand forecasting is displayed in the data column 42. Each piece of data listed in the data column 42 corresponds to a data candidate.

A sort button 42a and a narrow down button 42b are also displayed in the data column 42. By operating the sort button 42a, the user can change the display order of the data displayed in the data column 42. For example, the user can change the display order of the data to order by date and time, newest or oldest data, or type of data.

The user can also narrow down the data displayed in the data column 42 by operating the narrow down button 42b. For example, the user can narrow down the data by specifying the data period. Also, if different types of data are displayed, such as image data, table data, and time series data, the user can narrow down the data by specifying the type of data.

Furthermore, the data column 42 can display attribute information prepared for each data. For example, as shown in the example of Figure 5, when a user selects data "Data202104," the attribute information 42d for that data is displayed as additional information in the form of a speech bubble or the like. This allows the user to check the attribute information of each data before selecting data.

The data column 42 also has an OK button 42c, and after the user selects data in the list using the cursor C or the like, they confirm the selection by pressing the OK button 42c.

When the user selects data from the data column 42 and presses the OK button 42c to confirm the selection, a list of models is displayed in the model column 43. The list of models displayed here is a list of models that use the data selected by the user in the data column 42. Here, as one example, a model that uses the data selected by the user includes a model that is applicable to the data selected by the user. As another example, a model that uses the data selected by the user may include a model that has been trained with the data selected by the user. Furthermore, a model that uses the data selected by the user may include both a model that is applicable to the data selected by the user and a model that has been trained with the data selected by the user.

Specifically, the model search unit 123 references the correspondence information described above, extracts models that correspond to the data selected by the user and are actually stored in the model storage unit 122, and creates a list of models. This allows the user to search for models that are actually registered in the model storage unit 122, i.e., models that can be used immediately. In the example of Figure 5, the user has selected the data "Data202104" in the data column 42, and a list of models that use that data is displayed in the model column 43. Each model listed in the model column 43 corresponds to a candidate model.

The model column 43 also displays a sort button 43a and a narrow down button 43b. By operating the sort button 43a, the user can change the display order of the models displayed in the model column 43. For example, the user can change the display order of the models to the order of newest to oldest models (by version or update date). In addition, the model storage unit 122 stores the attributes of each model as attribute information, specifically, characteristics such as accuracy, interpretability, fairness, and operating costs. Therefore, the user can sort the models based on these attributes.

The user can also narrow down the models displayed in the model column 43 by operating the narrow down button 43b. For example, the user can narrow down the models by specifying whether the model is new or old (version or update date). As mentioned above, the user can also narrow down the models based on attributes such as accuracy, interpretability, fairness, and operational cost. Here, "accuracy" refers to the probability that the model output is correct. "Interpretability" refers to the ability to explain the basis for predictions and inferences made by the model, and to the ability of humans to understand them. "Fairness" refers to the absence of bias in predictions and inferences made by the model based on the characteristics of the target. "Operational cost" refers to the cost required when actually using the model for operational purposes.

Furthermore, the model column 43 can display attribute information prepared for each model. For example, when a user selects the model "LGBTM" as in the example of Figure 5, the model's attribute information 43d is displayed as additional information in the form of a speech bubble or the like. This allows the user to select a model after checking the attribute information of each model. When a user selects a model, related models of that model may also be displayed as additional information. Figure 6 shows an example in which additional information 43e is displayed indicating related models for the model "LGBTM" selected by the user. Related models include models that have undergone backward compatibility adjustments for the model selected by the user, as well as newer and older versions of that model.

The model column 43 also has an OK button 43c, and the user can select a model in the list using cursor C or the like, and then confirm the selection by pressing the OK button 43c.

In the above example, additional information is displayed when the user selects data or a model using cursor C, and the selection is confirmed when the user presses OK button 42c or 43c. Alternatively, the OK button may not be displayed on the model search screen, and additional information may be displayed when the user places cursor C over the text of the data or model (i.e., mouse-overs), and the selection may be confirmed when the user selects the data or model.

As such, with the model search screen of this embodiment, the user selects the desired task and then the data to be used, and a list of candidate data and models corresponding to the user's selection is displayed. This allows the user to efficiently search for a model suitable for the desired task. Furthermore, by selecting a task or data, candidate models suitable for those are displayed in a list, so even users with little knowledge of models can easily search for a model suitable for the desired task and the actual data they have already acquired.

[Model Search Process]
Next, a description will be given of the model search process executed by the model search device 100. Fig. 7 is a flowchart of the model search process. This process is realized by the processor 112 shown in Fig. 2 executing a program prepared in advance and operating mainly as the model search unit 123.

First, the model search unit 123 displays the initial screen of the model search screen on the display device 2 (step S10). On the initial screen, the model search unit 123 displays a default task list in the task column 41, and leaves the data column 42 and model column 43 blank.

Next, the model search unit 123 determines whether the user has selected a task (step S11). In this case, the selection of a task refers to a selection confirmed by the user, for example, by pressing the OK button 41c. If no task has been selected (step S11: No), the model search unit 123 waits for a selection by the user. On the other hand, if a task has been selected (step S11: Yes), the model search unit 123 refers to the correspondence information and data storage unit 121 described above, and displays a list of data corresponding to the selected task on the model search screen (step S12).

Next, the model search unit 123 determines whether the user has selected data (step S13). Note that in this case, data selection refers to a selection confirmed by the user by pressing the OK button 42c, for example. If no data has been selected (step S13: No), the model search unit 123 waits for a selection by the user. On the other hand, if data has been selected (step S13: Yes), the model search unit 123 refers to the correspondence information and model storage unit 122 described above, and displays a list of models corresponding to the selected data on the model search screen (step S14).

Next, the model search unit 123 determines whether the user has selected a model (step S15). Note that in this case, the selection of a model refers to a selection confirmed by the user, for example, by pressing the OK button 43c. If no model has been selected (step S15: No), the model search unit 123 waits for the user to make a selection. On the other hand, if a model has been selected (step S15: Yes), the model search unit 123 determines the model selected by the user as the search result (step S16) and terminates the processing.

The model search unit 123 accepts changes to the task and data until the user finalizes the model selection. This allows the user to change the task and data selection to check various patterns and find the optimal model. The user can also prepare the data and model to be used for the desired task by obtaining the data to be used for the determined model from the data storage unit 121 and obtaining the determined model from the model storage unit 122.

[Modification]
In the above embodiment, the model search device 100 is configured as an independent device such as a PC. However, instead, the model generation device may be configured by a server and a terminal device. FIG. 8 is a block diagram showing a schematic configuration of a model management system 1x using a server and a terminal device. In FIG. 8, the server 100x has the configuration of the model search device 100 shown in FIG. 3. Furthermore, the display device 2x and input device 3x of the terminal device 7 used by the user are used as the display device 2 and input device 3 shown in FIG. 3.

Second Embodiment
9 is a block diagram showing the functional configuration of a model search device 70 according to the second embodiment. The model search device 70 includes a task list output unit 71, a data list output unit 72, a model list output unit 73, and a model output unit 74.

Figure 10 is a flowchart of processing by the model search device 70 of the second embodiment. The task list output means 71 outputs a task list indicating candidate tasks (step S71). The data list output means 72 receives a selection of tasks included in the task list and outputs a data list indicating candidate data to be used in the selected tasks (step S72). The model list output means 73 receives a selection of data included in the data list and outputs a model list indicating candidate models that use the selected data (step S73). The model output means 74 receives a selection of models included in the model list and outputs the selected models (step S74).

The model search device 70 of the second embodiment enables a user to easily search for a model to achieve a desired task.

Some or all of the above embodiments may also be described as, but are not limited to, the following notes:

(Appendix 1)
a task list output means for outputting a task list showing candidate tasks;
a data list output means for receiving a selection of a task included in the task list and outputting a data list showing candidates of data to be used in the selected task;
a model list output means for receiving a selection of data included in the data list and outputting a model list showing candidate models that use the selected data;
a model output means for receiving a selection of a model included in the model list and outputting the selected model;
A model search device comprising:

(Appendix 2)
2. The model search device according to claim 1, wherein the data list output means rearranges the order of the data in the data list or narrows down the data in the data list based on an instruction from a user.

(Appendix 3)
3. The model search device according to claim 1, wherein the data list includes attribute information relating to the data included in the data list.

(Appendix 4)
4. The model search device according to claim 1, wherein the model list output means rearranges the order of models in the model list or narrows down the models in the model list based on an instruction from a user.

(Appendix 5)
5. The model search device according to claim 4, wherein the model list output means performs the sorting or the narrowing down based on at least one of accuracy, interpretability, fairness, and operation cost of the model.

(Appendix 6)
6. The model search device according to claim 1, wherein the model list includes attribute information about the models included in the model list.

(Appendix 7)
7. The model search device according to claim 1, wherein the model list includes related models that are related to the models included in the model list.

(Appendix 8)
8. The model search device according to claim 1, wherein the data list includes actual data stored in a data storage unit.

(Appendix 9)
Output a task list showing candidate tasks,
receiving a selection of a task included in the task list, and outputting a data list indicating candidates of data to be used in the selected task;
receiving a selection of data included in the data list, and outputting a model list indicating candidate models that use the selected data;
A model search method that receives a selection of a model to be included in the model list and outputs the selected model.

(Appendix 10)
Output a task list showing candidate tasks,
receiving a selection of a task included in the task list, and outputting a data list indicating candidates of data to be used in the selected task;
receiving a selection of data included in the data list, and outputting a model list indicating candidate models that use the selected data;
A recording medium having recorded thereon a program for causing a computer to execute a process of receiving a selection of a model included in the model list and outputting the selected model.

The present disclosure has been described above with reference to embodiments and examples, but the present disclosure is not limited to the above embodiments and examples. Various modifications that would be understood by a person skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure.

1, 1x Model management system 2, 2x Display device 3, 3x Input device 7 Terminal device 100 Model search device 112 Processor 121 Data storage unit 122 Model storage unit 123 Model search unit

Claims (11)

a task list output means for outputting a task list showing candidate tasks;
a data list output means for outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, by referring to correspondence information indicating correspondence between tasks, data, and trained models ;
a model list output means for outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, by referring to the correspondence information ;
a model output means for outputting the trained model selected by the user from the output model list ;
A model search device comprising: 2. The model search device according to claim 1, wherein the data list output means rearranges the order of the data in the data list or narrows down the data in the data list based on an instruction from the user. A model search device according to claim 1 or 2, wherein the data list includes attribute information relating to the data included in the data list. 4. The model search device according to claim 1, wherein the model list output means rearranges the order of trained models in the model list or narrows down the trained models in the model list based on an instruction from the user. The model search device according to claim 4 , wherein the model list output means performs the sorting or the narrowing down based on at least one of accuracy, interpretability, fairness, and operation cost of the trained models. The model search device according to claim 1 , wherein the model list includes attribute information about trained models included in the model list. The model search device according to claim 1 , wherein the model list includes related models that are related to the trained models included in the model list. A model search device according to any one of claims 1 to 7, wherein the data list includes actual data stored in a data storage unit. 9. The model search device according to claim 1, wherein the task list output means outputs major classifications of tasks and tasks as minor classifications corresponding to the major classifications. 1. A computer-implemented method for model exploration, comprising:
Output a task list showing candidate tasks,
outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models;
outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information ;
A model search method that outputs the trained model selected by the user from the output model list . Output a task list showing candidate tasks,
outputting a data list indicating candidates of the data associated with a task selected by a user from the output task list, with reference to correspondence information indicating correspondence between tasks, data, and trained models;
outputting a model list indicating candidates for the trained model associated with the data selected by the user from the output data list, with reference to the correspondence information ;
A program that causes a computer to execute a process of outputting the trained model selected by the user from the output model list .