JP7702298B2 - Data classification system, data classification method, program, and method for evaluating machine learning models - Google Patents

Description

The present invention relates to a data classification system, a data classification method , a program, and a method for evaluating a machine learning model.

Technologies for classifying target data using machine learning models have been known for some time. Classification-type machine learning models include single-label models that classify target data into one of multiple labels, and multi-label models that simultaneously classify target data into multiple labels.

For example, Patent Documents 1 and 2 are known as techniques related to single-label models and multi-label models. Patent Document 1 describes determining a threshold and/or a scale factor to improve classification of a multi-label model. Patent Document 2 describes that in order to create training data for a multi-label model and a single-label model, for input data, one of a positive evaluation indicating that the content of the input data matches the label, a negative evaluation indicating that the content of the input data does not match the label, and a disregard evaluation indicating that the input data is to be excluded from the learning target labels is obtained for each label, and the training data is created, and when training a neural network for training, the learning device adjusts the weight coefficient of the intermediate layer so that the recognition score of the positive evaluation or negative evaluation label approaches the correct answer score of the positive evaluation or negative evaluation, and prevents the recognition score of the disregard evaluation label from affecting the adjustment of the weight coefficient of the intermediate layer.

Special table 2018-528521 publication International Publication No. 2017/073373

Figures 14 and 15 are diagrams for explaining the single label model. The single label model performs learning and prediction so that the sum of the outputs for each of multiple labels is 1. Therefore, as shown in Figure 14 (A), when the correct label among labels 1 to 4 is label 3, label 3 can be accurately output as the correct label.

However, as shown in FIG. 14(B), the single-label model cannot be applied to multi-labeling problems where multiple labels (e.g., labels 2 and 3) apply to the target data. In other words, the single-label model outputs label 3, which has the highest score, as the correct label, even though the correct labels are labels 2 and 3. Also, in the single-label model, the sum of the scores for each label is set to 1, so the scores for labels 2 and 3 are small.

As shown in Figure 15, in a single-label model, the more correct labels there are, the smaller the score of each correct label becomes. This makes it difficult to extract the correct label by comparing the score of each label output by the single-label model with a threshold value. Furthermore, since it is necessary to calculate the threshold value based on the difference between the score of the correct label output by the single-label model and the scores of the other labels, there is a problem in that it is not possible to train the model.

FIG. 16 is a diagram for explaining a multi-label model. The multi-label model performs learning and prediction so that the score for each label is in the range of 0 to 1. Therefore, even if the correct label is label 3 as in FIG. 16(A) or the correct labels are labels 2 and 3 as in FIG. 16(B), a high score can be output. Furthermore, even if a single threshold value is set without considering the number of correct labels, the multi-label model can output correct labels with high accuracy. However, it is common to use a single-label model as a prediction model for classifying target data such as documents. However, the single-label model has a problem in that it is difficult to extract multiple correct labels.

When it is unclear whether the target data has one or multiple correct labels, it is possible to verify both the prediction results of the single-label model and the prediction results of the multi-label model. However, there is a problem in that annotation work for the multi-label model and annotation work for the single-label model must be performed separately, which increases the workload. In addition, because the labels for the multi-label model and the single-label model are separate, there is a problem in that it is not possible to compare the prediction accuracy of the multi-label model with that of the single-label model.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a data classification system, a data classification method , a program, and a method for evaluating a machine learning model that can predict one correct label from multiple labels and predict multiple correct labels from multiple labels without increasing the burden of annotation work.

(1) One aspect of the present invention is a data classification system including: a multi-label storage unit that stores information for assigning multiple correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; a conversion unit that generates one new integrated label from multiple labels in first learning data for training a first machine learning model that outputs multiple correct labels when target data is input, the multiple labels having the flag value of the first value being one new integrated label, sets the flag of the integrated label to the first value, and sets flags of the multiple labels that are the basis of the integrated label to a second value, thereby converting the first learning data into second learning data including the integrated label; a learning unit that uses the second learning data converted by the conversion unit to train a second machine learning model that outputs one correct label when target data is input; a prediction unit that inputs target data to the second machine learning model and acquires a prediction result including the correct label output from the second machine learning model; and an inverse conversion unit that, when the correct label acquired by the prediction unit is the integrated label, converts the flags of the multiple labels that are the basis of the integrated label to the first value and deletes the integrated label from the prediction result .

(2) One aspect of the present invention includes a step in which a storage unit stores information for assigning multiple correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; and a learning device converts the first learning data, which is used to train a first machine learning model that outputs multiple correct labels when target data is input , into second learning data including the integrated label by generating a new integrated label from multiple labels in which the flag value is a first value, setting the flag of the integrated label to the first value, and setting flags of the multiple labels that are the basis of the integrated label to a second value. a step of the learning device using the converted second learning data to train a second machine learning model that outputs one correct label when target data is input; a step of the prediction device inputting the target data to the second machine learning model and acquiring a prediction result including the correct label output from the second machine learning model; and a step of the prediction device, if the acquired correct label is the integrated label, converting flags of multiple labels that are the basis of the integrated label to the first value and deleting the integrated label from the prediction result .

(3) One aspect of the present invention is a program that causes a computer to execute the following steps: storing information for assigning multiple correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; generating a new integrated label from multiple labels in the first learning data, the flag value of which is a first value, for training a first machine learning model that outputs multiple correct labels when target data is input, setting the flag of the integrated label to the first value, and setting flags of the multiple labels that are the basis of the integrated label to a second value, thereby converting the first learning data into second learning data that includes the integrated label; training a second machine learning model using the converted second learning data, the second learning data being a model that outputs one correct label when target data is input; inputting target data into the second machine learning model and acquiring a prediction result including the correct label output from the second machine learning model; and, if the acquired correct label is the integrated label, converting the flags of the multiple labels that are the basis of the integrated label to the first value and deleting the integrated label from the prediction result .

(4) One aspect of the present invention includes a step in which a storage unit stores information for assigning multiple correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal; a step in which a learning device converts the first learning data into second learning data including the integrated label by generating one new integrated label from multiple labels in which the flag value is a first value among the first learning data for training a first machine learning model that outputs multiple correct labels when target data is input, setting the flag of the integrated label to the first value, and setting flags of the multiple labels that are the basis of the integrated label to a second value; a step in which the learning device uses the converted second learning data to train a second machine learning model that outputs one correct label when target data is input; and a prediction device converts the converted second learning data into second learning data including the integrated label. a step of inputting target data into a first machine learning model trained using first training data to which a plurality of correct labels are assigned, and acquiring a prediction result of the first machine learning model; a step of inputting target data into a second machine learning model trained by the prediction device using second training data including an integrated label obtained by integrating a plurality of labels included in the first training data based on flags of the plurality of labels, and converting flags of the plurality of labels that are the basis of the integrated label to the first value when the correct label is the integrated label, and deleting the integrated label from the prediction result, and acquiring a prediction result of the second machine learning model ; and a step of evaluating the second machine learning model by the prediction device by comparing the prediction result of the first machine learning model with the prediction result of the second machine learning model.
This is a method for evaluating machine learning models, including:

According to one aspect of the present invention, it is possible to perform a process of predicting one correct label from multiple labels and a process of predicting multiple correct labels from multiple labels without increasing the burden of annotation work.

1 is a block diagram showing an example of a configuration of a data classification system according to an embodiment. 1A and 1B are block diagrams showing an example of a learning device 40 in an embodiment, in which (A) is a block diagram relating to a multi-label model, and (B) is a block diagram relating to a single-label model. 1A and 1B are block diagrams showing an example of a prediction device 70 in an embodiment, in which (A) is a block diagram relating to a multi-label model, and (B) is a block diagram relating to a single-label model. FIG. 1 is a diagram illustrating an example of training data of a multi-label model according to an embodiment. FIG. 13 is a diagram illustrating an example of training data (second training data) of a converted single-label model in an embodiment. 11 is a flowchart illustrating an example of a learning process according to an embodiment. 11 is a flowchart illustrating an example of a conversion process (step S108) of learning data according to an embodiment. 11 is a flowchart illustrating an example of a prediction process according to an embodiment. 13 is a flowchart illustrating an example of an inverse conversion process (step S306) of a prediction result according to an embodiment. 1A and 1B are diagrams for explaining an example of a conversion process and an inverse conversion process of an embodiment, in which (A) is an example of training data of a multi-label model, (B) is an example of training data after conversion processing, (C) is an example of a prediction result, and (D) is an example of a prediction result after inverse conversion processing. 1A and 1B are diagrams for explaining an example of a conversion process and an inverse conversion process of an embodiment, in which (A) is an example of training data of a multi-label model, (B) is an example of training data after conversion processing, (C) is an example of a prediction result, and (D) is an example of a prediction result after inverse conversion processing. 11 is a flowchart illustrating an example of a model evaluation process according to the embodiment. FIG. 13 is a diagram showing the relationship between the number of labels included in the learning data of the multi-label model, the score predicted by the single-label model, and the score predicted by the multi-label model. FIG. 1 is a diagram for explaining a single-label model. FIG. 1 is a diagram for explaining a single-label model. FIG. 1 is a diagram illustrating a multi-label model.

Below, the learning device and method, prediction device and method, program, and machine learning model evaluation method to which the present invention is applied will be described with reference to the drawings.

[Overview of the embodiment]
A data classification system 1 to which the present invention is applied, for example, trains a single-label model to learn which label data to be classified belongs to, and classifies information to which label it belongs using the trained single-label model. Data to be classified in the embodiment is, for example, text data, document data, voice data, or image data. Hereinafter, data to be classified is referred to as "target data." In the embodiment, a label is information indicating a correct answer that is assigned to the target data. The single-label model in the embodiment is a machine learning model that predicts that the target data will be classified into one label when the target data is input. The multi-label model in the embodiment is a machine learning model that predicts that the target data will be classified into multiple labels when the target data is input.

In particular, the data classification system 1 of the embodiment generates an integrated label by integrating multiple labels in training data for a multi-label model that has been annotated for training the multi-label model, and converts the generated data into training data for a single-label model that includes the integrated label as a single correct label. The training data for the single-label model is data that is pseudo-labeled with a single integrated label instead of multiple correct labels in the multi-label model. The data classification system 1 trains a single-label model using the converted training data for the single-label model. The data classification system 1 inputs target data into the trained single-label model, and when the correct label output from the single-label model is an integrated label, it converts the integrated label into multiple labels.

As a result, the data classification system 1 of the embodiment can create training data for a multi-label model in a single annotation operation, and can convert the training data to train a single-label model. Furthermore, the data classification system 1 can output prediction results equivalent to those of a multi-label model by converting the correct answer label output from a trained single-label model. As a result, the data classification system 1 can perform prediction processing using a single-label model and prediction processing using a multi-label model without increasing the burden of annotation work. The data classification system 1 of the embodiment will be described in detail below.

[Configuration of Data Classification System 1]
1 is a block diagram showing an example of the configuration of a data classification system in an embodiment. The data classification system 1 includes, for example, a request terminal 10, an annotation terminal 20, a multi-label storage unit 30, a learning device 40, a single-label storage unit 50, a model parameter storage unit 60, a prediction device 70, and a prediction result storage unit 80. The request terminal 10, the annotation terminal 20, the multi-label storage unit 30, the learning device 40, the single-label storage unit 50, the model parameter storage unit 60, the prediction device 70, and the prediction result storage unit 80 are connected to a network NW. Each device connected to the network NW includes a communication interface such as a network interface card (NIC) or a wireless communication module. The network NW includes, for example, a general-purpose Internet, a wide area network (WAN), a local area network (LAN), Wifi (registered trademark), a cellular network, and the like.

The requesting terminal 10 is a mobile phone such as a smartphone, a tablet terminal, a personal computer, etc. The requesting terminal 10 shares with the prediction device 70 information requesting target data and a correct label for the target data, for example, in accordance with a user's operation. The requesting terminal 10 receives information on the prediction result in response to the request from the prediction device 70, and presents the prediction result to the user.

The annotation terminal 20 is, for example, a personal computer, and is a terminal for performing annotation work. The annotation work is a work of assigning a correct answer label to data. The annotation terminal 20 associates data with a correct answer label according to the operation of the worker, and stores the data in the multi-label storage unit 30. The annotation in the embodiment performs multi-labeling that allows multiple correct answer labels to be assigned to one data. Note that in the embodiment, the data to which the label is assigned may be text data, document data, image data, audio data, or a combination of these data, and is not particularly limited.

The multi-label storage unit 30 includes a storage device such as a HDD, control software, etc. The multi-label storage unit 30 performs processes such as registering data, registering labels, and registering the correspondence between data and labels in response to requests from the annotation terminal 20. In this way, the multi-label storage unit 30 stores training data for a multi-label model in which multiple correct answer labels are assigned to one piece of data.

The learning device 40 is, for example, a computer that executes a learning process by executing a learning program using a CPU or the like. The learning device 40 converts the learning data of the multi-label model into learning data of the single-label model, and trains the single-label model using the training data of the converted single-label model. The learning device 40 stores the converted single-label model in the single-label storage unit 50. The learning device 40 trains the multi-label model using the training data of the multi-label model stored in the multi-label storage unit 30.

The single label storage unit 50 includes a storage device such as an HDD, control software, etc. In response to a request from the learning device 40, the single label storage unit 50 performs processes such as registering the learning data of the multi-label model stored in the multi-label storage unit 30, registering the integrated label, changing flags in the multi-label model, and registering the correspondence between data and the integrated label. In this way, the single label storage unit 50 stores the learning data of the single label model in which one piece of data is assigned an integrated label as one correct answer label.

The model parameter storage unit 60 includes a storage device such as a HDD, control software, etc. The model parameter storage unit 60 stores model parameters of the multi-label model and the single-label model as a result of learning by the learning device 40.

The prediction device 70 is a computer that executes a prediction process by, for example, executing a prediction program using a CPU or the like. The prediction device 70 inputs the target data supplied from the request terminal 10 to a multi-label model and obtains a prediction result based on information output from the multi-label model. The prediction result of the multi-label model may include multiple correct labels. The prediction device 70 inputs the target data supplied from the request terminal 10 to a single-label model and obtains a prediction result based on information output from the single-label model. When the correct label included in the single-label model is an integrated label, the prediction device 70 converts the integrated model into multiple correct labels and generates a prediction result including the converted multiple correct labels. The prediction result of the prediction device 70 is stored, for example, in the prediction result storage unit 80.

The prediction result storage unit 80 includes a storage device such as an HDD, control software, etc. The prediction results stored in the prediction result storage unit 80 are provided to, for example, the requesting terminal 10. The prediction results stored in the prediction result storage unit 80 may be used to compare the prediction results of the multi-label model and the prediction results of the single-label model.

FIG. 2 is a block diagram showing an example of a learning device 40 in an embodiment, where (A) is a block diagram related to a multi-label model, and (B) is a block diagram related to a single-label model. FIG. 3 is a block diagram showing an example of a prediction device 70 in an embodiment, where (A) is a block diagram related to a multi-label model, and (B) is a block diagram related to a single-label model.

As shown in FIG. 2A, the learning device 40 includes, for example, a learning processing unit 42 and a multi-label model 44. The multi-label model 44 is a machine learning model, for example, a convolutional neural network (CNN). The learning processing unit 42 acquires learning data of the multi-label model from the multi-label storage unit 30. FIG. 4 is a diagram showing an example of learning data of the multi-label model in the embodiment. The learning processing unit 42 inputs the learning data of the multi-label model to the multi-label model 44 and acquires a prediction result output from the multi-label model 44. The learning processing unit 42 recursively updates the model parameters of the multi-label model 44 so that the correct answer label output from the multi-label model 44 matches the correct answer label in the learning data. That is, the learning processing unit 42 updates the processing parameters related to the multi-label model 44 in the model parameter storage unit 60 and reflects the updated processing parameters in the multi-label model 44. The processing parameters are, for example, at least one of the number of layers, the number of nodes in each layer, the node connection method between each layer, the activation function, the error function, the gradient descent algorithm, the pooling domain, the kernel, the weighting coefficient, and the weighting matrix in the convolutional neural network. As a result, the learning processing unit 42 performs, for example, deep learning to obtain the processing parameters. Deep learning is machine learning that uses a multi-layered neural network, particularly a neural network with three or more layers.

As shown in FIG. 2(B), the learning device 40 includes, for example, a conversion unit 46, a learning processing unit 42, and a single-label model 48. The conversion unit 46 generates a single integrated label by integrating multiple correct answer labels in the learning data (first learning data) of a multi-label model in which multiple correct answer labels are assigned to data, and converts the data into learning data of a single-label model in which the integrated label is assigned as a single correct answer label for the data (second learning data). The conversion unit 46 stores the converted learning data of the single-label model in the single-label storage unit 50.

5 is a diagram showing an example of training data (second training data) of a converted single-label model in an embodiment. For example, as shown in FIG. 4, for one piece of data "Hello, ...", the flag of the label "I can't get through on the phone" and the flag of the label "Please put the person in charge" are both "1". In this case, as shown in FIG. 5, the conversion unit 46 generates an integrated label "I can't get through on the phone_Please put the person in charge" by integrating the two labels, sets the flag of the integrated model, and changes the flags of the two labels "I can't get through on the phone" and "Please put the person in charge" that were the source of the integrated label to "0". Similarly, for data "Repaired yesterday...", if the flag of "I can't get through on the phone", the flag of "I can't connect to the Internet", and the flag of "Complaint" are "1", the conversion unit 46 generates an integrated label "I can't get through on the phone_I can't connect to the Internet_Complaint", sets the flag of the integrated model, and changes the flag of the label that was the source of the integrated model to "0". Furthermore, if the flags of all the labels for the data "Is it recording...?" are "0", the conversion unit 46 generates an integrated label called "NoMatch" and sets the flag for the integrated model. Note that the label name of the integrated label is just an example, and other label names may be used as long as they correspond to the multiple labels that were the basis for the integrated label. Also, the integrated label name "NoMatch" is just an example, and other label names may be used as long as they are the label name that integrates all the label flags for the data that are "0".

The learning processing unit 42 acquires the learning data of the single-label model stored in the single-label storage unit 50. The learning processing unit 42 inputs the learning data of the single-label model to the single-label model 48 and acquires the prediction result output from the single-label model 48. The learning processing unit 42 recursively updates the model parameters of the single-label model 48 so that the correct label output from the single-label model 48 matches the correct label in the learning data. That is, the learning processing unit 42 updates the processing parameters related to the single-label model 48 in the model parameter storage unit 60 and reflects the updated processing parameters in the single-label model 48. The processing parameters of the single-label model 48 may be the same as those of the multi-label model 44.

As shown in FIG. 3(A), the prediction device 70 includes, for example, a prediction unit 72 and a prediction result output unit 76. The prediction unit 72 acquires model parameters of the multi-label model from the model parameter storage unit 60. The prediction unit 72 inputs target data, predicts a correct label for the target data, and stores the predicted label in the prediction result storage unit 80. The prediction result output unit 76 returns the correct label for the target data stored in the prediction result storage unit 80 as a prediction result.

As shown in FIG. 3(B), the prediction device 70 includes, for example, a prediction unit 72, an inverse conversion unit 74, and a prediction result output unit 76. The prediction unit 72 acquires model parameters of the single label model from the model parameter storage unit 60. The prediction unit 72 inputs target data, predicts a correct label for the target data, and outputs the predicted label to the inverse conversion unit 74. The inverse conversion unit 74 refers to the single label storage unit 50, and if the output correct label is an integrated label, converts the integrated label into multiple correct labels and stores them in the prediction result storage unit 80. The prediction result output unit 76 returns the correct label for the target data stored in the prediction result storage unit 80 as a prediction result.

[Learning process]
6 is a flowchart showing an example of a learning process according to an embodiment. The data classification system 1 performs an annotation process (step S100) and stores learning data of a multi-label model (step S102). For example, when the time for the learning process arrives, the data classification system 1 acquires learning data of the multi-label model (step S104) and determines the type of model to be learned (step S106).

When the type of model to be trained is a single-label model, the data classification system 1 converts the training data of the multi-label model into training data of the single-label model (step S108), and performs training processing of the single-label model using the converted training data of the single-label model (step S110). This allows the data classification system 1 to store the model parameters of the single-label model (step S112).

When the type of model to be trained is a multi-label model, the data classification system 1 performs a training process for the multi-label model using the acquired training data for the multi-label model (step S114). This allows the data classification system 1 to store model parameters for the multi-label model (step S116).

FIG. 7 is a flowchart showing an example of the conversion process of the learning data (step S108) of the embodiment. First, the learning device 40 selects one row in the learning data of the multi-label model (step S200). In the learning data of the multi-label model in the embodiment, data and label flags are associated with each other in one row, as shown in FIG. 4. Next, the learning device 40 determines whether there are two or more flagged labels (step S202). If there are two or more flagged labels (step S202: YES), the learning device 40 generates an integrated label by integrating the two or more labels (step S204). Next, the learning device 40 adds the generated integrated label to the column of the learning data (step S206), sets the flag of the integrated label to "1", and sets the flag of the original label that is the basis of the integrated label to "0" (step S208).

If there are two or more labels with flags set (step S202: NO), the learning device 40 determines whether the flags are all "0" (step S210). If the flags are not all "0" (step S210: NO), the learning device 40 proceeds to step S216. If the flags are all "0" (step S210: YES), the learning device 40 adds a NoMatch label to the string of learning data (step S212), sets the flag of the NoMatch label to "1", and sets the flags of other labels after the NoMatch label to "0" (step S214).

In step S214, the learning device 40 determines whether or not all rows of the learning data of the multi-label model have been processed. If not all rows have been processed (step S214: NO), the learning device 40 returns to step S200. If all rows have been processed (step S214: YES), the learning device 40 ends the processing of this flowchart.

[Prediction Processing]
8 is a flowchart showing an example of a prediction process according to an embodiment. The prediction device 70 acquires target data from the request terminal 10 (step S300), and determines the type of model to be predicted (step S302). The type of model to be predicted may be preset based on, for example, a user operation.

When the type of model to be predicted is a single-label model, the prediction device 70 inputs the target data into the single-label model and obtains the prediction result output from the single-label model (step S304). Next, the prediction device 70 inversely transforms the prediction result output from the single-label model (step S306). Next, the prediction device 70 stores the inversely transformed prediction result in the prediction result storage unit 80 (step S308). Next, the prediction device 70 responds with the stored prediction result (step S314).

When the type of model to be predicted is a multi-label model, the prediction device 70 inputs the target data into the multi-label model and obtains the prediction result output from the multi-label model (step S310). Next, the prediction device 70 stores the prediction result output from the multi-label model in the prediction result storage unit 80 (step S312). Next, the prediction device 70 responds with the stored prediction result (step S314).

Figure 9 is a flowchart showing an example of the inverse conversion process (step S306) of the prediction result of the embodiment. First, the prediction device 70 selects one row in the prediction result of the single label model (step S400). For example, when a request includes multiple target data, or when the data included in the request is divided into multiple target data according to a predetermined rule, the prediction result of the single label model will be data in which each row corresponds to the target data and a score for each label. Next, the prediction device 70 obtains the label with the highest score among the labels included in the prediction result as the prediction result (step S402).

Next, the prediction device 70 determines whether the label with the highest score is a NoMatch label (step S406). If the label with the highest score is a NoMatch label (step S406: YES), the prediction device 70 sets the flags of all labels included in the prediction result to "0" (step S408) and proceeds to step S416. If the label with the highest score is not a NoMatch label (step S406: NO), the prediction device 70 proceeds to step S410.

In step S410, the prediction device 70 determines whether the label with the highest score is an integrated label. If the label with the highest score is an integrated label (step S410: YES), the prediction device 70 sets the flag of the original label of the integrated label to "1" and sets the flags of the other labels of the original label to "0" (step S412). If the label with the highest score is not an integrated label (step S410: NO), the prediction device 70 sets the flag of the label with the highest score to "1" and sets the flags of the other labels of the label with the highest score to "0" (step S414).

In step S416, the prediction device 70 determines whether all rows in the prediction result have been processed. If not all rows have been processed (step S416: NO), the prediction device 70 returns to step S400. If all rows have been processed (step S416: YES), the prediction device 70 discards the integrated label sequence included in the prediction result (step S418) and ends the processing of this flowchart.

Figure 10 is a diagram for explaining an example of the conversion process and inverse conversion process of the embodiment, where (A) is an example of training data for a multi-label model, (B) is an example of training data after conversion process, (C) is an example of a prediction result, and (D) is an example of a prediction result after inverse conversion process.

It is assumed that the annotation terminal 20 generates learning data for a multi-label model as shown in FIG. 10(A) through annotation work. Before training the single-label model, the learning device 40 extracts multiple labels 1 to 3 with flags set to "1" from the learning data for the multi-label model, and generates a new label x by integrating the extracted labels 1 to 3. If the flags of multiple labels other than labels 1 to 3 in the learning data for the multi-label model are set to "1", the learning device 40 may integrate the multiple labels to generate a new label x+1. As shown in FIG. 10(B), the learning device 40 sets the flags of labels 1 to 3 that were the basis of the new label x generated by integration to "0", and creates learning data for a single-label model with the flag of label x set to "1".

The prediction device 70 uses a single label model to predict the single most likely label among multiple labels. The prediction result is output so that the sum of the scores (likelihood, probability) of each label is 1, as shown in FIG. 10(C). The prediction device 70 determines that the score of label x is the highest among all the scores of labels. Because label x is an integrated label, the prediction device 70 converts the prediction result into one in which the flags of labels 1 to 3 that were the basis of label x are set to "1", as shown in FIG. 10(D), and deletes the integrated label.

Figure 11 is a diagram for explaining an example of the conversion process and inverse conversion process of the embodiment, where (A) is an example of training data for a multi-label model, (B) is an example of training data after conversion process, (C) is an example of a prediction result, and (D) is an example of a prediction result after inverse conversion process.

It is assumed that the annotation terminal 20 generates learning data for a multi-label model as shown in FIG. 11(A) through annotation work. Before training the single-label model, all flags in the learning data for the multi-label model are set to "0", so the learning device 40 generates a new NoMatch label by integrating all labels. The learning device 40 creates learning data for a single-label model in which the flags of the generated NoMatch labels are set to "1", as shown in FIG. 11(B).

The prediction device 70 uses a single label model to predict the single most likely label among multiple labels. The prediction result is output so that the sum of the scores (likelihood, probability) of each label is 1, as shown in FIG. 11 (C). The prediction device 70 determines that the score of the NoMatch label is the highest among the scores of all labels. Because the NoMatch label is an integrated label, the prediction device 70 converts the flag of the label that was the source of the NoMatch label to "0" as shown in FIG. 11 (D), and deletes the NoMatch label.

[Model evaluation process]
The data classification system 1 may evaluate a single-label model trained using training data of the single-label model.
12 is a flowchart illustrating an example of a model evaluation process according to an embodiment. The prediction device 70 acquires evaluation data (step S300#). The evaluation data is data including target data and a correct label. Next, the prediction device 70 determines a model type (step S302).

When the model type is a single-label model, the prediction device 70 inputs evaluation data to the single-label model and obtains the prediction result output from the single-label model (step S304). Next, the prediction device 70 inversely transforms the prediction result output from the single-label model (step S306). Next, the prediction device 70 stores the inversely transformed prediction result in the prediction result storage unit 80 (step S308). When the model type is a multi-label model, the prediction device 70 inputs evaluation data to the multi-label model and obtains the prediction result output from the multi-label model (step S310). Next, the prediction device 70 stores the prediction result output from the multi-label model in the prediction result storage unit 80 (step S312).

Next, the prediction device 70 compares the prediction result of the single-label model with the correct label in the evaluation data, and compares the prediction result of the multi-label model with the correct label in the evaluation data (step S320). This allows the prediction device 70 to evaluate the single-label model based on the prediction accuracy of the multi-label model and the prediction accuracy of the single-label model (step S322). For example, the prediction device 70 can evaluate whether the single-label model has a prediction accuracy close to that of the multi-label model.

Figure 13 is a diagram showing the relationship between the number of labels included in the training data of the multi-label model, the score predicted by the single-label model, and the score predicted by the multi-label model. In Figure 13, the scores corresponding to the labels are shown in a line graph. According to Figure 13, even if the same evaluation data is input to both the multi-label model and the single-label model, there is no significant difference between the scores predicted by the multi-label model and the single-label model. Therefore, it can be seen that even if the training data of the multi-label model is converted and a single-label model is trained as in the embodiment, predictions can be made with the same accuracy as the multi-label model.

As described above, according to the learning device 40 of the embodiment, a learning device can be realized that includes a conversion unit 46 that converts first learning data for learning a first machine learning model (multi-label model) that outputs multiple correct labels when target data is input, into second learning data including an integrated label obtained by integrating multiple labels included in the first learning data based on the flags of the multiple labels, and a learning unit (learning processing unit 42, model parameter storage unit 60, and single label model 48) that uses the second learning data converted by the conversion unit 46 to learn a second machine learning model (single label model) that outputs one correct label when target data is input. In addition, according to the learning device 40, a learning method corresponding to the learning device and a program that executes processing corresponding to the learning method can be realized. According to the learning device 40 of the embodiment, it is possible to learn a single label model using learning data of a multi-label model. As a result, according to the learning device 40 of the embodiment, by performing annotation work on the learning data of the multi-label model, it is possible to perform the learning process of a single-label model that predicts one correct label from multiple labels and the learning process of a multi-label model that predicts multiple correct labels from multiple labels without increasing the burden of the annotation work.

According to the prediction device 70 of the embodiment, a prediction device can be realized that includes a prediction unit 72 that inputs target data into a second machine learning model (single-label model) trained using second learning data including an integrated label obtained by integrating multiple labels included in the first learning data based on the flags of the multiple labels, and a reverse conversion unit 74 that converts the flags of the multiple labels that are the basis of the integrated label and deletes the integrated label from the prediction result when the correct label acquired by the prediction unit 72 is an integrated label. In addition, according to the prediction device 70, a prediction method corresponding to the prediction device and a program that executes processing corresponding to the prediction method can be realized. According to the prediction device 70 of the embodiment, a single-label model can be trained using training data converted from the training data of the multi-label model, and the prediction result of the single-label model can be reverse converted to output the prediction result of the multi-label model. As a result, according to the prediction device 70 of the embodiment, by performing annotation work on the learning data of the multi-label model, it is possible to perform prediction processing of a single-label model that predicts one correct label from multiple labels and prediction processing of a multi-label model that predicts multiple correct labels from multiple labels, without increasing the burden of the annotation work.

According to the data classification system 1 of the embodiment, a method for evaluating a machine learning model can be realized, including the steps of inputting target data into a first machine learning model (multi-label model) trained using first training data in which multiple correct labels are assigned to data, and obtaining a prediction result of the first machine learning model, inputting target data into a second machine learning model (single-label model) trained using second training data including an integrated label in which multiple labels included in the first training data are integrated based on the flags of the multiple labels, converting the flags of the multiple labels that are the basis of the integrated label when the correct label is the integrated label, deleting the integrated label from the prediction result, and obtaining a prediction result of the second machine learning model, and evaluating the second machine learning model by comparing the prediction result of the first machine learning model with the prediction result of the second machine learning model. According to this data classification system 1, when training data for a multi-label model is generated by a single annotation operation and training is performed for a multi-label model and a single-label model, the prediction result of the single-label model can be evaluated.

Note that although each embodiment and each modified example have been described, these are merely examples and are not intended to be limiting. For example, any of the embodiments and modified examples, or a part of each embodiment or a part of each modified example, may be combined with one or more other embodiments or one or more other modified examples to realize one aspect of the present invention.

In addition, the programs for executing the processes of the user terminal device 100, the search server device 200, and the batch server device 300 in this embodiment may be recorded on a computer-readable recording medium, and the programs recorded on the recording medium may be read into a computer system and executed to perform the various processes described above related to the user terminal device 100, the search server device 200, and the batch server device 300.

Note that the term "computer system" here may include hardware such as the OS and peripheral devices. In addition, if a WWW system is used, the term "computer system" also includes the homepage provision environment (or display environment). Furthermore, "computer-readable recording media" refers to storage devices such as flexible disks, magneto-optical disks, ROMs, writable non-volatile memories such as flash memory, portable media such as CD-ROMs, and hard disks built into computer systems.

Furthermore, the term "computer-readable recording medium" refers to a volatile memory (e.g., DRAM (Dynamic Random Access Memory)) within a computer system that is a server or a client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.
This also includes those that hold a program for a certain period of time, such as a random access memory (RDMA). The program may be transmitted from a computer system in which the program is stored in a storage device to another computer system via a transmission medium, or by a transmission wave in the transmission medium.

Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system, a so-called differential file (differential program).

The above describes an embodiment of the present invention in detail with reference to the drawings, but the specific configuration is not limited to this embodiment, and includes designs that do not deviate from the gist of the invention.

REFERENCE SIGNS LIST 10 Request terminal 20 Annotation terminal 30 Multi-label storage unit 40 Learning device 42 Learning processing unit 44 Multi-label model 46 Conversion unit 48 Single-label model 50 Single-label storage unit 60 Model parameter storage unit 70 Prediction device 72 Prediction unit 74 Inverse conversion unit 76 Prediction result output unit 80 Prediction result storage unit

Claims (4)

a multi-label storage unit that stores information for assigning a plurality of correct labels and a flag having a first value to one piece of learning data acquired from an annotation terminal;
a conversion unit that generates one new integrated label from a plurality of labels having the flag value of a first value in first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is input, sets a flag of the integrated label to the first value, and sets flags of the plurality of labels that are the basis of the integrated label to a second value, thereby converting the first learning data into second learning data including the integrated label;
a learning unit that uses the second learning data converted by the conversion unit to train a second machine learning model that outputs one correct label when target data is input;
A prediction unit that inputs target data to the second machine learning model and obtains a prediction result including a correct answer label output from the second machine learning model;
an inverse conversion unit that converts flags of a plurality of labels that are the source of the integrated label to the first value when the correct label acquired by the prediction unit is the integrated label, and deletes the integrated label from the prediction result;
A data classification system comprising: A step in which a storage unit stores information for assigning a plurality of correct labels and a flag having a first value to one piece of learning data acquired from the annotation terminal;
a step of converting the data into second learning data including the integrated label by generating a new integrated label from a plurality of labels having a first value of the flag in first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is inputted, setting a flag of the integrated label to the first value, and setting flags of the plurality of labels that are the basis of the integrated label to a second value;
A step in which the learning device uses the converted second learning data to train a second machine learning model that outputs one correct label when target data is input;
A step in which the prediction device inputs target data to the second machine learning model and obtains a prediction result including a correct answer label output from the second machine learning model;
a step of converting flags of a plurality of labels that are the source of the integrated label to the first value when the obtained correct label is the integrated label, and deleting the integrated label from the prediction result, by the prediction device;
Data classification methods , including: On the computer,
storing information for assigning a plurality of correct labels and a flag having a first value to one piece of learning data acquired from the annotation terminal;
a step of generating one new integrated label from a plurality of labels having the flag value of a first value among first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is input, setting a flag of the integrated label to the first value, and setting flags of the plurality of labels that are the basis of the integrated label to a second value, thereby converting the data into second learning data including the integrated label;
A step of training a second machine learning model that outputs one correct label when target data is input using the converted second training data;
inputting target data into the second machine learning model and obtaining a prediction result including a correct answer label output from the second machine learning model;
converting flags of a plurality of labels that are the basis of the integrated label to the first value when the acquired correct label is the integrated label, and deleting the integrated label from the prediction result;
A program to execute. a step of storing information for assigning a plurality of correct labels and a flag having a first value to one piece of learning data acquired from the annotation terminal, in a storage unit;
a step of converting the data into second learning data including the integrated label by generating a new integrated label from a plurality of labels having a first value of the flag in first learning data for training a first machine learning model that outputs a plurality of correct labels when target data is inputted, setting a flag of the integrated label to the first value, and setting flags of the plurality of labels that are the basis of the integrated label to a second value;
A step in which the learning device uses the converted second learning data to train a second machine learning model that outputs one correct label when target data is input;
A step in which a prediction device inputs target data to a first machine learning model trained using first training data in which a plurality of correct answer labels are assigned to the data, and obtains a prediction result of the first machine learning model;
a step of inputting target data to a second machine learning model trained using second learning data including an integrated label obtained by integrating multiple labels included in the first learning data based on flags of the multiple labels, converting flags of the multiple labels that are the basis of the integrated label to the first value when the correct label is the integrated label, and deleting the integrated label from the prediction result, thereby acquiring a prediction result of the second machine learning model;
The prediction device evaluates the second machine learning model by comparing a prediction result of the first machine learning model with a prediction result of the second machine learning model;
How to evaluate machine learning models, including:

Images