JP7763643B2 - Learning device, label estimation device and program - Google Patents

Description

The present invention relates to a learning device, a label estimation device, and a program.

For reasons such as making searches easier, it may be desirable to assign labels related to news articles. For example, if the news is about a company whose stock prices have fluctuated due to the impact of an infectious disease, terms such as infectious disease, stock price, and business may be assigned as labels.

JP 2019-53730 A

Grigorios, Tsoumakas, Ioannis Katakis, “Multi-Label Classification: An Overview” Ankit Pal, Muru Selvakumar and Malaikannan Sankarasubbu, “MAGNET: Multi-Label Text Classification using Attention-based Graph Neural Network” arXiv:2003.11644v1 Ashutosh Adhikari, Achyudh Ram, Raphael Tang, and Jimmy Lin, “Rethinking Complex Neural Network Architectures for Document Classification” Proceedings of NAACL-HLT 2019, pages 4046-4051

In some cases, it is sufficient to assign one label to a sentence, but since sentences often consist of multiple words, there are cases where a single label is insufficient. In other words, as shown in the example above, it may be desirable to assign multiple labels to a single sentence. However, the more labels that need to be assigned to a sentence, the greater the effort required to estimate the labels.

In light of the above circumstances, the present invention aims to provide technology that reduces the increase in labor required for label estimation work.

One aspect of the present invention is a learning device that includes a model learning unit that updates a label estimation model, which is a mathematical model that estimates a label to be assigned to a sentence indicated by input sentence information, using a machine learning method that uses model training data that includes sentence information indicating a sentence and label appropriateness information indicating the degree to which a plurality of labels, predetermined as candidate labels to be assigned to the sentence, are appropriate as labels for the sentence. The label appropriateness information is information obtained based on true/false information indicating a label that satisfies a predetermined condition regarding the likelihood of being assigned to the sentence indicated by the model training data, and label co-occurrence information, which is information indicating the probability of co-occurrence between any two of a plurality of labels predetermined as candidate labels to be assigned to the sentence.

One aspect of the present invention is a method for processing a document, comprising: an object acquisition unit that acquires object information that indicates a sentence to be processed;
The label estimation device includes: a model learning unit that updates a label estimation model, which is a mathematical model that estimates a label to be assigned to a sentence indicated by input sentence information, by a machine learning method using model learning data including sentence information that indicates a sentence and label appropriateness information that indicates the degree to which a plurality of labels, which are predetermined as candidate labels to be assigned to the sentence, are appropriate as labels for the sentence; and an estimation unit that estimates a label to be assigned to a sentence indicated by target information acquired by the sentence acquisition unit, using the trained label estimation model obtained by a learning device, which is information obtained based on true/false information that indicates a label that satisfies a predetermined condition regarding the likelihood of being assigned to the sentence indicated by the model learning data, and label co-occurrence information that is information that indicates the probability of co-occurrence between any two of a plurality of labels, which are predetermined as candidate labels to be assigned to the sentence.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned learning device.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned label estimation device.

This invention makes it possible to provide technology that reduces the increase in labor required for label estimation work.

FIG. 1 is an explanatory diagram illustrating a label estimation system according to an embodiment. FIG. 4 is a diagram showing an example of label co-occurrence information according to the embodiment. 5A to 5C are diagrams illustrating an example of label suitability information generation processing according to an embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a learning device 1 according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of a control unit 11 according to the embodiment. 10 is a flowchart showing an example of the flow of processing executed by the learning device 1 in the embodiment. FIG. 2 is a diagram showing an example of the hardware configuration of a label estimation device 2 according to an embodiment. FIG. 2 is a diagram showing an example of the configuration of a control unit 21 according to the embodiment. 10 is a flowchart showing an example of the flow of processing executed by a label estimation device 2 in the embodiment. FIG. 1 is a first diagram showing an example of experimental results using the label estimation system according to the embodiment. FIG. 2 is a second diagram showing an example of experimental results using the label estimation system according to the embodiment. FIG. 3 is a third diagram showing an example of experimental results using the label estimation system according to the embodiment. FIG. 10 is a diagram showing an example of the configuration of a control unit in a modified example.

(Embodiment)
FIG. 1 is an explanatory diagram illustrating a label estimation system 100 according to an embodiment. The label estimation system 100 is a system that estimates a label to be assigned to a sentence. The label estimation system 100 obtains a mathematical model for estimating a label to be assigned to a sentence using a machine learning method. The label estimation system 100 uses the obtained mathematical model to estimate a label to be assigned to an input sentence.

More specifically, when text information is input, the label estimation system 100, having acquired the mathematical model, uses the acquired mathematical model to estimate a label to be assigned to the text indicated by the text information, based on the text information. Text information is information that indicates a text. The label estimation system 100 will be described in more detail. The label estimation system 100 includes a learning device 1 and a label estimation device 2.

The learning device 1 obtains a trained label estimation model by updating the label estimation model using a machine learning method. The label estimation model is a mathematical model that estimates, based on input text information, the label to be assigned to a sentence indicated by the input text information (hereinafter referred to as the "target sentence"), and is a mathematical model before a predetermined termination condition for learning is satisfied.

More specifically, the result estimated by the label estimation model is label appropriateness information. Label appropriateness information is information indicating the label appropriateness for each label candidate. Label appropriateness is the degree to which a label is appropriate as a label for the target sentence. Label candidates are each of multiple labels that have been predetermined as candidates for labels to be assigned to the target sentence. Label candidates are terms that can be associated with the target sentence, such as "infectious disease," "business," "sports," and "stock prices."

A trained label estimation model is a label estimation model at the point in time when a predetermined termination condition for learning (hereinafter referred to as the "learning termination condition") is met. The learning termination condition is, for example, a condition that the change in the label estimation model due to learning is smaller than a predetermined change. The learning termination condition may also be, for example, a condition that the number of times learning has been performed has reached a predetermined number.

Hereinafter, the process by which the learning device 1 obtains a trained label estimation model is referred to as a model learning process. Any machine learning method may be used as long as it can obtain a trained label estimation model. The machine learning method may, for example, be a method using CNN (Convolutional Neural Networks), a method using LSTM (Long short-term memory), or a method using BERT (Bidirectional Encoder Representations from Transformers).

Machine learning methods for obtaining trained label estimation models use data that has text information as explanatory variables. The objective variable corresponding to the explanatory variables indicates label suitability information. Hereinafter, data that has text information as explanatory variables and label suitability information as objective variables will be referred to as model training data. Model training data is data used to obtain trained label estimation models. In other words, model training data is data used to train label estimation models. Hereinafter, the label suitability information contained in model training data will be referred to as training data.

<Examples of label appropriateness information>
A specific example of the representation of label appropriateness information will be described. When there are N label candidates (N is a natural number), the label appropriateness information is expressed, for example, as an N-dimensional vector. Each element of the N-dimensional vector corresponds to one of the N label candidates, and elements with different indexes n correspond to different label candidates. n is a natural number between 1 and N. Note that index n is an index that distinguishes between label candidates and also indicates the nth element of the N-dimensional vector. For ease of explanation, the label estimation system 100 will be described below using an example in which there are N label candidates.

Each element of the N-dimensional vector representing label suitability information indicates the label suitability of the corresponding label candidate. Label suitability is expressed, for example, as a value between 0 and 1. In such a case, the closer the value of each element of the N-dimensional vector representing label suitability information is to 0, the less appropriate the corresponding label candidate is as a label for the sentence indicated by the sentence information. On the other hand, the closer the value of each element of the N-dimensional vector representing label suitability information is to 1, the more appropriate the corresponding label candidate is as a label for the sentence indicated by the sentence information.

<Model learning process and loss function>
The model learning process will be described in more detail. As described above, the model learning process is a process of updating the label estimation model by a machine learning method using model learning data until the learning termination condition is met. In the model learning process, the label estimation model is updated so as to reduce the loss calculated using a loss function. Note that the loss calculated using the loss function is the value of the loss function, and is, for example, a value that represents the degree of mismatch between the output of the label estimation model and the training data.

The loss function is an index expressed using the degree of agreement and disagreement between the training data and the estimation results of the label estimation model. The loss function may be, for example, the binary cross-entropy defined by the following equation (1):

y _n is the label appropriateness indicated by the training data and indicates the label appropriateness of the label candidate with index n. y{^} _n is the label appropriateness estimated by the label estimation model and indicates the label appropriateness of the label candidate with index n. Note that A{^} indicates a symbol in which a circumflex is added to the symbol A. Therefore, y{^} _n means a symbol in which a circumflex is added to the symbol y and a subscript n is added. More specifically, y{^} _n means the symbol in the following formula (2).

The term expressed in the following equation (3) in equation (1) indicates the quantitative difference between the estimation results of the label estimation model and the training data when the estimation results of the label estimation model and the training data qualitatively match.

The term expressed by the following equation (4) in equation (1) indicates the quantitative difference between the estimation results of the label estimation model and the training data when there is a qualitative mismatch between the estimation results of the label estimation model and the training data.

A specific example of updating to reduce the loss function is a process that uses the loss function shown in equation (1) to update the label estimation model so as not to increase the probability that the label estimation model will infer incorrect label information. Incorrect label information is label appropriateness information that qualitatively disagrees with the training data.

The label estimation device 2 uses the trained label estimation model acquired by the learning device 1 to estimate the label to be assigned to the target sentence indicated by the input text information. More specifically, the label estimation device 2 uses the trained label estimation model acquired by the learning device 1 to estimate the label appropriateness of each label candidate for the target sentence indicated by the input text information.

<Generating label appropriateness information included in model training data>
An example of a method for generating label appropriateness information included in model learning data will be described below. The label appropriateness information is generated, for example, manually or by a device, based on correct/incorrect information and label co-occurrence information.

The correct/incorrect information is information indicating a label that satisfies a predetermined condition regarding the likelihood of being assigned to a sentence indicated by the sentence information included in the model training data. In other words, the correct/incorrect information is information indicating a label that satisfies a predetermined condition regarding the likelihood of label appropriateness for a sentence indicated by the model training data. The predetermined condition regarding the likelihood of label appropriateness (hereinafter referred to as the "label appropriateness condition") is, for example, the condition that the label appropriateness is the highest. If there are multiple labels that satisfy the label appropriateness condition, the correct/incorrect information may indicate multiple labels. The correct/incorrect information is expressed, for example, as an N-dimensional vector in which only the element corresponding to the label with the highest likelihood of being assigned has a value of 1, and the other elements have values of 0. For simplicity of explanation, the label estimation system 100 will be described below using an example in which the label appropriateness condition is the condition that the label appropriateness is the highest.

Label co-occurrence information is information that indicates the probability of co-occurrence between any two of N label candidates. Specifically, the probability of co-occurrence is the probability that one label candidate appears in a sentence when the other label candidate appears in the sentence. Note that label co-occurrence information may also indicate the probability of co-occurrence between identical label candidates. Since the probability of co-occurrence between identical label candidates refers to autocorrelation, the probability of co-occurrence between identical label candidates is 1. Note that label co-occurrence information does not necessarily have to indicate the probability of co-occurrence between identical label candidates; in such cases, the probability of co-occurrence between identical label candidates indicated by label co-occurrence information is, for example, 0.

Figure 2 shows an example of label co-occurrence information in an embodiment. The label co-occurrence information is expressed, for example, as a positive definite matrix whose elements have values between 0 and 1. In the example of Figure 2, the columns and columns each represent label candidates, and the diagonal elements represent autocorrelation.

More specifically, the label co-occurrence information in Figure 2 is a matrix showing the PPMI (Positive Pointwise Mutual Information) scores between label candidates. The PPMI score is defined by the following equation (5):

In formula (5), _ln indicates a label candidate for index n, and _lm indicates a label candidate for index m. Note that m is an integer between 1 and N. m may be the same as or different from n. C( _ln ) indicates the number of times the label candidate for index n appears in a set of multiple predetermined sentences prepared in advance (hereinafter referred to as the "pre-sent sentence set"). C( _lm ) indicates the number of times the label candidate for index m appears in the pre-sent sentence set. C( _ln , lm) indicates the number of times the label candidate for index n co-occurs with the label candidate for index m in the pre-sent sentence set.

Hereinafter, the process of generating label appropriateness information based on correct/incorrect information and label co-occurrence information is referred to as the label appropriateness information generation process. In the label appropriateness information generation process, for example, for a label candidate whose element of the vector representing the correct/incorrect information has a value of 1, the probability of co-occurrence with other label candidates is obtained using the label co-occurrence information. If there are multiple label candidates whose element of the vector representing the correct/incorrect information has a value of 1, for example, the probability of co-occurrence with other label candidates is obtained for the multiple label candidates whose element has a value of 1, and the sum of the co-occurrence probabilities for each other label candidate is calculated. Next, in the label appropriateness information generation process, a function that limits the value of an independent variable to a predetermined value between 0 and 1, such as a sigmoid function, is used to convert the probability of co-occurrence with other label candidates to a value between 0 and 1.

In the label appropriateness information generation process, a process (hereinafter referred to as the "replacement process") is performed in which the element values of the vector representing the correct/incorrect information, which had a value of 0, are replaced with converted values. The correct/incorrect information whose element values have been changed by the replacement process is the label appropriateness information.

Figure 3 is a diagram illustrating an example of the label appropriateness information generation process in an embodiment. More specifically, Figure 3 is an explanatory diagram illustrating an example of the label appropriateness information generation process in an example where the label co-occurrence information is a matrix indicating the PPMI scores of label candidates (hereinafter referred to as the "PPMI matrix").

Figure 3 shows images G1 to G5. Image G1 shows an example of correct/incorrect information. The correct/incorrect information for image G1 shows five label candidates: "sports," "business," "health," "vaccine," and "infectious disease." Image G1 shows that "business" and "infectious disease" have the highest label appropriateness. In Figure 3, "business" and "infectious disease" are label candidates that meet the label appropriateness conditions.

Image G2 shows an example of a PPMI matrix. Image G3 shows a sigmoid function. Image G4 shows the process of obtaining the vector sum of rows in the PPMI matrix that satisfy the label appropriateness conditions. Specifically, it shows the process of obtaining the vector sum of rows indicating the probability that a label candidate will co-occur with the label candidate "business" and rows indicating the probability that a label candidate will co-occur with the label candidate "infectious disease." Image G5 shows an example of label appropriateness information.

As shown in image G4, the label appropriateness information generation process involves adding up rows in the PPMI matrix that indicate the probability of co-occurrence between the correct label and label candidates (hereinafter referred to as "major co-occurrence rows"). Hereinafter, the process of adding up major co-occurrence rows in the PPMI matrix is referred to as the addition process. In the example of Figure 3, "business" and "infectious disease" are correct labels, and the process of adding up the "business" row and the "infectious disease" row is the addition process. The correct label is the label candidate that corresponds to an element with a value of 1 among the label candidates corresponding to the elements of the N-dimensional vector that indicates correctness information. In other words, the correct label is the label candidate that satisfies the label appropriateness conditions.

By performing the summation process, the PPMI scores of incorrect labels that tend to co-occur with correct labels are added together for each incorrect label. The information obtained as a result of the summation is expressed, for example, as an N-dimensional vector. In the example of Figure 3, the summation process adds together the PPMI scores of incorrect labels that tend to co-occur with "business" and the PPMI scores of incorrect labels that tend to co-occur with "infectious disease".

An incorrect label is a label candidate that corresponds to an element with a value of 0 among the label candidates corresponding to the elements of the N-dimensional vector indicating correct/incorrect information. In other words, an incorrect label is a label candidate that is not a correct label. In the example in Figure 3, the incorrect labels are "sports", "health", and "vaccine".

Next, in the label appropriateness information generation process, the N-dimensional vector obtained by executing the addition process (hereinafter referred to as the "addition result vector") is input into the sigmoid function shown in image G3, whereby the value of each element of the addition result vector is normalized to a value between 0 and 1. Hereinafter, the process of normalizing the value of each element of the addition result vector to a value between 0 and 1 by inputting the addition result vector into the sigmoid function is referred to as the first normalization process. The first normalization process is defined by the following equation (6).

P _nm denotes the PPMI matrix, and σ(·) denotes the sigmoid function.

In the label appropriateness information generation process, after the first normalization process is performed, a process of adding the obtained score S _n and vector y _n (hereinafter referred to as a "smoothing process") is performed. In the label appropriateness information generation process, after the first normalization process is performed, a normalization process (hereinafter referred to as a "second normalization process") shown by the following equations (7) and (8) is also performed.

p _{' nm} means the element in the nth row and mth column in the PPMI matrix P'. y _m is a vector indicating the row indicating the probability of co-occurrence between the correct label with index m and the label candidate. s _n is a vector. α is a hyperparameter (a coefficient between 0 and 1) indicating the strength of smoothing. α is, for example, a coefficient between 0 and 1 called a scaling rate. In this way, the replacement process includes an addition process, a first normalization process, a smoothing process, and a second normalization process. y _n ' obtained in this way is an example of label appropriateness information.

Figure 4 is a diagram showing an example of the hardware configuration of a learning device 1 in an embodiment. The learning device 1 has a control unit 11 including a processor 91 such as a CPU (Central Processing Unit) and memory 92 connected by a bus, and executes a program. By executing the program, the learning device 1 functions as a device including the control unit 11, input unit 12, communication unit 13, memory unit 14, and output unit 15.

More specifically, the processor 91 reads the program stored in the storage unit 14 and stores the read program in the memory 92. When the processor 91 executes the program stored in the memory 92, the learning device 1 functions as a device including a control unit 11, an input unit 12, a communication unit 13, a storage unit 14, and an output unit 15.

The control unit 11 controls the operation of the various functional units included in the learning device 1. The control unit 11 executes, for example, a model learning process. The control unit 11 may also execute, for example, a label appropriateness information generation process. As described above, the label appropriateness information generation process may be performed manually, or may be executed by a device. Below, the label estimation system 100 will be described using an example in which the learning device 1 executes the label appropriateness information generation process.

The control unit 11 controls, for example, the operation of the output unit 15. The control unit 11 records, for example, various information generated by executing the model learning process in the memory unit 14. The control unit 11 records, for example, the obtained label suitability information in the memory unit 14.

The input unit 12 includes input devices such as a mouse, keyboard, and touch panel. The input unit 12 may be configured as an interface that connects these input devices to the learning device 1. The input unit 12 accepts various types of information input to the learning device 1.

The communication unit 13 includes a communication interface for connecting the learning device 1 to an external device. The communication unit 13 communicates with the external device via wired or wireless connections. The external device is, for example, a device that transmits true/false information. The external device is, for example, a device that transmits label co-occurrence information. The external device is, for example, a device that transmits model training data. The external device is, for example, the label estimation device 2. Note that the true/false information, label co-occurrence information, and model training data do not necessarily need to be input via the communication unit 13, and may be input to the input unit 12.

The memory unit 14 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or semiconductor storage device. The memory unit 14 stores various information related to the learning device 1. The memory unit 14 stores information input via the input unit 12 or the communication unit 13, for example. The memory unit 14 stores various information generated by executing a model learning process, for example. The memory unit 14 stores label appropriateness information, for example. The memory unit 14 stores a label estimation model in advance. The memory unit 14 may also store the obtained trained label estimation model.

The output unit 15 outputs various types of information. The output unit 15 is configured to include a display device such as a CRT (Cathode Ray Tube) display, a liquid crystal display, or an organic EL (Electro-Luminescence) display. The output unit 15 may be configured as an interface that connects these display devices to the learning device 1. The output unit 15 outputs information input to the input unit 12, for example. The output unit 15 may display the execution results of the model learning process, for example. The output unit 15 may display label suitability information, for example.

Figure 5 is a diagram showing an example of the configuration of the control unit 11 in an embodiment. The control unit 11 includes a label appropriateness information acquisition unit 110, a model learning unit 120, a memory control unit 130, a communication control unit 140, and an output control unit 150.

The label suitability information acquisition unit 110 acquires label suitability information. The label suitability information acquisition unit 110 acquires label suitability information by executing a label suitability information generation process based on the correct/incorrect information and label co-occurrence information input to the input unit 12 or the communication unit 13.

The model learning unit 120 updates the label estimation model using the label appropriateness information and the model learning data input to the input unit 12 or the communication unit 13 until the learning termination condition is met. In other words, the model learning unit 120 obtains a learned label estimation model by performing a model learning process using the label appropriateness information and the model learning data input to the input unit 12 or the communication unit 13.

The memory control unit 130 records various information in the memory unit 14. The communication control unit 140 controls the operation of the communication unit 13. The output control unit 150 controls the operation of the output unit 15.

Figure 6 is a flowchart showing an example of the flow of processing executed by the learning device 1 in an embodiment. The label appropriateness information acquisition unit 110 acquires label appropriateness information (step S101). Next, model training data is input to the input unit or communication unit 13 (step S102). Next, the model training unit 120 estimates label appropriateness information by inputting sentence information indicated by the model training data into a label estimation model (step S103). Next, the model training unit 120 updates the label estimation model based on the label appropriateness information included in the model training data and the estimation result of step S103 (step S104). Next, the model training unit 120 determines whether a learning termination condition is satisfied (step S105). If the learning termination condition is satisfied (step S105: YES), the processing ends. On the other hand, if the learning termination condition is not satisfied (step S105: NO), the processing returns to step S102.

The processes from step S102 to step S105, which are repeated until the learning termination condition is met, are an example of a model learning process.

Figure 7 is a diagram showing an example of the hardware configuration of a label estimation device 2 in an embodiment. The label estimation device 2 has a control unit 21 including a processor 93 such as a CPU and memory 94 connected by a bus, and executes a program. By executing the program, the label estimation device 2 functions as a device including the control unit 21, input unit 22, communication unit 23, memory unit 24, and output unit 25.

More specifically, the processor 93 reads the program stored in the storage unit 24 and stores the read program in the memory 94. When the processor 93 executes the program stored in the memory 94, the label estimation device 2 functions as a device including a control unit 21, an input unit 22, a communication unit 23, a storage unit 24, and an output unit 25.

The control unit 21 controls the operation of the various functional units included in the label estimation device 2. The control unit 21, for example, executes a trained label estimation model. The control unit 21, for example, controls the operation of the output unit 25. The control unit 21, for example, records various information generated by the execution of the trained label estimation model in the memory unit 24.

The input unit 22 includes input devices such as a mouse, keyboard, and touch panel. The input unit 22 may be configured as an interface that connects these input devices to the label estimation device 2. The input unit 22 accepts input of various information for the label estimation device 2.

The communication unit 23 includes a communication interface for connecting the label estimation device 2 to an external device. The communication unit 23 communicates with the external device via wired or wireless connections. The external device is, for example, a device that transmits text information. The external device is, for example, the learning device 1. The communication unit 23 acquires a trained label estimation model by communicating with the learning device 1. Note that the text information does not necessarily have to be input to the communication unit 23, but may be input to the input unit 22.

The memory unit 24 is configured using a computer-readable storage medium device such as a magnetic hard disk drive or semiconductor storage device. The memory unit 24 stores various information related to the label estimation device 2. The memory unit 24 stores information input via, for example, the input unit 22 or the communication unit 23. The memory unit 24 stores various information generated by executing, for example, a trained label estimation model. The memory unit 24 stores the trained label estimation model.

The output unit 25 outputs various types of information. The output unit 25 is configured to include a display device such as a CRT display, a liquid crystal display, or an organic EL display. The output unit 25 may be configured as an interface that connects these display devices to the label estimation device 2. The output unit 25 outputs information input to the input unit 22, for example. The output unit 25 may also display the execution results of the trained label estimation model, for example.

Figure 8 is a diagram showing an example of the configuration of the control unit 21 in an embodiment. The control unit 21 includes an object acquisition unit 210, an estimation unit 220, a memory control unit 230, a communication control unit 240, and an output control unit 250. The object acquisition unit 210 acquires text information input to the input unit 22 or the communication unit 23.

The estimation unit 220 executes the learned label estimation model on the text information acquired by the object acquisition unit 210. By executing the learned label estimation model, the estimation unit 220 estimates label appropriateness information for the text information acquired by the object acquisition unit 210.

The memory control unit 230 records various information in the memory unit 24. The communication control unit 240 controls the operation of the communication unit 23. The output control unit 250 controls the operation of the output unit 25.

Figure 9 is a flowchart showing an example of the flow of processing executed by the label estimation device 2 in this embodiment. The object acquisition unit 210 acquires text information input to the input unit 22 or the communication unit 23 (step S201). Next, the estimation unit 220 executes a trained label estimation model to estimate label appropriateness information for the text information acquired by the object acquisition unit 210 (step S202). Next, the output control unit 250 controls the operation of the output unit 25 to cause the output unit 25 to output the acquired label appropriateness information (step S203).

(Experimental results)
Here, the results of an experiment using the label estimation system 100 will be described. In the experiment, benchmarks used in multi-label classification were used as datasets. Specifically, three datasets were used: Reuters-21578, the Arxiv Academic Paper Dataset (AAPD), and 20Newsgroups. In the experiment, machine learning models used in natural language processing were used as machine learning models. Specifically, Bidirectional Encoder Representations from Transformers (BERT), Long Short Term Memory (Bi-LSTM), and Convolution Neural Network (CNN) were used. In the experiment, Micro-f1 and Macro-f1 were used as evaluation indices.

Figure 10 is a first diagram showing an example of experimental results using the label estimation system 100 of the embodiment. In Figure 10, the rows with "BERT w/ALS", "LSTM w/ALS", and "CNN w/ALS" in the "Method" column indicate results using the label estimation system 100. The rows with "BERT only", "LSTM only", and "CNN only" in the "Method" column indicate estimation results using a trained label estimation model obtained using correct/incorrect information without using label appropriateness information.

Note that "BERT" in "BERT w/ALS" and "BERT only" indicates that the machine learning model used in the experiment was BERT. Note that "LSTM" in "LSTM w/ALS" and "LSTM only" indicates that the machine learning model used in the experiment was Bi-LSTM. Note that "CNN" in "CNN w/ALS" and "CNN only" indicates that the machine learning model used in the experiment was CNN.

"Macro-f1" in "Rueters-21578" indicates the value of Macor-f1 when the dataset used is Reuters-21578. "Micro-f1" in "Rueters-21578" indicates the value of Micor-f1 when the dataset used is Reuters-21578. "Macro-f1" in "AAPD" indicates the value of Macor-f1 when the dataset used is AAPD. "Micro-f1" in "AAPD" indicates the value of Micor-f1 when the dataset used is AAPD. "Macro-f1" in "20Newsgroups" indicates the value of Macor-f1 when the dataset used is 20Newsgroups. "Micro-f1" in "20Newsgroups" indicates the value of Micor-f1 when the dataset used is 20Newsgroups.

The results in Figure 10 show the results of five experiments conducted with different random seeds. The numbers in parentheses in Figure 10 indicate standard deviations. The results in Figure 10 demonstrate that the label estimation system 100 can estimate labels with high accuracy regardless of a specific machine learning model such as CNN or Bi-LSTM.

Figure 11 is a second diagram showing an example of experimental results using the label estimation system 100 of the embodiment. More specifically, Figure 11 shows the results of an experimental evaluation of the accuracy of estimating low-frequency label candidates. Note that Figure 10 shows the results of an experimental evaluation of the accuracy of estimating both low-frequency label candidates and non-low-frequency label candidates. Note that a low-frequency label candidate refers to a label candidate that, among multiple label candidates, is ranked below the median in terms of the number of times it appears in the dataset.

In Figure 11, rows with "BERT w/ALS," "LSTM w/ALS," or "CNN w/ALS" in the "Method" column indicate results using the label estimation system 100. Rows with "BERT only," "LSTM only," or "CNN only" in the "Method" column indicate estimation results using a trained label estimation model obtained using correct/incorrect information without using label appropriateness information.

Note that "BERT" in "BERT w/ALS" and "BERT only" indicates that the machine learning model used in the experiment was BERT. Note that "LSTM" in "LSTM w/ALS" and "LSTM only" indicates that the machine learning model used in the experiment was Bi-LSTM. Note that "CNN" in "CNN w/ALS" and "CNN only" indicates that the machine learning model used in the experiment was CNN.

"Macro-f1" in "Rueters-21578" indicates the value of Macor-f1 when the dataset used is Reuters-21578. "Micro-f1" in "Rueters-21578" indicates the value of Micor-f1 when the dataset used is Reuters-21578. "Macro-f1" in "AAPD" indicates the value of Macor-f1 when the dataset used is AAPD. "Micro-f1" in "AAPD" indicates the value of Micor-f1 when the dataset used is AAPD. "Macro-f1" in "20Newsgroups" indicates the value of Macor-f1 when the dataset used is 20Newsgroups. "Micro-f1" in "20Newsgroups" indicates the value of Micor-f1 when the dataset used is 20Newsgroups.

The results in Figure 11 show the results of five experiments conducted using different random seeds. The numbers in parentheses in Figure 11 indicate standard deviations. The results in Figure 11 demonstrate that the label estimation system 100 is capable of highly accurate estimation of low-frequency label candidates, regardless of the use of specific machine learning models such as CNN or Bi-LSTM.

Figure 12 is a third diagram showing an example of experimental results using the label estimation system 100 of an embodiment. The horizontal axis of Figure 12 represents the number of training rounds. The vertical axis of Figure 12 represents the Micro-f1 value. "CNN only (train)" represents the results of training data estimation using a trained label estimation model obtained using correct/incorrect information without label appropriateness information. "CNN only (valid)" represents the results of development data estimation using a trained label estimation model obtained using correct/incorrect information without label appropriateness information. "CNN with ALS (train)" represents the results of training data estimation using the label estimation system 100. "CNN with ALS (valid)" represents the results of development data estimation using the label estimation system 100. Note that development data refers to training data used in experiments to measure the estimation accuracy of the label estimation system 100 for each training round.

The results in Figure 12 show that learning using the label estimation system 100 is more effective at suppressing overfitting in the early stages of learning than learning using correct/incorrect information without using label appropriateness information.

In this embodiment, the learning device 1 configured in this manner obtains a trained label estimation model using label appropriateness information. As a result, it is possible to obtain a mathematical model that estimates more labels to be assigned with high accuracy than a device that obtains a trained label estimation model based only on correct/incorrect information rather than label appropriateness information. As a result, the learning device 1 can suppress an increase in the effort required for the task of estimating labels.

Furthermore, the label estimation device 2 in this embodiment configured estimates labels to be assigned to sentences using a trained label estimation model obtained using label appropriateness information. Therefore, compared to a device that obtains a trained label estimation model based only on correct/incorrect information rather than label appropriateness information, it can estimate more labels to be assigned with high accuracy. As a result, the label estimation device 2 can suppress an increase in the effort required for the task of estimating labels.

(Modification)
As described above, the label suitability information may be generated manually. In such a case, the label suitability information is input to the input unit 12 or the communication unit 13 instead of the correct/incorrect information and the label co-occurrence information. In such a case, the label suitability information acquisition unit 110 acquires the label suitability information input to the input unit 12 or the communication unit 13, instead of acquiring the label suitability information based on the correct/incorrect information and the label co-occurrence information.

The output control unit 250 may output a predetermined label from the label appropriateness information of the estimation result of the estimation unit 220 as a genre to the output unit 25.

As described above, the label appropriateness indicated by the label appropriateness information is, for example, expressed as a value between 0 and 1. However, the label appropriateness does not necessarily have to be expressed as a value between 0 and 1. Therefore, the values of each element of the N-dimensional vector representing the label appropriateness information may include negative values. An example of the label appropriateness information generation process was explained above, and the process used was an example of a process that uses a function, such as a sigmoid function, that limits the value of an independent variable to a predetermined value between 0 and 1. This is an example of a process that uses an example where the label appropriateness is expressed as a value between 0 and 1. Therefore, if the label appropriateness does not need to be a value between 0 and 1, there is no need to use a function, such as a sigmoid function, that limits the value of an independent variable to a predetermined value between 0 and 1.

The control unit 21 may further include either or both of a sentence similarity estimation unit 260 and a key phrase extraction unit 270. Hereinafter, a control unit 21 including a sentence similarity estimation unit 260 and a key phrase extraction unit 270 will be referred to as a control unit 21a. Figure 13 is a diagram showing an example of the configuration of a control unit 21a in a modified example. The control unit 21a differs from the control unit 21 in that it includes a sentence similarity estimation unit 260 and a key phrase extraction unit 270.

The text similarity estimation unit 260 estimates the degree of similarity between two pieces of text information (hereinafter referred to as "text similarity"). At least one of the two pieces of text information is text information acquired by the target acquisition unit 210. Therefore, both pieces of text information may be text information acquired by the target acquisition unit 210, or one may be text information acquired by the target acquisition unit 210 and the other may be text information previously stored in the storage unit 24. The text similarity estimation unit 260 inputs each of the two pieces of text information to the estimation unit 220 and causes the estimation unit 220 to estimate label appropriateness information for both pieces of text information. Based on the two pieces of label appropriateness information estimated by the estimation unit 220, the text similarity estimation unit 260 acquires the degree of agreement between the label appropriateness information as the text similarity between the two pieces of text information. For example, the text similarity estimation unit 260 acquires the value of the inner product of each vector corresponding to each of the two pieces of label appropriateness information as the text similarity.

The text similarity estimation unit 260 may determine that two pieces of text information are similar if the acquired text similarity is equal to or greater than a predetermined level. In such a case, the output control unit 250 may cause the output unit 25 to output one or both of the two pieces of text information determined by the text similarity estimation unit 260 to be similar.

The key phrase extraction unit 270 uses GiNZA, an open source library for Japanese natural language processing, to acquire key phrases in the text indicated by the text information. The output control unit 250 may cause the output unit 25 to output the key phrases acquired by the key phrase extraction unit 270.

The learning device 1 may be implemented using multiple information processing devices connected to each other via a network. In this case, the functional units of the learning device 1 may be distributed and implemented across multiple information processing devices.

The label estimation device 2 may be implemented using multiple information processing devices connected to each other so that they can communicate with each other via a network. In this case, the functional units of the label estimation device 2 may be distributed and implemented across multiple information processing devices.

All or part of the functions of the learning device 1 and the label estimation device 2 may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. The program may also be transmitted via telecommunications lines.

Note that the text information acquired by the object acquisition unit 210 is an example of object information. The text indicated by the text information acquired by the object acquisition unit 210 is an example of a processing object.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and includes designs that do not deviate from the gist of the present invention.

100...Label estimation system, 1...Learning device, 2...Label estimation device, 11...Control unit, 12...Input unit, 13...Communication unit, 14...Memory unit, 15...Output unit, 110...Label appropriateness information acquisition unit, 120...Model learning unit, 130...Memory control unit, 140...Communication control unit, 150...Output control unit, 21...Control unit, 22...Input unit, 23...Communication unit, 24...Memory unit, 25...Output unit, 210...Object acquisition unit, 220...Estimation unit, 230...Memory control unit, 240...Communication control unit, 250...Output control unit, 91...Processor, 92...Memory, 93...Processor, 94...Memory, 21a...Control unit, 260...Sentence similarity estimation unit, 270...Key phrase extraction unit

Claims (5)

a model learning unit that updates a label estimation model, which is a mathematical model that estimates a label to be assigned to a sentence indicated by input text information, by a machine learning method using model learning data that includes text information indicating a sentence and label appropriateness information indicating the degree to which a plurality of labels predetermined as candidates for labels to be assigned to the sentence are appropriate as labels for the sentence;
Equipped with
The label appropriateness information is information obtained based on true/false information indicating a label that satisfies a predetermined condition regarding the probability of being assigned to a sentence indicated by the model training data, and label co-occurrence information that is information indicating the probability of co-occurrence between any two of a plurality of labels that are predetermined as candidates for labels to be assigned to the sentence.
Learning device. The predetermined condition is a condition that the probability of being assigned to a sentence represented by the model training data is the highest.
The learning device according to claim 1 . an object acquisition unit that acquires object information that indicates a sentence to be processed;
a model learning unit that updates a label estimation model, which is a mathematical model that estimates a label to be assigned to a sentence indicated by input sentence information, by a machine learning method using model learning data including sentence information indicating a sentence and label appropriateness information indicating the degree to which a plurality of labels predetermined as candidate labels to be assigned to the sentence are appropriate as labels for the sentence, wherein the label appropriateness information is information obtained based on true/false information indicating a label that satisfies a predetermined condition regarding the likelihood of being assigned to the sentence indicated by the model learning data, and label co-occurrence information, which is information indicating the probability of co-occurrence between any two of a plurality of labels predetermined as candidate labels to be assigned to the sentence; and an estimation unit that estimates a label to be assigned to the sentence indicated by the target information acquired by the target acquisition unit, using the trained label estimation model obtained by a learning device, the label appropriateness information being information obtained based on true/false information indicating a label that satisfies a predetermined condition regarding the likelihood of being assigned to the sentence indicated by the model learning data, and label co-occurrence information, which is information indicating the probability of co-occurrence between any two of a plurality of labels predetermined as candidate labels to be assigned to the sentence.
A label estimation device comprising: A program for causing a computer to function as the learning device described in claim 1 or 2. A program for causing a computer to function as the label estimation device described in claim 3.