JP7822292B2 - Annotation device and method - Google Patents

Description

The present invention relates to a technique for annotation, or labeling, in machine learning.

Currently, in supervised machine learning, so-called annotators create (label) training data using a user interface. For this reason, manually labeling all data using a user interface by an annotator requires a huge amount of work. For this reason, a framework known as active learning is sometimes used to reduce this amount of work.

In active learning, an annotator first manually labels a portion of the target data. Next, the labeled data is used as training data to train a machine learning model. The trained machine learning model also estimates labels for unlabeled data and presents them to the annotator via a user interface. The annotator then checks the presented results and makes any corrections.

The machine learning model is then retrained using the confirmed labeled data, and the process of making inferences on unlabeled data, followed by confirmation and correction by an annotator, is repeated. This reduces the amount of work required compared to labeling everything manually.

Active learning is also being considered from the perspective of what kind of data should be presented when presenting results to annotators. For example, one method for this is to quantify the confidence in the estimation results and prioritize presenting those with low confidence, thereby increasing the rate of accuracy improvement when retraining a machine learning model. Specifically, one method uses the entropy of the estimation probability as the confidence level. Another method is to prioritize presenting unlabeled data with a large number of similar pieces of data. In this case, the similarity between data is calculated, and retraining efficiency can be improved by prioritizing the presentation of data with high similarity.

Technologies such as those described above that aim to reduce labeling work by combining machine learning models and user feedback have been proposed in, for example, Patent Documents 1 and 2. Patent Document 1 describes an information processing device that prevents a decrease in machine learning accuracy due to incorrect tagging. Patent Document 2 also describes extracting synonyms/related words with good efficiency and accuracy. Patent Documents 1 and 2 are described below.

Some machine learning tasks involve learning and inference for multiple subtasks, and the desired output is obtained by combining the inference results of each subtask. For example, in a natural language processing event extraction task, named entities are extracted from text using named entity extraction, and then relationships are inferred for pairs of named entities in the text using relationship extraction. Furthermore, in machine learning models that solve tasks consisting of multiple subtasks, the models for each subtask may share parts. For example, in a neural network that simultaneously infers named entity extraction and relationship extraction, several layers from the input are shared, and only the final layer is split between named entity extraction and relationship extraction. This model is constructed in this way. Learning in one subtask contributes to learning and inference in other subtasks, mutually improving accuracy.

In annotation systems that employ an active learning framework, the aforementioned Patent Documents 1 and 2 have proposed methods for improving re-learning efficiency for single tasks by determining presentation priorities according to quantitative priority scores for the estimation results of each piece of data. First, Patent Document 1 assumes that multiple labels (tags) can be attached to a single piece of data, and determines whether the estimation results are abnormal based on the probability of co-occurrence between tags, and notifies the system accordingly.

In addition, in Patent Document 2, structured data candidates related to relationships are generated for document data, and validity is determined based on the likelihood of the generated structured data candidates.

Japanese Patent Application Laid-Open No. 2018-185601 Japanese Patent Application Laid-Open No. 2020-095452

As described above, Patent Documents 1 and 2 describe processes for making judgments using values related to the co-occurrence probability and likelihood of labels in a single task. However, in machine learning models in which multiple subtasks share part of the model, it has been difficult to determine presentation priority using the same method as for a single task. Here, when presentation priority is determined by simply adding up the priority scores for each subtask, it is possible that only one subtask has not been properly trained, resulting in low accuracy. In this case, it is not possible to focus on data that will improve the accuracy of that subtask. This makes efficient re-learning difficult. Furthermore, because the mutual contribution of subtask learning to other subtasks is not taken into account, it is possible that sufficient improvement in accuracy during re-learning cannot be expected.

The present invention aims to provide technology that enables efficient labeling for machine learning models that execute tasks consisting of multiple subtasks.

With regard to machine learning labeling, the priority for confirming the inferred results, for example, the inferred labels, is determined based on the influence level, which indicates the degree of influence that the learning of a subtask included in multiple subtasks has on the subtask modules that learn other subtasks. Note that this priority includes the priority for presenting the results to the annotator.

A more specific configuration of the present invention is an annotation device that realizes machine learning labeling, comprising: a supervised machine learning module having multiple subtask modules that learn using labeled data and estimate estimated labels for unlabeled data for each of multiple subtasks that make up the machine learning task; a subtask mutual influence calculation module that calculates an influence level that indicates the degree to which learning in the subtask modules affects other subtask modules; and a label correction module that corrects the estimated labels in accordance with the confirmation results for the estimated labels presented in accordance with an overall priority score based on the influence levels. The present invention also includes an annotation method using this annotation device.

This invention enables efficient labeling in machine learning using multiple subtasks.

FIG. 1 is a diagram illustrating a functional configuration of an annotation device according to an embodiment of the present invention. FIG. 2 illustrates unlabeled data in one embodiment of the present invention. FIG. 2 is a diagram illustrating labeled data in one embodiment of the present invention. FIG. 2 is a diagram illustrating the processing of a supervised machine learning module in one embodiment of the present invention. FIG. 10 is a diagram illustrating subtask accuracy information in one embodiment of the present invention. FIG. 10 illustrates an example output of a subtask priority score calculation module in one embodiment of the present invention. FIG. 10 illustrates an example output of a subtask priority score calculation module in one embodiment of the present invention. FIG. 1 is an overall processing flow diagram according to an embodiment of the present invention. FIG. 10 is a diagram showing priority scores for one piece of data in one embodiment of the present invention. FIG. 10 is a diagram showing a total priority score for each piece of data in one embodiment of the present invention. FIG. 10 is a diagram showing a confirmation screen for all data to be confirmed in this embodiment. FIG. 10 is a diagram showing a details confirmation screen in the present embodiment. 1 is a configuration diagram of an annotation system according to an embodiment of the present invention.

An embodiment of the present invention will be described below. In this embodiment, an event extraction task from a document will be described as an example. That is, the event extraction task is used as a task in the supervised machine learning module 103 described below. Here, the supervised machine learning module 103 has three subtask modules. First, the first subtask module extracts named entities from the document. Furthermore, the second subtask module extracts relationships from the document. Furthermore, the third subtask module performs name matching in the document. That is, in this embodiment, the subtasks performed are named entity extraction, relationship extraction, and name matching.

Unlabeled data 101 and labeled data 102 are used to execute these subtasks. These unlabeled data 101 and labeled data 102 are failure response history data. In this embodiment, the supervised machine learning module 103 extracts event information on the actions taken from these data by combining part names and action names.

In this embodiment, to execute these event extraction tasks and their subtasks, estimated labels are estimated for the unlabeled data 101. Then, an overall priority score is calculated, indicating the priority for checking these estimated labels. As a result, the annotator can check and correct the estimated labels in a more appropriate order. Details of this are explained below using the drawings.

Figure 1 is a diagram showing the functional configuration of the annotation device 1 in this embodiment. In Figure 1, the annotation device 1 has a supervised machine learning module 103, a subtask accuracy calculation module 106, a subtask mutual influence calculation module 107, a subtask priority score calculation module 108, an overall priority score calculation module 109, a label correction module 110, and an overall priority score recalculation module 111. Note that each of these components (modules) and the subtask module 104 described below are components that execute processing, and can also be expressed as a ~ unit. In addition, this embodiment uses unlabeled data 101 and labeled data 102.

First, unlabeled data 101 is data that has not been assigned a label. In this embodiment, it is text data describing the failure response history in natural language. However, in the present invention, the target data may be in the form of image data, table data, time-series data, or other data other than text data. Details of unlabeled data 101 will be described later with reference to Figure 2.

Furthermore, labeled data 102 is data to which labels for each subtask and task of the supervised machine learning module have been assigned. Details will be described later in Figure 3.

The supervised machine learning module 103 also performs learning, verification, and estimation for each subtask module 104. In other words, it uses labeled data 102 as input to perform learning and verification for each subtask module 104. The supervised machine learning module 103 also uses unlabeled data 101 as input to perform estimation for each subtask module 104. Note that this estimation refers to estimating an estimated label, which is a label for the unlabeled data 101.

For this purpose, the supervised machine learning module 103 has multiple subtask modules 104 and a shared model 105. Here, the shared model 105 performs common processing within the subtask modules 104. Note that the shared model 105 may be realized as the processing entity, or may be realized as information indicating the model.

Here, each subtask module 104 may perform not only machine learning but also preprocessing of the input data and postprocessing of the estimation results. Details of the supervised machine learning module 103, subtask module 104, and shared model 105 will be described later in Figure 4.

The subtask accuracy calculation module 106 also calculates the estimation accuracy of the estimated labels estimated by the supervised machine learning module 103. To do this, the subtask accuracy calculation module 106 calculates the estimation accuracy of the estimated labels in the subtasks of each subtask module 104 from the estimated labels, which are the estimation results output from the supervised machine learning module 103. Details of this will be described later in Figure 5.

The subtask mutual influence calculation module 107 also calculates the influence, which indicates the degree of influence that learning about a subtask has on other subtask modules 104. To do this, the subtask mutual influence calculation module 107 uses the estimated labels output from the supervised machine learning module 103. This will be described in more detail later in Figure 6.

The subtask priority score calculation module 108 also calculates a subtask priority score related to the confirmation order for each data unit of data handled by each subtask module 104. Here, a data unit refers to each piece of unlabeled data 101 itself, or each element of the unlabeled data 101. Any calculation method may be used as long as it outputs a scalar value for the data or data elements. For example, in the case of a multi-class classification problem, the entropy of the output probability is calculated as the subtask priority score, resulting in a higher priority score in descending order of the confidence of the estimation. Alternatively, a similarity measure such as cosine similarity between the target data and other unlabeled data is calculated as the subtask priority score, resulting in a higher priority score for data similar to other data.

The overall priority score calculation module 109 also calculates an overall priority score for each subtask for each unlabeled data 101 using at least one of the estimation accuracy, influence, and subtask priority score described above. More preferably, the overall priority score is calculated based on at least the estimation accuracy. Alternatively, influence may be used. For example, the overall priority score can be calculated by multiplying each subtask priority score by the weight of each subtask, which is calculated as a function of the estimation accuracy and influence. The overall priority score indicates the priority in confirming the estimated label, and the estimated label is presented, for example, according to the overall priority score. This presentation includes presenting in an order according to the overall priority score and presenting in a format (size, layout) according to the overall priority score. Details of the calculation of the overall priority score are described in Figure 8.

The label correction module 110 also presents the estimated labels according to the overall priority score calculated by the overall priority score calculation module 109. As a result, the annotator can confirm the corresponding estimated labels. Then, when the label correction module 110 receives a correction instruction from the annotator based on the confirmation results, it corrects the estimated labels and adds them to the labeled data. For this purpose, the annotation device 1 is provided with a user interface (GUI) function, allowing the annotator to check and correct the labels. For example, the annotation device 1 may be equipped with an input device or display device itself, or may be connected to these.

As described above, in this embodiment, the annotator confirms and corrects the estimated labels, but the label correction module 110 may also be configured to perform confirmation and correction in cooperation with another system. The GUI in this embodiment will be described later with reference to Figure 9.

The overall priority score recalculation module 111 also recalculates the overall priority score in response to confirmation and correction of the estimated labels described above. For example, it updates the accuracy calculated by the subtask accuracy calculation module 106 using the accuracy of the data already confirmed by the label correction module 110. Then, it recalculates the influence calculated by the subtask mutual influence calculation module 107 and the overall priority score calculated by the overall priority score calculation module 109 in accordance with the update results. The overall priority score may also be recalculated by manually adjusting parameters related to the importance of accuracy and influence that appear in the overall priority score calculation formula using the GUI. The functionality of the overall priority score recalculation module 111 may also be provided in the overall priority score calculation module 109. This concludes the explanation of Figure 1.

Next, we will explain each type of data handled in this embodiment. Figure 2 is a diagram showing unlabeled data 101 in this embodiment. The unlabeled data 101 in this embodiment is text data of a failure response history. Therefore, as shown in Figure 2, the unlabeled data 101 is composed of a data ID 201 that uniquely identifies each piece of data and its corresponding data content 202.

Here, the data ID 201 identifies the corresponding unlabeled data 101. Here, the data ID 201 shown in Figure 2 is an integer value, but it can be in any format, such as a number or a character string, as long as it does not overlap with multiple data. Furthermore, the data ID 201 must not overlap with the data ID of the labeled data 102 described below.

Data content 202 is text data of the failure response history. While data content 202 shown in Figure 2 consists of only one piece of data, it may also consist of multiple elements, such as dividing it into items such as the error code that occurred, the details of the inspection results, and the estimated cause.

Next, Figure 3 is a diagram showing labeled data 102 in this embodiment. As shown in Figure 3, labeled data 102 is composed of a data ID 301, data content 302, and labels 303, 304, and 305 corresponding to each subtask. Data ID 301 and data content 302 are the same items as in unlabeled data 101. Labels 303, 304, and 305 indicate the three subtasks of named entity extraction, relationship extraction, and name matching. Here, each label 303, 304, and 305 may have any structure as long as it corresponds to the data content and the label is clearly identifiable. In Figure 3, named entity extraction label 303 stores the named entity extraction label ID (for example, "3-0"), the corresponding string, the position of the string, and the named entity label for the data content.

Here, the relationship extraction label stores a pair of named entity extraction label IDs and the corresponding relationship labels. The matching label 305 lists the named entity extraction label ID and the corresponding word after matching. For example, for data with data ID = 3, the expression "Part D" in data content 302 is a named entity for "part." There is a "part treatment relationship" between the two named entities, "Part D" and "Replacement." When matching is performed, the expression "Part D" in data content 302 is matched to the expression "AAA."

Next, the supervised machine learning module 103 including the shared model 105 will be described.
FIG. 4 is a diagram illustrating the processing of the supervised machine learning module 103 in this embodiment. As described above, the subtasks in this embodiment are named entity extraction, relationship extraction, and name matching. To this end, the supervised machine learning module 103 is composed of a shared model 105, a named entity extraction model 401, a relationship extraction model 402, and a name matching model 403. Each of these is assumed to be in the form of a neural network, but any method can be used as long as it can learn the label of each subtask from the labeled data 102 and estimate the label of each subtask from the unlabeled data 101. Each subtask performs common processing in the shared model 105 before the models 401, 402, and 403, and the parameters of the shared model 105 also change through learning. By learning each subtask in parallel, the learning of each subtask can contribute to improving the accuracy of the other subtasks. However, each of these models may be stored as template data. FIG. 4 will now be described.

First, text data 404 in the figure indicates that text data to be used as the task is input. In this example, two pieces of text data are prepared: "Error..." and "Inspection...." These are then divided into morphemes ("Error," "Part A," "to," "Replace," ".") using morphological analysis. For evaluation using training and validation data for the supervised machine learning module 103, data extracted from the labeled data 102 is used as this text data. For estimation by the supervised machine learning module 103, data extracted from the unlabeled data 101 is used.

Next, in vector representation 405, each morpheme of the text data is converted into a vector representation. Methods such as word2vec are available as conversion methods. Furthermore, in shared model 105, feature extraction is performed on the vector representation 405 of each morpheme using a neural network or the like, and a feature vector 406 is output. BERT is an example of a shared model 105.

The named entity extraction model 401 also takes the feature vector 406 as input and estimates the named entity label 407 of the target morpheme, i.e., whether it corresponds to a part name, a procedure name, or something else. If there is a part name that spans multiple morphemes, estimation may be made using BIO notation or the like rather than a label name such as part name.

Furthermore, the relationship extraction model 402 estimates the relationship label 408 for a pair of two morphemes in the same text data whose named entity label 407 is estimated to be a named entity such as a part name or a procedure name, by inputting a pair of feature vectors 406. In other words, the relationship extraction model 402 estimates whether there is a part-procedure relationship or no relationship. Here, in addition to the two morphemes, the feature vector 406 of the morpheme between them, or a feature vector related to the context of the entire text that can be calculated within the shared model, may also be input.

In addition, for pairs of morphemes that are estimated to have the same named entity label 407 within two pieces of text data, the name matching model 403 estimates a name matching label 409 indicating whether the two morphemes should be merged.

During estimation, the supervised machine learning module 103 uses the data content 202 of the unlabeled data 101 as input data and outputs named entity labels 407, relationship labels 408, and name identification labels 409. During learning, the data content 302 of the labeled data 102 is used as input, and learning is performed so that the output labels 407, 408, and 409 match the subtask labels 303, 304, and 305, respectively. During this learning, the parameters of each model 105, 401, 402, and 403 are changed using a method such as backpropagation, so that labels 303, 304, and 305 can be output for the data content 302 of the labeled data. During evaluation using labeled data 102, the degree to which the output labels 407, 408, and 409 match the labels 303, 304, and 305 in the labeled data 102 is quantitatively calculated.

Next, the processing details of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 will be explained. Note that in this embodiment, the influence is calculated from the difference in accuracy between when all subtasks are learned and when the subtasks are learned excluding those subtasks, but any calculation method can be used as long as it can output the influence quantitatively. Note that "during learning" refers to when or at the time of learning.

Here, subtask accuracy information 204 is created through the processing of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107. Figure 5 is a diagram showing the subtask accuracy information 204 in this embodiment. Below, the processing of the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 will be explained with reference to Figure 5.

First, the subtask accuracy calculation module 106 and the subtask mutual influence calculation module 107 divide the labeled data 102 into training data and validation data. The subtask accuracy calculation module 106 then performs training on the supervised machine learning module 103 using the training data from the labeled data 102, and evaluates (i.e., calculates) the estimation accuracy when estimating using the validation data. The subtask accuracy calculation module 106 then displays this estimation accuracy (evaluation value) in the column 501 for all subtask training and the row 503 for each subtask accuracy in Figure 5. For example, the accuracy a_NER for named entity extraction during all subtask training is 0.85. This accuracy a is calculated by the subtask accuracy calculation module 106. The accuracies a_NER, a_RE, and a_CR for each subtask are used by the overall priority score calculation module 109, described below. NER, RE, and CR represent named entity extraction, relationship extraction, and name matching, respectively. The subtask accuracy calculation module 106 then records the calculated items described above in the subtask accuracy information 204.

Furthermore, for the three subtasks this time, the subtask accuracy calculation module 106 also calculates the estimation accuracy when only the remaining two subtasks are learned without learning each of them, in the same way as when all subtasks are learned. Then, when the subtask accuracy calculation module 106 calculates, it can record the results in the column 502 for learning when each subtask is excluded and the row 503 for the accuracy of each subtask.

In addition, the difference in accuracy of subtasks other than the excluded subtasks when learning without each subtask and when learning all subtasks is a value that reflects the influence of the excluded subtask on other subtasks. In other words, it indicates the influence calculated by the subtask mutual influence calculation module 107. Here, the average value of the difference (accuracy difference) in the accuracy row 503 of each subtask in column 502 when learning without each subtask and column 501 when learning all subtasks can be recorded in the influence on other subtasks 504. In other words, the subtask mutual influence calculation module 107 calculates the average value of the difference in the accuracy row 503 as the influence. The above calculation of influence c is realized by the subtask mutual influence calculation module 107 as described above. c_NER, c_RE, and c_CR are used in the overall priority score calculation module 109, which will be described later.

Next, we will explain the processing of the subtask priority score calculation module 108. Here, the output of the subtask priority score calculation module 108 is shown in FIGS.
The processing of the subtask priority score calculation module 108 will be described below with reference to FIGS.

In this embodiment, the priority score for named entity extraction is output for each morpheme obtained by morphologically analyzing text data. The maximum value of these scores is then used to determine the priority score for the target text data. The priority score for relationship extraction is calculated for each combination of all named entities in the text data, and the maximum value of these scores is used to determine the priority score for the target text data. For name matching, the priority score for the target morpheme is the average of the priority scores output when determining whether the morpheme to be matched should be matched with other morphemes, and the maximum value of these scores is used to determine the priority score for the target text data.

Figure 6 shows an example of priority score information 205 for unlabeled data 101 with a data ID 201 of 1. Here, the named entity extraction priority score 601 for one piece of data is the calculation result of the priority score for the named entity extraction subtask for one piece of data. In addition, the relationship extraction priority score 602 indicates the value of the relationship priority score for pairs of entities inferred as named entities by named entity extraction. The name identification priority score 603 indicates the name identification priority score for entities inferred as named entities by named entity extraction.

Figure 7 also shows the priority score for each subtask for each data. Here, priority scores 703, 704, and 705 for each subtask are shown for the same data ID 701 and data content 702 as for the unlabeled data 101. Note that in this embodiment, the maximum value of the priority scores for each subtask of one data element shown in Figure 6 is used as the priority score for each subtask of the data, but this may also be the overall average value or the average value of the top few scores. The output, i.e., information, shown in Figures 6 and 7 above may be managed together with the overall priority score as priority score information 205, which will be described below.

Next, the processing of the overall priority score calculation module 109 will be explained. The overall priority score calculation module 109 calculates the priority score and overall priority score for a single piece of data. Therefore, the processing of the overall priority score calculation module 109 will be explained below with reference to this. Note that the priority score and overall priority score for a single piece of data can be managed as priority score information 205.

Figure 9 is a diagram showing the priority score for one piece of data in this embodiment. The overall priority score calculation module 109 integrates the priority scores resulting from each subtask within one piece of data to calculate the priority score for all subtasks. Specifically, the priority score for each element of the data in each subtask is multiplied by the weight calculated for each subtask to calculate the priority score for all subtasks. Note that Figure 6 lists the target data 901 for each listed subtask, the target data for which subtask 902, and the weighted priority score 903.

Figure 10 is a diagram showing the overall priority score for each data item in this embodiment. Figure 10 shows the overall priority score 1006 obtained by weighting and adding up the priority scores 1003, 1004, and 1005 of each subtask for the data ID 1001 and data content 1002 of the unlabeled data 101 obtained in Figure 7. In other words, the overall priority score calculation module 109 calculates the overall priority score 1006 by weighting and adding up the priority scores 1003, 1004, and 1005 of each subtask for the data ID 1001 and data content 1002 of the unlabeled data 101. For this purpose, (Equation 1) is used.

The overall priority score calculation module 109 can calculate the weight of each subtask used in the calculations in Figures 9 and 10 as a function of the estimation accuracy of each subtask and the influence of each subtask on other subtasks. The formula used for this calculation is shown in (Equation 2).

In (Equation 2), by multiplying the result obtained by subtracting the estimated accuracy from 1 by the impact, a higher weight is assigned to subtasks with poor accuracy or subtasks with a large impact on other subtasks. Furthermore, α and β in (Equation 2) are parameters related to accuracy and impact, respectively, and can be adjusted in the user interface described below. Formulas other than (Equation 2) may also be used. In other words, the weight calculation formula is a function of accuracy and impact, and any formula in which α and β are parameters related to accuracy and impact, respectively, will suffice.

Next, the processing flow in this embodiment will be described. Figure 8 is an overall processing flow diagram in this embodiment. First, in step S801, the supervised machine learning module 103 accepts input of labeled data 102. In step S802, the supervised machine learning module 103 divides the accepted labeled data 102 into training data and validation data. Note that steps S802 to S808 may be performed using multiple division patterns, and the accuracy and impact values may use the average values of the multiple patterns.

Furthermore, in step S803, the supervised machine learning module 103 performs learning using the learning data for each subtask module 104. In other words, learning is performed for the subtasks. Furthermore, in step S804, the supervised machine learning module 103 trained in step S803 estimates the validation data.

Furthermore, in parallel with steps S803 and S804, steps S805 and S806 are executed. First, in step S805, the supervised machine learning module 103 learns about the subtask modules 104 that make up itself, excluding one subtask module. In other words, it learns about the subtasks. In this embodiment, it learns about patterns excluding each of the three subtasks: named entity extraction, relationship extraction, and name matching. Furthermore, in step S806, the supervised machine learning module 103 estimates estimated labels based on the learning results of S805.

Furthermore, in step S807, the subtask accuracy calculation module 106 inputs the estimated labels, which are the estimation results for the validation data obtained in S804, and calculates the estimation accuracy of each subtask during all subtask learning. In other words, row 503 of the subtask accuracy during all subtask learning column 501 shown in Figure 5 is calculated.

Furthermore, in step S808, the subtask mutual influence calculation module 107 calculates the influence of each subtask on other subtasks using the estimation results obtained in steps S804 and S806. This calculation method has already been explained using Figure 5. Furthermore, an example of the calculated influence is shown as influence on other subtasks 504 in Figure 5.

In step S809, the supervised machine learning module 103 accepts the unlabeled data 101 and estimates the corresponding labels. This estimation uses the model used during all-task learning. In step S810, the subtask priority score calculation module 108 calculates the priority scores for each data and each subtask based on the estimation results obtained in step S809. Examples of the calculation results are shown in Figures 6 and 7.

Furthermore, in step S811, the overall priority score calculation module 109 calculates an overall priority score based on the estimation accuracy obtained in step S807, the influence obtained in step S808, and the estimation results for each unlabeled data 101 obtained in S810. It is also desirable that the overall priority score calculation module 109 also calculates the priority score for each element within each piece of data. Examples of this calculation result are shown in Figures 9 and 10.

Furthermore, in step S812, the label correction module 110 outputs the estimation results in order of overall priority score. For example, the label correction module 110 outputs the estimation results to a display device, which then displays a confirmation screen including the results. As a result, the annotator can check the estimation results and input corrections as necessary. The label correction module 110 then corrects the estimation results, i.e., the estimated labels, in accordance with the input corrections, etc. The specific contents of the confirmation screen will be described later with reference to Figures 11 and 12.

Next, in step S814, the label correction module 110 determines whether additional data to be checked is required. The determination of whether additional data is required may be made based on input from the annotator, or may be made based on criteria prepared in advance. If the result indicates that additional data is required (Yes), the process proceeds to step S813. If additional data is not required (No), the process proceeds to step S815.

Furthermore, in step S813, the overall priority score recalculation module 111 recalculates the overall priority score. In the recalculation, for the confirmed data, the estimation result is compared with the confirmed and corrected data to calculate the estimation accuracy of each subtask. In response, the overall priority score recalculation module 111 updates the accuracy of each subtask using a weighted average weighted by the estimation accuracy calculated in step S807 and the number of target data items. The overall priority score recalculation module 111 then updates the influence of each subtask on other tasks in step S808 according to the updated estimation accuracy of each subtask. In this way, the overall priority score recalculation module 111 recalculates the weight for the priority score of each subtask in step S811. As a result, the overall priority score can be recalculated to reflect the recalculated weight. Once the overall priority score has been recalculated, the processing of step S812 is executed for data that has not yet been confirmed.

In step S815, the label correction module 110 determines whether re-learning is necessary. Regarding this determination, the label correction module 110 may prepare criteria in advance and make the determination, or may make the determination based on input from the annotator. If re-learning is necessary (Yes), the process proceeds to step S816. If re-learning is not necessary (No), the process ends.

Furthermore, in step S816, the supervised machine learning module 103 adds the confirmed data to the labeled data 102. Then, in step S801, the supervised machine learning module 103 re-trains using the added labeled data 102. Thereafter, the processes of steps S801 to S815 are repeated. This concludes the explanation of this flow diagram.

Next, the confirmation screen displayed on the display device in step S812 will be described. Figures 11 and 12 are diagrams showing confirmation screens in this embodiment. First, Figure 11 is a diagram showing a confirmation screen 1100 for all data to be confirmed in this embodiment. In Figure 11, the confirmation screen 1100 has display areas for the data to be confirmed 1101, data ID 1102, data content 1103, and label 1104. In addition, the confirmation screen 1100 displays designation tools such as an edit button 1105, a total priority score adjustment slide bar 1106, an add confirmation data button 1107, a re-learn button 1108, and a complete button 1109.

First, the data to be confirmed 1101 displays the data to be confirmed and its estimation results. At this time, the unlabeled data 101 and its estimation results are sorted in descending order of the overall priority score 1006 calculated for each data by the overall priority score calculation module 109 and overall priority score recalculation module 111, and a pre-specified number of data items are displayed. Furthermore, data ID 1102 and data content 1103 display information corresponding to the data ID 201 and data content 202 in the unlabeled data 101.

Label 1104 displays the inference result, i.e., the inferred label. At this time, the inference result may be displayed for each subtask, but in this example, the results of named entity extraction, relationship extraction, and name matching are combined to display the result of event extraction inferred from the data. Furthermore, edit button 1105 is a button used to check and edit the details of each piece of data. When the annotator selects edit button 1105, the screen transitions to a details confirmation screen 1200 for each piece of data. The details confirmation screen 1200 after the transition will be described later with reference to Figure 12.

The overall priority score adjustment slide bar 1106 is also used to make adjustments to the overall priority score. In other words, the overall priority score for each piece of data is recalculated according to the values of the slide bars for the subtask accuracy and impact on other subtasks specified by the annotator. To do this, the overall priority score recalculation module 111 changes the values of α and β in (Equation 1) according to the specified values, and recalculates the overall priority score for each piece of data. After recalculation, the overall priority score recalculation module 111 re-sorts the data in order of overall priority score and displays it in the data to be confirmed 1101 on the confirmation screen 1100.

Also, when the annotator presses the Add Confirmation Data button 1107, recalculation is performed in step S813. That is, the overall priority score recalculation module 111 recalculates the overall priority score using the accuracy of the data 1101 to be confirmed up to that point, and the process returns to step S812.

Furthermore, when the annotator presses the Relearn button 1108, the processing of step S816 is executed. That is, the supervised machine learning module 103 adds the verified data to the labeled data 102, and the processing returns to step S801. Furthermore, when the annotator presses the Done button 1109, the processing flow described in Figure 8 ends.

Next, we will explain the details confirmation screen 1200 that is displayed when the edit button 1105 is pressed, and the processing that is performed at that time. Figure 12 is a diagram showing the details confirmation screen 1200 in this embodiment. This example shows the case where the data ID in Figure 11 is 1. In Figure 12, the details confirmation screen 1200 displays display areas for the data body 1201 and confirmation items 1202, as well as a YES button 1203 and a NO button 1204. Note that the details confirmation screen 1200 will also be displayed on the display device described above.

First, the data content is displayed in the data body 1201. In this example, the confirmation items are highlighted, and it is also indicated which parts of the text to focus on.

In addition, confirmation item 1202 displays the content to be confirmed on this screen. This display is performed in the order of the weighted priority score 903 shown in Figure 9. In Figure 12, a display is shown for confirming named entity extraction for "Part A", which has the highest weighted priority score. For this display, it is desirable to pre-determine the question format for each subtask, such as named entity extraction, relationship extraction, and name matching. For example, for named entity extraction, the question format is defined as "Is 'Part A' a part name?"; for relationship extraction, "Is there a relationship between 'Part A' and 'Replace'?"; and for name matching, "Can 'Part A' be rephrased as 'AAA'?".

The YES button 1203 and NO button 1204 accept the annotator's feedback regarding the detailed confirmation. If the YES button 1203 is pressed, the label correction module 110 adds the data to the confirmed data as is. If the NO button 1204 is pressed, the label correction module 110 may save only the information that the label is incorrect as labeled data, or may have the annotator correct the label. As described above, when each button is pressed, the label correction module 110 displays the confirmation item with the next highest priority score in the confirmation item 1202, in accordance with the weighted priority score 903 shown in Figure 9. As a result, the annotator can confirm this content. Furthermore, when a certain number of confirmations have been completed for one confirmation data, the screen returns to the screen shown in Figure 11. This concludes the explanation of the processing in this embodiment.

The annotation device 1 of this embodiment can be configured as an annotation system realized by a cloud system. This configuration will be described below. Figure 13 is a configuration diagram of the annotation system of this embodiment. In this embodiment, a corporate system consisting of the annotation device 1, terminal device 2, corporate terminal device 3, and corporate server 4 is connected via network 5. In other words, the annotation system is configured as a cloud system.

First, the annotation device 1 can be realized by a computer such as a server. To this end, the annotation device 1 has a processing device 11, a communication device 12, a memory 13, and an auxiliary storage device 14, which are connected to each other via a communication path. Furthermore, the annotation device 1 in FIG. 13 is provided with a group of calculation devices 15 that perform calculations (such as voice synthesis) on documents for which an event extraction task has been executed, but this can be omitted. Furthermore, the group of calculation devices 15 may be realized by a dedicated circuit such as an FPGA (Field-Programmable Gate Array), or the processing device 11 described below may perform calculations on documents. Furthermore, calculations on documents may be performed by a device other than the annotation device 1, such as a terminal device 2 or a functional system.

First, the processing device 11 can be realized by a processor such as a CPU (Central Processing Unit), and performs processes such as calculations in accordance with programs stored in the auxiliary storage device 14 (described below). The communication device 12 communicates with other devices via the network 5. The memory 13 stores the programs stored in the auxiliary storage device 14 and information used for processing by the processing device 11.

The auxiliary storage device 14 can be realized as a so-called storage device, and stores the above-mentioned programs and information. The auxiliary storage device 14 may also be realized as various storage media, such as an external HDD (Hard Disk Drive), SSD (Solid State Drive), or memory card. Furthermore, at least one of this information may be stored in a database system other than this device.

In the example shown in Figure 13, each module in Figure 1 is implemented as a computer program. To this end, the auxiliary storage device 14 stores a program consisting of a supervised machine learning module 103, a subtask accuracy calculation module 106, a subtask mutual influence calculation module 107, a subtask priority score calculation module 108, an overall priority score calculation module 109, a label correction module 110, and an overall priority score recalculation module 111. Therefore, the processing device 11 executes the processing of each module in accordance with the computer program. Note that each module may be implemented as an independent program, or a program may be configured by combining parts of these programs.

Furthermore, the auxiliary storage device 14 stores unlabeled data 101, labeled data 102, subtask accuracy information 204, and priority score information 205. These are as described above. This concludes the description of the annotation device 1.

Next, terminal device 2 is a terminal device used by an annotator, and can be realized by a computer such as a PC, tablet, or smartphone. For this reason, it is desirable for terminal device 2 to have the input device and display device described above. In addition, the functions of any of the modules described above may be provided in terminal device 2.

A corporate system is composed of corporate terminal devices 3-1 and 3-2 (hereinafter simply referred to as corporate terminal devices 3) and business servers 4-1 and 4-2 (hereinafter simply referred to as corporate servers 4), and is used to carry out the business of the relevant company. In this embodiment, business is carried out using documents, which are the target data. This embodiment is configured so that the annotation device 1 can be accessed and used from the corporate systems of multiple companies via a network 5. This network 5 can be realized as a wide area network such as the Internet. However, the annotation device 1 may also be constructed as a dedicated system for a specific company.

The company terminal device 3 may also be used by an employee of the company as an annotator. In other words, the company terminal device 3 may perform the same functions as the terminal device 2. In this way, the company terminal device 3 may function as an input/output device for the annotation device 1, or may perform functions for carrying out document-related tasks. Furthermore, the company terminal device 3 may perform both of these functions. This concludes the explanation of Figure 13. The annotation device 1 of this embodiment may be configured as a so-called standalone type, or this function may be provided on the terminal device side, such as a PC.

This concludes the explanation of this embodiment, but the present invention is not limited to the above-mentioned aspects. For example, it can also be applied to data other than documents. Furthermore, in this embodiment, processing is performed on a subtask-by-subtask basis, but it may also be performed on a combination of tasks or subtasks.

101 Unlabeled data 102 Labeled data 103 Supervised machine learning module 104 Subtask module 105 Shared model 106 Subtask accuracy calculation module 107 Subtask mutual influence calculation module 108 Subtask priority score calculation module 109 Overall priority score calculation module 110 Label correction module 111 Overall priority score recalculation module 401 Named entity extraction model 402 Relationship extraction model 403 Name matching model 404 Text data 405 Vector representation 406 Feature vector 407 Named entity label 408 Relationship label 409 Name matching estimation label

Claims (10)

In an annotation device that realizes machine learning labeling,
a supervised machine learning module having a plurality of subtask modules that learn using labeled data and estimate estimated labels for unlabeled data for each of a plurality of subtasks that make up the machine learning task;
a subtask mutual influence calculation module that calculates an influence indicating the degree of influence of learning in the subtask module on other subtask modules;
An annotation device having a label correction module that corrects the estimated label in accordance with a confirmation result of the estimated label presented in accordance with a total priority score based on the influence degree. The annotation device according to claim 1,
Further, a subtask accuracy calculation module is provided for calculating an estimation accuracy indicating the accuracy of the estimated label,
The label correction module corrects the estimated label in accordance with a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy and the influence. The annotation device according to claim 2,
further comprising a subtask priority score calculation module that calculates a subtask priority score relating to the order of confirmation for each data unit of the data handled by the plurality of subtask modules;
The label correction module corrects the estimated label in accordance with a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy, the influence, and the subtask priority score. The annotation device according to claim 3,
The annotation device further includes an overall priority score calculation module that calculates an overall priority score for the unlabeled data as the overall priority score using the estimation accuracy, the influence, and the subtask priority scores. The annotation device according to claim 4,
The annotation device further includes a re-overall priority score calculation module that recalculates the overall priority score in response to correction of the estimated label by the label correction module. An annotation method for realizing machine learning labeling using an annotation device,
A supervised machine learning module having a plurality of subtask modules for estimating estimated labels for unlabeled data is used to learn, using labeled data, each of a plurality of subtasks constituting the machine learning task;
Calculating an influence level indicating the degree of influence of learning in the subtask module on other subtask modules;
The annotation method modifies the estimated label according to a confirmation result of the estimated label presented according to the overall priority score based on the influence. The annotation method according to claim 6,
Furthermore, an estimation accuracy indicating the accuracy of the estimated label is calculated,
An annotation method that modifies the estimated label according to a confirmation result for the estimated label presented according to an overall priority score based on the estimation accuracy and the influence. The annotation method according to claim 7,
Furthermore, a subtask priority score is calculated for each data unit of the data handled by the plurality of subtask modules, and the subtask priority score is calculated for each data unit of the data handled by the plurality of subtask modules.
An annotation method that modifies the estimated label according to a confirmation result of the estimated label presented according to an overall priority score based on the estimation accuracy, the influence, and the subtask priority score. The annotation method according to claim 8,
Furthermore, the annotation method calculates an overall priority score for the unlabeled data using the estimation accuracy, the influence, and the subtask priority score as the overall priority score. The annotation method according to claim 9,
Furthermore, the annotation method recalculates the overall priority score in response to modifications of the estimated label .