JP7673757B2 - ANNOTATION DEVICE, ANNOTATION METHOD, AND ANNOTATION PROGRAM - Google Patents

Description

The present invention relates to an annotation device, an annotation method, and an annotation program.

Traditionally, supervised learning in machine learning requires training data and corresponding correct labels. In many studies, multiple people view the data and assign metadata (annotation).

For example, when annotating audio or video, a worker (sometimes called an "annotator") listens to a few to several tens of seconds of the audio or video provided and adds metadata to meet the specifications. Specifically, if the annotation is for research and development of emotion recognition from audio, the worker would select the emotion most appropriate for the audio heard, and if it is object detection or recognition for an image, the worker would select the area of the object in the image and add a description of the object.

Conventional annotation methods can be divided into those that have a comparison target for the task and those that do not. When there is no comparison target, the annotator watches still images or a few seconds of audio or video and assigns metadata. This method has a low time cost because the number of times data is viewed = the total number of samples N. In addition, accurate annotation can be performed for tasks that can be reliably understood by anyone even in a short time (e.g., transcription, tagging objects, or a state in which the person is clearly angry no matter who sees or hears it).

On the other hand, when there is something to compare, annotators can listen to audio or video for a long period of time (tens of seconds to several minutes) and assign metadata about continuous and relative changes in events (see, for example, Non-Patent Document 3), or listen to multiple audio or video clips and assign relative rankings or scores (see, for example, Non-Patent Document 4). This method reduces variation between annotators and allows for more accurate metadata to be assigned, since there is something to compare.

Mohammad Soleymani, and Martha Larson, “Crowdsourcing for Affective Annotation of Video: Development of a Viewer-reported Boredom Corpus”, 2010. Ryutaro Tanno, Ardavan Saeedi, Swami Sankaranarayanan, Daniel C. Alexander, and Nathan Silberman, “Learning From Noisy Labels By Regularized Estimation Of Annotator Confusion”, 2019. David Melhart, Antonios Liapis, and Georgios N. Yannakakis “PAGAN: Video Affect Annotation Made Easy”, 2019. Lifang Yang, and Rui Zhu, “Subjective Evaluation of Cooling Fan Sound based on Grade Scoring and Paired Comparison”, 2016.

However, the above-mentioned conventional techniques cannot perform low-cost and highly accurate annotation in supervised learning in machine learning. This is because annotation methods without a comparison target cannot perform accurate annotation for tasks that are difficult to understand through short viewing, and there is a large variance in the labels assigned by annotators, resulting in low reliability of annotation results.

To address this issue, there are approaches such as taking into consideration the quality of the annotators, answer trends, and reducing the effects of noise by having multiple people annotate, but these do not fundamentally solve the problem of increasing the reliability of the annotation itself (see, for example, Non-Patent Documents 1 and 2). For example, when annotating the degree of concentration, if the person appears to be very focused or not focused at all, that is, if it is clear to anyone who sees or hears, then the votes of multiple annotators are likely to match, but if it is difficult to tell whether someone is focused or not, accurate annotation is difficult, and as a result, subtle differences cannot be expressed.

On the other hand, annotation methods without a comparison target require viewing a long time or a large amount of data, which requires huge costs for annotation. For example, when viewing several combinations of data simultaneously, if n pieces of data are selected from all N samples, a maximum of _N C _n combinations may exist. It may be possible to reduce the number of combinations while maintaining the quality of annotation by referring to psychological experiment methods, but careful consideration is required to determine which combinations to exclude.

In order to solve the above-mentioned problems and achieve the objective, the annotation device of the present invention is characterized in that it comprises an acquisition unit that acquires first learning data to be used in machine learning, a first distribution unit that distributes the first learning data acquired by the acquisition unit to a plurality of annotators, a classification unit that classifies the first learning data based on the reliability of a first correct label that is assigned to the first learning data by each annotator, and a second distribution unit that distributes the classification result of the first learning data classified by the classification unit.

The annotation method according to the present invention is an annotation method executed by an annotation device, and is characterized in that it includes an acquisition step of acquiring first learning data to be used in machine learning, a first distribution step of distributing the first learning data acquired by the acquisition step to a plurality of annotators, a classification step of classifying the first learning data based on the reliability of a first correct label assigned to the first learning data by each annotator, and a second distribution step of distributing the classification result of the first learning data classified by the classification step.

The annotation program of the present invention is characterized in that it causes a computer to execute an acquisition step of acquiring first learning data to be used in machine learning, a first distribution step of distributing the first learning data acquired by the acquisition step to a plurality of annotators, a classification step of classifying the first learning data based on the reliability of a first correct label assigned to the first learning data by each annotator, and a second distribution step of distributing the classification result of the first learning data classified by the classification step.

The present invention enables lower-cost and more accurate annotation in supervised learning in machine learning.

FIG. 1 is a diagram showing an example of the configuration of an annotation system according to the first embodiment. FIG. 2 is a block diagram showing an example of the configuration of the annotation device according to the first embodiment. FIG. 3 is a diagram illustrating an example of learning data according to the first embodiment. FIG. 4 is a diagram illustrating an example of first learning data and a first correct label according to the first embodiment. FIG. 5 is a diagram illustrating an example of second learning data and a second correct label according to the first embodiment. FIG. 6 is a flowchart showing an example of the flow of the annotation process according to the first embodiment. FIG. 7 is a flowchart showing an example of the flow of the first learning data classification process according to the first embodiment. FIG. 8 is a diagram illustrating a computer that executes a program.

Below, an annotation device, an annotation method, and an annotation program according to the present invention will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

First Embodiment
The configuration of the annotation system according to this embodiment, the configuration of the annotation device, a specific example of the annotation process, the flow of the annotation process, and the flow of the data classification process will be described below in this order, and finally the effects of this embodiment will be described.

[Configuration of annotation system]
The configuration of an annotation system (or "this system" as appropriate) 100 according to this embodiment will be described in detail with reference to Fig. 1. Fig. 1 is a diagram showing an example of an annotation system according to a first embodiment. The annotation system 100 includes an annotation device 10 such as a server, annotators 20 (20A, 20B, 20C) such as various terminals, and various databases 30 (30A, 30B, 30C).

Here, the annotation device 10, the annotator 20, and the database 30 are connected to each other via a predetermined communication network (not shown) so as to be able to communicate with each other by wire or wirelessly. Note that the annotation system 100 shown in FIG. 1 may include multiple annotation devices 10.

First, the annotation device 10 acquires learning data required for research and development as first learning data from various databases 30 (step S1). Here, the acquired learning data is data such as audio, images, and videos, and is acquired in a medium and on a scale appropriate to the purpose of the research and development.

Next, the annotation device 10 distributes the acquired first learning data to the annotator 20 (step S2). Here, the annotator 20 is a terminal and a user of the terminal that respectively assign correct answer labels to the distributed learning data, but is not limited thereto. The annotator 20 may be a machine learning model that can assign a specific correct answer label that has been created separately.

Next, the annotator 20 assigns a correct label (first correct label) to the distributed first learning data (step S3). The annotation device 10 also acquires the first learning data to which the correct label has been assigned (step S4).

Thereafter, the annotation device 10 classifies the first learning data based on the first correct answer label (step S5). At this time, the annotation device 10 selects the learning data to which reliable correct answer data has been assigned based on the answer obtained from the annotator 20 as a reference point (suitably, "reference data") S. The annotation device 10 also further classifies the learning data other than the reference point S into data to which it is easy to assign an accurate correct answer label (suitably, "data D") and data to which it is difficult to assign an accurate correct answer label (suitably, "data E").

Furthermore, the annotation device 10 generates second learning data from the classified first learning data (step S6). At this time, the annotation device 10 generates a data group including the reference point S, data E, and data D that have the same source. The classification of the first learning data and the generation of the second learning data will be described later.

Then, the annotation device 10 distributes the generated second learning data to the annotator 20 (step S7). At this time, when distributing the data group of the second learning data, the annotation device 10 distributes each data to the annotator 20 so that the user views the reference point S, followed by data E and data D. The annotator 20 also assigns a correct answer label (second correct answer label) to the distributed second learning data (step S8). Finally, the annotation device 10 acquires the second learning data to which the correct answer label has been assigned (step S9).

In the annotation system 100 according to this embodiment, the annotation device 10 includes reliable data for the event to which the correct label is to be assigned in the data group and indicates it clearly. This allows the annotator 20 to use the data as a comparison target, enabling more accurate annotation.

[Configuration of annotation device]
The configuration of the annotation device 10 according to this embodiment will be described in detail with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the configuration of the annotation device according to this embodiment. The annotation device 10 has an input unit 11, an output unit 12, a communication unit 13, a storage unit 14, and a control unit 15.

The input unit 11 is responsible for inputting various types of information to the annotation device 10. The input unit 11 is, for example, a mouse or a keyboard, and accepts input of setting information, etc. to the annotation device 10. The output unit 12 is responsible for outputting various types of information from the annotation device 10. The output unit 12 is, for example, a display, and outputs setting information, etc. stored in the annotation device 10.

The communication unit 13 is responsible for data communication with other devices. For example, the communication unit 13 performs data communication with each communication device. The communication unit 13 can also perform data communication with an operator's terminal (not shown).

The memory unit 14 stores various information referenced when the control unit 15 operates and various information acquired when the control unit 15 operates. Here, the memory unit 14 is, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Note that in the example of FIG. 2, the memory unit 14 is installed inside the annotation device 10, but it may be installed outside the annotation device 10, or multiple memory units may be installed.

The memory unit 14 stores the first learning data obtained from the database 30 described below, the first learning data to which a first correct answer label obtained from the annotator 20 has been assigned, the classification results classified by the classification unit 15c of the control unit 15, the second learning data generated by the generation unit 15d, the second learning data to which a second correct answer label obtained from the annotator 20 has been assigned, etc., as well as information on the annotator 20, such as the user name and the identification number of the machine learning model.

The control unit 15 is responsible for controlling the entire annotation device 10. The control unit 15 has an acquisition unit 15a, a first distribution unit 15b, a classification unit 15c, a generation unit 15d, and a second distribution unit 15e. Here, the control unit 15 is, for example, an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

The acquisition unit 15a acquires first learning data used for machine learning. For example, the acquisition unit 15a acquires first learning data including audio, images, or videos. The acquisition unit 15a also acquires the first learning data from the database 30. The acquisition unit 15a also acquires learning data to which a correct answer label has been assigned from the annotator 20. The acquisition unit 15a further stores the first learning data, the learning data to which a correct answer label has been assigned, etc. in the memory unit 14.

The first distribution unit 15b distributes the first learning data acquired by the acquisition unit 15a to multiple annotators 20. For example, the first distribution unit 15b distributes the first learning data in a format in which a predetermined number is assigned as a first correct answer label. The first distribution unit 15b also distributes the first learning data to a machine learning model as an annotator 20. Detailed processing of the first learning data and the first correct answer label will be described later.

The classification unit 15c classifies the first learning data based on the reliability of the first correct label assigned to the first learning data by each annotator. For example, the classification unit 15c classifies the first learning data into reference data, data to which the correct label is easily assigned accurately, and data to which the correct label is difficult to assign accurately, based on the variance of the first correct label as the reliability. The classification unit 15c also classifies the first learning data based on the posterior probability of the first correct label as the reliability. Furthermore, the classification unit 15a stores the calculation result of the reliability of the first correct label and the classification result based on the reliability in the storage unit 14.

Here, when the annotator 20 is a human, the reliability is the variance of the numerical values of the correct labels of each annotator for certain training data, but is not particularly limited thereto. The index used for the reliability may be one that represents the variance of the numerical values, and the smaller the variance of the numerical values, the higher the reliability of the correct labels. Also, when the annotator 20 is a machine learning model, the reliability is the posterior probability of the numerical values that are the estimation results of the machine learning model for certain training data, but is not particularly limited thereto. The index used for the reliability may be one that represents the accuracy of the estimation results of the machine learning model, and the higher the accuracy of the estimation results, the higher the reliability of the correct labels.

The generating unit 15d generates, as a classification result, second learning data that includes a plurality of reference data having different extreme values, data that is easy to accurately assign a correct answer label to, and data that is difficult to accurately assign a correct answer label to, and the data are a data group whose source is the same. Furthermore, the generating unit 15d stores the classification result of the second learning data, etc., in the memory unit 14.

Here, the extreme values are, for example, the minimum number "1" and the maximum number "5" in the case of learning data in a format in which the degree of a particular state such as concentration is judged as a correct label using a number on a 5-point scale {1, 2, 3, 4, 5}, but are not particularly limited thereto. The extreme values are not limited to the minimum or maximum value of the range of numbers set in advance as the correct label, as long as they are numbers that indicate that the annotator 20 can clearly judge that the state is extreme.

The second distribution unit 15e distributes the classification results of the first learning data classified by the classification unit 15c. For example, the second distribution unit 15e first distributes multiple reference data with different extreme values. In addition, the second distribution unit 15e distributes the classification results to multiple annotators who distributed the first learning data, or to a specified annotator other than the multiple annotators who distributed the first learning data.

Here, the classification result is the first learning data classified by the classification unit 15c based on the reliability of the assigned correct answer label, and is, for example, learning data labeled with three categories: reference point S (reference data), data E (data to which it is easy to assign an accurate correct answer label), and data D (data to which it is difficult to assign an accurate correct answer label), but is not particularly limited thereto. The classification result may be learning data labeled with the reliability of the correct answer label, or may be learning data selected by the generation unit 15d.

[Specific example of annotation processing]
A specific example of the annotation process of the annotation device 10 according to the present embodiment will be described with reference to Fig. 3 to Fig. 5. Fig. 3 is a diagram showing an example of training data according to the first embodiment. Fig. 4 is a diagram showing an example of first training data and a first correct label according to the first embodiment. Fig. 5 is a diagram showing an example of second training data and a second correct label according to the first embodiment.

(First annotation process)
First, the first annotation process from acquisition of the first learning data to acquisition of the first learning data to which the first correct answer label is assigned will be described. First, the first learning data acquired by the annotation device 10 from the database 30 or the like will be described with reference to FIG. 3. Here, the acquired first learning data is data such as audio, images, videos, etc., and is data acquired in a medium and on a scale according to the purpose of research and development. For example, when performing annotation to realize concentration level estimation from audio, the annotation device 10 acquires audio data from an audio database that stores audio data in the database 30.

Fig. 3 shows a data set X that holds voice data, and the data set X includes voice data { _x0 , _x1 , _x2 , _x3 , _x4 , ... _xN }. The voice data shown in Fig. 3 represents a voice waveform as a relationship between the passage of time and the voice signal strength.

In the following explanation, annotation processing using audio data as the first training data will be described, but the type of training data is not particularly limited. The first training data may be image data, video data, or a combination thereof in addition to audio data. Furthermore, the first training data may be data obtained by digitizing or converting the above-mentioned audio data, etc., into text.

Next, the first learning data that the annotation device 10 distributes to the annotator 20 and the first correct label assigned to the first learning data that the annotation device 10 acquires from the annotator 20 will be described with reference to FIG. 4. For example, when performing annotation to realize concentration level estimation from voice, the annotation device 10 pre-sets five levels of concentration levels ("1": not concentrated, "2": slightly not concentrated, "3": neither concentrated nor unconcentrated (flat), "4": slightly concentrated, "5": concentrated), and distributes the first learning data to the annotator 20, which assigns a correct label indicating which concentration level is most appropriate for each voice data held by the dataset X.

When annotation device 10 performs annotation to estimate concentration level from voice, it distributes learning data for judging, for example, from voice data regarding questions and dialogue between a teacher and a student during class, whether the student to whom a question was asked was concentrating on the class, or not, and how concentrated he or she was, on a five-point scale. When annotation device 10 assigns correct answer labels regarding concentration level from image data or video data, it may distribute learning data for judging the concentration level by having annotator 20 read the facial expressions of students from images or videos during class.

Then, the annotation device 10 acquires the first learning data to which the correct labels have been assigned by the annotator 20. In Fig. 3, the correct labels for the speech data " _x0 " to " _xN " of the data set X of "annotator 01" to "annotator 03" are shown (see "ANNOT1(X)" in Fig. 4). For example, the correct labels assigned by "annotator 01" to "annotator 03" to the speech data _x0 are "2", "1", and "1", respectively.

(Second annotation process)
Secondly, the second annotation process from classification of the first learning data assigned with the first correct label to acquisition of the second learning data assigned with the second correct label will be described. First, a specific example of classification process of the first learning data based on the reliability of the correct label acquired from the annotator 20 will be described with reference to Fig. 4. The annotation device 10 calculates the average and variance values of the correct label assigned to each piece of speech data.

In Fig. 4, _x0 has a mean of 1.3 and a variance of 0.3 (small variance). Similarly, _x1 has a mean of 5 and a variance of 0 (all annotators' answers match), _x2 has a mean of 1 and a variance of 0 (all annotators' answers match), _x3 has a mean of 3.3 and a variance of 0.3 (small variance), _x4 has a mean of 4.0 and a variance of 1.0 (large variance), and _xN has a mean of 1.6 and a variance of 1.3 (large variance).

At this time, the annotation device 10 selects, from the answers acquired from the annotator 20, learning data to which reliable correct answer data has been assigned as a reference point S. In the example of Fig. 4, _x1 (extreme value "5") and _x2 (extreme value "1") are selected as the learning data in which the answers of all annotators match and the numerical values of the assigned correct answer labels are extreme values.

Furthermore, the annotation device 10 further classifies the learning data other than the reference point S into data D, which is easy to accurately assign a correct label to, and data E, which is difficult to accurately assign a correct label to. For example, the annotation device 10 sets a reliability threshold, and classifies data D if the variance is 1.0 or more, and classifies data E if not. In the example of Fig. 4, the annotation device 10 classifies _x0 and _x3 into data E because their variances are less than 1.0, and classifies _x4 and _xN into data D because their variances are 1.0 or more.

When the annotation device 10 uses the estimation results from a separately created machine learning model as the correct label, it classifies, for example, learning data in which the posterior probability of the estimated numerical value is 80% or more as reference point S, learning data in which the posterior probability is 50% or more but less than 80% as data E, and learning data in which the posterior probability is less than 50% as data D. Furthermore, the annotation device 10 can statically or dynamically change the classification method, such as the number of classifications and thresholds, of the learning data.

Next, a specific example of the generation and distribution process of the second learning data based on the classification of the first learning data will be described with reference to FIG. 5. The annotation device 10 generates a data group including three types of data, namely, a reference point S, data E, and data D, as the second learning data. At this time, the reference point S is always ensured to include data of "1" and "5", which are the respective extreme values. In addition, the source of the data for each data group (speaker, person appearing in a video, object, etc.) is assumed to be the same.

For example, the annotation device 10 generates a data group set P having data groups { _p0 , _p1 , ..., _pM } as elements. Here, the data group _p0 includes { _x0 , _x1 , _x2 , _x3 , _x4 , _xN } (see Figs. 3 and 4) as elements, and the data group _pM includes { _xa , xb, _xc , _xd , _xe , _{xf }} (not shown in Figs. 3 and 4) as elements. In the example of Figure 5, the annotation device 10 selects reference point S{ _x1 , _x2 }, data E{ _x0 , _x3 }, and data D{ _x4 , _xN } as data group _p0 , and selects reference point S{ _xa , _xb }, data E{ _xc , _xd }, and data D{ _xe , _xf } as data group _pM .

The number of data in each data group and the selection method can be changed arbitrarily as long as they include reference points S that are different extreme values as described above and satisfy the condition of being from the same source. For example, the number of data may be a random number within a certain range. Also, the included data may be two each of data E and data D, and the average value of each annotation result may be closer to "1" or "5".

Thereafter, the annotation device 10 distributes the data group selected as described above to the annotator 20 as the second learning data. At this time, when distributing the data group of the second learning data, the annotation device 10 distributes the reference point S for each data group first, and then distributes the data E and the data D to the annotator _20. In the example of Fig. 5, the annotation device 10 distributes the reference point S{ _x1 , _x2 }, the data E{ _x0 , _x3 }, and the data D{ _x4 , _xN } in this order when distributing the data group _p0 , and distributes the reference point S{ _xa , _xb }, the data E{ _xc , _xd }, and the data D{ _xe , _xf } in this order when distributing the data group pM.

The annotation device 10 may instruct the annotator 20 to view the data in the order of data E and data D. The annotation device 10 may also distribute data E and data D to be viewed randomly. Furthermore, the annotation device 10 may present a correct label for the reference point S that was distributed first at the same time as distribution and instruct the annotator 20 not to assign a correct label, or may instruct the annotator 20 to assign a correct label to all learning data regardless of the classification of the learning data.

Finally, the annotation device 10 acquires the second learning data to which the correct labels have been assigned by the annotator 20. In the example of Fig. 5, the annotation device 10 acquires the correct labels of data E and data D excluding the reference point S for each data group for each annotator, and for example, for "annotator 01", the annotation device ₁₀ acquires the correct labels of { ₁ , _{4, 3} , 2} for the learning data of {x0, x3, x4 _{, xN} } in the data group p0, and acquires the correct labels of {2, 4 _, 3, 3} for the learning data of { _xc , _xd , _xe , _xf } in the data group _pM (see "ANNOT2(X)" in Fig. 5).

Note that there is no particular limitation on the final processing of the correct labels assigned to the second training data. The annotation device 10 may take a majority vote for each training data and determine the most common correct label as the final correct label, or may calculate the average score of the numerical values and determine the numerical value as the final correct label.

[Annotation process flow]
The flow of annotation processing according to this embodiment will be described in detail with reference to Fig. 6. Fig. 6 is a flowchart showing an example of the flow of annotation processing according to the first embodiment.

First, the acquisition unit 15a of the annotation device 10 acquires first learning data including audio, images, videos, etc. from the database 30, etc. (step S101). At this time, the acquisition unit 15a may acquire the first learning data from the memory unit 14. The acquisition unit 15a may also process the original data, such as audio data, acquired from the database 30 or the memory unit 14, and divide the original data into an appropriate size as learning data or perform appropriate classification. Furthermore, the acquisition unit 15a may acquire audio data, etc. from the outside via the input unit 11.

Next, the first distribution unit 15b distributes the first learning data to the annotator 20 (step S102). At this time, the first distribution unit 15b may select the annotator 20 to distribute the first learning data according to the first learning data. In addition, the acquisition unit 15a acquires the first learning data to which the first correct answer label has been assigned by the annotator 20 (step S103).

Then, the classification unit 15c classifies the first learning data based on the reliability of the first correct label (step S104). The generation unit 15d generates second learning data from the classified first learning data (step S105). Next, the second distribution unit 15e distributes the second learning data to the annotator 20 (step S106).

In addition, the second distribution unit 15e can also distribute the second learning data to an annotator other than the annotator 20 that distributed the first learning data. For example, the second distribution unit 15e can distribute the first learning data to a human annotator and distribute the second learning data to an annotator that is a machine learning model.

Finally, the acquisition unit 15a acquires the second learning data to which the second correct answer label has been assigned by the annotator 20 (step S107), and the process ends. Note that if the accuracy of the acquired second correct answer label is not sufficient, the processes of steps S104 to S107 may be performed again.

[Flow of classification process of first learning data]
The flow of the classification process of the first learning data according to the present embodiment will be described in detail with reference to Fig. 7. Fig. 7 is a flowchart showing an example of the flow of the classification process of the first learning data according to the first embodiment. First, the acquisition unit 15a of the annotation device 10 acquires the first correct answer label assigned to the first learning data from the annotator 20 (step S201). Next, when the annotator 20 is a human (step S202: annotator is a human), the classification unit 15c performs the classification process of steps S208 to S210 based on the processes of steps S203 to S205.

If the answers of all annotators match (step S203: YES) and the answers are extreme values (step S204: YES), the classification unit 15c classifies the first learning data to which the correct answer label has been assigned as the reference point S (step S208). If the answers of the annotator 20 include ones that do not match (step S203: NO) or if the answers of the annotator 20 are not extreme values (step S204: NO), the classification unit 15c performs the process of step S205.

If the variance of the annotator 20's answer is 1.0 or more (step S205: Yes), the classification unit 15c classifies the first learning data to which the correct answer label has been assigned as data E (step S209). If the variance of the annotator 20's answer is less than 1.0 (step S205: No), the classification unit 15c classifies the first learning data to which the correct answer label has been assigned as data D (step S210). When the classification process of steps S208 to S210 is completed, the classification unit 15c ends the process.

On the other hand, if the annotator 20 is a machine learning model (step S202: annotator is a machine learning model), the classification unit 15c performs classification processing of steps S208 to S210 based on the processing of steps S206 to S207. If the posterior probability of the value that is the estimated result of the annotator 20 is 80% or more (step S206: Yes), the classification unit 15c classifies the first learning data to which the correct label has been assigned as the reference point S (step S208).

Furthermore, if the posterior probability of the value that is the estimation result of the annotator 20 is less than 80% (step S206: negative) and the posterior probability is 50% or more (step S207: positive), the classification unit 15c classifies the first learning data to which the correct label has been assigned as data E (step S209). Furthermore, if the posterior probability of the value that is the estimation result of the annotator 20 is less than 50% (step S207: negative), the classification unit 15c classifies the first learning data to which the correct label has been assigned as data D (step S210). When the classification process of steps S208 to S210 is completed, the classification unit 15c ends the process.

[Effects of the First Embodiment]
First, in the annotation process according to the present embodiment described above, first learning data used in machine learning is acquired, the acquired first learning data is distributed to a plurality of annotators, the first learning data is classified based on the reliability of the first correct answer labels respectively assigned to the first learning data by each annotator, and the classification results of the classified first learning data are distributed. Therefore, in this process, annotation can be performed at a lower cost and with higher accuracy in supervised learning in machine learning.

Secondly, in the annotation process according to the present embodiment described above, first learning data including audio, images or videos is acquired, the first learning data is distributed in a format in which a predetermined number is assigned as a first correct label, and the first learning data is classified into reference data, data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the variance of the first correct label as the reliability. Therefore, in the present process, in supervised learning in machine learning, it is possible to assign a highly reliable correct label even in the absence of a comparison target, and annotation can be performed at a lower cost and with higher accuracy.

Thirdly, in the annotation process according to the present embodiment described above, the annotator distributes first learning data to the machine learning model, and classifies the first learning data based on the posterior probability of the first correct label as the reliability. Therefore, in this process, in supervised learning in machine learning, even if the annotator is not human, it is possible to assign a highly reliable correct label, and annotation can be performed at a lower cost and with higher accuracy.

Fourth, in the annotation process according to the present embodiment described above, as a classification result, second learning data is generated that is a data group that includes multiple reference data with different extreme values, data that is easy to accurately assign a correct answer label, and data that is difficult to accurately assign a correct answer label to, and each data source is the same, and the multiple reference data are distributed first. Therefore, in supervised learning in machine learning, this process enables reliable and efficient assignment of correct answer labels even in the absence of a comparison target, and can perform annotation at a lower cost and with higher accuracy.

Fifth, in the annotation process according to the present embodiment described above, the classification results are distributed to the multiple annotators who distributed the first learning data, or to a specific annotator other than the multiple annotators who distributed the first learning data. Therefore, in supervised learning in machine learning, this process enables reliable, efficient, and more flexible assignment of correct answer labels even in the absence of a comparison target, and enables annotation to be performed at lower cost and with higher accuracy.

[System configuration, etc.]
Each component of each device shown in the figures according to the above embodiment is a functional concept, and does not necessarily have to be physically configured as shown in the figures. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figures, and all or a part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

〔program〕
It is also possible to create a program in which the processing executed by the annotation device 10 described in the above embodiment is written in a language executable by a computer. In this case, the same effect as in the above embodiment can be obtained by the computer executing the program. Furthermore, such a program may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read and executed by a computer to realize the same processing as in the above embodiment.

Figure 8 is a diagram showing a computer that executes a program. As illustrated in Figure 8, the computer 1000 has, for example, a memory 1010, a CPU (Central Processing Unit) 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070, and each of these components is connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012, as illustrated in FIG. 8. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090, as illustrated in FIG. 8. The disk drive interface 1040 is connected to a disk drive 1100, as illustrated in FIG. 8. For example, a removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, as illustrated in FIG. 8. The video adapter 1060 is connected to a display 1130, as illustrated in FIG. 8.

8, the hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, the above programs are stored, for example, in the hard disk drive 1090 as program modules in which instructions to be executed by the computer 1000 are written.

In addition, the various data described in the above embodiment are stored as program data, for example, in memory 1010 or hard disk drive 1090. Then, CPU 1020 reads out program module 1093 and program data 1094 stored in memory 1010 or hard disk drive 1090 into RAM 1012 as necessary, and executes various processing procedures.

In addition, the program module 1093 and program data 1094 related to the program are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via a disk drive or the like. Alternatively, the program module 1093 and program data 1094 related to the program may be stored in another computer connected via a network (such as a LAN (Local Area Network), WAN (Wide Area Network)), and read by the CPU 1020 via the network interface 1070.

The above embodiments and their variations are included in the scope of the invention and its equivalents described in the claims, as well as in the technology disclosed in this application.

REFERENCE SIGNS LIST 10 Annotation device 11 Input unit 12 Output unit 13 Communication unit 14 Storage unit 15 Control unit 15a Acquisition unit 15b First distribution unit 15c Classification unit 15d Generation unit 15e Second distribution unit 20, 20A, 20B, 20C Annotator 30, 30A, 30B, 30C Database 100 Annotation system

Claims (14)

an acquisition unit that acquires first learning data to be used in machine learning;
a first distribution unit that distributes the first learning data acquired by the acquisition unit to a plurality of annotators;
a classification unit that classifies the first training data based on a reliability of a first correct label that is assigned to each of the first training data by each annotator;
a second distribution unit that distributes a classification result of the first learning data classified by the classification unit to the plurality of annotators or a predetermined annotator other than the plurality of annotators ;
The acquisition unit acquires the first learning data including audio, images, or videos,
The first distribution unit distributes the first learning data in a format in which a predetermined number is assigned as the first correct label;
The classification unit classifies the first learning data into reference data, data that is easy to accurately assign a correct label, or data that is difficult to accurately assign a correct label based on the variance of the first correct label as the reliability. The first distribution unit distributes the first training data to a machine learning model as the annotator;
The annotation device according to claim 1 , wherein the classification unit classifies the first learning data based on a posterior probability of the first correct label as the reliability. a generation unit that generates second learning data as the classification result, the second learning data including a plurality of reference data having different extreme values, the data to which the correct label is easily assigned accurately, and the data to which the correct label is difficult to assign accurately, the second learning data being generated from a same data group;
The annotation device according to claim 1 , wherein the second distribution unit distributes the plurality of reference data first. An annotation method performed by an annotation device, comprising:
An acquisition step of acquiring first learning data to be used in machine learning;
a first distribution step of distributing the first training data acquired by the acquisition step to a plurality of annotators;
a classification step of classifying the first training data based on the reliability of a first correct label assigned to each of the first training data by each annotator;
a second distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
The acquiring step acquires the first learning data including audio, an image, or a video;
The first distribution step distributes the first learning data in a format in which a predetermined number is assigned as the first correct label;
The annotation method is characterized in that the classification step classifies the first learning data into reference data, data that is easy to accurately assign a correct label, or data that is difficult to accurately assign a correct label based on the variance of the first correct label as the reliability. An acquisition step of acquiring first learning data used in machine learning;
a first distribution step of distributing the first training data acquired by the acquisition step to a plurality of annotators;
a classification step of classifying the first training data based on a reliability of a first correct label assigned to each of the first training data by each annotator;
a second distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ;
The acquiring step acquires the first learning data including audio, an image, or a video;
The first distribution step distributes the first learning data in a format in which a predetermined number is assigned as the first correct label;
The classification step classifies the first learning data into reference data, data that is easy to accurately assign a correct label, or data that is difficult to accurately assign a correct label based on the variance of the first correct label as the reliability. a classification unit that classifies first learning data used in machine learning based on the reliability of first correct labels that are respectively assigned by a plurality of annotators to the first learning data;
a distribution unit that distributes a classification result of the first learning data classified by the classification unit to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
The annotation device, wherein the reliability represents a variance of the first correct label. a classification unit that classifies first learning data used in machine learning based on the reliability of first correct labels that are respectively assigned by a plurality of annotators to the first learning data;
a distribution unit that distributes a classification result of the first learning data classified by the classification unit to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
An annotation device characterized in that the classification result includes reference data selected based on the reliability. a classification unit that classifies first learning data used in machine learning based on the reliability of first correct labels that are respectively assigned by a plurality of annotators to the first learning data;
a distribution unit that distributes a classification result of the first learning data classified by the classification unit to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
The annotation device, characterized in that the classification result includes classification of the first learning data into data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the reliability. An annotation method performed by an annotation device, comprising:
a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
The annotation method, wherein the reliability represents a variance of the first correct label. An annotation method performed by an annotation device, comprising:
a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
An annotation method, characterized in that the classification result includes reference data selected based on the reliability. An annotation method performed by an annotation device, comprising:
a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ,
The annotation method, characterized in that the classification result includes classification of the first learning data into data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the reliability. a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ;
The annotation program, wherein the reliability represents a variance of the first correct label. a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ;
The annotation program, wherein the classification result includes reference data selected based on the reliability. a classification step of classifying first learning data used for machine learning based on the reliability of first correct labels respectively assigned by a plurality of annotators to the first learning data;
a distribution step of distributing the classification result of the first learning data classified by the classification step to the plurality of annotators or a predetermined annotator other than the plurality of annotators ;
The annotation program, characterized in that the classification result includes classification of the first learning data into data that is easy to accurately assign a correct label to, or data that is difficult to accurately assign a correct label to, based on the reliability.

Images