JP6989485B2 - Multi-label data learning support device, multi-label data learning support method and multi-label data learning support program - Google Patents

Description

The present invention relates to a multi-label data learning support device, a multi-label data learning support method, and a multi-label data learning support program.

In recent years, in the field of image recognition, systems and services utilizing machine learning (especially deep learning) are increasing. Deep learning (hereinafter referred to as "DL") includes supervised learning in which learning is performed by attaching a correct answer label to image data for learning, and unsupervised learning in which learning is performed without a correct answer label. For example, DL can be used to solve image classification problems that recognize "something" about an object in an input image, failure diagnosis that determines whether an object in an image is "normal or abnormal", and regression problems such as estimating age from a person's image. Supervised learning is often used to solve.

In supervised learning, the more image data is generally used for learning, the better the performance (recognition accuracy, generalization performance, etc.) of the classifier generated after learning. However, supervised learning has a problem that the work (annotation) of labeling image data with a correct answer requires a very large number of man-hours.

In response to the above problem, in the technique described in Patent Document 1, data having a high learning effect is extracted from unlabeled image data, and the image data is presented to an annotation worker (hereinafter referred to as "annotator"). We are proposing a system that uses a method called active learning.

Specifically, first, a classifier is generated using a small amount (for example, tens to hundreds of sheets) of labeled image data, and a large amount (for example, thousands to tens of thousands of sheets) of labels are generated on the generated classifier. None Image data is input and recognition processing is performed. Next, image data having a high learning effect (for example, image data having a large entropy by calculating the entropy of the output value) is extracted by calculation processing based on the output value of the recognition process. The image data having a high learning effect here means the image data in which the result estimated by the discriminator is likely to be incorrect. Then, the annotator assigns the correct label to the image data that is likely to have an incorrect estimation result, generates a classifier again, and uses a method called active learning that gradually improves the performance of the classifier. ..

JP-A-2017-167834

The technique described in Patent Document 1 can be expected to improve the efficiency of annotation for a single label (one correct label for one image data). However, for a multi-label that assigns multiple correct labels to one image data (for example, attaching multiple labels such as age, gender, clothes, etc. to a person's image), a single label is used. Compared with, the man-hours for annotation are doubled by the number of labels. Therefore, the burden on the annotator that assigns the correct label to the image data also increases.

The present invention has been made in view of the above problems, and is capable of reducing the amount of annotation work on image data having multi-labels, a multi-label data learning support device, a multi-label data learning support method, and multi-label data. The challenge is to provide a learning support program.

In order to achieve the above object, the multi-label data learning support device of the present invention includes a labeled image data DB (DataBase) for storing labeled image data, an unlabeled image data DB for storing unlabeled image data, and a plurality of unlabeled image data DBs. The storage unit that stores the label-related information DB that stores the label-related information that shows the relationship between the labels and shows the tendency of the contents of the multi-label attached to the image data, and the storage unit that stores the label-related information DB, and the machine that acquires the labeled image data. A learning unit that generates a classifier by learning, a reasoning unit that performs inference processing for each label that constitutes a multi-label for unlabeled image data using the generated classifier, and no label obtained by inference processing. Annotated image that selects the unlabeled image data as the annotation target image that requires annotation when the inference result of each label for the image data and the relationship between the labels indicated by the label-related information are compared and different. It is provided with a selection unit and an annotation processing unit that displays the selected image to be annotated on the display device and accepts the input of the correct answer label of the image data.

According to the present invention, it is possible to provide a multi-label data learning support device, a multi-label data learning support method, and a multi-label data learning support program that reduce the amount of annotation work on image data having multi-labels.

It is a block diagram which shows the structure of the multi-label data learning support apparatus which concerns on this embodiment. It is a figure which shows the data structure example of the labeled image data information stored in the labeled image data DB which concerns on this embodiment. It is a figure which shows the data composition example of the unlabeled image data information (before inference) stored in the unlabeled image data DB which concerns on this embodiment. It is a figure which shows the data composition example of the unlabeled image data information (after inference) stored in the unlabeled image data DB which concerns on this embodiment. It is a figure which shows the data structure example of the label-related information stored in the label-related information DB which concerns on this embodiment. It is a flowchart which shows the flow of the whole processing executed by the multi-label data learning support apparatus which concerns on this embodiment. It is a flowchart which shows the flow of the annotation target image selection process executed by the annotation image selection part of the multi-label data learning support apparatus which concerns on this embodiment. It is a figure which shows the data composition example of the unlabeled image data information (after the annotation candidate image is determined) stored in the unlabeled image data DB which concerns on this embodiment. It is a figure which illustrates the label-related information definition UI (User Interface) which concerns on this embodiment. It is a figure which illustrates the label-related information history confirmation UI which concerns on this embodiment. It is a figure which illustrates the label-related information definition UI which concerns on this embodiment.

Hereinafter, embodiments of the present invention (hereinafter referred to as “the present embodiment”) will be described in detail with reference to the drawings.

The multi-label data learning support device 1 (described later, see FIG. 1) according to the present embodiment defines the “relationship between labels” of the multi-label, which differs depending on the shooting location of the image, as “label-related information”. By utilizing label-related information when extracting data with a high learning effect for active learning, the amount of annotation work on image data having multiple labels is reduced.
The "relationship between labels" in the present embodiment means the "tendency" of the content of the multi-label attached to the image data to be photographed at the place where the image is photographed. To explain with an example of a person image having three types of multi-labels, age label, gender label, and clothing label, "In this place, a man (gender label) in his twenties (age label) wears a suit (clothing label). "In this place, people in their thirties (age label) who wear skirts (clothes label) are more likely to be women (gender label)." Depending on where you do it, you may find a tendency in the content of the multi-label. This tendency is defined in advance as "label related information".

In the multi-label data learning support device 1, the above label-related information is defined for each shooting location, and when the classifier infers the unlabeled image data, for example, "age: 20s, gender: male," If you infer "clothes: work clothes", the inference for this unlabeled image data is incorrect because the inference is different from the defined tendency that "men in their twenties are likely to wear suits". It is more likely that you are there. For the result inferred by this classifier, the image judged to be highly likely to be incorrect is extracted, and the image data having the multi-label is annotated. As a result, according to the multi-label data learning support device 1, active learning can be performed on an image having a multi-label, and the performance of the classifier can be efficiently improved, so that the work man-hours can be reduced.
Hereinafter, the multi-label data learning support device 1 according to the present embodiment will be described in detail.

<Multi-label data learning support device>
FIG. 1 is a block diagram showing a configuration of a multi-label data learning support device 1 according to the present embodiment.
As shown in FIG. 1, the multi-label data learning support device 1 is composed of a computer including a control unit 10, an input unit 20, an output unit 30, and a storage unit 40.

The input unit 20 has a function of receiving information input via a communication interface (not shown) for transmitting / receiving (inputting) information via a network or the like, or an input device (not shown) such as a touch panel or keyboard. Be prepared. The input of information to the input unit 20 may be realized by a user interface (UI: User Interface) so that the annotator 9 can be visually operated, or may be input by a medium such as a CSV file. good.

The input unit 20 includes an annotation processing unit 21, an image data reading unit 22, and a label-related information definition unit 23.
The annotation processing unit 21 generates labeled image data by receiving information on the correct label given by the annotator 9 or the like to the unlabeled image data. Then, the annotation processing unit 21 stores the generated labeled image data in the labeled image data DB (DataBase) 41 described later in the storage unit 40.

The image data reading unit 22 accepts input of labeled image data and unlabeled image data. Then, when the image data reading unit 22 receives the labeled image data, it generates one record indicated by the labeled image data information 410 (see FIG. 2) described later, and then stores the labeled image data in the storage unit. It is stored in the labeled image data DB 41 in 40. Further, when the image data reading unit 22 receives the unlabeled image data, it generates one record shown in the unlabeled image data information 420 (see FIG. 3) described later, and then stores the unlabeled image data in the storage unit. It is stored in the unlabeled image data DB 42 in 40.
The image data reading unit 22 may accept labeled image data and unlabeled image data one by one by operating the annotator 9, or collectively as image data contained in a specific directory. You may accept it.
Further, the image data reading unit 22 has a function of determining whether or not the image data is labeled when the input of the image data is received. Then, when the image data reading unit 22 is labeled, the image data is stored in the labeled image data DB 41 as the labeled image data. If the image data reading unit 22 is not labeled, the image data reading unit 22 stores the data as unlabeled image data in the unlabeled image data DB 42.

The label-related information definition unit 23 indicates the relationship between the labels from the annotator 9 and the like, and also indicates the "tendency" of the content of the multi-label attached to the image data to be photographed, that is, the label-to-label information. Accepts input of relevance definition information. The label-related information definition unit 23 stores the received label-related information in the label-related information DB 43 in the storage unit 40. The details of the label-related information will be described later with reference to FIG. Further, with reference to FIGS. 9 to 11, a user interface (UI) used when the annotator 9 or the like inputs label-related information will be described.

The output unit 30 has a communication interface (not shown) for transmitting / receiving (outputting) information via a network or the like, and an interface function for outputting information to a display device (not shown) such as a monitor.
The output unit 30 is an image data selected as an image to be annotated from the accuracy evaluation result of the classifier 111 and the unlabeled image data, which are the results obtained based on the functions of the input unit 20 and the control unit 10. The display device displays the screen of the label-related information definition user interface 200 (FIGS. 9 and 11), the screen of the label-related information history confirmation user interface 250 (FIG. 10), and the like, which will be described later.

The storage unit 40 is composed of a hard disk, a flash memory, a RAM (Random Access Memory), and the like. The storage unit 40 stores the labeled image data DB 41, the unlabeled image data DB 42, the label-related information DB 43, and other information necessary for the processing executed by the multi-label data learning support device 1.

FIG. 2 is a diagram showing a data configuration example of the labeled image data information 410 stored in the labeled image data DB 41 according to the present embodiment.
In the labeled image data DB 41, together with the labeled image data itself (captured image), information related to the image data is stored as the labeled image data information 410 for each image data (for each record), for example, in a table format. It is stored in (see Fig. 2).

As shown in FIG. 2, the labeled image data information 410 includes a shooting location 411, a shot image storage destination path 412, and a multi-label for the shot image (here, as an example, gender 413, age 414, clothing). (Upper body) 415 and clothes (lower body) 416 labels) are stored.
For example, as shown in the first line (first record) of the labeled image data information 410 in FIG. 2, "Factory A" is stored as the shooting location 411 of the image data. "A_001.jpg" is stored as the captured image storage destination path 412 indicating the storage destination of the image data. As label information of the label (multi-label) attached to the image data, gender 413 is "male", age 414 is "30s", clothes (upper body) 415 is "work clothes", and clothes (lower body) 416 is "work". Each information of "arrival" is stored.

FIG. 3 is a diagram showing a data configuration example of the unlabeled image data information (before inference) 420 (420A) stored in the unlabeled image data DB 42 according to the present embodiment.
In the unlabeled image data DB 42, along with the unlabeled image data itself (photographed image), information related to the image data is stored as unlabeled image data information 420 as, for example, in a table format (see FIG. 3).
The unlabeled image data information 420 has different data items in a stored state due to the treatment of the multi-label data learning support device 1. FIG. 3 shows a state before (before inference) the inference processing by the control unit 10 (inference unit 13) described later, which is the stage where the image data reading unit 22 acquires the unlabeled image data and stores it in the storage unit 40. ing. The information of the unlabeled image data in the pre-inference state is shown as reference numeral 420A in FIG.

As shown in FIG. 3, the unlabeled image data information (before inference) 420 (420A) includes the shooting location 421, the shot image storage destination path 422, and the multi-label for the shot image (gender 423, age) as the data items. 424, clothes (upper body) 425, clothes (lower body) 426 labels), annotation candidates 427, and relevance degree 428 are stored.
The shooting location 421, the shooting image storage destination path 422, and the multi-label (gender 423 to clothes (lower body) 426) have the same contents as the labeled image data information 410 shown in FIG. However, the field of each label in the multi-label is blank (“−”). Annotation candidate 427 and relevance degree 428 are data items required when selecting an annotation image (“annotation target image” described later) to execute annotation from unlabeled image data, and details will be described later.

Returning to FIG. 1, the control unit 10 controls the entire multi-label data learning support device 1, and includes a learning unit 11, an evaluation unit 12, an inference unit 13, and an annotation image selection unit 14. ..

The learning unit 11 uses the labeled learning data to generate the classifier 111 by using machine learning such as DL. Specifically, the learning unit 11 selects a predetermined amount of image data (for example, 90% of the image data) from the labeled image data stored in the labeled image data DB 41, and the classifier 111 is subjected to machine learning. To generate. The learning unit 11 selects a predetermined amount of learning data from the labeled learning data, for example, at random.

The evaluation unit 12 evaluates the accuracy of the classifier 111 generated by the learning unit 11. Specifically, the evaluation unit 12 uses the labeled image data that the learning unit 11 does not use to generate the classifier 111, and the performance of the classifier 111 reaches a predetermined recognition accuracy (target recognition accuracy). Determine if it has been done.
Information on the predetermined recognition accuracy may be described in a parameter file or the like in advance and stored in the storage unit 40, or the input unit 20 inputs from the annotator 9 or the like when the multi-label data learning support device 1 is started. It may be received and stored in the storage unit 40.

When the evaluation unit 12 determines that the performance of the classifier 111 does not reach a predetermined recognition accuracy, the reasoning unit 13 uses the discriminator 111 generated by the learning unit 11 at that time to generate an unlabeled image. Inference processing is performed for all unlabeled image data stored in the data DB 42.

FIG. 4 is a diagram showing a data configuration example of the unlabeled image data information (after inference) 420 (420B) stored in the unlabeled image data DB 42 according to the present embodiment.
The inference unit 13 uses the classifier 111 to infer (estimate) each label (multi-label) for each unlabeled image data, and uses the result as unlabeled image data information of the record corresponding to the unlabeled image data. Store. In FIG. 4, the inference results of the labels of the gender 423, the age 424, the clothes (upper body) 425, and the clothes (lower body) 426 are stored in the unlabeled image data information 420 as label information. The information of the unlabeled image data in the post-inference state is shown as reference numeral 420B in FIG.

Returning to FIG. 1, the annotation image selection unit 14 uses the result of deduction by the inference unit 13 for each label of the unlabeled image data and the label-related information 430 (see FIG. 5) stored in the label-related information DB 43. , Select a photographed image (hereinafter referred to as "annotation target image") to be preferentially annotated.

First, the annotation image selection unit 14 refers to the label-related information DB 43 and reads the label-related information used for selecting the annotation image. Here, when the label-related information is undefined, the annotation image selection unit 14 accepts the input of the label-related information from the annotator 9 or the like via the label-related information definition unit 23. The details of the definition of the label-related information will be described with reference to FIGS. 9 to 11 (label-related information definition user interface, label-related information history confirmation user interface) described later.

FIG. 5 is a diagram showing a data configuration example of the label-related information 430 stored in the label-related information DB 43 according to the present embodiment.
The label-related information 430 has ID431, a shooting location 432, and a multi-label for a shot image as its data items (here, for example, gender 433, age 434, clothes (upper body) 435, clothes (lower body) 436 labels). , Description label 437, target label 438, and relevance degree 439 are stored.

The ID 431 stores a unique identifier attached to the label-related information (1 record). The shooting place of the shot image is stored in the shooting place 432. Gender 433, age 434, clothing (upper body) 435, and clothing (lower body) 436 are examples of multi-labels (each label). The explanatory label 437 (first label) and the target label 438 (second label) define which labels are related to each other, as shown below. Further, the degree of association 439 defines the strength of the association between the explanatory label 437 and the target label 438 (for example, in the range of 0 to 1.0).

For example, as shown in the record of ID431 "A_001", the explanatory label 437 is defined as "gender" and "age", the target label 438 is defined as "clothes (upper body)", and the relevance degree 439 is defined as "0.9". , The record of "A_001" is defined as "At Factory A, 90% of men in their twenties wear work clothes on their clothes (upper body)". Also, like the record with ID 431 of "A_005", the explanation label 437 is "age", "clothes (upper body)", "clothes (lower body)", the target label 438 is "gender", and the relevance degree 439 is "0.8". When defined as, the record of "A_005" has the definition content that "80% of people in their 50s who can shoot at Factory A and whose clothes (upper and lower body) are suits are men". For irrelevant labels, the field may be blank (“−”), as in the clothing (lower body) 436 of the record “A_001”.

On the other hand, when the label-related information 430 is defined, the annotation image selection unit 14 records the same shooting location 432 as the image data to be annotated from the label-related information 430 stored in the label-related information DB 43. To extract. Then, the annotation image selection unit 14 selects unlabeled image data (annotation target image) to be annotated preferentially using the result of the inference processing by the inference unit 13 and the extracted label-related information 430. The process of selecting the annotation target image by the annotation image selection unit 14 will be described in detail with reference to FIG. 7 described later.

The annotation image selected by the annotation image selection unit 14 is output to a display device or the like by the output unit 30. Then, the information of the correct answer label given by the annotator 9 or the like is received, and the newly labeled image data is stored in the labeled image data DB 41. At this time, the unlabeled image data selected as the annotation image is deleted from the unlabeled image data DB 42.
The learning unit 11 of the multi-label data learning support device 1 improves the performance of the classifier 111 by further performing machine learning including the added labeled image data to generate the classifier 111, and the classifier 111 The above process is repeated until a predetermined recognition accuracy is reached.

<Processing flow>
Next, the flow of processing executed by the multi-label data learning support device 1 will be described. First, the entire flow will be described with reference to FIG. Then, the details of the selection process of the annotation target image will be described with reference to FIG. 7.

FIG. 6 is a flowchart showing the flow of the entire process executed by the multi-label data learning support device 1 according to the present embodiment.
It is assumed that the storage unit 40 of the multi-label data learning support device 1 stores information of a predetermined recognition accuracy (target recognition accuracy) for determining the performance of the classifier 111 in advance.

First, the image data reading unit 22 of the multi-label data learning support device 1 accepts input of labeled image data and unlabeled image data (step S1). Then, the image data reading unit 22 generates the labeled image data information 410 shown in FIG. 2 for the received labeled image data, and then stores it in the labeled image data DB 41. Further, the image data reading unit 22 generates the unlabeled image data information (before inference) 420 (420A) shown in FIG. 3 for the received unlabeled image data, and then stores it in the unlabeled image data DB 42.

Here, for example, the multi-label data learning support device 1 has a small amount (for example, tens to hundreds of sheets) of labeled image data and a large amount (for example, thousands to tens of thousands of sheets) of unlabeled image data. accept. If the image data received by the image data reading unit 22 is all unlabeled image data, the annotation processing unit 21 randomly selects and outputs the image data from the received unlabeled image data. By outputting to the unit 30, the annotator 9 or the like is made to execute the annotation, and the image data with the level with the correct answer label is generated.

Next, the learning unit 11 generates the classifier 111 by machine learning such as DL using the labeled learning data stored in the labeled image data DB 41 (step S2). Here, the learning unit 11 randomly selects a predetermined amount (for example, 90%) of learning data from the labeled image data, and generates the classifier 111 by machine learning.

Subsequently, the evaluation unit 12 uses the labeled image data that the learning unit 11 does not use to generate the classifier 111, and determines whether or not the performance of the classifier 111 is equal to or higher than a predetermined recognition accuracy (target recognition accuracy). Is determined (step S3). Here, when the performance of the classifier 111 is equal to or higher than the predetermined recognition accuracy (target recognition accuracy) (step S3 → Yes), the process is terminated assuming that the learning of the classifier 111 is completed. On the other hand, if the predetermined recognition accuracy is not reached (step S3 → No), the process proceeds to the next step S4.

In step S4, the inference unit 13 executes inference processing for each label of all the image data stored in the unlabeled image data DB 42 by using the classifier 111 generated in step S2. Then, the inference unit 13 stores the information of each label as the inference result in the unlabeled image data information (after inference) 420 (420B) (see FIG. 4).

Next, the annotation image selection unit 14 refers to the label-related information DB 43 (FIG. 1), and determines whether or not the label-related information 430 (see FIG. 5) used for selecting the annotation image has been defined (step S5). ).
Here, for example, the annotation image selection unit 14 sets the same shooting location as the shooting location 421 of the unlabeled image data information 420 (FIG. 4) in the item of the shooting location 432 of the label-related information 430 (FIG. 5). Whether or not it has been defined can be determined depending on whether or not it has been defined.

Then, when the label-related information 430 is defined (step S5 → Yes), the annotation image selection unit 14 reads the defined label-related information (step S6). Specifically, the annotation image selection unit 14 selects, for example, a record of the same shooting location 432 as the image data to be annotated from the label-related information 430 (FIG. 5) stored in the label-related information DB 43. Extract as label related information to be used this time. Note that the annotation image selection unit 14 may receive information that specifies the information of ID 431 among the information shown in the label-related information 430 (FIG. 5), and may use it as the label-related information to be used this time.
On the other hand, if the annotation image selection unit 14 has not been defined (step S5 → No), the annotation image selection unit 14 performs a process of defining label-related information (step S7). Details regarding the definition of label-related information will be described later (see FIGS. 9 to 11).

Next, the annotation image selection unit 14 preferentially annotates the unlabeled image data (annotation) using the inference result of each label for the unlabeled image data performed in step S4 and the defined label-related information. The target image) is selected (step S8: annotation target image selection process). The details of this annotation target image selection process will be described later with reference to FIG. 7.

Subsequently, the annotation image selection unit 14 displays the selected annotation target image on the screen of the display device via the output unit 30 (step S9). Then, the annotation processing unit 21 accepts the input of the correct answer label from the annotator 9 and the like. That is, the annotation of the selected image to be annotated is executed (step S10).

The annotation processing unit 21 adds image data (labeled image data) to which a correct answer label is newly added to the labeled image data DB 41, and deletes the information of the image data from the unlabeled image data DB 42 (. Step S11).
At this time, the annotation processing unit 21 returns the information stored in the fields of each label of the unlabeled image data information (after inference) 420 (420B) (see FIG. 4) to the state before inference in FIG. You may reset it, or you may keep it as it is and overwrite it every time the inference process is repeated.

Subsequently, the multi-label data learning support device 1 returns to step S2 and repeats the above processing until the performance of the classifier 111 becomes equal to or higher than a predetermined recognition accuracy (target recognition accuracy) (see step S3).

<Annotation target image selection process>
Next, the annotation target image selection process executed by the annotation image selection unit 14 of the multi-label data learning support device 1 will be described. The annotation target image selection process is a process executed in step S8 of FIG.
FIG. 7 is a flowchart showing the flow of the annotation target image selection process executed by the annotation image selection unit 14 of the multi-label data learning support device 1 according to the present embodiment.

First, the annotation image selection unit 14 acquires the defined label-related information 430 (FIG. 5) (step S81).
Then, the annotation image selection unit 14 selects one unlabeled image data from the unlabeled image data DB 42 (step S82).

Subsequently, the annotation image selection unit 14 compares the inference result of the selected unlabeled image data (see FIG. 4) with the defined label-related information, that is, is the inference result different from the defined label-related information? It is determined whether or not (step S83).
Specifically, the record of the selected unlabeled image data of the unlabeled image data information (after inference) 420 (420B) shown in FIG. 4 is compared with the record of the label-related information 430 of FIG. Search for records in which the label information of 437 (first label) matches but the label information of the target label (second label) is different. For example, a record whose captured image storage path 422 is "A_101.jpg" in FIG. 4 has an explanatory label (gender, age) and label information of a record whose ID 431 is "A_002" in FIG. 5 is "gender: male, age:". Although they match in their thirties, the label information of the target label (clothes (upper body)) is different.

In step S83, the label information of the explanatory label 437 matches the record of the unlabeled image data information (after inference) 420 (420B) and the record of the label-related information 430, but the label information of the target label 438 is different. When a different record is found (step S83 → Yes), the unlabeled image data is determined as an annotation candidate image (step S84).
For the unlabeled image data determined as the annotation candidate image, a flag (for example, "YES") is given to the annotation candidate 427 as shown in the unlabeled image data information (after the annotation candidate image is determined) 420 (420C) in FIG. "). Further, the annotation image selection unit 14 has a label-related information 430 (FIG. 5) determined to be different in the column of the relevance degree 428 of the unlabeled image data information (after the annotation candidate image is determined) 420 (420C). The value of the degree of association 439 stored in is extracted and stored. Here, the value "0.9" of the relevance degree 439 in the record in which the ID 431 of FIG. 5 is "A_002" is the value of the relevance degree 428 of the record in which the captured image storage destination path 422 of FIG. 8 is "A_101.jpg". It is stored as "0.9".

On the other hand, if the corresponding record is not found in step S83 (step S83 → No), the annotation candidate 427 is shown in the unlabeled image data information (after the annotation candidate image is determined) 420 (420C) in FIG. For example, the flag of is set to "NO" (without setting the flag), and the process proceeds to the next step S85.

In step S85, the annotation image selection unit 14 determines whether or not all of the unlabeled image data stored in the unlabeled image data DB 42 has been processed. Then, if there is unlabeled image data that has not been processed yet (step S85 → No), the process returns to step S82, and processing is continued for the next unlabeled image data. On the other hand, when all the processing of the unlabeled image data is completed (step S85 → Yes), the process proceeds to step S86.

In step S86, the annotation image selection unit 14 determines whether or not the number of the determined annotation candidate images exceeds a predetermined threshold value. The predetermined threshold value regarding the number of annotation candidate images is a threshold value for limiting the number of unlabeled image data for executing annotation processing, and is stored in the storage unit 40 in advance.
Then, when the predetermined threshold value is not exceeded (step S86 → No), the annotation image selection unit 14 selects all of the annotation candidate images as annotation target images for executing annotation (step S87), and performs processing. Finish.

On the other hand, when the predetermined threshold is exceeded (step S86 → Yes), the annotation image selection unit 14 refers to the unlabeled image data information (after the annotation candidate image is determined) 420 (420C) (FIG. 8) and annotates. Among the records of unlabeled image data on which the candidate 427 flag is set, annotation candidate images up to the number reaching a predetermined threshold in descending order of the value of the degree of association 428 are selected as annotation target images to be annotated (step S88). ), Finish the process.
The reason for selecting in descending order of the relevance degree 428 is that the higher the relevance value is, the stronger the relationship between the explanatory label and the target label is, so that the inference result is different from the defined label-related information. In some cases, the reasoning is likely to be wrong.

<User interface>
Next, a user interface (hereinafter, referred to as “UI”) provided by the label-related information definition unit 23 of the multi-label data learning support device 1 according to the present embodiment will be described. The label-related information definition unit 23 displays the label-related information definition UI 200 (see FIGS. 9 and 11) and the label-related information history confirmation UI 250 on the display device via the output unit 30, and the label-related information is displayed by the annotator 9 or the like. The input of the setting information of the definition regarding is received and stored as the label-related information 430 (FIG. 5) in the label-related information DB 43.

FIG. 9 is a diagram illustrating the label-related information definition UI 200 according to the present embodiment. The annotator 9 uses this label-related information definition UI 200 when setting a new or additional definition of label-related information.

First, the annotator 9 selects a captured image at a photographing location that defines label-related information by reading an image (menu) 201. The selected captured image is displayed in the image display area 202. Further, the shooting location of the selected shot image is displayed in the shooting location text box 203. FIG. 9 shows an example in which "Factory A" is displayed as a shooting location. Here, when newly defining label-related information, a unique ID is automatically displayed in the ID text box 204. If this ID is unique, the annotator 9 or the like may freely set it.

Next, a method for setting the definition of label-related information performed by the annotator 9 or the like will be described.
The label-related information definition unit 23 accepts selection by the label pull-down 205 for the label information of the related label. FIG. 9 shows an example in which "male" is selected in the gender label. Then, the selection of whether the selected label is an explanatory label (“description”) (first label) or a target label (“purpose”) (second label) is accepted by the related type pull-down 206. At this time, one or more items are selected for each of the explanatory label and the target label.

Next, the annotator 9 sets the degree of relevance of the label-related information. The relevance may be set by the relevance slider 207 or by the relevance text editor 208.

The defined label-related information is newly registered in the label-related information 430 (FIG. 5) by pressing the registration button 209. The registered information can also be output in a CSV file format or a binary file format.
Further, the output label-related information can be loaded into the label-related information DB 43 (see FIG. 1) by selecting a file from the label-related information definition read (menu) 210. When the history (button) 211 is pressed, the loaded label-related information can be displayed as the label-related information history confirmation UI 250 in FIG. 10, and the label-related information can be confirmed in a list.
The annotator 9 specifies the label-related information displayed on the label-related information history confirmation UI 250 of FIG. 10 with a mouse or the like (for example, a record of ID “A_001” is specified), so that the label is as shown in FIG. It is possible to confirm or modify the definition contents on the related information definition UI 200.

As described above, the annotator 9 can input the label-related information by the label-related information definition UI 200 by a visually easy-to-understand and simple method. Therefore, it is possible to reduce the burden on the annotator 9 regarding the definition of label-related information.

As described above, according to the multi-label data learning support device 1, the multi-label data learning support method, and the multi-label data learning support program according to the present embodiment, the multi-label "between labels" that differs depending on the shooting location of the image. By defining "relevance" in advance as label-related information, it is possible to select an image to be annotated by utilizing this label-related information when extracting data having a high learning effect in active learning. Therefore, it is possible to efficiently perform annotation even in image data having multiple labels, that is, to reduce the amount of annotation work.

In this embodiment, the shooting location is "Factory A", and as an example of adding multi-labels of "gender", "age", "clothes (upper body)", and "clothes (lower body)" to the photographed person. explained. However, the present invention is not limited to this, and for example, for an object (industrial product, food, etc.) photographed in a visual inspection process in a factory, for example, "type of scratch (scratch)", "position of scratch", "dirt". It can also be used when adding a multi-label related to the above. At this time, the relationship and tendency between the labels regarding "type of scratch", "position of scratch", "dirt", etc. are defined as label-related information. For example, if the type of scratch includes line scratches, pinholes, scratches, dents, etc., "In the visual inspection line (photographing location) of product B, the line scratches (type of scratches) are on the upper surface (scratch position). ), Red foreign matter (dirt) is attached. ”, The label-related information is defined. As described above, the object to be photographed is not limited to a person, and the present invention can be applied even when a multi-label is attached to various objects and the like.

Further, the "shooting place" defined by the label-related information according to the present invention is not limited to a place (area) such as a factory, a commercial facility, a park, or a station, and if the shooting area shown in the image is specified. Often, it can be arbitrarily defined by the annotator 9 depending on the target to which the multi-label is added. For example, the inside of a store, the entrance of a store, a conference room, one process (visual inspection) of a production line, an area where a part of a product is specified and photographed, and the like can be set as a "photographing place" as a photographing area.

Further, the present invention can also be realized by a program (multi-label data learning support program) that operates the hardware resources of a general computer as each function of the multi-label data learning support device 1. The program (multi-label data learning support program) can be distributed via a communication line, or can be recorded and distributed on a recording medium such as a CD-ROM.

1 Multi-label data learning support device 10 Control unit 11 Learning unit 12 Evaluation unit 13 Reasoning unit 14 Annotation image selection unit 20 Input unit 21 Annotation processing unit 22 Image data reading unit 23 Label-related information definition unit 30 Output unit 40 Storage unit 41 Label Attached image data DB
42 Unlabeled image data DB
43 Label-related information DB
111 Discriminator 200 Label Related Information Definition User Interface (UI)
250 Label-related information History confirmation user interface (UI)
410 Labeled image data information 420 Unlabeled image data information 430 Label-related information 437 Description label (first label)
438 Purpose label (second label)

Claims (5)

It is a multi-label data learning support device that supports learning of a classifier that assigns multi-labels indicating multiple labels to image data.
A labeled image data DB (DataBase) that stores labeled image data, an unlabeled image data DB that stores unlabeled image data, and a multi that shows the relationship between the plurality of labels and is attached to the image data. A storage unit that stores a label-related information DB that stores label-related information that indicates the tendency of the contents of the label, and a storage unit that stores the label-related information DB.
A learning unit that acquires the labeled image data and generates the classifier by machine learning.
An inference unit that performs inference processing for each label constituting the multi-label on the unlabeled image data using the generated discriminator.
When the inference result of each label for the unlabeled image data obtained by the inference processing and the relationship between the labels shown in the label-related information are compared and different, the unlabeled image data is imaged. An annotation image selection unit that selects as an annotation target image that indicates an image that requires annotation, which is the work of labeling data correctly.
The selected image to be annotated is displayed on the display device, the input of the correct answer label of the image data is accepted, and the image data with the accepted correct answer label is stored in the labeled image data DB as the labeled image data. An annotation processing unit to store and
A multi-label data learning support device characterized by being equipped with. The label-related information is provided for each shooting location indicating the shooting area of the image data using the first label and the second label, in which one or more labels are selected from the labels constituting the multi-label. It is information in which the first label and the second label are associated with each other.
The annotation image selection unit extracts the information of the first label and the information of the second label from the information of each label obtained as the inference result of the unlabeled image data, and stores the information as the label-related information. The multi-label data learning support device according to claim 1, wherein the comparison is performed depending on whether or not the relationship between the labels indicated by the information of the second label associated with the first label matches. Label-related information that displays an input screen for defining a label corresponding to each of the first label and the second label among the labels constituting the multi-label on the display device and accepts the input of the label-related information. The multi-label data learning support device according to claim 2, further comprising a definition unit. It is a multi-label data learning support method of a multi-label data learning support device that supports learning of a classifier that assigns a multi-label indicating a plurality of labels to image data.
The multi-label data learning support device is
A labeled image data DB that stores labeled image data, an unlabeled image data DB that stores unlabeled image data, and the contents of a multi-label attached to the image data while showing the relationship between the plurality of labels. It is provided with a storage unit for storing a label-related information DB that stores label-related information indicating the tendency of the above.
The step of acquiring the labeled image data and generating the classifier by machine learning, and
A step of inferring each label constituting the multi-label on the unlabeled image data using the generated classifier, and
When the inference result of each label for the unlabeled image data obtained by the inference processing and the relationship between the labels shown in the label-related information are compared and different, the unlabeled image data is referred to as an image. The step of selecting an image that requires an annotation, which is the work of labeling the data correctly, as an image to be annotated, and
The selected image to be annotated is displayed on the display device, the input of the correct answer label of the image data is accepted, and the image data with the accepted correct answer label is stored in the labeled image data DB as the labeled image data. Steps to remember and
A multi-label data learning support method characterized by executing. A multi-label data learning support program for operating a computer as the multi-label data learning support device according to any one of claims 1 to 3.

Images