JP6916849B2 - Information processing equipment, information processing methods and information processing programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and an information processing program.

Conventionally, there is an image recognition technique that recognizes an object or the like in an image by using machine learning. For example, in the medical field, there is a technique for recognizing a lesion or the like from various medical images such as an X-ray image or a CT image of a human body. Further, in the field of the manufacturing industry, there is a technique for recognizing a foreign substance mixed in a product or a defect of the product from an image of the product taken (see, for example, Patent Document 1).

In machine learning, annotations may be performed to learn the judgment result judged by a human in advance as teacher data. For example, in the annotation in recognizing the text content, text data such as words and idioms for classifying the text are annotated as teacher data.

Further, in the annotation in image recognition, image data for classifying images is annotated as teacher data. For example, in image recognition that classifies and recognizes a specific object part and other parts included in an image, the image part of the specific object is annotated as a "normal example", or the other image part is ". Annotate as "negative example". The result of annotation is called the annotation result. The learning model (engine) of machine learning can classify and recognize a specific object included in an image and others based on the annotation result of a positive example or the annotation result of a negative example.

In addition, a large number of accurately annotated teacher data may be required to improve recognition accuracy in machine learning. In particular, the amount of data per image is increasing due to the improvement in image quality of captured images in recent years. Therefore, a computer with high processing power may be required to analyze a large amount of high-quality teacher data.

JP-A-2018-032071

For example, in the medical field, when annotating a part such as a lesion included in a photographed image, the person performing the annotation is required to have a high degree of specialization in being able to recognize the lesion or the like by looking at the photographed image. For this reason, when there is a shortage of highly specialized humans (for example, doctors), the load may be concentrated on highly specialized humans in the annotation work, and the workload may increase.

In addition, if the classification result of the image by machine learning is incorrect, it is necessary for a human to manually correct the classification result. Further, if the processing power of the computer is not sufficient, the machine learning process takes time. For this reason, the person who makes the correction cannot correct the classification result until the machine learning process is completed, and the workability is reduced. The decrease in workability may increase the work time of the person performing the annotation, resulting in an increase in the workload.

The present invention has been made in view of the above circumstances, and one object of the present invention is to provide an information processing apparatus, an information processing method, and an information processing program capable of reducing the workload of annotation.

(1) In order to solve the above problems, the information processing device provides a user interface (UI) that displays an image processed in machine learning and enables the user to annotate the image. Classify images by learning the acquisition section that acquires the annotation result that the user specifies the area for the image displayed on the UI provided by the section and the providing section, and the annotation result acquired by the acquisition section. An information processing device including a prediction unit for predicting The annotation result is acquired, the prediction unit predicts the first image classification based on the first annotation result acquired by the acquisition unit, and the providing unit predicts the second image including the first image classification predicted by the prediction unit. The UI to be displayed is provided, the acquisition unit acquires the second annotation result in which the user specifies the area for the second image, and the prediction unit further learns the second annotation result acquired in the acquisition unit. The second image classification is predicted, and the providing unit provides a UI for displaying the third image including the second image classification predicted by the prediction unit.

(2) Further, in the information processing apparatus of the embodiment, the acquisition unit acquires the second annotation result in which the user modifies the classification for the first image classification in the second image and specifies the area. The prediction unit may further learn the second annotation result in which the classification of the first image classification is modified, and predict the second image classification in which the classification of the first image classification is modified.

(3) Further, in the information processing apparatus of the embodiment, the acquisition unit acquires the annotation result including the size of the area specified by the user, and the prediction unit is the area for predicting the classification of the image according to the size. The size may be changed.

(4) Further, in the information processing apparatus of the embodiment, the acquisition unit may acquire the annotation result including the length of the area specified by the user as the size.

(5) Further, in the information processing apparatus of the embodiment, the prediction unit extracts the feature amount of the area specified by the user based on the annotation result, and predicts the classification of the image of the area similar to the extracted feature amount. It may be a thing.

(6) Further, in the information processing apparatus of the embodiment, the prediction unit determines the average value of the pixel information, the dispersion value of the pixel information, or the dispersion of the pixel information with edge enhancement from the area specified by the user. The feature amount may be extracted based on at least one of the values.

(7) Further, in the information processing apparatus of the embodiment, the prediction unit sets each weight of the average value of the pixel information, the dispersion value of the pixel information, and the dispersion value of the pixel information with edge enhancement. The feature amount may be extracted based on.

(8) In order to solve the above problems, the information processing method is an information processing method in an information processing device, and an image processed in machine learning is displayed so that the user can annotate the image. A provision step that provides a user interface (UI) to be processed, an acquisition step in which the user acquires an annotation result in which an area is specified for an image displayed on the UI provided in the provision step, and an acquisition step. An information processing method including a prediction step of learning an annotation result and predicting the classification of an image, the step of providing a UI for displaying the first image, and a user area for the first image. Provides a step of acquiring the first information processing result in which is specified, a step of predicting the first image classification based on the acquired first information processing result, and a UI for displaying the second image including the predicted first image classification. A step of acquiring a second information processing result in which the user specifies an area for the second image, a step of further learning the acquired second information processing result, and a step of predicting the second image classification. Includes a step of providing a UI for displaying a third image that includes the second image classification.

(9) In order to solve the above problems, the information processing program displays an image processed in machine learning on a computer and provides a user interface (UI) that enables the user to annotate the image. By learning the provided function to be provided, the acquisition function to acquire the annotation result in which the user specifies the area for the image displayed on the UI provided in the provided function, and the annotation result acquired in the acquisition function, It is an information processing program for realizing a prediction function for predicting the classification of images. The provided function provides a UI for displaying the first image, and the acquisition function is provided by the user for the first image. The first annotation result in which the area is specified is acquired, the prediction function predicts the first image classification based on the first annotation result acquired in the acquisition function, and the providing function predicts the first image classification predicted in the prediction function. The UI for displaying the second image including the above is provided, the acquisition function acquires the second annotation result in which the user specifies the area for the second image, and the prediction function is the second acquired in the acquisition function. The annotation result is further learned to predict the second image classification, and the providing function provides a UI for displaying the third image including the second image classification predicted by the prediction function.

According to one embodiment of the present invention, the user acquires the first annotation result in which the area is specified for the first image, and predicts and predicts the first image classification based on the acquired first annotation result. A UI for displaying the second image including the first image classification is provided, the second annotation result in which the user specifies the area for the second image is acquired, and the acquired second annotation result is further learned. By providing a UI that predicts the second image classification and displays the third image including the predicted second image classification, the workload of annotation can be reduced.

It is a block diagram which shows an example of the software structure of the information processing apparatus in embodiment. It is a flowchart which shows an example of the operation of the information processing apparatus in embodiment. It is a block diagram which shows an example of the hardware composition of the information processing apparatus in embodiment. It is a figure which shows the brush stroke (A) and the prediction (B) of an image classification in 1st Example. It is a figure which shows the brush stroke (A) and the prediction (B) of an image classification in the 2nd Example. It is a figure which shows the brush stroke (A) and the prediction (B) of an image classification in a 3rd Example. It is a figure which shows the initial annotation in 4th Example. It is a figure which shows the prediction of the image classification in 4th Example. It is a figure which shows the correction annotation in 4th Example. It is a figure which shows the prediction of the repredicted image classification in the 4th Example. It is a figure which shows the division of the area in 4th Example. It is a figure which shows the original image (A), the annotation result (B), and the prediction (C) of an image classification in 5th Example.

Hereinafter, the information processing apparatus, the information processing method, and the information processing program according to the embodiment of the present invention will be described in detail with reference to the drawings.

First, the function of the information processing apparatus will be described with reference to FIG. FIG. 1 is a block diagram showing an example of a software configuration of an information processing device according to an embodiment.

In FIG. 1, the information processing device 1 is connected to the display device 2 and the input device 3. Further, the information processing device 1 is communicably connected to the server 4 via the network 9. The display device 2 is a device that displays a user interface (UI) that enables annotation by a user on an image processed in machine learning, and is, for example, a liquid crystal display. The input device 3 is a device in which a user of the information processing device 1 (hereinafter, may be simply referred to as a “user”) inputs an instruction in annotation, and is, for example, a keyboard, a mouse, a touch panel, or the like. The server 4 communicates data with the information processing device 1. For example, the server 4 may provide image data to be processed by the information processing device 1 or store the processing result processed by the information processing device 1. Although FIG. 1 illustrates the configuration of the system including the information processing device 1, the configuration of the system in the embodiment is not limited to this. For example, it may be a system in which a plurality of information processing devices 1 are connected to the server 4. Further, the network 9 is a communication path via wire or wireless, and the communication means such as the communication protocol is arbitrary.

The information processing device 1 has each functional unit of a providing unit 11, an acquisition unit 12, a storage unit 13, a prediction unit 14, and a communication control unit 15. Each of the above-mentioned functional units of the information processing apparatus 1 in the present embodiment will be described as being a functional module realized by the information processing program (software) in the present embodiment.

The providing unit 11 provides a UI that enables the user to annotate the image (processed image) processed in machine learning. Annotation is a work by a user who creates teacher data for a processed image, and in the present embodiment, the area to be teacher data in the processed image is classified into binary or multi-valued (“image classification”). In some cases), it means the work specified in. In the annotation, tagging is performed to link the feature amount of the image in the specified area with the tag indicating the image classification.

Here, the feature amount of the image is, for example, an average value of pixel information, a dispersion value of pixel information, or a dispersion value of pixel information with edge enhancement. The pixel information is, for example, the RGB value of the pixel. The RGB values represent, for example, each color of red, green, and blue as a numerical value from 0 to 255. The pixel information may be information such as a numerical value of each color in another color space (for example, YMCK value, HSV value, etc.).

Tagging is to add metadata indicating image classification to the feature amount of an image. For example, in the annotation for recognizing a specific object included in the processed image, a region of the image of the specific object in the processed image is specified, and the feature amount of the image in that region is "positive". Tag the image classification shown in "Example". In addition, tagging of image classification indicating a "negative example" is performed on the feature amount of the image in the other region. In the following embodiment, an annotation that specifies a binary image classification of a "positive example" region and a "negative example" region is performed on the image to be processed, but a multi-valued image is illustrated. It does not exclude the implementation of annotations that specify classification.

The UI provided by the providing unit 11 is a region where the user is a positive example (hereinafter, may be abbreviated as "positive example") or a negative example area (hereinafter, "negative example") with respect to the image to be processed. May be omitted.) Enables annotation to specify the image classification. For example, in the annotation of the image recognition process for recognizing a specific object in an image, the image area of the specific object is designated as a positive image classification, and the image area other than the object is negative. Designated as an example image classification.

In addition, the providing unit 11 displays the image to be processed and provides a UI including a designation tool for designating an image classification of a positive example or a negative example with respect to the image to be processed. The designated tool is, for example, a tool similar to a paint brush in a paint application. In the paint application, the user can move the cursor in the image to be processed and draw the trajectory of the cursor. The thickness of the locus can be changed, for example, according to the size or thickness of the cursor. Further, the length of the locus can be changed according to the moving distance of the cursor. The UI allows you to specify the image classification of an area by the same operation as a paint brush in a paint application. The designation of the image classification of the area is performed, for example, by the user operating a mouse or the like on the image to be processed displayed on the display device 2. That is, the user can arbitrarily specify the classification of positive and negative examples by drawing with a tool similar to the paint brush on an arbitrary area of the image to be processed. Further, the classification of the once designated area may be canceled by a cancel operation such as undo. In the following examples, a tool similar to the paint brush in the paint application provided in the UI may be referred to as a "paint brush", and an object drawn by the "paint brush" may be referred to as an "object". be.

In addition, the providing unit 11 displays an image including the image classification predicted by the prediction unit 14 described later, and provides a UI that enables the user to re-annotate the displayed image. The predicted image classification is information indicating which image classification the region of the processed image belongs to, and the image including the predicted image classification is the predicted image classification for the processed image. This is the image to be displayed. The predicted image classification can be displayed, for example, by superimposing information indicating the image classification on the image to be processed. The information indicating the image classification is, for example, color information color-coded for each image classification. Further, the information indicating the image classification may be hatching information in which the types are classified for each image classification, grayscale information in which the densities are divided for each image classification, and the like. The image classification displayed on the UI enables the user to identify which area in the processed image was predicted for which image classification. For example, when displaying the predicted image classification by superimposing color-coded color information, the user can use the area of the image to be processed by coloring the area showing the positive example in red and the area showing the negative example in blue. It is possible to distinguish between the region predicted as a positive example and the region predicted as a negative example. The image including the predicted image classification may be generated by the providing unit 11 (including the UI) or may be generated by the predicting unit 14, and the subject of generation is not limited.

The image containing the predicted image classification may include a region where the image classification is not predicted. That is, in the image including the predicted image classification, the area in which the image classification is predicted and the area in which the image classification is not predicted are displayed so as to be distinguishable by the user. The user can re-annotate the area where the image classification is not predicted, and can re-annotate the area where the image classification is already predicted.

Here, the image displayed first in the UI is referred to as a first image, the result of annotation to the first image is referred to as the first annotation result, and the image classification based on the first annotation result is referred to as the first image classification. Further, including the first image classification, the image displayed in the UI after the first image is the second image, the result of re-annotating the second image is the second annotation result, and the image classification based on the second annotation result is further performed. This is called the second image classification. Further, an image including the second image classification and displayed in the UI after the second image is referred to as a third image. That is, the providing unit 11 provides a UI for displaying the first image, the second image, and the third image. The first image is an image displayed before the second image, and may include, for example, an image classification based on the nth annotation result (n is an integer). Further, the second image is an image displayed after the first image, and may include, for example, an image classification based on the n + 1th annotation result. Similarly, the third image is an image displayed after the second image, and may include, for example, an image classification based on the n + second annotation result.

The UI allows annotations on the first image by displaying the first image. By displaying the second image, the UI displays a prediction of image classification based on the result of the first annotation, and enables re-annotation of the second image. Further, the UI displays the third image to display the prediction of the image classification based on the second annotation result, and enables re-annotation of the third image. That is, the user can interactively perform the annotation work by repeatedly modifying the annotation while checking the prediction result of the image classification, improving the workability and reducing the annotation workload. Can be done.

Note that the image classification may be displayed separately according to the number of annotations. For example, in the first image classification based on the first annotation result, the region predicted as a positive example is displayed in orange, and the region predicted as a negative example is displayed in blue. Further, in the second image classification based on the second annotation result, the region predicted as a positive example may be displayed in red, and the region predicted as a negative example may be displayed in dark blue.

The acquisition unit 12 acquires the annotation result in which the user specifies the area classification for the image displayed on the UI provided by the providing unit 11. For example, the acquisition unit 12 acquires the coordinate data of the region designated as the classification of the positive example or the negative example. The annotation result acquired by the acquisition unit 12 is, for example, information on the position indicated by the tip of the object drawn by the paint brush of the UI. The annotation result is information on the length of the object drawn by the paint brush. The acquisition unit 12 may acquire the annotation result each time the user operates the annotation via the UI. For example, every time the user draws with the paint brush, information on the specified positive / negative classification and the drawn area is acquired.

In addition, the acquisition unit 12 acquires the first annotation result and the second annotation result. The acquisition unit 12 acquires the annotation result each time the annotation is performed, thereby enabling real-time prediction in the prediction unit 14.

Although the acquisition unit 12 has been described as acquiring the annotation result each time the user performs annotation, for example, the acquisition unit 12 may acquire the annotation result at a predetermined time interval. Further, the acquisition unit 12 may acquire the annotation result when the annotation is performed a predetermined number of times.

The storage unit 13 stores the annotation result acquired by the acquisition unit 12. The storage unit 13 stores, for example, a plurality of annotation results for the same processed image in chronological order. Further, the storage unit 13 may store the annotation results for different processed images. For example, when recognizing a lesion from an image of a specific organ, a plurality of annotation results for images of a plurality of organs may be stored. Further, the storage unit 13 may store the classification of the image predicted by the prediction unit 14 described below.

The prediction unit 14 learns the annotation result acquired by the acquisition unit 12 and predicts the classification of the image. The prediction unit 14 extracts the feature amount of the area designated by the user based on the annotation result, and predicts the classification of the image of the area similar to the extracted feature amount.

Here, the fact that the feature amounts of the images are similar means that, for example, in the images to be compared, at least one of the average value of the pixel information, the dispersion value of the pixel information, and the dispersion value of the pixel information with edge enhancement is used. It means that one value is within a predetermined range. The similarity of the feature amounts of the images may be, for example, the similarity of information of one pixel or the similarity of a plurality of pixels. In the present embodiment, a case where the image to be processed is divided into blocks of several pixels to several tens of pixels and the statistic of the RGB value of the pixels in each block is used as a feature amount is illustrated. When the pixel information is an RGB value, the average value of the pixel information mainly represents the color of the block, and the dispersion value of the pixel information mainly represents the fineness of the geometric pattern contained in the block. can.

The prediction unit 14 weighted and weighted each value of the average value of the pixel information, the dispersion value of the pixel information, and the dispersion value of the pixel information with edge enhancement in the feature amount of the image. It may predict the image classification of similar regions in the feature amount. For example, the prediction unit 14 weights the average value of the pixel information, the variance value of the pixel information, and the variance value of the pixel information with edge enhancement by 50:30:20, respectively. Image classification is predicted using the calculated numerical value as a feature amount.

The prediction unit 14 may select a preset weighting according to the type of the image. For example, in the process of recognizing an object in an image in an image type including an object having a color different from that of the surroundings, the prediction unit 14 selects a setting for increasing the weight of the average value of pixel information. Further, in the process of recognizing an object in an image in an image type including an object having a different shape pattern from the surroundings, the prediction unit 14 selects a setting for increasing the weight of the dispersion value of pixels. May be good. Further, in the process of recognizing an object in the image in the type of image including an object having an unclear outline, the prediction unit 14 selects a setting for increasing the weight of the dispersion of the information of the pixel with edge enhancement. It may be.

For example, the prediction unit 14 uses a statistic (for example, an average value or a variance value) of RGB values of pixels in an annotation result as a feature quantity, and predicts image classification using a support vector machine (SVM). SVM is a technique (engine) that classifies binary values based on teacher data. It learns annotation results to build a model, and predicts which of the binary values a new image will be classified into. The machine learning technique in the prediction unit 14 is not limited to SVM, and for example, a technique such as a neural network, clustering, or Bayesian network may be used.

The prediction unit 14 predicts the first image classification based on the first annotation result, and predicts the second image classification based on the second annotation result. That is, the prediction unit 14 predicts the image classification each time for the repeatedly executed annotation result, thereby feeding back the prediction of the image classification, which is the result of the annotation, to the user in real time (image display). ) Is possible.

Further, the prediction unit 14 may learn the feature amount in the block (for example, a block of several tens of pixels) at the position indicated by the tip of the object drawn by the paint brush in the annotation as the annotation result. The prediction unit 14 predicts a region having a feature amount similar to the feature amount of the block at the position indicated by the tip of the drawing as the classification specified in the annotation. For example, by moving the cursor of the paint brush to the position of the image learned by the user as a positive example and drawing the object, the classification intended by the user can be correctly learned.

Further, the prediction unit 14 may change the size of the region for predicting the classification according to the length of the object (stroke) drawn by the paint brush in the annotation. For example, when the drawing stroke is short, the prediction unit 14 reduces the size of the region for predicting the classification. Further, when the drawing stroke is long, the prediction unit 14 increases the size of the region for predicting the classification. By changing the size of the area for which the classification is predicted according to the length of the drawing stroke, it is possible to predict the classification within the range intended by the user. For example, the user can predict a wide range of classifications at once by lengthening the drawing stroke. This makes it possible to improve the work efficiency of annotation and reduce the work load. In addition, the user can make an accurate prediction of classification in a narrow range by shortening the drawing stroke. As a result, it is possible to reduce the modification of annotations for correcting erroneously predicted classifications, improve the work efficiency of annotations, and reduce the work load.

Further, the prediction unit 14 may predict the classification of the area (the part not painted with the paint brush) not specified by the annotation. The area specified by the annotation is an area whose classification has been confirmed by the user, and it is not necessary to predict the classification. By predicting the classification of the area other than the area specified by the annotation, the prediction unit 14 reduces the number of areas to be predicted and improves the processing efficiency in the prediction.

The communication control unit 15 controls communication between the information processing device 1 and the server 4.

The above-mentioned functional units of the information processing device 1 show an example of the functional units of the information processing device 1, and do not limit the functions of the information processing device 1. For example, the information processing device 1 does not have to have all the above-mentioned functional parts, and may have some of the functional parts. Further, the information processing device 1 may have a function other than the above. For example, the information processing device 1 may have an input function for inputting information and an output function for notifying the operating state of the device by an LED lamp or the like.

Further, each of the above-mentioned functional units included in the information processing apparatus 1 has been described as being realized by software as described above. However, at least one or more of the functional units included in the information processing device 1 may be realized by hardware.

Further, any of the above-mentioned functional units included in the information processing device 1 may be implemented by dividing one functional unit into a plurality of functional units. Further, any two or more of the above-mentioned functional units included in the information processing device 1 may be integrated into one functional unit. That is, FIG. 1 shows the functions of the information processing apparatus 1 represented by functional blocks, and does not show, for example, that each functional unit is composed of a separate program file or the like.

Further, the information processing device 1 may be a device realized by one housing or a system realized by a plurality of devices connected via a network or the like. For example, the information processing device 1 may realize a part or all of its functions by another virtual device such as a cloud service provided by a cloud computing system. That is, the information processing device 1 may realize at least one or more of the above-mentioned functional units in another device. Further, the information processing device 1 may be a general-purpose computer such as a desktop PC, or may be a dedicated device having limited functions.

For example, the server 4 may have a part or all of the above-mentioned information processing device 1.

Next, the operation of the information processing device 1 will be described with reference to FIG. FIG. 2 is a flowchart showing an example of the operation of the information processing device 1 in the embodiment. Although the operation described here will be executed mainly by the information processing device 1, it may be realized by each function of the information processing device 1 described with reference to FIG.

In FIG. 2, the information processing apparatus 1 determines whether or not an image to be processed in machine learning has been selected (step S11). The image to be processed (original image) is selected, for example, by the user selecting an image file from a predetermined folder. When it is determined that the image to be processed has not been selected (step S11: NO), the information processing apparatus 1 repeats the process of step S11 and waits for the image to be processed to be selected.

On the other hand, when it is determined that the image to be processed has been selected (step S11: YES), the information processing apparatus 1 displays the image to be processed and enables annotation by the user (step S12). To enable annotation by the user means, for example, to enable the user to specify an area for the displayed image and tag the specified area.

After executing the process of step S12, the information processing apparatus 1 determines whether or not the annotation result has been acquired (step S13). When it is determined that the annotation result has not been acquired (step S13: NO), the information processing apparatus 1 repeats the process of step S13 and waits for the annotation result to be acquired.

On the other hand, when it is determined that the annotation result has been acquired (step S13: YES), the information processing apparatus 1 learns the acquired annotation result and predicts the image classification (step S14).

After executing the process of step S14, the information processing device 1 determines whether or not to end the machine learning process (step S15). Whether or not to end the machine learning process can be determined, for example, by whether or not the user has explicitly instructed the end of the process. Whether or not to end the machine learning process may be determined, for example, by whether or not the area not classified as an image is equal to or less than a predetermined ratio. When it is determined that the machine learning process is not completed (step S15: NO), the information processing apparatus 1 returns to the process of step S12, displays an image including the image classification predicted in step S14 as a processing target, and displays the image as a processing target. Allows user annotation. When the image displayed in step S12 is an image including the image classification predicted in the process of step S14, the image displayed in step S12 is color-coded for each predicted image classification with respect to the original image, for example. It is generated by superimposing color information.

As described above, the first image is an image displayed before the second image. For example, when the first image is displayed in the process of step S12, the information processing apparatus 1 predicts the first image classification based on the first annotation result in the process of step S14, and the first image in the process of step S12. The second image including the classification is displayed in an annotable manner (step S15: in the case of NO), the second image classification is predicted based on the second annotation result in the process of step S14, and the second image is further displayed in the process of step S12. The third image including the classification is displayed in an annotable manner (step S15: in the case of NO). The first image is, for example, an original image determined to be selected in the process of step S11, or an image including an image classification predicted in the process of step S14. The second image is an image including the first image classification predicted in the process of step S14. That is, the information processing apparatus 1 can repeat the annotation and predict the image classification corresponding to each of the repeated annotations unless the processing is completed in the process of step S15. The above-mentioned display of the first image, acquisition of the first annotation result, and prediction of the first image classification can be performed by executing the processes of steps S11 to S14, and further, the display of the second image. , The acquisition of the second annotation result, and the prediction of the second image classification can be performed by repeatedly executing the processes of steps S12 to S14.

On the other hand, when it is determined that the machine learning process is finished (step S15: YES), the information processing apparatus 1 ends the process shown in the flowchart.

The illustrated flowchart shows an example of the processing executed by the information processing apparatus 1, and does not limit the processing to be executed. For example, the illustrated processes include display of the first image and display of the second image (step S12), acquisition of the first annotation result and acquisition of the second annotation result (step S13), and prediction of the first image classification and the first. An example is shown in which the prediction of the two image classifications (step S14) is executed in the same process (step). However, the display of the first image and the display of the second image, the acquisition of the first annotation result and the acquisition of the second annotation result, and the prediction of the first image classification and the prediction of the second image classification are executed as separate processes. You may try to do it. Further, the end of the process may be executed in the middle of the process. For example, the process may be interrupted during the standby in step S11 or step S13.

Next, the hardware configuration of the information processing apparatus 1 will be described with reference to FIG. FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing apparatus 1 according to the embodiment.

The information processing device 1 includes a CPU (Central Processing Unit) 101, a RAM (Random Access Memory) 102, a ROM (Read Only Memory) 103, an I / O device 104, and a communication I / F (Interface) 105. The information processing device 1 is a device that executes the information processing program described with reference to FIG.

The CPU 101 controls the user terminal by executing an information processing program stored in the RAM 102 or the ROM 103. The information processing program is acquired from, for example, a recording medium on which the program is recorded, a program distribution server via a network, or the like, installed in the ROM 103, read from the CPU 101, and executed.

The I / O device 104 has an operation input function and a display function (operation display function). The I / O device 104 is, for example, a touch panel. The touch panel enables the user of the information processing device 1 to input operations using a fingertip, a touch pen, or the like. The case where the I / O device 104 in the present embodiment uses a touch panel having an operation display function will be described, but the I / O device 104 separates the display device having the display function and the operation input device having the operation input function. It may have. In that case, the display screen of the touch panel can be performed as the display screen of the display device, and the operation of the touch panel can be performed as the operation of the operation input device. The I / O device 104 may be realized by various forms such as a head mount type, a glasses type, and a wristwatch type display.

The communication I / F 105 is an I / F for communication. The communication I / F 105 executes short-range wireless communication such as wireless LAN, wired LAN, and infrared rays. Although the figure shows only the communication I / F 105 as the communication I / F, the information processing device 1 may have each communication I / F in a plurality of communication methods.

Next, the annotation by the designated tool will be described with reference to FIGS. 4 to 6. FIGS. 4 to 6 show a case where the fountain in front of the hotel building is annotated as a positive example. Since both the fountain and the building are white and the difference in RGB values is small, teacher data by annotation is required.

FIG. 4 is a diagram showing a brush stroke (A) and an image classification prediction (B) in the first embodiment. The first embodiment shows the prediction of image classification when short stroke annotation is performed in the paint brush included in the designated tool.

In FIG. 4 (A), the stroke sa1 is an annotation using the fountain as a positive example, and specifies an image of the fountain in a short stroll. In FIG. 4B, the image classification sy1 shows the prediction of the image classification judged to be a normal example (fountain) in the size (area) corresponding to the length of the stroke sa1 centering on the tip portion of the stroke sa1. ing. The similarity of the feature amount can be calculated by, for example, the cosine similarity of the feature amount (feature vector). Here, the image classification sy1 is predicted to protrude from the building where a part of it is a negative example, but a large part correctly predicts a positive example. Therefore, when annotation is performed in a short stroke, the accuracy of prediction of image classification can be improved, and the time and effort for modifying the annotation can be reduced.

FIG. 5 is a diagram showing a brush stroke (A) and an image classification prediction (B) in the second embodiment. The second embodiment shows the prediction of image classification when long stroke annotation is performed in the paint brush included in the designated tool.

In FIG. 5 (A), the stroke sa2 is a positive annotation and specifies an image of the fountain in a stroll that is longer than the stroke sa1. In FIG. 5B, the image classification sy2 shows the prediction of the image classification determined to be a fountain in a size corresponding to the length of the stroke sa2 centering on the tip portion of the stroke sa2. Here, in the image classification sy2, many parts are predicted to protrude from the building (negative example), but the fountain (positive example) also correctly predicts the fountain in a large area. Therefore, when annotation is performed in a long stroke, image classification can be predicted in a large area at a time, and work efficiency can be improved.

FIG. 6 is a diagram showing a brush stroke (A) and an image classification prediction (B) in the third embodiment. In the third embodiment, in the state where the image classification is performed by the negative example, the prediction of the image classification when the long stroke annotation is performed by the paint brush included in the designated tool as in the second embodiment is predicted. show.

In FIG. 6A, the stroke sa3 is a positive annotation and specifies an image of the fountain in a stroll that is longer than the stroke sa1. The image classification phy1 shows the prediction of the image classification of the negative example based on the annotation on the wall surface of the building which is the negative example. The image classification fy1 shows the prediction of the image classification determined to be a negative example (other than the fountain) in the size corresponding to the length of the stroke (not shown) centering on the tip portion of the stroke (not shown).

In FIG. 6B, the image classification sy3 is an image classification in which a portion having a similarity higher than the similarity with the image classification fy1 is predicted as a positive example in comparison with the feature amount of the image classification phy1 which is a negative example. .. In the image classification of the positive example, the image classification is predicted by comparing the similarity with the feature amount of the block of the negative example annotated in the past (for example, the latest one). On the other hand, in the image classification of the negative example, the image classification is predicted by comparing the similarity with the feature amount of the block of the positive example annotated in the past (for example, the latest one). For example, the initial annotation is performed by the user freely specifying a positive example or a negative example in the image in a state where the image classification is not predicted. On the other hand, when the image classification has already been predicted, there are a mixture of regions that are correctly predicted as positive or negative examples and regions that are incorrectly predicted. Therefore, in the re-annotation, the annotation that corrects the erroneously predicted area is performed. Therefore, the feature amount in the re-annotation is often similar to the feature amount in the latest one annotation, and it is possible to improve the prediction accuracy of the image classification by comparing the two. Therefore, accurate image classification can be predicted by saving the feature amount of the block specified by the latest one annotation and comparing the feature amount of the newly predicted area with the similarity.

Next, the work flow of annotation will be described with reference to FIGS. 7 to 11. 7 to 11 show a case where the fountain in front of the hotel building is annotated as a positive example, as in the images described with reference to FIGS. 4 to 6.

FIG. 7 is a diagram showing initial annotations in the fourth embodiment. In the fourth embodiment, in the initial annotation, the positive and negative annotations are performed to a certain extent, and then the image classification is predicted based on the explicit instruction of the user. When performing additional predictions, it is periodically determined whether or not the user has newly annotated, and if it is determined that annotations have been made (annotation results have been obtained), image classification is automatically predicted. do.

In FIG. 7, the image field sy4 and the image field sy5 are the prediction results of the positive examples in the initial annotation. Further, the four locations of the image field fy2, the image field fy3, the image field fy4, and the image field fy5 are the prediction results of negative examples in the initial annotation. In normal image recognition, negative cases often have a higher variety of features. Therefore, in the fourth embodiment, the accuracy of image classification prediction is improved by increasing the amount of annotations in the negative example (area, number of blocks, or number of samples) as compared with the amount of annotations in the positive example. Can be made to. The ratio of the amount of annotations in the positive example and the negative example may be changed according to the type of the image.

FIG. 8 is a diagram showing a prediction of image classification in the fourth embodiment. In FIG. 8, the region (fountain) predicted as a positive example is displayed in a semi-transparent manner, and the region (part other than the fountain) predicted as a negative example is filled in and displayed. Since the user can visually recognize the area where the fountain is filled, it is possible to easily confirm the area predicted by mistake.

In the fourth embodiment, the block for calculating the feature amount is limited to, for example, 2000 blocks in the region of the positive example or the negative example annotated by the user. As a result, it is possible to significantly improve the processing speed as compared with the case where the feature amount is calculated for all the blocks specified by the user by annotation. By limiting the number of blocks for calculating the feature amount, the prediction accuracy of the image classification is slightly lowered, but in the present embodiment, the user can easily recognize the error of the prediction classification by the interactive UI. , Since the annotation can be easily modified, the decrease in prediction accuracy does not matter.

Further, in the present embodiment, the result of the determination of the positive example or the negative example manually annotated by the user may be prioritized over the result of the prediction by machine learning to predict the next image classification. .. By prioritizing the result of manual judgment by the user over the result of prediction by machine learning, the judgment of the user is prioritized, and by increasing the number of annotations, the prediction error is converged. Therefore, the decrease in prediction accuracy does not become a problem. On the other hand, the improvement of the processing speed makes it possible to improve the response from the annotation to the prediction that the user feels, and to improve the interactive operability in the annotation work.

FIG. 9 is a diagram showing modified annotations in the fourth embodiment. In FIG. 9, the user re-annotates the region of the lower part of the fountain with the wrong prediction in the example stroke sa4. As a negative example, the part where the prediction is wrong is the part where the features are more similar than the positive example. By re-annotating the part where the prediction is wrong as a negative example with a positive example, the feature amount can be learned more accurately, and the prediction accuracy of image classification can be improved.

For example, consider the case where the feature quantity is represented by the magnitude of one parameter. Assuming that the feature amount correctly predicted as a positive example is a1 or more and the feature amount correctly predicted as a negative example is less than a2, the region where the feature amount to be predicted is n (n: a1 ≧ n> a2). Is predicted to be positive or negative depending on whether n is similar to a1 or a2. Here, when a3 (a3: a1> a3) is annotated as a new positive example, it is determined whether the region having the feature amount n is predicted depending on whether it is similar to a3 or a2. Since it is | a1-a2 |> | a3-a2 |, by annotating a3 as a new positive example, the area erroneously predicted as a negative example decreases, and the area correctly predicted as a positive example increases. .. As a result, the prediction accuracy can be improved.

When the correction work by re-annotation by the user is not performed for a certain period of time or longer, the information processing device 1 acquires the feature amount of the newly annotated area as the annotation result and classifies the image based on the newly acquired annotation result. Execute the prediction of.

FIG. 10 is a diagram showing the prediction of the repredicted image classification in the fourth embodiment. In FIG. 10, the region of the fountainized portion is correctly predicted as a positive example as a result of re-annotation by the stroke sa4. By repeating the annotation by the user, the erroneously predicted area gradually becomes smaller, and many parts of the positive example are extracted.

FIG. 11 is a diagram showing the division of regions in the fourth embodiment. In FIG. 11, the first to sixth regions are divided into regions according to a combination of whether or not the region is annotated by the user as a positive example or a negative example, and whether or not the positive or negative example is correctly predicted. Is. The first area is an area correctly predicted by the user annotating as a positive example. The second area is an area correctly predicted by the user annotating as a negative example. The third area is an area that the user annotates as a positive example and is erroneously predicted. The fourth area is an area that the user annotates as a negative example and is erroneously predicted. The fifth area is an area where the correct example is correctly predicted without the annotation of the user. The sixth area is an area in which a negative example is correctly predicted without the annotation of the user. In the fourth embodiment, by repeating the annotation and the prediction, the region erroneously predicted without the user's annotation is not included. That is, in the area where there is no user annotation, all the erroneously predicted areas are corrected by the annotation.

As shown in FIG. 11, even when the user's annotation is prioritized, the area annotated by the user may be erroneously predicted. That is, even if the teacher data is increased by the annotation, the prediction may be incorrect. However, by repeating annotation and prediction, the area that is erroneously recognized gradually decreases and converges. In the fourth embodiment, almost all images of the fountain, which is a positive example, are extracted, and a large number of regions erroneously predicted as a positive example are included. In image recognition, it is important to extract all the objects to be extracted as positive examples, and it is often not a problem to erroneously predict negative examples as positive examples. That is, it is possible to reduce the workload of the user in annotation by allowing a region to be erroneously predicted to remain as a positive example as a negative example.

Next, an example in a medical image will be described with reference to FIG. FIG. 12 is a diagram showing an original image (A), an annotation result (B), and a prediction (C) of image classification in the fifth embodiment. In the fifth embodiment, the blood vessel portion is extracted as a positive example in the fundus photograph.

FIG. 12A is an original image of the fundus photograph before processing. In the fundus photograph, it has a red tint as a whole. Medical images have the same tendency as a whole, and the difference in the amount of features depending on the color is often small. FIG. 12B is an image of the result of annotating the original image. The lower left portion of FIG. 12B is an enlarged view of a part of the illustrated image. In FIG. 12B, stroke sa5 and stroke sa6 are portions annotated as positive examples. The image classification fi6 is a portion annotated with a portion other than the blood vessel as a negative example. Note that annotations as positive examples are made in several places other than stroke sa5 and stroke sa6. In addition, annotations as a negative example are made in several places other than the image classification fi6. FIG. 12C is a prediction result of image classification. In FIG. 12C, most of the blood vessels are correctly predicted as a positive example. That is, it is shown that even in a medical image having a small difference in the feature amount, highly accurate prediction can be made by appropriately performing the annotation of the positive example and the annotation of the negative example. In the present embodiment, since the annotation and the prediction result can be confirmed interactively, the user can immediately confirm the quality of the prediction result by the annotation, so that the skill of the annotation work can be improved. Efficient annotation makes it possible to reduce the workload.

It should be noted that the program for realizing the function constituting the apparatus described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into the computer system and executed. Therefore, the above-mentioned various processes of the present embodiment may be performed. The "computer system" referred to here may include hardware such as an OS and peripheral devices. Further, the "computer system" includes a homepage providing environment (or a display environment) if a WWW system is used. The "computer-readable recording medium" includes a flexible disk, a magneto-optical disk, a ROM, a writable non-volatile memory such as a flash memory, a portable medium such as a CD-ROM, a hard disk built in a computer system, and the like. It refers to the storage device of.

Furthermore, the "computer-readable recording medium" is a volatile memory inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line (for example, DRAM (Dynamic)). It also includes those that hold the program for a certain period of time, such as Random Access Memory)). Further, the program may be transmitted from a computer system in which this program is stored in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the "transmission medium" for transmitting a program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. Further, the above program may be for realizing a part of the above-mentioned functions. Further, it may be a so-called difference file (difference program) that realizes the above-mentioned function in combination with a program already recorded in the computer system.

Although the embodiments of the present invention have been described above with reference to the drawings, the specific configuration is not limited to this embodiment and includes various modifications within a range not deviating from the gist of the present invention. Is done.

1 Information processing device 11 Providing unit 12 Acquisition unit 13 Storage unit 14 Prediction unit 15 Communication control unit 2 Display device 3 Input device 4 Server 9 Network 101 CPU
102 RAM
103 ROM
104 I / O equipment 105 Communication I / F
sa1, sa2, sa3, sa4, sa5, sa6 strokes (normal example)
sy1, sy2, sy3, sy4, sy5 Image classification (normal example)
fy1, fy2, fy3, fy4, fy5, fy6 Image classification (negative example)

Claims (8)

A provider that displays an image processed in machine learning and provides a user interface (UI) that enables the user to annotate the image.
An acquisition unit that acquires an annotation result in which a user specifies an area for the image displayed on the UI provided by the provision unit, and an acquisition unit.
A prediction unit that learns the annotation results acquired by the acquisition unit and predicts the classification of the image.
It is an information processing device equipped with
The providing unit provides the UI for displaying the first image, and provides the UI.
The acquisition unit acquires the first annotation result in which the user specifies the area for the first image, and obtains the result.
The prediction unit predicts the first image classification based on the result of the first annotation acquired by the acquisition unit.
The providing unit provides the UI for displaying a second image including the first image classification predicted by the prediction unit.
The acquisition unit acquires the second annotation result in which the user specifies the area for the second image, and obtains the result.
The prediction unit further learns the second annotation result acquired in the acquisition unit to predict the second image classification, and predicts the second image classification.
The providing unit may provide the UI to display a third image including the predicted second image classification in the prediction unit,
The acquisition unit acquires the annotation result including the size of the area specified by the user.
The prediction unit is an information processing device that changes the size of a region for predicting image classification according to the size. The acquisition unit acquires the second annotation result in which the user modifies the classification with respect to the first image classification and specifies the area in the second image.
The second image classification according to claim 1, wherein the prediction unit further learns the second annotation result in which the classification of the first image classification is modified, and predicts the second image classification in which the classification of the first image classification is modified. Information processing device. The information processing device according to claim 1 or 2 , wherein the acquisition unit acquires an annotation result including the length of a region specified by the user as the size. The prediction unit extracts the feature amount of the area specified by the user based on the annotation result, and predicts the classification of the image of the area similar to the extracted feature amount, according to any one of claims 1 to 3. The information processing device described. The prediction unit is based on at least one of the average value of the pixel information, the dispersion value of the pixel information, and the dispersion value of the pixel information with edge enhancement from the area specified by the user. The information processing apparatus according to claim 4 , wherein the information processing apparatus is extracted. The prediction unit extracts the average value of the information of the pixel, the variance value of the information of the pixels, and the feature amount each on the basis of the weight set of variance information of the edge enhancement was subjected to pixel, claim 4 Or the information processing apparatus according to 5. It is an information processing method in an information processing device.
A provision step that displays an image processed in machine learning and provides a user interface (UI) that enables the user to annotate the image.
An acquisition step of acquiring an annotation result in which a user specifies an area for the image displayed on the UI provided in the provision step, and an acquisition step.
A prediction step that learns the annotation result acquired in the acquisition step and predicts the classification of the image,
Information processing method including
The step of providing the UI for displaying the first image and
A step of acquiring the first annotation result in which the user specifies an area for the first image, and
The step of predicting the first image classification based on the acquired first annotation result,
The step of providing the UI to display the second image including the predicted first image classification, and
The step of acquiring the second annotation result in which the user specifies the area for the second image, and
The step of further learning the acquired second annotation result and predicting the second image classification,
Providing said UI for displaying a third image including the predicted second image classification,
Steps to get the annotation result including the size of the area specified by the user,
An information processing method including a step of changing the size of an area for predicting image classification according to the size. On the computer
A providing function that displays an image processed in machine learning and provides a user interface (UI) that enables the user to annotate the image.
An acquisition function for acquiring an annotation result in which a user specifies an area for the image displayed on the UI provided in the provided function, and an acquisition function.
A prediction function that learns the annotation results acquired by the acquisition function and predicts the classification of the image.
It is an information processing program to realize
The provided function provides the UI for displaying the first image, and provides the UI.
The acquisition function acquires the first annotation result in which the user specifies an area for the first image.
The prediction function predicts the first image classification based on the result of the first annotation acquired by the acquisition function.
The provided function provides the UI for displaying a second image including the first image classification predicted by the prediction function.
The acquisition function acquires the second annotation result in which the user specifies the area for the second image.
The prediction function further learns the second annotation result acquired by the acquisition function to predict the second image classification, and predicts the second image classification.
The provided function provides the UI for displaying a third image including the second image classification predicted by the prediction function.
The acquisition function acquires the annotation result including the size of the area specified by the user.
The prediction function is an information processing program that changes the size of an area for predicting image classification according to the size.

Images