JP7502055B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program, and in particular to a technology for efficiently creating image data to be used in machine learning.

Segmentation is one of the image processing techniques. Segmentation is the process of distinguishing between areas of interest and non-interest areas in an image, and is also called region extraction, region division, or image division. Various methods related to segmentation have been proposed, but in recent years, methods based on machine learning have been attracting attention.

The accuracy of distinguishing regions of interest using machine learning-based techniques also depends on the number of images used in the training data, so it is desirable to prepare as many images as possible as training data. However, creating correct images as correct data to be included in the training data is a labor-intensive task, so there is a demand for technology to efficiently create correct images. Therefore, for example, Patent Document 1 discloses a technology in which correct images are created manually or automatically, and then more accurate correct images are created based on the created correct images.

JP 2019-101535 A

However, in the above technology, creating a correct answer image requires manually labeling each pixel in the image as being an area of interest or not, which increases the workload of the person creating the correct answer image to a significant extent.

The present invention was made in consideration of the above circumstances, and aims to provide a technology that can create highly accurate correct images in learning to identify areas within an image while reducing the workload.

The present disclosure provides a region extraction unit that extracts, as an extraction region, a region that is identified as a region of interest from a target image using a classifier;
an area specifying unit that specifies a learning area to be used for learning the classifier based on the extracted area extracted by the area extracting unit;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area;
a teacher data adding unit that adds a pair of the target image and the correct image to teacher data;
a learning unit that learns the classifier by referring to the learning area in the correct image included in the training data;
Equipped with
The learning unit calculates a value of a loss function for the target image and the correct image using weights.
The image processing device includes:
The present disclosure also provides a region extraction unit that extracts a region identified as a region of interest from a target image using a classifier as a first extraction region, and extracts a second extraction region from the target image as a region of interest by performing image processing different from the image processing used to extract the first extraction region;
a region specifying unit that specifies a learning region to be used for learning the classifier based on a comparison between the first extracted region and the second extracted region;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area or is an attention area;
The image processing device includes:
The present disclosure also provides a method for extracting an area that is identified as an attention area from a target image using a classifier, as an extraction area;
an area specifying unit that specifies a learning area to be used for learning the classifier based on the extracted area extracted by the area extracting unit;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area;
Equipped with
The region identification unit creates a connected region by connecting the extracted regions extracted by the region extraction unit, identifies an erroneously extracted region based on an area of the created connected region, and identifies the erroneously extracted region as the learning region.
The image processing device includes:

Further, the present disclosure provides a method for extracting a region identified as a region of interest from a target image using a classifier as an extracted region;
a region specifying step of specifying a learning region to be used for learning the classifier based on the extracted region extracted by the region extracting step;
a generation step of generating a correct answer image indicating that the learning area identified by the area identification step is not an area of interest;
a teacher data adding step of adding a pair of the target image and the correct image to teacher data;
a learning step of learning the classifier by referring to the learning region in the correct image included in the training data;
Including,
The learning step calculates a loss function value for the target image and the correct image using weights.
The present invention also includes an image processing method comprising the steps of:
The present disclosure also provides a method for extracting a region identified as a region of interest from a target image using a classifier as a first extracted region, and extracting a second extracted region from the target image as a region of interest by performing image processing different from the image processing used for extracting the first extracted region;
a region specifying step of specifying a learning region to be used for training the classifier based on a comparison between the first extracted region and the second extracted region;
a generating step of generating a correct answer image indicating that the learning area identified by the area identifying step is not an area of interest or is an area of interest;
The present invention also includes an image processing method comprising the steps of:
In addition, the present disclosure provides a method for extracting an area that is identified as an attention area from a target image using a classifier.
A region extraction step of extracting an extracted region;
a region specifying step of specifying a learning region to be used for learning the classifier based on the extracted region extracted by the region extracting step;
a generation step of generating a correct answer image indicating that the learning area identified by the area identification step is not an area of interest;
Equipped with
The region identification step creates a connected region by connecting the extracted regions extracted by the region extraction step, identifies an erroneously extracted region based on an area of the created connected region, and identifies the erroneously extracted region as the learning region.
The present invention also includes an image processing method comprising the steps of:

Furthermore, the present disclosure includes a program for causing a computer to execute each step of the image processing method.

The present invention provides a technology that can create highly accurate images as ground truth data for learning to determine regions within an image while minimizing the workload.

FIG. 1 is a functional block diagram showing an example of a functional unit of an image processing apparatus according to a first embodiment; FIG. 11 is a diagram showing a result of image processing in the first embodiment. FIG. 4 is a diagram illustrating a process of determining an attention area according to the first embodiment; FIG. 11 is another diagram showing the determination process of an attention area in the first embodiment; FIG. 1 is a diagram showing a flow of image processing in the first embodiment; FIG. 1 is a diagram showing a flow of a learning process in the first embodiment; FIG. 13 is a functional block diagram showing an example of a functional unit of an image processing apparatus according to a second embodiment. FIG. 13 is a diagram showing a process flow in a second embodiment. FIG. 13 is a diagram showing a result of image processing in the second embodiment. FIG. 13 is a diagram illustrating an example of a determination process of an attention area according to a second embodiment;

Below, an embodiment of the present invention will be described with reference to the drawings. The same or equivalent components, parts, and processes shown in each drawing will be given the same reference numerals, and duplicated explanations will be omitted as appropriate. Also, some of the components, parts, and processes will be omitted in each drawing.

In the following, the following embodiment will be described by taking the liver depicted in an abdominal CT image taken by an X-ray computed tomography (X-ray CT) device as an example. However, the following embodiment can be applied not only to the liver but also to any structure of the human body such as other organs, bones, and muscles. The following embodiment can also be applied to tomographic images taken by a magnetic resonance imaging (MRI) device, a positron emission tomography (PET) device, and an ultrasonic imaging device, and to normal images taken by a camera. In addition, the following embodiment can be applied to both two-dimensional images and three-dimensional images. Furthermore, the following embodiment can also be applied to images taken of general objects. In addition, the embodiments of the present invention are not limited to the following embodiment.

First Embodiment
The configuration of the image processing device according to the first embodiment will be described with reference to FIG. 1. The image processing device 100 according to the first embodiment is composed of a correct image generating unit 101 and a learning unit 102. For example, the CPU reads and executes a program stored in the memory, thereby providing the functions of the correct image generating unit 101 and the learning unit 102 in the image processing device 100. In addition to the CPU, the image processing device 100 may also include a processor (GPU, FPGA, etc.) that is responsible for the calculations of the correct image generating unit 101 and the learning unit 102. The memory may include a memory for non-temporarily storing a program, a memory for temporarily storing data such as a received signal, a working memory used by the CPU, etc.

The correct image generating unit 101 generates a correct image indicating a region (attention image) to be extracted in an image captured by an X-ray CT device. More specifically, the correct image generating unit 101 applies a predetermined classifier to an image that is the basis of the image as the correct data of the learning data, and extracts the attention region from the image. Hereinafter, the image processed by the correct image generating unit 101 is referred to as a target image. Hereinafter, the classifier will be described. The classifier is a classifier for performing segmentation, and in the first embodiment, a classifier trained using teacher data is used. Hereinafter, the region of the organ that the classifier is to extract from among the anatomical structures depicted in the target image is set as the attention region. Although the classifier can generally correctly extract the attention region in the target image, there is a possibility that it cannot completely extract all the attention regions. Therefore, the correct image generating unit 101 identifies the region in the target image that the classifier could not correctly extract as a learning region to be used for learning the classifier. Then, based on the identified learning region, a correct image corresponding to the target image is generated. In addition, the learning unit 102 trains the classifier using training data that is a pair of the target image and the correct image created by the correct image generation unit 101.

Here, the differences between the creation of a correct answer image and the training of a classifier according to the conventional technology and the creation of a correct answer image and the training of a classifier executed by the image processing device 100 according to the first embodiment will be explained. In the conventional technology, when creating a correct answer image, the creator of the correct answer image assigns a label to all pixels in the image on which the correct answer image is based as information indicating whether or not the pixel is an area of interest. Then, in the training of the classifier, the parameters of the classifier are determined so that the output of the classifier for all pixels in the image in the created correct answer image matches the label assigned to the pixel. In other words, in the conventional technology, regardless of whether the classifier can correctly identify the area of interest, a label indicating whether or not the pixel is an area of interest is assigned to all pixels in the image, and the classifier is trained.

However, in order to save the effort of creating a correct image and efficiently train the classifier, it is not essential to assign labels to pixels that allow the classifier to correctly identify the region of interest. Therefore, in this embodiment, the correct image generation unit 101 first causes the classifier to classify the image (target image) that is the basis of the correct image. Then, the correct image generation unit 101 assigns labels only to pixels that the classifier has not correctly identified the region of interest among the pixels that constitute the target image. Furthermore, the learning unit 102 trains the classifier by referring only to the pixels that have been assigned labels by the correct image generation unit 101. Therefore, in the creation of the correct image and the training of the classifier in the first embodiment, a label is assigned only to pixels in the target image that the classifier has misclassified, and the classifier is trained by referring only to the labeled pixels.

The configuration of the image processing device 100 according to the first embodiment will be described in more detail with reference to FIG. 1. As shown in FIG. 1, the correct image generating unit 101 has a data acquiring unit 110 for acquiring a target image, and a region extracting unit 120 for extracting an attention region from the target image using the above-mentioned classifier. Furthermore, the correct image generating unit 101 has a region identifying unit 130 for acquiring information on a region used for learning the classifier, a generating unit 150 for generating a correct image for learning, and a teacher data adding unit 160 for adding a pair of the created correct image and target image to teacher data. The region identifying unit 130 also has a function as a region selecting unit 140 for considering the region extracted by the region extracting unit 120 as a candidate region and selecting a learning region to be used for learning from the candidate region. Note that in the first embodiment, the correct image generating unit 101 can obtain the effect of this embodiment even if it does not have the region selecting unit 140. Note that the learning region is a region in the correct image to which a label indicating whether or not it is a region of interest is attached, and means a region other than a region to which a label indicating that it is not used for learning the classifier is attached.

The learning unit 102 has a data acquisition unit 170 that acquires training data, and a classifier learning unit 180 that performs training of the classifier by referring to the training data. The image processing device 100 is also connected to a data server 200 for storing data to be input to the image processing device 100 and data output by the image processing device 100. The data server 200 functions as a storage unit, is an example of a computer storage medium, and is a large-capacity information storage device represented by a hard disk drive (HDD) or a solid-state drive (SSD). The data server 200 may be held within the image processing device 100, or may be provided separately outside the image processing device 100 and configured to be able to communicate via a network.

The data acquisition unit 110 acquires a target image from the data server 200. The target image may be an image in which a region of interest is depicted within the target image, or an image in which a region of interest is not depicted. The data acquisition unit 110 transmits the acquired target image to the region extraction unit 120, the region identification unit 130, and the teacher data addition unit 160, respectively.

Now, referring to Figures 2A to 2C, the target image processed by the correct image generating unit 101 will be described. Image 310 in Figure 2A is an example of a target image. Target image 310 is a cross section (transverse section) of a 3D abdominal CT image obtained by photographing the abdomen of a patient. The portion indicated by region 311 and region 312 in target image 310 is the liver, and is the region to be classified by the classifier (region of interest). Organs other than the liver (not shown), such as the spleen and left kidney, are also depicted in target image 310. Here, the regions of these organs other than the liver are not considered to be regions of interest.

The area extraction unit 120 applies the above-mentioned classifier to the target image and extracts an area of interest depicted in the target image. The area extraction unit 120 transmits the extracted area to the area identification unit 130. In the following description, the processing result (extracted area) of the area extraction unit 120 is referred to as the extraction result. In addition, the area identified (extracted) as an area of interest by the area extraction unit 120 is referred to as the extracted area.

The classifier in the first embodiment may be any classifier that can be used for image segmentation. Examples of classifiers that can be used by the region extraction unit 120 include a Support Vector Machine (SVM), Adaptive Boosting (AdaBoost), and a neural network.

Image 320 shown in FIG. 2B is an example of an extraction result. Regions 321 and 322 in image 320 are extracted regions obtained by extracting attention regions 311 and 312 of image 310 by a classifier. Comparing attention region 311 with extracted region 321 and attention region 312 with extracted region 322, respectively, it can be seen that the classifier correctly extracts most of attention regions 311 and 312. However, there are also regions such as region 323 in image 320 (the boundary is shown by a dotted line for convenience of explanation) that the classifier could not correctly extract as attention regions. In addition, there are also regions such as region 324 in image 320 that are not part of the attention region but are erroneously extracted as attention regions by the classifier.

The region identification unit 130 refers to the extraction result by the region extraction unit 120 and identifies a learning region in the target image to be used by the learning unit 102 when training the classifier. Therefore, the learning region is also the extraction region extracted by the region extraction unit 120. The region identification unit 130 identifies a region that has been erroneously extracted from the extraction region, assigns information (label) indicating that the extraction result is erroneous to the identified region, and identifies the region as a learning region for the classifier. For example, the region identification unit 130 identifies the region excluding the region with the largest area from the extraction region as the learning region. In the case of the example shown in FIG. 2B, the region identification unit 130 calculates the area of two regions in the image 320, that is, (1) a region obtained by combining the region 321 and the region 322, and (2) a region 324. In FIG. 2B, the region 321 and the region 322 are distinguished from each other for convenience of explanation, but these two regions are adjacent to each other. Therefore, the region identification unit 130 creates a connected region by connecting these adjacent regions 321 and 322, and calculates the area of the two regions 321 and 322 as one connected region. In this case, it is found that the region with the largest calculated area is the connected region 321 and 322. Therefore, the region identification unit 130 identifies the region that does not occupy the largest area in the image, that is, region 324, out of the connected regions 321 and 322 and region 324, as the learning region.

Image 330 shows the result of processing the image 320 by the region identification unit 130. As shown in the figure, in image 330, the regions corresponding to regions 321 and 322 in image 320 are excluded, and region 334 corresponding to region 324 is identified as the learning region. Note that the region identification unit 130 may create a connected region of the extracted regions using a connection method such as 4-connection or 8-connection. In addition, when the attention region is a three-dimensional region, a connection method such as 6-connection or 26-connection may be used to create a connected region of the extracted regions. In addition, the number of learning regions identified by the region identification unit 130 is not limited to one. For example, when identifying a learning region from three or more regions in the target image, the region identification unit 130 may identify the Nth (N is an integer of 2 or more) region from the region with the largest area as the learning region. This makes it possible to identify each region as a learning region, for example, even if the attention region is divided into two or more regions in the image.

The region selection unit 140 regards the learning regions identified by the region identification unit 130 as candidate regions and determines whether or not each candidate region is to be used for learning the classifier in the learning unit 102. Therefore, the candidate regions are also extracted regions extracted by the region extraction unit 120. Here, the evaluation of the characteristics described below for the candidate regions identified by the region identification unit 130 and the determination of whether or not the evaluation result satisfies a predetermined condition are referred to as a determination process. The region selection unit 140 executes at least one of the following five determination processes 1 to 5. By executing at least one of the determination processes 1 to 4, the region selection unit 140 evaluates at least one of the characteristics related to one or more of the length, area, and shape of the region and the characteristics related to the grayscale values of the pixels that make up the region. Then, if the evaluated characteristics satisfy a predetermined condition, the region selection unit 140 determines and selects the region to be used for learning. In addition to this/instead of this, the region selection unit 140 executes the determination process 5 to more reliably exclude the region correctly extracted as the region of interest and then specify the learning region. The area selection unit 140 then transmits information about the candidate areas determined to be used for learning to the generation unit 150.

Next, we will explain the determination process performed by the area selection unit 140.

[Determination process 1: Determination based on region length]
The region selection unit 140 calculates the length of the candidate region along each of the X-axis direction and the Y-axis direction in the XY orthogonal coordinate system set in the image. Fig. 4A shows an example of a candidate region 411 in an image 410 identified by the region identification unit 130. The region selection unit 140 obtains the length 412 in the X-axis direction and the length 413 in the Y-axis direction for the region 411. If the respective lengths 412, 413 satisfy a predetermined threshold or range condition, the region selection unit 140 selects the region 411 as a learning region.

The region selection unit 140 may calculate the lengths along the principal axis of inertia of the region and an axis perpendicular thereto. FIG. 4B shows an example of a candidate region 421 in another image 420 identified by the region identification unit 130. FIG. 4B also shows the principal axis of inertia 422 of the candidate region 421 and an axis 423 perpendicular thereto. The region selection unit 140 identifies the two axes 422, 423 of the region 421 and calculates the lengths 424, 425 along each of the axes. If the lengths 424, 425 satisfy a predetermined threshold or range condition, the region selection unit 140 selects the region 421 as a learning region.

If the image is a three-dimensional image and the candidate region is a three-dimensional region, mutually orthogonal XYZ coordinates are set in the image, and the region selection unit 140 may calculate the lengths along the X, Y, and Z axis directions, or may calculate the lengths along the principal axis of inertia and the direction perpendicular to it in three dimensions.

[Determination process 2: Determination based on area of region]
The region selection unit 140 calculates the area of the candidate region. If the area of the candidate region satisfies a predetermined threshold or range condition, the region selection unit 140 selects the candidate region as a learning region. In the determination process 2, if the candidate region is a three-dimensional region, the region selection unit 140 may calculate the volume of the candidate region, and if the volume satisfies the threshold or range condition, select the candidate region as a learning region.

[Determination process 3: Determination based on the shape of the region]
The region selection unit 140 calculates a feature quantity representing the shape of the candidate region. Examples of the feature quantity representing the shape include the perimeter, circularity, and moment of the candidate region. When the feature quantity representing the shape of the candidate region satisfies a threshold or range condition, the region selection unit 140 selects the candidate region as a learning region. In the determination process 3, when the candidate region is a three-dimensional region, the region selection unit 140 may calculate, for example, the surface area or sphericity of the candidate region, and select the candidate region as a learning region when the surface area or sphericity satisfies a threshold or range condition.

The area selection unit 140 may also determine whether to select the candidate area as a learning area by comparing a template indicating the shape of the area to be used for learning, which has been created in advance, with the shape of the candidate area. In this case, the area selection unit 140 acquires a template indicating the shape of the area to be used for learning the classifier from the data server 200. The area selection unit 140 then calculates the similarity between the shape indicated by the acquired template and the shape of the candidate area. For example, the similarity includes the Dice coefficient, Jaccard index, and Sum of Squared Difference. Then, the area selection unit 140 selects the candidate area as a learning area if the similarity satisfies a threshold or range condition.

[Determination process 4: Determination based on pixel values of pixels constituting the region]
The region selection unit 140 calculates statistics of pixel values of pixels constituting the candidate region. Examples of the statistics include a maximum value, a minimum value, an average value, a standard deviation of pixel values, and a histogram showing the frequency of occurrence of pixel values. When the statistics satisfy a threshold or range condition, the region selection unit 140 selects the candidate region as a learning region. When the statistics are a histogram, the region selection unit 140 may calculate the Kullback-Leibler divergence to evaluate the similarity between the histogram indicated by the template and the histogram calculated for the candidate region. In this case, the region selection unit 140 acquires a histogram of pixel values of the region used for training the classifier from the data server 200. Then, the region selection unit calculates the Kullback-Leibler divergence between the histogram calculated from the candidate region and the histogram acquired from the data server 200. Note that the Kullback-Leibler divergence is well known, and a detailed explanation of the calculation will be omitted.

In the above-mentioned determination processes 1 to 4, the threshold and range conditions that the area selection unit 140 uses when selecting a learning area may be, as appropriate, above the threshold, below the threshold, within the range, outside the range, etc. This makes it possible to select as the learning area an area that deviates from the above criteria for the size, area, feature amount, statistical value, etc. expected for the area of interest.

[Determination process 5: Determination based on distance 1 from the boundary of the maximum connected region]
In this determination process, the region selection unit 140 selects a region to be used for training the classifier based on the features related to the position of the training region within the image. Specifically, the region selection unit 140 calculates the distance of the candidate region from the boundary of the maximally connected region extracted by the region extraction unit 120. The region selection unit 140 then selects, within the candidate region, a region that is away from the boundary of the maximally connected region by a predetermined distance or more as the training region.

The details of the determination process 5 will be described with reference to Figures 4A to 4D. Image 510 shown in Figure 4A is the extraction result obtained by the area extraction unit 120. In image 510, two areas, the combined area of areas 511 and 512 and area 513, are the extracted areas. For convenience of explanation, areas 511 and 512 are considered as separate areas. Also, image 520 shown in Figure 4B is the processing result by the area identification unit 130. The area identification unit 130 creates a connected area by connecting areas that are in contact with each other among the extracted areas (areas 511 and 512 in the figure). As described above, the connected area may be created by 4-connection or 8-connection. As shown in Figure 4A, the connected area of areas 511 and 512 is the maximum connected area that occupies the maximum area in image 510. For this reason, the area identification unit 130 excludes areas 511 and 512, which are the maximum areas, from the learning area. On the other hand, since region 513 is not the maximum connected region, region identification unit 130 identifies region 513 as a candidate region for the learning region. As a result, region identification unit 130 identifies regions other than the maximum region, which has the largest area, among the created connected regions as candidate regions for the learning region. Candidate region 521 in FIG. 4B is a region that corresponds to region 513 in FIG. 4A.

The region selection unit 140 selects, as the learning region, a region of the candidate region 521 that exists at a position that is more than a predetermined distance away from the boundary of the maximally connected regions 511 and 512. FIG. 4C shows an image 530 that shows a schematic representation of the result of this process. In the image 530, the contour lines 531 and 532 represent the boundary between the maximally connected regions 511 and 512. Also, the dotted line 533 represents a position that is a predetermined distance away from the maximally connected regions 511 and 512. The region 535 is more than the predetermined distance away from the maximally connected regions 511 and 512. Therefore, the region selection unit 140 selects the region 535 as the learning region. On the other hand, the region 534 exists at a position within the predetermined distance from the maximally connected regions 511 and 512. Therefore, the region selection unit 140 excludes the region 534 from the learning region.

Figure 4D shows image 540, which is a schematic representation of the learning area selected by the above-mentioned determination process 5 of area selection unit 140. Area 541 in image 540 corresponds to area 535 in image 530. Area selection unit 140 executes at least one of the above-mentioned determination processes 1 to 5, and selects the learning area from among the areas identified by area identification unit 130.

The correct image generating unit 101 can generate a correct image indicating a learning area without performing a determination process by the area selecting unit 140. Therefore, the correct image generating unit 101 may be configured not to include the area selecting unit 140. When the correct image generating unit 101 does not include the selecting unit 140, the area identifying unit 130 transmits the identified area to the generating unit 150.

Next, the processing of the generation unit 150 will be described. The generation unit 150 generates a correct answer image corresponding to the target image based on the learning area transmitted from the area identification unit 130 or the area selection unit 140. The generated correct answer image is transmitted to the teacher data addition unit 160.

The region that the generating unit 150 receives from the region identifying unit 130 or the region selecting unit 140 is a region that is not a region of interest and that the classifier could not identify as a "region of interest", that is, a region that was erroneously identified as a region of interest. Therefore, the generating unit 150 assigns a label of "not a region of interest" to each pixel in the received region. Using the image 330 shown in FIG. 2C as an example, the generating unit 150 assigns a label of "not a region of interest" to each pixel constituting the region 334 in the image 330. On the other hand, the generating unit 150 assigns a label of "not used for learning" to each pixel in a region other than the learning region, that is, the region 335 other than the region 334 in the image 330. Also, using the image 540 shown in FIG. 4D as an example, the generating unit 150 assigns a label of "not a region of interest" to each pixel constituting the region 541 in the image 540. On the other hand, the generating unit 150 assigns a label of "not used for learning" to each pixel constituting the area 542 other than the area 541 in the image 540. The generating unit 150 then generates an image that holds the assigned label as the pixel value of each pixel as the correct image.

The teacher data adding unit 160 associates the target image received from the data acquiring unit 110 with the correct image received from the generating unit 150 as one pair. The teacher data adding unit 160 then adds the pair of the target image and correct image to the teacher data stored in the data server 200. In this embodiment, the teacher data stored in the data server 200 may include a correct image in which a region of interest in the image is labeled as "region of interest" and other regions are labeled as "not a region of interest". In the learning of the classifier described below, not only images to which the classifier is applied, but also such pre-created correct images can be used.

The teacher data adding unit 160 may output the correct image paired with the target image to a display unit (not shown). An example of the display unit is a display. The display unit may display only the correct image, or may display the correct image and the target image paired with the correct image.

Next, the teacher data creation process executed by the correct image generation unit 101 will be described with reference to FIG. 5.

(S1010)
In step S1010, the data acquisition unit 110 acquires a target image from the data server 200. Then, the data acquisition unit 110 transmits the acquired target image to the region extraction unit 120, the region selection unit 140, and the teacher data addition unit 160, respectively.

(S1020)
In step S1020, the region extraction unit 120 applies the classifier to the target image received from the data acquisition unit 110 in step S1010, and extracts, as an extraction region, a region that the classifier has identified as a region of interest from the target image. Then, the region extraction unit 120 transmits the extracted extraction region to the region identification unit 130 as an extraction result.

(S1030)
In step S1030, the region identifying unit 130 identifies a candidate region for the learning region of the classifier by referring to the extraction result received from the region extraction unit 120. Here, if the correct image generation unit 101 has the region selection unit 140, the region identifying unit 130 transmits the identified region to the region selection unit 140 as a candidate region for the learning region. On the other hand, if the correct image generation unit 101 does not have the region selection unit 140, the region identifying unit 130 transmits the identified region to the generation unit 150 as a learning region.

(S1040)
In step S1040, the region selection unit 140 receives candidate regions for the learning region from the region identification unit 130. That is, the region selection unit 140 considers the region received from the region identification unit 130 to be a candidate region. The region selection unit 140 determines the learning region by performing at least one of the determination processes 1 to 5 for each candidate region. Then, the region selection unit 140 transmits information indicating the learning region determined to be used for learning the classifier to the generation unit 150. Details of the determination processes 1 to 5 are as described above regarding the processing of the region selection unit 140. Note that if the correct answer image generation unit 101 does not have the region selection unit 140, the processing of step S1040 is not performed, and the processing proceeds from step S1030 to S1050.

(S1050)
In step S1050, the generating unit 150 generates a correct image corresponding to the target image based on the learning region selected by the region identifying unit 130 or the region selecting unit 140. Then, the generating unit 150 transmits the correct image to the teacher data adding unit 160. Details of the correct image generation process are as described in the description of the generating unit 150.

(S1060)
In step S1060, the teacher data adding unit 160 associates the target image received from the data acquiring unit 110 with the correct image received from the generating unit 150 to form a pair. Then, the teacher data adding unit 160 adds the pair of the target image and the correct image to the teacher data stored in the data server 200.

(S1070)
In step S1070, the correct image generating unit 101 determines whether or not there is a target image for which no teacher data has been created, i.e., a target image that is not paired with a correct image. If there is a target image for which no teacher data has been created (S1070: Yes), the correct image generating unit 101 returns the process to S1010 and acquires the target image. On the other hand, if there is no target image for which no teacher data has been created (S1070: No), the correct image generating unit 101 ends the process of this flowchart.

By processing the above flowchart, the correct image generation unit 101 generates a correct image that corresponds to the target image.

Next, the learning unit 102 will be described with reference to FIG. 1. As shown in FIG. 1, the learning unit 102 has a data acquisition unit 170 and a classifier learning unit 180.

The data acquisition unit 170 acquires training data from the data server 200 and transmits the acquired training data to the classifier training unit 180. The training data includes pairs of target images and correct images added by the training data addition unit 160 of the correct image generation unit 101 described above. Then, the classifier is trained using the target images as training data and the correct images as correct data.

The learning unit 102 acquires training data from the data server 200 by the data acquisition unit 170, and uses the acquired training data to train the classifier. The learning unit 102 acquires parameters for adjusting the classifier through learning. Note that the parameters of the classifier are publicly known, so a detailed description of the parameters will be omitted. The learning unit 102 then stores the acquired parameters of the classifier in the data server 200.

The learning algorithm applied to the classifier is uniquely determined according to the type of classifier used in the image processing device 100. The learning algorithm applied to the classifier is publicly known. The learning unit 102 executes an algorithm that is a modified version of one of these publicly known learning algorithms. The difference between the publicly known learning algorithm and the learning algorithm executed by the classifier learning unit 180 is the method of calculating the loss function that is calculated when determining the parameters of the classifier.

The learning unit 102 calculates the value of the loss function by referring to only pixels (learning area) in the correct answer image that are labeled as "area of interest" or "not an area of interest". On the other hand, the learning unit 102 does not use pixels labeled as "not used for learning" in calculating the value of the loss function because they are not in the learning area. Therefore, in this embodiment, the learning unit 102 omits the loss function processing described below for pixels in the correct answer image that are labeled as "not used for learning". This is expected to reduce the processing load of learning using teacher data in the image processing device 100 compared to the processing load associated with conventional learning.

Next, the calculation method of the loss function in this embodiment will be described by taking binary cross entropy, which is generally used in neural networks, as an example. Here, it is assumed that the teacher data includes a set of target images _Ii , which are T sets of learning data, and correct images _Ri , which are correct data (i = 1, 2, ... T). An extracted image _Oi is the result of extracting an attention area obtained by applying a classifier to the target image _Ii . The image sizes of the correct image _Ri and the extracted image _Oi are M (pixels) x N (pixels), the pixel value of the correct image is _Ri (x, y), and the pixel value of the extracted image is _Oi (x, y) (1 ≤ x ≤ M, 1 ≤ y ≤ N).

As described above, the generation unit 150 expresses the pixel values of the correct image with three labels ("area of interest", "not an area of interest", and "not used for learning"). In addition, the learning unit 102 uses an expression format based on a known learning algorithm to set the pixel value for the label "area of interest" to 1 and the pixel value corresponding to the label "not an area of interest" to 0. In the following description, for convenience, the pixel value of a pixel in an area other than the learning area is expressed as "not used for learning", but any pixel value may be used as the pixel value corresponding to the label "not used for learning" as long as it indicates that the pixel is not used for learning. The calculation formula for the value E of the loss function in this embodiment is expressed by the following formula (1).

On the right side of equation (1), the value of R _i (x, y) log O _i (x, y) is set to "0" for pixels not used in learning. This prevents the pixels not used in learning from affecting the loss function value E. Equation (1) is an example of an equation indicating that pixels in the target image that have been labeled "not used in learning" by the correct image generation unit 101 are not used in calculating the loss function value in learning the classifier.

The calculation method of the loss function value E is not limited to the calculation method using formula (1). For example, the loss function value E may be calculated by referring to only the pixels of the target image corresponding to the pixels in the correct answer image whose pixel values are other than "not used for learning". The learning unit 102 scans the correct answer image R _i and creates a list whose elements are pixels whose pixel values are other than "not used for learning". Here, the created list is represented by P. P is expressed by the following formula (2).

Next, the learning unit 102 calculates the value E of the loss function using the following equation (3).

In the above explanation, the loss function is calculated using binary cross entropy, but the loss function can also be calculated using other calculation methods (cross entropy, mean square error, etc.).

The learning unit 102 may also calculate the value of the loss function using a weight calculated from the ratio of the number of pixels in the extraction region depicted in the target image _Ii to the number of pixels in the region other than the extraction region (background region). In this case, the learning unit 102 determines an equation w(l) representing the weight. The equation w(l) representing the weight is expressed by the following equation (4).

ここで、Ｗ_ｆｇとＷ_ｂｇは、それぞれ重みの値を表す実数であり、定数である。Ｗ_ｆｇとＷ_ｂｇは、識別器が処理する典型的な画像に描出されている注目領域の画素数と注目領域以外の領域（背景領域）の画素数の比に基づいて決定される。Ｗ_ｆｇとＷ_ｂｇを決定する方法の１つとして、データサーバ２００に格納されている教師データに含まれる正解画像から決定する方法が挙げられる。具体的には、以下の式（５）、（６）によってＷ_ｆｇとＷ_ｂｇが決定される。

ここで、Ｇは集合であり、Ｇは正解画像生成部１０１により追加された正解画像以外の画像を示すインデックスを要素に持つ。＃_ｆｇ（Ｒ_ｊ）は、正解画像Ｒ_ｊの画素のうち、画素値が１である画素の数である。＃_ｂｇ（Ｒ_ｊ）は、正解画像Ｒ_ｊの画素のうち、画素値が０である画素の数である。また、学習部１０２は、損失関数の値を計算する前に、式
（５）および式（６）を用いてＷ_ｆｇとＷ_ｂｇの値を計算してもよいし、事前に計算されたＷ_ｆｇとＷ_ｂｇの値をデータサーバ２００から取得してもよい。

Here, W _fg and W _bg are real numbers representing weight values and are constants. W _fg and W _bg are determined based on the ratio of the number of pixels in a region of interest depicted in a typical image processed by the classifier to the number of pixels in a region other than the region of interest (background region). One method for determining W _fg and W _bg is to determine them from a correct image included in the teacher data stored in the data server 200. Specifically, W _fg and W _bg are determined by the following formulas (5) and (6).

Here, G is a set, and G has an index indicating an image other than the correct image added by the correct image generating unit 101 as an element. # _fg (R _j ) is the number of pixels having a pixel value of 1 among the pixels of the correct image R _j . # _bg (R _j ) is the number of pixels having a pixel value of 0 among the pixels of the correct image R _j . In addition, the learning unit 102 may calculate the values of W _fg and W _bg using equations (5) and (6) before calculating the value of the loss function, or may obtain the values of W _fg and W _bg calculated in advance from the data server 200.

ここで、Ｗ_ｆｇ（ｉ）とＷ_ｂｇ（ｉ）は、重みの値を表す実数であり、ターゲット画像Ｉ_ｉに応じて変化する変数である。学習部１０２は、複数の画像間における画像の類似度に基づいて、複数の画像から、ターゲット画像Ｉ_ｉと画像の類似度が近似する類似画像としての画像を特定する。画像の類似度が互いにどの程度近似するものを類似画像とするかは適宜決定することができる。そして、学習部１０２は、特定した類似画像内に描出されている注目領域の画素数と注目領域以外の領域（背景領域）の画素数との比を基にＷ_ｆｇ（ｉ）とＷ_ｂｇ（ｉ）の値を決定する。 Furthermore, the equation representing the weight may be changed depending on the target image I _i . In this case, the equation w _i (l) representing the weight is expressed by the following equation (7).

Here, W _fg (i) and W _bg (i) are real numbers representing weight values, and are variables that change according to the target image I _i . The learning unit 102 identifies, from among the multiple images, an image that is similar to the target image I _i in image similarity based on the image similarity between the multiple images. It is possible to appropriately determine the degree to which images with similar similarity to each other are to be regarded as similar images. Then, the learning unit 102 determines the values of W fg (i) and W _bg (i) based on the ratio between the number of pixels in the attention area depicted in the identified similar image and the number of pixels in the area other than the attention area (background _area ).

Next, a specific example of a method for determining the values of W _fg (i) and W _bg (i) will be described. The learning unit 102 calculates the image similarity between the target image I _i and a target image I _k other than I _i among the target images included in the teacher data stored in the data server 200. Then, the target image with the highest similarity (here, the target image I _{k '} ) is specified. Next, the learning unit 102 determines the values of W _fg (i) and W _bg (i) from the correct image R _{k '} corresponding to I _{k '} using the following formulas (8) and (9).

The method of determining the values of W _fg (i) and W _bg (i) by the learning unit 102 is not limited to the above method using the teacher data stored in the data server 200. For example, a typical image processed by the classifier, the number of pixels of the attention region depicted in the image, and the number of pixels of the region other than the attention region (background region) may be paired and stored in the data server 200. Here, examples of typical images include images of a type of image that is processed in large numbers by the learning unit 102, such as an image of a part of the target image that is included in a target image and an image having the same feature values such as hue, saturation, brightness, and luminance as the target image. In this case, the learning unit 102 calculates the similarity for each of the typical images stored in the data server 200. Then, the learning unit 102 acquires from the data server 200 the number of pixels N _fg of the attention region and the number of pixels N _bg of the region other than the attention region (background region) in the image with the highest similarity. Then, the learning unit 102 can determine the values of W _fg (i) and W _bg (i) using the following equations (10) and (11).

After determining w(l) or _wi (l) using equation (4) or equation (7), the learning unit 102 calculates the loss function value E using w(l) or _wi (l) and the following equation (12).

ここで、式（１３）に示す損失関数の計算式は、式（３）に示す損失関数の計算式に対応する式である。 In the example of calculating the loss function by formula (12), _wi (l) is used, but when determining w(l) by formula (4), _wi (l) can be replaced with w(l) in formula (12). The loss function value E may be calculated by using the following formula (13) instead of formula (12).

Here, the calculation formula for the loss function shown in formula (13) is an equation corresponding to the calculation formula for the loss function shown in formula (3).

The learning unit 102 calculates the loss function value E using the above formulas (1), (3), (8), and (9). This concludes the explanation of how the learning unit 102 calculates the loss function value.

Next, the learning process of the classifier by the learning unit 102 will be described with reference to FIG. 6. The learning unit 102 learns the classifier using a known learning algorithm. Here, the learning process of the learning unit 102 will be described using a neural network learning algorithm as an example. Note that, of the processing steps described below, in the processing steps other than step S2040, processing similar to that in a known neural network learning algorithm is executed.

(S2010)
In step S2010, the data acquisition unit 170 of the learning unit 102 acquires teacher data from the data server 200.

(S2020)
In step S2020, the learning unit 102 sets initial values of the parameters of the classifier. In this step, the initial values are set using a known method, such as the method of determining initial values proposed by He et al. ([Kaiming He, et al., “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” ICCV, 2015]) or a method of setting constant values or random values as initial values.

(S2030)
In step S2030, the learning unit 102 applies a classifier to each target image included in the training data to extract a region of interest depicted in the target image. As shown in Fig. 6, steps S2030 to S2060 are repeatedly executed. When step S2030 is executed for the first time, the learning unit 102 uses the parameters set in step S2020 as the parameters of the classifier. When step S2030 is executed for the second or subsequent times, the learning unit 102 uses the parameters calculated in step S2050 as the parameters of the classifier.

(S2040)
In step S2040, the learning unit 102 calculates the loss function value E[t]. Here, the subscript "t" means that the process from step S2030 to step S2060 is executed for the tth time. The learning unit 102 calculates the loss function value E[t] according to the above-mentioned method for calculating the loss function.

(S2050)
In step S2050, the learning unit 102 calculates the parameters of the classifier. Specifically, the learning unit 102 calculates the parameters using a known calculation method such as gradient descent, Stochastic Gradient Descent (SGD), Adam, etc. After that, the learning unit 102 updates the parameters of the classifier by replacing the parameters of the classifier with the parameters obtained in this step.

(S2060)
In step S2060, the learning unit 102 judges whether or not to end the update of the parameters of the classifier, that is, whether or not to end the learning of the classifier. For example, the learning unit 102 calculates the difference between the value E[t-1] of the loss function calculated in the previous process of step S2040 and the value E[t] of the loss function calculated in the current process of step S2040. If the calculated difference is greater than a predetermined value, the learning unit 102 returns the process to step S2030. On the other hand, if the calculated difference is equal to or less than the predetermined value, the learning unit 102 advances the process to step S2070.

(S2070)
In step S2070, the learning unit 102 transmits the parameters of the classifier updated in S2050 to the data server 200.

The image processing device 100 according to this embodiment can efficiently acquire areas in a target image that the classifier cannot correctly extract. The image processing device 100 then creates a correct answer image based on the acquired areas, excluding areas that the classifier can correctly extract from the learning target. As a result, the image processing device 100 can reduce the processing load associated with learning the classifier and improve the learning efficiency by training the classifier using only areas that the classifier cannot correctly extract.

Second Embodiment
An image processing device 700 according to the second embodiment will be described with reference to FIG. 7. The image processing device 700 has a generated image generating unit 701 and a learning unit 702. The learning unit 702 of the image processing device 700 performs the same operation as the learning unit 102 of the image processing device 100 in the first embodiment, and therefore a detailed description of the learning unit 702 is omitted here. Note that a correct answer image for which the correct answer is known may be extracted by a first extraction means, and a learning area may be specified from the difference between the extraction result by the first extraction means and the correct answer image. Furthermore, a correct answer image may be generated based on the specified learning area, and a loss function may be calculated.

The correct image generating unit 701 has a data acquiring unit 710, a first region extracting unit 720, a second region extracting unit 730, a region identifying unit 740, a region selecting unit 750, a generating unit 760, and a teacher data adding unit 770. Of the units of the correct image generating unit 701, the first region extracting unit 720, the region identifying unit 740, and the teacher data adding unit 770 perform the same operations as the region extracting unit 120, the region identifying unit 130, and the teacher data adding unit 160 of the image processing device 100 in the first embodiment. For this reason, detailed descriptions of these processing units will be omitted here.

The data acquisition unit 710 acquires a target image from the data server 200, which corresponds to the data server according to the first embodiment. The target image may be an image in which an attention area is depicted within the target image, or an image in which an attention area is not depicted. The data acquisition unit 710 transmits the acquired target image to the first area extraction unit 720, the second area extraction unit 730, the area selection unit 750, and the teacher data addition unit 770.

The second region extraction unit 730 applies image processing to the target image using a method different from the image processing executed by the first region extraction unit 720 to extract a region of interest depicted in the target image. The second region extraction unit 730 transmits the extracted extraction region to the region identification unit 740. In the following description, the region extracted by the first region extraction unit 720 is referred to as the first extraction region, and the region extracted by the second region extraction unit 730 is referred to as the second extraction region. In addition, the result of the processing by the first region extraction unit 720 is referred to as the first extraction result, and the result of the processing by the second region extraction unit 730 is referred to as the second extraction result.

Here, the image processing executed by the second region extraction unit 730 will be described. The second region extraction unit 730 may use any image processing method as long as it is different from the image processing method used by the first region extraction unit 720, but a preferred image processing method is the graph cut segmentation method. The graph cut segmentation method is a well-known image processing method, and is disclosed in detail, for example, in Patent Publication US6973212. Below, the same description as that disclosed in this patent publication will be omitted, and the differences will be mainly described.

In the graph cut segmentation method, a value (likelihood) indicating that the pixel corresponding to the pixel node is a pixel in the region of interest is set as energy to the edge (t-link) connecting the pixel node and the terminal node S. Also, a value (likelihood) indicating that the pixel corresponding to the pixel node is not a pixel in the region of interest is set as energy to the edge (t-link) connecting the pixel node and the terminal node T. The output value of the classifier executed in the first extraction unit 720 is used for each likelihood.

The pixels included in the first extraction region extracted by the first region extraction unit 720 are set as foreground seeds. The pixels located at a position that is more than a predetermined distance away from the pixels included in the first extraction region are set as background seeds. Significantly large values are then set to the edge (t-link) between the pixel node corresponding to the foreground seed and the terminal node S, and the edge (t-link) between the pixel node corresponding to the background seed and the terminal node T. By setting such values and executing the graph cut segmentation method, the second region extraction unit 730 extracts a second extraction region as a region of interest from the target image.

With reference to FIG. 9, the second extraction region extracted by the second region extraction unit 730 will be described. An image 910 shown in FIG. 9A is a target image. Regions 911 and 912 in the image 910 are regions of interest. An image 920 shown in FIG. 9B is an image showing a result (first extraction result) of the first region extraction unit 720 extracting a region of interest from the target image 910 using a classifier. Regions 921 and 922 in the image 920 are regions obtained by extracting the regions of interest 911 and 912 by the classifier. Region 923 in the image 920 is a region that was not correctly extracted as a region of interest by the classifier. Region 924 in the image 920 is a region that is not part of the region of interest but is erroneously extracted as a region of interest by the classifier.

9C shows an image 930 that shows the result (second extraction result) of the second region extraction unit 730 extracting the second extraction region as a region of interest from the target image 910 by the graph cut segmentation method. As can be seen by comparing the region of interest 911 of the image 910 with the second extraction region 931 of the image 930, and the region of interest 912 of the image 910 with the second extraction region 932 of the image 930, most of the regions of interest 911 and 912 are correctly extracted by the second region extraction unit 730. Also, as can be seen by comparing the image 920 (first extraction result) with the image 930 (second extraction result), the region 923 that was not correctly extracted in the first extraction result is correctly extracted in the second extraction result. Also, in the image 930, the region 924 that was erroneously extracted by the classifier is not extracted. Therefore, the second region extraction unit 730 can extract the region of interest with higher accuracy than the first region extraction unit 720.

The region identification unit 740 refers to the first extraction result extracted by the first region extraction unit 720 and the second extraction result extracted by the second region extraction unit 730 to acquire a region to be used by the learning unit 702 when training the classifier. The region identification unit 740 transmits the acquired region to the region selection unit 750.

The region identification unit 740 acquires the erroneously extracted region included in the first extraction result as a region to be used for learning. For example, the region identification unit 740 acquires a region that is extracted as a region of interest by the first region extraction unit 720 (first extracted region) and is not extracted as a region of interest by the second region extraction unit 730 as a region to be used for learning the classifier. In the example of FIG. 9B, the regions 921, 922, and 924 in the image 920 are regions extracted as regions of interest by the first region extraction unit 720 (first extracted regions). In the example of FIG. 9C, the region 935 in the image 930 is a region that is not extracted as a region of interest by the second region extraction unit 730.

Furthermore, region 924 in image 920 is the first extracted region extracted as a region of interest by first region extraction unit 720, and is also a region that was not extracted as a region of interest by second region extraction unit 730. Based on a comparison of regions 921, 922, and 924 with region 935, region identification unit 740 identifies region 924 as a region that is not a region of interest but was erroneously extracted as a region of interest by first region extraction unit 720 (hereinafter referred to as an over-extracted region). Then, region identification unit 740 identifies region 924 as a region to be used for training the classifier.

The region identification unit 740 also identifies the second extracted region, which is not extracted as a region of interest by the first region extraction unit 720 and is extracted as a region of interest by the second region extraction unit 730, as a region to be used for learning the classifier. For example, regions 923 and 925 in image 920 are regions that are not extracted as regions of interest by the first region extraction unit 720. Regions 931 and 932 in image 930 are second extracted regions that are extracted as regions of interest by the second region extraction unit 730. Here, region 923 is a region that is not extracted as a region of interest by the first region extraction unit 720 and is a second extracted region that is extracted as a region of interest by the second region extraction unit 730. Therefore, based on a comparison of regions 923, 925, 931, and 932, the region identifying unit 740 identifies region 923 as a region that is of interest but was not correctly extracted as a region of interest by the first region extraction unit 720 (hereinafter referred to as an unextracted region). Then, the region identifying unit 740 identifies region 923 as a region to be used for training the classifier.

9D is an image showing the regions to be used for training the classifier, which have been identified by the above-described processing of the region identification unit 740. As shown in image 940, the region identification unit 740 identifies regions 923 and 924 in the image 920 as regions 943 and 944 to be used for training the classifier, respectively.

Note that the area identification unit 740 does not necessarily need to perform the process of identifying area 943 and the process of identifying area 944, and may perform one of these processes to identify the area to be used for training the classifier.

The region selection unit 750 regards the regions identified by the region identification unit 740 as candidate regions, and determines for each candidate region whether or not to use the region in training of a classifier by the training unit 702. The region selection unit 750 then transmits information indicating the regions determined to be used for training to the generation unit 760.

The area selection unit 750 performs the above-mentioned determination processes 1 to 4 performed by the area selection unit 140 of the image processing device 100 according to the first embodiment, as well as the following determination process 6. Note that in the following description, each area acquired by the area identification unit 740 is referred to as a candidate area.

[Determination process 6: Determination based on distance 2 from the boundary of the maximum connected region]
The region selection unit 750 measures the distance of the candidate region from the boundary of the maximum connected region within the region acquired by the second region extraction unit 730. As an example, the distance from the boundary of the maximum connected region to the candidate region may be the shortest distance between the boundaries of the regions. Then, the region selection unit 750 selects, from among the candidate regions, a candidate region that is away from the boundary of the maximum connected region by more than a predetermined distance as a region to be used for learning.

The process of the region selection unit 750, which identifies the region to be used for training the classifier from the candidate regions, will be described with reference to Figures 10A to 10F. Image 1010 shown in Figure 10A is the first extraction result obtained by the first region extraction unit 720. In image 1010, regions 1011, 1012, and 1013 are the first extracted regions. Next, image 1020 is the second extraction result obtained by the second region extraction unit 730. Regions 1021 and 1022 are the second extracted regions. Image 1030 is the processing result of the region identification unit 740. By comparing images 1010 and 1020, the region identification unit 740 acquires regions 1031 and 1032 as regions to be used for training (candidate regions).

The region selection unit 750 first identifies the largest connected region having the largest area among the second extracted regions acquired by the second region extraction unit 730. In the image 1020 shown in FIG. 10B, the regions 1021 and 1022 are the largest connected regions identified by the region selection unit 750.

Next, the region selection unit 750 identifies a region of the second extracted region that exists inside the maximum connected region 1021 and 1022. The region selection unit 750 then selects, from the identified region, a region that exists further inside than a predetermined distance from the boundary of the maximum connected region formed by connecting regions 1021 and 1022, as a region (candidate region) to be used for learning. An image that shows this process is image 1040 in FIG. 10D. In image 1040, contour lines 1041 and 1042 correspond to the boundary between regions 1021 and 1022 that form the maximum connected region. Then, dotted lines 1043 and 1044 represent positions that are a predetermined distance away from the boundary between the maximum connected regions 1021 and 1022 toward the inside of the region.

Of the second extracted regions, the region selection unit 750 identifies region 1031 as a region existing inside maximally connected regions 1021 and 1022. Then, of region 1031, the region selection unit 750 selects region 1045, which exists away from the boundary between maximally connected regions 1021 and 1022 by more than a predetermined distance, as a region to be used for learning (candidate region). On the other hand, of region 1031, the region selection unit 750 excludes region 1046, which exists within a predetermined distance from the boundary between maximally connected regions 1021 and 1022, from the regions to be used for learning (candidate region) and does not select it.

The region selection unit 750 also identifies regions of the second extraction region that exist outside the maximally connected regions 1021, 1022. The region selection unit 750 then selects, from the identified regions, regions that exist away from the boundary between the maximally connected regions 1021, 1022 by more than a predetermined distance as regions to be used for learning (candidate regions). An image that illustrates this process is image 1050 in FIG. 10E. In image 1050, contour lines 1051 and 1052 correspond to the boundary between regions 1021 and 1022 that form the maximally connected region. Dotted line 1053 indicates a position that is a predetermined distance away from the boundary between the maximally connected regions 1021, 1022 toward the outside of the regions.

The region selection unit 750 identifies region 1032 of the second extracted region as a region that exists outside the maximal connected regions 1021, 1022. The region selection unit 750 then selects region 1055 of region 1032 that exists more than a predetermined distance away from the boundary between the maximal connected regions 1021, 1022 as a region to be used for learning (candidate region). On the other hand, the region selection unit 750 excludes region 1054 of region 1032 that exists within a predetermined distance from the boundary between the maximal connected regions 1021, 1022 from the regions to be used for learning (candidate region) and does not select it.

Finally, image 1060 shown in FIG. 10F is an image showing the area to be used for learning selected by area selection unit 750 by executing judgment process 6 described above. In image 1060, area 1061 corresponds to area 1045 in image 1040 shown in FIG. 10D. Also, in image 1060, area 1062 corresponds to area 1055 in image 1050 shown in FIG. 10E.

The area selection unit 750 executes at least one of the judgment processes 1, 2, 3, 4, and 6, and selectively acquires an area to be used for learning from the areas acquired by the area identification unit 740.

In this embodiment, the correct image generating unit 701 does not need to execute the judgment process by the area selecting unit 750 to obtain the area to be used for training the classifier. Therefore, the correct image generating unit 701 may be configured not to include the area selecting unit 750. If the correct image generating unit 701 does not include the area selecting unit 750, the area identifying unit 740 transmits the area identified by the above process to the generating unit 760.

Next, the generation unit 760 generates a correct answer image corresponding to the target image based on the area to be used for learning transmitted from the area identification unit 740 or the area selection unit 750.

Specifically, the generation unit 760 assigns a label of "area of interest" to pixels constituting a region that was not extracted as a region of interest by the first region extraction unit 720 and was extracted as a region of interest by the second region extraction unit 730. Using image 940 shown in FIG. 9D as an example, the generation unit 760 assigns a label of "area of interest" to pixels constituting region 943. Using image 1060 shown in FIG. 10F as an example, the generation unit 760 assigns a label of "area of interest" to pixels constituting region 1061.

Furthermore, the generating unit 760 assigns a label of "not a region of interest" to pixels constituting a region that is extracted as a region of interest by the first region extracting unit 720 and is not extracted as a region of interest by the second region extracting unit 730. Taking the image 940 shown in FIG. 9D as an example, the generating unit 760 assigns a label of "not a region of interest" to pixels constituting a region 944. Taking the image 1060 shown in FIG. 10F as an example, the generating unit 760 assigns a label of "not a region of interest" to pixels constituting a region 1062.

Finally, the generating unit 760 assigns the label "not used for learning" to pixels that have not been assigned the label "area of interest" or "not an area of interest". Using the image 940 shown in FIG. 9D as an example, the generating unit 760 assigns the label "not used for learning" to pixels that make up the area 945 other than the areas 943 and 944. Using the image 1060 shown in FIG. 10F as an example, the generating unit 760 assigns the label "not used for learning" to pixels that make up the area 1063 other than the areas 1061 and 1062. The generating unit 760 then generates an image that holds the labels assigned by the above process as the pixel values of each pixel as a correct image. The generating unit 760 transmits the generated correct image to the teacher data adding unit 770.

Next, the process executed by the correct image generating unit 701 will be described with reference to the flowchart in FIG. 8.

(S3010)
In step S3010, the data acquisition unit 710 acquires teacher data from the data server 200. Then, the data acquisition unit 710 transmits the acquired target image to the first region extraction unit 720, the second region extraction unit 730, the region selection unit 750 (if present), and the teacher data addition unit 770, respectively.

(S3020)
In step S3020, the first region extraction unit 720 acquires a target image from the data acquisition unit 710. Then, the first region extraction unit 720 applies a classifier to the target image to extract a region of interest depicted in the target image. The first region extraction unit 720 transmits the extracted region to the second region extraction unit 730.

(S3030)
In step S3030, the second region extraction unit 730 acquires the target image from the data acquisition unit 710. The second region extraction unit 730 also acquires the first extraction result from the first region extraction unit 720. The second region extraction unit 730 then extracts a second extraction region from the target image as a region of interest by using an image processing method different from the image processing method used by the first region extraction unit 720 on the target image. The second region extraction unit 730 transmits the extracted region to the second region extraction unit 740.

(S3040)
In step S3040, the region identifying unit 740 identifies a region to be used for training the classifier by referring to the first extraction result received from the first region extraction unit 720 and the second extraction result received from the second region extraction unit 730. If the correct image generation unit 701 has a region selection unit 750, the region identifying unit 740 transmits the identified region to the region selection unit 750 as a candidate region for the region to be used for training. On the other hand, if the correct image generation unit 701 does not have a region selection unit 750, the region identifying unit 740 transmits the identified region to the generation unit 760 as a region to be used for training.

(S3050)
In step S3050, the region selection unit 750 acquires candidate regions for regions used for learning the classifier from the region identification unit 740. That is, the region selection unit 750 regards the region received from the region identification unit 740 as a candidate region. The region selection unit 750 judges whether each candidate region is a region to be used for learning based on at least one of the features related to the shape of the region, the features related to the gray value of the region, and the features related to the position of the region in the image. Then, the region selection unit 750 transmits information on the region judged as a region to be used for learning to the generation unit 760. Here, the judgment process executed by the region selection unit 750 is as described above for the region selection unit 750. Note that, if the correct image generation unit 701 does not have the region selection unit 750, the process of step S3050 is not executed, and the process proceeds from step S3040 to S3060.

(S3060)
In step S3060, the generating unit 760 generates a correct answer image corresponding to the target image based on the region to be used for learning selected by the region identifying unit 740 or the region selecting unit 750. Here, the process of generating the correct answer image by the generating unit 760 is as described above for the generating unit 760. Then, the generating unit 760 transmits the generated correct answer image to the teacher data adding unit 770.

(S3070)
In step S3070, the teacher data adding unit 770 associates the target image received from the data acquiring unit 710 with the correct image received from the generating unit 760 to form a pair. Then, the teacher data adding unit 770 adds the pair of the associated target image and correct image to the teacher data stored in the data server 200.

As described above, the correct image generation unit 701 generates a correct image corresponding to the target image obtained from the data server 200.

Note that the processing performed by the learning unit 702 is the same as the processing performed by the learning unit 102 of the image processing device 100 according to the first embodiment, specifically steps S2010 to S2070, and therefore will not be described here.

As described above, the image processing device 700 according to the second embodiment can efficiently obtain two types of regions in a target image that the classifier cannot correctly extract. Here, the two types of regions are (1) regions that are of interest but are not correctly extracted as such, and (2) regions that are not of interest but are erroneously extracted as such. Then, by creating a correct answer image to which a label has been assigned based on the obtained regions, the regions that the classifier cannot correctly extract can be used for training the classifier, while the regions that the classifier can correctly extract are excluded from training, thereby improving the learning efficiency of the classifier.

Other Embodiments
The above-described embodiment merely shows a specific example of the present invention. The scope of the present invention is not limited to the configuration of the above-described embodiment, and various embodiments can be adopted without departing from the gist of the present invention.

The disclosed technology can be embodied, for example, as a system, device, method, program, or recording medium (storage medium). Specifically, it may be applied to a system consisting of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or it may be applied to an apparatus consisting of a single device.

Needless to say, the object of the present invention can be achieved by the following: A recording medium (or storage medium) on which is recorded software program code (computer program) that realizes the functions of the above-mentioned embodiments is supplied to a system or device. Such a storage medium is, needless to say, a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or device reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-mentioned embodiments, and the recording medium on which the program code is recorded constitutes the present invention.

100: Image processing device 120: Region extraction unit 130: Region identification unit 150: Generation unit 160: Teacher data addition unit

Claims (24)

a region extraction unit that extracts a region that is identified as a region of interest from a target image using a classifier as an extracted region;
an area specifying unit that specifies a learning area to be used for learning the classifier based on the extracted area extracted by the area extracting unit;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area;
a teacher data adding unit that adds a pair of the target image and the correct image to teacher data;
a learning unit that learns the classifier by referring to the learning area in the correct image included in the training data;
Equipped with
The learning unit calculates a value of a loss function for the target image and the correct image using weights.
13. An image processing device comprising: The image processing device according to claim 1, characterized in that the region identification unit creates a connected region by connecting the extracted regions extracted by the region extraction unit, and identifies the learning region based on the area of the created connected region. The image processing device according to claim 2, characterized in that the region identification unit identifies the Nth (N is an integer equal to or greater than 2) or subsequent region from the largest area among the created connected regions as the learning region. 4. The image processing device according to claim 1 , wherein the region identification unit selects a learning region to be used for training the classifier from the extracted region extracted by the region extraction unit, based on at least one of a feature related to a length of the extracted region, a feature related to an area of the extracted region, a feature related to a shape of the extracted region, and a feature related to pixel values of pixels constituting the extracted region. The image processing device according to any one of claims 1 to 4, characterized in that the region identification unit selects, from the extracted region extracted by the region extraction unit, a region that is more than a predetermined distance away from the boundary of the region with the largest area among the extracted regions, as a learning region to be used for training the classifier. The image processing device according to any one of claims 1 to 5, characterized in that the generation unit generates the correct image so as to indicate that areas other than the learning area selected by the area identification unit are not used in training the classifier. a region extraction unit that extracts a region identified as a region of interest from a target image using a classifier as a first extracted region, and extracts a second extracted region from the target image as a region of interest by executing image processing different from the image processing used for extracting the first extracted region;
a region specifying unit that specifies a learning region to be used for learning the classifier based on a comparison between the first extracted region and the second extracted region;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area or is an attention area;
An image processing device comprising: The image processing device according to claim 7, characterized in that the region identification unit identifies, as the learning region, at least one of a region that is the first extraction region and is not the second extraction region, and a region that is not the first extraction region and is the second extraction region. The image processing device according to claim 8, characterized in that the generation unit generates the correct image so as to indicate that an area of the learning area identified by the area identification unit that is the first extraction area and is not the second extraction area is not a region of interest, and that an area of the learning area identified by the area identification unit that is not the first extraction area and is the second extraction area is a region of interest. The image processing device according to any one of claims 7 to 9, characterized in that the region identification unit selects a learning region to be used for training the classifier from the first extracted region extracted by the region extraction unit based on at least one of the following features: a feature related to the length of the first extracted region, a feature related to the area of the first extracted region, a feature related to the shape of the first extracted region, and a feature related to the pixel values of the pixels constituting the first extracted region. The image processing device according to any one of claims 7 to 10, characterized in that the region identification unit selects, from the first extracted region extracted by the region extraction unit, a region that is away from the boundary of the region having the largest area among the second extracted regions by more than a predetermined distance toward the inside or outside of the region, as a learning region to be used for training the classifier. The image processing device according to any one of claims 7 to 11, characterized in that the generation unit generates the correct image so as to indicate that areas other than the learning area identified by the area identification unit are not used for learning. 13. The image processing device according to claim 7, wherein the region extraction unit extracts the second extracted region by a graph cut segmentation method without using the classifier. a storage unit that stores a weight calculated based on a ratio of a number of pixels corresponding to the region of interest to a number of pixels in a region other than the region of interest;
The image processing device according to claim 1 , wherein the learning unit acquires the weights from the storage unit. The image processing device according to claim 1, characterized in that the learning unit calculates a value of a loss function for the target image and the correct image using a ratio between the number of pixels in a region of interest in a similar image that has image features most similar to those of the target image and the number of pixels in a region other than the region of interest. The image processing device according to claim 15, characterized in that the learning unit calculates the value of the loss function by applying a weight based on a ratio between the number of pixels in a region of interest in the similar image and the number of pixels in a region other than the region of interest to pixel values of a correct image for the target image. a region extraction step of extracting, as an extracted region, a region identified as a region of interest from the target image using a classifier;
a region specifying step of specifying a learning region to be used for learning the classifier based on the extracted region extracted by the region extracting step;
a generation step of generating a correct answer image indicating that the learning area identified by the area identification step is not an area of interest;
a teacher data adding step of adding a pair of the target image and the correct image to teacher data;
a learning step of learning the classifier by referring to the learning region in the correct image included in the training data;
Including,
The learning step calculates a loss function value for the target image and the correct image using weights.
13. An image processing method comprising: a region extraction step of extracting a region identified as a region of interest from a target image using a classifier as a first extracted region, and extracting a second extracted region from the target image as a region of interest by performing image processing different from the image processing used for extracting the first extracted region;
a region specifying step of specifying a learning region to be used for training the classifier based on a comparison between the first extracted region and the second extracted region;
a generation step of generating a correct answer image indicating that the learning area identified by the area identification step is not an attention area or is an attention area;
13. An image processing method comprising: 20. The image processing method according to claim 18 , further comprising a teacher data adding step of adding a pair of the target image and the correct answer image to teacher data. A program for causing a computer to execute each step of the image processing method according to any one of claims 17 to 19 . a region extraction unit that extracts a region that is identified as a region of interest from a target image using a classifier as an extracted region;
A learning unit for learning the classifier based on the extracted region extracted by the region extraction unit.
A region specifying unit that specifies a learning region to be processed;
a generation unit that generates a correct answer image indicating that the learning area identified by the area identification unit is not an attention area;
Equipped with
The region identification unit creates a connected region by connecting the extracted regions extracted by the region extraction unit, identifies an erroneously extracted region based on an area of the created connected region, and identifies the erroneously extracted region as the learning region.
13. An image processing device comprising: 22. The image processing apparatus according to claim 21, wherein the region specifying unit specifies, as the learning region, an Nth (N is an integer equal to or greater than 2) or subsequent region from the largest area among the created connected regions. a region extraction step of extracting, as an extracted region, a region identified as a region of interest from the target image using a classifier;
a region specifying step of specifying a learning region to be used for learning the classifier based on the extracted region extracted by the region extracting step;
a generation step of generating a correct answer image indicating that the learning area identified by the area identification step is not an area of interest;
Equipped with
The region identification step creates a connected region by connecting the extracted regions extracted by the region extraction step, identifies an erroneously extracted region based on an area of the created connected region, and identifies the erroneously extracted region as the learning region.
23. An image processing method comprising: 24. The image processing method according to claim 23, wherein said region specifying step specifies, as said learning region, an Nth (N is an integer of 2 or more) or subsequent region counting from the largest area among said created connected regions.

Images