JP7542802B2 - Image recognition device using neural network and program used in the image recognition device - Google Patents

Description

Patent Act Article 30, paragraph 2 applies http://mprg.jp/research/abn_j Posted June 20, 2019, July 2019 http://cvim.ipsj.or.jp/MIRU2019/ (Website of the 22nd Symposium on Image Recognition and Understanding (MIRU2019)) Posted July 2019 22nd Symposium on Image Recognition and Understanding (MIRU2019) Oral Session OS2A-5, Grand Cube Osaka (Osaka Prefectural International Convention Center) July 31, 2019

The present invention relates to an image recognition device and a training device that use a neural network.

Conventionally, in image recognition technology using neural networks such as CNN (Convolutional Neural Network), a technology is known that generates an attention map that represents the gaze area during inference by the neural network (see, for example, Non-Patent Documents 1 and 2).

Ramprasaath, R., S., Michael, C., Abhishek, D., Ramakrishna, V., Devi, P. and Dhruv, B.: Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization, International Conference. on Computer Vision, pp. 618-626 (2017). Zhou, B., Khosla, A., Lapedriza, A., Oliva, A. and Torralba, A.: Learning Deep Features for Discriminative Localization, Computer Vision and Pattern Recognition, pp. 2921-2929 (2016).

However, according to the inventor's research, when a neural network generates a recognition result and an attention map for a certain image, there are cases where the recognition result does not match the gaze area in the attention map. For example, if the recognition result is a "smile" but the gaze area is hair, the recognition result does not match the gaze area in the attention map. If there is a mismatch between the recognition result and the gaze area in the attention map, it is a problem that can lead to errors in the image recognition itself. And currently, there is no way to correct this mismatch.

In view of the above, the present disclosure aims to incorporate human knowledge into the recognition function or learning of a neural network in an image recognition technology that uses a neural network that outputs an attention map.

According to one aspect of the present disclosure, an image recognition device includes an input unit (120) for inputting an image (51) to a neural network (10), a feature map (52) including features of the input image and an attention map (53) expressing a gaze area of the input image , a map correction unit (140) for correcting the generated attention map in response to a correction operation by a person when the neural network generates the feature map (52) including features of the input image and an attention map (53) expressing a gaze area of the input image, and an output unit (160) for outputting the generated recognition result when the neural network (10) generates a recognition result of the image based on a synthesis map (54) in which the corrected attention map and the feature map are synthesized and the image. According to another aspect, a program causes the image recognition device to function.

In this way, by using human knowledge to modify the attention map 53, the recognition unit 14 places emphasis on the area intended by the human. As a result, image recognition can be performed in line with the human's intention. Furthermore, at this time, the parameters of the neural network 10 are not changed. In other words, there is no need to re-learn the neural network 10, and the recognition results intended by the human can be obtained.

The reference symbols in parentheses attached to each component indicate an example of the correspondence between the component and the specific components described in the embodiments described below.

FIG. 1 is a configuration diagram of an image recognition device. FIG. 1 is a diagram illustrating a configuration of a neural network. FIG. 2 is a diagram illustrating a configuration of an attention unit. 11 is a flowchart of a process executed by a processing unit. FIG. 13 is a diagram showing a display in which an input image and an unmodified attention map are superimposed. FIG. 13 is a diagram showing a display in which an input image and an attention map from which a gaze area has been deleted are superimposed. FIG. 13 is a diagram showing a display in which an input image and an attention map with an added gaze area are superimposed. FIG. 13 illustrates the relationship between the modified attention map and the neural network. 13 is a flowchart of a process executed by a processing unit in the second embodiment. FIG. 1 is a diagram showing an outline of relearning.

First Embodiment
Hereinafter, a first embodiment will be described. An image recognition device 1 according to this embodiment includes an operation device 2, a display device 3, a memory 4, and a processing unit 5, as shown in FIG.

The operation device 2 is a device that receives operations from a person and outputs a signal corresponding to the received operation to the processing unit 5. The operation device 2 may be, for example, a mouse, a keyboard, a touch panel, etc. The display device 3 is a device that displays an image to a person.

The memory 4 includes RAM, which is a rewritable volatile storage medium, ROM, which is a non-rewritable non-volatile storage medium, and flash memory, which is a rewritable non-volatile storage medium. RAM, ROM, and flash memory are non-transient tangible storage media. The flash memory stores pre-recorded data of the trained neural network 10.

The processing unit 5 executes a program (not shown) stored in ROM or flash memory, and uses the RAM as a working area during execution to realize various processes described below.

Here, we will explain the neural network 10. As shown in FIG. 3, the neural network 10 is a deep neural network that includes a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a recognition unit 14.

When an input image 51 is input, the neural network 10 generates an attention map 53. The attention map 53 is data that represents the area of attention during inference by the neural network 10. In other words, the attention map 53 is visual explanatory data that explains which area of the input image 51 is emphasized during inference by the neural network 10.

The neural network 10 also outputs a classification result for the input image 51 based on the input image 51 and the attention map 53. The classification result for the input image 51 is a number of likelihoods corresponding to a number of classes (e.g., dalmatian, crayfish, finch, frog, etc.) that correspond to the recognition target of the image. Here, the number of classes is K.

The feature extraction unit 11 is a neural network having multiple layers. These multiple layers include at least multiple convolutional layers. Furthermore, these multiple layers may be components of multiple residual blocks, or may have multiple pooling layers, etc. Then, the feature extraction unit 11 generates a feature map 52 by propagating information of the input image 51 that has been input to these multiple layers.

The feature map 52 is a map with K resolutions h×w, each of which corresponds to one of the K classes. h and w are any integers. Thus, the number of channels in the feature map 52 is K. The resolution of the feature map 52 may be the same as the resolution of the input image 51, or may be lower than the resolution of the input image 51.

The feature extraction unit 11 may be configured by a portion of the baseline model starting from the input layer and preceding the first fully connected layer. As the baseline model, one that has multiple convolutional layers and generates the same types of multiple class likelihoods as the neural network 10 is selected. For example, as the baseline model, the VGGNet shown in Non-Patent Document 3 may be used, the ResNet shown in Non-Patent Document 4 may be used, or another CNN (Convolutional Neural Network) may be used.

The attention unit 12 generates an attention map 53 from the feature map 52 generated by the feature extraction unit 11. The attention unit 12 is a neural network having multiple layers. As shown in FIG. 3, these multiple layers include a first part 12a having one or more convolutional layers or one or more residual blocks, and a K×1×1 convolutional layer 12b at the rear of the first part. Here, if L, a, and b are any natural numbers, an L×a×b convolutional layer means a convolutional layer using an a×b kernel in each of the L channels.

The attention unit 12 has two K×1×1 convolutional layers 12c and 1×1×1 convolutional layers 12e that branch out after the convolutional layer 12b. The attention unit 12 also has a GAP (Global Average Pooling) layer 12d after the convolutional layer 12c.

The information of the feature map 52 input to the attention unit 12 propagates through the first part 12a, the convolution layer 12b, the convolution layer 12c, and the GAP layer 12d, and the output of the GAP layer 12d is input to the Softmax function, generating the likelihood of multiple classes of the same type as the neural network 10 as classification results. The classification results are a type of recognition result.

In addition, the information of the feature map 52 input to the attention unit 12 is propagated to the first part 12a, the convolutional layer 12b, and the convolutional layer 12e to generate the attention map 53. By generating the attention map 53 via the convolutional layer 12b rather than the fully connected layer, the information of the attention area is propagated to the attention map 53 while remaining localized. In addition, by passing through the 1x1x1 convolutional layer 12e, a one-channel attention map 53 is generated as a weighted sum of the attention areas corresponding to all classes. The kernel values of the convolutional layer 12e may all be 1, or may be other values.

The resolution of each map in the feature map 52 is the same as that of the attention map 53. The attention unit 12 is configured so that this is the case. In the attention map 53, pixels that fall within the attention area are given relatively high pixel values, and pixels that do not fall within the attention area are given lower pixel values than those within the attention area. The possible values of each pixel in the attention map 53 may be binary or may be a value in 256 levels. The higher the pixel value of a pixel, the higher the attention level at that pixel's position.

The synthesis unit 13 synthesizes the feature map 52 and the attention map 53. Specifically, the attention map 53 is multiplied by a map of resolution h×w in each of the K channels in the feature map 52. The attention map 53 is multiplied by the map of resolution h×w between pixels at the same position coordinates. Note that the synthesis may be multiplication as described above, addition, or a calculation consisting of a combination of addition and multiplication. A synthesis map 54 is obtained by this synthesis. The number of channels and resolution of the synthesis map 54 are the same as those of the feature map 52.

The cognition unit 14 outputs the likelihood of each class based on the composite map 54. The cognition unit 14 is a neural network having multiple layers. These multiple layers include at least multiple convolution layers. In addition, these multiple layers include one or both of a fully connected layer and a GAP layer. Furthermore, these multiple layers may be components of multiple residual blocks or may have multiple pooling layers. The cognition unit 14 outputs the likelihood of each class as a classification result by propagating the information of the input composite map 54 to these multiple layers. The classification result is also a recognition result. The cognition unit 14 may be configured from the part of the above-mentioned baseline model immediately following the part used in the attention unit 12 to the output layer.

The functions described above as being performed by the neural network 10, feature extraction unit 11, attention unit 12, synthesis unit 13, and recognition unit 14 are actually realized by the processing unit 5 performing processing in accordance with the structure and parameters of the neural network 10.

The feature extraction unit 11, attention unit 12, and synthesis unit 13 are trained in advance by backpropagation learning with a supervised method so as to realize the above-mentioned functions. In the training, L = Latt + Lper is used as the training error (also called a loss function) L. Here, Latt is the training error regarding the classification result output by the attention unit 12, and Lper is the training error regarding the classification result output by the recognition unit 14. Latt and Lper may be calculated by applying a combination of the Softmax function and cross entropy to each classification result. The feature extraction unit 11 is trained by passing through the gradients of the attention unit 12 and the recognition unit 14 in the backpropagation method.

The following describes the image classification process performed by the processing unit 5 using the trained neural network 10 configured in this way.

When a predetermined condition is satisfied, such as a human performing an operation to start execution on the operation device 2, the processing unit 5 starts the process shown in FIG. 4, which is defined in a predetermined program recorded in the memory 4. In this process, the processing unit 5 first reads the neural network 10 from the memory 4 in step 110.

Next, in step 120, an input image 51 is acquired and input to the neural network 10. The input image 51 may be an image selected from a plurality of images previously recorded in the memory 4 by a person operating the operation device 2, or may be an image received from another device via a communication network (not shown).

When an input image 51 is input to the neural network 10, as described above, the feature extraction unit 11 of the neural network 10 generates a feature map 52 and a classification result from the input image 51, and the attention unit 12 generates an attention map 53 from the feature map 52.

In step 130, which follows step 120, the processing unit 5 acquires the attention map 53 thus generated. That is, the processing unit 5 copies or moves the attention map 53 generated in the memory 4 by the neural network 10 to another area in the memory 4.

Next, in step 140, the processing unit 5 modifies the acquired (i.e., copy or move destination) attention map 53 based on the human modification operation on the operation device 2. In this way, the attention map 53 is modified based on human knowledge.

Specifically, the processing unit 5 displays the attention map 53 and the pointer before correction on the display device 3. The pointer is an image that moves within the display range of the attention map 53 displayed on the display device 3 in response to a person's operation on the operation device 2. The person corrects the value of the position range that overlaps with the pointer in the displayed attention map 53 by performing a predetermined correction operation (e.g., a delete operation, an add operation, etc.) on the operation device 2.

At this time, the processing unit 5 may transparently align and overlay the input image 51 on the attention map 53 as shown in FIG. 5 and display the input image 51 on the display device 3, and then reflect the correction corresponding to the above correction operation in the attention map 53. At this time, if the input image 51 and the attention map 53 have different resolutions, the processing unit 5 lowers the resolution of the input image 51 to match the attention map 53, and then transparently overlays the input image 51 on the attention map 53.

In Figure 5, an input image 51 of a Dalmatian holding a soccer ball is transparently overlaid on an attention map 53.

In this attention map 53, the gaze area is in the soccer ball area. If attention map 53 is input as is to synthesis unit 13, and synthesis map 54, which is the result of synthesizing attention map 53 and feature map 52, is input to the first layer of recognition unit 14, the classification result generated by recognition unit 14 will have the highest likelihood of being a soccer ball. In other words, neural network 10 recognizes input image 51 as an image of a soccer ball.

However, if a person using the image recognition device 1 wants the input image 51 to be recognized as an image of a Dalmatian, the fixation area in this attention map 53 should be the area where the Dalmatian is located.

In such a case, therefore, the person uses the operation device 2 to modify the gaze area in the attention map 53. Specifically, first, the person uses the operation device 2 to erase the gaze area in the attention map 53. For example, the person moves the pointer while pressing a specific erase button on the operation device 2 to scan the entire gaze area in the attention map 53. As a result, the processing unit 5 lowers the pixel values of the attention map 53 in the area scanned by the pointer while pressing the erase button, to pixel values that do not become part of the gaze area, as shown in FIG. 6.

Then, the person uses the operation device 2 to set a new area in the attention map 53 that the person wants to set as the attention area. For example, while pressing a specific add button on the operation device 2, the person moves the pointer to scan the entire area in the attention map 53 that the person wants to set as the attention area. As a result, the processing unit 5 increases the pixel values of the attention map 53 in the area scanned by the pointer while pressing the add button, and sets the pixel values to the attention area as shown in FIG. 7. In the example of FIG. 7, the new attention area specified by the person is the face of the Dalmatian.

In this way, by displaying the input image 51 on the display device 3 superimposed on the attention map 53, if a person can determine which part of the input image 51 should be the gaze area, that knowledge can be efficiently used to easily specify the gaze area in the attention map 53. By the processing of step 140 in this way, the attention map 53 obtained in step 130 is corrected in the memory 4.

Next, in step 150, the processing unit 5 inputs the attention map 53 corrected in the previous step 140 to the synthesis unit 13. The synthesis unit 13 then generates a synthesis map 54 by synthesizing the feature map 52 and the attention map 53 as described above, and inputs the synthesis map 54 to the first layer of the recognition unit 14. The recognition unit 14 to which the synthesis map 54 has been input generates a classification result based on the synthesis map 54 as described above. In this classification result, the likelihood of a Dalmatian is the highest. In other words, the neural network 10 recognizes the input image 51 as an image of a Dalmatian.

In step 160 following step 150, the processing unit 5 acquires and outputs the classification result thus generated by the recognition unit 14. The output destination may be another device via a communication network (not shown), the memory 4, or the display device 3.

In this way, by using human knowledge to modify the attention map 53, the recognition unit 14 assigns a higher weight to areas intended by the human. As a result, image recognition can be performed in line with the human's intention. In other words, by using an attention map that has been manually modified based on human knowledge, it becomes possible to adjust the recognition results.

In addition, at this time, the parameters of the neural network 10 are not changed. In other words, there is no need to re-train the neural network 10, and the recognition results intended by the person can be obtained.

For example, when a fundus image is input to the neural network 10 as the input image 51, a doctor can use knowledge based on his or her own experience to modify the gaze area of the attention map 53, thereby making it possible to more accurately identify the grade of eye disease as a class. In this way, the function of this embodiment is useful, for example, in medical image diagnosis.

As described above, the processing unit 5 of the image recognition device 1 modifies the attention map 53 generated by the neural network 10 to which the input image 51 is input in accordance with the human modification operation (step 140), as shown in FIG. 8. The processing unit 5 then outputs the recognition result of the input image 51 generated by the neural network 10 based on the modified attention map 53 and the input image 51 (step 160).

In this way, by using human knowledge to modify the attention map 53, the recognition unit 14 places emphasis on the area intended by the human. As a result, image recognition can be performed in line with the human's intention. Furthermore, at this time, the parameters of the neural network 10 are not changed. In other words, there is no need to re-learn the neural network 10, and the recognition results intended by the human can be obtained.

In addition, the path along which image information is propagated to generate attention map 53 and the path along which image information is propagated to generate recognition results are shared in one part (i.e., feature extraction unit 11) and separated in other parts (i.e., attention unit 12 and recognition unit 14). Then, synthesis unit 13 inputs composite map 54 reflecting the corrected attention map 53 to the recognition unit 14 side of the separated part. In this way, the input point of composite map 54 based on corrected attention map 53 is suited to the structure of neural network 10, thereby improving the degree of improvement in recognition results by corrected attention map 53.

Furthermore, the processing unit 5 reflects corrections made by the person to the attention map 53 in a state in which the input image 51 is transparently overlaid on the attention map 53 and displayed on the display device 3. A person can relatively easily determine which part of the input image 51 should be the gaze area by looking at the input image 51. Therefore, by displaying the input image 51 on the display device 3 and overlaid on the attention map 53, a person can visually and efficiently use his or her own knowledge to easily specify the gaze area in the attention map 53.

In this embodiment, the processing unit 5 functions as an input unit by executing step 120, functions as a map correction unit by executing step 140, and functions as an output unit by executing step 160.

Second Embodiment
Next, a second embodiment will be described. In this embodiment, learning parameters such as weights and biases of the neural network 10 are corrected based on the attention map corrected according to a human correction operation. That is, the neural network 10 is re-learned based on the corrected attention map.

The hardware configuration of this embodiment is the same as that shown in FIG. 1 in the first embodiment. The configuration of the trained neural network 10 stored in the memory 4 is also the same as that in the first embodiment. Note that the image recognition device 1 of this embodiment corresponds to a training device.

One of the differences between this embodiment and the first embodiment is that the processing unit 5 does not execute the process of FIG. 4, but instead causes the neural network 10 to generate a classification result corresponding to the input image 51 without modifying the attention map 53.

That is, the processing unit 5 first reads the neural network 10 from the memory 4, similar to the first embodiment, and then acquires the input image 51 and inputs this input image 51 to the neural network 10.

In the neural network 10, the feature extraction unit 11 and attention unit 12 function in the same manner as in the first embodiment, and the attention unit 12 generates an attention map 53 and a classification result. This attention map 53 is input to the synthesis unit 13 without being modified by a human. The synthesis unit 13 generates a composite map 54 by combining the feature map 52 and the attention map 53 that has not been modified by a human. The recognition unit 14 generates a classification result based on this composite map 54 in the same manner as in the first embodiment. The processing unit 5 acquires and outputs this classification result in the same manner as in the first embodiment.

Then, in addition to the process of acquiring the classification result of the input image 51 from the input image 51 using the neural network 10 as described above, the processing unit 5 executes the process shown in FIG. 9 to re-train the neural network 10. Through this re-training, the neural network 10 is fine-tuned.

The processing unit 5 starts the process of FIG. 9 based on a predetermined relearning start operation being performed by a human on the operation device 2. In this process, the processing unit 5 uses a dataset for relearning. The dataset for relearning has multiple groups (10, 100, or 100,000 groups) each consisting of learning images and teacher labels.

The learning image is data input to the feature extraction unit 11, such as the input image 51. The teacher label is data that is considered to be the correct answer value of the classification result output from the attention unit 12 and the recognition unit 14 when learning images of the same group are input to the feature extraction unit 11.

The re-learning dataset may be generated in advance and recorded in a non-volatile storage medium of memory 4, or may be acquired from a data server via a communication network (not shown). In addition, the training images and teacher labels of the re-learning dataset may be the same as the training dataset used during initial training of neural network 10, or may be different from the training dataset.

In the process of FIG. 9, the processing unit 5 first executes the loop process of steps 210 and 220 for each group included in the re-learning dataset. In each loop process, the processing unit 5 first inputs the learning images in the target group to the feature extraction unit 11 in step 210. Next, in step 220, the processing unit 5 acquires the attention map 53 and classification results generated by the attention unit 12 based on the input learning images, as well as the classification results generated by the recognition unit 14 based on the learning images, and records them in the memory 4.

The method by which the neural network 10 outputs the attention map 53 and the two types of classification results based on the input learning images is the same as the above-mentioned method in which the learning images are replaced with the input images 51. After step 220, one loop of processing is completed.

When the loop processing has been completed for the number of groups, the processing unit 5 proceeds to step 230. At this point, for each group in all re-learning datasets, the attention map 53 and classification results generated by the attention unit 12 and the classification results generated by the recognition unit 14 are associated and recorded in the memory 4.

In step 230, the processing unit 5 extracts a group in which a recognition error has occurred from among the multiple groups. A group in which a recognition error has occurred is extracted in which the class with the highest likelihood in the classification result output by the recognition unit 14 does not match the class indicated by the teacher label (i.e., the class with the highest likelihood in the teacher label). Alternatively, a group in which the class with the highest likelihood in the classification result output by the attention unit 12 does not match the class indicated by the teacher label may be extracted in which a recognition error has occurred. Alternatively, both of these may be extracted. In most cases, multiple groups are extracted.

Next, in step 240, the attention map 53 recorded in the memory 4 corresponding to each of the groups extracted in the previous step 230 is modified based on human knowledge. Specifically, the attention map 53 is modified based on the human modification operation on the operation device 2 by a process similar to that of step 140 in FIG. 4. The processing unit 5 then stores the modified attention map 53 in the memory 4 as a teacher attention map belonging to the group.

The teacher attention map created in this way is data that is considered to be the correct answer value of the attention map 53 output from the attention unit 12 when learning images of the same group are input to the feature extraction unit 11. Through this process, the teacher attention map is added to the re-learning dataset.

Next, in step 250, the processing unit 5 retrains the neural network 10 based on the two types of classification results obtained in the current process of FIG. 9, the attention map 53, and the re-learning dataset. As described above, the re-learning dataset includes a teacher attention map and a teacher label.

Specifically, as shown in FIG. 10, the learning error is determined by the sum of three learning errors Latt, Lper, and Lmap, that is, L=Latt+Lper+Lmap, and the learning parameters such as weights and biases of the attention unit 12 and the recognition unit 14 are updated by the error backpropagation method. In FIG. 10, the output layer 14b of the recognition unit 14 and the part 14a preceding the output layer 14b of the recognition unit 14 are shown. Note that in this embodiment, the learning parameters such as weights and biases of the feature extraction unit 11 are not updated.

Here, Latt is an amount indicating the error between the classification result output by the attention unit 12 when a training image 61 is input to the neural network 10 and the teacher label 60 that belongs to the same group as the training image 61.

In addition, Lper is an amount indicating the error between the classification result output by the feature extraction unit 11 when the training image 61 is input to the neural network 10 and the teacher label 60 that belongs to the same group as the training image 61.

In addition, Lmap is an amount indicating the error between the attention map 53 output by the attention unit 12 when a learning image 61 is input to the neural network 10 and the teacher label 60 that belongs to the same group as the learning image 61.

As the learning error Lmap, an L2 norm error may be adopted as shown in the following formula, or another form of error may be adopted.
Lmap=γ×||M'-M|| ₂
Here, M indicates the value of the attention map 53 output by the attention unit 12 when the learning image 61 is input to the neural network 10. M' indicates the value of the corrected attention map corresponding to the same group as the learning image 61. By calculating the error for each element of these two attention maps, the attention unit 12 is trained to output an attention map close to human knowledge.

Here, γ is a coefficient that adjusts the learning error Lmap. Lmap has a larger error value than Latt and Lper. Therefore, by multiplying Lmap by γ, the magnitudes of the three learning errors Lmap, Latt, and Lper can be adjusted. After step 250, the process of FIG. 9 ends, and the retrained neural network 10 is recorded in memory 4.

In this way, fine-tuning the neural network 10 based on the attention map modified based on human knowledge improves the image recognition function of the neural network 10. In other words, the accuracy rate of the recognition results generated by the recognition unit 14 when the processing unit 5 inputs various input images 51 to the fine-tuned neural network 10 improves.

As described above, the processing unit 5 retrains the neural network 10 using the retraining dataset (step 250). The retraining dataset includes multiple teacher attention maps.

In this way, when retraining the neural network 10 that generates the attention map, the teacher attention map, which is regarded as the correct value of the attention map, is used. Since the teacher attention map is created based on human knowledge, this makes it possible to incorporate human knowledge into the learning of the neural network 10.

The processing unit 5 also inputs a plurality of learning images to the neural network 10 to obtain a plurality of attention maps corresponding to the plurality of learning images (steps 210, 220). The processing unit 5 then modifies the plurality of attention maps in response to a human correction operation to obtain a teacher attention map (step 240).

In this way, a teacher attention map can be generated based on human correction operations made to the attention map generated by the neural network 10. This makes it possible to more directly incorporate human knowledge into the learning of the neural network 10. Moreover, the correction operations are simpler than when creating a teacher attention map from scratch.

The multiple teacher attention maps included in the re-learning dataset are only those training images that were misrecognized by the neural network 10 before re-learning. In this way, by using many teacher attention maps corresponding to misrecognized training images for re-learning, re-learning can be performed more efficiently. This is because the attention map generated using misrecognized training images as input is likely to be itself erroneous.

Furthermore, neural network 10 generates a recognition result for input image 51 based on input image 51 and attention map 53. In this way, in a neural network 10 in which not only input image 51 but also attention map 53 are fed back as information for image recognition, there is a strong correlation between the recognition result for input image 51 and attention map 53. Therefore, in such a neural network 10, the effect of re-learning using the teacher attention map contributes to a high degree to which the recognition result for input image 51 is improved.

The processing unit 5 also re-learns the attention unit 12 without re-learning the feature extraction unit 11. In this way, the parts of the neural network 10 that are strongly related to the generation of the attention map 53 are re-learned, thereby achieving highly efficient fine-tuning of the neural network 10.

In this embodiment, the processing unit 5 functions as a readout unit by executing step 205, as a training unit by executing step 250, as an acquisition unit by executing steps 210 and 220, and as a map correction unit by executing step 240.

Other Embodiments
The present invention is not limited to the above-mentioned embodiment, and can be modified as appropriate. The above-mentioned embodiments are not unrelated to each other, and can be combined as appropriate, except when the combination is clearly impossible. In each of the above-mentioned embodiments, the elements constituting the embodiment are not necessarily essential, except when it is specifically stated that they are essential or when it is clearly considered essential in principle. In each of the above-mentioned embodiments, when the numbers, values, amounts, ranges, etc. of the components of the embodiment are mentioned, they are not limited to the specific numbers, except when it is specifically stated that they are essential or when it is clearly limited to a specific number in principle. In addition, when multiple values are exemplified for a certain amount, it is also possible to adopt a value between those multiple values, except when it is specifically stated otherwise or when it is clearly impossible in principle.

The present invention also allows for the following modifications and modifications within an equivalent scope to each of the above embodiments. Note that each of the following modifications can be independently applied or not applied to the above embodiments. In other words, any combination of the following modifications can be applied to the above embodiments.

(Variation 1)
The image recognition device 1 may have both the functions of the first embodiment (i.e., image recognition using an attention map modified based on human knowledge) and the functions of the second embodiment (i.e., re-learning using an attention map modified based on human knowledge).

(Variation 2)
In the above embodiment, a classification result is given as an example of the recognition result output by the attention unit 12 and the recognition unit 14. However, the recognition result output by the attention unit 12 and the recognition unit 14 is not limited to a classification result and may be a result by regression. In other words, the image recognition performed by the neural network 10 may be classification or regression.

(Variation 3)
In the first embodiment, the neural network 10 has a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a recognition unit 14. However, the neural network for realizing image recognition using an attention map corrected based on human knowledge is not limited to such a configuration. In other words, if the neural network generates an attention map based on an input image and generates an image recognition result based on the input image and the attention map, the image recognition function can be improved by correcting the attention map.

(Variation 4)
In the second embodiment, the neural network 10 has a feature extraction unit 11, an attention unit 12, a synthesis unit 13, and a recognition unit 14. However, it may be modified based on human knowledge. However, the neural network for realizing re-learning using the attention map modified based on human knowledge is not limited to such a configuration. That is, if the neural network generates an attention map and an image recognition result based on an input image, the image recognition function can be improved by re-learning using the modified attention map. For example, a neural network such as CAM (Class Activation Mapping) described in Non-Patent Document 2 may be re-learned using an attention map modified based on human knowledge.

(Variation 5)
In the first embodiment, the processing unit 5 reflects the correction corresponding to the correction operation by the person on the attention map in a state where the input image 51 is transparently superimposed on the attention map 53 and displayed on the display device 3. However, this is not necessarily required. For example, the processing unit 5 may reflect the correction corresponding to the correction operation by the person on the attention map in a state where the input image 51 and the attention map 53 are displayed side by side without overlapping on the display device 3. Also, for example, the processing unit 5 may reflect the correction corresponding to the correction operation by the person on the attention map in a state where the attention map 53 is displayed on the display device 3 and the input image 51 is not displayed on the display device 3.

(Variation 6)
In the first and second embodiments, as a method for correcting the attention map, a method is shown in which only the values of some pixels in the attention map generated by the attention unit 12 are changed and the values of the remaining pixels are not changed. In other words, a method is shown in which a change is made to the attention map generated by the attention unit 12.

However, the method of correcting the attention map is not necessarily limited to this method. For example, a new attention map may be created from scratch in addition to the attention map output by the attention unit 12 when an image is input to the neural network 10. In this case, in the first embodiment, this new attention map is input to the synthesis unit 13, and in the second embodiment, this new attention map becomes the teacher attention map.

For example, a method for creating a new attention map is as follows. First, a person looks at the image input to the neural network 10 and determines the position range of the gaze area. The person may then operate a computer to create a new attention map that reflects the determined position range of the gaze area. This computer may be the image recognition device 1 or another device.

(Variation 7)
In the above second embodiment, the teacher attention map used for re-learning is only the teacher attention map corresponding to the training image erroneously recognized by the neural network 10 before re-learning. However, the teacher attention map used for re-learning may include the teacher attention map corresponding to the training image correctly recognized by the neural network 10 before re-learning.

Even in this case, if the number of teacher attention maps corresponding to misrecognized training images is greater than the number of teacher attention maps corresponding to correctly recognized training images, re-learning can be performed with high efficiency.

Alternatively, the number of teacher attention maps corresponding to misrecognized training images may be fewer than the number of teacher attention maps corresponding to correctly recognized training images.

(Variation 8)
In the above embodiment, when re-learning the neural network 10, the feature extraction unit 11 is not re-learned, and only the attention unit 12 and the recognition unit 14 are re-learned. Re-learning of the neural network 10 is not limited to this form. For example, the feature extraction unit 11 and the recognition unit 14 may not be re-learned, and only the attention unit 12 may be re-learned. Alternatively, for example, only the feature extraction unit 11 may be re-learned, and the attention unit 12 and the recognition unit 14 may not be re-learned. Alternatively, for example, the feature extraction unit 11 and the recognition unit 14 may be re-learned, and the attention unit 12 may not be re-learned.

It is also permissible for the feature extraction unit 11, attention unit 12, and recognition unit 14 to be retrained. In this case, the retraining of the neural network 10 is not fine tuning.

(Variation 9)
In the above embodiment, the re-learning is performed by using the error backpropagation method using three learning errors, Lmap, Latt, and Lper. However, it is not necessary to use all of Lmap, Latt, and Lper. For example, only Lmap may be used.

1...image recognition device, 2...operation device, 3...display device, 4...memory, 5...processing unit, 10...neural network, 11...feature extraction unit, 12...attention unit, 13...synthesis unit, 14...recognition unit, 51...input image, 52...feature map, 53...attention map, 54...synthesis map, 60...teacher label, 61...learning image

Claims (4)

an input unit (120) for inputting an image (51) to the neural network (10);
a map correction unit (140) that corrects the generated attention map in response to a correction operation by a person when the neural network generates a feature map (52) including features of the input image and an attention map (53) expressing a gaze area of the input image;
An image recognition device comprising : a composite map (54) in which the corrected attention map and the feature map are combined ; and an output unit (160) that outputs the generated recognition result when the neural network (10) generates a recognition result for the image based on the image. The neural network includes a feature extraction unit (11), an attention unit (12), a synthesis unit (13), and a recognition unit (14),
The feature extraction unit includes a plurality of convolution layers, and generates the feature map (52) by propagating information of the image through the plurality of convolution layers;
The attention unit generates the attention map based on the feature map;
The synthesis unit synthesizes the feature map and the modified attention map to generate the synthesis map ;
The image recognition device according to claim 1 , wherein the recognition unit generates the recognition result based on the composite map. The image recognition device according to claim 1 or 2, characterized in that the map correction unit reflects the correction corresponding to the correction operation in the attention map while the image is transparently superimposed on the attention map and displayed on a display device (3). A program used in an image recognition device that outputs a recognition result of an input image (51),
An input unit (120) for inputting the image into the neural network (10);
A program that causes the image recognition device to function as a map correction unit (140 ) that, when the neural network generates a feature map (52) including features of the input image and an attention map (53) expressing the gaze area of the input image, corrects the generated attention map in response to a human correction operation, and an output unit (160) that outputs the generated recognition result when the neural network (10) generates a recognition result of the image based on a composite map (54) in which the corrected attention map and the feature map are combined and the image.

Images