JP7384238B2 - Inference device, inference method, and inference program - Google Patents

Description

The present invention relates to an inference device, an inference method, and an inference program.

In recent years, neural networks have been used in various fields. For example, a technology (object detection technology) that detects an object in an image based on a trained neural network (for example, a trained neural network based on deep learning) is attracting attention. Various techniques are known as object detection techniques (see Non-Patent Documents 1 to 4). In such object detection technology, a class (object class) to which an object appearing in an image belongs is identified.

Xiongwei Wu, Doyen Sahoo, and Steven C.H. Hoi, "Recent Advances in Deep Learning for Object Detection", [online], 5 July 2020, Neurocomputing, [Retrieved November 2, 2020], Internet <https:// www.sciencedirect.com/science/article/abs/pii/S0925231220301430＞ Joseph Redmon, Ali Farhadi, "YOLO9000:Better, Faster, Stronger", [online], 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), [Retrieved November 2, 2020], Internet <https:/ /ieeexplore.ieee.org/document/8100173＞ Aditya Krishna Menon, 3 others, "Multilabel reductions: what is my loss optimizing?", [online], 2019 Neural Information Processing Systems (NIPS), [Retrieved November 2, 2020], Internet <https:/ /papers.nips.cc/paper/9245-multilabel-reductions-what-is-my-loss-optimizing＞ , Rafael Muller, Simon Kornblith and Geoffrey Hinton, "When Does Label Smoothing Help?", [online], 2019 Neural Information Processing Systems (NIPS), [Retrieved November 2, 2020], Internet <https:// papers.nips.cc/paper/8717-when-does-label-smoothing-help＞

However, it is desired to provide a technique that can improve the accuracy of identifying the class to which an object in an image belongs.

In order to solve the above problem, according to one aspect of the present invention, classes forming the first class group are provided based on information regarding the classes forming the first class group that can be identified by a first neural network. A hierarchical structure indicating the hierarchical relationship generated when it is determined that there is a hierarchical relationship , the value of the element corresponding to the correct answer class, and the correct answer belonging to the same layer as the layer to which the correct answer class belongs. The teacher data after conversion has been performed to bring the values of elements corresponding to classes other than the class closer together, and the output data output from the first neural network corresponding to the training data corresponding to the teacher data. A trained neural network obtained by a learning process based on the training process is obtained, output data output from the trained neural network for the entire class constituting the first class group in response to test data, and the hierarchy an inference unit that calculates the reliability of each class in the entire class based on the structure, and performs a process of identifying a class to which the test data belongs based on the reliability of each class in the entire class , An inference device is provided, wherein the inference unit performs the identification process depending on whether there is a class among the entire classes that has a higher reliability than a first threshold value corresponding to a layer to which the class belongs. Ru.

The reasoning section isIf there is a class whose reliability is higher than the first threshold, the class may be output as the identification result of the class to which the test data belongs.
Furthermore, when there is no class whose reliability is higher than the first threshold, and when the reliability of the upper class of the class belonging to the layer is higher than the second threshold, the inference unit determines whether the upper class may be output as the identification result of the class to which the test data belongs.

According to another aspect of the present invention, the classes forming the first class group have a hierarchical relationship with each other based on information regarding the classes forming the first class group that can be identified by the first neural network. A hierarchical structure showing the above-mentioned hierarchical relationship, which is generated when it is determined that obtained through a learning process based on the training data that has been transformed to bring the values of the corresponding elements closer to each other, and the output data output from the first neural network corresponding to the training data corresponding to the training data. based on the output data outputted from the trained neural network for the entire class constituting the first class group in response to the test data, and the hierarchical structure. , calculating the reliability of each class in the whole class, and performing a process of identifying the class to which the test data belongs based on the reliability of each class in the whole class, and in the whole class, An inference method is provided, which includes performing the above-mentioned identification process depending on whether there is a class whose reliability is higher than a first threshold value corresponding to a layer to which the class belongs.

Further, according to another aspect of the present invention, the computer is configured to identify classes that make up the first class group based on information about the classes that make up the first class group that can be identified by the first neural network. A hierarchical structure indicating the above-mentioned above-mentioned above-mentioned above-mentioned above-mentioned above-mentioned hierarchical structure generated when it is determined that there is a above-mentioned above-mentioned relation, the value of the element corresponding to the correct answer class, and the above-mentioned correct answer class belonging to the same layer as the above-mentioned correct answer class belongs. learning based on teacher data after conversion has been performed to bring values of elements corresponding to classes closer to each other, and output data output from the first neural network corresponding to training data corresponding to the teacher data. A trained neural network obtained by the processing is obtained, and output data output from the trained neural network for the entire class constituting the first class group in response to test data, and the hierarchical structure. an inference unit that calculates reliability of each class in the entire class based on the reliability of each class in the entire class, and performs processing to identify a class to which the test data belongs based on the reliability of each class in the entire class, the inference unit is an inference program that functions as an inference device that performs the identification process depending on whether or not there is a class among the entire classes that has a reliability higher than a first threshold value corresponding to a layer to which the class belongs. is provided.

As described above, according to the present invention, a technique is provided that makes it possible to improve the accuracy of identifying the class to which an object appearing in an image belongs.

FIG. 2 is a diagram for explaining an overview of object detection technology. FIG. 6 is a diagram for explaining an example where exclusive class identification is not suitable. FIG. 6 is a diagram for explaining an example where exclusive class identification is not suitable. 1 is a diagram showing an example of a functional configuration of a class identification system according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of a neural network input from a data section to a hierarchical structure generation section. FIG. 3 is a diagram illustrating an example of a neural network after conversion by a hierarchical structure generation unit. FIG. 3 is a diagram illustrating an example of a hierarchical structure generated by a hierarchical structure generation unit. FIG. 3 is a diagram for explaining an overview of neural network replacement and training data conversion. FIG. 7 is a diagram illustrating a comparison of the amount of correction of the neural network before and after replacement of the neural network. FIG. 6 is a diagram for explaining a case where converted teacher data is used for learning. FIG. 6 is a diagram for explaining a case where pre-conversion teacher data is used for learning. 1 is a diagram showing a hardware configuration of an information processing device as an example of a class identification system according to an embodiment of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configurations are designated by the same reference numerals and redundant explanation will be omitted.

Further, in this specification and the drawings, a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers after the same reference numeral. However, if there is no particular need to distinguish between a plurality of components having substantially the same functional configuration, only the same reference numerals are given. Further, similar components in different embodiments may be distinguished by attaching different alphabets after the same reference numerals. However, when it is not necessary to particularly distinguish between similar components of different embodiments, only the same reference numerals are given.

(0. Background)
First, the background of the embodiment of the present invention will be explained.

In recent years, neural networks have been used in various fields. For example, a technology (object detection technology) that detects an object in an image based on a trained neural network (for example, a trained neural network based on deep learning) is attracting attention. It is expected that various applications will be realized using such object detection technology. For example, based on the detection results of objects in images obtained by in-vehicle cameras, we can create applications that identify people in images, recognize license plates in images, and recognize situations in images. is expected.

Technology for detecting objects in images (object detection technology) includes a region estimation subtask of estimating one or more object candidate regions based on the image, and identifying the class of the object in each of the one or more object candidate regions. This can be realized by a class identification task (for example, see Non-Patent Document 1). Here, an overview of such object detection technology will be explained with reference to FIG.

FIG. 1 is a diagram for explaining an overview of object detection technology. Referring to FIG. 1, an image 50 of an object is shown. More specifically, a bus as an example of the object is shown in the object area 51A in the image 50. Furthermore, a person as an example of an object is shown in an object area 52A in the image 50. FIG. 1 shows a two-stage detector N1 and a one-stage detector N2 as examples of object detectors that detect objects from such an image 50.

The two-stage detector N1 has a first stage (feature extraction task N11) of extracting a feature map F1 based on the image 50, and a second stage (feature extraction task N11) of estimating object candidate regions h1 to h3 based on the feature map F1. The position and feature amount of the object candidate regions h1 to h3 are detected by the region estimation task N12). On the other hand, the one-stage detector N2 detects the positions of predetermined object candidate regions (for example, the positions of object candidate regions comprehensively determined in advance over the entire image ("i1, j1", "i1, j2", " i2, j1", "i2, j2", . . . ) and the image 50, the feature amount for each object candidate region is detected in one step (feature extraction & region estimation task N13).

In addition, in each of the two-stage detector N1 and the one-stage detector N2, the class identification task N21 identifies object candidate regions that do not contain objects as background regions based on the feature amount of each object candidate region, and The object candidate region that includes is identified as an object region. If an object is included in the object candidate area, the class identification task N21 identifies the class (object class) to which the object included in the object candidate area belongs.

In the example shown in FIG. 1, since the object appearing in the object candidate area 51B (corresponding to the object area 51A in which the bus is shown) is identified as a truck, incorrect class identification may occur in the object candidate area 51B. It is being done. On the other hand, in the example shown in FIG. 1, since the object appearing in the object candidate area 52B (corresponding to the object area 52A in which a person is shown) is identified as a person, correct class identification is performed in the object candidate area 52B. is being carried out.

In the class identification task N21, learning is performed to exclusively identify the class (object class) to which the object included in the object candidate region belongs, using cross-entropy error, based on the classification results for each object candidate region. is common. However, there may be cases where exclusive class identification is not suitable. Here, an example in which exclusive class identification is not suitable will be described with reference to FIGS. 2 and 3.

FIGS. 2 and 3 are diagrams for explaining an example where exclusive class identification is not suitable. Referring to FIGS. 2 and 3, images 21A, 22A, 23A, and 24A are shown that are associated with teacher data in which the correct class is "Car". Also shown are images 21B, 22B, 23B, and 24B associated with teacher data in which the correct answer class is "Truck." Also shown are images 21C, 22C, 23C, and 24C associated with teacher data in which the correct answer class is "Bus."

Here, "Car" is shown in the object candidate area (within the thick rectangular frame) of images 21A and 22A, which are respectively associated with teacher data in which the correct answer class is "Car". On the other hand, "Truck" is shown in the object candidate area of the image 23A, which is associated with the teacher data in which the correct answer class is "Car". That is, there is an error in the labeling (annotation) for the image 23A. Further, a "Van (a vehicle with a roof for transporting cargo)" is shown in the object candidate region of the image 24A, which is associated with the teacher data in which the correct answer class is "Car." It is ambiguous which class "Van" belongs to. That is, the class to which the object appearing in the object candidate region of image 24A belongs is ambiguous.

Similarly, "Truck" is shown in the object candidate regions of images 21B and 22B, which are respectively associated with teacher data in which the correct answer class is "Truck." On the other hand, "Car" is shown in the object candidate region of the image 23B, which is associated with the teacher data in which the correct answer class is "Truck". That is, there is an error in the labeling (annotation) for the image 23B. Furthermore, "Van" is shown in the image 24B associated with the teacher data in which the correct answer class is "Truck". That is, the class to which the object appearing in the object candidate region of image 24B belongs is ambiguous.

Similarly, "Bus" appears in the object candidate regions of images 21C and 22C, which are respectively associated with teacher data in which the correct answer class is "Bus." On the other hand, "Car" is shown in the object candidate area of the image 23C, which is associated with the teacher data in which the correct answer class is "Bus". That is, there is an error in the labeling (annotation) for the image 23C. Furthermore, "Van" is shown in the object candidate area of the image 24C, which is associated with the teacher data in which the correct answer class is "Bus". That is, the class to which the object appearing in the object candidate region of image 24C belongs is ambiguous.

As described above, an example of a case where exclusive class identification is not suitable is a case where an error exists in the annotation (labeling of an image). Another example of a case where exclusive class identification is not suitable is a case where the class to which an object in an image belongs is ambiguous. Even though there are cases where exclusive class identification is not suitable, if exclusive class identification is learned, there is a possibility that errors will occur in class identification.

Therefore, in order to reduce the possibility of class identification errors, abstract classes are generated based on information about concrete classes that can be identified, and whether concrete classes or abstract classes are A technique for identifying whether an object belongs to a class (that is, a technique for performing hierarchical class identification) has been proposed (for example, see Non-Patent Document 2). In the technique described in Non-Patent Document 2, hierarchical class identification makes it possible to identify an abstract class as a class to which an object appearing in an image belongs.

However, with the technology described in Non-Patent Document 2, over-fitting occurs when exclusive class identification is not suitable (for example, when there is an error in the annotation or when the class to which an object in an image belongs is ambiguous). is performed, and the accuracy of class identification may deteriorate. One of the causes of overfitting is that exclusive discrimination learning is carried out in the learning stage. Another cause of overfitting is that the teacher data used for learning is exclusive.

Therefore, in the following description, a technique is proposed in which a technique for learning non-exclusive discrimination (for example, the technique described in Non-Patent Document 3) is applied to hierarchical class discrimination. According to this configuration, the possibility that overfitting will be performed when exclusive class identification is not suitable is reduced, so the possibility that the accuracy of class identification will deteriorate can be reduced. In the embodiment of the present invention, a technique called "Sigmoid Activation" is used as an example of a technique for performing non-exclusive discrimination learning.

Furthermore, in the following description, a technique is proposed in which a technique that uses non-exclusive training data for learning (for example, the technique described in Non-Patent Document 4) is applied to hierarchical class identification. According to this configuration, the possibility that overfitting will be performed when exclusive class identification is not suitable is reduced, so the possibility that the accuracy of class identification will deteriorate can be reduced. In an embodiment of the present invention, a technique called "label smoothing" is used as an example of a technique that uses non-exclusive training data for learning.

In the following description, an example in which both sigmoid activation and label smoothing are applied to hierarchical class identification will be mainly described. However, both sigmoid activation and label smoothing do not necessarily have to be applied to hierarchical class identification. For example, only one of sigmoid activation and label smoothing may be applied to hierarchical class identification.

The background of the embodiments of the present invention has been described above.

(1. Details of embodiment)
Next, details of embodiments of the present invention will be described.

(1-1. Explanation of configuration)
First, a configuration example of a class identification system according to an embodiment of the present invention will be described. FIG. 4 is a diagram showing an example of the functional configuration of the class identification system according to the embodiment of the present invention. As shown in FIG. 4, the class identification system 1 according to the embodiment of the present invention includes a data section 11, a hierarchical structure generation section 12, a learning section 13, and an inference section 14. The class identification system 1 may be realized by the same computer, or may be realized by being distributed among multiple computers. Note that the hierarchical structure generation unit 12 and the learning unit 13 may constitute a learning device. Further, the inference unit 14 may constitute an inference device.

The hierarchical structure generation unit 12, the learning unit 13, and the inference unit 14 include arithmetic units such as a CPU (Central Processing Unit), and programs (a learning program and an inference program) stored in a memory (not shown) are stored in a RAM (Random) by the CPU. The function can be realized by being deployed and executed in Access Memory. At this time, a computer-readable recording medium on which the program is recorded may also be provided. Alternatively, the hierarchical structure generation section 12, the learning section 13, and the inference section 14 may be configured by dedicated hardware, or may be configured by a combination of multiple pieces of hardware.

(Data part 11)
The data section 11 may be configured by memory. The data section 11 stores a neural network (first neural network). Further, the data unit 11 stores data (learning data set) used for training the neural network. Further, the data unit 11 stores information regarding classes that constitute a class group (first class group) that can be identified by a neural network. Furthermore, the data section 11 stores test images as examples of test data.

In the embodiments of the present invention, it is mainly assumed that the neural network is a convolutional neural network. However, the configuration of the neural network is not particularly limited. Note that the trained neural network can become the two-stage detector N1 (FIG. 1). Alternatively, the trained neural network can be a one-stage detector N2 (FIG. 1).

The learning data set includes teacher data and training images as examples of training data. The teacher data includes information regarding the correct class to which the subject in the training image belongs. Here, information regarding the correct class is indicated by a vector (so-called one-hot expression) in which "1" is set as an element corresponding to the correct class and "0" is set as an element corresponding to a class other than the correct answer class. . In addition, the teacher data may include information regarding the object candidate region in which the subject appears in the training image (for example, the position and size of the object candidate region, etc.).

The information regarding the classes constituting the class group that can be identified by the neural network may be, for example, a class name. However, the information regarding the class is not limited to the class name, and may be any information that allows the class to be identified.

The data section 11 outputs information regarding the class and the neural network to the hierarchical structure generation section 12 in the learning stage. Further, the data unit 11 outputs a learning data set to the learning unit 13 in the learning stage. On the other hand, the data section 11 outputs the test image to the inference section 14 in the inference stage.

(Hierarchical structure generation unit 12)
The hierarchical structure generation unit 12 receives input from the data unit 11 of information regarding classes (hereinafter also referred to as “existing classes”) that constitute a class group (hereinafter also referred to as “existing classes”) that can be identified by a neural network. Then, based on information about existing classes, a hierarchical structure indicating the hierarchical relationship between existing classes is generated. Here, the hierarchical relationship may be a conceptual hierarchical relationship. Furthermore, the timing at which the hierarchical structure is generated is mainly assumed to be during the learning stage. However, the hierarchical structure may be generated in advance prior to the learning phase.

The hierarchical structure generation unit 12 may add a new class (hereinafter also referred to as "additional class") to the existing class group based on information regarding existing classes. The number of additional classes is not limited and may be one or more. At this time, the hierarchical structure generation unit 12 generates a hierarchical structure indicating the hierarchical relationship between the classes that constitute the class group (second class group) after the addition of the new class. Then, the hierarchical structure generation unit 12 converts the neural network input from the data unit 11 (into a second neural network) based on the addition of the new class.

An example of generation of a hierarchical structure and an example of conversion of a neural network will be described with reference to FIGS. 5 to 7. FIG. 5 is a diagram showing an example of a neural network input from the data section 11 to the hierarchical structure generation section 12. FIG. 6 is a diagram showing an example of a neural network after conversion by the hierarchical structure generation unit 12. FIG. 7 is a diagram illustrating an example of a hierarchical structure generated by the hierarchical structure generation unit 12.

Referring to FIG. 5, an example of a neural network input from the data section 11 to the hierarchical structure generation section 12 is shown. However, the example shown in FIG. 5 shows the class identification task N21 and the output layer L0, which correspond to a part of the neural network. An object candidate region h2 is shown as an example of input to the class identification task N21. Each node of the output layer L0 corresponds to a plurality of classes that can be identified by the neural network. Each node of the output layer L0 is a node corresponding to an existing class (in FIG. 5, the node is not filled in).

The output layer L0 forms a full connection with the class identification task N21. In FIG. 5, a part of the connection formed between the output layer L0 and the class identification task N21 is shown by a solid line segment. That is, in the example shown in FIG. 5, not all the connections formed between the output layer L0 and the class identification task N21 are shown, but the connections between the output layer L0 and the class identification task N21 are not shown. Some of the bonds formed in are omitted.

Referring to FIG. 5, examples of existing classes include the "Background" class, "Car" class, "Truck" class, "Bus" class, "Pedestrian" class, "Rider" class, and "Person" class. ing. These existing classes correspond to a group of existing classes that can be identified by a neural network. The hierarchical structure generation unit 12 identifies the hierarchical relationship between the classes that make up the existing class group, based on information about the existing class group.

As an example, it is assumed that the data section 11 includes dictionary data in which a combination of a name of a lower class and a name of its higher class is registered in advance. In such a case, the hierarchical structure generation unit 12 determines that if a combination of names that matches or is similar to the combination of names of two classes constituting the existing class group is registered in the dictionary data, the two classes are in a hierarchical relationship. It may also be specified. For example, referring to FIG. 7, two existing classes, the "Person" class and the "Pedestrian" class, are specified to be in a hierarchical relationship. Note that in FIG. 7, existing classes are shown as unfilled classes.

Further, the hierarchical structure generation unit 12 can generate a hierarchical structure in which a new class (additional class) is added to the existing class group based on information regarding existing classes. For example, it is assumed that one name that matches or is similar to the name of a class constituting an existing class group is registered in the dictionary data. In such a case, the hierarchical structure generation unit 12 sets the other name forming a combination with the one name as a new class name, and based on the hierarchical relationship between the one name and the other name, the hierarchical structure generation unit 12 A hierarchical structure with new classes added may also be generated. For example, referring to FIG. 7, the "Vehicle" class has been added as an upper class of the existing class "Car" class. Note that in FIG. 7, additional classes are shown as shaded classes.

If a combination of names that match or are similar to the combination of names of two additional classes is registered in the dictionary data, the hierarchical structure generation unit 12 may specify that the two additional classes are in a hierarchical relationship. For example, referring to FIG. 7, two additional classes, the "Foreground" class and the "Vehicle" class, are identified as having a hierarchical relationship. Note that the hierarchical structure 60 (FIG. 7) generated in this way is formed by layers L1 to L5. Each of the layers L1 to L5 is a collection of classes in which the same class is a higher class.

When a new class is added, the hierarchical structure generation unit 12 converts the neural network input from the data unit 11 based on the addition of the new class. In FIG. 6, there are additional classes such as a "Foreground" class, a "Vehicle" class, a "Motorist" class, and a "Cyclist" class. These additional classes and existing classes correspond to a group of classes that can be identified by the neural network after conversion.

The hierarchical structure generation unit 12 adds nodes corresponding to these additional classes to the output layer as nodes corresponding to new classes. In FIG. 6, nodes corresponding to additional classes in the output layer are shown as filled nodes. Note that, in the same way as between the existing class and the class identification task N21, a connection (dashed line segment in FIG. 6) is also formed between the additional class and the class identification task N21.

The hierarchical structure generation unit 12 outputs the converted neural network (second neural network) to the learning unit 13. Note that when the hierarchical structure generation unit 12 does not convert the neural network (first neural network) input from the data unit 11 (that is, when it does not add a new class), The neural network input from the data section 11 may be output to the learning section 13 as is. Further, the hierarchical structure generating section 12 outputs the generated hierarchical structure 60 to the learning section 13.

(Learning part 13)
Prior to performing the learning process (that is, updating the weight parameters of the neural network based on the learning data set), the learning unit 13 performs a replacement for a technique (sigmoid activation) that performs learning for non-exclusive discrimination. This is applied to the neural network input from the hierarchical structure generation unit 12. Furthermore, before performing the learning process, the learning unit 13 performs a transformation on the teaching data input from the data unit 11 for a technique (label smoothing) that uses non-exclusive teaching data for learning.

FIG. 8 is a diagram for explaining an overview of neural network replacement and training data conversion. Here, we will mainly focus on the layer L3 of the hierarchical structure 60 (FIG. 7). Also, among the nodes of the output layer of the neural network input from the hierarchical structure generation unit 12, the nodes corresponding to each class ("Car" class, "Truck" class, and "Bus" class) constituting layer L3 are mainly Focus on.

(Neural network replacement)
Referring to FIG. 8, a training image 40 is shown as an example of training data included in the learning data set. The class to which the object shown in the training image 40 belongs is the "Truck" class. Assuming that the training image 40 is input to the neural network input from the hierarchical structure generation unit 12, the result calculated at the node of the output layer of the neural network corresponding to the training image 40 is Logits. It becomes as shown in C10. However, the logit C10 is a calculation result based on the input to the output layer node, the weight, and the bias, and is the calculation result before the activation function is applied.

Here, a softmax function C11 is introduced as an activation function in the output layer of the neural network. However, when the softmax function C11 is applied to the logit C10, the output from the neural network is as shown in output data C12, as an example. That is, between the output value from the node corresponding to the "Car" class with the largest logit and the output value from the nodes corresponding to the "Truck" and "Bus" classes, respectively, which are the classes with the second largest logit. The difference becomes large. Increasing the difference in output values in this manner may lead to exclusive identification learning being performed.

Therefore, the learning unit 13 replaces the softmax function C11 included in the neural network with the sigmoid function E11. More specifically, a softmax function C11 is introduced as an activation function to a node corresponding to an existing class in the output layer of the neural network. Therefore, the learning unit 13 converts the softmax function C11 introduced into the node corresponding to the existing class in the output layer of the neural network into a sigmoid function E11.

On the other hand, activation functions have not yet been introduced to nodes corresponding to additional classes in the output layer of the neural network. Therefore, the learning unit 13 introduces the sigmoid function E11 as the activation function of the node corresponding to the additional class in the output layer of the neural network.

In the following, the neural network after the softmax function C11 introduced to the node corresponding to the existing class is converted to the sigmoid function E11 and the sigmoid function E11 is introduced as the activation function of the node corresponding to the additional class is described as " It is also called ``neural network after replacement''. However, if a new class is not added by the hierarchical structure generation unit 12, the neural network after the softmax function C11 introduced in the node corresponding to the existing class is converted to the sigmoid function E11 is It is also called "neural network after replacement."

When the sigmoid function E11 is applied to the logit C10, the output from the neural network is as shown in output data E12, as an example. That is, between the output value from the node corresponding to the "Car" class with the largest logit and the output value from the nodes corresponding to the "Truck" and "Bus" classes, respectively, which are the classes with the second largest logit. The difference becomes smaller. A decrease in the difference in output values in this manner may lead to learning to identify non-exclusively.

FIG. 9 is a diagram showing a comparison of the amount of correction of the neural network before and after replacement of the neural network. Referring to FIG. 9, a training image 40A is shown as an example of training data A corresponding to teacher data in which the correct answer class is "Truck". Furthermore, a training image 40B is shown as an example of training data B corresponding to teacher data in which the correct answer class is "Car".

The modification amount C13 is the modification amount of the neural network before replacement by learning using the training data A. On the other hand, the modification amount E13 is the modification amount of the neural network after replacement by learning using the training data A. Further, the modification amount C14 is the modification amount of the neural network before replacement by learning using the training data B. On the other hand, the modification amount E14 is the modification amount of the neural network after replacement by learning using the training data B.

Note that "+ve" shown in FIG. 9 is an abbreviation for "Positive" and indicates a modification of the neural network in a direction that makes it easier to identify the class. On the other hand, "-ve" shown in FIG. 9 is an abbreviation for "Negative" and indicates a modification of the neural network in a direction that makes it difficult to identify the class.

Referring to the correction amounts C13 and C14, in the neural network before replacement, the neural network is corrected in a direction that makes it easier to identify the correct class, and also makes it difficult to identify classes other than the correct class. It is understood that the neural network will be modified in the direction of That is, it is understood that exclusive class identification learning is performed in the neural network before replacement.

On the other hand, referring to the correction amounts E13 and E14, it can be seen that in the neural network after replacement, only corrections are made to the neural network in a direction that makes it easier to identify the correct class. That is, it is understood that non-exclusive class identification learning is performed in the neural network after replacement.

(Conversion of training data)
Returning to FIG. 8, the explanation will be continued. Referring to FIG. 8, teacher data C20 is shown as an example of teacher data. In the teacher data C20, "1" is set as the value of the element corresponding to the "Truck" class which is the correct answer class, and the value of the element corresponding to each of the "Car" class and "Bus" class which are classes other than the correct answer class. It is indicated by a vector (one-hot expression) in which "0" is set as "0".

The learning unit 13 determines that the element value "1" corresponding to the correct class "Truck" is a class other than the correct class that belongs to the same layer L3 (FIG. 7) as the layer to which the correct class "Truck" belongs. Conversion is performed to bring the values of the elements corresponding to the "Car" and "Bus" classes closer to each other. This allows non-exclusive training data to be used for learning.

For example, the learning unit 13 may subtract a predetermined value λ from the value "1" of the element corresponding to the correct answer class. At this time, the learning unit 13 divides the result "λ/N" obtained by dividing by the number N of classes other than the correct class belonging to the same layer as the layer to which the correct answer class belongs, and calculates the result "λ/N" corresponding to the class other than the correct answer class. It may be added to each element value "0".

In the example shown in FIG. 8, it is assumed that λ=0.4 and the number of classes other than the correct class N=2. Therefore, the value of the element corresponding to the correct class "Truck" class is "1-λ" = "1-0.4" = "0.6", and the value of the element corresponding to the "Car" class, which is a class other than the correct class, and The value of the element corresponding to each "Bus" class is "0+λ/N" = "0+0.4/2" = "0.2".

FIG. 10 is a diagram for explaining a case where converted teacher data is used for learning. On the other hand, FIG. 11 is a diagram for explaining a case where pre-conversion teacher data is used for learning.

Referring to FIG. 10, a feature space 71 shows changes in the feature amount of the training data when the teacher data before conversion is used for learning. Referring to FIG. 11, the feature space 72 shows changes in the feature amount of the training data when the transformed teacher data is used for learning.

In the feature space 71 and the feature space 72, the features of the training data whose correct answer class is the "Car" class are indicated by dark dots, and the features of the training data whose correct answer class is the "Truck" class are indicated by medium dots. The features of the training data whose correct answer class is the "Bus" class are indicated by light dots.

When the pre-conversion teacher data is used for learning, learning based on the training data of each correct answer class is performed so that the distances D21 to D23 between the feature quantities in the feature space 72 (FIG. 11) are not too far apart. be exposed. Therefore, it is understood that class identification to "Vehicle", which is a higher class of the "Car" class, "Truck" class, and "Bus" class, is difficult to be performed, and the accuracy of class identification is not improved.

On the other hand, when the converted training data is used for learning, learning based on the training data of each correct answer class is performed so that the distances D11 to D13 between the feature quantities in the feature space 71 (FIG. 10) become larger. It will be done. Therefore, it is understood that class identification to "Vehicle", which is a higher class of the "Car" class, "Truck" class, and "Bus" class, is easily performed, and the accuracy of class identification is improved.

(Learning process)
Returning to FIG. 8, the explanation will be continued. The learning unit 13 performs a process based on the converted teacher data, the output data output from the replaced neural network corresponding to the training data, and the hierarchical structure 60 (FIG. 7) input from the hierarchical structure generating unit 12. Then, calculate the error for each layer. Here, it is assumed that the learning unit 13 uses a cross-entropy error C30 as an example of the error. However, the error used by the learning unit 13 is not limited to the cross-entropy error. For example, when a is the layer number, the entropy error Loss _La of the layer La can be expressed as in the following equation (1).

Here, n indicates the number of classes configuring the layer, and k indicates the number of the class configuring the layer. t _k is the value of the element corresponding to class k in the converted teacher data. P(x _k ) is the ratio of the output value of the node corresponding to class k to the sum of the output values of the nodes corresponding to the n classes composing that layer among the nodes of the output layer of the neural network after replacement. (probability). That is, P(x _k ) corresponds to the reliability of class k in layer units.

The learning unit 13 performs learning processing of the neural network after replacement using the sum of cross-entropy errors for each layer as a loss function. As an example, the total cross-entropy error per layer may be a weighted sum of cross-entropy errors per layer. For example, when the weight of the layer La is α _a , the weighted sum Loss of cross-entropy errors for each layer can be expressed as in equation (2) below.

However, m indicates the number of layers. For example, the learning unit 13 performs a learning process for the neural network after replacement using an error backpropagation method based on the loss function Loss. As a result, the weights of the replaced neural network are adjusted to adapt to the learning data set.

If the learning unit 13 determines that the loss function Loss has converged, it ends the learning process of the neural network after replacement. For example, when the loss function Loss or its change becomes smaller than a threshold value, it may be determined that the loss function Loss has converged. The trained neural network is output to the inference section 14.

Note that the learning unit 13 mainly assumes a case where the converted teacher data and the replaced neural network are used for learning. However, it is also assumed that only one of training data transformation and neural network replacement is performed (i.e., it is also assumed that only one of sigmoid activation and label smoothing is applied to hierarchical class identification). .

For example, when the learning unit 13 only converts the teacher data, the learning unit 13 may use the teacher data itself input from the data unit 11 for learning. Further, when the learning section 13 only performs neural network replacement, the learning section 13 may use the neural network itself input from the hierarchical structure generating section 12 for learning.

(Inference part 14)
The inference unit 14 acquires the trained neural network input from the learning unit 13, and outputs output data corresponding to the test data from the trained neural network and the hierarchy generated by the hierarchical structure generation unit 12. Based on the structure 60 (FIG. 7), the class to which the test data belongs is identified.

More specifically, the inference unit 14 calculates, based on the output data corresponding to the test data and the hierarchical structure 60, the sum of the output values of the nodes corresponding to all the classes that constitute the class group that can be identified by the trained neural network. The ratio (probability) of the output value of the node corresponding to each class is calculated. Such probability corresponds to the reliability of the class relative to the class as a whole.

The reliability of a class for the entire class may be calculated in any manner. As an example, the inference unit 14 may calculate the class reliability for the entire class based on the class reliability for each layer. For example, the inference unit 14 calculates the reliability of class k with respect to the entire class by the product of the reliability of each layer of each class passed through until reaching class k in the hierarchical structure 60 starting from the highest class. You may.

As an example, the inference unit 14 calculates the reliability of the "Car" class for the entire class by calculating the reliability of the "Foreground" class in each layer, the reliability of the "Vehicle" class in each layer, and the reliability of the "Vehicle" class in each layer. It can be calculated by multiplying the reliability of the "Car" class.

Then, the inference unit 14 performs class identification processing based on the reliability of each class among the entire classes. Class identification based on the reliability of each class may be performed in any manner. As an example, the inference unit 14 may identify classes depending on whether there is a class whose reliability is higher than a first threshold value corresponding to the layer. The inference unit 14 outputs the class identification results.

For example, if there is a class whose reliability is higher than the first threshold value corresponding to the layer, the inference unit 14 may use that class as the identification result. As an example, if the reliability of the "Car" class belonging to the layer L3 is higher than the first threshold, the inference unit 14 may determine the "Car" class as the identification result.

On the other hand, if there is no class in the layer whose reliability is higher than the first threshold corresponding to the layer, and the reliability of the upper class of the class belonging to the layer is higher than the second threshold, If it is higher than , the upper class may be used as the identification result. As an example, if the "Car" class, "Truck" class, and "Bus" class belonging to layer L3 are all below the first threshold, and "Vehicle" which is a higher class of these classes, If the reliability of the ``Vehicle'' class is higher than the second threshold, the ``Vehicle'' class, which is a higher class, may be used as the identification result.

(1-3. Explanation of effects)
The above describes a technique in which a technique for learning non-exclusive discrimination is applied to hierarchical class discrimination. In particular, in the above description, the case where a technique called "sigmoid activation" is used is mainly explained as an example of a technique for performing non-exclusive discrimination learning. According to this configuration, the possibility of overfitting being performed when exclusive class identification is not suitable is reduced, so it is possible to improve the accuracy of class identification.

Furthermore, we have also explained a technique that applies the technique of using non-exclusive training data for learning to hierarchical class identification. In particular, above, the case where a technique called "label smoothing" is used is mainly described as an example of a technique that uses non-exclusive training data for learning. According to this configuration, the possibility of overfitting being performed when exclusive class identification is not suitable is reduced, so it is possible to improve the accuracy of class identification.

More specifically, examples of cases where exclusive class identification is not suitable include cases where an error exists in the annotation and cases where the class to which an object in an image belongs is ambiguous, as described above. In cases where such exclusive class identification is not suitable, embodiments of the present invention provide for abstract class identification rather than concrete classes (e.g., "Car" class, "Truck" class, and "Bus" class, etc.). classes (for example, “Vehicle” class, etc.) are more likely to be identified as identification results.

That is, the class identification system 1 according to the embodiment of the present invention is a system that can perform robust class identification even when there is an error in the annotation and when the class to which an object in an image belongs is ambiguous. It can be said that

(2. Various modifications)
Although preferred embodiments of the present invention have been described above in detail with reference to the accompanying drawings, the present invention is not limited to such examples. It is clear that a person with ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea stated in the claims. It is understood that these also naturally fall within the technical scope of the present invention.

In the above example, the class identification system 1 identifies the vehicle class (for example, "Car" class, "Truck" class, "Bus" class, "Vehicle" class, etc.) as the class to which the object in the image belongs. I mainly explained examples. However, the class identification system 1 is also capable of identifying a class other than a vehicle as a class to which an object appearing in an image belongs. For example, the class identification system 1 can also identify other object classes, such as a person class (for example, a "Pedestrian" class, a "Rider" class, and a "Motorist" class), as a class to which an object in an image belongs. It is.

As mentioned above, the trained neural network can become a two-stage detector N1 (FIG. 1). Alternatively, the trained neural network can be a one-stage detector N2 (FIG. 1). That is, in embodiments of the present invention, a general detector such as a two-stage detector N1 (FIG. 1) or a one-stage detector N2 (FIG. 1) may be used as the neural network architecture.

(3. Hardware configuration example)
Next, an example of the hardware configuration of the class identification system 1 according to the embodiment of the present invention will be described. An example of the hardware configuration of the information processing device 900 will be described below as an example of the hardware configuration of the class identification system 1 according to the embodiment of the present invention. Note that the example hardware configuration of the information processing device 900 described below is only an example of the hardware configuration of the class identification system 1. Therefore, in the hardware configuration of the class identification system 1, unnecessary configurations may be deleted from the hardware configuration of the information processing device 900 described below, or new configurations may be added.

FIG. 12 is a diagram showing a hardware configuration of an information processing device 900 as an example of the class identification system 1 according to the embodiment of the present invention. The information processing device 900 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 904, a bridge 905, an external bus 906, and an interface. With 907 , an input device 908, an output device 909, a storage device 910, and a communication device 911.

The CPU 901 functions as an arithmetic processing device and a control device, and controls overall operations within the information processing device 900 according to various programs. Further, the CPU 901 may be a microprocessor. The ROM 902 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 903 temporarily stores programs used in the execution of the CPU 901 and parameters that change as appropriate during the execution. These are interconnected by a host bus 904 composed of a CPU bus and the like.

The host bus 904 is connected via a bridge 905 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus. Note that the host bus 904, bridge 905, and external bus 906 do not necessarily need to be configured separately, and these functions may be implemented in one bus.

The input device 908 includes input means for the user to input information, such as a mouse, keyboard, touch panel, button, microphone, switch, and lever, and an input control circuit that generates an input signal based on the user's input and outputs it to the CPU 901. It is composed of etc. By operating the input device 908, a user operating the information processing device 900 can input various data to the information processing device 900 and instruct processing operations.

The output device 909 includes, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display (LCD) device, an OLED (Organic Light Emitting Diode) device, a display device such as a lamp, and an audio output device such as a speaker.

The storage device 910 is a device for storing data. The storage device 910 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, a deletion device that deletes data recorded on the storage medium, and the like. The storage device 910 is configured with, for example, an HDD (Hard Disk Drive). This storage device 910 drives a hard disk and stores programs executed by the CPU 901 and various data.

The communication device 911 is, for example, a communication interface configured with a communication device for connecting to a network. Further, the communication device 911 may support either wireless communication or wired communication.

The example hardware configuration of the class identification system 1 according to the embodiment of the present invention has been described above.

1 Class Identification System 11 Data Section 12 Hierarchical Structure Generation Section 13 Learning Section 14 Inference Section 60 Hierarchical Structure

Claims (5)

The previous class generated when it is determined that the classes constituting the first class group have a hierarchical relationship based on information regarding the classes constituting the first class group that can be identified by the first neural network. A transformation is performed to bring the values of elements corresponding to the correct class closer to the values of elements corresponding to classes other than the correct class that belong to the same layer as the layer to which the correct class belongs. a trained neural network obtained by a learning process based on the training data corresponding to the training data corresponding to the training data from the first neural network and the training data corresponding to the training data corresponding to the training data, and Calculating the reliability of each class in the entire class based on the output data output from the trained neural network for the entire class constituting the first class group in response to the test data and the hierarchical structure. and an inference unit that performs a process of identifying a class to which the test data belongs based on the reliability of each class among the entire classes ,
The inference unit performs the identification process depending on whether there is a class among the entire classes that has a higher reliability than a first threshold value corresponding to a layer to which the class belongs.
Reasoning device. The inference unit outputs the class as an identification result of the class to which the test data belongs, if there is a class whose reliability is higher than the first threshold.
The inference device according to claim 1. The inference unit selects the upper class from the above when there is no class whose reliability is higher than the first threshold, and when the reliability of the upper class of the class belonging to the layer is higher than the second threshold. Output as the identification result of the class to which the test data belongs,
The inference device according to claim 1. The previous class generated when it is determined that the classes constituting the first class group have a hierarchical relationship based on information regarding the classes constituting the first class group that can be identified by the first neural network. A transformation is performed to bring the values of elements corresponding to the correct class closer to the values of elements corresponding to classes other than the correct class that belong to the same layer as the layer to which the correct class belongs. a trained neural network obtained by a learning process based on the training data corresponding to the training data corresponding to the training data from the first neural network and the training data corresponding to the training data corresponding to the training data, and Calculating the reliability of each class in the entire class based on the output data output from the trained neural network for the entire class constituting the first class group in response to the test data and the hierarchical structure. and performing a process of identifying a class to which the test data belongs based on the reliability of each class among the entire classes ,
performing the identification process depending on whether there is a class among the entire classes that has a reliability higher than a first threshold value corresponding to a layer to which the class belongs;
Reasoning method. computer,
The previous class generated when it is determined that the classes constituting the first class group have a hierarchical relationship based on information regarding the classes constituting the first class group that can be identified by the first neural network. A transformation is performed to bring the values of elements corresponding to the correct class closer to the values of elements corresponding to classes other than the correct class that belong to the same layer as the layer to which the correct class belongs. a trained neural network obtained by a learning process based on the training data corresponding to the training data corresponding to the training data from the first neural network and the training data corresponding to the training data corresponding to the training data, and Calculating the reliability of each class in the entire class based on the output data output from the trained neural network for the entire class constituting the first class group in response to the test data and the hierarchical structure. and an inference unit that performs a process of identifying a class to which the test data belongs based on the reliability of each class among the entire classes ,
The inference unit performs the identification process depending on whether there is a class among the entire classes that has a higher reliability than a first threshold value corresponding to a layer to which the class belongs.
An inference program that functions as an inference device.

Images