JP7623868B2 - Data processing device and method - Google Patents

Description

The present invention relates to a data processing device and method.

To build an inference model that performs image recognition (such as object detection or object classification) using supervised learning, which is a type of machine learning, a large number of images (thousands to millions of images, or even more) with annotation information are generally required. The annotation information is created, for example, by manual annotation by humans or automatic annotation using a trained model. Supervised learning can be performed using a dataset that includes image data and annotation information for a large number of images.

Special table 2017-515189 publication

However, in both manual and automatic annotation, objects in an image may be annotated with incorrect class information (labels). Manual (human eye) checking of each image can identify incorrect class information (labels), but such checking incurs huge costs (time and human effort). If the device could automatically extract data with erroneous class information, it would be beneficial as it would reduce the cost of searching for defects in a dataset.

The present invention aims to provide a data processing device and method that contributes to identifying errors in class information.

The data processing device according to the present invention is configured (first configuration) to include an image classification dataset acquisition unit that acquires an image classification dataset including a plurality of object images each including image data of an object and class information that identifies the class of the object in each object image; a feature derivation unit that derives features of each object image in the image classification dataset; a clustering processing unit that performs clustering of the m object images (m is an integer of 3 or more) based on the features of the m object images included in the plurality of object images and the class information for the m object images; an index derivation unit that derives, for each cluster obtained by the clustering, the center of gravity of a plurality of feature points belonging to the cluster and the distance between the center of gravity and each feature, and derives the average and variance of the distances derived for the plurality of feature amounts; and an error identification unit that identifies, based on the derivation results of the average and the variance, a cluster to which an object image having an error in the class information belongs as an error cluster.

In the data processing device according to the first configuration, the error identification unit may be configured to identify, as the error cluster, a cluster whose derived mean is equal to or greater than a predetermined first threshold and whose derived variance is equal to or greater than a predetermined second threshold (second configuration).

In the data processing device according to the first or second configuration, the clustering processing unit may classify the m object images into k clusters in the clustering, where k is equal to the number of object classes in the m object images represented by the class information for the m object images (third configuration).

In the data processing device according to any one of the first to third configurations, the error identification unit may be configured to perform reclustering of n object images belonging to the error cluster based on n feature amounts for the n object images, and identify an object image having an error in the class information from among the n object images based on the result of the reclustering (n is an integer equal to or greater than 3) (fourth configuration).

In the data processing device according to the fourth configuration, the error identification unit may be configured to classify the n object images into first and second clusters by the reclustering, and, when the total number of object images belonging to the second cluster is smaller than the total number of object images belonging to the first cluster, identify the object image belonging to the second cluster as an object image having an error in the class information (fifth configuration).

In the data processing device according to the fourth configuration, the error identification unit may be configured to classify the n object images into first and second clusters by the reclustering, and, when the distance between the center of gravity of the n feature amounts for the n object images and the second cluster is greater than the distance between the center of gravity of the n feature amounts for the n object images and the first cluster, identify the object image belonging to the second cluster as an object image having an error in the class information (sixth configuration).

In the data processing device according to any one of the fourth to sixth configurations, the plurality of object images are images cut out from a plurality of input images, and the data processing device is provided with a learning dataset generation unit that uses the plurality of input images as a plurality of learning images and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies the position and class of an object in each learning image, and the learning dataset generation unit may be configured to extract, as an annotation target, an input image including an object image having an error in the class information from the plurality of input images based on the identification result of the error identification unit, and to correct the annotation information for the annotation target using information provided from outside for the annotation target (seventh configuration).

In the data processing device according to any one of the fourth to sixth configurations, the plurality of object images are images cut out from a plurality of input images, and the data processing device is provided with a learning dataset generation unit that extracts a portion of the plurality of input images as a plurality of learning images and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies the position and class of an object in each learning image, and the learning dataset generation unit may be configured to exclude input images including object images with errors in the class information from the plurality of learning images based on the identification result of the error identification unit (eighth configuration).

The data processing device according to any one of the first to eighth configurations may further include a machine learning processing unit that creates an inference model for object classification by executing machine learning of a neural network using the image classification dataset, and the feature derivation unit may be configured to derive features of each object image by inputting the image classification dataset to the inference model (ninth configuration).

The data processing method according to the present invention includes an image classification dataset acquisition step of acquiring an image classification dataset including a plurality of object images each including image data of an object and class information identifying a class of an object in each object image; a feature amount derivation step of deriving feature amounts of each object image in the image classification dataset; a clustering processing step of clustering the m object images based on the feature amounts of the m object images included in the plurality of object images and the class information for the m object images (m is an integer of 3 or more); an index derivation step of deriving, for each cluster obtained by the clustering, the center of gravity of a plurality of feature points belonging to the cluster and the distance between the center of gravity and each feature amount, and deriving the average and variance of the distances derived for the plurality of feature amounts; and an error identification step of identifying a cluster to which an object image having an error in the class information belongs as an error cluster based on the results of deriving the average and the variance (tenth configuration).

The present invention makes it possible to provide a data processing device and method that contributes to identifying errors in class information.

1 shows a configuration diagram of a data processing device according to an embodiment of the present invention. FIG. 2 illustrates a number of input images contemplated in an embodiment of the present invention. FIG. 2 is a diagram showing how a plurality of object images are cut out from one input image according to an embodiment of the present invention. FIG. 2 illustrates multiple object images contemplated in an embodiment of the present invention. FIG. 1 is a diagram showing an evaluation dataset (ES1) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. FIG. 6 is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images of FIG. 5 according to a first example belonging to an embodiment of the present invention. FIG. 11 is a diagram showing an evaluation dataset (ES2) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. FIG. 8 relates to a first example belonging to an embodiment of the present invention and is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images of FIG. 7. FIG. 11 is a diagram showing an evaluation dataset (ES3) including a plurality of object images and corresponding class information according to a first example belonging to an embodiment of the present invention. FIG. 10 relates to a first example belonging to an embodiment of the present invention and is a diagram showing a distribution of a plurality of feature points derived for the plurality of object images of FIG. 9. FIG. 8 is a diagram showing the results of reclustering performed on the evaluation dataset of FIG. 7 according to a second example belonging to an embodiment of the present invention. FIG. 10 is a diagram showing the results of reclustering performed on the evaluation dataset of FIG. 9 according to a second example belonging to an embodiment of the present invention. 13 is a partial flowchart showing the operation of a data processing device according to a fourth embodiment of the present invention.

Below, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are given the same reference numerals, and duplicated descriptions of the same parts will be omitted as a general rule. In this specification, for the sake of simplicity, a symbol or code referring to information, a signal, a physical quantity, a member, etc. may be written, and the name of the information, signal, physical quantity, member, etc. corresponding to the symbol or code may be omitted or abbreviated. For example, the image classification dataset acquisition unit referred to by "10" described below (see FIG. 1) may be written as image classification dataset acquisition unit 10 or abbreviated as acquisition unit 10, but they all refer to the same thing.

Figure 1 shows a configuration diagram of a data processing device 1 according to this embodiment. The data processing device 1 includes an image classification dataset acquisition unit 10, a machine learning processing unit 20, an inference unit 30, a clustering processing unit 40, an index derivation unit 50, an error identification unit 60, a learning dataset generation unit 70, and a database 80. The data processing device 1 may be configured as a single computer device, or may be configured as multiple physically separated computer devices. The data processing device 1 may also be configured using so-called cloud computing.

The image classification dataset acquisition unit 10 acquires an image classification dataset based on multiple input images. Here, as shown in FIG. 2, it is assumed that the image classification dataset is acquired based on L input images. L is an arbitrary integer equal to or greater than 2, and has a value of, for example, several thousand to several million. Each input image includes one or more objects. For example, a large number of still images selected from images captured by a camera mounted on a vehicle such as an automobile may be used as the multiple input images. The i-th input image of the L input images may be referred to by the symbol "II[i]". i represents an arbitrary integer.

In this embodiment, when image data of an object is included in an image, this may be expressed as the object being included or present in the image. Similarly, when image data of an object is included in a focused image area in an image (e.g., an object area described below), this may be expressed as the object being included or present in the focused image area. In this embodiment, an object refers to a recognition target object that is the subject of image recognition by the data processing device 1.

Image 610 in Figure 3 is an example of one input image. Input image 610 contains three objects 611 to 613. Objects 611, 612, and 613 are a vehicle, a human, and a traffic sign, respectively. In this embodiment, the vehicle is primarily assumed to be an automobile that can travel on the road.

The acquisition unit 10 may have an object detector generated by machine learning. In this case, the acquisition unit 10 can generate an image classification dataset by performing object detection for each input image using the object detector. In object detection, position identification is performed to identify the position of an object in an image that is the subject of image recognition, and class identification is performed to identify the class (type) of an object in an image that is the subject of image recognition.

Therefore, in object detection for the input image 610, the object detector can set object region 611B including object 611, object region 612B including object 612, and object region 613B including object 613 in the input image 610. Object regions 611B to 613B are set by identifying the positions of objects 611 to 613 in the input image 610. The acquisition unit 10 generates object images 621 to 623 by cutting out object regions 611B to 613B from the input image 610. The object region of a certain object is a rectangular region (preferably the smallest rectangular region) that surrounds the image of the object, and is also called a bounding box. Furthermore, in class identification, the object detector identifies the class (type) of the object in each object region.

There are multiple object classes. For the sake of concreteness and clarity, any object that is focused on as a target for any process, such as location identification or class identification, may be referred to as a focused object. In object detection, each object in an input image can be a focused object. In class identification, it is determined to which of the multiple classes the focused object belongs. The class of an object is synonymous with the type of object. Here, for the sake of concreteness, it is assumed that the multiple classes include first to third classes, and the first, second, and third classes are a vehicle, a human, and a traffic sign, respectively. In this case, class identification determines whether the focused object is a vehicle, a human, or a traffic sign, or whether it is an object of a class that does not belong to the first to third classes (for example, the fourth class).

In class identification of input image 610, the objects in object regions 611B, 612B, and 613B are identified as vehicles (i.e., first class objects), humans (i.e., second class objects), and traffic signs (i.e., third class objects), respectively (assuming there is no error in the class identification).

By performing object detection for each input image by the object detector, a large number of object images, each including image data of one object, are generated from the input images II[1] to II[L], and class information is assigned to each object image. The class information assigned to a certain object image is information that identifies the class of the object in the object image. Therefore, assuming that there is no error in the class identification, the class information assigned to the object images 621, 622, and 623 represents the first, second, and third classes, respectively. Note that the class information is also called a label. In addition, information that represents the result of the position identification is called object position information. The object position information for a certain input image identifies the position of the object region of each object in the input image (more specifically, the coordinate values of both ends of one diagonal line in the object region). Information including the class information and object position information is called annotation information. In object detection by the object detector, annotation information is generated for each input image.

As shown in FIG. 4, in the following, a total of H object images OI[1] to OI[H] are generated from input images II[1] to II[L], and the class information assigned to object image OI[i] is represented by the symbol "CLS[i]". H is an arbitrary integer of 2 or more, and since it is assumed here that one or more object regions are cut out from each input image, "H≧L". In reality, for example, H has a value of several thousand to several million. The image classification dataset includes image data of object images OI[1] to OI[H] and class information CLS[1] to CLS[H], and in the image classification dataset, the image data of object image OI[i] and the class information CLS[i] are associated with each other. In addition, extraction source information indicating which input image each object image is based on is shared and recognized by each part in the data processing device 1. Therefore, for example, if object images OI[1] to OI[5] are extracted from input image II[1], this is specified in the extraction source information. The extraction source information may be included in the image classification dataset.

Although the method for generating an image classification dataset when the acquisition unit 10 has an object detector has been described, it is not essential that the acquisition unit 10 has an object detector. For example, the image classification dataset may be generated by inputting annotation information into the data processing device 1 based on a manual operation by a human. That is, for example, an object region may be set for each input image and class information for each object region (in other words, each object image) may be specified based on operation information provided to the data processing device 1 by an administrator of the data processing device 1 through a predetermined man-machine interface (not shown), thereby generating annotation information for each input image and generating and acquiring an image classification dataset. Alternatively, for example, the acquisition unit 10 may receive input of a dataset created in advance from an external device (not shown) different from the data processing device 1 through wired or wireless communication. The dataset here includes image data of the input images II[1] to II[L] and annotation information for each input image, and the acquisition unit 10 may generate and acquire an image classification dataset based on the input dataset. In the following description, unless otherwise specified, the image classification dataset refers to the image classification dataset acquired by the acquisition unit 10.

The machine learning processing unit 20 has a neural network 21 (hereinafter referred to as NN21) and executes machine learning processing. In the machine learning processing, the machine learning processing unit 20 creates an inference model that performs a predetermined inference by executing machine learning of the NN21 using an image classification dataset. The machine learning here may be classified as deep learning, and therefore the NN21 may be a deep neural network. During the machine learning, the parameters (weights and biases) of the NN21 are appropriately set. The NN21 after the parameters (weights and biases) have been appropriately set corresponds to the inference model.

The inference unit 30 performs a predetermined inference using an inference model by the NN21. The NN21 is a neural network for image classification, and therefore the inference in the inference model is for image classification. That is, when an object image including an object is given, the inference unit 30 performs class discrimination in inference to identify (discriminate) the class of the object included in the object image. At this time, the inference unit 30 performs a feature derivation process to derive features useful for realizing class discrimination from the object image, and identifies (discriminates) the class of the object included in the object image based on the features. The features are multidimensional vector quantities, and may be read as feature vectors. The feature derivation process is performed by the feature derivation unit 31 included in the inference unit 30. An image classification dataset is input to the inference model of the NN21, and inference is performed for each object image in the image classification dataset using the inference model of the NN21. In this inference, a feature derivation process is performed for each object image in the image classification dataset, thereby deriving the features of each object image.

The clustering processing unit 40 executes clustering processing based on the results of the feature derivation processing. That is, in the clustering processing, the clustering processing unit 40 performs clustering of multiple object images in the image classification dataset based on the features derived for each object image in the image classification dataset. The clustering processing unit 40 may perform dimensionality reduction of the features derived in the feature derivation processing, and perform clustering based on the features after dimensionality reduction. For example, dimensionality reduction can be performed using principal component analysis. The features shown below refer to the features themselves derived in the feature derivation processing if dimensionality reduction is not performed, and refer to the features after dimensionality reduction if dimensionality reduction is performed.

Clustering is performed based on the distance between multiple feature amounts derived for multiple object images in an image classification dataset. The distance between feature amounts is a distance in a feature space in which the feature amounts are defined, and is, for example, Euclidean distance or cosine similarity. By the clustering process, object images with similar feature amounts are classified into a common cluster in the image classification dataset. Hierarchical clustering may be used as a clustering method. However, clustering may also be performed using non-hierarchical clustering (such as k-means).

The clustering in the clustering processing unit 40 is not performed collectively on all object images included in the image classification dataset. In this embodiment, the object images OI[1] to OI[H] are divided into a number of groups, and clustering of the object images belonging to each group is performed for each group. In other words, the input images [1] to [L] are divided into a number of groups, and clustering of the object images in the input images belonging to each group is performed for each group. Specific examples and technical significance of such clustering will be described later.

Creating an image classification dataset requires annotation of the location of objects in the input images and the object classes. Whether an image classification dataset is created using an object detector or through manual operations, the object classes may be incorrectly annotated. For example, an object of interest may be erroneously assigned as a vehicle (i.e., a first class object) when in fact it is a human (i.e., a second class object), resulting in incorrect class information being included in the image classification dataset.

The index derivation unit 50 derives indices that are useful for identifying such errors, and the error identification unit 60 identifies erroneous data, etc. based on the derived indices (details will be described later).

The training dataset generator 70 generates a training dataset that includes image data of multiple training images and annotation information that identifies the position and class of objects in each training image.

For convenience, the dataset including the image data of input images II[1] to II[L] handled by the acquisition unit 10 and the annotation information of each input image is referred to as the provisional dataset. The provisional dataset is the original dataset for the learning dataset to be generated by the generation unit 70, and if there are no errors in the annotations in the provisional dataset, the provisional dataset itself becomes the learning dataset. However, as described above, a huge number of annotations may contain errors, and the generation unit 70 corrects or removes the errors, and generates the provisional dataset after the errors have been corrected or removed as the learning dataset.

The database 80 is made of a non-volatile recording medium, and stores the learning dataset generated by the generation unit 70 under the control of the generation unit 70. The provisional dataset may also be stored in the database 80. It may also be considered that the provisional dataset stored in the database 80 is rewritten or updated into the learning dataset by the generation unit 70. The database 80 may be configured from a magnetic disk, an optical disk, or a semiconductor memory, or a combination of these. The database 80 may also be an external recording device provided outside the data processing device 1.

Below, several specific examples of operation, application techniques, modification techniques, etc. related to the data processing device 1 will be described in multiple examples. The matters described above in this embodiment are applied to each of the following examples unless otherwise specified and unless there is a contradiction. If there are any matters in each example that contradict the matters described above, the description in each example may take precedence. Furthermore, unless there is a contradiction, the matters described in any of the multiple examples shown below can also be applied to any of the other examples (i.e., any two or more of the multiple examples can also be combined).

<<First Example>>
A first embodiment will be described. As described above, the clustering in the clustering processing unit 40 is not performed collectively on all object images included in the image classification dataset, but is performed on an evaluation dataset including a portion of all object images included in the image classification dataset.

First, referring to Fig. 5, clustering for an evaluation dataset ES1 including image data of object images OI[11] to OI[20], which are part of object images OI[1] to OI[H], will be described. The evaluation dataset ES1 includes class information (i.e., CLS[11] to CLS[20]; see Fig. 4) of object images OI[11] to OI[20] extracted from an image classification dataset. Assume that object images OI[11] to OI[20] are object images cut out from one or more input images II[J _A1 ] to II[J _A2 ]. "0<J _A1 ≦J _A2 <L" holds, and the difference (J _A2 -J _A1 ) has an integer value of 0 or 1 or more.

The object contained in each of object images OI[11] to OI[14] is a vehicle, and the object contained in each of object images OI[15] to OI[20] is a human. In the image classification dataset and evaluation dataset ES1, the class information of each of object images OI[11] to OI[14] indicates a first class corresponding to a vehicle, and the class information of each of object images OI[15] to OI[20] indicates a second class corresponding to a human. There is no error in this class information. A feature derivation process is performed on each object image to derive the features of each object image. Hereinafter, the feature of object image OI[i] will be referred to by the symbol "FT[i]".

Figure 6 shows feature quantities FT[11] to FT[20] corresponding to object images OI[11] to OI[20] plotted in feature space. In reality, each feature quantity is a multidimensional vector quantity, but in Figure 6, each feature quantity is shown as if it were a two-dimensional vector quantity. This is also true for other figures described below that show how feature quantities are plotted in feature space. However, the number of dimensions of the feature quantities may be reduced to "2" by the above-mentioned dimensionality reduction.

In the following, for any natural number u, the fact that the object image OI[u] is classified into or belongs to a certain cluster of interest may be expressed as the feature FT[u] being classified into or belonging to the cluster of interest. In other words, the fact that the object image OI[u] is classified into or belongs to the cluster of interest is equivalent to the fact that the feature FT[u] is classified into or belongs to the cluster of interest.

The clustering processing unit 40 classifies the object images OI[11] to OI[20] into k _A clusters based on the class information of the object images OI[11] to OI[20] in the image classification dataset (evaluation dataset ES1) (i.e., the number of clusters is set to k _A and then clustering is performed). k _A is the number of types of classes represented by the class information for the object images OI[11] to OI[20] in the image classification dataset (i.e., the number of types of objects in the object images OI[11] to OI[20] represented by the class information). In other words, k _A is equal to the number of types of classes (total number) indicated by the total of 10 pieces of class information assigned to the object images OI[11] to OI[20]. In the example of FIG. 5, the classes indicated by the total of 10 pieces of class information assigned to the object images OI[11] to OI[20] are the first class and the second class, so "k _A = 2". When non-hierarchical clustering is performed as the clustering, the total number of clusters can be set to "k _A = 2" before performing the clustering. When hierarchical clustering is performed as the clustering, the clustering can be stopped when the total number of clusters reaches "k _A = 2".

As shown in Fig. 6, two clusters CT _A1 and CT _A2 are set for the features FT[11] to FT[20]. The features FT[11] to FT[14] are classified into the cluster CT _A1 , and the features FT[15] to FT[20] are classified into the cluster CT _A2 . In Fig. 6, point G _A1 represents the center of gravity of the features FT[11] to FT[14] belonging to the cluster CT _A1 , and point G _A2 represents the center of gravity of the features FT[15] to FT[20] belonging to the cluster CT _A2 .

The index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40 and derives the distance between the center of gravity and each feature amount belonging to the cluster. The distance between the center of gravity and the feature amount is a distance in the feature space in which the feature amount is defined, similar to the distance between two feature amounts, and is, for example, Euclidean distance or cosine similarity. The distances derived for the feature amounts FT[11] to FT[20] are referred to by the symbols "d[11] to d[20]". The distances d[11] to d[14] respectively represent the distance between the center of gravity G _A1 and the feature amounts FT[11] to FT[14]. The distances d[15] to d[20] respectively represent the distance between the center of gravity G _A2 and the feature amounts FT[15] to FT[20].

The index derivation unit 50 further derives the average and variance of the distances derived for each cluster generated by the clustering processing unit 40. The average and variance derived by the index derivation unit 50 are hereinafter referred to as the distance average and distance variance. The distance average and distance variance derived for the cluster CT _A1 are referred to as "AVE _A1 " and "σ ² _A1 ", respectively. Then, the distance average AVE _A1 is the average of the distances d[11] to d[14], and the distance variance σ ² _A1 is the variance of the distances d[11] to d[14]. Similarly, the distance average and distance variance derived for the cluster CT _A2 are referred to as "AVE _A2 " and "σ ² _A2 ", respectively. Then, the distance average AVE _A2 is the average of the distances d[15] to d[20], and the distance variance σ ² _A2 is the variance of the distances d[15] to d[20].

The error identification unit 60 judges whether each cluster is an error cluster based on the distance average and distance variance derived for each cluster by the index derivation unit 50. An error cluster is a cluster to which an object image with erroneous class information belongs. An object image with erroneous class information is, in other words, an object image to which erroneous class information has been assigned, and is hereinafter also referred to as an object image with an error label. Conversely, an object image to which correct class information has been assigned is referred to as an object image with a correct label. In addition, hereinafter, a cluster that has not been identified as an error cluster by the error identification unit 60 may be referred to as a correct cluster. It can also be said that the error identification unit 60 has a function of judging whether each cluster is a correct cluster or an error cluster.

The error identifying unit 60 compares the distance average and distance variance with predetermined thresholds _TH1 and _TH2 , respectively, for each cluster generated by the clustering processing unit 40, and identifies a cluster that satisfies a predetermined error condition as an error cluster based on the comparison result. When the corresponding distance average is equal to or greater than the predetermined threshold _TH1 and the corresponding distance variance is equal to or greater than the predetermined threshold _TH2 for a certain cluster, the error condition is satisfied for that cluster.

The thresholds TH ₁ and TH ₂ may be set as follows. That is, a plurality of object images for threshold setting, each of which includes a recognition target object, may be prepared in advance, and the thresholds TH ₁ and TH ₂ may be set based on the clustering results for the plurality of object images for threshold setting. At that time, the clustering processing unit 40 classifies the plurality of object images for threshold setting into a plurality of clusters. Next, the index derivation unit 50 derives the average and variance of the plurality of distances for each cluster. Next, for the plurality of object images for threshold setting, a value obtained by adding an appropriate offset value (which may be 0) to the average of the plurality of distances is set as the threshold TH ₁ , and a value obtained by adding an appropriate offset value (which may be 0) to the variance of the plurality of distances is set as the threshold TH _2. Then, the set thresholds TH ₁ and TH ₂ may be used in the above comparison with the corresponding clusters (clusters related to the same type of object). The image data of the plurality of object images for threshold setting may be obtained from an existing and arbitrary dataset, and may be, for example, image data included in the Microsoft COCO dataset.

For the cluster CT _A1, the distance average AVE _A1 and the distance variance σ ² _A1 are compared with the thresholds TH ₁ and TH _2, respectively, and for the cluster CT _A2 , the distance average AVE _A2 and the distance variance σ ² _A2 are compared with the thresholds TH ₁ and TH ₂ , respectively. In the examples of Figs. 5 and 6, there is no error in the class information of the object images OI[10] to OI[20] in the image classification dataset. Therefore, the distance average AVE _A1 and the distance variance σ ² _A1 in the cluster CT _A1 are smaller than the thresholds TH ₁ and TH ₂ , and the distance average AVE _A2 and the distance variance σ ² _A2 in the cluster CT _A2 are also smaller than the thresholds TH ₁ and TH _2. Therefore, neither the cluster CT _A1 nor the cluster CT _A2 is identified as an incorrect cluster (in other words, it is determined to be a correct cluster).

Next, referring to Fig. 7, clustering for the evaluation dataset ES2 including image data of body images OI[31] to OI[40] which are part of the object images OI[1] to OI[H] will be described. The evaluation dataset ES2 includes class information (i.e., CLS[31] to CLS[40]; see Fig. 4) of the object images OI[31] to OI[40] extracted from the image classification dataset. Assume that the object images OI[31] to OI[40] are object images cut out from one or more input images II[J _B1 ] to II[J _B2 ]. "J _A2 <J _B1 ≦J _B2 <L" holds, and the difference (J _B2 -J _B1 ) has an integer value of 0 or 1 or more.

The object included in each of the object images OI[31] to OI[34] is a vehicle, the object included in each of the object images OI[35] to OI[38] is a human, and the object included in each of the object images OI[39] and OI[40] is a traffic sign. In the image classification dataset and the evaluation dataset ES2, the class information of each of the object images OI[31] to OI[34] indicates a first class corresponding to a vehicle, and the class information of each of the object images OI[35] to OI[38] indicates a second class corresponding to a human. There is no error in this class information. However, in the image classification dataset and the evaluation dataset ES2, the class information of the object images OI[39] and OI[40] indicates a second class corresponding to a human, and therefore the class information of the object images OI[39] and OI[40] contains an error.

FIG. 8 is a diagram showing a state where feature quantities FT[31] to FT[40] corresponding to object images OI[31] to OI[40] are plotted on a feature space. The clustering processing unit 40 classifies object images OI[31] to OI[40] into k _B clusters based on class information of object images OI[31] to OI[40] in the image classification dataset (evaluation dataset ES2) (i.e., the number of clusters is set to k _B and then clustering is performed). k _B is the number of types of classes represented by class information for object images OI[31] to OI[40] in the image classification dataset (i.e., the number of types of objects in object images OI[31] to OI[40] represented by class information). In other words, k _B is equal to the number of types of classes (total number) indicated by a total of 10 pieces of class information assigned to object images OI[31] to OI[40]. 7, the classes indicated by the 10 pieces of class information assigned to the object images OI[31] to OI[40] should be three, the first to third classes, but are mistakenly indicated as two, the first and second classes, resulting in "k _B = 2." When non-hierarchical clustering is performed as the clustering, the total number of clusters can be set to "k _B = 2" before clustering is performed, and when hierarchical clustering is performed as the clustering, the clustering can be stopped when the total number of clusters reaches "k _B = 2."

As shown in Fig. 8, two clusters CT _B1 and CT _B2 are set for the features FT[31] to FT[40]. Here, it is assumed that the features FT[31] to FT[34] are classified into the cluster CT _B1 , and the features FT[35] to FT[40] are classified into the cluster CT _B2 . In Fig. 8, point G _B1 represents the center of gravity of the features FT[31] to FT[34] belonging to the cluster CT _B1 , and point G _B2 represents the center of gravity of the features FT[35] to FT[40] belonging to the cluster CT _B2 .

As described above, the index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40, and derives the distance between the center of gravity and each feature belonging to the cluster. The distances derived for the features FT[31] to FT[40] are referred to by the symbols "d[31] to d[40]", respectively. The distances d[31] to d[34] represent the distances between the center of gravity G _B1 and the features FT[31] to FT[34], respectively. The distances d[35] to d[40] represent the distances between the center of gravity G _B2 and the features FT[35] to FT[40], respectively.

The index derivation unit 50 further derives the distance average and distance variance for each cluster as described above. The distance average and distance variance derived for the cluster CT _B1 are referred to by the symbols "AVE _B1 " and "σ ² _B1 ", respectively. Then, the distance average AVE _B1 is the average of the distances d[31] to d[34], and the distance variance σ ² _B1 is the variance of the distances d[31] to d[34]. Similarly, the distance average and distance variance derived for the cluster CT _B2 are referred to by the symbols "AVE _B2 " and "σ ² _B2 ", respectively. Then, the distance average AVE _B2 is the average of the distances d[35] to d[40], and the distance variance σ ² _B2 is the variance of the distances d[35] to d[40].

The error identification unit 60 judges whether the error condition is satisfied for each of the clusters CT _B1 and CT _B2 . For the cluster CT _B1 , the distance average AVE _B1 and the distance variance σ ² _B1 are compared with the thresholds TH ₁ and TH _2, respectively, and for the cluster CT _B2, the distance average AVE _B2 and the distance variance σ ² _B2 are compared with the thresholds TH ₁ and TH ₂ , respectively. In the example of Fig. 7 and Fig. 8, the distance average AVE _B1 and the distance variance σ ² _B1 in the cluster CT _B1 are smaller than the thresholds TH ₁ and TH ₂ , respectively, and as a result, the cluster CT _B1 is not identified as an error cluster (in other words, it is determined to be a correct cluster). However, the distance average AVE _B2 and the distance variance σ ² _B2 in the cluster CT _B2 are larger than the thresholds TH ₁ and TH ₂ , respectively. Therefore, cluster CT _B2 satisfies the error condition, and as a result, cluster CT _B2 is identified as an error cluster.

Next, referring to Fig. 9, clustering for an evaluation dataset ES3 including image data of body images OI[51] to OI[60] which are part of object images OI[1] to OI[H] will be described. The evaluation dataset ES3 includes class information (i.e., CLS[51] to CLS[60]; see Fig. 4) of object images OI[51] to OI[60] extracted from an image classification dataset. Assume that object images OI[51] to OI[60] are object images cut out from one or more input images II[J _C1 ] to II[J _C2 ]. "J _B2 <J _C1 ≦J _C2 <L" holds, and the difference (J _C2 -J _C1 ) has an integer value of 0 or 1 or more.

The object contained in each of object images OI[51] to OI[58] is a vehicle, and the object contained in each of object images OI[59] and OI[60] is a human. In the image classification dataset and evaluation dataset ES3, the class information of each of object images OI[51] to OI[58] indicates the first class corresponding to a vehicle. There is no error in this class information. However, in the image classification dataset and evaluation dataset ES3, the class information of object images OI[59] and OI[60] also indicates the first class corresponding to a vehicle, and therefore the class information of object images OI[59] and OI[60] contains an error.

FIG. 10 is a diagram showing a state where feature quantities FT[51] to FT[60] corresponding to object images OI[51] to OI[60] are plotted on a feature space. The clustering processing unit 40 classifies object images OI[51] to OI[60] into k _C clusters based on class information of object images OI[51] to OI[60] in the image classification dataset (evaluation dataset ES3) (i.e., the number of clusters is set to k _C and then clustering is performed). k _C is the number of types of classes represented by the class information for object images OI[51] to OI[60] in the image classification dataset (i.e., the number of types of objects in object images OI[51] to OI[60] represented by class information). In other words, k _C is equal to the number of types of classes (total number) indicated by a total of 10 pieces of class information assigned to object images OI[51] to OI[60]. In the example of Fig. 9, the classes indicated by the total of 10 pieces of class information assigned to the object images OI[51] to OI[60] should be two classes, the first and second classes, but are mistakenly indicated as only the first class, so " _kC = 1". When non-hierarchical clustering is performed as the clustering, the total number of clusters may be set to " _kC = 1" before clustering is performed, and when hierarchical clustering is performed as the clustering, clustering may be continued until the total number of clusters becomes " _kC = 1". However, when " _kC = 1", it can be said that a substantial clustering operation is not necessary, and it is sufficient to simply set a single cluster that includes all of the feature amounts FT[51] to FT[60].

As shown in Fig. 10, one cluster CT _C1 is set for the features FT[51] to FT[60]. In Fig. 10, a point G _C1 represents the center of gravity of the features FT[51] to FT[60] belonging to the cluster CT _C1 .

As described above, the index derivation unit 50 derives the center of gravity of each cluster generated by the clustering processing unit 40, and derives the distance between the center of gravity and each feature amount belonging to the cluster. Since a single cluster CT _C1 is set for the evaluation data set ES3, each distance is derived focusing only on the cluster CT _C1 . The distances derived for the features FT[51] to FT[60] are referred to by the symbols "d[51] to d[60]", respectively. The distances d[51] to d[60] represent the distance between the center of gravity G _C1 and the features FT[51] to FT[60], respectively.

The index derivation unit 50 further derives the distance average and distance variance for the cluster CT _C1 . The distance average and distance variance derived for the cluster CT _C1 are referred to by the symbols "AVE _C1 " and "σ ² _C1 ", respectively. Then, the distance average AVE _C1 is the average of the distances d[51] to d[60], and the distance variance σ ² _C1 is the variance of the distances d[51] to d[60].

The error identification unit 60 judges whether the error condition is satisfied for the cluster CT _C1 . The distance average AVE _C1 and the distance variance σ ² _C1 for the cluster CT _C1 are compared with the thresholds TH ₁ and TH ₂ , respectively. In the examples of Fig. 9 and Fig. 10, the distance average AVE _C1 and the distance variance σ ² _C1 for the cluster CT _C1 are greater than the thresholds TH ₁ and TH ₂ , respectively. Therefore, the cluster CT _C1 satisfies the error condition, and as a result, the cluster CT _C1 is identified as an error cluster.

The features in the correct cluster are for objects of the same class, so the variation in the features in the correct cluster is relatively small. In contrast, the features in the error cluster include features for objects of different classes, so the variation in the features in the error cluster is relatively large. Taking these factors into consideration, as described above, the distance mean and distance variance are derived for each cluster, and the error cluster is identified based on the derived results. This method makes it possible for the data processing device 1 itself to detect error clusters well without relying on human eyes, and reduces the cost (time and human effort) of searching for defects in the dataset compared to when relying on human eyes. If an error cluster can be identified, it becomes possible to take measures such as correcting the class information of the object image belonging to the error cluster to generate a learning dataset or excluding an input image including the object image from the learning dataset, and as a result, it becomes possible to improve the quality of the learning dataset. By improving the quality of the learning dataset, the performance of an inference model for image recognition constructed by machine learning using the learning dataset can be improved.

<<Second Example>>
A second embodiment will be described. The second embodiment is implemented in combination with the above-described first embodiment. In the second embodiment, an error label specification process that can be performed by the error specification unit 60 when an error cluster is specified will be described.

In the error label identification process, the error identification unit 60 performs clustering to group object images belonging to an error cluster into a first and a second cluster within the error cluster. The clustering in the error label identification process is called reclustering to clearly distinguish it from the above-mentioned clustering performed by the clustering process 40. The reclustering is performed based on the feature amounts of each object image belonging to the error cluster. In other words, if n object images belong to a certain error cluster, reclustering is performed based on the n feature amounts derived in the feature derivation process for the n object images. However, this reclustering is performed on the premise that "n ≧ 3" is satisfied. Therefore, the error label identification process can be performed on an error cluster that satisfies "n ≧ 3". The reclustering can be realized by non-hierarchical clustering (such as k-means) with the number of clusters set to 2.

The error identification unit 60 involved in the error label identification process identifies an erroneously labeled object image (i.e., an object image with an error in the class information) from among the n object images based on the result of reclustering. It can also be said that the error identification unit 60 has a function of determining whether each object image belonging to an error cluster is an object image with a correct label or an object image with an error label. As the error label identification process, the following first or second error label identification process can be adopted.

The first error label identification process will be described with attention to the evaluation data set ES2 corresponding to FIG. 7 and FIG. 8. Please refer to FIG. 11. For the evaluation set ES2, the cluster CT _B2 is identified as an error cluster as described in the first embodiment. The object images OI[35] to OI[40] belong to the error cluster CT _{B2 .} In other words, the feature amounts FT[35] to FT[40] belong to the error cluster CT B2. Therefore, "n=6" in the error cluster CT _B2 . The error identification unit 60 classifies the object images OI[35] to OI[40] into the first and second clusters by re-clustering the object images OI[35] to OI[40] based on the feature amounts FT[35] to FT[40]. In FIG. 11, one of the clusters CT _{B2_1} and CT _{B2_2} corresponds to the first cluster, and the other corresponds to the second cluster. The features FT[35] to FT[38] are classified into a cluster CT _{B2_1} , and the features FT[39] and FT[40] are classified into a cluster CT _{B2_2} . The point G _B2 in FIG. 11 coincides with the center of gravity G _B2 in FIG.

In the first error label identification process, of the first and second clusters formed by reclustering each object image in the error cluster, the object image in the cluster with fewer object images is identified as an object image with an error label. That is, in the first error label identification process, after forming the first and second clusters by reclustering each object image in the error cluster, if the total number of object images belonging to the second cluster is smaller than the total number of object images belonging to the first cluster, the object image belonging to the second cluster is identified as an object image with an error label (i.e., an object image with an error in class information), and if the total number of object images belonging to the first cluster is smaller than the total number of object images belonging to the second cluster, the object image belonging to the first cluster is identified as an object image with an error label.

11 , the total number of object images belonging to the cluster CT _{B2_1} is 4, and the total number of object images belonging to the cluster CT _{B2_2} is 2. Therefore, when the first erroneous label identification process is applied to the erroneous clusters of the evaluation data set ES2, each object image belonging to the cluster CT _{B2_2} is identified as an erroneously labeled object image.

The second error label identification process will be described with a focus on the evaluation data set ES2. The process up to classifying the feature quantities FT[35] to FT[40] into the clusters CT _{B2_1} and CT _{B2_2} is common to the first and second error label identification processes. In the second error label identification process, of the first and second clusters formed by reclustering each object image in the error cluster, the object image in the cluster farther from the center of gravity of the error cluster is identified as an object image with an error label. That is, in the second error label identification process, the first and second clusters are formed by reclustering each object image in the error cluster. Then, if the distance between the center of gravity of the error cluster and the second cluster is greater than the distance between the center of gravity of the error cluster and the first cluster, the object image belonging to the second cluster is identified as an object image with an error label (i.e., an object image with an error in class information). Conversely, if the distance between the center of gravity of the error cluster and the first cluster is greater than the distance between the center of gravity of the error cluster and the second cluster, the object image belonging to the first cluster is identified as an object image with an error label. The distance between the center of gravity of the error cluster and the first cluster may be the distance between the center of gravity of the error cluster and the center of gravity of all features belonging to the first cluster, and the distance between the center of gravity of the error cluster and the second cluster may be the distance between the center of gravity of the error cluster and the center of gravity of all features belonging to the second cluster.

11 (see also FIG. 8 ), the distance between the center of gravity G _B2 of the error cluster CT _B2 and the cluster CT _{B2_2} (more specifically, the distance between the center of gravity G _B2 and the centers of gravity of the features FT[39] and FT[40] belonging to the cluster CT _{B2_2} ) is greater than the distance between the center of gravity G _B2 of the error cluster CT _B2 and the cluster CT _{B2_1} (more specifically, the distance between the center of gravity G _B2 and the centers of gravity of the features FT[35] to FT[38] belonging to the cluster CT _{B2_1} ). Therefore, when the second error label identification process is applied to the error clusters of the evaluation dataset ES2, each object image belonging to the cluster CT _{B2_2} is identified as an erroneously labeled object image, as in the case of applying the first error label identification process.

The first error label identification process will be described with attention to the evaluation data set ES3 corresponding to FIG. 9 and FIG. 10. Please refer to FIG. 12. For the evaluation data set ES3, the cluster CT _C1 is identified as an error cluster as described in the first embodiment. The object images OI[51] to OI[60] belong to the error cluster CT _{C1 .} In other words, the feature amounts FT[51] to FT[60] belong to the error cluster CT C1. Therefore, "n=10" in the error cluster CT _C1 . The error identification unit 60 classifies the object images OI[51] to OI[60] into the first and second clusters by re-clustering the object images OI[51] to OI[60] based on the feature amounts FT[51] to FT[60]. In FIG. 12, one of the clusters CT _{C1_1} and CT _{C1_2} corresponds to the first cluster, and the other corresponds to the second cluster. The features FT[51] to FT[58] are classified into a cluster CT _{C1_1} , and the features FT[59] and FT[60] are classified into a cluster CT _{C1_2} . The point G _C1 in FIG. 12 coincides with the center of gravity G _C1 in FIG.

12 , the total number of object images belonging to the cluster CT _{C1_1} is 8, and the total number of object images belonging to the cluster CT _{C1_2} is 2. Therefore, when the first erroneous label identification process is applied to the erroneous clusters of the evaluation dataset ES3, each object image belonging to the cluster CT _{C1_2} is identified as an erroneously labeled object image.

12 (see also FIG. 10 ), the distance between the center of gravity G _C1 of the error cluster CT _C1 and the cluster CT _{C1_2} (more specifically, the distance between the center of gravity G _C1 and the centers of gravity of the features FT[59] and FT[60] belonging to the cluster CT _{C1_2} ) is greater than the distance between the center of gravity G _C1 of the error cluster CT _C1 and the cluster CT _{C1_1 (more specifically, the distance between the center of gravity G C1 and the centers of gravity of the features FT[51] to FT[58] belonging to the cluster CT C1_1 ) .} For this reason, even when the second error label identification process is applied to the error clusters of the evaluation dataset ES3, each object image belonging to the cluster CT _{C1_2} is identified as an erroneously labeled object image, as in the case of applying the first error label identification process.

As shown in this embodiment, if reclustering is performed on the erroneous clusters, the erroneous clusters are expected to be decomposed into clusters to which correctly labeled object images belong and clusters to which erroneously labeled object images belong. Therefore, by referring to the results of reclustering, it becomes possible to easily identify erroneously labeled object images.

More specifically, when reclustering is performed on the erroneous clusters to form the first and second clusters, it is expected that the object images belonging to one of the first and second clusters are correctly labeled object images, and the object images belonging to the other cluster are incorrectly labeled object images. On the other hand, it is expected that the majority of annotation information in the image classification dataset is basically correct. Therefore, of the first and second clusters, the object images in the cluster that contains fewer object images are more likely to be incorrectly labeled object images. By using the first erroneous label identification process based on this knowledge, it is possible to accurately identify incorrectly labeled object images.

In addition, since most of the annotation information in an image classification dataset is expected to be correct, of the first and second clusters formed by reclustering each object image in an erroneous cluster, the object image in the cluster farther from the center of gravity of the erroneous cluster is likely to be an erroneously labeled object image. By using the second erroneous label identification process based on this knowledge, erroneously labeled object images can also be identified with high accuracy.

<<Third Example>>
A third embodiment will be described. The third embodiment is implemented in combination with the first and second embodiments. As described above, the learning dataset generation unit 70 generates a learning dataset including image data of a plurality of learning images and annotation information that identifies the position and class of an object in each learning image. Each of the input images II[1] to II[L] is a candidate for a learning image, and if there is no erroneously labeled object image in a dataset (provisional dataset) including the image data of the input images II[1] to II[L] and the annotation information of each input image handled by the acquisition unit 10, the provisional dataset becomes the learning dataset as it is. However, in reality, there are often erroneously labeled object images to a certain extent. In the third embodiment, an erroneous label correspondence process that can be executed by the learning dataset generation unit 70 when an erroneously labeled object image is identified will be described.

First, attention is paid to the evaluation data set ES1 corresponding to FIG. 5 and FIG. 6. For the evaluation data set ES1, neither of the clusters CT _A1 and CT _A2 generated by the clustering processing unit 40 is identified as an erroneous cluster, and therefore none of the object images OI[11] to OI[20] are identified as erroneously labeled object images. The generation unit 70 does not perform erroneous label correspondence processing on the evaluation data set (ES1 in this case) in which no erroneously labeled object images are identified. The generation unit 70 regards each object image in the evaluation data set in which no erroneous clusters are identified as a correctly labeled object image, and sets each input image including a correctly labeled object image as a learning image. The image data of each input image set as a learning image is included in the learning data set.

For example, if object images OI[11] to OI[14] are included in input image II[J _A1 ], object images OI[15] to OI[17] are included in input image II[J _A1 +1], and object images OI[18] to OI[20] are included in input image II[J _A1 +2] (assuming here that "J _A1 +2 = J _A2 "), then the three input images II[J _A1 ] to II[J _A1 +2] are set as three learning images, and the image data of input images II[J _A1 ] to II[J _A1 +2] and the annotation information of input images II[J _A1 ] to II[J _A1 +2] in the provisional dataset are included as they are in the learning dataset.

Next, attention is paid to the evaluation data set ES2 corresponding to Figures 7, 8, and 11. Regarding the evaluation data set ES2, of the clusters CT _B1 and CT _B2 generated by the clustering processing unit 40, only the cluster CT _B2 is identified as an erroneous cluster, and of the object images OI [31] to [40], only the object images OI [39] and [40] are identified as erroneously labeled object images. The generating unit 70 executes an erroneous label correspondence process on the evaluation data set (ES2 in this case) in which erroneously labeled object images have been identified. As the erroneous label correspondence process, the following first erroneous label correspondence process can be adopted.

Now, assume case CASE1 in which object images OI[31] to OI[34] are included in input image II[J _B1 ], object images OI[35] to OI[37] are included in input image II[J _B1 +2], and object images OI[38] to OI[40] are included in input image II[J _B1 +2] (in this case, "J _B1 +2 = J _B2 ").

In the first error label correspondence process for case CASE1, the generation unit 70 extracts the input image II[J _B1 +2] including the object images OI[39] and [40], which are object images with an error label, as an annotation target, and notifies the administrator of the data processing device 1 that the annotation information of the input image II[J _B1 +2] in the provisional data set may contain an error (this notification is referred to as notification NT1). The notification NT1 may be performed through a display on a display screen (not shown) provided in or connected to the data processing device 1. The notification NT1 may include a correct answer request notification for requesting a correct answer for the annotation information of the input image II[J _B1 +2]. The administrator who has received the notification NT1 inputs correct answer information indicating the correct answer for the annotation information of the input image II[J _B1 +2] into the data processing device 1 through a predetermined man-machine interface (not shown) such as a pointing device. The correct answer information here indicates that the objects in the object images OI[39] and [40] are traffic signs (i.e., objects of the third class). The generation unit 70 corrects the annotation information for the input image II[J _B1 +2] (annotation information in the provisional data set) based on the correct answer information, and includes the image data of the input image II[J _B1 +2] and the corrected annotation information in the learning data set as a set of image data of one teacher image and the annotation information of the teacher image.

In case CASE 1, input images II[J _B1 ] and II[J _B1 +1] are set as two learning images, and the image data of input images II[J _B1 ] and II[J _B1 +1] and the annotation information of input images II[J _B1 ] and II[J _B1 +1] in the provisional dataset are included directly in the learning dataset.

There may also be a case CASE2 in which all of the object images OI[31] to OI[40] are included in a single input image II[J _B1 ] (in this case, "J _B1 =J _B2 ").

In the first error label correspondence process related to case CASE2, the generation unit 70 extracts the input image II[J _B1 ] including the object images OI[39] and [40], which are object images with an error label, as an annotation target, and notifies the administrator of the data processing device 1 that the annotation information of the input image II[J _B1 ] in the provisional data set may contain an error (this notification is referred to as notification NT2). The notification NT2 may be performed through a display on a display screen (not shown) provided in or connected to the data processing device 1. The notification NT2 may include a correct answer request notification for requesting a correct answer for the annotation information of the input image II[J _B1 ]. The administrator who has received the notification NT2 inputs correct answer information indicating the correct answer for the annotation information of the input image II[J _B1 ] into the data processing device 1 through a predetermined man-machine interface (not shown) such as a pointing device. The correct answer information here indicates that the objects in the object images OI[39] and [40] are traffic signs (i.e., objects of the third class). The generation unit 70 corrects the annotation information for the input image II[J _B1 ] (annotation information in the provisional dataset) based on the correct answer information, and includes the image data of the input image II[J _B1 ] and the corrected annotation information in the learning dataset as a set of image data of one teacher image and the annotation information of the teacher image.

The following second error label correspondence process can also be adopted as the error label correspondence process. In the second error label correspondence process according to the above case CASE1, the generation unit 70 does not set the input image II[J _B1 +2] including the object images OI[39] and [40], which are the object images with the error labels, as the learning image. That is, the image data of the input image II[J _B1 +2] is excluded from the learning dataset (not included in the learning dataset). When the second error label correspondence process is adopted, in case CASE1, similarly to the case where the first error label correspondence process is adopted, the input images II[J _B1 ] and II[J _B2 ] are set as two learning images, and the image data of the input images II[J _B1 ] and II[J _B1 +1] and the annotation information of the input images II[J _B1 ] and II[J _B1 +1] in the provisional dataset are included in the learning dataset as they are. In the second error label handling process for CASE 2, the generating unit 70 does not set the input image II[J B1 ] including the object images OI[39] and [40], which are object images with an error label, as a learning image. That is, the image data of the input image II[J _B1 ] is excluded from the learning data set (is not included in the learning data set ₎ . Note that even when the second error label handling process is adopted, the notifications NT1 and NT2 may be performed in CASE 1 and CASE 2 (however, the correct answer request notification is not executed).

The first and second erroneous label response processes have been described with a focus on the evaluation dataset ES2, but similar processes are also performed on other evaluation datasets, including the evaluation dataset ES3.

In this way, the generation unit 70 that employs the first error label correspondence process sets multiple input images (II[1] to II[L]) as multiple learning images, and generates a learning dataset that includes image data of the multiple learning images as the multiple input images (II[1] to II[L]) and annotation information that identifies the position and class of an object in each learning image. At this time, the generation unit 70 that employs the first error label correspondence process extracts input images that include erroneously labeled object images as annotation targets based on the identification results (identification results of error clusters and erroneously labeled object images) of the error identification unit 60, and corrects the annotation information for the annotation targets using information provided from outside for the annotation targets (correct answer information). The corrected annotation information is included in the learning dataset.

In response to this, the generation unit 70 that employs the second error label correspondence process extracts a portion of the multiple input images (II[1] to II[L]) as multiple training images, and generates a training dataset that includes image data of the multiple training images obtained by the extraction and annotation information that identifies the position and class of objects in each training image. At this time, the generation unit 70 that employs the second error label correspondence process excludes input images that include erroneously labeled object images from the multiple training images based on the identification results of the error identification unit 60 (identification results of error clusters and erroneously labeled object images). For example, if there are input images II[1] to II[10000], and none of the input images II[1] to II[9800] contain an erroneously labeled object image, and each of the input images II[9801] to II[10000] contains an erroneously labeled object image, then only the input images II[1] to II[9800] are extracted as a total of 9,800 training images, and a training dataset containing image data of the 9,800 training images is generated.

Whether the first or second label correspondence process is adopted, it is possible to generate a high-quality training dataset, i.e., a training dataset with few errors in the annotation information (ideally no errors). By improving the quality of the training dataset, it is possible to improve the performance of an inference model for image recognition constructed by machine learning using the training dataset. When the second label correspondence process is adopted, there is an advantage that the manual input of correct answer information is not required. On the other hand, when the first label correspondence process is adopted, there is an advantage that the number of training images can be increased compared to when the second label correspondence process is adopted.

<<Fourth Example>>
A fourth embodiment will now be described. Fig. 13 shows a partial flowchart of the operation of the data processing device 1. It is assumed that the image classification dataset acquisition process, the machine learning process, and the feature derivation process have been executed before reaching step S11. However, the feature derivation process can be executed at any time before reaching step S12 described later.

In the operation of FIG. 13, in order to perform clustering processing on an evaluation dataset basis, first, in step S11, the clustering processing unit 40 sets an evaluation dataset including image data of m object images and class information of the m object images. The m object images are object images cut out from M input images in the image classification dataset. M is an integer equal to or greater than 1.

m is an integer of 3 or greater. However, m may be an integer of 4 or greater, an integer of 5 or greater, or an integer even greater than this. m may be a fixed value m _CNST (e.g., "m = m _CNST = 10). When "m = m _CNST ", the value of M is indefinite, and the evaluation dataset is set by extracting M input images including the fixed value m _CNST number of object images from the image classification dataset. Instead of "m = m _CNST ", "M = M _CNST " may be used. M _CNST is a fixed value and is basically an integer of 2 or more, but may be 1. When "M = M CNST ", the total number of object regions (bounding boxes) set in M _CNST input images is the value of m, so the value of _{m may vary each time the evaluation dataset is set. While "M = M CNST "} is the general rule, if the total number of object regions (bounding boxes) set in M input images is less than the required number (at least 3), the value of M may be increased until the total number becomes equal to or greater than the required number.

When the evaluation data set is set in step S11, the processes of steps S12 to S15 or steps S12 to S16 are sequentially performed on the set evaluation data set. That is, in step S12, the clustering processing unit 40 performs clustering of the m object images in the evaluation data set based on the feature amounts and class information of the m object images, thereby classifying the m object images into one or more clusters. In the following step S13, the index derivation unit 50 derives the center of gravity, distance average, and distance variance of the feature amounts for each cluster. Then, in step S14, the error identification unit 60 performs a process for identifying an error cluster based on the distance average and distance variance. In step S15, it is checked whether an error cluster has been identified by this process. If an error cluster has been identified (Y in step S15), the process of step S16 is performed and then the process proceeds to step S17, but if an error cluster has not been identified (N in step S15), the process proceeds directly to step S17 without going through the process of step S16. In step S16, the error identification unit 60 identifies object images with erroneous labels through reclustering of the error clusters.

In step S17, the data processing device 1 (e.g., the clustering processing unit 40) determines whether or not the evaluation of all object images included in the image classification dataset has been completed. If all object images included in the image classification dataset have been included in any one of the evaluation datasets and the processing of steps S12 to S15 or the processing of steps S12 to S16 has been performed, it is determined that the evaluation has been completed (Y in step S17) and the process proceeds to step S18.
If one or more object images included in the image classification dataset have not yet been included in the evaluation dataset, the process returns to step S11 and repeats the processes from step S11 onward.

In step S18, the training dataset generation unit 70 generates a training dataset based on the provisional dataset, while performing the erroneous label handling process described in the third embodiment as necessary. The generated training dataset is stored in the database 80.

<<Fifth Example>>
A fifth embodiment will now be described. In the fifth embodiment, applied techniques, modified techniques, or supplementary matters to the above-mentioned matters will be described.

In the above embodiment, a specific example was given in which the number of erroneously labeled object images belonging to an error cluster was two (see Figures 7 to 12), but the number of erroneously labeled object images belonging to one error cluster could be one, or three or more.

The data processing device 1 includes, as hardware, a central processing unit (CPU), a graphics processing unit (GPU), a read only memory (ROM), and a random access memory (RAM), which are arithmetic processing devices. The data processing device 1 may realize the functions of each part shown in FIG. 1 by executing, in the CPU, a program stored in the ROM or a program acquired from another device through communication, and thus may realize each process of steps S11 to S18 in FIG. 13.

An inference model for image recognition can be constructed by machine learning any neural network using the learning dataset generated by the data processing device 1. For example, a learning device including a DNN (Deep Neural Network) may be formed in the in-vehicle device, and the learning dataset may be used as training data to cause the learning device of the in-vehicle device to perform machine learning. After machine learning, an inference model capable of object detection is formed by the learning device of the in-vehicle device. Then, object detection is performed by the inference model in the in-vehicle device, and the results of the object detection may be used for autonomous driving or driving assistance that may be implemented in the vehicle. The in-vehicle device here refers to a device installed in a vehicle such as an automobile.

The data processing device 1 itself may be an in-vehicle device. Depending on the vehicle (e.g., a broadcast relay vehicle), an in-vehicle device with abundant computing resources may be installed, and in this case, it is particularly possible for the data processing device 1 itself to be an in-vehicle device.

Part or all of the processing executed by the data processing device 1 may be realized by a mixture of software and hardware. A computer program that causes a computer to execute the above-mentioned method and a computer-readable recording medium on which the program is recorded are included in the scope of this embodiment. Here, computer-readable recording media are, for example, flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, large-capacity DVDs, next-generation DVDs, and semiconductor memories.

The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above embodiments are merely examples of the present invention, and the meanings of the terms of the present invention or each of the constituent elements are not limited to those described in the above embodiments. The specific numerical values shown in the above description are merely examples, and can, of course, be changed to various numerical values.

REFERENCE SIGNS LIST 1 Data processing device 10 Image classification dataset acquisition unit 20 Machine learning processing unit 21 Neural network 30 Inference unit 31 Feature amount derivation unit 40 Clustering processing unit 50 Index derivation unit 60 Error identification unit 70 Learning dataset generation unit 80 Database

Claims (7)

A data processing device having a calculation processing unit,
The arithmetic processing unit is
creating an inference model for object classification by performing machine learning on a neural network using an image classification dataset including a plurality of object images, each of which includes image data of an object, and class information that identifies a class of an object in each of the object images;
Derive features of each object image by inputting the image classification dataset into the inference model ;
performing clustering to classify the m object images into k clusters based on feature amounts of the m object images included in the plurality of object images and the class information for the m object images;
deriving, for each cluster obtained by the clustering, a center of gravity of a plurality of feature points belonging to the cluster and a distance between the center of gravity and each feature amount, and deriving an average and a variance of the distances derived for the plurality of feature amounts;
identifying a cluster to which an object image having an error in the class information belongs as an error cluster based on the derived results of the mean and the variance ;
performing reclustering of the n object images belonging to the erroneous cluster based on the n feature amounts for the n object images, and identifying an object image having an error in the class information from among the n object images based on a result of the reclustering;
m represents an integer of 3 or more, n represents an integer of 3 or more,
k is equal to the number of object classes in the m object images represented by the class information for the m object images.
, data processing device. 2 . The data processing apparatus according to claim 1 , wherein the calculation processing unit identifies, as the erroneous cluster, a cluster whose derived mean is equal to or greater than a predetermined first threshold and whose derived variance is equal to or greater than a predetermined second threshold. 3. The data processing device according to claim 1, wherein the calculation processing unit classifies the n object images into first and second clusters by the reclustering, and when the total number of object images belonging to the second cluster is smaller than the total number of object images belonging to the first cluster, identifies the object image belonging to the second cluster as an object image having an error in the class information. 3. The data processing device according to claim 1, wherein the arithmetic processing unit classifies the n object images into first and second clusters by the reclustering, and when a distance between the center of gravity of the n feature amounts for the n object images and the second cluster is greater than a distance between the center of gravity of the n feature amounts for the n object images and the first cluster, identifies an object image belonging to the second cluster as an object image having an error in the class information. the plurality of object images are images cut out from a plurality of input images,
the arithmetic processing unit uses the plurality of input images as a plurality of learning images, and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies a position and a class of an object in each learning image ;
The data processing device according to any one of claims 1 to 4, wherein the calculation processing unit extracts an input image including an object image having an error in the class information as an annotation target from the multiple input images based on the result of identifying the error cluster, and corrects the annotation information for the annotation target using information provided from outside for the annotation target. the plurality of object images are images cut out from a plurality of input images,
the arithmetic processing unit extracts a portion of the plurality of input images as a plurality of learning images, and generates a learning dataset including image data of the plurality of learning images and annotation information that identifies a position and a class of an object in each learning image ;
5. The data processing device according to claim 1 , wherein the arithmetic processing unit excludes, from the plurality of learning images, input images including object images having erroneous class information, based on the result of identifying the erroneous clusters . a feature derivation step of creating an inference model for object classification by executing machine learning of a neural network using an image classification dataset including a plurality of object images each including image data of an object and class information identifying a class of an object in each object image, and deriving features of each object image by inputting the image classification dataset to the inference model ;
a clustering processing step of performing clustering to classify the m object images into k clusters based on feature amounts of the m object images included in the plurality of object images and the class information for the m object images;
an index deriving step of deriving, for each cluster obtained by the clustering, a center of gravity of a plurality of feature points belonging to the cluster and a distance between the center of gravity and each feature amount, and deriving an average and a variance of the distances derived for the plurality of feature amounts;
and an error identifying step of identifying, as an error cluster, a cluster to which an object image having an error in the class information belongs, based on the derived results of the mean and the variance,
In the error identifying step, reclustering is performed on the n object images belonging to the error cluster based on the n feature amounts for the n object images, and an object image having an error in the class information is identified from among the n object images based on a result of the reclustering;
m represents an integer of 3 or more, n represents an integer of 3 or more,
k is equal to the number of object classes in the m object images represented by the class information for the m object images.
,Data Processing Methods.