JP7580178B2 - Face recognition learning data generation device and program - Google Patents

Description

The present invention relates to a face recognition training data generating device and a program for generating training data for face recognition.

In recent years, the technology to recognize and identify people in images and videos has become more accurate with the advancement of AI technology, and has reached the level of practical use in the security field. In addition, efforts to use face recognition technology to add metadata such as keywords to videos are spreading, mainly among broadcasting stations.
These facial recognition technologies are based on a mechanism in which face images for learning are collected, facial features that can recognize differences between people are calculated through machine learning processing, and faces are recognized based on the similarity of the facial features (see Patent Document 1).
The performance of this face recognition process is greatly affected by the quantity and quality of the training data used in the training process. The method of generating this training data is generally to collect face images through many manual processes.

On the other hand, as a method for collecting face images, a technology has been disclosed in which a CG model of a 3D face image is generated from collected 2D face images, and CG parameters are manipulated to generate more face images (see Patent Document 2). However, this method requires precise tuning of the CG parameters in order to generate a CG model of an appropriate face image.
In addition, a commonly used practical method is to collect face images using web scraping technology.

JP 2017-33372 A Special table 2016-501396 publication

As described above, most of the methods for collecting face images involve manual labor, which is problematic in that it is time-consuming. Even if a method for generating face images is used, such as the invention described in Patent Document 2, the generation method requires precise tuning of CG parameters, which is problematic in that it is time-consuming.
Furthermore, the conventional method is a method for collecting face images, and in order to use the collected face images as learning data for face recognition, it is necessary to add metadata such as person names to the images.

However, with conventional methods, adding such metadata must be done manually after collecting face images, which is time-consuming and inefficient.
Furthermore, in face recognition technology, in order to utilize face recognition results from multiple angles, it is desirable to associate a face image with category information to which the person belongs.

The present invention has been made in consideration of such problems, and has an objective of providing a face recognition learning data generation device and a program therefor that are capable of collecting face images and generating learning data for face recognition with metadata attached, without any human effort.

To solve the above problems, the face recognition learning data generating device according to the present invention is a face recognition learning data generating device that generates face recognition learning data from a data set that associates keywords including person names with hierarchical categories, and is configured to include a person keyword extraction means, a category information extraction means, a collection means, a face image extraction means, a feature extraction means, a relevance setting means, a clustering means, a determination means, and a data integration means.

In such a configuration, the face recognition learning data generation device extracts keywords for person names from the data set using the person keyword extraction means.
In addition, the face recognition learning data generation device extracts, as category information, hierarchical categories corresponding to keywords that identify people from the data set, by the category information extraction means, thereby associating the person's name with the hierarchical category.

The face recognition learning data generating device then collects documents related to the keywords that identify a person using a collection means. For example, the collection means searches for documents related to the keywords on the Web. These collected documents (Web documents) often contain images related to the names of people.
Therefore, the face recognition training data generating device extracts a face image from an image included in a document by face recognition using a face image extraction means, and then extracts features from the face image using a feature extraction means.

Then, the face recognition learning data generation device uses the relevance setting means to set the relevance between the feature and the person identified by the keyword in correspondence with the feature, based on a document including the face image from which the feature was extracted.
In addition, the face recognition learning data generating device clusters the feature amounts of the face images using a clustering means.
Then, the face recognition learning data generating device determines, by a determination means, whether or not each class corresponds to an identified person based on the relevance set for the features belonging to each clustered class.

Then, the face recognition learning data generation device uses a data integration means to integrate, for each keyword that identifies a person, features belonging to a class determined to correspond to the identified person with category information to generate face recognition learning data.
In this way, a keyword for one person's name is associated with the facial features of the person and the category information to which the person belongs.
This face recognition learning data generating device can be operated by a face recognition learning data generating program for causing a computer to function as each of the means described above.

The present invention provides the following excellent effects.
According to the face recognition training data generating device of the present invention, it is possible to collect a large amount of face images without manual effort and generate training data for face recognition. In addition, according to the present invention, since hierarchical category information can be added to each training data, it is possible to organize the training data into a database with the category information as metadata .

1 is a block diagram showing a configuration of a face recognition learning data generating device according to an embodiment of the present invention. FIG. 2 is a data structure diagram of a keyword table showing an example of a correspondence relationship between IDs (identifiers) and keywords in the data structure of a data set. FIG. 2 is a data structure diagram of a category table showing an example of a correspondence relationship between IDs (identifiers) and category IDs in the data structure of a data set. FIG. 11 is a data structure diagram of a hierarchical table showing an example of a correspondence relationship between an ID (identifier) and a higher hierarchical ID in the data structure of a data set. FIG. 2 is an explanatory diagram for explaining an example of a hierarchical structure of categories. FIG. 11 is an explanatory diagram for explaining the relationship between person KWs and category information. FIG. 1 is a network diagram showing an example of the configuration of a neural network model that extracts facial features from a facial image. FIG. 1 is an explanatory diagram for explaining a concept of clustering of facial feature amounts. 1 is a data structure diagram showing an example of a structure of learning data for face recognition. 4 is a flowchart showing the operation of the face recognition learning data generating device according to the embodiment of the present invention. 1 is a block diagram showing a configuration of a face recognition device according to an embodiment of the present invention. 4 is a flowchart showing the operation of the face recognition device according to the embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[Configuration of the face recognition learning data generation device]
First, with reference to FIG. 1, a configuration of a face recognition learning data generating device 1 according to an embodiment of the present invention will be described.
The face recognition learning data generating device 1 generates face recognition learning data.
As shown in FIG. 1, the face recognition learning data generating device 1 includes a data analysis unit 10, a face image collection unit 11, a person-KW correspondence determination unit 12, and a data integration unit 13.

The data analysis means 10 analyzes the data set and extracts person name keywords (hereinafter, person KWs) and keywords for categories (attributes) to which the person belongs (hereinafter, category KWs) from among keywords that include person names.

The dataset is a large-scale data set that associates keywords, including people's names, with hierarchical categories. Any dataset may be used as long as the data structure is known, but here, the Wikipedia data will be used as an example.
First, an example of the data structure of a data set will be described with reference to FIGS.

2 is a data structure diagram of a keyword table Tk showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. An ID is an identifier such as a number that is individually assigned to a keyword. A keyword is a character string of various entry words including a person's name, and is a page entry word of Wikipedia. In the example of FIG. 2, the keyword "Japanese politician" is associated with the ID "id ₁ ", the keyword "XXXX Taro" is associated with the ID "id ₆ ", etc.
As a result, various keywords including a person's name are uniquely identified by the ID.

3 is a data structure diagram of a category table Tc showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a data set. The category IDs are a list of IDs of categories to which keywords belong. In the example of FIG. 3, the category IDs "id ₁ , id ₂ , id ₃ , id ₄ , ..., id ₈ , ..." are associated with the ID "id ₆ ". This means that the keyword "〇〇 Taro" corresponding to the ID "id ₆ " belongs to the keyword (category KW) "Japanese politicians" corresponding to the ID "id ₁ ", the keyword (category KW) "politicians of each prefecture" corresponding to the ID "id ₂ ", etc.
This identifies which category the keyword belongs to.

FIG. 4 is a data structure diagram of a hierarchical table Tu showing an example of the correspondence between IDs (identifiers) of category keywords and higher-level IDs in the data structure of a data set. The higher-level IDs are a list of IDs in a higher level in a hierarchical structure for a certain ID. In the example of FIG. ₄ , IDs "id ₁ , id ₂ , id ₃ " are associated with ID "id ₄ " as higher-level IDs. This means that for the category keyword "Diet members elected from Tokyo" corresponding to ID "id _{4 ", the keyword "Japanese politicians" corresponding to ID "id 1} ", the keyword "politicians of each prefecture" corresponding to ID "id ₂ ", and the keyword "Tokyo politicians" corresponding to ID "id ₃ " are in the higher level as categories.
This allows the keywords to be organized into a hierarchy.
When the data set is Wikipedia data, there is also a table in which an explanation for a keyword is associated with an ID, but this is not shown here.

Returning to FIG. 1, the configuration of the face recognition learning data generating device 1 will be further described.
The data analysis means 10 includes a category hierarchy means 100, a person KW extraction means 101, and a category information extraction means 102.

The category hierarchy means 100 generates category hierarchy information by associating category IDs with position information representing hierarchical categories from a data set.
Specifically, the category hierarchical division means 100 refers to the hierarchy table Tu in Fig. 4, and sequentially and recursively associates each ID (identifier) with a higher hierarchy ID, thereby generating a tree structure graph as shown in Fig. 5. Here, an example of a tree structure in which the category KW "Japanese politicians" is the highest hierarchy will be described.
In addition, in Wikipedia, the hierarchy of categories is structured as a directed acyclic type, and since there is a possibility that the same category keyword may appear in the recursive process, the maximum number of recursions is determined in advance and the process is stopped.

Then, the category hierarchy means 100 generates category hierarchy information by associating the IDs of the category KWs with the position information of each category KW configured in the tree-structured graph. This position information is information indicating the position of a node (corresponding to a category KW) in the tree-structured graph, and can be expressed in, for example, a tuple format. In the tuple format of the position information, each number in the order of the numeric sequence represents the position of the node to be traced for each level of the tree-structured graph.

For example, suppose the position information of a certain node is (0, 2, 25, 11, 2). Each value indicates the position of the child node from the left of the parent node, starting from the top layer. The leftmost value in the position information is '0' to indicate the topmost (first layer) node (root node). The second value in the position information, '2', indicates the second child node from the left of the parent node in the layer (second layer) below the root node.
For example, in FIG. 5, the location information of the category KW "Japanese politicians" of id ₁ is (0), and the location information of the category KW "Tokyo politicians" of id ₃ is (0, 1, 1).
The category hierarchy generating means 100 outputs the generated category hierarchy information (ID and location information) to the category information extracting means 102 .

The person KW extraction means (person keyword extraction means) 101 extracts person name keywords (person KWs) from the data set. Here, the person KW extraction means 101 extracts keywords corresponding to categories including predetermined character strings that identify a person from the data set as person KWs.
Specifically, the person KW extraction means 101 searches for keywords that include predetermined words that can be used to identify a person, such as "year of birth,""year of death," and "living person," and extracts keywords that include an ID corresponding to the keyword as a category ID as a person KW.

For example, the person KW extraction means 101 searches for IDs containing the keyword phrase "grade" in the keyword table Tk of Fig. 2. Then, the person KW extraction means 101 further searches for IDs containing the searched ID (id ₈ in this case) as a category ID in the category table Tc of Fig. 3. Then, the person KW extraction means 101 extracts "XXXX Taro" corresponding to the searched ID (id ₆ in this case) as a person KW in the keyword table Tk of Fig. 2.
In addition, in Wikipedia, the process of identifying keywords that include in the category any of the phrases "birth year,""death," and "living person" as person KWs has been experimentally confirmed to be 100% accurate.

Of course, a keyword containing a phrase that identifies a person in the person name may be used as a keyword for a person in a data set. For example, a keyword containing a predetermined phrase "person" such as "XXXX Taro (person)" may be used. In this case, the person KW extraction means 101 may extract a keyword containing the phrase "person" from the keyword table Tk as a person KW.
The person KW extraction means 101 outputs the extracted person KWs to the category information extraction means 102 and the face image collection means 11 .

The category information extraction means 102 extracts, as category information, hierarchical categories corresponding to keywords of person names from the data set, based on the category hierarchical information hierarchized by the category hierarchization means 100 .
The category information extraction means 102 extracts upper IDs in the tree structure from the category hierarchical information using as a reference a category ID corresponding to the person KW extracted by the person KW extraction means 101. Then, the category information extraction means 102 generates keywords corresponding to the reference category ID and the upper category ID as category information.

For example, even if only "id ₅ " is set as the category ID for a certain person keyword "△△Hanako" in the category table Tc (Figure 3), the category information extraction means 102 generates, as category information, keywords corresponding to the ID "id ₅ " and the IDs "id ₁ , id ₂ , id ₃ , id ₄ " that are hierarchically ranked higher in the category hierarchy information, as shown in Figure 6.
This allows the category information extraction means 102 to associate not only category KWs directly corresponding to the person KWs, but also category KWs higher up in the hierarchical structure with the person KWs.
The category information extraction means 102 associates the person KW with the category information and outputs it to the data integration means 13 .

Note that tree-structured position information may be added to each keyword in the category information. This makes it possible to create a hierarchical database of face recognition training data.

The face image collecting means 11 collects face images corresponding to persons KW.
The face image collecting means 11 includes a web search means 110 and a face image extracting means 111 .

The Web search means 110 searches for and collects documents (Web documents) related to keywords related to person names on the Web, which is a system for publishing and viewing documents on a network (Internet) N.
The web search means 110 searches and collects web documents related to the person KWs extracted by the person KW extraction means 101 of the data analysis means 10 from the web server S by a general search engine via the network N. Note that the search for person KWs by the search engine may use a general web scraping technique.
The Web documents collected by the Web search means 110 include images including the face of the person KW, images other than the face, text data, and the like.
The web search means 110 outputs the collected web documents to the face image extraction means 111 .

The facial image extraction means 111 extracts facial images from images contained in Web documents collected by the Web search means 110 by facial recognition.
The face image extraction means 111 detects faces in images contained in a Web document by general face recognition processing, and extracts only images in which one face is shown.
The face image extraction means 111 outputs the extracted face image and the Web document from which the face image was extracted, together with the person KW, to the person KW correspondence determination means 12 .

The person KW correspondence determination means (person keyword correspondence determination means) 12 determines whether or not the face image collected by the face image collection means 11 is a face image of a person corresponding to the person KW.
The person KW correspondence determination means 12 includes a feature amount extraction means 120 , a clustering means 121 , a relevance setting means 122 , and a determination means 123 .

The feature extraction means 120 extracts features (facial feature amounts) from the face images collected by the face image collection means 11 .
For example, the feature extraction means 120 extracts facial feature values F by performing multiple convolution operations on a facial image I using a pre-trained convolutional neural network NN equipped with multiple convolution layers CL (CL ₁ , CL ₂ , ...) as shown in Fig. 7. This facial feature value F has a dimensionality of, for example, about 500.
This convolutional neural network NN can use, for example, a network for extracting features of a general object recognition network such as VGG (Visual Geometry Group) 19.
The feature extraction means 120 outputs facial feature amounts {f ₁ , f ₂ , . . . , f _n } (n is the number of facial images) extracted from a plurality of facial images corresponding to the person KW to the clustering means 121 for each person KW.

The clustering means 121 clusters the facial features extracted by the feature extraction means 120 .
A general method may be used as the clustering method in the clustering means 121. For example, the clustering means 121 clusters the facial feature amounts {f ₁ , f ₂ , ..., f _n } of the number of face images using a DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, which can also calculate the number of classes at the same time.

Fig. 8 shows the concept of facial feature amounts clustered by the clustering means 121. Fig. 8 shows a state in which facial feature amounts { _f1 , _f2 , ..., _f9 } are classified into three classes ( _C1 , _C2 , _C3 ), with facial feature amounts { _f1 , _f4 , _f5 , _f9 } classified into class _C1 , facial feature amounts { _f6 , _f8 } classified into class _C2 , and facial feature amounts { _f2 , _f3 , _f7 } classified into class _C3 .
This enables the clustering means 121 to divide a plurality of facial features into a plurality of groups, such as a group of persons corresponding to the person KW and other groups, or even if the persons correspond to the same person KW, groups of faces facing forward and faces facing sideways, etc.
The clustering means 121 outputs the clustering result to the determination means 123 .

The relevance setting unit 122 sets a relevance indicating a degree of relevance between the facial feature amount and a person's name in association with the facial feature amount based on a Web document including a face image from which the facial feature amount has been extracted.
The relevance setting unit 122 sets the relevance based on a predetermined criterion. Here, the relevance setting unit 122 increases the relevance based on the criterion from a predetermined initial value (e.g., “0”).

For example, when the Web document from which the face image has been extracted is a Wikipedia document that has a person keyword as an entry word, the relevance setting unit 122 adds a predetermined number to the relevance.
Also, for example, when a tag indicating an image in a Web document contains a description of a person keyword, the relevance setting unit 122 adds a predetermined number to the relevance.
Furthermore, for example, when a Web document is similar to a description that describes a person keyword included in the data set, the relevance setting unit 122 adds a predetermined number to the relevance. This similarity can be determined, for example, when the number of words common to the Web document and the description is greater than a predetermined number, or when the ratio of the number of common words to the total number of words in the Web document and the description is greater than a predetermined ratio.
Furthermore, when Web documents are categorized in advance, such as in Wikipedia, it is possible to update the relevance by calculating the degree of overlap with the category information of the person keyword and adding it to the relevance.
In the above relevance calculations, it is also possible to obtain the relevance by vector converting the words and image features in the sentences, which are the elements for calculating the relevance, into a common dimensional space using WORD2VEC, PCA (Principal Component Analysis), etc., and then performing a similarity calculation process such as cosine similarity.
The relevance setting means 122 outputs the relevance W{w ₁ , w ₂ , . . . , w _n } (n is the number of face images) set for a plurality of face feature amounts corresponding to the person KW to the determination means 123 for each person KW.

The determining means 123 determines whether or not each class corresponds to a person's name based on the degree of association set for the facial feature amounts belonging to each class clustered by the clustering means 121 .
This determination means 123 calculates the sum of the relevance set by the relevance setting means 122 for the facial features belonging to each clustered class, and determines that a class whose sum satisfies a predetermined criterion is the class corresponding to the person KW.

For example, suppose that the clustering result for a certain person KW is as shown in Fig. 8. Also suppose that the relevance of the person KW set to the facial feature amount { _f1 , _f2 , ..., _f9 } is { _w1 , _w2 , ..., _w9 }. In this case, the determination means 123 calculates the sum _W1 of the relevance of class _C1 as _w1 + _w4 + _w5 + _w9 , the sum _W2 of the relevance of class _C2 as _w6 + _w8 , and the sum _W3 of the relevance of class _C3 as _w2 + _w3 + _w7 .

In addition, the judgment means 123 judges whether each class corresponds to a person KW based on, for example, the sum of the relevance degrees for each class exceeds a predetermined threshold value, or the difference between the sum of the relevance degrees for each class and the sums of the other classes is greater than a predetermined threshold value.
The determination means 123 associates the person KW with the face feature amount determined to be the class of the person KW, and outputs the result to the data integration means 13 .

The data integration means 13 integrates category information corresponding to the person KW generated by the data analysis means 10 with facial features determined to correspond to the person KW by the person KW correspondence determination means 12 to generate learning data for face recognition.
9, the data integration means 13 generates face recognition learning data by linking the facial feature amount F and the category information CI for each person KW. Note that position information in a tree structure may be added to each keyword in the category information CI.
Furthermore, the data integration means 13 may add a face image from which facial features have been extracted to the face recognition learning data in place of or together with the facial features.

By configuring as described above, the face recognition learning data generation device 1 can collect face images without any human effort and generate learning data for face recognition to which person KWs and category information have been added.
Furthermore, the face recognition training data generation device 1 can collect face images in a wide variety of categories and also collect face images in hierarchical categories, thereby enabling the face recognition training data generation device 1 to generate comprehensive and systematically organized training data.
The face recognition learning data generating device 1 can be operated by a program (face recognition learning data generating program) for causing a computer to function as each of the means described above.

[Operation of the Face Recognition Learning Data Generation Device]
Next, the operation of the face recognition learning data generating device 1 according to an embodiment of the present invention will be described with reference to FIG. 10 (for the configuration, refer to FIG. 1 as appropriate).
In step S1, the category hierarchical division means 100 of the data analysis means 10 analyzes the data set and divides categories into a hierarchy. Here, the category hierarchical division means 100 refers to a hierarchy table Tu (FIG. 4), sequentially and recursively associates a higher hierarchy ID with each ID (identifier), thereby dividing the categories into a hierarchy, and generates category hierarchy information (FIG. 5) in which the IDs and position information are associated with the nodes of a tree-structure graph.

In step S2, the person KW extraction means 101 of the data analysis means 10 analyzes the dataset and extracts person KWs from the keywords in the dataset. Here, the person KW extraction means 101 extracts, as person KWs, keywords in the keyword table Tk (Figure 2) whose category table Tc (Figure 3) contains an ID that includes any of the words "birth date," "death," and "living person."

In step S3, the category information extraction means 102 of the data analysis means 10 extracts category KWs corresponding to the person KWs extracted in step S2 from the dataset based on the category hierarchy information generated in step S1, and generates category information of the person KWs (Figure 6).

In step S4, the web search means 110 of the face image collection means 11 collects web documents corresponding to the person KW extracted in step S2 from the web server S via the network N. Here, the web search means 110 collects web documents by web scraping using the person KW.

In step S5, the facial image extraction means 111 of the facial image collection means 11 extracts facial images from the images included in the Web documents collected in step S4. Here, the facial image extraction means 111 detects faces from the Web documents using facial recognition, and extracts only images that show one person's face.

In step S6, the feature extraction means 120 of the person KW correspondence determination means 12 extracts features (facial features) from the face image extracted in step S5. Here, the feature extraction means 120 extracts facial features from the face image using a pre-trained convolutional neural network.
In step S7, the clustering means 121 of the person KW correspondence determination means 12 clusters the face feature amounts extracted in step S6.

In step S8, the relevance setting means 122 of the person KW correspondence determination means 12 sets the relevance between the facial feature extracted in step S6 and the person KW. For example, the relevance setting means 122 sets a higher relevance value the stronger the relevance between the Web document and the person KW, such as when the Web document is a Wikipedia document that uses the person KW as a heading word.

In step S9, the determination means 123 of the person KW correspondence determination means 12 determines, for each class clustered in step S7, whether or not the class corresponds to a person KW, based on the relevance set in step S8. Here, the determination means 123 calculates the sum of the relevance for the facial features included in the class, and determines that a class whose sum of the relevance is greater than a predetermined threshold is a class corresponding to a person KW.

In step S10, the data integration means 13 integrates, for each person KW, the category information generated in step S3 corresponding to the person KW and the facial feature values belonging to the class determined in step S9 to be the class of the person KW, to generate learning data for face recognition ( FIG. 9 ).
Through the above operations, the face recognition learning data generating device 1 can automatically collect face images corresponding to person KWs and generate face recognition learning data.

[Configuration of face recognition device]
Next, the configuration of the face recognition device 2 according to the embodiment of the present invention will be described with reference to FIG.
The face recognition device 2 recognizes the faces of people appearing in the video.
As shown in FIG. 11, the face recognition device 2 includes a face recognition learning data storage means 20, a face detection means 21, a feature extraction means 22, a similarity calculation means 23, and a person identification means 24.

The face recognition learning data storage means 20 stores face recognition learning data, and can be configured with a general storage medium such as a hard disk.
The face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generating device 1 described with reference to FIG.
The face recognition learning data is a database in which a person KW is associated with a face feature amount F and category information CI, as shown in FIG.

The face detection means 21 detects faces appearing in the input image. The face detection means 21 recognizes faces in the image by using a general face recognition process.
The face detection means 21 extracts the detected face area (face image) from the video for each frame, and outputs it to the feature extraction means 22 .

The feature extraction means 22 extracts facial features from the face image detected by the face detection means 21. The feature extraction means 22 extracts facial features using the same method as the feature extraction means 120 of the face recognition learning data generation device 1 described in FIG.
The feature extraction means 22 outputs the extracted one or more facial feature amounts to the similarity calculation means 23 .

The similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted by the feature extraction means 22 and the facial features stored in the face recognition learning data storage means 20 .
Here, the similarity calculation means 23 calculates the similarity for facial features stored in the face recognition learning data storage means 20 that correspond to category information (keywords) specified from outside.
For example, when the keyword “politician in Tokyo” is specified as the category information, the similarity calculation means 23 calculates the similarity only from the facial features of the learning data for face recognition stored in the face recognition learning data storage means 20, the category information of which includes the keyword “politician in Tokyo.”

The similarity calculation method used by the similarity calculation means 23 may be a general method, and for example, the similarity can be calculated using the cosine similarity between the vector of facial features extracted by the feature extraction means 22 and the vector of facial features stored in the face recognition learning data storage means 20.
The similarity calculation means 23 outputs the calculated similarity and the person KW of the face recognition learning data used to calculate the similarity to the person identification means 24.

The person identification means 24 identifies a person based on the similarity calculated by the similarity calculation means 23 .
The person identification means 24 extracts those whose similarities calculated by the similarity calculation means 23 are greater than a predetermined threshold. The person identification means 24 then sorts the extracted similarities and outputs the person KWs of the face recognition learning data used to calculate the similarities as recognition results in descending order of similarity. Of course, for simplicity, it is also possible to output only the person KW with the highest similarity.

In addition, the person identification means 24 may output a keyword of the specified category information in association with the person KW, or keywords of all category information associated with the person KW stored in the face recognition learning data storage means 20.
This makes it possible to add metadata based on category information to a person KW recognized by the face recognition device 2.
In addition, when tree-structured location information is added to the category information of the face recognition learning data stored in the face recognition learning data storage means 20, the person identification means 24 may output the location information in addition to the category information in association with the person KW.

With the above-described configuration, the face recognition device 2 can perform face recognition by specifying category information, thereby improving the accuracy of face recognition.
The face recognition device 2 can be operated by a program (face recognition program) for causing a computer to function as each of the above-mentioned means.

[Facial Recognition Device Operation]
Next, the operation of the face recognition device 2 according to the embodiment of the present invention will be described with reference to Fig. 12 (for the configuration, refer to Fig. 11 as appropriate). It is assumed that the face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generation device 1 (Fig. 1).

In step S20, the face detection means 21 detects a face appearing in the input video image, and extracts the area of the detected face as a face image.
In step S21, the feature extraction means 22 extracts features (facial feature amounts) from the face area (facial image) detected in step S20.

In step S22, the similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted in step S21 and the facial features stored in the face recognition learning data storage means 20 and corresponding to externally specified category information (keywords).
In step S23, the person identification means 24 identifies a person based on the similarity calculated in step S22. At this time, the person identification means 24 extracts and sorts those having a similarity greater than a predetermined threshold, and outputs the person KWs with the highest similarity as the recognition result.
Through the above operations, the face recognition device 2 can limit the categories and recognize faces from the video.

Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment.
In this embodiment, the dataset has keywords including a person's name, and the target person is identified using the keyword of the person's name. However, the keyword does not have to be a person's name as long as it can identify the target person. For example, the keyword may be a number representing a person. For example, it may be an index such as a My Number that can identify a person. Furthermore, the text of the dataset itself may have a person's name replaced with another index so that the individual cannot be identified.

In addition, in this embodiment, documents containing facial images are described as being searched and collected on the Web. However, documents containing facial images are not limited to those searched and collected on the Web, and can be anything that contains a facial image and document data related to that face. For example, documents containing facial images can be documents that have been saved in advance on a storage medium, or documents that are read from an external source. Therefore, the Web search means 110 is not limited to searches on the Web, and can be configured as a collection means. Note that the documents in this case are multimedia documents that include text, images, and audio.

In addition, in this embodiment, the determination means 123 is described as calculating the sum of the relevance for each clustered class and making a determination based on the calculated value. However, the index used for the determination is not limited to the sum of the relevance for each class. The index used for the determination may be any index that can be determined to correspond to a person KW. For example, the index may be the average relevance of the class, the maximum value of the facial feature value belonging to the class, etc.

REFERENCE SIGNS LIST 1 Face recognition learning data generating device 10 Data analysis means 100 Category hierarchical means 101 Person KW extraction means 102 Category information extraction means 11 Face image collection means 110 Web search means (collection means)
REFERENCE SIGNS LIST 111 Facial image extraction means 12 Person-KW correspondence determination means 120 Feature extraction means 121 Clustering means 122 Association setting means 123 Determination means 13 Data integration means 2 Face recognition device 20 Face recognition learning data storage means 21 Face detection means 22 Feature extraction means 23 Similarity calculation means 24 Person identification means

Claims (8)

A face recognition learning data generation device that generates face recognition learning data from a data set in which keywords are associated with hierarchical categories, comprising:
A person keyword extraction means for extracting person name keywords from the data set;
a category information extraction means for extracting, from the data set, the hierarchical categories corresponding to keywords that identify a person as category information;
A collection means for collecting documents related to a keyword that identifies the person;
A facial image extraction means for extracting a facial image from an image included in the document by facial recognition;
A feature extraction means for extracting features from the face image;
an association degree setting means for setting an association degree between the feature amount and a person specified by the keyword in association with the feature amount based on the document including the face image from which the feature amount is extracted;
A clustering means for clustering the feature amounts of the face image;
a determination means for determining whether each of the clustered classes corresponds to the specified person based on the relevance level set for the feature amount belonging to each of the clustered classes;
a data integration means for integrating, for each keyword that identifies the person, a feature value belonging to a class determined to correspond to the identified person and the category information to generate the face recognition learning data;
A face recognition learning data generating device comprising: The face recognition learning data generation device according to claim 1, characterized in that the person keyword extraction means extracts keywords corresponding to a category including a predetermined character string that identifies a person as a keyword for the person's name. The face recognition learning data generation device according to claim 1 or 2, characterized in that the category information extraction means adds position information of the hierarchical categories represented by a tree structure graph to the category information. The face recognition learning data generation device according to any one of claims 1 to 3, characterized in that the relevance setting means increases the relevance when the document is a web document in which the person's name is an entry word. The face recognition learning data generating device according to any one of claims 1 to 4, characterized in that the relevance setting means increases the relevance when the document contains a tag indicating the person's name. The dataset includes keywords and explanatory text,
The face recognition learning data generating device according to any one of claims 1 to 5, characterized in that the relevance setting means increases the relevance when the description corresponding to the keyword of the person's name and the document are similar. The face recognition learning data generating device according to any one of claims 1 to 6, characterized in that the data integration means uses a face image from which the feature values have been extracted as the face recognition learning data instead of the feature values. A face recognition learning data generation program for causing a computer to function as a face recognition learning data generation device according to any one of claims 1 to 7.

Images