JP7808918B2 - Facial recognition device and its program - Google Patents

Description

The present invention relates to a face recognition device and a program therefor.

In recent years, advances in AI technology have made it possible to identify people by recognizing their faces in images and videos, and this has led to practical applications in the security field. Furthermore, efforts to use facial recognition technology to add metadata such as keywords to videos are becoming more widespread, particularly among broadcasting stations.
These facial recognition technologies are based on a mechanism in which face images for learning are collected, facial features that can recognize differences between people are calculated through machine learning processing, and faces are recognized based on the similarity of those facial features (see Patent Document 1).
The performance of this facial recognition process is greatly affected by the quantity and quality of the training data used in the training process. The method of generating this training data is usually to manually collect and generate facial images.

On the other hand, as a method for collecting face images, a technology has been disclosed in which a CG model of a 3D face image is generated from collected 2D face images and CG parameters are manipulated to generate a larger number of face images (see Patent Document 2). However, this method requires precise tuning of the CG parameters in order to generate a CG model of an appropriate face image.
In addition, a commonly used method in practice is to collect facial images using web scraping technology.

JP 2017-33372 A Special table 2016-501396 publication

As mentioned above, most of the methods for collecting face images involve manual labor, which is a problem. Even if a method for generating face images is used, such as the invention described in Patent Document 2, the generation method requires precise tuning of CG parameters, which still leaves the problem of being time-consuming.
Furthermore, conventional methods are methods for collecting face images, and in order to use the collected face images as learning data for face recognition, it is necessary to add metadata such as person names to the collected face images.

However, conventional methods have the problem that adding such metadata must be done manually after collecting face images, which is time-consuming and inefficient.
Furthermore, in face recognition technology, in order to utilize face recognition results from multiple angles, it is desirable that face images be associated with category information to which the person belongs.

The present invention was made in consideration of these problems, and aims to provide a face recognition device and a program therefor that can recognize the faces of people captured in video footage by collecting face images and using face recognition training data with metadata attached, without any manual effort.

In order to solve the above problems, the face recognition device of the present invention includes a person keyword extraction means for extracting keywords of person names from a data set in which keywords are associated with hierarchical categories, a category information extraction means for extracting the hierarchical categories corresponding to the keywords that identify people from the data set as category information, a collection means for collecting documents related to the keywords that identify the people, a face image extraction means for extracting face images from images included in the documents by face recognition, a feature extraction means for extracting features from the face images, a relevance setting means for setting a relevance between the features and the people identified by the keywords in association with the features based on the documents including the face images from which the features have been extracted, a clustering means for clustering the features of the face images, and a relevance setting means for setting a relevance between the features and the people identified by the keywords in association with the features. a data integration means for integrating, for each keyword that identifies the person, features belonging to the class determined to correspond to the identified person with the category information to generate face recognition learning data; a face recognition learning data storage means for storing the face recognition learning data; a face detection means for detecting faces shown in a video and extracting face images; a second feature extraction means for extracting features from the face images extracted by the face detection means ; a similarity calculation means for calculating a similarity between a feature of a face image corresponding to specified category information in the face recognition learning data and the feature extracted by the second feature extraction means; and a person identification means for recognizing, as a recognition result, a person name corresponding to a feature whose similarity is greater than a predetermined threshold .

The face recognition device can be operated by a face recognition program that causes a computer to function as each of the above-mentioned means.

The present invention provides the following excellent effects.
According to the face recognition device of the present invention, face recognition can be performed by specifying a category, so that highly accurate face recognition can be performed.

1 is a block diagram showing the configuration of a face recognition learning data generation device according to an embodiment of the present invention; FIG. 10 is a data structure diagram of a keyword table showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. FIG. 10 is a data structure diagram of a category table showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a data set. FIG. 10 is a data structure diagram of a hierarchical table showing an example of the correspondence between IDs (identifiers) and upper hierarchical IDs in the data structure of a data set. FIG. 2 is an explanatory diagram illustrating an example of a hierarchical structure of categories. FIG. 10 is an explanatory diagram for explaining the relationship between person KWs and category information. FIG. 1 is a network diagram showing an example of the configuration of a neural network model that extracts facial features from a facial image. FIG. 10 is an explanatory diagram illustrating the concept of clustering of facial feature amounts. FIG. 2 is a data structure diagram showing an example of the structure of face recognition learning data. 4 is a flowchart showing the operation of the face recognition learning data generation device according to the embodiment of the present invention. 1 is a block diagram showing the configuration of a face recognition device according to an embodiment of the present invention. 4 is a flowchart illustrating an operation of the face recognition device according to the embodiment of the present invention.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[Configuration of the face recognition learning data generation device]
First, with reference to FIG. 1, the configuration of a face recognition learning data generating device 1 according to an embodiment of the present invention will be described.
The face recognition learning data generating device 1 generates face recognition learning data.
As shown in FIG. 1, the face recognition learning data generating device 1 includes a data analysis means 10, a face image collection means 11, a person-KW correspondence determination means 12, and a data integration means 13.

The data analysis means 10 analyzes the dataset and extracts person name keywords (hereinafter referred to as person KWs) and keywords for categories (attributes) to which the person belongs (hereinafter referred to as category KWs) from keywords containing person names.

The dataset is a large-scale data set that associates keywords, including people's names, with hierarchical categories. Note that any dataset can be used as long as the data structure is known, but here we will use Wikipedia data as an example.
First, an example of the data structure of a data set will be described with reference to FIGS.

FIG. 2 is a data structure diagram of a keyword table Tk showing an example of the correspondence between IDs (identifiers) and keywords in the data structure of a data set. An ID is an identifier such as a number assigned individually to a keyword. A keyword is a string of various headwords including people's names, and is a Wikipedia page headword. In the example of FIG. 2, the keyword "Japanese politician" is associated with ID "id ₁ ", the keyword "Taro _X ", and so on.
As a result, various keywords including people's names are uniquely identified by the ID.

FIG. 3 is a data structure diagram of a category table Tc showing an example of the correspondence between IDs (identifiers) and category IDs in the data structure of a data set. The category IDs are a list of IDs of categories to which keywords belong. In the example of FIG. 3, the category IDs "id ₁ , id ₂ , id ₃ , id ₄ , ..., id ₈ , ..." are associated with the ID "id _6. " This means that the keyword "Taro X" corresponding to the ID "id ₆ " belongs to the keyword (category KW) "Japanese politicians" corresponding to the ID "id ₁ ," the keyword (category KW) "politicians of each prefecture" corresponding to the ID "id ₂ ," etc.
This identifies which category the keyword belongs to.

FIG. 4 is a data structure diagram of a hierarchical table Tu showing an example of the correspondence between IDs (identifiers) of category keywords and upper-level IDs in the data structure of a dataset. Upper-level IDs are a hierarchical list of IDs in the upper level for a certain ID. In the example of FIG. 4, IDs "id ₁ , id ₂ , id ₃ " are associated with ID "id ₄ " as upper-level IDs. This means that for the category keyword "Diet members elected from Tokyo" corresponding to ID "id ₄ ," the keyword "Japanese politicians" corresponding to ID "id ₁ ," the keyword "politicians from each prefecture" corresponding to ID "id ₂ ," and the keyword "Tokyo politicians" corresponding to ID "id ₃ " are in the upper level as categories.
This allows the keywords to be organized into a hierarchy.
If the dataset is Wikipedia data, there will be a table in which IDs are associated with explanatory text that explains the keywords, but this will not be shown here.

Returning to FIG. 1, the configuration of the face recognition learning data generation device 1 will be further described.
The data analysis means 10 includes a category hierarchy means 100, a person KW extraction means 101, and a category information extraction means 102.

The category hierarchy means 100 generates category hierarchy information by associating category IDs with position information representing hierarchical categories from the data set.
Specifically, the category layering means 100 refers to the layer table Tu in Fig. 4 and sequentially associates each ID (identifier) with a higher layer ID in a recursive manner, thereby generating a tree structure graph as shown in Fig. 5. Here, an example of a tree structure in which the category keyword "Japanese politicians" is at the top layer will be described.
In addition, in Wikipedia, the hierarchy of categories is a directed acyclic structure, and since there is a possibility that the same category keyword will appear in the recursive process, the maximum number of recursions is set in advance and the process is stopped.

The category hierarchy means 100 then generates category hierarchy information by associating the IDs of category keywords with the position information of each category keyword configured in a tree-structured graph. This position information indicates the position of a node (corresponding to a category keyword) in the tree-structured graph, and can be expressed, for example, in tuple format. In tuple-format position information, each number in the order of the numeric sequence represents the position of a node traversed at each level in the tree-structured graph.

For example, suppose the position information of a certain node is (0, 2, 25, 11, 2). Each value indicates the position of the child node from the left of the parent node, starting from the top layer. The leftmost value in the position information is '0' to indicate the topmost (first layer) node (root node). The second value in the position information, '2', indicates the second child node from the left of the parent node in the layer (second layer) below the root node.
For example, in FIG. 5, the location information of the category keyword "Japanese politicians" of id ₁ is (0), and the location information of the category keyword "Tokyo politicians" of id ₃ is (0, 1, 1).
The category hierarchy unit 100 outputs the generated category hierarchy information (ID and location information) to the category information extraction unit 102 .

The person KW extraction means (person keyword extraction means) 101 extracts person name keywords (person KWs) from the dataset. Here, the person KW extraction means 101 extracts keywords corresponding to categories including predetermined character strings that identify people from the dataset as person KWs.
Specifically, the person KW extraction means 101 searches for keywords containing predetermined words that can be used to identify a person, such as "year of birth,""year of death," and "living person," and extracts keywords that contain the ID corresponding to the keyword as a category ID as person KW.

For example, the person KW extraction means 101 searches for IDs containing the keyword phrase "grade" in the keyword table Tk of Fig. 2. Then, the person KW extraction means 101 further searches for IDs containing the searched ID (id ₈ in this case) as a category ID in the category table Tc of Fig. 3. Then, the person KW extraction means 101 extracts "XXXX Taro" corresponding to the searched ID (id ₆ in this case) as a person KW in the keyword table Tk of Fig. 2.
In addition, in Wikipedia, the process of identifying keywords that include in the category any of the phrases "year of birth,""year of death," and "living person" as person KWs has been experimentally confirmed to be 100% accurate.

Of course, keywords for people in a data set may also be keywords that include a phrase that identifies the person in the person's name. For example, a keyword that includes the predetermined phrase "person" such as "Taro X (person)" may be used. In this case, the person KW extraction means 101 may extract keywords that include the phrase "person" from the keyword table Tk as person KWs.
The person KW extraction means 101 outputs the extracted person KW to the category information extraction means 102 and the face image collection means 11 .

The category information extraction means 102 extracts, as category information, hierarchical categories corresponding to keywords of person names from the data set based on the category hierarchical information hierarchized by the category hierarchization means 100 .
The category information extraction means 102 extracts higher-level IDs in the tree structure from the category hierarchy information, based on the category ID corresponding to the person KW extracted by the person KW extraction means 101. Then, the category information extraction means 102 generates keywords corresponding to the reference category ID and the higher-level category ID as category information.

For example, even if only "id ₅ " is set as the category ID for a certain person KW "△△Hanako" in the category table Tc (Figure 3), the category information extraction means 102 generates, as category information, keywords corresponding to the ID "id ₅ " and the IDs "id ₁ , id ₂ , id ₃ , id ₄ " that are hierarchically ranked higher in the category hierarchy information, as shown in Figure 6.
This allows the category information extraction means 102 to associate not only category keywords directly corresponding to person keywords but also category keywords higher in the hierarchical structure with person keywords.
The category information extraction means 102 associates the person KW with the category information and outputs it to the data integration means 13 .

Note that tree-structured location information may be added to each keyword in the category information. This makes it possible to create a hierarchical database of face recognition training data.

The face image collecting means 11 collects face images corresponding to person KW.
The facial image collecting means 11 includes a web search means 110 and a facial image extracting means 111 .

The Web search means 110 searches for and collects documents (Web documents) related to keywords of people's names on the Web, which is a system for publishing and viewing documents on a network (Internet) N.
The web search means 110 uses a general search engine via the network N to search and collect web documents related to the person keywords extracted by the person keyword extraction means 101 of the data analysis means 10 from the web server S. Note that the search for person keywords using a search engine can be performed using general web scraping technology.
The web documents collected by the web search means 110 include images including the face of the person KW, images other than the face, text data, and the like.
The web search means 110 outputs the collected web documents to the face image extraction means 111 .

The facial image extraction means 111 extracts facial images from images contained in the web documents collected by the web search means 110 by facial recognition.
The face image extraction means 111 detects faces in images contained in a Web document by using a general face recognition process, and extracts only images that show one face.
The facial image extraction means 111 outputs the extracted facial image and the Web document from which the facial image was extracted, together with the person KW, to the person KW correspondence determination means 12 .

The person keyword correspondence determination means (person keyword correspondence determination means) 12 determines whether or not the face image collected by the face image collection means 11 is a face image of a person of the person keyword.
The person KW correspondence determination means 12 includes a feature extraction means 120 , a clustering means 121 , a relevance setting means 122 , and a determination means 123 .

The feature extraction means 120 extracts features (face feature amounts) from the face images collected by the face image collection means 11 .
For example, the feature extraction means 120 extracts facial feature values F by performing multiple convolution operations on a facial image I using a pre-trained convolutional neural network NN equipped with multiple convolution layers CL (CL ₁ , CL ₂ , ...) as shown in Fig. 7. This facial feature value F has a dimensionality of, for example, about 500.
This convolutional neural network NN can use, for example, a network for extracting features from a general object recognition network such as VGG (Visual Geometry Group) 19.
The feature extraction means 120 outputs facial feature amounts {f ₁ , f ₂ , . . . , f _n } (n is the number of facial images) extracted from a plurality of facial images corresponding to a person keyword to the clustering means 121 for each person keyword.

The clustering means 121 clusters the facial feature amounts extracted by the feature amount extraction means 120 .
A general method may be used as the clustering method in the clustering means 121. For example, the clustering means 121 uses the DBSCAN (Density-Based Spatial Clustering of Applications with Noise) method, which can also determine the number of classes, to cluster the facial feature amounts {f ₁ , f ₂ , ..., f _n } of the number of face images.

Fig. 8 shows the concept of facial features clustered by the clustering means 121. Fig. 8 shows a state in which facial features { _f1 , _f2 , ..., _f9 } are classified into three classes ( _C1 , _C2 , _C3 ), with facial features { _f1 , _f4 , _f5 , _f9 } classified into class _C1 , facial features { _f6 , _f8 } classified into class _C2 , and facial features { _f2 , _f3 , _f7 } classified into class _C3 .
This allows the clustering means 121 to divide multiple facial features into multiple groups, such as a group of people corresponding to the person KW and other groups, or even if the people correspond to the same person KW, groups of faces facing forward and faces facing sideways, etc.
The clustering means 121 outputs the clustering result to the determining means 123 .

The relevance setting means 122 sets a relevance indicating the degree of relevance between the facial feature and a person's name in association with the facial feature, based on a Web document including a face image from which the facial feature has been extracted.
The relevance setting unit 122 sets the relevance based on a predetermined criterion. Here, the relevance setting unit 122 increases the relevance based on the criterion from a predetermined initial value (e.g., "0").

For example, if the web document from which the face image has been extracted is a Wikipedia document that uses a person keyword as an entry word, the relevance setting unit 122 adds a predetermined number to the relevance.
Furthermore, for example, when a tag indicating an image in a Web document contains a description of a person keyword, the relevance setting unit 122 adds a predetermined number to the relevance.
Furthermore, for example, when a web document and a description that explains a person keyword included in the data set are similar, the relevance setting unit 122 adds a predetermined number to the relevance. This similarity can be determined, for example, when the number of words common to the web document and the description is greater than a predetermined number, or when the ratio of the number of common words to the total number of words in the web document and the description is greater than a predetermined ratio.
Furthermore, when web documents are categorized in advance, such as in Wikipedia, it is possible to update the relevance by calculating the degree of overlap with the category information of the person keyword and adding it to the relevance.
In the above relevance calculation, it is also possible to obtain the relevance by vector converting the words and image features in the text, which are the elements for calculating the relevance, into a common dimensional space using WORD2VEC, PCA (Principal Component Analysis), etc., and then performing a similarity calculation process such as cosine similarity.
The relevance setting means 122 outputs the relevance W{w ₁ , w ₂ , . . . , w _n } (n is the number of face images) set for a plurality of facial feature amounts corresponding to the person KW to the determination means 123 for each person KW.

The determining means 123 determines whether each class corresponds to a person's name based on the relevance set for the facial feature amounts belonging to each class clustered by the clustering means 121 .
This determination means 123 calculates the sum of the relevance set by the relevance setting means 122 for the facial features belonging to each clustered class, and determines that the class whose sum satisfies a predetermined criterion is the class corresponding to the person KW.

For example, suppose the clustering result for a certain person KW is as shown in Fig. 8. Also, suppose the relevance of the person KW set to the facial feature amount { _f1 , _f2 , ..., _f9 } is { _w1 , _w2 , ..., _w9 }. In this case, the determination means 123 calculates the sum _W1 of the relevance of class _C1 as _w1 + _w4 + _w5 + _w9 , the sum _W2 of the relevance of class _C2 as _w6 + _w8 , and the sum _W3 of the relevance of class _C3 as _w2 + _w3 + _w7 .

In addition, the determination means 123 determines whether each class corresponds to a person KW based on, for example, whether the sum of the relevance degrees for each class exceeds a predetermined threshold, or whether the difference between the sum of the relevance degrees for each class and the sums of the other classes is greater than a predetermined threshold.
The determination means 123 associates the person KW with the face feature determined to be the class of the person KW, and outputs the result to the data integration means 13 .

The data integration means 13 integrates category information corresponding to the person KW generated by the data analysis means 10 with facial features determined to correspond to the person KW by the person KW correspondence determination means 12, to generate learning data for face recognition.
9, the data integration means 13 generates face recognition learning data by linking the facial feature amount F and the category information CI for each person KW. Note that tree-structured position information may be added to each keyword in the category information CI.
Furthermore, the data integration means 13 may add a face image from which facial features have been extracted to the face recognition learning data in place of or together with the facial features.

By configuring as described above, the face recognition learning data generation device 1 can collect face images without any human effort and generate learning data for face recognition with person KW and category information added.
Furthermore, the face recognition training data generation device 1 can collect face images in a wide variety of categories, as well as in hierarchical categories, thereby enabling the face recognition training data generation device 1 to generate comprehensive and systematically organized training data.
The face recognition learning data generation device 1 can be operated by a program (face recognition learning data generation program) that causes a computer to function as each of the above-mentioned means.

[Operation of the face recognition learning data generation device]
Next, with reference to FIG. 10 (for the configuration, refer to FIG. 1 as appropriate), the operation of the face recognition learning data generation device 1 according to an embodiment of the present invention will be described.
In step S1, the category layering means 100 of the data analysis means 10 analyzes the data set and layers the categories. Here, the category layering means 100 refers to a hierarchy table Tu (FIG. 4) and sequentially and recursively associates upper-level hierarchical IDs with each ID (identifier), thereby layering the categories and generating category hierarchy information (FIG. 5) in which IDs and position information are associated with nodes in a tree-structure graph.

In step S2, the person KW extraction means 101 of the data analysis means 10 analyzes the dataset and extracts person KWs from the keywords in the dataset. Here, the person KW extraction means 101 extracts, as person KWs, keywords in the keyword table Tk (Figure 2) whose category table Tc (Figure 3) contains an ID that includes any of the phrases "year of birth," "year of death," or "living person."

In step S3, the category information extraction means 102 of the data analysis means 10 extracts category keywords corresponding to the person keywords extracted in step S2 from the dataset based on the category hierarchy information generated in step S1, and generates category information for the person keywords (Figure 6).

In step S4, the web search means 110 of the face image collection means 11 collects web documents corresponding to the person keywords extracted in step S2 from the web server S via the network N. Here, the web search means 110 collects web documents by web scraping using the person keywords.

In step S5, the facial image extraction means 111 of the facial image collection means 11 extracts facial images from the images contained in the web documents collected in step S4. Here, the facial image extraction means 111 detects faces from the web documents using facial recognition and extracts only images that show one face.

In step S6, the feature extraction means 120 of the person KW correspondence determination means 12 extracts features (facial feature amounts) from the facial image extracted in step S5. Here, the feature extraction means 120 extracts facial feature amounts from the facial image using a pre-trained convolutional neural network.
In step S7, the clustering means 121 of the person KW correspondence determination means 12 clusters the facial feature amounts extracted in step S6.

In step S8, the relevance setting means 122 of the person KW correspondence determination means 12 sets the relevance between the facial feature extracted in step S6 and the person KW. For example, if the web document is a Wikipedia document that uses the person KW as a headword, the relevance setting means 122 sets a higher relevance value the stronger the relevance between the web document and the person KW.

In step S9, the determination means 123 of the person KW correspondence determination means 12 determines, for each class clustered in step S7, whether the class corresponds to a person KW based on the relevance set in step S8. Here, the determination means 123 calculates the sum of the relevance degrees for the facial features included in the class, and determines that a class whose sum of the relevance degrees is greater than a predetermined threshold is a class corresponding to a person KW.

In step S10, the data integration means 13 integrates, for each person KW, the category information generated in step S3 corresponding to the person KW and the facial features belonging to the class determined in step S9 to be the class of the person KW, to generate learning data for face recognition (FIG. 9).
Through the above operations, the face recognition learning data generation device 1 can automatically collect face images corresponding to person KWs and generate face recognition learning data.

[Configuration of face recognition device]
Next, the configuration of the face recognition device 2 according to the embodiment of the present invention will be described with reference to FIG.
The face recognition device 2 recognizes the faces of people appearing in the video.
As shown in FIG. 11, the face recognition device 2 includes a face recognition learning data storage unit 20, a face detection unit 21, a feature extraction unit 22, a similarity calculation unit 23, and a person identification unit 24.

The face recognition learning data storage means 20 stores face recognition learning data, and can be configured as a general storage medium such as a hard disk.
The face recognition learning data storage means 20 stores in advance the face recognition learning data generated by the face recognition learning data generation device 1 described with reference to FIG.
As shown in FIG. 9, the face recognition learning data is a database in which person KW is associated with face feature amounts F and category information CI.

The face detection means 21 detects faces appearing in the input video image. The face detection means 21 recognizes faces in the video image by using a general face recognition process.
The face detection means 21 extracts the detected face area (face image) from the video for each frame and outputs it to the feature extraction means 22 .

The feature extraction means 22 extracts facial features from the face image detected by the face detection means 21. The feature extraction means 22 extracts facial features using the same method as the feature extraction means 120 of the face recognition training data generation device 1 described in FIG.
The feature extraction means 22 outputs the extracted facial feature or features to the similarity calculation means 23 .

The similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted by the feature extraction means 22 and the facial features stored in the face recognition learning data storage means 20.
Here, the similarity calculation means 23 calculates the similarity for facial features stored in the face recognition learning data storage means 20 that correspond to category information (keywords) specified from outside.
For example, if the keyword "politicians in Tokyo" is specified as category information, the similarity calculation means 23 will calculate the similarity only for the facial features of the learning data for face recognition stored in the face recognition learning data storage means 20, whose category information includes the keyword "politicians in Tokyo."

The similarity calculation method used by the similarity calculation means 23 may be a general method, and for example, the similarity may be calculated using the cosine similarity between the vector of facial features extracted by the feature extraction means 22 and the vector of facial features stored in the face recognition learning data storage means 20.
The similarity calculation means 23 outputs the calculated similarity and the person KW of the face recognition learning data used to calculate the similarity to the person identification means 24 .

The person identification means 24 identifies a person based on the similarity calculated by the similarity calculation means 23 .
The person identification means 24 extracts those whose similarities calculated by the similarity calculation means 23 are greater than a predetermined threshold. The person identification means 24 then sorts the extracted similarities and outputs, in descending order of similarity, the person KWs of the face recognition learning data used to calculate the similarities as recognition results. Of course, for simplicity, it is also possible to output only the person KW with the highest similarity.

In addition, the person identification means 24 may output keywords of the specified category information in association with the person KW, or keywords of all category information associated with the person KW stored in the face recognition learning data storage means 20.
This makes it possible to add metadata based on category information to a person KW recognized by the face recognition device 2.
In addition, if tree-structured location information is added to the category information of the face recognition learning data stored in the face recognition learning data storage means 20, the person identification means 24 may output the location information in addition to the category information in association with the person KW.

With the configuration described above, the face recognition device 2 can perform face recognition by specifying category information, thereby improving the accuracy of face recognition.
The face recognition device 2 can be operated by a program (face recognition program) that causes a computer to function as each of the above-mentioned means.

[Facial Recognition Device Operation]
Next, the operation of the face recognition device 2 according to an embodiment of the present invention will be described with reference to Fig. 12 (for the configuration, see Fig. 11 as appropriate). It is assumed that the face recognition learning data storage means 20 has previously stored therein face recognition learning data generated by the face recognition learning data generation device 1 (Fig. 1).

In step S20, the face detection means 21 detects a face appearing in the input video, and extracts the area of the detected face as a face image.
In step S21, the feature extraction means 22 extracts features (face feature amounts) from the face area (face image) detected in step S20.

In step S22, the similarity calculation means 23 calculates the degree of similarity (similarity) between the facial features extracted in step S21 and the facial features stored in the face recognition learning data storage means 20 and corresponding to externally specified category information (keywords).
In step S23, the person identification means 24 identifies a person based on the similarity calculated in step S22. At this time, the person identification means 24 extracts and sorts person keywords whose similarity is greater than a predetermined threshold, and outputs the keyword keywords as recognition results in descending order of similarity.
Through the above operations, the face recognition device 2 can recognize faces from video by limiting the categories.

Although the embodiment of the present invention has been described above, the present invention is not limited to this embodiment.
In this embodiment, the dataset contains keywords including person names, and the target person is identified using the person name keyword. However, the keyword does not have to be a person's name as long as it can identify the target person. For example, the keyword may be a number representing a person. For example, it may be an index such as a My Number that can identify a person. Furthermore, the text of the dataset itself may have person names replaced with other indexes so that individuals cannot be identified.

Furthermore, in this embodiment, documents containing facial images are described as being searched and collected on the web. However, documents containing facial images are not limited to those searched and collected on the web, and can be anything that contains a facial image and document data related to that face. For example, documents containing facial images can be documents that have been saved in advance on a storage medium, or they can be documents that are read from an external source. Therefore, the web search means 110 is not limited to searches on the web, and can also be configured as a collection means. Note that the documents in this case are multimedia documents that include text, images, and audio.

Furthermore, in this embodiment, the determination means 123 is described as calculating the sum of the relevance for each clustered class and making a determination based on this calculated value. However, the index used for the determination is not limited to the sum of the relevance for each class. The index used for the determination may be any index that can be determined to correspond to a person keyword. For example, the index may be the average relevance of the class, the maximum value of the facial feature values belonging to the class, etc.

REFERENCE SIGNS LIST 1 Face recognition learning data generation device 10 Data analysis means 100 Category hierarchy means 101 Person keyword extraction means 102 Category information extraction means 11 Face image collection means 110 Web search means (collection means)
111 Facial image extraction means 12 Person-KW correspondence determination means 120 Feature extraction means 121 Clustering means 122 Association degree setting means 123 Determination means 13 Data integration means 2 Face recognition device 20 Face recognition learning data storage means 21 Face detection means 22 Feature extraction means (second feature extraction means)
23 Similarity calculation means 24 Person identification means

Claims (2)

a person keyword extraction means for extracting keywords of person names from a data set in which keywords are associated with hierarchical categories; and a category information extraction means for extracting, as category information, the hierarchical categories corresponding to the keywords that identify people from the data set;
a collection means for collecting documents related to a keyword that identifies the person; and a face image extraction means for extracting a face image from an image included in the document by face recognition;
a feature extraction means for extracting features from the face image;
a relevance setting means for setting a relevance between the feature amount and the person specified by the keyword in association with the feature amount based on the document including the face image from which the feature amount has been extracted;
clustering means for clustering the feature quantities of the face image;
a determination means for determining whether each of the clustered classes corresponds to the specified person based on the relevance set for the feature amounts belonging to each of the clustered classes;
a data integration means for integrating, for each keyword that identifies the person, features belonging to a class determined to correspond to the identified person with the category information to generate face recognition learning data;
a face recognition learning data storage means for storing the face recognition learning data;
a face detection means for detecting a face in the video and extracting a face image;
a second feature extraction means for extracting features from the face image extracted by the face detection means;
a similarity calculation means for calculating a similarity between a feature of a face image corresponding to designated category information in the face recognition learning data and the feature extracted by the second feature extraction means;
a person identification means for identifying a person name corresponding to a feature value having a similarity greater than a predetermined threshold value as a recognition result;
A face recognition device comprising: A facial recognition program for causing a computer to function as the facial recognition device described in claim 1.