JP4092059B2 - Image recognition device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image recognition apparatus and a recording medium for recognizing the orientation of a face image included in a moving image or a still image.
[0002]
[Prior art]
Conventionally, documents that disclose this type of image recognition method include “Face Tracking and Pose Representation”, Stephen MCKENNA, Shaogang GONG, John J. et al. COLLINS, British Machine Vision Conference, Edinburgh, 1996. This document describes expressing the orientation of a face in an image and proposes the following image recognition method.
[0003]
(Processing 1) An image set is prepared in which faces of a plurality of persons are shown in several orientations with respect to the vertical axis. A Gabor wavelet count having four orientations is measured for each image pixel of this image set, and a feature vector is constructed from the absolute value of the count. Calculate the average of feature vectors for all persons in each orientation. The principal component by direction is calculated from the average feature vector. In the calculation of the process 2, the first several principal component vectors are used in descending order of the eigenvalues.
[0004]
(Process 2) A face image to be recognized is input, the coefficients of Gabor wavelets having four orientations are measured at all pixels of the input image, and a feature vector is constructed from the absolute values of the coefficients. Projection is performed on the principal component vector obtained in the above processing 1, and the orientation of the face in the input image with respect to the vertical axis is expressed by the result.
[0005]
[Problems to be solved by the invention]
In the image recognition method, the face direction can be expressed by a small number of variables (that is, projection coefficients). However, the proposed method has the following drawbacks.
[0006]
(A) Since the Gabor wavelet feature is obtained in all the pixels of each learning image and the input image and the feature vector of the image is obtained by rearranging the features of all the pixels of each image, the number of dimensions of the feature vector is large, and therefore the amount of calculation Will become very large.
[0007]
(B) Even on pixels having the same coordinates, the position on the face differs depending on the learning image or the input image. For this reason, even if the center position and size of the face area are normalized, a difference in the shape of the face by a person has an unfavorable effect on the expression of the face direction. As a result, the direction estimation accuracy decreases.
[0008]
(C) Since the wavelet coefficients are measured for all the pixels of the learning image and the input image, if there is a background in the image, the background affects the image recognition, and the accuracy of estimating the face orientation decreases.
[0009]
SUMMARY OF THE INVENTION An object of the present invention is to provide an image recognition method and a recording medium that can reduce the amount of calculation of image recognition without impairing the accuracy of face orientation estimation.
[0010]
[Means for Solving the Problems]
In order to achieve such an object, the invention of claim 1 extracts features from learning image data whose face orientation is known in advance and image data of a face to be recognized, and uses the extracted features. Then, in the image recognition apparatus for recognizing the face orientation of the image data to be recognized, the first feature pre-extracted from the learning image data with respect to the specific position of the face, from the specific positions of the plurality of learning image data A memory storing a first feature, which is a principal component vector calculated by extracting Gabor wavelet coefficients, calculating an average feature vector of a plurality of persons for each face orientation, and analyzing the calculated average feature vector. means and, designating means for designating the specific position corresponding to the position of the image data of the recognition target, a second feature of the specified location, the image data of the recognition target A feature vector using a Gabor wavelet coefficients extracted from the position corresponding to the specific position, the second feature comprises a feature extracting means for extracting from the image data of the recognition target, extracted by said feature extracting means the second characteristic, and estimates the direction of said first feature and is the principal component vectors facial recognition target by the projection coefficients obtained by projecting the stored in the storage means.
[0011]
According to a second aspect of the present invention, in the image recognition apparatus according to the first aspect, an input unit that inputs the learning image data and a face direction, and a first feature at the specific position from the input learning image data. It further comprises learning image data feature extracting means for extracting and storing in the storage means.
[0013]
According to a third aspect of the present invention, in the image recognition apparatus according to the first or second aspect , the specific position includes any of both ends of the eye, between the eyebrows, the top of the nasal head, both ends of the lips, and the upper center of the lips. And
[0018]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
[0019]
In the present embodiment, the disadvantages of the conventional image recognition method are eliminated based on the following image recognition method.
[0020]
(1) As a feature vector of a learning image or an input image, in the case of the above-mentioned document, Gabor wavelet coefficients are measured in four directions at all pixels (128 × 128 = 16384 points). The Gabor wavelet coefficient is measured only (for example, in the embodiment described later, nine points such as the corner of the eye, the corner of the eye, and both ends of the mouth). Thus, in this embodiment, only a small number of feature points in the face are used, so even if eight orientations are used, the number of feature vector dimensions and the amount of calculation are overwhelming in this embodiment. Very few.
[0021]
(2) In the conventional method, even if the size or the like is normalized by processing an image without aligning facial features (for example, the eyes), the personality of the shape of the person's face is not preferable for expressing the orientation. This can affect the accuracy of the direction estimation. However, in the present embodiment, by specifying the position of each feature point in the face, the feature points are always located at the same place (for example, the eyes) of the face. This
(A) When the feature vectors for each direction of the learning image are averaged, the positions of the same features are matched even in different faces, and the average feature vector and its principal component vector are not affected by the personality of the human face shape. It is possible to express only changes due to the orientation of the face.
(B) Similarly, when projecting a feature vector measured from an input image onto a principal component vector, the feature vector measurement location is not the same location of the image but the same location of the face, so that the orientation of the face is estimated. The value is not affected by the personality of the person's face shape.
[0022]
(3) In the conventional image recognition method, since the wavelet coefficient is measured for all the pixels of the learning image and the input image, if the image has a background, it may affect the result. On the other hand, in this embodiment, even if there is a background in the learning image or the input image, the influence of the background is minimized by measuring the feature vector using only the neighborhood for each feature point.
[0023]
FIG. 1 shows a functional configuration of an embodiment of the present invention. In FIG. 1, the learning image set is an image set of a plurality of faces of a plurality of persons, and images having different face orientations are included in the image set. The orientation of the face indicated by each learning image is given in advance. For example, a learning image is given in the form of “person ID = n, direction = p”. In the present embodiment, the learning image set includes a plurality of images whose directions are different by 10 degrees between −40 degrees and +40 degrees. (The front face is called 0 degrees.)
[0024]
The number of learning image sets collected in the present embodiment is 17, and the size of the learning image is 128 × 128 pixels. Details of normalization will be described later. It is also assumed that a plurality of feature point coordinates (9 points in the present embodiment) for each learning image are also given. It is assumed that the coordinates of the feature points are obtained manually or by another method. In this embodiment, the learning image is displayed on the display screen of the image processing apparatus, the operator designates the position of the feature point with a mouse or the like, and the position of the feature point of the learning image is acquired on the image processing apparatus side.
[0025]
In the present embodiment, the position of each feature point on the face is as follows.
[0026]
Feature point 0: The vertical center line of the face, the deepest place in the upper part of the nose. Feature point 1: Red and white boundary point in the middle side of the upper lip on the vertical center line of the face. Point 3: Human right eye feature point 4: Human left eye feature point 5: Human left eye feature point 6: Nose tip (frontmost in 3D)
Feature point 7: The right corner of the person's right corner. Feature point 8: The left corner of the person. Whereas the conventional feature point is the number of all pixels, in this embodiment, there are nine points. Note that there are few feature points. FIG. 2 schematically shows a given learning image set and the positions of feature points in each learning image.
[0027]
As a pre-processing of face orientation recognition processing, the face area indicated by the original learning image is translated to the center of the image, the length of the line from the feature point 0 to the feature point 1 is increased to approximately 56 pixels, and the inclination is increased. In the image recognition apparatus, normalization processing (enlargement, rotation, translation) is performed in which the image is vertically rotated for the front face and rotated 10 degrees for the side face. The processing shown in FIG. 1 is executed in the image recognition apparatus using the learning image and the feature point coordinate values that have been preprocessed in this way.
[0028]
The feature vector measuring unit 1 in FIG. 1 convolves the neighborhood of the learning image with some Gabor wavelet functions at each feature point, and calculates the absolute value of the result (wavelet coefficient). The definition of the wavelet used is based on the following document.
-Simon CLIPPINGDALE, Takayuki Ito, "Feature selection experiment of moving image face tracking / recognition system" FAVRET "", IEICE PRMU 99-109, November 1999, minimum resolution ((1) to (3) Only those having the resolution parameter r = 4) in the formula are used, although in this embodiment, eight Gabor wavelet functions are used, but the present invention is not limited to this. This is a generating function that is rotated for each learning image.8 features × 9 feature points used in this embodiment = 72 coefficient absolute values are represented as feature vectors, and represents one learning image. Is shown in the upper half of FIG.
[0029]
The average calculation unit 2 for each direction calculates an average vector of feature vectors of all persons in each direction. This process is shown as the thick arrow in FIG.
[0030]
The principal component analysis unit 3 obtains a principal component vector based on the direction of the average feature vector. This process is generally well known and is described, for example, in the following references.
・ Press, Teukolsky, Vetterling & Flannery, “Numeric Recipe in Sea, Numerical Computation Recipe in C” (Japanese version), Technical Review, ISBN 4-87408-560-1
・ Toyo Tanaka and Kazumasa Wakimoto, “Multivariate Statistical Analysis”, Chapter 2, “Principal Component Analysis” of Contemporary Mathematics Company, Kyoto (1983)
The principal component vector obtained by the principal component analysis unit 3 is a unit eigenvector of the sample covariance matrix of the average feature vector set. A rearranged order of the eigenvalues is shown in the lower part of FIG. Only the two principal component vectors e ₀ e ₁ having the largest eigenvalues are stored in the memory inside the image processing apparatus.
(I) Assume that an input image and its feature point coordinates are given. In the embodiment, the following reference, Japanese Patent Application No. 11-206764, Simon CLIPPINGDALE, Takayuki Ito, “A unified approach to face detection / tracking / recognition of moving images”, IEICE PRMU 98-200, The feature point coordinates are obtained by a unified tracking / recognition method described in January 1999. Similar to the learning image normalization process described above, the input image and the feature point coordinates are normalized in advance (see the upper right in FIG. 4).
[0031]
The feature vector measuring unit 4 obtains a feature vector representing the input image as in the above-described feature vector measuring unit 1 (see the upper left in FIG. 4). The projection unit 5 projects the feature vector obtained from the input image onto the principal component vectors e ₀ and e ₁ . That is, the inner product with each of the principal component vectors e ₀ and e ₁ is calculated (see the upper right in FIG. 4), and the projection coefficient u. Get v. Let the projection coefficient be coordinates in the principal component space (see the lower right in FIG. 4).
[0032]
In vector quantization VQ6, the direction of the face appearing in the input image is estimated from the values of the projection coefficients u and v. An example of the input / output function of the vector quantization V06 is shown in FIG.
[0033]
In order to set the input / output function of the vector quantization unit VQ6, data obtained from an actual face image is used. As an example, in the principal component space obtained from actual data for 17 persons, the same face image for 17 persons is used as an input image, and the obtained projection coefficients are shown in FIG. FIG. 6 shows the average of the projection coefficients for 17 persons. The region corresponding to each direction can be defined as shown in FIG. 5 by the position of the average value in this graph (indicated by □ in the figure). Projection coefficients obtained by the image recognition method of the present embodiment in accordance with changes in the orientations of input face images for several people are represented as lines. Most of them show a gentle trajectory with a similar shape, which makes it easier to estimate the orientation of the face.
[0034]
An example of a specific system configuration of the image recognition apparatus is shown in FIG. As the image recognition device, a general-purpose image processing device, a personal computer, or the like can be used.
[0035]
The image recognition apparatus includes a CPU 7, a system memory 8, a hard disk drive (HDD) 9, an input / output processing unit (I / O) 10 connected to a scanner, a keyboard 11, a mouse 12, a display 13, and the like. A program is read from a recording medium (not shown) and installed in the HDD 9. When executing the program, the program is read from the recording medium or the HDD 9, stored (loaded) in the system memory 8, and executed by the CPU 7.
[0036]
The system operation of the image recognition apparatus in FIG. 7 will be described with reference to FIG.
[0037]
A menu selection screen is displayed on the display 13 in response to the activation of the program. In this embodiment, a learning image registration mode and a face orientation recognition mode are prepared as menus. When the user selects the learning image registration mode with the keyboard 11 or the mouse 12 (step S10), the selection content is determined, and the procedure proceeds from step S20 to step S100.
[0038]
The user sets a plurality of image originals to be input to the scanner and executes image reading. The read image is stored in the system memory 8 and displayed sequentially on the display 13. The user uses the mouse 12 to specify facial feature points on the display screen, that is, both ends of the eyes, the top of the nose, between the eyebrows, both ends of the lips, and the upper center of the lips. Further, the face direction and the image identification number (ID) are input by the keyboard 11 and the mouse 12. Image data corresponding to the designated position is extracted from the input image stored in the system memory 8. The extracted image data and the face orientation are stored in another area in the system memory 8 in units of read images.
[0039]
The CPU 7 collects image data of the same face orientation with respect to the extracted and stored image data, and calculates features using these image data. Here, the processing 1 to processing 3 described with reference to FIG. 1 is performed, and the calculated feature (principal component of the average feature vector) and the face orientation in that case are stored in the HDD 9 (steps S100 → S110 → S120). This completes the learning image registration process.
[0040]
If the user wants to recognize the face orientation, the user selects the face orientation recognition mode on the menu screen with the mouse 12 or the like (step S10). By this selection process, the procedure proceeds from step S10 to step S30 to step S200.
[0041]
The user sets an image document to be recognized on the scanner and causes the scanner to read an image. The image thus read is stored in the system memory 8 and also displayed on the display 13 (step S200).
[0042]
The user designates feature points of the face of the displayed image, that is, feature points at the same positions as the registration of the learning image, such as both ends of the eyes, the top of the nasal head and the eyebrows (step S210). The CPU 7 extracts the image data at the designated position from the system memory 8 and calculates a feature vector. This process corresponds to the process 4 in FIG. The orientation of the image is recognized using the principal component and orientation of the average feature vector stored in the HDD 9 and the feature vector acquired from the recognition target image. This processing is performed through processing 5 and processing 6 in FIG. The recognition result is displayed on the display 13 in the form of a face orientation angle, for example.
[0043]
As described above, according to the present embodiment, the amount of calculation is small compared to the conventional method, and the orientation of the face with respect to the vertical axis can be efficiently expressed and estimated.
[0044]
In addition to the above embodiment, the following embodiment can be implemented.
1) In the above-described embodiment, the specification of the feature points of the input image to be recognized and the learning image is manually performed with the mouse, but can be automatically performed on the image recognition apparatus side. As an example, the contour line of the image is extracted and the part of the extracted contour line is determined. The part may be determined by using an image template of color or a facial part such as eyes, nose and mouth. If colors are used, it is better to use the fact that lips have a lot of red components and eyes have a lot of black and white components. The feature points may be selectively used such as eyes, lips, and nose, and are not limited to the above-described embodiments. It is also possible to use facial parts other than the eyes, lips, and nose as the position (specific position) of the feature point, but a position where the change of the position becomes clear every time the face direction is different is preferable.
2) In the above-described embodiment, the image recognition apparatus has a learning function, that is, a function for extracting features from learning image data. However, features are extracted from learning image data by another image processing device in advance and extracted. The feature and the direction thereof may be stored in the hard disk of the above-described image processing apparatus via a portable recording medium or communication.
3) As a recording medium for recording a program for recognizing a face, an IC memory such as a ROM and a RAM, a hard disk, a CDROM, and other known storage devices can be stored. When this program is downloaded from another device to the image recognition device by communication, a storage device that stores the program in the other device becomes the recording medium of the present invention.
4) Although the recognition process using a still image has been described in the above embodiment, it goes without saying that an image to be recognized or a learning image can be a moving image. In the case of a moving image, a plurality of still images constituting the moving image are to be processed.
5) A face database can be constructed for face image processing (detection, tracking, area segmentation, person recognition, etc.) using the image recognition method described above. See references below.
[0045]
-Japanese Patent Application No. 11-206764-Simon CLIPPINGDALE, Takayuki Ito, "Unified Approach to Face Detection / Tracking / Recognition of Moving Images", IEICE PRMU 98-200, Implementation mentioned above from January 1999 Various modifications in addition to the forms are possible, but the modifications are within the technical scope of the present invention as long as the modifications are in accordance with the technical idea described in the claims.
[0046]
【The invention's effect】
As described above, since the feature of the image is extracted from the specific position of the face where the change of the position clearly appears when the face direction is different, such as eyes, nose, lips, etc., the image data pixel Compared with the conventional method of extracting features from all, the amount of calculation related to face orientation recognition processing is greatly reduced, and the recognition performance is not lowered.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a functional configuration of an embodiment of the present invention.
FIG. 2 is an explanatory diagram for explaining a feature point of an embodiment of the present invention.
FIG. 3 is an explanatory diagram showing information processing contents according to the embodiment of the present invention.
FIG. 4 is an explanatory diagram showing information processing contents according to the embodiment of the present invention.
FIG. 5 is an explanatory diagram showing information processing contents according to the embodiment of the present invention.
FIG. 6 is an explanatory diagram showing a projection coefficient according to an embodiment of the present invention.
FIG. 7 is a block diagram showing a system configuration of an embodiment of the present invention.
FIG. 8 is a flowchart showing a processing procedure according to the embodiment of the present invention.
[Explanation of symbols]
7 CPU
8 System memory 9 HDD
10 I / O
11 Keyboard 12 Mouse 13 Display

Claims (3)

An image recognition device that extracts features from learning image data whose face orientation is known in advance and image data of a face to be recognized, and recognizes the face orientation of the image data to be recognized using the extracted features In
A first feature pre-extracted from the learning image data for a specific position of the face, Gabor wavelet coefficients are extracted from the specific positions of the plurality of learning image data, and an average feature vector of a plurality of persons is obtained for each face orientation. Storage means for storing a first feature that is a principal component vector calculated by performing principal component analysis on the calculated average feature vector ;
Designation means for designating a position corresponding to the specific position of the image data to be recognized;
The second feature of the designated position is a feature vector using a Gabor wavelet coefficient extracted from a position corresponding to the specific position of the image data to be recognized. Comprising feature extraction means for extracting from the image data,
The second feature extracted by said feature extracting means, to estimate the first is characteristic principal component vector direction of the face to be recognized by the projection coefficients obtained by projecting the a stored in the storage means A featured image recognition apparatus. 2. The image recognition apparatus according to claim 1, wherein an input unit that inputs the learning image data and a face orientation, and a first feature at the specific position is extracted from the input learning image data and stored in the storage unit. An image recognition apparatus, further comprising: feature extraction means for learning image data to be stored. 3. The image recognition apparatus according to claim 1, wherein the specific position includes any one of both ends of the eye, between the eyebrows, the top of the nasal head, both ends of the lips, and the upper center of the lips.

Images