JP5528172B2 - Face image processing device - Google Patents

Description

  The present invention relates to a face image processing apparatus that creates face image data to be registered for collation processing from an image obtained by photographing a person's face.

Conventionally, there has been proposed a face authentication device that authenticates a target person by collating a face image acquired by photographing the face of the target person with a registered face image. Such a face authentication apparatus registers the face image of the subject person in advance, and determines whether or not to authenticate based on the result of comparison with the face image acquired at the time of use. However, the orientation of the face when the registration face image is captured may differ from the orientation of the face of the subject in the image captured for verification. When collation processing is performed between images with different face orientations, even if images of the same subject are captured, the arrangement of characteristic parts such as eyes, nose, and mouth varies from image to image. Will fall.
Therefore, a technique for creating a three-dimensional face model by mapping a registration two-dimensional face image obtained by photographing a subject to three-dimensional face shape data representing a three-dimensional shape of a human face. Has been proposed (see, for example, Patent Document 1).

  The face image processing apparatus disclosed in Patent Document 1 adjusts the face orientation of a plurality of 3D face shape data in accordance with a person's 2D face image, estimates the light source direction from the 2D face image, and determines the face orientation. The estimated light source direction is applied to the adjusted plurality of three-dimensional face shape data to create a two-dimensional shadow image of the face direction, and to the two-dimensional shadow image most similar to the two-dimensional face image Corresponding 3D face shape data is selected. The face image processing apparatus synthesizes the selected 3D face shape data and the 2D face image to create a 3D face model corresponding to the face of the person.

JP 2009-2111151 A

  The face authentication apparatus using the technique disclosed in Patent Document 1 can create a three-dimensional face model of the subject. For this reason, when performing the matching process, such a face authentication device creates a face image having the same face direction as that of the person shown in the image acquired at the time of matching from the 3D face model. Since the positional shift of the characteristic part by the difference in direction can be suppressed, collation accuracy can be improved.

The technique disclosed in Patent Document 1 does not consider the shadows due to things other than the structure of the human face, and it is assumed that the light reflected by the face is not strong enough to saturate on the image. However, depending on the environment in which the subject is photographed to create a registration face image, a part of the face may be exposed to direct sunlight that cannot be ignored. In such a case, a bright area is generated on the registration face image so that the luminance value of the pixel is saturated. In addition, a shadow such as a pillar or window frame or a shadow caused by planting may be applied to the face. In such a case, a dark region having a very low luminance value of pixels compared to other regions is generated on the registration face image. If an attempt is made to generate a pre-registered 3D face model using a face image including such a bright area or dark area, the face orientation cannot be estimated correctly, and an incorrect 3D face model is created. There is a fear.
Further, when the matching process is executed using a three-dimensional face model created based on a face image including such a bright region or dark region, the dark region or bright region or its boundary is erroneously extracted as a facial feature. Or the shadow of the face formed on the image photographed at the time of collation is greatly different, and the collation accuracy may be lowered.
Therefore, there is a need for a technique for determining whether a face image is an image taken in an inappropriate lighting environment as described above.

  SUMMARY An advantage of some aspects of the invention is that it provides a face image processing apparatus capable of determining whether a face image is captured in an inappropriate lighting environment.

  As one aspect of the present invention, a face image processing apparatus is provided. This face image processing apparatus is included in an input face image, an imaging unit that captures an image of the subject's face and creates an input face image, a storage unit that stores three-dimensional face shape data representing the shape of a person's face, and the like. In the target person's face area, a shadow abnormal area extracting unit that extracts a shadow abnormal area corresponding to a shadow other than the structure of the target person's face, and the ratio of the shadow abnormal area to the face area is greater than a predetermined value And a shadow abnormality determining unit that determines that a shadow caused by a structure other than the target person's face exists in the face of the target person in the input face image, and the face of the target person by the shadow abnormality determining unit. A 3D face model of the subject is generated by aligning the input face image determined to have no shading due to other than the structure of the 3D face shape data and mapping it to the 3D face shape data 3D face model generation With the door.

  The abnormal shadow region extraction unit extracts pixels that are out of the normal shadow luminance range corresponding to the luminance range caused by the shadow caused by the structure of the subject's face among the pixels included in the facial region. It is preferable to extract as

  The face image processing apparatus according to the present invention has an effect that it can be determined whether or not the face image is captured in an inappropriate lighting environment.

It is a schematic block diagram of the face image processing apparatus which concerns on one Embodiment of this invention. It is an arrangement | positioning figure showing an example of the positional relationship of a registration subject and illumination. (A) is a figure which shows an example of the input face image image | photographed at the point A in FIG. 2, (b) is a figure which shows an example of the input face image image | photographed at the point B in FIG. (C) is a figure which shows an example of the input face image image | photographed at the point C in FIG. It is a flowchart which shows the operation | movement of the three-dimensional face model production | generation process containing a shadow abnormality determination process.

  Hereinafter, a face image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. This face image processing device maps a face image obtained by photographing a registration target person's face onto 3D face shape data representing a three-dimensional shape of a human face, thereby obtaining the registration target person's 3D face. Create a model. This face image processing apparatus determines whether or not a dark area such as a shadow due to a structure or a bright area bright enough to saturate a pixel value exists in an area corresponding to a face in a face image used for mapping, A three-dimensional face model is created using a face image without such dark areas and bright areas.

  FIG. 1 is a diagram showing a schematic configuration of a face image processing apparatus 1 according to one embodiment. As illustrated in FIG. 1, the face image processing apparatus 1 includes an imaging unit 2, an interface unit 3, a storage unit 4, and an image processing unit 5.

The imaging unit 2 includes, for example, a camera, and creates a two-dimensional input face image obtained by capturing the face of a registration target person who has entered the monitoring area. For this purpose, the imaging unit 2 is installed at a position where the monitoring area can be imaged. For example, when the monitoring area is set indoors, such as in a store or a hallway, the imaging unit 2 is installed at a position higher than the height of the person on the wall near the ceiling in the monitoring area or on the boundary of the monitoring area. The Moreover, the imaging part 2 may be installed so that the vicinity of the common entrance of a monitoring area may be image | photographed. Further, the imaging unit 2 is not limited to one, and a plurality of imaging units 2 may exist.
Note that the camera included in the imaging unit 2 has a sensitivity in the visible light range and creates a color multi-tone image so that it is easy to extract the skin color portion on the input face image. Is preferred. However, the camera included in the imaging unit 2 may be a camera having sensitivity in the near infrared region and creating a gray image. Moreover, it is preferable that the imaging unit 2 has a number of pixels that can distinguish facial features such as the eyes, nose, and mouth of the face shown on the input face image.
The imaging unit 2 outputs the input face image to the interface unit 3 every time an input face image is created.

The interface unit 3 is an interface circuit connected to the imaging unit 2, and passes the input face image received from the imaging unit 2 to the image processing unit 5.
The storage unit 4 includes at least one of a semiconductor memory, a magnetic recording medium, its access device, an optical recording medium, and its access device. The storage unit 4 stores a computer program, various parameters, data, and the like for controlling the face image processing apparatus 1. The storage unit 4 also stores 3D face shape data and 3D position information of 3D face feature points corresponding to the 3D face shape data. There may be a plurality of three-dimensional face shape data, or one. The three-dimensional face shape data is a frame model representing the three-dimensional shape of a person's face, and a wire frame model or a surface model is used. Note that the 3D face shape data is created, for example, from the face shapes of a plurality of persons, or by averaging face shape models simulating the face shapes of many persons. Furthermore, as the three-dimensional face shape data, a part of the three-dimensional face shape data created by another method, such as a technique disclosed in Japanese Patent Application Laid-Open No. 2009-2111148, is used. You may use what changed the ratio of the length of the upper part and the length of a lower part from the center of this.

  In addition, the 3D face feature point represents any point of a characteristic part such as an eye, a nose, or a mouth that is different from the other in terms of shape or color component, for example, a center point or an end point of those parts. For example, the 3D face feature points include an eyebrow head, an eyebrow butt, a black eye center, an eye region center, an eye head, an eye corner, a nasal apex, a nostril center, a mouth center, and a mouth edge. The type and number of 3D face feature points to be stored are not limited, but preferably include at least all the feature points of the same part as the face feature points that can be extracted by the 3D face model generation unit 53. In this embodiment, as the 3D face feature points, the left and right eye area centers, the eyes and corners of the eyes, the nose apex point, the mouth center, and the left and right mouth ends, tragus points, jaw angle points, and mental points are stored. To do.

  The image processing unit 5 includes one or a plurality of processors and their peripheral circuits. When the image processing unit 5 receives the input face image from the imaging unit 2, the image processing unit 5 associates the input face image with the identification code of the person to be registered. The identification code of the person to be registered is input to the image processing unit 5 via a user interface (not shown), for example. Alternatively, the image processing unit 5 is already set when the face of the person to be registered in the input face image does not match any 3D face model of the registrant already registered in the storage unit 4. An identification code different from any of the identification codes may be newly generated. In this case, the image processing unit 5 uses any one of known face image matching methods, and the registration target person's face reflected in the input face image is the 3D of the registered person registered in the storage unit 4. It can be determined whether or not it matches the face model.

The image processing unit 5 creates a three-dimensional face model of the registration target person using the input face image for the registration target person. For this purpose, the image processing unit 5 includes a shadow abnormal region extracting unit 51, a face image selecting unit 52, and a three-dimensional face model generating unit 53 as functional modules implemented by software operating on the processor.
Note that these units included in the image processing unit 5 may be configured by independent integrated circuits, firmware, a microprocessor, and the like.
Hereinafter, each part of the image processing unit 5 will be described in detail.

  Each time the shadow abnormal region extraction unit 51 acquires an input face image from the imaging unit 2 via the interface unit 3, the shadow abnormal region extraction unit 51 extracts a shadow abnormal region existing in a region where the face of the person to be registered on the input face image is shown. Extract. Here, the abnormal shadow region is a region where shadows caused by structures or planting such as pillars or windows located between the light source and the person to be registered, or a region where the luminance value is extremely high due to direct sunlight, etc. An area corresponding to a shadow caused by the structure other than the structure of the face of the person to be registered.

With reference to FIG. 2 and FIG. 3, the manner in which an abnormal shadow region is generated on the input face image will be described.
FIG. 2 is a layout diagram illustrating an example of the positional relationship between the person to be registered and the illumination.
In FIG. 2, the imaging unit 2 is located near the ceiling so as to photograph the face of a person walking toward the door 230 provided in front of the hallway 220 through the hallway 220 provided with a window 210 on one side wall. 2, the optical axis of the imaging optical system included in the imaging unit 2 is set so as to be substantially parallel to the longitudinal direction of the corridor 220 and obliquely downward. The window 210 is a sliding window, and a window frame 211 is formed around the window 210 and at a substantially central portion. Then, it is assumed that sunlight enters the corridor 220 through the window 210 and illuminates the area 240 surrounded by a dotted line.
Here, while the registration target person 250 is walking along the arrows X1 and X2 so as to gradually approach the door 230 in the order of point A → point B → point C, the registration target person 250 moves to the points A and B. , Suppose that the imaging unit 2 has photographed the person to be registered 250 while in each of C and C. At points A and B, the head of the registration target person 250 is included in the region 240 irradiated with sunlight. On the other hand, at the point C, the head of the registration target person 250 is out of the region 240. It shall be.

3A is a diagram showing an example of an input face image of the person to be registered 250 photographed at the point A in FIG. 2, and FIG. 3B is a registration object photographed at the point B in FIG. FIG. 3C is a diagram showing an example of the input face image of the person to be registered 250 taken at the point C in FIG.
As shown in FIG. 3A, in the input face image 310, a shadow 312 is generated near the right end of the region 311 in which the head of the person to be registered is shown. The shade 312 is generated when the sunlight of the person who is registered through the window 210 blocks the head of the person to be registered. As shown in FIG. 3C, in the input face image 330, since the head of the person to be registered is not irradiated with sunlight, the person to be registered faces obliquely downward. There is no strong shadow in the entire region 331 where the head is reflected.
On the other hand, as shown in FIG. 3B, in the input face image 320, a shadow generated by blocking the registration subject person's head in the area 321 in which the registration subject person's head is reflected. Not only 322 exists, but also a shadow 323 generated by the window frame 211 exists so as to overlap the vicinity of the left eye and the left cheek. As described above, when there is a certain structure that covers only a part of the face between the light source and the person to be registered, an input face image in which the shadow of the structure overlaps the face is obtained. Such a shadow 323 becomes a shadow abnormal region on the input face image 320. Since the value of the pixel included in this abnormal shadow region is significantly different from the values of other pixels corresponding to the face of the person to be registered, the presence of the abnormal shadow region adversely affects the generation or matching accuracy of the three-dimensional face model. .

  Therefore, the abnormal shadow region extraction unit 51 extracts a facial region in which the face is reflected in the input facial image, and measures the luminance distribution in the facial region, thereby causing a shadow caused by the structure of the subject's face. A normal shadow luminance range corresponding to the luminance range is determined. Then, the abnormal shadow region extraction unit 51 extracts a region including a pixel having a luminance outside the normal shadow luminance range as the abnormal shadow region.

  First, the abnormal shadow area extraction unit 51 extracts a face area from the input face image. Since the skin of the human face has a skin color, the shadow abnormal region extraction unit 51 extracts a region having a skin color in the input face image as a face region. For example, when the input face image is an RGB color image, the shadow abnormal region extraction unit 51 converts the value of each pixel of the input face image into the HSV color system, and the color information of the component of hue H is converted to skin color. A set of pixels included in a range of equivalent values (for example, 0 to 30 when the color information of the component of hue H is represented by 0 to 359) is set as a face region. Alternatively, the shadow abnormal region extraction unit 51 can project the three-dimensional face shape data on the input face image after aligning the three-dimensional face shape data stored in the storage unit 4 and the input face image. May be used as the face area. The alignment process between the 3D face shape data and the input face image can be the same as the alignment process executed by the 3D face model generation unit 53. Therefore, for details of the alignment processing, refer to the description of related processing of the three-dimensional face model generation unit 53.

Next, the abnormal shadow region extraction unit 51 calculates the average value μ and the variance σ ² of the luminance values assuming that the distribution of the luminance values of the pixels included in the face region follows a normal distribution. The abnormal shadow region extraction unit 51 sets the normal shadow luminance range to a range of ± ασ with the average value μ as the center. That is, the abnormal shadow region extraction unit 51 determines, among the pixels in the face region, a pixel in which the absolute value | I−μ | of the difference between the luminance value I and the average value μ satisfies the following relationship: It is extracted as a pixel having a luminance deviating from the above.
| I-μ |> ασ (1)
Here, α is a constant, and is set to any value within the range of 2.0 to 4.0, for example. As a result, a pixel having a luminance value greatly deviating from the luminance average value μ is extracted. For this reason, not only pixels with extremely low luminance values due to overlapping shadows and face areas caused by structures, etc., but also direct sunlight hits a part of the face and the reflected light from that part is reflected on the face. The pixel having a very high luminance value is also extracted by, for example, deviating from the dynamic range of the image sensor of the imaging unit 2.
The shadow abnormal region extraction unit 51 extracts a region constituted by pixels extracted according to the equation (1) as a shadow abnormal region.

  In addition, the image processing unit 5 calculates in advance the luminance average value and variance of the face area extracted from one or more input face images without the abnormal shadow area, which are captured by the imaging unit 2 under the same illumination conditions. In addition, the luminance average value and the variance may be stored in the storage unit 4. Then, the shadow abnormal region extraction unit 51 normally uses the average luminance value and the variance read from the storage unit 4 when the input face image of the registration target person is obtained to create the registration target person's three-dimensional face model. An abnormal shadow region may be extracted by defining the shadow luminance range and performing the calculation of equation (1). As described above, the abnormal shadow region extracting unit 51 can set the normal shadow luminance range more appropriately than the normal shadow luminance range by defining the normal shadow luminance range using the luminance average value and variance calculated from the face region without the abnormal shadow region. Therefore, it is possible to extract an abnormal shadow region more accurately.

In this case, the image processing unit 5 may change the illumination conditions in various ways, calculate the luminance average value and variance of the face area without the shadow abnormal area for each illumination condition, and store them in the storage unit 4. Good. For example, each illumination condition is classified according to a shooting time zone. Then, the shadow abnormal region extraction unit 51 uses the luminance average value and the variance associated with the illumination condition corresponding to the shooting time period including the shooting time of the input face image for extraction of the shadow abnormal region.
Or each illumination condition may be classified by the brightness | luminance on the input face image of the fixed structure contained in the monitoring area | region image | photographed by the imaging part 2. FIG. In this case, the shadow abnormal area extraction unit 51 obtains the average brightness value of the area in which the fixed structure appears in the input face image, and the average brightness value and the variance associated with the illumination condition corresponding to the average brightness value. Is used for extraction of shadow abnormal areas.
Thereby, the shadow abnormal area extraction unit 51 determines the normal shadow luminance range, and the average luminance value of the face area without the shadow abnormal area calculated under the lighting conditions similar to the lighting conditions at the time of photographing the input face image And dispersion can be used, so that the normal shadow luminance range can be set more appropriately.

Further, the abnormal shadow region extraction unit 51 may extract the abnormal shadow region by comparing the shadow image created from the three-dimensional face shape data stored in the storage unit 4 with the input face image.
For this purpose, a hypothetical light source is assumed to be a predetermined position, and a shaded three-dimensional face model having a shadow formed when light is emitted from the virtual light source at the assumed position to the three-dimensional face shape data is created in advance. A plurality of such shaded three-dimensional face models are created in advance by changing the position and irradiation direction of the virtual light source, and stored in the storage unit 4.
The shadow abnormal area extraction unit 51 aligns the input face image with each shadow three-dimensional face model, and then projects the shadow three-dimensional face model onto a two-dimensional plane to create an ideal shadow image. . When creating a shaded three-dimensional face model, a structure that blocks light from the virtual light source is not defined. Therefore, no shadow other than the shadow caused by the face shape represented in the three-dimensional face shape data is generated on the ideal shadow image. The alignment between the input face image and the shaded 3D face model is the same as the alignment process between the input face image and the 3D face shape data executed by the 3D face model generation unit 53 described later, for example. It can be executed by processing. Therefore, for details of the alignment process, refer to the related description about the three-dimensional face model generation unit 53.

The shadow abnormal area extraction unit 51 calculates a difference ΔI in luminance value between each pixel in the face area in the input face image and a pixel at a corresponding position in each ideal shadow image. Shadow abnormal region extraction unit 51, usually a shade luminance range, and less than the threshold value theta _l theta _h for [Delta] I. That is, the abnormal shadow region extraction unit 51 extracts pixels in which ΔI exceeds the predetermined threshold θ _h and pixels in which ΔI is less than the predetermined threshold θ _l as pixels outside the normal shadow luminance range. Then, the shadow abnormal region extraction unit 51 sets a region constituted by the extracted pixels as a shadow abnormal region.
Note that the threshold values θ _h and θ _l are preferably determined based on the distribution of the difference value ΔI calculated for each pixel in the face area of the input face image. For example, the shadow abnormal region extraction unit 51 calculates the difference average value μ ′ and the difference variance σ ′ ² on the assumption that the distribution of the difference value ΔI follows a normal distribution. At this time, the threshold values θ _h and θ _l are set as follows.
θ _h = μ '+ βσ'
θ _l = μ'-βσ '
However, (beta) is a constant and is set to any value within the range of 2.0-4.0, for example.
Alternatively, the threshold values θ _h and θ _l may be values determined in advance by simulation or experiment, for example. However, θ _l <θ _h .

Note that, as disclosed in Japanese Patent Application Laid-Open No. 2009-2111151, the abnormal shadow region extraction unit 51 estimates a light source direction when an input face image is captured, and an ideal corresponding to the estimated light source direction. An abnormal shadow region may be extracted using only the shadow image.
The shadow abnormal region extraction unit 51 passes information representing the extracted shadow abnormal region to the face image selection unit 52. Note that the information indicating the shadow abnormal region is, for example, a ternary image having the same size as the input face image, and is included in the face value other than the pixel value (eg, “0”) included in the shadow abnormal region and the shadow abnormal region. The pixel value (for example, “1”) and the pixel value (for example, “2”) included in other regions may have different values.

The face image selection unit 52 is an example of a shadow abnormality determination unit, and determines whether or not a shadow due to a part other than the face structure of the person to be registered exists in the face area of the input face image.
For example, when the ratio (S _abn / S _face ) of the area S _abn of the abnormal shadow area to the area S _face of the face area of the input face image is equal to or less than a predetermined threshold θ _s , the face image selection unit 52 inputs that In the face area of the face image, it is determined that there is no shadow due to other than the face structure of the person to be registered, and the input face image is stored in the storage unit 4 for use in generating a three-dimensional face model. On the other hand, when the ratio (S _abn / S _face ) is larger than the predetermined threshold value θ _s , the face image selection unit 52 causes the face area of the input face image to be shaded due to other than the face structure of the person to be registered. The input face image is discarded without being used for generating the three-dimensional face model.
The threshold value θ _s is set to any value within the range of 0.1 to 0.3, for example.

In addition, the face image selection unit 52 performs three-dimensional input face image processing depending on the position of the abnormal shadow area, even if the ratio (S _abn / S _face ) of the abnormal shadow area to the facial area is equal to or less than the threshold θ _s. You may discard, without using for the production | generation of a face model. For example, the face image selection unit 52 determines whether the center of the shadow abnormal region in the horizontal direction is within a predetermined distance from the center of the face region in the horizontal direction, or the face, such as an eye, nose, or mouth, in the shadow abnormal region. If any of the facial feature points related to the characteristic part is included, the input face image is discarded without being used for generating the three-dimensional face model. The predetermined distance can be any value within a range of 1/10 to 1/3 of the horizontal width of the face area, for example. Further, the method for extracting the facial feature points from the input face image can be the same as the method executed as part of the alignment process executed by the three-dimensional face model generation unit 53 described later. Therefore, please refer to the related description about the three-dimensional face model generation unit 53 for details of the method of extracting the face feature points from the input face image.

The three-dimensional face model generation unit 53 maps the input face image determined by the face image selection unit 52 that there is no shadow due to other than the face structure of the person to be registered in the face area to the three-dimensional face shape data. By doing so, a three-dimensional face model of the person to be registered is generated.
Therefore, the three-dimensional face model generation unit 53 extracts face feature points from the input face image.
In the present embodiment, the three-dimensional face model generation unit 53 extracts face feature points corresponding to each 3D face feature point (15 locations such as the eye region center, nose tip, mouth corner point) stored in the storage unit 4. To do. The three-dimensional face model generation unit 53 can use various known methods for extracting face feature points from the input face image. For example, the three-dimensional face model generation unit 53 performs edge extraction processing on the input face image to extract edge pixels having a large difference in pixel value from surrounding pixels. Then, the three-dimensional face model generation unit 53 checks whether or not the feature amount obtained based on the position and pattern of the edge pixel satisfies a predetermined condition for the part such as the eyes, the nose, and the mouth. Each facial feature point can be extracted by specifying the position of. In addition, the 3D face model generation unit 53 performs Gabor transform processing or wavelet transform processing instead of performing edge extraction processing to extract edge pixels, and detects pixels having large local changes in a plurality of different spatial frequency bands. It may be extracted. Further, the three-dimensional face model generation unit 53 may extract face feature points by performing template matching between a template corresponding to each part of the face and the input face image to identify the position of each part of the face. . Furthermore, the three-dimensional face model generation unit 53 causes the user to specify the position of each face feature point via a user interface (not shown) configured by a keyboard, a mouse, a display, and the like, thereby May be obtained.

  The 3D face model generation unit 53 uses each face feature point extracted from the input face image and the 3D face feature point related to the 3D face shape data to rotate and enlarge / reduce the 3D face shape data. The input face image and the 3D face shape data are aligned. Then, the three-dimensional face model generation unit 53 creates alignment information as a result of the alignment. The alignment information includes, for example, a rotation angle along each axis of the three-dimensional orthonormal coordinate system (X, Y, Z) of the three-dimensional face shape data, a translation amount, and an enlargement / reduction ratio. In this orthonormal coordinate system (X, Y, Z), for example, the center of gravity of a plurality of 3D face feature points on the 3D face shape data is set as the origin, and the horizontal direction is 3 (3 (From left to right in front of the three-dimensional face model) The X, X, and Z axes are orthogonal to the X-axis, X-axis, and Z-axis. The Z axis is set in the direction from the apex of the nose to the back of the head.

  For example, the three-dimensional face model generation unit 53 calculates the alignment information as follows. First, a 3D face feature point on the 3D face shape data is projected onto an input face image which is a 2D image. Then, the three-dimensional face model generation unit 53 obtains the sum of squares of differences between the positions of the projected 3D face feature points and the positions of the corresponding face feature points as a positional deviation amount. The three-dimensional face model generation unit 53 rotates, translates, enlarges or reduces the three-dimensional face shape data along each axis of the three-dimensional orthonormal coordinate system so that the amount of positional deviation is minimized. . The three-dimensional face model generation unit 53 uses the rotation angle, the translation amount, and the enlargement / reduction ratio of the three-dimensional face shape data when the positional deviation amount is minimized as the alignment information.

  When a plurality of 3D face shape data is stored in the storage unit 4, the 3D face model generation unit 53 sets the minimum value of the positional deviation amount with respect to the input face image for each 3D face shape data. Ask. Then, the three-dimensional face model generation unit 53 applies the three-dimensional face shape data that minimizes the minimum value of the positional deviation amount with respect to the input face image as three-dimensional face shape data to be applied to the registration target person reflected in the input face image. select. For the selection, for example, a technique disclosed in Japanese Patent Application Laid-Open No. 2009-212151 can be appropriately employed.

  The three-dimensional face model generation unit 53 projects each face feature point extracted from the input face image into the three-dimensional space, and then the amount of positional deviation from the 3D face feature point on the three-dimensional face shape data is minimized. The alignment information may be calculated by performing processing such as rotation and enlargement / reduction on the three-dimensional face shape data.

  The 3D face model generation unit 53 determines the face orientation of the 3D face shape data selected by the 3D face model generation unit 53 among the 3D face shape data stored in the storage unit 4 as an input face image. Adjust so that the face orientation is the same as the face orientation shown in. Therefore, the three-dimensional face model generation unit 53 rotates or translates the selected three-dimensional face shape data along each axis of the three-dimensional orthonormal coordinate system according to the corresponding alignment information, or enlarges it. Or reduce the size and create an orientation-adjusted shape model.

  The three-dimensional face model generation unit 53 creates a three-dimensional face model of the person to be registered in the input face image by mapping the input face image as a texture image on the orientation-adjusted shape model. That is, the registration information is used so that each face feature point extracted for the input face image and the 3D face feature point for the 3D face shape data selected as the orientation-adjusted shape model are aligned. The input face image is mapped as a texture image to the orientation-adjusted shape model. Note that the mapping is a process in which the value of each pixel of the input face image is set to a pixel value at a corresponding position in the three-dimensional face shape data.

  The 3D face model generation unit 53 stores the created 3D face model in the storage unit 4 in association with the identification code of the person to be registered. Alternatively, the 3D face model generation unit 53 may output the created 3D face model and the identification code of the person to be registered to another device via a communication interface (not shown).

FIG. 4 is a flowchart showing the operation of the three-dimensional face model generation process including the shadow abnormality determination process in the face image processing apparatus 1 according to one embodiment. This three-dimensional face model generation process is controlled by the image processing unit 5.
First, the image processing unit 5 acquires an input face image in which the face of the person to be registered is shown from the imaging unit 2 via the interface unit 3 (step S101). Then, the image processing unit 5 associates the acquired input face image with the identification code of the person to be registered, and passes it to the shadow abnormal region extraction unit 51 of the image processing unit 5.
The abnormal shadow area extraction unit 51 extracts a face area in which the face of the person to be registered is shown from the input face image (step S102). Then, the shadow abnormal area extraction unit 51 extracts pixels whose luminance values are out of the normal shadow luminance range in the face area, and sets an area constituted by the extracted pixels as a shadow abnormal area (step S103). The shadow abnormal region extraction unit 51 passes information representing the shadow abnormal region to the face image selection unit 52 of the image processing unit 5.

The face image selection unit 52 determines whether or not the ratio (S _abn / S _face ) of the area S _abn of the abnormal shadow area to the area S _{face of the} face area is equal to or less than a predetermined threshold θ _s (step S104). If the ratio (S _abn / S _face ) is larger than the predetermined threshold θ _s , the face image selection unit 52 discards the input face image (step S105). Then, the image processing unit 5 repeats the processes after step S101.
On the other hand, when the ratio (S _abn / S _face ) is equal to or less than the predetermined threshold θ _s , the face image selection unit 52 stores the input face image in the storage unit 4 (step S106).

Thereafter, the three-dimensional face model generation unit 53 of the image processing unit 5 aligns the input face image stored in the storage unit 4 with the three-dimensional face shape data, and positions each pixel value of the input face image. By mapping onto the combined 3D face shape data, a 3D face model of the person to be registered is generated (step S107).
The image processing unit 5 stores the created three-dimensional face model in the storage unit 4 in association with the registration target person identification code, or stores the three-dimensional face model together with the registration target person identification code via the communication network. Output to other devices.
Thereafter, the image processing unit 5 ends the three-dimensional face model generation process.

As described above, this face image processing apparatus has an abnormal shadow area by detecting pixels having an abnormally high or abnormally low luminance value in a face area in which a face is reflected in an input face image. It is determined whether or not to do. Thereby, this face image processing apparatus can determine whether it is an input face image image | photographed with the inappropriate illumination environment.
The face image processing apparatus creates an 3D face model by selecting an input face image in which no abnormal shadow area exists in the face area, or even if it exists, the proportion of the abnormal shadow area in the face area is small. . As described above, the face image processing apparatus creates a three-dimensional face model by selectively using an input face image photographed in an appropriate illumination environment, so that a three-dimensional face model suitable for collation can be created.
Further, since this face image processing apparatus selectively discards the input face image in which the shadow abnormality has occurred and does not use it for the creation of the three-dimensional face model, such a shadow abnormality may occur. An imaging unit can be installed to photograph a place. Therefore, the conditions regarding the place where the face image processing apparatus can be installed can be relaxed as compared with the conventional case.

  In addition, this invention is not limited to said embodiment. For example, the three-dimensional face model generation unit maps pixels included in a region excluding the abnormal shadow region (hereinafter referred to as a normal shadow region) within the face region of the input face image to the three-dimensional face shape data. A three-dimensional face model may be generated. In this case, a plurality of input face images are used for generating a three-dimensional face model. Then, in the shadow abnormal region that was not mapped when the first input face image was mapped to the 3D face shape data, the corresponding pixels in the normal shadow region of the other input face images were mapped to the 3D face shape data. Is done. As described above, the three-dimensional face model generation unit can generate a single three-dimensional face model by synthesizing normal shadow regions of a plurality of input face images.

Further, the face image processing apparatus may register the input face image determined to be stored by the image selection unit as it is as the registered face image of the person to be registered. In this case, the three-dimensional face model generation unit is omitted.
The present invention may also be applied to a face image matching device. In this case, the image processing unit includes a matching unit instead of the three-dimensional face model generation unit. Then, the registered face image of the user registered in advance is stored in the storage unit. Then, the shadow abnormal region extraction unit of the image processing unit extracts a shadow abnormal region included in the face region in the input face image photographed for verification by the imaging unit. When the image selection unit determines that the ratio of the area of the shadow abnormal region to the area of the face region is equal to or less than the threshold value, the matching unit performs a matching process between the input face image and the registered face image. To do. On the other hand, when the image selection unit determines that the ratio of the area of the shadow abnormal region to the area of the face region is larger than the threshold value, the image processing unit acquires the input face image from the imaging unit again, and the shadow is obtained. Anomaly extraction and verification processing is performed.
As described above, those skilled in the art can make various modifications in accordance with the embodiment to be implemented within the scope of the present invention.

DESCRIPTION OF SYMBOLS 1 Face image processing apparatus 2 Imaging part 3 Interface part 4 Memory | storage part 5 Image processing part 51 Shading abnormal area extraction part 52 Face image selection part 53 3D face model production | generation part

Claims (2)

An imaging unit that shoots the face of the subject and creates an input face image;
A storage unit for storing three-dimensional face shape data representing the shape of a person's face;
In the face area of the subject included in the input face image, a shadow abnormal area extracting unit that extracts a shadow abnormal area corresponding to a shadow other than the structure of the subject's face;
If the ratio of the shadow abnormal area to the face area is greater than a predetermined value, the shadow abnormality is determined to include a shadow due to other than the structure of the target person's face on the face of the target person in the input face image. A determination unit;
The input face image determined by the shadow abnormality determination unit that there is no shadow other than the structure of the subject's face in the subject's face is aligned with the three-dimensional face shape data, A 3D face model generation unit that generates a 3D face model of the subject by mapping to 3D face shape data;
A face image processing apparatus comprising:   The abnormal shadow region extraction unit includes pixels out of a normal shadow luminance range corresponding to a luminance range caused by a shadow generated due to a structure of the subject's face among pixels included in the face region. The face image processing apparatus according to claim 1, wherein the facial image processing apparatus extracts the shadow abnormal area.

Images