JP3135201B2 - Method and apparatus for extracting human mouth region - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for extracting a person mouth region from a front image of a person.

[0002]

2. Description of the Related Art Conventionally, when a person's mouth region is detected from a person image in a TV (television) conference or a TV telephone, or when a portrait of a person image is made, a person's mouth region is extracted from the person image. As a mouth region extraction method, for example, there is a cleft line detection method as disclosed in Japanese Patent Application Laid-Open No. 4-1863.

In this cleft line detection method, an image around the mouth is cut out from a person image obtained by photographing a normal face of a person. And
Pixels presenting image data darker than any of the pixels located vertically adjacent to each other in the y direction are detected, and a cleft line, which is a portion where the upper lip and the lower lip meet, is determined from a series of such pixels in the x direction. It is to detect.

[0004]

However, in the above-mentioned conventional method for extracting a person's mouth region, when a person image is photographed, the face of the person is a three-dimensional object, so that a highlight is placed under the nose due to the influence of lighting or the like. There are many cases where parts occur. In such a case, the contour of the image itself is lost, and there is a problem that it is very difficult to detect a cleft line which is a part of this contour.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a person / mouth area extracting method and apparatus which can easily extract a mouth area based on a person image whose contour has been lost due to the influence of illumination or image noise.

[0006]

According to a first aspect of the present invention, a rectangular area including a mouth is set based on a distance between both eyes in a front image of a person. The pixel value of a pixel adjacent to the target pixel based on the set binary image data of the rectangular area.
Then, the pixel value of the target pixel is reset and the outline of the image is changed.
Smoothing process for sharp and smooth
Thereafter, one or more partial regions of the rectangular region formed by connecting pixels having pixel values representing the mouth region are obtained, and the area and position of the obtained one or more partial regions are calculated. A partial area whose extracted area is larger than a predetermined value is extracted as a mouth candidate area, and a mouth candidate area located in a predetermined range in the rectangular area is determined as a mouth area based on the position of the extracted mouth candidate area. It is characterized by:

According to a second aspect of the present invention, in the method for extracting a person's mouth region according to the first aspect of the present invention, when a plurality of the determined mouth regions are present, the mouth region having the maximum area is determined. The feature is that it is adopted as the mouth area to be used.

According to a third aspect of the present invention, in the person / mouth region extracting method according to the first aspect of the present invention, if all of the candidate mouth regions are not determined to be mouth regions, the predetermined value is set to the predetermined value. Extracts a partial region having an area larger than a different second predetermined value as a mouth candidate region, and based on a position of the extracted mouth candidate region, a second predetermined range including the predetermined range in the rectangular region. A region obtained by integrating all the mouth candidate regions located in is defined as a mouth region.

According to a fourth aspect of the present invention, there is provided a person / mouth region extracting apparatus for extracting a person / mouth region based on image data of a front image of a person,
A rectangular area including the mouth is extracted based on the distance between the eyes, and extracted.
The binary image data of the output rectangular area is converted to rectangular area image data.
A rectangular area extracting unit to be stored in the storage unit, and the rectangular area image
Based on the binary image data stored in the data storage,
The above target pixel according to the pixel value of the pixel adjacent to the target pixel
Reset the pixel value of
Smoothing process to perform smoothing process
The step and the rectangular area after performing the smoothing process are
A region classifying unit that classifies one or more partial regions and other regions obtained by connecting pixels having pixel values representing the mouth region, a region area calculating unit that calculates the area of the partial region, and the partial region An area position calculation unit that calculates the position of
A mouth candidate region extraction unit for extracting a partial region having the calculated area larger than a predetermined value as a mouth candidate region; and a mouth located in a predetermined range in the rectangular region based on the position of the extracted mouth candidate region. A mouth area determination unit that determines that the candidate area is a mouth area is provided.

According to a fifth aspect of the present invention, in the human mouth area extracting apparatus according to the fourth aspect of the present invention, the rectangular area
The extraction unit obtains a reference point and a size of the rectangular area using a distance between both eyes in the front image of the person as a reference length, and calculates the rectangular shape from the image data of the front image based on the coordinates and the size of the reference point. binarizing cutting out image data of the area, and the obtained binary image data characterized by being adapted to store in the rectangular region image data storage unit.

According to a sixth aspect of the present invention, in the person / mouth area extracting apparatus according to the fourth aspect of the present invention, when the plurality of mouth candidate areas are determined to be mouth areas by the mouth area determination section, And a second mouth area determining unit that determines that the mouth area having the largest area is the target mouth area.

According to a seventh aspect of the present invention, in the person / mouth region extracting apparatus according to the fourth aspect of the present invention, when all the mouth candidate regions are determined not to be a mouth region by the mouth region determining section, A second mouth candidate region extracting unit that extracts a partial region having an area larger than a second predetermined value different from the predetermined value as a mouth candidate region, and a position of the extracted mouth candidate region, A mouth candidate area integrating unit is provided which integrates all mouth candidate areas located in a second predetermined range including the predetermined range in the rectangular area to obtain a mouth area.

[0013]

According to the first aspect of the present invention, a front image of a person is added to the image .
A rectangular area including the mouth is set based on the distance between the eyes and the target image is set based on the binary image data of the rectangular area.
The pixel of the target pixel according to the pixel value of the pixel adjacent to the element
Reset the value to sharpen and smooth the outline of the image
Is performed. After that, one or a plurality of partial regions of the rectangular region, which are obtained by connecting pixels having pixel values representing the mouth region, are obtained. Further, the area and the position of the obtained one or a plurality of partial regions are calculated. Then, a partial region having the calculated area larger than a predetermined value is extracted as a mouth candidate region. Based on the position of the extracted mouth candidate region, a mouth candidate region located in a predetermined range in the rectangular region is determined as a mouth candidate region. The area is determined. In this way, the mouth region is extracted regardless of the contour in the front image of the person.

In the invention according to claim 2, when a plurality of the mouth regions are determined, the mouth region having the largest area is adopted as the target mouth region. Thus, even if one large mouth region is not determined due to the influence of image noise, the mouth region is optimally extracted.

According to the third aspect of the present invention, if all of the mouth candidate areas are not determined to be mouth candidates,
All partial regions having an area larger than a second predetermined value different from the predetermined value are extracted as mouth candidate regions, and a second region in the rectangular region is determined based on the position of the extracted mouth candidate region. A region obtained by integrating the mouth candidate regions located in the predetermined range is determined as a mouth region. In this manner, the mouth area is accurately extracted by changing the reference area for extracting the mouth candidate area and the reference position for determining the mouth area.

Further, in the invention according to claim 4, the rectangular area
The distance between the eyes in the front image of the person by the extraction unit
The rectangular area including the mouth is extracted based on
Is stored in the rectangular area image data storage unit.
It is. Then, the rectangular processing is performed by the smoothing processing means.
Adjacent to the target pixel based on the binary image data of the
Resets the pixel value of the target pixel according to the pixel value of the target pixel
To make the outline of the image sharp and smooth
Zing processing is performed. Further, by the area classification unit,
The rectangular area after the above-described smoothing processing is classified into one or more partial areas formed by connecting pixels having pixel values representing the mouth area and other areas. Then, the area of the partial region is calculated by the region area calculation unit, and the position of the partial region is calculated by the region position calculation unit. After that, the partial region having the area larger than the predetermined value is extracted as a candidate region by the candidate mouth region extracting unit, and the candidate mouth region located in a predetermined range in the rectangular region is defined by the mouth region determining unit. Is determined. In this way, the mouth region is extracted regardless of the contour in the front image of the person.

Further, in the invention according to claim 5, the image of the person image is obtained by the rectangular area extraction unit based on the coordinates and dimensions of the reference point obtained using the distance between the eyes in the front image of the person as the reference length. The image data of the rectangular area is cut out from the data, binarized, and stored in the rectangular area image data storage. Thus, the rectangular area is automatically set based on the image data of the front image of the person, and the mouth area is extracted based on the rectangular area.

In the invention according to claim 6, when the plurality of mouth candidate areas are determined to be mouth areas by the mouth area determination section, the mouth area having the maximum area is determined by the second mouth area determination section. It is determined that the region is the target mouth region. Thus, even if one large mouth region cannot be obtained due to the influence of image noise, the mouth region is extracted optimally.

Further, in the invention according to claim 7, when all of the mouth candidate regions are determined not to be mouth regions by the mouth region determination unit, the second mouth candidate region extraction unit determines
A partial region having an area larger than the second predetermined value is extracted as a mouth candidate region. Then, the mouth candidate area integration unit integrates all the mouth candidate areas located in the second predetermined range in the rectangular area to obtain a mouth area. In this manner, the mouth area is accurately extracted by changing the reference area for extracting the mouth candidate area and the reference position for determining the mouth area.

[0020]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the illustrated embodiments. FIG. 1 is a block diagram of the person / mouth area extracting apparatus according to the present embodiment.

Reference numerals 4 to 7 denote image memories. here,
The image memory 4 is an image memory for storing an original image, the image memory 5 is an image memory for storing an initial processing area including a mouth area in the original image, and the image memory 6 stores image data of the initial processing area. The image memory 7 stores the binarized data. The image memory 7 stores the binary image data of the label area, which is a labeled partial area of the initial processing area based on the binary image data stored in the image memory 6. This is an image memory for storing.

Reference numerals 8 to 10 denote work memories. here,
The work memory 8 is a work memory that stores the area of each of the label areas. The work memory 9 is a work memory that stores the position of each of the label areas.
Reference numeral 0 denotes a work memory that stores a flag array described later in detail based on the binary image data of each of the label areas stored in the image memory 7.

The above image memories 4 to 7
Each of the work memories 8 to 10 is constituted by a random access memory (RAM).

The initial processing area extracting unit 12 extracts the image data of the initial processing area from the image data of the original image stored in the image memory 4 and stores it in the image memory 5.
The binarization processing unit 13 converts the image data of the initial processing area stored in the image memory 5 into a value of 0 or 1 according to the density value, and stores the obtained binary image data in the image memory 6. The density histogram calculator 15 calculates a density histogram based on the image data of the initial processing area stored in the image memory 5. Threshold calculator 14
Calculates the threshold value of the density used in the binarization processing based on the density histogram,
To supply.

The 4-neighbor expansion unit 16 expands the binary image data of the initial processing area stored in the image memory 6 in the vicinity of four as described later in detail, and updates the contents of the image memory 6. I do. The four-neighbor contraction unit 17 contracts the binary image data of the initial processing area stored in the image memory 6 near four, and updates the contents of the image memory 6.

The area classifying unit 18 stores the image memory 6
As described in detail below, the binary image data of the initial processing area stored in the above is subjected to an 8-neighborhood area connection process to connect the target pixel and the nearby pixel and add a label to the label area. The binary image data is stored in the image memory 7. The area area calculation unit 19 calculates the area of each label area based on the binary image data for each label area stored in the image memory 7 and stores the area in the work memory 8. The area position calculation unit 20 calculates the position of each label area based on the binary image data for each label area stored in the image memory 7 and stores the calculated position in the work memory 9.

The mouth candidate area extracting unit 21 extracts a label area whose area exceeds a predetermined value as a mouth candidate area based on the value of the area of the label area stored in the work memory 8. The mouth region first determination unit 22 determines whether the mouth candidate region extracted by the mouth candidate region extraction unit 21 is appropriate as a mouth region based on the area and position values of the mouth candidate region. The mouth region second determination unit 23 is a mouth region exhibiting the maximum area when there are a plurality of mouth regions extracted by the mouth candidate region extraction unit 21 and determined to be valid by the mouth region first determination unit 22. Is determined to be the mouth area for the purpose. The mouth candidate area integration unit 24 selects the mouth candidate area by changing the above-mentioned predetermined value when all the label areas are determined not to be valid as the mouth area by the mouth area first determination unit 22. All the mouth candidate areas in which the positions of the mouth candidate areas are within a predetermined range are integrated and extracted as mouth candidate areas.

The control unit 1 has a read only memory (RO)
M) The respective sections are controlled according to the program stored in M) 11 to execute a person / mouth area extraction processing operation. The control section 1 and each section are constituted by a CPU (Central Processing Unit). Reference numeral 3 denotes a control memory used for various processes executed under the control of the control unit 1.
The exchange of various data and the like between the control unit 1, the control memory 3, and each unit is performed via the CPU bus 2.

FIGS. 2 to 6 are flowcharts of a person / mouth area extraction processing operation executed by each section of the initial processing area extraction section 12 to the mouth area first determination section 22 under the control of the control section 1. . FIG. 2 shows the initial processing area extraction unit 12.
FIG. 3 is a flowchart of an initial processing area extraction processing operation executed by the density histogram extraction unit 15.
4 is a flowchart of a binarization processing operation executed by a threshold value calculation unit and a binarization processing unit; FIG. 4 is a flowchart of the label area extraction processing operation executed by the 4-neighbor expansion section 16, the 4-neighbor contraction section 17, the area classification section 18, the area area calculation section 19, and the area position calculation section 20. FIG. 5 is a flowchart of the mouth candidate region extraction processing operation performed by the mouth candidate region extraction unit 21. FIG. 6 is a flowchart of the mouth candidate region validity determination processing operation performed by the mouth region first determination unit 22. It is.

In the following description, the value of each pixel of each image in the image data is described by i (x, y). here,
i (x, y) is a density value in the range of 0 ≦ i (x, y) ≦ (Np−1). (X, y) represents two-dimensional coordinates with the origin at the upper left of the screen, and the range of the coordinate values is 0 ≦ x ≦ (xsize−
1), 0 ≦ y ≦ (ysize−1). Here, Np is the number of density values i of pixels, xsize is the number of pixels in the x direction, and ysize is the number of pixels in the y direction. Hereinafter, each processing operation will be sequentially described with reference to FIGS.

First, according to FIG. 2, the initial processing region extracting unit 12 extracts a rectangular initial processing region R0 including a mouth image based on the image data of the original image as shown in FIG. The region extraction processing operation will be described in detail. In step S1, the image data of the original image is stored in the image memory 4. Here, the original image is
It is the image which image | photographed the upper body of the person who turned to the front as shown in FIG. In step S2, the coordinates (x1, y1) of the center point P1 of the right eye and the coordinates (x2, y2) of the center point P2 of the left eye are input based on the image data of the original image. Step S3
Then, based on the coordinates of the center point P1 of the right eye and the coordinates of the center point P2 of the left eye, the reference length L is calculated as the distance P1P2.

In step S4, the width W and height H of the rectangular area
Is calculated as follows. W = 6 * L / 5 H = 3 * L / 5 In step S5, the coordinates of the upper left point P3 in the above rectangular area
(x3, y3) is calculated as follows. x3 = (x1 + x2) / 2-W / 2 y3 = (y1 + y2) / 2 + 4 * L / 5) In step S6, the coordinates of the point P3 calculated as described above.
The values of (x3, y3), the width W and the height H of the rectangular area are stored in the image memory 5. Thus, the rectangular area (W, H) with the point P3 at the upper left is registered as the initial processing area R0.

In this embodiment, the initial processing area R0 is extracted from the center coordinates of the eyes in the original image.
It does not matter if the operator directly specifies the position of the point P3 and the rectangular area (W, H) to extract the initial processing area R0, or directly obtains the initial processing area R0 from the original image and stores it in the image memory 5. Absent.

As described above, when the initial area extraction processing operation by the initial processing area extraction section 12 is completed, the binarization executed by the density histogram extraction section 15, the threshold value calculation section 14, and the binarization processing section 13 is performed. Move to processing operation. Next, the binarization processing operation will be described in detail with reference to FIG. In step S7, the density histogram of the initial processing region R0 is extracted by the density histogram extracting unit 15. Here, the density histogram is a one-dimensional array indicating the number of pixels presenting each density value i in the image, and is represented by h [i] {i = 0,..., Np−1}. After that, the process proceeds to the threshold calculation process by the threshold calculator 14.

In step S8, first, the area S of the initial processing region R0 is calculated. Here, S = W * H. In step S9, the sum value Sum of the density value i and the density histogram is initialized to "0", and the threshold value Sth of the area S is calculated by the following equation. Sth = S * per Note that “per” is a constant satisfying 0.0 ≦ per ≦ 1.0, and in this embodiment, it is assumed that per = 0.2. In step S10,
It is determined whether or not the density value i is the maximum value (Np-1). As a result, if the density value i is the maximum value (Np-1), the process proceeds to step S14; otherwise, the process proceeds to step S11. In step S11, the value of the density histogram h [i] of the density i is added to the sum value Sum of the density histogram. In step S12, it is determined whether or not the sum Sum of the density histogram calculated in step S11 is equal to or greater than the area threshold Sth calculated in step S9. As a result, if it is equal to or larger than the area threshold value Sth, step S14 is performed.
If the value is smaller than the area threshold value Sth, the process proceeds to step S13. After the density value i is incremented in step S13, the process returns to step S10. In step S14, the density value i is determined as the threshold value th for binarizing the image data. After that, the process proceeds to the binarization processing by the binarization processing unit 13.

In step S15, the binarization processing section 13 binarizes the image data in the initial processing area R0 based on the threshold value th determined in step S14. The binary image data thus obtained is stored in the image memory 6. Here, the binarization of the image data is a process of comparing the image data of each pixel in the image with a threshold th, and converting the image data into a binary value based on the comparison result. Here, the pixel value of a pixel having a density value equal to or less than the threshold value th is converted to “1”, while the pixel value of a pixel having a density value larger than the threshold value th is converted to “0”.

That is, the rectangular area extracting unit according to the fifth aspect is the initial processing area extracting unit 1 in this embodiment.
2, a concept including a density histogram extraction unit 15, a threshold value calculation unit 14, and a binarization processing unit 13.

As described above, when the binarization processing operation executed by the density histogram extraction unit 15, the threshold value calculation unit 14, and the binarization processing unit 13 is completed, the 4-neighbor expansion unit 16, the 4-neighbor contraction unit The process proceeds to a label region extraction processing operation executed by the region classification unit 18, the region area calculation unit 19, and the region position calculation unit 20. Next, the label region extraction processing operation will be described in detail with reference to FIG.

In step S 16, the binary image data is read from the image memory 6 by the 4-neighbor expansion unit 16, and the binary image R 1 is read and stored in the control memory 3. In step S17, 4-neighbor expansion processing is performed on the read binary image data, and the image memory 6 is updated with the result of the four-neighbor expansion processing. Here, the 4-neighbor expansion processing refers to a case where the pixel value of the target pixel is set to 1 when at least one pixel presenting a pixel value of 1 exists in 4 pixels adjacent to the pixel to be processed, and In this case, the pixel value of the target pixel is set to 0. In step S18, the 4-neighborhood shrinking unit 17 performs 4-neighborhood shrinkage processing on the binary image data read from the image memory 6 and stored in the control memory 3 after the 4-neighborhood expansion processing. The image memory 6 is updated with the result of the neighborhood contraction processing. Here, the four-neighbor contraction processing is to set the pixel value of the target pixel to 0 when at least one pixel presenting a pixel value of 0 exists in four pixels adjacent to the target pixel; This is processing for setting the pixel value of a pixel to 1. In this way, by performing the above-described four-neighbor expansion processing and four-neighbor contraction processing, the pixels having the pixel value “0” scattered at the peripheral end of the mouth shape formed by the set of the pixel values “1” are converted to the pixel value “1” To sharpen the outline of the mouth area.

In step S19, the area classifying unit 18 performs an 8-neighboring operation on the binary image data that has been read from the image memory 6 and stored in the control memory 3 after the 4-neighbor expansion processing and the 4-neighbor contraction processing. An area connection process is performed. Here, the 8-neighborhood region connection process is a process of connecting a target pixel and pixels having the same pixel value as the target pixel in eight pixels adjacent to the target pixel to form the same region. In step S20, a label number j is sequentially added to each area obtained in step S19 to obtain a label area L [j] {j = 1,..., Nb}. Then, the obtained label area L [j] is stored in the image memory 7 as a processing target area. Here, Nb is the number of the label number j.

In step S21, the area area calculation unit 19
Thus, the area Larea [j] {j = 1, 1, of each label area L [j]
.., Nb} are calculated and stored in the work memory 8. In step S22, the region position calculation unit 20 causes the left end x coordinate Sp [j] {j = 1,..., Nb} and the right end x coordinate Ep [j] {j = 1,. , Nb} is calculated and stored in the work memory 9.

As described above, the above-described 4-neighbor expansion portion 1
When the label region extraction processing operation performed by the 6, 4 neighborhood contraction unit 17, the region classification unit 18, the region area calculation unit 19, and the region position calculation unit 20 ends, the mouth candidate region extraction unit 2
Then, the process proceeds to the mouth candidate region extraction processing operation executed by Step 1. Next, the mouth candidate region extraction processing operation will be described in detail with reference to FIG.

In step S23, the mouth candidate area extracting unit 2
1, the area Larea [j] {j = 1,..., Nb} of each label area L [j] is read from the work memory 8 and stored in the control memory 3. In step S24, the label number j is initialized to "1". In step S25, the area Lare
It is determined whether a [j] is greater than a predetermined threshold th1.
As a result, if it is larger than the threshold th1, the process proceeds to step S27, and if it is smaller than the threshold th1, the process proceeds to step S26.

In step S26, it is determined whether or not the area Larea [j] is larger than a predetermined threshold th2 different from the threshold th1. As a result, if it is larger than the threshold th2, step S2
The process proceeds to step S8, and if it is equal to or smaller than the threshold th2, the process proceeds to step S29. In step S27, the flag array Lflag [j] {j = 1,.
"on 1" is set to the element Lflag [j] of the label number j in {b}. In step S28, "ON 2" is set in the flag array element Lflag [j]. Step S
At 29, "off" is set in the flag array element Lflag [j].

In step S30, it is determined whether or not the label number j is equal to the number of labels (the maximum value of the label number j) Nb. As a result, when they are equal, the process proceeds to step S32, and when they are different, the process proceeds to step S31. Step S31
Then, the label number j is incremented, and the process returns to step S25, where the area Larea [j] of the next label area L [i] is obtained.
Shifts to processing related to In step S32, the flag array Lflag [j] {j = 1,..., Nb} is stored in the work memory 10. As a result, the flag area L [i] corresponding to the flag array element Lflag [j] in which the “ON 1” is set is:
It is extracted as a mouth candidate area.

As described above, when the mouth candidate area extraction processing operation by the mouth candidate area extraction unit 21 ends, the process proceeds to a mouth candidate area validity determination processing operation executed by the mouth area first determination unit 22. Next, the operation of the mouth candidate area validity determination processing will be described in detail with reference to FIG.

In step S33, the mouth area first determination unit 2
2, the flag array Lflag
[j] {j = 1,..., Nb} are read out and stored in the control memory 3. In step S34, the mouth area first determination unit 22
Thus, from the work memory 9, the left end x coordinate Sp [j] {j = 1,..., Nb} and the right end x coordinate Ep [j] of each label area L [j]
{j = 1,..., Nb} are read and stored in the control memory 3. In step S35, the flag flag is initialized to "0". In step S36, the label number j is initialized to "1".

In step S37, the flag array element Lfl
It is determined whether or not “on 1” is set in ag [j]. As a result, if "ON 1" has been set, the process proceeds to step S38, and if not, the process proceeds to step S40. In step S38, the left end x of the label area L [j]
The coordinate Sp [j] is smaller than the predetermined value sx, and the right end x coordinate E
It is determined whether or not the condition that p [j] is larger than a predetermined value ex is satisfied. As a result, if the condition is satisfied, the process proceeds to step S39; otherwise, the process proceeds to step S40. In step S39, 1 is added to the value of the flag flag.

In step S40, it is determined whether or not the label number j is equal to the label number (maximum value) Nb. As a result, if they are equal, the process proceeds to step S42; otherwise, the process proceeds to step S41. In step S41, the label number j
Is incremented. After that, step S
Return to 37.

In step S42, it is determined whether the content of the flag flag is "0". As a result, if “0”, the process proceeds to the mouth candidate region integration processing operation; otherwise, the process proceeds to step S43. In step S43, the flag flag
Is determined as to whether or not the content is "1". as a result,
If "1", the process proceeds to step S44, and if not, the process proceeds to the mouth region determination processing operation. In step S44, the fact that the content of the flag flag is "1" means that the area Larea [j] is larger than the predetermined threshold th1 and the left end x coordinate Sp
[j] is smaller than a predetermined value sx, and the right end x coordinate Ep [j] is a predetermined value.
The label area L [j] satisfying the condition of being larger than ex is
It means that there is only one. Therefore, the only label area L [j] in which “ON 1” is set in the flag array element Lflag [j] is determined to be valid as a mouth area, and “ON 3” is set in the flag array element Lflag [j]. "Is reset. In step S45, the flag array element Lflag [j]
The image memory 7 is updated with the label area L [j] in which "ON 3" is set, and the mouth area is registered. As described above, when the mouth candidate region validity determination processing operation by the mouth region first determination unit 22 ends, the human mouth region extraction processing operation ends.

FIG. 7 is a flowchart of the mouth area determination processing operation executed by the mouth area second determination section 23 under the control of the control section 1. Hereinafter, the above-described mouth region determination processing operation will be described in detail with reference to FIG. If it is determined in step S43 in the flowchart of the above-mentioned person / mouth region extraction processing operation that the number greater than or equal to “2” is set in the flag “flag”, the process proceeds to a mouth region determination processing operation.

In step S46, the mouth area second determination unit 2
3, the flag array Lflag
[j] {j = 1,..., Nb} are read out and stored in the control memory 3. In step S47, the mouth area second determination unit 23
Thus, the area Larea [j] {j = 1,..., Nb} of each label area L [j] is read from the work memory 8 and stored in the control memory 3. In step S48, the label number j
Is set to the initial value “1”, and the initial value “0” is set to the maximum area max.

In step S49, the flag array element Lfl
It is determined whether or not “on 1” is set in ag [j]. As a result, if “ON 1” is set, the process proceeds to step S50, and if not, the process proceeds to step S52. In step S50, it is determined whether or not the area Larea [j] is larger than the maximum area max. As a result, if it is larger, the process proceeds to step S51, and if not, the process proceeds to step S52. In step S51, the area Larea [j] is set to the maximum area max, and the label number j is set to the mouth area label number mj.

In step S52, it is determined whether or not the label number j is equal to the label number (maximum value) Nb. As a result, if they are equal, the process proceeds to step S54; otherwise, the process proceeds to step S53. In step S53, the label number j
Is incremented. After that, step S
Return to 49.

In step S54, the label area L [mj] corresponding to the label number exhibiting the maximum area max (that is, the mouth area label number mj) is determined to be a mouth area, and the flag array element Lflag [mj] is determined. ] Is reset to “ON 3”. When the mouth area determination processing operation is completed, the process proceeds to step S45 in the flowchart of the above-described person / mouth area extraction processing operation, and the label area L [mj] is registered in the image memory 7 as a mouth area.

FIG. 8 is a flowchart of the mouth candidate area integration processing operation executed by the mouth candidate area integration section 24 under the control of the control section 1. Hereinafter, the mouth candidate area integration processing operation will be described in detail with reference to FIG. If it is determined in step S42 in the flowchart of the person / mouth area extraction processing operation that the flag "flag" is set to "0", the process proceeds to a mouth candidate area integration processing operation.

In step S55, the mouth candidate area integrating unit 2
4, the flag array Lflag
[j] {j = 1,..., Nb} are read out and stored in the control memory 3. In step S56, the mouth candidate area integration unit 24
Thus, from the work memory 9, the left end x coordinate Sp [j] {j = 1,..., Nb} and the right end x coordinate Ep [j] of each label area L [j]
{j = 1,..., Nb} are read and stored in the control memory 3. In step S57, the label number j is initialized to "1".

In step S58, the flag array element Lfl
It is determined whether "on 1" or "on 2" is set in ag [j]. As a result, if set, the process proceeds to step S59; otherwise, the process proceeds to step S63. In step S59, whether or not the condition that the left end x coordinate Sp [j] of the label area L [j] is smaller than the predetermined value sx and the right end x coordinate Ep [j] is larger than the predetermined value ex is satisfied. Is determined. As a result, if the condition is satisfied, the process proceeds to step S62; otherwise, the process proceeds to step S60.
In step S60, the left end x coordinate Sp of the label area L [j]
[j] is larger than the predetermined value sx and the right end x coordinate Ep
It is determined whether or not the condition that [j] is smaller than the predetermined value ex is satisfied. As a result, if it is satisfied, step S
Proceeding to step 62, if not satisfied, proceeding to step S61.
In step S61, the left end x coordinate Sp of the label area L [j]
[j] is smaller than the predetermined value ex and the right end x coordinate Ep
It is determined whether or not the condition that [j] is larger than the predetermined value sx is satisfied. As a result, if it is satisfied, step S
Proceed to 62, and if not satisfied, proceed to step S63.

In step S62, the label area L [j] that satisfies any of the conditions in step S59, step S60 or step S61 is determined to be a mouth area, and the flag array element Lflag [j] is set to "ON". 3 "is reset.
In step S63, the label number j is the number of labels (maximum value).
It is determined whether it is equal to Nb. As a result, if they are equal, the mouth candidate area integration processing operation ends, and if not, the process proceeds to step S64. In step S64, the label number j
Is incremented, and the process returns to step S58. If it is determined in step S63 that the label number j is equal to the label number Nb, the mouth candidate area integration processing operation is terminated.

When the mouth candidate area integration processing operation is completed in this manner, the process proceeds to step S45 in the flowchart of the above-described person / mouth area extraction processing operation, and all label areas L [j] for which “ON 3” is set are displayed. It is integrated as a candidate area and registered in the image memory 7.

As described above, in the present embodiment, the coordinates (x1, y1) of the center P1 of the right eye and the coordinates (x2, y) of the center P2 of the left eye are obtained from the front image of the person by the initial processing area extraction unit 12.
The rectangular area including the mouth of the person is obtained by setting the distance P1P2 between 2) and the reference length L as the initial processing area R0. The binarization processing unit 13 uses the threshold value calculated by the threshold value calculation unit 14 based on the density histogram h [j] {j = 0,..., Np−1} calculated by the density histogram extraction unit 15. The image data in the initial processing area R0 is binarized. Further, the four-neighbor expansion unit 16 performs four-neighbor expansion processing and four-neighbor contraction unit 17 on the binary image data in the initial processing region R0.
After performing the four-neighbor contraction process, the region classifying unit 18 performs the eight-neighbor region linking process and assigns a label to obtain a label region L [j]. Then, the area Larea [j] calculated by the area area calculation unit 19 by the mouth candidate area extraction unit 21
A mouth candidate region is extracted based on a comparison result between the threshold value th1 and the threshold value th2.

As a result, the area La larger than the threshold th1 is obtained.
When a label region L [j] exhibiting rea [j] is extracted,
A label region that satisfies the condition that the left end x coordinate Sp [j] calculated by the region position calculation unit 20 by the mouth region first determination unit 22 is smaller than the predetermined value sx and the right end x coordinate Ep [j] is larger than the predetermined value ex. If there is only one L [j], it is determined that the label area L [j] is a mouth area. When there are a plurality of label areas L [j] satisfying the above conditions, the mouth area second determination unit 23 determines that the label area L [j] having the largest area is the mouth area. Also,
When the label area L [j] exhibiting the label area Larea [j] larger than the threshold th1 is not extracted, but the label area L [j] exhibiting the label area Larea [j] larger than the threshold th2 is extracted. Is calculated by the mouth candidate area integration unit 24.
All label areas L [j] in which the left end x coordinate Sp [j] and the right end x coordinate Ep [j] calculated by the area position calculation unit 20 have a specific positional relationship with the above-mentioned predetermined values ex, sx are integrated. To be the mouth area.

As described above, the initial processing area R set based on the distance L between the center coordinates of the right and left eyes in the human image
Since the mouth area is extracted based on the binary image data of 0,
The mouth region can be extracted regardless of the outline of the person image. Therefore, according to the present embodiment, it is possible to easily extract the mouth region based on a human image whose contour has been lost due to the influence of illumination and image noise.

[0064]

As is apparent from the above description, the method for extracting a person's mouth region according to the first aspect of the present invention includes a method for extracting a frontal image of a person.
A rectangular area including the mouth based on the distance between the eyes
Neighboring based on the binary image data of the set rectangular area
Smoothin that resets the pixel value according to the pixel value of the touching pixel
After performing the pixel processing, pixels having pixel values representing the mouth region are connected to each other to obtain one or more partial regions of the rectangular region, the area and the position of the partial region are calculated, and the area is a predetermined value. A larger partial area is extracted as a mouth candidate area,
Since the mouth candidate area located in the predetermined range in the rectangular area is determined as the mouth area, the mouth area can be extracted from the image data of the front image of the person regardless of the outline of the image.
Therefore, according to the present invention, it is possible to easily extract a mouth region based on a human image whose contour has been lost due to the effects of lighting and image noise.

In the person / mouth region extracting method according to the second aspect of the present invention, when there are a plurality of determined mouth regions, the mouth region having the largest area is adopted as the target mouth region. Even if one large mouth area cannot be determined due to the influence of image noise, the mouth area can be optimally extracted.

Further, in the person / mouth region extracting method according to the third aspect of the present invention, if all of the mouth candidate regions are not determined to be mouth regions, the person / mouth region extracting method is performed based on a second predetermined value different from the predetermined value. A partial region having a large area is extracted as a mouth candidate region, and a second region including the predetermined range in the rectangular region is extracted.
Since the mouth area is determined by integrating all the mouth candidate areas located in the predetermined range, even if the influence of image noise is large and the mouth area cannot be determined, the mouth area can be accurately extracted.

According to a fourth aspect of the present invention, in the person / mouth region extracting apparatus, the distance between the eyes of the person is determined by the rectangular region extracting unit.
Extract a rectangular area including the mouth based on the
Based on the binary image data, the smoothing processing means
To reset the pixel value according to the pixel value of the adjacent pixel
After the smoothing process, the
The rectangular area after the jing process is classified into one or more partial areas and other areas formed by connecting pixels having pixel values representing the mouth area, and the area is larger than a predetermined value by the mouth candidate area extracting unit. A partial region is extracted as a mouth candidate region, and a mouth region determining unit determines that a mouth candidate region located in a predetermined range in the rectangular region is a mouth region. The mouth region can be extracted from the data regardless of the outline of the image. Therefore, according to the present invention, it is possible to easily extract a mouth region based on a human image whose contour has been lost due to the effects of lighting and image noise.

According to a fifth aspect of the present invention, there is provided the person / mouth region extracting apparatus, wherein the rectangular region extracting unit uses the reference point and the size determined based on the distance between the eyes in the front image of the person as the reference length. The image data of the rectangular area is cut out from the image data of the front image and binarized, and the obtained binary image data is stored in the rectangular area image data storage unit. The mouth area can be extracted by setting the above rectangular area. Therefore, according to the present invention, it is possible to easily extract the mouth region from the image data of the front image of the person without having to set the rectangular region.

Further, in the person / mouth region extracting apparatus according to the present invention, when the plurality of mouth candidate regions are determined to be mouth regions by the mouth region determining unit, the second mouth region determining unit determines whether the plurality of mouth candidate regions is a mouth region. As a result, the mouth region having the largest area is determined to be the target mouth region, so that even if one large mouth region cannot be determined due to the influence of image noise, the mouth region can be extracted optimally.

In the person / mouth region extracting apparatus according to the present invention, if all the mouth candidate regions are not determined to be mouth regions by the mouth region determination section, the second mouth candidate region extraction device may extract the second mouth candidate region. The unit extracts a partial region having an area larger than the second predetermined value as a mouth candidate region, and integrates all the mouth candidate regions located in the second predetermined range in the rectangular region by the mouth candidate region integrating unit. Thus, even when the influence of image noise is large and the mouth area cannot be obtained, the mouth area can be accurately extracted.

[Brief description of the drawings]

FIG. 1 is a block diagram of an embodiment of a person / mouth region extracting apparatus according to the present invention.

FIG. 2 is a flowchart of an initial processing area extraction processing operation executed by an initial processing area extraction unit in FIG. 1;

FIG. 3 is a flowchart of a binarization processing operation executed by a density histogram extraction unit, a threshold value calculation unit, and a binarization processing unit in FIG. 1;

FIG. 4 is a flowchart of a label area extraction processing operation executed by a 4-neighbor expansion part, a four-neighbor contraction part, a region classification unit, a region area calculation unit, and a region position calculation unit in FIG.

FIG. 5 is a flowchart of a mouth candidate region extraction processing operation executed by a mouth candidate region extraction unit in FIG. 1;

FIG. 6 is a flowchart of a mouth candidate area validity determination processing operation executed by a mouth area first determination unit in FIG. 1;

FIG. 7 is a flowchart of a mouth area determination processing operation executed by a mouth area second determination unit in FIG. 1;

FIG. 8 is a flowchart of a mouth candidate region integration processing operation executed by the mouth candidate region integration unit in FIG. 1;

FIG. 9 is an explanatory diagram of an original image.

FIG. 10 is an explanatory diagram of an initial processing area.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 ... Control part, 4-7 ... Image memory, 8-10 ... Work memory, 12 ... Initial processing area extraction part, 13 ... Binarization processing part, 1
4: threshold value calculation unit, 15: density histogram extraction unit,
16 ... Near expansion part, 17 ... Near contraction part,
18: region classification unit, 19: region area calculation unit,
20: region position calculation unit, 21: mouth candidate region extraction unit,
22: mouth region first determination unit, 23: mouth region second determination unit, 24: mouth candidate region integration unit.

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-5-232908 (JP, A) JP-A-4-101280 (JP, A) JP-A-2-14386 (JP, A) JP-A 64-64 7787 (JP, A) (58) Field surveyed (Int. Cl. ⁷ , DB name) G06T ⁷ /00-7/60 G06T 1/00

Claims (7)

(57) [Claims] 1. A rectangular area including a mouth is set based on a distance between both eyes in a front image of a person, and based on the binary image data of the set rectangular area,
The above target pixel according to the pixel value of the pixel adjacent to the target pixel
Reset the pixel value of
After performing the smoothing process to make the first and second partial regions, one or more partial regions of the rectangular region formed by connecting pixels having pixel values representing the mouth region are obtained. Calculating an area and a position, extracting a partial area in which the calculated area is larger than a predetermined value as a mouth candidate area, and positioning the partial area in a predetermined range in the rectangular area based on the position of the extracted mouth candidate area. A person / mouth region extraction method, wherein a mouth candidate region is determined as a mouth region. 2. The method according to claim 1, wherein when a plurality of determined mouth regions exist, the mouth region having the largest area is adopted as a target mouth region. Method for extracting a person-mouth area. 3. The person / mouth area extraction method according to claim 1, wherein when all of the mouth candidate areas are not determined to be mouth areas, the mouth area is larger than a second predetermined value different from the predetermined value. A partial region having an area is extracted as a mouth candidate region. Based on the position of the extracted mouth candidate region, all the mouth candidate regions located in a second predetermined range including the predetermined range in the rectangular region. A person-mouth region extraction method, characterized in that a region obtained by integrating the above is determined as a mouth region. 4. A person / mouth region extracting apparatus for extracting a person / mouth region based on image data of a front image of a person, wherein the mouth and mouth region is extracted based on a distance between both eyes in the front image of the person.
Extracts a rectangular area that contains, and a binary image of the extracted rectangular area
Rectangular area for storing data in the rectangular area image data storage
An extraction unit and a binary image data stored in the rectangular area image data storage unit.
The pixel value of the pixel adjacent to the target pixel based on the
Then, the pixel value of the target pixel is reset and the outline of the image is changed.
For smoothing and smoothing
Smoothing processing means, and an area classifying unit that classifies the rectangular area after performing the smoothing processing into one or more partial areas and other areas formed by connecting pixels having pixel values representing the mouth area. A region area calculation unit that calculates the area of the partial region; a region position calculation unit that calculates the position of the partial region; and a mouth candidate that extracts a partial region whose calculated area is larger than a predetermined value as a mouth candidate region An area extraction unit, comprising a mouth area determination unit that determines a mouth candidate area located in a predetermined range in the rectangular area to be a mouth area based on the position of the extracted mouth candidate area. Person mouth region extraction device. 5. The person / mouth area extracting apparatus according to claim 4, wherein the rectangular area extracting unit obtains a reference point and a size of the rectangular area using a distance between both eyes in a front image of the person as a reference length. Based on the coordinates and dimensions of the reference point, the image data of the rectangular area is cut out from the image data of the front image and binarized, and the obtained binary image data is stored in the rectangular area image data storage unit. People outlet region extraction apparatus characterized by being made as to. 6. The person / mouth region extracting apparatus according to claim 4, wherein when the plurality of mouth candidate regions are determined to be mouth regions by the mouth region determination unit, the mouth region having the largest area is determined. A second mouth area determination unit that determines a mouth area to be extracted. 7. The person / mouth region extracting apparatus according to claim 4, wherein when the mouth region determination unit determines that all the mouth candidate regions are not mouth regions, a second value different from the predetermined value is used.
A second mouth candidate region extracting unit that extracts a partial region having an area larger than a predetermined value as a mouth candidate region, and includes the predetermined range in the rectangular region based on the position of the extracted mouth candidate region. A person / mouth region extracting apparatus, comprising: a mouth candidate region integration unit that integrates all mouth candidate regions located in a second predetermined range to obtain a mouth region.