JP3465566B2 - Eye position detection device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention can be used for automatically controlling a mirror and a seat, and for identifying the eye position necessary for detecting a dozing driving state from the open / closed state of a driver's eye. The present invention relates to a position detection device.

[0002]

2. Description of the Related Art By calculating facial image data by extracting facial feature amounts, various persons can be identified, such as identifying a person, detecting the direction of a line of sight, and determining whether to fall asleep or awake. The device is being developed. Image data is a huge amount of data that expresses the brightness of each pixel with a large number of gradation levels (density value, brightness), so in such a device, the image data is narrowed down early to reduce the number of data. The processing speed is secured by saving the number of calculations.

For example, in the doze warning device disclosed in Japanese Unexamined Patent Publication No. 7-181012, the eye opening index, which is one of the facial feature values, is calculated from the facial image data. Binarize the density values of image data in black and white, set black mask areas in unnecessary areas of the screen to reduce the number of black areas, and treat each island-shaped black area as a group and label it. To reduce the number of data.

[0004]

[Patent Document 1] Japanese Patent Application Laid-Open No. 7-18101
In the process disclosed in Japanese Patent Laid-Open No. 2 publication, it is necessary to perform the following three steps in order to detect the position of the eye, and therefore it takes a considerable time for image processing and calculation.

(1) Preprocessing of image data mainly for binarization of density values and noise removal (2) Face width detection (labeling, particle removal, white pixel detection for specifying the lateral position of the eye) (Continuity judgment, etc.) (3) Eye longitudinal position detection (judgment of barycentric coordinates of each label, area value, fillet diameter, etc.)

That is, although the density value is binarized, each individual black portion of the island includes several hundreds to tens of thousands of pixels, and therefore each calculation of these processes is performed. It takes a lot of time. Therefore, it is difficult to shorten the processing cycle to improve the followability and reliability of the determination, and it is also difficult to downsize the arithmetic device and interrupt the other arithmetic processing.

Further, since the labeling is performed in the state where the black portions on the face are individually separated, when other black portions are continuous with the black portion corresponding to the eye, for example, when long hair touches the outer corner of the eye. A black portion corresponding to an eye on a black-and-white binarized image is integrated with another black portion, and a black portion corresponding to an eye cannot be specified.

Therefore, the applicant of the present invention, as an eye position detecting device, can reduce the amount of data to be calculated, increase the processing speed, and accurately specify the feature amount corresponding to the eye even when the long hair touches the outer corner of the eye. For example, Japanese Patent Application No. 8-10190
I am proposing something like that described in No. 4.

This eye position detecting device detects the density of pixels along a pixel row in the vertical direction of the face, and determines one pixel as an extraction point for each local increase in the density in the pixel row. In this configuration, the extraction points adjacent to each other in the pixel row direction of adjacent pixel rows are connected to detect the eye position from the curve data extending in the lateral direction of the face.

The present invention is a further improvement of the above-mentioned eye position detecting device. Even if there is a curtain rail or a sunroof which causes noise in the background portion of the person who is subject to eye position detection, the eye position detection accuracy is improved. An object of the present invention is to provide an eye position detecting device that can maintain the eye position.

[0011]

According to a first aspect of the present invention, image input means for inputting image data including a face, and pixel densities are read out along a vertical pixel row of the input image data, Point extraction means for specifying the pixel of the unidirectional peak of the density value as the extraction point, provided that the differential value of the density change before the unidirectional peak of the density value on the pixel row exceeds a predetermined value. Curve data extracting means for extracting curve data that continuously extends in the lateral direction of the face from the extraction points that are adjacent to each other in the pixel row direction, and movement state determination means for determining the movement state of the curve data. And an eye position detecting means for detecting the eye position based on the curve data when it is determined that the curve data is dynamic.

Therefore, the curve data to be judged for movement is narrowed down by the curve data extracting means, and the moving state of the corresponding curve data is judged by the moving state judging means.
When it is determined that the curve data is dynamic, the eye position can be detected by the eye position detecting means based on the curve data.

The invention of claim 2 is the eye position detecting apparatus according to claim 1, wherein an eye position search area 1 for setting an eye position search area 1 including the curve data extracted by the curve data extracting means is set. 1 setting means, an uppermost end curve data detection means for detecting curve data appearing at the uppermost end in the eye position search area 1, and an eye for setting the eye position search area 2 based on the uppermost end curve data. Location search area 2
The setting means, the movement state determination means of the curve data that determines the movement state of the curve data that appears in the eye position search area 2, and the eye movement when the target curve data is determined to be dynamic. The eye position search area 3 setting means for setting the position search area 3, and the setting of the eye position search area 1 when the target curve data is determined to be static curve data, are applicable. Relative position of the curve data appearing in the eye position search area 3 and a reduction means of the eye position search area 1 for vertically reducing the range of the eye position search area 1 based on the position information of the curve data. It is characterized by comprising eye position detecting means for detecting the eye position from the relationship.

Therefore, in addition to the operation of the invention of claim 1, first, the eye position search area 1 setting means sets the eye position search area 1 containing the curve data extracted by the curve data extraction means, and then the eye position search area 1 setting means. The upper end curve data detection means detects the curve data appearing at the uppermost end in the eye position search area 1, and the eye position search area 2 setting means sets the eye position search area 2 based on the uppermost curve data. To do. In this way, the curve data is narrowed down, and the movement state determination means determines the movement state of the curve data appearing in the eye position search area 2. When the curve data is determined to be dynamic, the eye position search area 3 The eye position search area 3 is set by the setting means. When it is determined that the target curve data is static curve data, the setting of the eye position search area 1 is returned to, and the eye position search area 1 of the eye is searched based on the position information of the corresponding curve data. The reduction unit reduces the range of the eye position search area 1 in the vertical direction. The eye position detecting means detects the eye position from the relative positional relationship of the curve data appearing in the eye position search area 3. Therefore, the eye position can be detected by narrowing down the eye position search area 1, the eye position search area 2, the eye position search area 3 and the curve data.

A third aspect of the invention is the eye position detecting apparatus according to the first or second aspect, wherein the movement state determination means determines the movement state of the curve data based on the amount of vertical movement of the curve data. It is characterized by

Therefore, in addition to the operation of the first or second aspect of the invention, it is possible to determine whether or not the curve data is dynamic based on the amount of vertical movement of the curve data by the movement state determination means. Therefore, it is possible to make a determination according to the movement of the occupant during driving.

According to a fourth aspect of the present invention, in the eye position detecting apparatus according to the third aspect, the movement state determining means determines the movement state of the curve data from the vertical position of each extraction point constituting the curve data. It is characterized in that it is determined by the average value of.

Therefore, in addition to the action of the invention of claim 3, the moving state judging means can judge the moving state of the curve data by the average value of the vertical position of each extraction point forming the curve data.

A fifth aspect of the present invention is the eye position detecting apparatus according to any one of the first to fourth aspects, wherein the movement state determining means determines the movement state by the extraction points forming curve data. It is characterized in that it is performed at the uppermost position.

Therefore, in addition to the operation of any one of the first to fourth aspects of the invention, the movement state can be determined at the uppermost position of the extraction points forming the curve data.

A sixth aspect of the present invention is the eye position detecting apparatus according to any one of the first to fourth aspects, wherein the movement state determination means is for the curve data appearing in the eye position search area 2. It is characterized by having a shape determination means for determining a shape and a movement state determination time changing means for changing the determination time of the movement state determination means of the curve data according to the output result of the shape determination means.

Therefore, in addition to the operation of any of the first to fourth aspects of the present invention, the determination time of the moving state determination means is changed by determining the shape of the curve data appearing in the eye position search area 2. For example, if the curve data is convex upward, it is highly likely that it is an eye, and the determination time can be lengthened.

The invention according to claim 7 is the eye position detecting apparatus according to any one of claims 1 to 4, wherein the movement state determining means determines the shape of the curve data appearing in the position search area 2. It is characterized by having a shape determining means for determining and a movement state determining movement amount changing means for changing the determination movement amount of the movement state determining means of the curve data according to an output result of the shape determination means.

Therefore, in addition to the operation of the invention of any one of claims 1 to 4, it is possible to change the determination movement amount of the movement state determination means by determining the shape of the curve data. Therefore, if the curve data is convex upward, there is a high possibility that it is an eye, and the amount of determination movement can be reduced.

An eighth aspect of the present invention is the eye position detecting apparatus according to any one of the first to fourth aspects, wherein the movement state determination means is for the curve data appearing in the eye position search area 2. It is characterized by comprising shape determining means for determining the shape, and setting condition changing means for the area 3 for changing the setting condition of the eye position search area 3 according to the output result of the shape determining means.

Therefore, in addition to the operation of the invention of any one of claims 1 to 4, the setting condition of the eye position search region 3 can be changed by determining the shape of the curve data. Therefore, even when the curve data is under the spectacle frame, the curve data corresponding to the eye can be captured by changing the setting condition of the eye position search area 3.

According to a ninth aspect of the present invention, in the eye position detecting apparatus according to the second aspect, the moving state determining means determines the face image data based on the curve data appearing in the eye position search area 2. Eye position search area 4 setting means for setting the horizontally long eye position search area 4 so as to extend to the left and right of the eye, and whether or not there are two or more curve data on the left and right in the set eye position search area 4. Area 3 for changing the setting condition of the eye position search area 3 by specifying the curve data based on the size of the vertical step of the determined left and right data
And setting means changing means.

Therefore, in addition to the operation of the invention of claim 2, the laterally long eye position search area 4 is set from the curve data appearing in the eye position search area 2. It is determined whether or not there are two or more right and left curve data in the set eye position search area 4, and the curve data is specified based on the size of the upper and lower steps of the determined left and right data to determine the eye position search area. The setting conditions of 3 can be changed. Therefore, the positions of the eyes can be detected by determining the bilaterally symmetrical features of the frames of the eyebrows, eyes, and glasses.

According to a tenth aspect of the present invention, in the eye position detecting apparatus according to the ninth aspect, the left and right data determining means selects two curve data from the upper part in the eye position search area 4. Characterize.

Therefore, in addition to the effect of the ninth aspect of the invention, two curve data can be selected from the upper part in the eye position search area 4. Therefore, the left and right eyebrows, eyes, and glasses can be captured in the eye position search area.

An invention according to claim 11 is the eye position detecting apparatus according to claim 9 or 10, wherein the setting condition changing means of the area 3 is the above-mentioned when the step of the left and right data is less than a predetermined value. It is characterized by further comprising shape determining means for determining the shape of the curve data appearing in the eye position search area 4 and specifying the curve data.

Therefore, in addition to the action of the invention of claim 9 or claim 10, when the step difference of the left and right data is less than a predetermined value, the shape of the curve data appearing in the eye position search area 4 is judged to determine the curve data. Can be specified. Therefore, it is possible to distinguish between the spectacle frame and the eye.

The invention according to claim 12 is the eye position detecting device according to claim 2, 6 or 7, or 8 or 11, wherein the shape determining means is a shape of the curve data. Is connected to the rightmost point and the leftmost point of each extraction point forming the curve data by a straight line, and the shape is recognized by the position of the extraction point at the center of the curve data with respect to the straight line.

Therefore, in addition to the action of the invention described in claim 2, claim 6, claim 7, claim 8, or claim 11, the rightmost point and the leftmost point of the extraction points are connected by a straight line to form a straight line. The shape can be recognized by the position of the extraction point in the center of the curve data for, and it can be recognized whether the curve data is convex upward or convex downward.

[0035]

According to the first aspect of the present invention, by determining the movement of the corresponding curve data, even if a curtain rail, a sunroof, or the like that causes noise is present in the background portion of the person whose eye position is to be detected, The detection accuracy of the position of can be maintained, and accurate detection can be performed.

According to the invention of claim 2, in addition to the effect of the invention of claim 1, by narrowing down the eye position search area in three stages, more accurate determination can be performed and a more accurate eye position can be determined. Detection can be performed.

According to the invention of claim 3, in addition to the effect of the invention of claim 1 or 2, the determination of the moving state can be accurately performed by using more reliable data, and more accurate eye position detection can be performed. Can be done.

According to the invention of claim 4, in addition to the effect of the invention of claim 3, more accurate by determining the moving state of the curve data by the average value of the vertical position of each extraction point forming the curve data. The determination can be performed, and more accurate eye position detection can be performed.

According to the invention of claim 5, in addition to the effect of any one of the inventions of claims 1 to 4, the determination of the moving state can be performed at the uppermost position of the extraction points forming the curve data, and the accuracy is accurate. Therefore, it is possible to perform accurate determination and accurately detect the eye position.

According to the invention of claim 6, in addition to the effect of the invention of any one of claims 1 to 4, when it is highly likely that it is an eye based on the shape determination of the curve data, a longer time is used to determine more. Accurate determination can be performed, and more accurate eye position detection can be performed.

According to the invention of claim 7, in addition to the effect of the invention of any one of claims 1 to 4, when there is a high possibility of being an eye by shape determination, the determination movement amount is reduced to make more accurate determination. Can be performed, and more accurate eye position detection can be performed.

According to the invention of claim 8, in addition to the effect of any one of the inventions of claims 1 to 4, by changing the setting condition of the eye position search region 3 by determining the shape of the curve data, the curve data can be converted into, for example, Even under the spectacle frame or the like, the more accurate eye position search area 3 can be set, and more accurate eye position detection can be performed.

According to the ninth aspect of the invention, in addition to the effect of the second aspect of the invention, by setting the laterally long eye position search area 4, the characteristics of the eyebrows, the eyes, and the spectacle frames that are present symmetrically are provided. The determination can be performed, and the more accurate determination can detect the eye position more accurately.

According to the tenth aspect of the invention, in addition to the effect of the ninth aspect of the invention, by selecting two curve data from the upper part in the eye position retrieving region 4, the curve positions are sequentially selected from the upper part to retrieve the eye position. The determination in the area 4 can be performed more accurately, and more accurate eye position detection can be performed.

According to the invention of claim 11, claim 9 or 10
In addition to the effect of the invention described above, more accurate determination can be performed by the shape determination of the curve data in the eye position search region 4, and more accurate eye position detection can be performed.

According to the invention of claim 12, in addition to the effect of the invention of claim 2 or claim 6 or claim 7 or claim 8 or claim 11, the shape of the curve data is accurately recognized by the minimum necessary information. Therefore, more accurate eye position detection can be performed in a short time.

[0047]

1 is a functional block diagram of an eye position detecting apparatus according to an embodiment of the present invention. The eye position detecting apparatus includes an image input means CL1, a density detecting means CL2, and a point extracting means CL3. Curve data extraction means CL4, predetermined area 1 setting means CL5, and curve data detection means CL6
And the setting means CL7 for the predetermined area 2 and the movement state determining means CL
8 and a predetermined area 3 setting means CL9, a predetermined area 1 reduction means CL10, and an eye position detecting means CL11.

The image input means CL1 inputs image data including the driver's face. The concentration detecting means CL2
Detects the density of the vertical pixel row of the image data input from the image input means CL1. The point extracting means CL3 extracts points according to the increase in the density of the pixel row and the changing state thereof. The curve data extracting means CL4 specifies pixels having a unidirectional peak of the density value on condition that the differential value of the density change before the unidirectional peak of the density value on the pixel array exceeds a predetermined value. As the extraction point. That is, the curve data extraction unit CL4 continuously extracts the extraction points that are adjacent to each other in the pixel row direction of the adjacent pixel rows,
Curve data extending in the lateral direction of the face is extracted. The setting unit CL5 of the predetermined area 1 sets the eye position search area 1 including the extracted curve data. The uppermost curve data detecting means CL6 detects the curve data appearing at the uppermost end in the eye position search area 1. The setting means CL7 of the predetermined area 2 sets the eye position search area 2 based on the uppermost curve data. The moving state determination means CL
Reference numeral 8 determines the movement state of the curve data that appears in the eye position search area 2. The setting means CL9 of the predetermined area 3 is
When it is determined that the target curve data is dynamic, the eye position search area 3 is set. The predetermined area 1
When it is determined that the target curve data is static curve data, the reducing means CL10 returns to the setting of the eye position search area 1, and based on the position information of the corresponding curve data, The range of the position search area 1 is reduced vertically. The eye position detecting means CL11 detects the eye position from the relative positional relationship of the curve data appearing in the eye position search area 3.

FIG. 2 is a block diagram of the configuration according to an embodiment of the present invention. As shown in FIG. 6, a camera 21 is provided as an image input unit CL1 that is installed in the instrument as shown in FIG. 2 and that captures the driver's face portion from the front, as shown in FIG. 6 in the present embodiment. It consists of 512 pixels in the horizontal (X) direction and 480 pixels in the vertical (Y) direction. The input image captured by the camera 21 is input as digital input image data via the A / D converter 22.
It is stored in the image memory 23. An image data arithmetic circuit 24 is connected to the image memory 23. The image data arithmetic circuit 24 includes the density detecting means CL2, the point extracting means CL3, the curve data extracting means CL4, the predetermined area 1 setting means CL5, and the curve data detecting means CL6.
The predetermined area 2 includes setting means CL7, curve data movement state determining means CL8, predetermined area 3 setting means CL9, and predetermined area 1 reducing means CL10.

An eye position detecting circuit 25 as an eye position detecting means CL11 for detecting the eye position in the eye position searching area 3 is connected to the image data arithmetic circuit 24.

Next, the flow of operation in the above configuration will be described with reference to the flowcharts of FIGS. 3 to 5 and the explanatory diagrams of FIGS. 6 to 14.

First, in step S301, the eye position search area 1 is set. This eye position search area 1
Is an area that can cover the entire image as shown in FIG. This is to allow for changes in the position of the eyes due to differences in physique.

In step S302, pixel data in the eye search area is read out, and in step S303, candidate eye data to be curve data (in the transition step, curve data is referred to as continuous data) is extracted. Step S
The processing contents of 302 and S303 will be described with reference to the flowchart of FIG. 5 and the explanatory diagrams of FIGS.

First, in step S501, the face portion of the driver is photographed by the camera, and an input image for one frame is converted by a digital signal and stored in the image memory. In step S502, point extraction processing is performed on the eye position search area as shown in FIG.
After finishing the line, move on to the processing of the next line,
It is determined whether or not the point extraction has been completed for all lines in the predetermined direction. If it is determined in step S502 that points have not been extracted on all lines, the process proceeds to step S503.

In step S503, the arithmetic mean of the density values of one line in a predetermined direction, for example, from top to bottom is calculated. This processing is intended to eliminate small variations in changes in density value during image data capturing, and is for capturing global changes in density value. In FIG.
The processing result of the arithmetic mean calculation of the line data of Xa is shown.

In step S504 of FIG. 5, step S
A differential operation is performed on the arithmetic mean value which is the operation result of 503. The result of this processing is shown in FIG. In step S505 of FIG. 5, point extraction is performed using the differential value that is the calculation result of step S504. The extraction method for that point is that the point where the differential value changes from negative to positive (p1
.. p3), in FIG. 7A, points at which the graph is convex downward are extracted.

Whether or not the change (q1 to q3) in the density value until reaching the point is below a predetermined value or not is shown in FIG.
It is determined whether or not the differential value of (b) is equal to or less than the stable reference value, and the Y coordinate values (A1 to A3) are extracted with respect to points having a change in density value of a predetermined value or less. This process is 1
After the line is completed, the process is switched to the next line in step S506.

When it is judged in step S502 that the points of all the lines have been extracted, the points shown in FIG. 8 are extracted. That is, on the Xa line in FIG.
This means that three points 1, A2 and A3 have been extracted.

After that, the process proceeds to step S507, and Y of the extraction points (A1, A2, A3 ...) Of adjacent lines are Y.
The coordinate values are compared, and if the Y coordinate value is within a predetermined value, the continuous data group number, continuous start line number, and continuous data number are stored as continuous data.

Since the detection target is the eye here, it can be assumed that the feature amount is data that continues laterally for a relatively long time. Therefore, continuous data is extracted under the condition that it continues for a predetermined value or more in the horizontal direction. To do. Therefore, step S5
As a result of the processing in 07, continuous data of G1 to G6 as shown in FIG. 9 can be recognized as curve data.

After extracting continuous data in step S303,
The process moves to step S304. In step S304,
The judgment value of each continuous data is read. The judgment value of each continuous data is the representative X coordinate of continuous data calculated by calculating the median value of continuous start line and end line of continuous data Refers to the representative Y coordinate.

In step S305, it is determined whether or not the eye position search area 3 is set. At the stage immediately after the start, since the eye position search area 3 is not set, the process proceeds to step S306. In step S306, it is determined whether or not the eye position search area 2 is set. Also in step S306, as in step S305, since the eye position search area 2 is not set at the stage immediately after the start, the process proceeds to step S307.

In step S307, step S30
5, since it is determined in S306 that neither the eye position search area 3 nor the eye position 2, the continuous data is detected in the eye position search area 1. In this step, the uppermost continuous data appearing in the eye position search area 1 as shown in FIG. 10 is detected. In this process, instead of suddenly patterning a large number of continuous data that appear in the entire image, the continuous data that appears at the top end is first focused, and the corresponding continuous data is the facial feature amount (continuous data of eyebrows and eyes). ), So that the eye can be searched as easily as possible.

In step S308, as shown in FIG. 10, the eye position search area 2 based on the representative coordinate value of the uppermost continuous data is set. The eye position search area 2 repeatedly determines for a predetermined time whether the corresponding continuous data is a facial feature amount such as an eyebrow or an eye, and obtains it from a change in the appearance state of the corresponding continuous data. Therefore, it is desirable that only one continuous data appears in the eye position search area 2, and the extraction points at both ends of the corresponding continuous data appear when the row density to be searched is set roughly. Since there is a considerable amount of variation in the lateral direction (X coordinate direction) depending on the presence or absence of the item, it is important to set the lateral width of the eye position search region 2 as small as possible.

Thereafter, in step S309, a timer for measuring the predetermined time is activated and step S302.
Return to. The region in step S302 at this time is performed only within the eye position search region 2 as shown in FIG. After that, the processes of steps S303 to S305 are similarly performed.

In step S306, since the eye position search area 2 has been set, the process moves to step S401 in FIG. In step S401, it is determined whether the timer started in step S309 has passed a predetermined time. If it is determined in step S401 that the predetermined time has not elapsed, the process proceeds to step S402.

In step S402, the maximum value of the average values of the Y coordinate values (y1 to y9) of the extraction points of the continuous data appearing in each frame as shown in FIG. 12 is updated and stored in memory until a predetermined time is reached. . In step S403, the minimum value of the average value of the Y coordinate values is updated and stored until a predetermined time is reached. If it is desired to further increase the sensitivity of movement in updating the minimum and maximum values of the Y coordinate value, the uppermost point (y2 in the case of FIG. 12) of each extraction point of continuous data can be used as the representative value. .

In step S404, the movement of each frame that appears in the eye position detection area 2 with the passage of time is determined by the difference between the maximum value and the minimum value of the Y coordinate values stored in steps S402 and S403. The vertical movement of is determined. At this time, the vertical direction is used to determine the movement of the continuous data, as described in the size of the eye position search region 2, the extraction point in the horizontal direction (X coordinate direction). This is because the connection in the horizontal direction, in other words, the number of continuous data varies from frame to frame, which is further disadvantageous when the column density is set roughly in order to speed up the processing. If the movement amount of the continuous data does not exceed the determination movement amount in step S404, step S3
Returning to 02, the same processing is repeated.

At this time, if the uppermost continuous data as shown in FIG. 10 is a static object of a vehicle interior structure such as a curtain rail, its movement is not determined in step S404 and step S401 is performed. It is determined that the predetermined time has elapsed. At this time, the process proceeds to step S406,
Clear the timer. After that, as shown in FIG. 13 in step S407, when the eye position search area 2 is set,
The eye position search area 1 which is reduced in the vertical direction so as not to include the coordinate value A as the reference is set.

In step S408, the eye position search area 1
It is determined whether or not the reduced area of No. 2 has become infinitely small in the vertical direction, and when a face is present in the detection target area, it is very unlikely to be NO. NO in this step S408
Is a loop when only a static object is detected and no dynamic continuous data can be detected even if it goes to the lowermost end when there is no person to be detected. At this time, step S30
After returning to 1, the eye position search area 1 is initialized, and the same processing is repeated.

After the eye position search area 1 is reduced in step S407, continuous data corresponding to the uppermost end in the eye position search area 1 is captured as the face feature amount (eyebrow in this example) shown in B of FIG. If it is determined that the change of the Y coordinate exceeds the determination movement amount in step S404 within a predetermined time after the timer starts. At this time, step S40
Go to 5.

In step S405, the eye search area 3 including the corresponding continuous data as shown in FIG. 14 is set. The eye position search area 3 has a size including one eye and one eyebrow. This is because it is known that the continuous data in which the movement is recognized in the eye position search area 2 is the feature amount of the face, but it is not possible to specify the eyebrow, the eye, or the frame of the eyeglass. After setting the eye position search area 3 in step S405, the process returns to step S302 to detect continuous data in the area.

At this time, in step S305, since it is determined that the eye position search area 3 is set, the process moves to step S310 and corresponds to the eye from the continuous data appearing in the eye position search area 3. Select continuous data. The selection method at this time is determined based on the relative positional relationship of each continuous data. The selection type can be configured very easily because the target of the eye is narrowed down to a certain degree, and accordingly high eye detection accuracy can be secured. After that, in step S311, the eye position coordinates are output, and the eye position detection processing from the entire image ends.

Next, a method of reliably catching an eye by adding the shape determination of continuous data will be described with reference to the flowcharts of FIGS. 15 and 16 and FIG.

The flowchart of FIG. 15 is obtained by adding step S1504 to the flowchart of FIG.

The shape determination method will be described by using an example of continuous data appearing in the eye position search area 2. As shown in FIG. 17, the continuous data shape determination method connects a continuous start point ys and a continuous end point ye with a straight line, and calculates a Y coordinate value yc on the X coordinate with an extraction point near its center. At this time, by determining the positional relationship between the extraction points and yc on the same line, it is determined whether the corresponding continuous data has an upward convex shape, a downward convex shape, or a substantially linear shape. In the case of continuous data in which a curtain rail, a sunroof, etc. are detected by this determination, the data are often almost horizontal as shown in FIG. 17 (a), whereas in the case of data of eyes, eyebrows, etc., they are projected upward. Often becomes.

In the above-described method for determining the shape of continuous data, the shape feature of the object to be detected can be surely grasped by the minimum necessary information, and the processing is simple. Therefore, the shape determination time can be shortened. There is an advantage.

The flowchart of FIG. 16 is obtained by adding steps S1602 to S1605 to the flowchart of FIG.

The above steps are intended to more reliably catch the eye by utilizing the shape information of the continuous data.
That is, when the shape of the continuous data is convex upward, the possibility of eyes is high, and the determination time is lengthened or the determination movement amount is reduced. If they are in the same straight line, there is a high possibility that the curtain rail or sunroof is behind the driver, and the judgment time is shortened or the judgment movement amount is increased. This improves the detection accuracy.

In step S1602, it is determined whether or not the shape of the continuous data appearing in the eye position search area 2 is convex upward. At this time, if it is convex upward, the process proceeds to step S1603, and the upward convex counter is counted up. This process focuses on the fact that, when the continuous data to be detected is eyebrows or eyes, most of the data is convex upward except when the eyes are closed. .

Therefore, in step S1604, it is determined whether or not the upward convex counter has reached the predetermined number by the processing repeated within the predetermined time A in step S1601. At this time, as described above, in the case of continuous data for the eyes, eyebrows, etc., the upward convex counter is counted up early and the process proceeds to step S1605.

In step S1605, the predetermined time A is extended or the determination movement amount B is decreased. The reason why the predetermined time A is extended and the determination movement amount B is reduced is to cope with the case where the motion of the detection target person is extremely small. The upward convex counter is cleared when the eye position search area 3 is set or when the predetermined time A is exceeded.

Next, a method for surely catching the eyes even when the motion under the frame of the eyeglasses is the object of judgment by adding the shape judgment of the continuous data will be described with reference to FIGS.

In a person to be detected wearing glasses as shown in FIG. 18, the movement of the face happens to be small and the eye position search area 1 is reduced as shown in FIG. When the continuous data at the upper end is detected under the spectacle frame, the eye position search area 2 is set and the determination of the movement is made, and when it is determined that the corresponding continuous data has moved, the eye position search is performed according to FIG. When the region 3 is set, the eyes on the spectacle frame are out of the region and cannot be detected. Further, as a measure of simply expanding the eye position search area 3 upward, continuous data such as hair may newly appear as noise, which may reduce the eye detection accuracy.

Therefore, in this embodiment, the eye position search area 2
This problem is solved by determining the shape of the continuous data that appears in. As shown in FIG. 21, in the case of being under the spectacle frame as shown in FIG. 20, continuous data having a downward convex shape is often obtained, and by determining the appearance frequency of the continuous data having a downward convex shape, as shown in FIG. The set position of the eye position search area 3 can be changed. By the shape determination of the continuous data, the continuous eye data can be captured in the eye position search area 3 even if the movement in the eye position search area 2 is determined under the spectacle frame. Detection is possible.

Next, the eye position search area 4 shown in FIG.
A method for surely correcting the set position of the eye position search region 3 by adding the shape determination of the continuous data in (3) will be described.

Details of this embodiment will be described with reference to the flow charts of FIGS. 22 to 24 and FIGS. 25 to 36.

FIG. 22 is a flowchart obtained by adding step S2207 to the flowchart of FIG. In the above-described embodiment, the moving amount of the continuous data appearing in the eye position search area 2 is determined, and the eye search area 3 is set when the condition is satisfied, but in this embodiment, before that, Figure 2
As shown in step S2309 of the flowchart of FIG. 3, the eye position search area 4 is set, and the relative positional relationship and shape of the two continuous data appearing in the area are determined. Therefore, in step S2309 of FIG.
22 is set, the process proceeds from step S2207 in FIG. 22 to step S2401 in FIG.

The contents of the judgment in the eye search area 4 will be described below with reference to the flow chart of FIG.

In FIG. 25, the continuous data appearing at the uppermost end in the eye position search area 1 is captured as G1, and the movement of the continuous data in the eye position search area 2 satisfies the determination condition. Is shown.

At this time, the eye position search area 4 in step S2309 of FIG. 23 is set as a horizontally long area having left, right, up, and down equal widths based on the center coordinates of the continuous data G1 as shown in FIG. . In step S2401 of FIG. 24, it is determined whether or not there are two or more continuous data satisfying the shape and position conditions in the eye position search area 4, and if there are two or more continuous data satisfying the conditions. Moves to step S2402.

It is to be noted that the continuous data G2 described in a little light in FIG. 25 indicates that the detection as continuous data is unstable, and the continuous data G2 may or may not appear in the processing for each frame. . Therefore, the two continuous data appearing in the eye position search area 4 in FIG. 26 may be G1 and G4. Further, when three or more continuous data appear in this area, two continuous data are selected from the upper part of the area.

At the stage of detecting the two left-right symmetric continuous data, it is still unknown whether the corresponding continuous data is an eyebrow or an eye. Therefore, FIG.
The eye position search area 3 shown in is set, and the final continuous eye data is detected.

However, there may be a problem when setting the eye search region 3. Although one of the countermeasures has been described in the previous embodiment, it is a case where continuous data can be recognized only under the frame of the eyeglasses and the eye position search area 3 is set. In the eye position search area 3, since it is not yet known whether the corresponding continuous data is an eyebrow or an eye, the area is enlarged and set only in the downward direction on the basis of the continuous data, and appears in the area. The eye continuous data is detected from the continuous data.

The reason for expanding only in the downward direction has been described in the previous embodiment, but when continuous data of eyebrows is used as a reference, continuous data of hair is included or a hat is worn. This is because continuous data of a hat may appear, the determination in the eye search area 3 may become complicated, and the accuracy of eye detection may deteriorate.

Therefore, it is necessary to shift and set the eye position search area 3 upward as in the previous embodiment shown in FIG. Even with the method based on only the shape determination in the eye position search area 2 as in the previous embodiment, it is possible to determine to some extent continuous data under the frame of the eyeglasses, and the eye search area 3 cannot be corrected and set. However, the continuous data under the frame of the glasses may be detected as horizontal continuous data, and it may not be recognized that the continuous data is under the frame of the glasses.

In this embodiment, in order to more reliably determine that the continuous data is under the frame of the spectacles, two determinations for two continuous data appearing in the eye position search area 4 are performed. I am doing it.

One of the judgments is step S2402.
Then, it is determined whether the continuous data corresponding to the right side of the corresponding two data is the eye or the frame below the glasses. The determination of the presence or absence of the appearance of these two continuous data is based on the fact that there are two feature amounts at which the eyes are laterally symmetrical.

Two sets of bilaterally symmetrical data appearing in the eye existence region 4 are the eyebrow on the left side (in some cases, it may be on the frame of eyeglasses when wearing glasses) and the eye on the right side. May be left eye and right eye. Furthermore, FIG.
As shown in FIG. 6, there are cases where detection is performed in different steps such as the right eyebrow and the left eye. This is because the appearance of continuous data corresponding to the eyebrows becomes unstable in the case of a person who has his bangs down and is on his eyebrows.

In step S2402, it is first determined whether the data on the right side of the two continuous data is under the eye or the frame of the spectacles. The determination method here is shown in FIG.
As shown in FIG. 6, a step L between the left and right continuous data G1 and G4
It is determined whether 1 is large. If the position of the continuous data on the right side is lower than that on the left side as shown in FIG.
There are cases in which the left eyebrow and the right eye as shown in FIG. 6 are targeted, and cases in which the left eye and the right under the frame of the spectacles as shown in FIG. 29 are targeted.

In the situation where two continuous data appear in the combination as shown in FIG. 29, it is difficult to capture the continuous data because both eyebrows are affected by the bangs as shown in FIG. This is the case when the detection is unstable due to such reasons. By determining the combination of two such continuous data by the steps L1 and L2 of both continuous data, the eye search region 3 is set in a state of being upwardly corrected as shown in FIGS. In the case of FIG. 27, since the corresponding continuous data is an eyebrow, it is not necessary to intentionally correct the set position of the eye position search area 3, but in this case, it is equivalent to the case where the left and right eyebrows are symmetrically set and set. There is no problem with the detection of. Therefore, if there is a step between the two continuous data in step S2402, the process moves to step S2403,
The set position of the eye position search area 3 is corrected. If there is no step between the two continuous data, the eye position search area 3 shown in FIG. 14 is left as it is.

Next, the reason why the eye position search area 4 is set to widen on both sides with reference to the continuous data specified in the eye position search area 1 will be described. Since the continuous data specified in the eye position search area 1 captures the uppermost continuous data that appears in the same area, it cannot be specified which of the left and right eyebrows, eyes, and eyeglass frames. Therefore, both sides of the eyebrows, the eyes, and the glasses need to be viewed in order to capture another continuous data having a symmetrical positional relationship between the frames.

The merit of this area setting is that the eye detection accuracy is improved by determining the bilaterally symmetrical features of the eyebrow, eye, and eyeglass frames, and that the ability to respond to changes in the light environment of the automobile is improved. There is also. More specifically, in the case of a right-hand drive vehicle, the right side of the driver's face may be hit by direct light from the side window, which may make it difficult to capture the shape of the eyes on the image. Therefore, in order to accurately capture the shape of the eyes, it can be said that the eyes inside the vehicle, that is, the driver's left eye is preferable. Add the eye position search area 4,
It can be said that capturing the frames of the left and right eyebrows, the eyes, and the glasses can reliably detect the driver's left eye, which is relatively stable in terms of the light environment, and is thus resistant to changes in the light environment of the vehicle interior.

Next, a method of determining the bottom of another spectacle frame will be described. The determination is made in step S2404. In step S2402, it is a method of recognizing that it is under the frame of the spectacles by performing shape determination on two continuous data appearing in the eye position search area 4. The processing flow is shown in FIGS.
Will be explained.

When the eyebrows cannot be captured due to the effect of the bangs and the lens of the eyeglasses due to the influence of the bangs, the continuous data of the frame G6 of the eyeglasses may be the detection target as shown in FIG.
In such a case, the appearance of two consecutive data in a symmetrical relationship in the eye position search area 4 is caused by the step difference L on the left and right.
Since 3 is small, it cannot be recognized in step S2402 that the data is continuous data below the frame of the glasses. Therefore, in step S2404, as described with reference to FIG. 20, it is determined whether or not the downward convex shape, which is the characteristic of the continuous data under the frame of the eyeglasses, is formed.

Since the shape determination in this embodiment can be performed on the left and right continuous data, the shape can be determined downward with the continuous data on both sides shown in FIG. It can be recognized that the continuous data on the left side only has a downward convex shape and the continuous data on the right side has a downward convex shape as shown in FIG. 36. Therefore, the recognition accuracy can be greatly improved as compared with the previous embodiment.

If the above conditions are satisfied in step S2404, the flow moves to step S2405, and the eye search region 3
Correct the setting position of. If the conditions of both steps S2402 and S2404 are not satisfied,
The normal eye position search area 3 is set in step S2406, and the process returns to step S2202. In addition, step S24
If there is no two continuous data satisfying the shape and position condition in the eye position search area 4 at 01, step S2407.
And the reduced eye position search area 1 is set to an area that does not include the setting reference of the eye position search area 2 and step S24 is performed.
Move to 08. In step S2408, the reduction limit of the eye position search area 1 is determined. If the reduction limit is exceeded, the detection area is returned to the initial state. Step S240
In the processing of S1, S2407, and S2408, by determining the appearance of two symmetric consecutive data items that appear in the eye position search area 4, hair or nose that becomes noise in detection,
Continuous data by mouth can be excluded. Therefore, when the eyes cannot be detected due to the temporary deterioration of the image state, the continuous data of the nose and mouth under the eyes are not detected, and the eye position search area 1 reaches the lower end. Again, the eye position can be detected from the upper part of the screen.

Finally, a method of selecting continuous eye data that appears in the eye position search area 3 using shape determination of continuous data will be described.

The set position of the eye position search area 3 is shown in FIG.
There are patterns as shown in (a) to (c). (A)
Indicates that the continuous region data of the eyebrows is used as a reference and the same region is enlarged and set downward, and (b) is set that the continuous region of eye is used as a reference and the same region is enlarged and set (c).
Indicates that the continuous data under the spectacle frame is used as a reference and the same region is enlarged in the upward direction. There are cases where eyebrows and eyes are present in the same area as in (a) and (b), and cases where eyes and eyeglass frames are underneath as in (c).

With these two patterns, it is difficult to make a judgment based only on the positional relationship between the two consecutive data. However, if the shape data of each continuous data is used, the distinction can be surely made. That is, the eyebrows and eyes are often convex upwards, and the eyebrows are not convex downwards. In addition, the bottom of the spectacle frame often has a downward convex shape, and does not have an upward convex shape. In this way, by adding the shape combination determination to the positional relationship of the continuous data that appears in the same region, the accuracy of eye detection can be further improved.

The determination in the eye position search area 3 is as follows.
Step S310 in FIG. 3 and step S151 in FIG.
0, the expression of the flowchart of step S2212 in FIG. 22 is a single loop, but naturally, it is also possible to turn the loop several times and determine from the appearance frequency of the continuous data of interest. Therefore, FIG. 30 and FIG. 33 show a state in which the continuous data corresponding to the eye does not appear, but when the processing is repeated several times, the continuous data of the eye is at a recognizable level.

In this way, by accurately detecting the position of the driver's eyes regardless of the presence of the curtain rails and sunroofs behind the driver, it is possible to accurately detect the driver's dozing driving or looking aside. Appropriate measures such as alarms and vehicle stops can be taken. Further, by accurately detecting the position of the driver's eyes, it is possible to appropriately control the angle of the mirror, the seat height, and the like.

Further, in the above embodiment, the AD converter 22 is used.
Although the following vehicles are all digitally calculated, the analog signal circuit may perform the processing until the arithmetic mean of the density values along the pixel row is obtained. For example, set a scan line in the vertical direction with a camera that uses an image,
After passing a code signal along a scan line that vertically traverses the face through a low-pass filter to remove high-frequency components, a code signal for one scan line is extracted for every 10 scan lines,
A-D conversion is performed by sampling.

[Brief description of drawings]

FIG. 1 is a functional block diagram of an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 3 is a flowchart showing the operation of one embodiment.

FIG. 4 is a flowchart showing an operation of one embodiment.

FIG. 5 is a flowchart showing the operation of one embodiment.

FIG. 6 is an explanatory diagram related to processing of a flowchart.

FIG. 7 is an explanatory diagram relating to the processing of the flowchart.

FIG. 8 is an explanatory diagram relating to the processing of the flowchart.

FIG. 9 is an explanatory diagram relating to the processing of the flowchart.

FIG. 10 is an explanatory diagram relating to the processing of the flowchart.

FIG. 11 is an explanatory diagram relating to the processing of the flowchart.

FIG. 12 is an explanatory diagram relating to the processing of the flowchart.

FIG. 13 is an explanatory diagram relating to the processing of the flowchart.

FIG. 14 is an explanatory diagram relating to the processing of the flowchart.

FIG. 15 is a flowchart according to one embodiment.

FIG. 16 is a flowchart showing the operation of one embodiment.

FIG. 17 is an explanatory diagram related to the processing of the flowchart, (a) is an explanatory diagram of continuous data on a straight line, and (b) is an explanatory diagram of continuous data having an upward convex.

FIG. 18 is an explanatory diagram relating to the processing of the flowchart.

FIG. 19 is an explanatory diagram relating to the processing of the flowchart.

FIG. 20 is an explanatory diagram relating to the processing of the flowchart.

FIG. 21 is an explanatory diagram relating to the processing of the flowchart.

FIG. 22 is a flowchart according to one embodiment.

FIG. 23 is a flowchart according to one embodiment.

FIG. 24 is a flowchart according to one embodiment.

FIG. 25 is an explanatory diagram relating to the processing of the flowchart.

FIG. 26 is an explanatory diagram relating to the processing of the flowchart.

FIG. 27 is an explanatory diagram relating to the processing of the flowchart.

FIG. 28 is an explanatory diagram relating to the processing of the flowchart.

FIG. 29 is an explanatory diagram relating to the processing of the flowchart.

FIG. 30 is an explanatory diagram relating to the processing of the flowchart.

FIG. 31 is an explanatory diagram relating to the processing of the flowchart.

FIG. 32 is an explanatory diagram relating to the processing of the flowchart.

FIG. 33 is an explanatory diagram relating to the processing of the flowchart.

FIG. 34 is an explanatory diagram relating to the processing of the flowchart.

FIG. 35 is an explanatory diagram of a process of the flowchart.

FIG. 36 is an explanatory diagram related to processing of the flowchart.

FIG. 37 is an explanatory diagram related to the processing of the flow chart, (a) is an explanatory diagram of a state in which the continuous data of the eyebrows is used as a reference, and the same area is enlarged downward and set, and (b) is used as a reference of continuous data of the eyes. Is an explanatory diagram in which the same region is enlarged and set in the upward direction, and (c) is an explanatory diagram in which the same region is enlarged in the upward direction with reference to continuous data below the spectacle frame.

[Explanation of symbols]

CL1 Image inputting means CL2 Density detecting means CL3 Point extracting means CL4 Curve data extracting means CL5 Setting means for predetermined area 1 (eye position search area 1 setting means) CL6 Curve data detecting means CL7 Setting means for predetermined area 2 (eye position Search area 2 setting means) CL8 Curve data movement state determining means CL9 Predetermined area 3 setting means (eye position search area 3 setting means) CL10 Predetermined area 1 reduction means (eye position search area 1)
CL11 Eye position detecting means 21 TV camera 22 A / D converter 23 Image memory 24 Image data arithmetic circuit 25 Eye position detecting circuit

Front page continuation (58) Fields surveyed (Int.Cl. ⁷ , DB name) G01B 11/00-11/30 G06T 7/20

Claims (12)

(57) [Claims] 1. An image input unit for inputting image data including a face, and a pixel density read out along a vertical pixel row of the input image data, and the density value on the pixel row in one direction. Point extraction means that specifies the pixel of the peak in one direction of the density value as the extraction point, provided that the differential value of the density change before the peak exceeds a predetermined value, and in the pixel row direction of the adjacent pixel row Curve data extraction means for extracting curve data continuously extending in the lateral direction of the face from the adjacent extraction points, movement state determination means for determining a movement state of the curve data, and the curve data being dynamic An eye position detecting device comprising an eye position detecting means for detecting the eye position based on the curve data when the eye position is detected. 2. The eye position detecting apparatus according to claim 1, wherein the eye position searching area 1 including the curve data extracted by the curve data extracting means is set. The position search area 1 setting means, the uppermost end curve data detection means for detecting the curve data appearing at the uppermost end in the eye position search area 1, and the eye position search area 2 based on the uppermost end curve data. The eye position search area 2 setting means to be set, the movement state determination means of curve data for determining the movement state of the curve data appearing in the eye position search area 2, and the target curve data are dynamic. If it is determined that the eye position search area 3 is set, the eye position search area 3 is set.
Setting means, and when it is determined that the target curve data is static curve data, returns to the setting of the eye position search area 1 and searches the eye position based on the position information of the corresponding curve data. A means for reducing the eye position search area 1 for reducing the range of the area 1 in the vertical direction, and an eye position detection means for detecting the eye position from the relative positional relationship of the curve data appearing in the eye position search area 3. An eye position detecting device comprising: 3. The eye position detecting device according to claim 1, wherein the movement state determination means determines the movement state of the curve data based on a vertical movement amount of the curve data. Eye position detection device. 4. The eye position detecting device according to claim 3, wherein the movement state determination means determines the movement state of the curve data by an average value of vertical positions of extraction points forming the curve data. An eye position detecting device characterized by: 5. The eye position detecting apparatus according to claim 1, wherein the movement state determination means determines the movement state of an uppermost end position of the extraction points forming curve data. An eye position detecting device characterized in that it is performed with a thing. 6. The eye position detecting device according to claim 1, wherein the movement state determination unit determines a shape of curve data that appears in the eye position search area 2. Determination means,
An eye position detecting apparatus comprising: a moving state determination time changing unit that changes a determination time of the moving state determining unit of the curve data according to an output result of the shape determining unit. 7. The eye position detection device according to claim 1, wherein the movement state determination unit determines a shape of curve data that appears in the position search area 2. And a movement state determination movement amount changing unit that changes the determination movement amount of the movement state determination unit of the curve data according to the output result of the shape determination unit. 8. The eye position detecting device according to claim 1, wherein the movement state determination unit determines a shape of curve data that appears in the eye position search area 2. Determination means,
An eye position detecting apparatus comprising: a setting condition changing unit for changing a setting condition of the eye position searching region 3 according to an output result of the shape determining unit. 9. The eye position detecting device according to claim 2, wherein the movement state determining unit extends to the left and right of the face image data based on the curve data that appears in the eye position search area 2. Eye position search area 4 setting means for setting the horizontally long eye position search area 4 and left and right data for determining whether or not there are two or more left and right curve data in the set eye position search area 4 It is characterized by comprising a judging means and a setting condition changing means of the area 3 for changing the setting condition of the eye position search area 3 by specifying the curved line data based on the size of the upper and lower steps of the judged left and right data. Eye position detection device. 10. The eye position detecting apparatus according to claim 9, wherein the left and right data determining unit selects two curve data from an upper portion in the eye position search area 4. Detection device. 11. The eye position detecting device according to claim 9, wherein the setting condition changing means of the region 3 is a position search region of the eye when the step difference of the left and right data is less than or equal to a predetermined value. 4. An eye position detecting device comprising shape determining means for determining the shape of curve data appearing in 4 to identify the curve data. 12. The eye position detecting device according to claim 2, claim 6, claim 7, claim 8, or claim 11, wherein the shape determining unit determines the shape of the curve data from the curve data. Connect the rightmost and leftmost points of each of the constituent points with a straight line,
An eye position detecting device characterized in that a shape is recognized based on a position of an extraction point at the center of the curve data with respect to the straight line.