JP3531503B2 - Eye condition detection device and drowsy driving alarm device - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an eye condition detecting device used for automatically controlling a mirror and a seat and for identifying a position of an eye necessary for detecting a dozing condition from a driver's eye open / closed condition. , And a drowsiness alarm device capable of warning drowsiness.

[0002]

2. Description of the Related Art As a conventional drowsiness driving warning device based on image processing, there is, for example, a device described in JP-A-10-143669. This is configured such that the eye position is detected for a grayscale image, and the eye is tracked while performing open / closed eye determination based on the eye opening value. In addition, the conventional device is configured to recognize an eye tracking error from the output state of the eye opening value and automatically return to the eye position detection processing.

[0003]

However, in the conventional device, when the eye tracking fails and the eye position is detected again, the eye position is detected by returning to the initial state. There was a problem that it took a long processing time again. In addition, even if the initial eye position detection succeeds and there is a state in which eye open / close determination is continued stably while tracking the eye, there is a risk of detection error due to the eye position detection process in the initial state again. There is a problem with it.

An object of the present invention is to provide an eye condition detecting device and a drowsy driving warning device which can detect the detection speedily and accurately.

[0005]

According to a first aspect of the present invention, a face image input means for inputting a face image, and a local density of pixels in a vertical pixel array of face image data input by the face image input means. A pixel for each increasing height as an extraction point, and an eye position specifying means for specifying an eye position based on continuous data in which the extraction points of a pixel row adjacent to the extraction point are close to each other and extend in the lateral direction of the face. A predetermined area setting means for setting a predetermined area including the specified eye position, and an eye position is specified based on the continuous data in the predetermined area set by the predetermined area setting means. In an eye state detection device including an eye opening degree detection unit that detects the eye opening degree from a concentration change state, a reference area of the eye that can be predicted to exist based on the continuous data is determined and stored. A first eye reference region determining means, and Eye tracking state determination means for tracking continuous eye data within a predetermined area and determining whether the tracking state is normal based on the state of the continuous data, and eye tracking by the eye tracking state determination means When judged to be abnormal,
First reference means for returning the predetermined area to the reference area is provided.

According to a second aspect of the present invention, a face image input means for inputting a face image, and a pixel for each local increase in density in a vertical pixel row of the face image data input by the face image input means are provided. As an extraction point, and an eye position specifying means for specifying an eye position based on continuous data in which the extraction points of a pixel row adjacent to the extraction point are close to each other and extend in the lateral direction of the face, and the specified eye. A predetermined area setting means for setting a predetermined area including the position of the eye, the eye position is specified based on the continuous data within the predetermined area set by the predetermined area setting means, and the eye position is determined based on the density change state of the continuous data. In an eye condition detection device including an eye opening detection unit that detects an opening, a first eye reference that determines and stores a reference area of an eye that can be predicted to exist based on the continuous data. Area determining means and the area within the predetermined area. Eye tracking state determining means for determining whether or not the tracking state is normal based on the state of the continuous data, and the eye tracking state determining means for determining an eye tracking abnormality When, the left and right area setting means for setting the left and right areas that can detect the continuous data of the left and right eyes based on the reference area, and the continuous data for determining whether continuous data appears in the left and right areas Appearance state determination means and re-detection means for causing the eye opening detection means to perform the detection again based on the continuous data when it is determined that continuous data appears in the left and right regions It is characterized by that.

According to a third aspect of the present invention, a face image input means for inputting a face image, and a pixel for each local increase in density in a vertical pixel row of the face image data input by the face image input means are provided. As an extraction point, and an eye position specifying means for specifying an eye position based on continuous data in which the extraction points of a pixel row adjacent to the extraction point are close to each other and extend in the lateral direction of the face, and the specified eye. A predetermined area setting means for setting a predetermined area including the position of the eye, the eye position is specified based on the continuous data within the predetermined area set by the predetermined area setting means, and the eye position is determined based on the density change state of the continuous data. In an eye condition detecting device having an eye opening detecting means for detecting the degree of opening, continuous eye data in the predetermined area is tracked, and whether or not the tracking state is normal based on the state of the continuous data. Eye tracking state determination means for determining whether When the eye tracking state determination means determines that the eye tracking is abnormal, a left and right area setting means for setting left and right areas capable of detecting continuous data of the left and right eyes based on the reference area, and within the left and right areas. Appearance state determination means of continuous data for determining whether continuous data has appeared, and when it is determined that the continuous data has appeared, the detection is performed by the eye position specifying means based on the continuous data. And a second eye reference area determination means that determines and stores a reference area of the eye that can be predicted to exist based on the eye position information identified by the re-identification means. Second reference means for returning the predetermined area to the reference area when the eye tracking state determination means determines that the eye tracking is abnormal.

A fourth aspect of the present invention is the eye state detecting device according to the first, second and third aspects, wherein the first or second reference area determining means repeats the eye position detecting process. The feature is that the reference region of the eye is learned.

According to a fifth aspect of the present invention, there is provided the eye state detecting device according to the first, second and third aspects, wherein the first or second reference area determining means is a time series of eye opening value. The feature is that the reference state of the eye is learned by determining the change state.

According to a sixth aspect of the present invention, there is provided the eye state detecting device according to the first, second, and third aspects, wherein the first or second reference area determining means performs a plurality of eye position detecting processes. It is characterized in that the reference area of the eye is determined when the stable output of the eye position is performed.

The invention of claim 7 is the eye state detecting device according to claim 2, wherein the re-detecting means is capable of stably detecting two consecutive laterally within the left and right regions at predetermined intervals. The feature is that re-detection is performed when there is data.

An eighth aspect of the present invention is the eye condition detecting apparatus according to the third aspect, wherein the re-identifying means is capable of stably detecting two consecutive laterally within the left and right regions at predetermined intervals. The feature is that re-detection is performed when there is data.

The invention of claim 9 is the invention of claims 2, 3, 7, and 8.
The eye state detection device described above is characterized in that a determination extension means is provided for extending the determination for a predetermined time when continuous data in the left and right regions cannot be stably detected.

The invention of claim 10 is the invention of claims 2, 3, 7,
The eye condition detection device according to item 8, wherein a determination extension means is provided for extending the determination for a predetermined time when continuous data in the left and right regions cannot be detected.

The invention of claim 11 is based on claims 2, 3, 7,
9. The eye condition detection device according to claim 8, wherein when the continuous data in the left and right regions cannot be stably detected and when the continuous data in the left and right regions cannot be detected, a determination extension means for extending the determination for a predetermined time. Is provided, and the determination extension means sets a longer extension time when continuous data cannot be detected stably than when it cannot be detected.

A twelfth aspect of the present invention is the eye state detecting apparatus according to any one of the first to eleventh aspects, wherein the tracking state determining means moves the continuous data in the predetermined area and stays at that position for a predetermined time. It is characterized in that it is determined that the tracking abnormality of the eye when.

A thirteenth aspect of the present invention is the eye state detecting apparatus according to any one of the first to eleventh aspects, wherein the tracking state determination means detects an abnormal eye tracking when continuous data in the predetermined area cannot be detected for a predetermined time. It is characterized by determining that.

A fourteenth aspect of the present invention is the eye state detection apparatus according to any one of the first to eleventh aspects, wherein the tracking state determination means constantly detects that the eye opening value exceeds a predetermined range. It is characterized in that it is determined to be an eye tracking abnormality when it occurs.

A fifteenth aspect of the present invention is the eye state detecting device according to the first to fourteenth aspects, wherein the eye opening state output by the eye opening detecting means is determined, and the eye opening / closing state is determined. It is characterized in that an alertness degree determining means for determining the alertness degree from the change and an alerting means for issuing an alert when the alertness degree determining means determines that the alertness level is decreasing are provided.

[0020]

According to the first aspect of the present invention, the first eye reference region determining means determines the eye reference region in which the presence of the eye can be predicted based on the continuous data for specifying the eye position, and the eye tracking is performed. When the state determining unit determines that the eye tracking is abnormal, the returning unit can return the predetermined region for detecting the eye opening to the reference region. Therefore, when the detection is performed again, the processing from the initial state to the setting of the predetermined area can be omitted, and the quick processing can be performed. Further, there is no risk of detection error in the omitted processing portion, and more accurate detection can be performed.

According to the second aspect of the present invention, the first eye reference area determining means determines the eye reference area that can predict the presence of the eye based on the continuous data for specifying the eye position, and determines the eye tracking state. When the eye tracking abnormality is determined by the means, the left and right areas capable of detecting continuous data of the left and right eyes based on the reference area are set by the left and right area setting means, and the appearance state determination means continuously sets the left and right areas within the left and right areas. When it is determined that the data has appeared, the eye opening detection means can be made to perform the detection again based on the continuous data. Therefore, when the detection is performed again, the processing from the initial state to the setting of the predetermined area can be omitted, and the quick processing can be performed. Further, there is no risk of detection error in the omitted processing portion, and more accurate detection can be performed. Further, the condition of left-right symmetry of continuous data can be used, and more accurate detection can be performed.

According to the third aspect of the present invention, when the eye tracking state determination means determines that the eye tracking is abnormal, the left and right regions in which the left and right eye continuous data can be detected based on the initially detected eye position continuous data. Set by the left and right area setting means, and when it is determined that continuous data appears in the left and right areas, re-identification means causes the eye position identification means to perform the detection again based on the continuous data, The reference region of the eye that can be predicted to exist based on the position information of the eye identified by the re-identifying unit is determined by the reference region determining unit of the second eye, and the eye tracking state determining unit determines the eye tracking. When it is determined that the abnormality is present, the second returning means can return the predetermined region for detecting the eye opening to the reference region. Therefore, in addition to the value corresponding to the opening value as information for determining whether the re-detected continuous data is an eye,
Since it is possible to use the condition of left-right symmetry of continuous data, it is possible to improve the detection accuracy of the reference region of the eye.
In addition, since the area where the re-detection of the eye is performed is limited to the left and right areas, it is possible to shorten the detection time of the reference position and perform a quick process.

According to the invention of claim 4, in addition to the effect of the invention of claim 1 or 2, the eye reference region can be learned by repeating the eye position detection processing, and more accurate processing can be performed. You can

According to the invention of claim 5, in addition to the effect of the invention of claim 1 or 2, it is possible to learn the reference region of the eye by judging the time series change state of the eye opening value. It is possible to perform accurate processing.

According to a sixth aspect of the invention, in addition to the effect of the first or second aspect of the invention, the eye reference region is output when a stable eye position is output by a plurality of eye position detection processes. Can be determined, and more accurate processing can be performed.

According to the invention of claim 7, in addition to the effect of the invention of claim 2, re-detection is performed when there are two continuous data which can be stably detected at a predetermined interval in the lateral direction in the left and right regions. It is possible to perform more accurate processing by using the condition of left-right symmetry of continuous data.

According to the invention of claim 8, in addition to the effect of the invention of claim 3, re-detection is carried out when there are two continuous data which can be stably detected at predetermined intervals in the lateral direction in the left and right regions. It is possible to perform more accurate processing by using the condition of left-right symmetry of continuous data.

In the invention of claim 9, claims 2, 3, 7,
In addition to the effect of the invention described in 8, when the continuous data in the left and right regions cannot be stably detected, the determination of the appearance state determination means can be extended for a predetermined time, and the continuous data can be captured more accurately, thereby improving accuracy. Good processing can be done.

According to the invention of claim 10, claims 2, 3 and
In addition to the effects of the invention described in 7 and 8, when the continuous data in the left and right regions cannot be detected, the determination of the appearance state determination means can be extended for a predetermined time, and the continuous data can be captured more accurately, thereby achieving higher accuracy. Processing can be performed.

According to the invention of claim 11, claims 2, 3 and
In addition to the effects of the invention described in 7 and 8, when the continuous data in the left and right regions is not stable and when the continuous data in the left and right regions does not exist, the determination by the appearance state determination means can be extended for a predetermined time. By capturing the data more accurately, more accurate processing can be performed. Moreover, the determination extension means sets the extension time longer when the continuous data does not exist than when the continuous data is not stable. Therefore, even when the continuous data does not exist, the continuous data can be more accurately captured after the extension time, and thus the accuracy can be improved. Processing can be performed.

According to the twelfth aspect of the invention, in addition to the effects of the first to eleventh aspects of the invention, when continuous data in a predetermined area moves and stays at that position for a predetermined time, it can be determined that eye tracking is abnormal. Even if continuous data of eyeglasses or the like, which is noise around the eyes due to a rapid movement of the face, is likely to be tracked, this can be accurately canceled and highly accurate re-detection can be performed.

In the thirteenth aspect of the invention, in addition to the effects of the first to eleventh aspects of the invention, it is possible to determine that the eye tracking abnormality has occurred when continuous data in a predetermined area cannot be detected for a predetermined time, and a change in the light environment. Even if there is such a problem, it is possible to perform re-detection with high accuracy.

According to the fourteenth aspect of the invention, in addition to the effects of the first to eleventh aspects of the invention, when the eye opening value exceeds a predetermined range and is constantly detected, it can be determined that the eye tracking abnormality. Therefore, it is possible to perform re-detection with high accuracy even when the face is moved largely to the side.

According to the fifteenth aspect of the present invention, in addition to the effects of the first to fourteenth aspects of the present invention, an alarm can be accurately issued when it is determined that the arousal level has decreased.

[0035]

BEST MODE FOR CARRYING OUT THE INVENTION The present invention can be used as a drowsiness driving alarm for railway cars, ships, operators of plants, etc. in addition to automobiles, but this embodiment will be described when applied to automobiles.

FIG. 1 is a functional block diagram of an eye condition detecting device and a drowsy driving alarm device to which an embodiment of the present invention is applied. The device includes an image input means CL1 and a pixel row density detecting means CL2. A point extracting means CL3, a continuous data extracting means CL4, an eye position detecting means CL5, a point extracting means CL6, an eye opening detecting means CL7, and a first eye reference area determining means CL8. , Eye tracking state determination means CL9, first return means CL10, awakening degree determination means CL11, and alarm means
It is equipped with CL12.

The image input means CL1 inputs a face image. The pixel row density detecting means CL2 detects a local increase in the density of a pixel row in the vertical direction of the input face image data. The point extracting means CL3 detects points based on the increase in the density of the pixel row and the changing state thereof. The continuous data extracting means CL4 continuously extracts the continuous data in the lateral direction of the face by successively adjoining points of adjacent pixel rows in the pixel row direction. The eye position detecting means CL5 detects the eye position based on the continuous data. The pixel row density detecting means CL2, the point extracting means CL3, the continuous data extracting means CL4, and the eye position detecting means CL5 constitute the eye position specifying means of the present invention.

The point extraction means CL6 sets a predetermined area including the specified eye position, determines a pixel for each increase in the density of the pixel row in the vertical direction within the predetermined area, and sets the pixel as an extraction point, which is adjacent to each other. The position of the eye is specified based on continuous data that continuously extends the extraction points in the lateral direction of the face. The continuous data detected by the point extracting means CL6 is finer than the continuous data detected by the point extracting means CL3, and is designed to perform accurate detection, but the latter is detected with the same roughness as the former. By doing so, it is possible to adopt a configuration that accelerates the processing. The eye opening detection means CL7 detects the eye opening based on the density change state of continuous data in a predetermined area.
These point extraction means CL6 and eye opening detection means CL7
And constitute the predetermined area setting means and the eye opening detection means of the present invention.

The eye reference area determining means CL8 determines a reference position as an eye reference area that can be predicted to exist based on the continuous data.

The eye tracking state determining means CL9 tracks the continuous data of the eyes in the predetermined area and determines whether the tracking state is normal based on the state of the continuous data.

The first returning means CL10 returns the predetermined area to the reference position when the eye tracking state determination means determines that the eye tracking is abnormal.

The awakening degree determining means CL11 determines the eye opening state output by the eye opening determining means, and determines the awakening degree from the change in the open / closed state.

The alarm means CL12 issues an alarm such as a buzzer when the awakening degree determining means determines that the awakening degree is low.

FIG. 2 is a block diagram showing the configuration of an embodiment of the present invention.

As shown in the same figure, the TV camera 21 is installed in the instrument of the automobile and equipped with a TV camera 21 as an image input means CL1 for photographing the driver's face from the front.
In the present embodiment, the input image of the camera 21 includes 512 pixels in the horizontal (X) direction and 480 pixels in the vertical (Y) direction.

The input image captured by the TV camera 21 is stored in the image memory 23 as digital input image data via the A / D converter 22. The output of the image memory 23 is input to the image data arithmetic circuit 24.

The image data calculation circuit 24 detects the density of the pixel row in the vertical direction of the face based on the input image data,
Points are extracted (extraction points) according to the increase in the density of the pixel row and the change state thereof, and the points adjacent to each other in the pixel row direction of the adjacent pixel rows are continuously extracted to extract continuous data in the lateral direction of the face. That is, the image data arithmetic circuit 24 constitutes the density detecting means CL2 for the pixel column, the point extracting means CL3, and the extracting means CL4 for the continuous data. The output of the image data calculation circuit 24 is input to the eye position detection circuit 25.

The eye position detection circuit 25 detects the eye position by selecting the eye from the continuous data. That is, the eye position detecting circuit 25 is used for the eye position detecting means CL.
Make up 5. The output of the eye position detection circuit 25 is input to the open / closed eye detection circuit 26.

The open / closed eye detection circuit 26 sets a predetermined area including the eyes (eye tracking area), and extracts points from the vertical density increase and its change state within the predetermined area (extraction point). , Continuous data in the lateral direction of the face is continuously extracted from points adjacent to each other in the pixel row direction of adjacent pixel rows, and the eye opening is detected from the density change state of the continuous data in the range specified by the points. To do. That is, the open / closed eye detection circuit 26 constitutes the eye opening degree detection means CL7.
The output of the open / closed eye detection circuit 26 is input to the backup circuit 27.

The backup circuit 27 determines a reference position of the eye that can predict the presence of the eye based on the continuous data, tracks the continuous data of the eye within the predetermined area, and based on the state of the continuous data. It is determined whether the eye tracking state is normal, and when the eye tracking abnormality is determined to be the eye tracking abnormality and the eye tracking failure is confirmed, the eye tracking area is previously learned and the predetermined area is returned to the reference position of the eye. . That is, the backup circuit 27 is provided with the first eye reference area determining means CL.
8, the eye tracking state determining means CL9, and the first returning means CL10. The output of the backup circuit 27 is input to the awakening degree determination circuit 28.

The awakening degree judging circuit 28 judges the awakening degree from the detection result of the open / closed eyes sent from the open / closed eye detecting circuit 26. That is, the awakening degree determining circuit 28 constitutes the awakening degree determining means CL11. The output of the awakening degree determination circuit 28 is input to the alarm device 29. The alarm device 29 is
When it is determined that the arousal level has decreased, a warning is issued to call the driver's attention. That is, the alarm device 29 constitutes the alarm means CL12.

Next, the flow of the entire operation in the above configuration will be described based on the flowcharts of FIGS. 3 and 4.
First, in step 301, the face portion is photographed by the TV camera 21, and in step 302, one frame of the input image is converted into a digital signal by the AD converter and stored in the image memory.

Next, in step 303, it is checked whether or not the eye tracking area is set. The eye tracking area refers to an area set as a predetermined area including the eye in the entire image, and recognizes the detailed position of the eye by detecting the position of continuous data that appears in this area, and identifies the position. The eye opening is detected by reading the density data at the different positions. Further, by setting the processing area in the next captured image based on the detailed position of the continuous data appearing in the predetermined area, it is possible to deal with the change in the eye position.

When the eye tracking area is not set,
In step 304 and step 305, the width in the horizontal direction (X direction) and the width in the vertical direction (Y direction) that are the eye tracking regions are set. Details of eye position detection are shown in FIG.
Will be described later with reference to the flowchart of FIG.

When it is determined in step 303 that the eye tracking area is set, in step 306 the eye opening and detailed position detection are performed. Details of this processing will be described later with reference to the flowchart of FIG. 12 and the explanatory diagrams shown in FIGS. 13 to 15.

Then, in step 307, it is judged whether or not the eye opening value is normal, and if normal, the process proceeds to step 401 in FIG. In step 401, it is determined whether or not the reference position of the eye has been learned. Since the determination in this step in the initial state is naturally not learned, the process shifts to step 403 to start learning the eye reference position. The method of learning the reference position of the eye is based on the condition that the output state of the eye opening value is continuously changing within the range considered to be the size of the eye and that there is a position where the data is retained. Memorize the reference position. In this case, the retention of data means that the reference position of the eye is too long (for example, 3
It says that it has stopped for more than 2 seconds.

While learning the reference position of the eye in step 403, the eye open / closed eye is determined based on the eye opening value in step 404, and the eye tracking area is updated in step 405.
Further, in step 406, the awakening degree is determined based on the appearance pattern of the eye-closed output. If it is determined in step 406 that the eyes are closed and the eyes are closed for a long time, an alarm is issued to alert the driver.

After the awakening level is determined in step 406, the process returns to step 301 to input the next image and continue the same processing. In the flow of this process, if the eye opening value deviates from the normal eye size range and becomes extremely large (such as a person with thick eyebrows) or becomes small (such as the line of the mouth), In step 307, it is recognized that the eye opening value is abnormal, and the process proceeds to step 308. In step 308,
Whether the abnormal state of the eye opening value is temporary or steady is confirmed by a plurality of processes, and if it is determined to be steady, the eye tracking area is determined in step 309. clear.

Then, steps 301 to 303
Although the same processing is repeated as described above, in the determination in step 303, since the eye tracking area is cleared due to the occurrence of a steady eye opening value abnormality, in the case of flowing this flow, from the entire image. The eye position will be detected again.

After learning the eye reference position in step 403,
In the case of the flow from step 401 to step 402, when it is determined in step 402 that eye tracking has failed, the process proceeds to step 407, and the tracking area to the reference position of the eye is returned to step 301. Then, the next image is input and the same processing is continued.

In other words, if the driver makes an eye tracking error due to the rapid movement of the driver's face, the deterioration of the light environment, and the noise around the eyes when wearing glasses, the amount of movement and its movement amount are determined in step 402. The accuracy of accuracy can be improved by recognizing a tracking error by the dwell time at the position and returning the eye tracking area to the eye reference position where the eye is most likely to exist when the driver is looking ahead of the vehicle and driving. High eye re-detection can be done in an instant. Details of the operating state by this processing will be described later with reference to the explanatory diagrams shown in FIGS. Details of a specific eye tracking method will be described later with reference to the explanatory diagram of FIG.

Next, details of eye position detection will be described.

The flow of the eye position detecting process will be described with reference to the flowchart of FIG. First, in step 501, as shown in FIG. 6, a point extraction process is performed on the data on the line in the Y-axis direction, and after the completion of one line, the process proceeds to the process of the next adjacent line, and all lines in the predetermined direction are processed. It is determined whether or not the point extraction has been completed.

If it is determined in step 501 that points have not been extracted on all lines, the process proceeds to step 502. In this step 502, the arithmetic mean of the density values of one line in the predetermined direction is calculated. This processing is intended to eliminate small variations in changes in density value during image data capturing, and is for capturing global changes in density value. FIG. 7A shows the processing result of the arithmetic mean calculation of the line data of Xa in FIG.

In step 503 of FIG. 5, step 50
Differentiation is performed on the arithmetic mean value, which is the calculation result of 2. The result of this processing is shown in FIG. In step 504 of FIG. 5, points are extracted by the differential value that is the calculation result of step 503. How to extract that point
Points where the differential value changes from negative to positive (p1 to p5),
In FIG. 7A, points at which the graph is convex downward are extracted. Next, it is determined whether or not the change (q1 to q5) in the density value until reaching that point is less than or equal to a predetermined value, that is, whether or not it enters the gray portion of FIG. The Y coordinate values (A1 to A3) are extracted for the points having the change in density value. After this processing is completed for one line, the processing is switched to the processing for the next line in step 505.

When this process is repeated, in the case of a line such as Xb shown in FIG. 6, there may be no extraction point, as can be seen from FIGS. 8 (a) and 8 (b).

When it is determined in step 501 that the points of all the lines have been extracted, the points shown in FIG. 9 are extracted. That is, on the Xc line in FIG. 9, A1,
Two points A2 are extracted, and four points A1, A2, A3, and A4 are extracted on the Xd line.

Then, the process proceeds to step 506 in FIG.
Extraction points of adjacent lines (A1, A2, A3 ...)
If the Y coordinate value is within a predetermined value, the continuous data group number, continuous start line number, and continuous data number are stored as continuous data. The specific processing contents will be described with reference to FIG.

Line 1 has Y coordinate values 192 and 229.
There are two extraction points. Y coordinate value of line 1 is 1
Since the line adjacent to the left does not exist at the point 92, there is no continuous data at this stage, so the group number of the continuous data is "1". Also, for the point of the Y coordinate value 229, there is no continuous data at this stage for the same reason, so the group number of the continuous data is set to the next “2”. Next, in the line 2 on the right side, there are two extraction points with Y coordinate values of 191 and 224. Since the point of the Y coordinate value 191 of the line 2 is within 10 with the Y coordinate value 192 of the left adjacent line 1, the group number of continuous data is set to "1". At this time, the number of continuous data is 2. When the same determination is made at the point of the Y coordinate value 224 on the line 2, the group number of continuous data is "2" and the number of continuous data is 2.

The continuous start line number in step 506 of FIG. 5 is a line number having a point where the number of continuous data is judged to be 1.

Next, in FIG. 10, the Y coordinate value 3 of line 3
At 60 points, since there is no point within 360 and 10 in the line 2 on the left side, the group number of continuous data is “3” and the number of continuous data is 1.

In step 506 of FIG. 5, the continuity of the points of each line is determined in this way until all the lines are completed, and the process proceeds to step 507.

In step 507, the average value of the Y coordinate values of the points having the same continuous data group number is stored in the average value of the continuous points. This value can be used as the representative Y coordinate value of the group. Also, a continuous end line is obtained from the continuous start line and the number of continuous data, and the average value of the continuous start line and the continuous end line is stored. This value can be used as the representative X coordinate value of the group.

In step 508, the length of each continuous group,
The position of the eye can be specified by making a determination based on the (X, Y) coordinate values.

Here, a specific eye position detecting method will be described with reference to FIG.

First, considering the eye feature amount, it can be defined that the shape is a horizontally long, upwardly convex arcuate shape, and if the continuous data is narrowed down based on this definition, the eye will become horizontal. From the condition that the continuous point is long, the number of continuous points continues for 5 points or more, and from the condition that the continuous shape is an arcuate shape, the difference between the Y coordinate values of the continuous start point and the continuous end point can be narrowed to small continuous data. If continuous data is narrowed down based on this determination, FIG.
Groups G1 to G6 as shown in (a) are extracted.

Next, considering the positions of the representative coordinate values of X and Y of each group described above, as shown in FIG.
ZONE: L, ZON from the degree of approach in the coordinate direction
It can be classified into E: C and ZONE: R. This is because the left eye and the left eyebrow are not greatly separated in the X coordinate direction, and the right eye and the right eyebrow are not greatly separated in the X coordinate direction. The continuous data of the mouth and the continuous data of the mouth are located near the center.

In this way, the eye position can be easily detected by further classifying the data according to the degree of approach in the X coordinate direction and narrowing the data. The continuous data included in ZONE: L includes the left eye and the left eyebrow.
When it is determined that the continuous data included in E: R is the right eye and the right eyebrow, the eye positions are G3 and G4, and their coordinate values can also be specified. In this way, eye position detection can be performed for both eyes, but for the purpose of doze detection, it is considered that no driver sleeps with only one eye closed, so opening and closing described below Eye detection is limited to one eye (left eye). The reason for limiting to one eye (left eye) of this driver is not only to save the calculation processing time, but in the case of a right-hand drive vehicle, the side light is closer to the side window than the left eye and the right light is more likely to hit. Since it is highly possible that the shape of the eye becomes difficult to capture due to the intensity of light,
The accuracy of detecting the open / closed eye is improved when the left eye is less likely to be hit by the direct light than the right eye. Therefore, in the case of a left-hand drive vehicle, it is preferable to limit it to the right eye.

Next, details of the eye opening detection in step S306 of FIG. 3 will be described.

A method for detecting the eye opening value is shown in FIG.
As shown in 4, the Q point where the density change from the white part of the skin to the black part of the eye is maximum and the R point where the density change from the black part of the eye to the white part of the skin is maximum are There are a method of obtaining an interval and a method of obtaining a vertical width of a black portion of an eye by setting a binarization threshold value which will be described with reference to the flowchart of FIG.

Now, the details of the eye opening detection processing by setting the binarization threshold will be described.

First, the flow of a binarization threshold value setting method for converting into a binarized image for detecting the opening will be described with reference to the flowchart of FIG. First, step 12
In 01, as shown in FIG. 13A, the point extraction process is performed on the data on the line in the Y-axis direction, and after one line is completed, the process proceeds to the process for the next adjacent line, and all lines in the predetermined direction are processed. It is determined whether or not the point extraction has been completed. If it is determined in step 1201 that points have not been extracted on all lines, step 120
Move to 2.

In this step 1202, 1 in the predetermined direction
The arithmetic mean of the density values of the line is calculated. This processing is intended to eliminate small variations in changes in density value during image data capturing, and is for capturing global changes in density value. FIG. 13B shows the processing result of the arithmetic mean calculation of the line data of Xa in FIG.

In step 1203 of FIG. 12, differential calculation is performed on the arithmetic mean value which is the calculation result of step 1202. The result of this processing is shown in FIG. In step 1204 of FIG. 12, point extraction is performed using the differential value that is the calculation result of step 1203. The extraction method of that point is the point P at which the differential value changes from negative to positive,
In FIG. 13B, points at which the graph is convex to the left are extracted. Next, it is judged whether the change of the density value before and after the point is less than or equal to a predetermined value or more, and whether it enters the gray portion of FIG. 13C, and the change of the density value satisfying the above condition is determined. The possessed point P is extracted.

In step 1205 of FIG.
When it is determined that the extraction point P exists in the Xa column as shown in (a), the process proceeds to step 1206 of FIG. 12 and the maximum and minimum differential values before and after the point P, that is, the Q point and the R point.
The density value of the Y coordinate of the point is stored in Nal (the density value of the Y coordinate with the maximum differential value) and Nah (the density value of the Y coordinate with the maximum differential value), and the process proceeds to step 1207.

In step 1207, the process is switched to the next line, and the same process is repeated until it is confirmed in step 1201 that all lines have been processed. That is, FIG.
As shown in FIG. 4A, in the Xb column after the Xa column, the density values of the Y coordinates of the Q point and the R point, which are the maximum and minimum differential values before and after the extraction point P point of the Xb column, are set to Nbl and Nbh. As memory goes. The minimum differential value of the density change before the extraction point stored in each line, N_1, is the point at which the density value changes most from the bright part to the dark part, and the part darker than this density value, that is, the extraction point P point. It can be said that it is the density value of the part corresponding to the eye toward. Further, N_h, which is the maximum differential value of the density change after the extraction point stored in each line, is the point where the density value changes most from the dark part to the bright part, and the extraction point P
It can be said that the part that becomes bright from the point to this density value is the density value that corresponds to the eye. Therefore, the values of N_1 and N_h should be used as threshold setting information for surely converting the eye part into the black pixel (0) and the skin part around the eye into the white pixel (1) in the binarization process. You can

When it is confirmed in step 1201 of FIG. 12 that the processing of all lines has been completed, the process proceeds to step 1208, and the density values (N_1, N_) of the extraction points of each line.
A binarization threshold value is set from the information of h). The following method can be considered for setting the binarization threshold value.

The minimum value of the density N_1 of the minimum differential value of each line is updated and the value is set as a reference.

The maximum value of the density N_h of the maximum differential value of each line is updated and the value is set as a reference.

The average value of the density N_1 of the minimum differential value of each line is set as a reference.

The average value of the density N_h of the maximum differential value of each line is set as a reference.

The average value of all N_1 and N_h of each line is set as a reference.

The gradation increase amount is corrected and set using the level of the differential value with reference to the brightest density value of the point P of each line extracted from the continuous eye data. (When the level of the differential value is large, the gradation added to the brightest density value at point P is set large, and when the level is small, the gradation added to the brightest density value at point P is set small. Next, a method of detecting the eye opening will be described with reference to FIG.

The binarization processing is performed by further limiting the region within the range in which continuous data appears using the binarization threshold value obtained by the above method. The binarized image is as shown in FIG. 15 when the detection target person is normal and when the person is asleep.
At this time, when the maximum number of consecutive vertical eye portions that are converted to black pixels is counted, the value becomes large in the normal state (open eyes) and becomes small in the dozing state (closed eyes). In this way, the eye opening is detected.

Next, details of the method for determining the open / closed eye in step 404 of FIG. 4 will be described.

When the driver is specified, the output value of the eye opening changes between the opened state and the closed state. Therefore, the reference value for determining whether the eye is open or closed, that is, the threshold for determining whether the eye is open or closed is within the range. As a result of examining how it is appropriate to set the threshold here, some people in the state of dozing are not in a deep sleep state and some drivers do not completely close their eyes. It seems desirable to use a threshold.

Next, step 40 in the flow chart of FIG.
Details of the eye tracking method in 5 will be described with reference to FIG. In the initial state of this process, that is, in the first frame, the eye tracking area is not set, so the eye tracking area is determined in steps 304 and 305 of FIG. At this time, the center coordinates of the eye and the center coordinates of the eye tracking area are shown in FIG.
They match as shown in (a).

After the completion of the series of processes, the process moves to the second frame, and the process proceeds to step 303 in FIG. 3. Since the eye tracking region has already been set here, the process proceeds to step 306 and the eye opening degree. And position detection. At this time, the detected eye position is as shown in FIG. FIG.
While the eye tracking area in (b) is located at the position set in the first frame, the current eye position is the image data captured in the second frame. The dots are offset from the eye tracking area. However, as long as the eye is within the eye tracking area, it is possible to detect the eye opening and the detailed position of the eye.

After that, when the eyes are correctly detected, step 307 in FIG. 3 to step 4 in FIG.
The process proceeds from 01 to step 405. Step 40
In FIG. 5, the eye tracking area is updated based on the center coordinates shown in FIG. 16B, and is set as the image capturing area in the next frame. Therefore, unless the eye moves extremely fast, the eye tracking area can be made to follow the movement. FIGS. 16C and 16D show the positional relationship between the eye position and the eye tracking area in the face image data captured in the third frame and the fourth frame.

Next, an example in which an eye tracking error is committed and a backup method at that time will be described with reference to FIGS. After learning the eye reference position, in the determination of the eye tracking situation in step 402 of FIG. 4, the eye tracking region moves to a position more than a predetermined distance away from the previously learned eye reference position, and the position When it stays for a predetermined time or more, it is determined that an eye tracking error has occurred, and the process proceeds to step 407 to restore the eye tracking area to the eye reference position.

This phenomenon will be described with reference to FIG.
In the case of a driver wearing eyeglasses as shown in FIG. 7, there is noise that can be eye candidate data around the eyes, and a slight up and down movement of the face causes the eye tracking area at the position as shown in FIG. May come. At this time, the continuous data 1 corresponding to the eye, which is the original detection target, is out of the tracking area, and the continuous data 2 corresponding to the frame of the glasses, which is noise, enters the same area and the continuous data 2 must be selected. The situation may not be obtained. If erroneous continuous data is detected in the area shown in FIG. 18, the continuous data will be tracked, and the eye tracking area will move. In such a case, the eye tracking area can be automatically restored as shown in FIG. 19 by performing the processing of steps 402 and 407 of FIG.

Therefore, when the detection is performed again, the processing up to the detection of the eye tracking area can be omitted, and the rapid processing can be performed. Further, there is no risk of detection error due to eye position detection in the initial state again, and highly accurate processing can be performed.

In the above embodiment, the continuous eye data is always present, its movement is confirmed, and it is a backup method for a detection error staying at that position. However, in steps 307 and 308 of FIG. When the eye opening degree abnormality constantly occurs, it is necessary to detect the eye position from the initial state again. Explaining the case where such a state occurs, as shown in FIG. 20 (a), strong sunlight hits the right half of the driver's face,
When the contrast of the left eye, which is the shaded part, has become extremely low and it is no longer possible to confirm it as continuous data, or as shown in (b), the driver moves his face quickly and greatly,
If the state continues for a long time and the eye tracking region cannot follow the movement and stays at that position, a state in which the eye opening value is abnormal occurs constantly.

Therefore, an embodiment for solving this problem will be described.
This will be described with reference to the functional block diagram of FIG. 21, the flowcharts of FIGS. 22 to 24, and the explanatory diagrams of FIGS. 25 to 34.

In the present embodiment, as shown in FIG. 21, the image input means CL2101, the density detection means CL2102,
Point extraction means CL2103, continuous data extraction means CL2104, eye position detection means CL2105, first eye reference area determination means CL2106, point extraction means CL2107, eye opening detection means CL2108, eye opening value status Judgment means CL2109, area setting means CL2
110, appearance state determination means CL2111, re-detection means C
L2112, eye tracking state determination means CL2113, first
Return means CL2114, awakening degree determination means CL211
5, the alarm means CL2116 is provided.

The image input means CL2101 inputs a face image. The density detecting means CL2102 detects the density of the vertical pixel array of the face image input from the image inputting means CL1201. The point extracting means CL21
Reference numeral 03 extracts points according to the increase in the density of the pixel row and the change state thereof. The continuous data extracting means CL2104 continuously extracts the continuous data in the lateral direction of the face by successively adjoining the adjacent pixel rows in the pixel row direction. An eye is selected from the eye position detecting means CL2105 and continuous data.

The eye reference area determining means CL2106
Determines the reference position as the eye reference region by a plurality of times of detection processing by the eye position detection means CL2105. The point extracting means CL2107 extracts points based on the vertical density increase in a predetermined area including the eyes (eye tracking area) and the change state thereof. The eye opening detection means CL2108 detects the detailed position of the eye by extracting continuous data that extends in the lateral direction of the face in succession in the pixel row direction of the pixel rows adjacent to the extraction point. The range in which the density value is read out is specified, and the eye opening is detected from the density change state within the range. The eye opening value status determination means CL2109 detects a steady eye opening value abnormal state.

The area setting means CL2110 is a left / right area for the purpose of detecting left-right symmetry data based on the reference position of the eye when the eye opening value status determination means CL2109 detects an abnormal eye opening value state. To set. The continuous data appearance state determination unit CL2111 determines the continuous data appearance state in the left and right regions where the left-right symmetrical data is detected.

The re-detection means CL2112 is an eye opening detection means CL based on the appearance state determination result of continuous data.
Re-detection by 2108 is performed. The eye tracking state determination unit CL2113 is a predetermined area including the eye (eye tracking area).
The tracking state is determined by tracking the data based on the appearance position of continuous data corresponding to the eye. The first
The return means CL2114 is an eye tracking state determination means CL.
When 2113 determines that the eye tracking is abnormal, a predetermined area including the eye (eye tracking area) is returned to the reference position of the eye.

The awakening degree determining means CL2115 determines the open / closed state of the eye based on the opening value output from the eye opening degree detecting means CL2108, and determines the awakening degree from the change in the open / closed state. The alarm means CL2116 issues an alarm when the awakening degree determining means CL2115 determines that the awakening degree is decreasing.

Next, the flow of operation in the above configuration will be described in detail with reference to the flow charts of FIGS. 22 to 24 and the explanatory diagrams of FIGS. Note that the processing content in each step to be described below will be performed focusing on the points different from the above-described embodiment.

In step 2203 of FIG. 22, it is confirmed whether or not the left and right regions in which the left and right symmetrical continuous data are detected are set. This area is set based on the reference position of the eyes as shown in FIG. 25, and its size is an area in which continuous data of both eyes are completely included.

Next, in step 2204, it is checked whether the eye tracking area is set. If the eye tracking area is not set, step 2205
Then, the eye position is detected by the same method as in the above-described embodiment. After that, in step 2206, the reference position of the eye is learned, and it is determined in step 2207 whether the learning is completed.
Check with.

An example of the method for learning the eye reference position will be explained. In the eye position detection results from the entire image for a plurality of times, the eye reference position is determined when the eye position is detected a predetermined number of times at the same position. A method of confirming it can be considered. Therefore, when the eye reference value is being learned, the process returns from step 2207 to step 2201 and the eye position detection is repeated a plurality of times.

If it is confirmed in step 2207 that the reference position of the eye has been learned, the process proceeds to step 2208 and the width in the horizontal direction (X direction) and the width in the vertical direction (Y direction) to be the tracking region of the eye. To set.

Thereafter, in step 2210, it is judged whether the eye opening value is normal or not.
The process moves to step 2301 of 3. Since the processing contents of the shift to step 2301 have the same processing contents as those in the above-described embodiment, the description thereof will be omitted here.

Next, due to the causes shown in the example of FIG.
The contents of processing when an eye opening degree abnormality occurs will be described.

In this state, it is recognized that the eye opening value is abnormal in step 2210 of FIG. 22, and the process proceeds to step 2211. In step 2211, the left and right regions where the left and right symmetrical continuous data are to be detected are set. The setting method and the size of the area have been described in the processing content of step 2203.

Thereafter, the process returns to step 2201, the image is input with the symmetrical data as the detection target, and the step 2
The process moves from step 203 to step 2401 in FIG. In step 2401, continuous data is detected in the left / right symmetrical data detection area as shown in FIG.
Move to 02. In step 2402, it is determined whether or not the waiting time for determining the appearance of continuous data has been exceeded.
When entering the flow for the first time, of course, the waiting time is not set, and thus the process proceeds to step 2403.

When the state in which it is determined that the eye opening value is abnormal in step 2210 of FIG. 22 is a temporary phenomenon, the detection of continuous data in the left and right regions for detecting left-right symmetric data is detected. At the stage of performing, the continuous data of both eyes can be stably detected as shown in FIGS. Therefore,
Two or more stable appearances can be confirmed in step 2403, and the process proceeds to step 2408.

In the determination in step 2403, the reason why there are two or more continuous data points that appear stably is that the continuous data of the eyes is changed due to the shift of the detection area of the symmetrical data or the movement of the driver's face. This is because continuous eyebrow data may also be included. In step 2408, when the vehicle is a right-hand drive vehicle, the eye tracking area is set again by setting the eye tracking area based on the continuous data of the left eye of the driver having relatively good light environment conditions. Then, step 240
By clearing the left and right regions where the left-right symmetric data is detected in 9, the flow returns to the open / closed eye determination and the awakening degree determination.

Next, description will be made on the case where the light environment is bad or the driver looks sideways for a relatively long time (for example, 1 to 2 minutes).

Due to the above reasons, step 2210 of FIG.
If the eye opening value is abnormal in step S2403, it is not possible to stably confirm the appearance of two or more continuous data in the detection area of the symmetrical data in step S2403 of FIG. . In step 2404 as well, as in step 2402, since the appearance determination waiting time of continuous data is not set even when the process first enters this flow, the process proceeds to step 2405. In step 2405, the number of consecutive data occurrences is 0 to 2.
We are checking to see if it is constantly changing within the range of individual items.
27 to 32 in the case where the light environment deteriorates as a result of this determination.
Will be explained. Further, a case where the driver's face direction is not fixed and is sloppy and the time other than the front of the vehicle is long is described with reference to FIGS. 33 and 34.

The deterioration of the light environment when an automobile is assumed is
When the vehicle is running, it is constantly changing, as shown in FIG.
As shown in Fig. 29, the driver's left eye part becomes a little shadow but can be recognized as continuous data, or as shown in Fig. 29, the driver's left eye part becomes dark and becomes continuous data. The case where it cannot be recognized, or the case where the direct light to the right eye becomes stronger and the right eye cannot be recognized as continuous data as shown in FIG. 31, is repeated. Therefore, in the appearance state of continuous data, the number of appearances of continuous data constantly changes as shown in FIGS. 28, 30, and 32. In addition, the upper limit of the number of appearances of continuous data is when the continuous data of eyebrows is included,
Assuming a driver wearing glasses, there may be two or more.

Next, description will be made regarding a situation in which the driver is looking at a place other than the front for a long time. When the driver is facing sideways as shown in FIG. 33, only one piece of continuous data appears in the left-right symmetrical continuous data detection area as shown in FIG. So the driver looks at the right,
The number of appearances of continuous data constantly changes even when looking to the left or looking around. Re-detecting the eyes in such a situation may cause the eyes to be erroneously detected, so the process proceeds to step 2410 and the determination is extended by a predetermined time by setting the continuous data appearance determination waiting time t1. Allows re-detection of the eyes waiting for improvement of the light environment or waiting for the driver's face position to settle down.

If the stable appearance of two or more continuous data is not recognized in step 2403 of FIG. 24, and that the appearance position of continuous data is not constantly changed in step 2405, the process proceeds to step 2406. It is confirmed whether continuous data does not appear at all even after the process is performed a predetermined number of times.

As a scene corresponding to step 2406, there is a case where the movement of the driver's face is so large that the driver's face is no longer visible in the camera. For example,
For example, when the driver is looking for something in the glove box. If it is considered that the situation is under such a condition, the process proceeds to step 2411, the continuous data appearance determination waiting time t2 is set, and the determination is extended by a predetermined time. This waiting time t2 is set with reference to the time required for changing the driver. In short, if the appearance of continuous data disappears at all, in addition to the case of a large movement of the face, the driver may change, and if it seems that the driver has changed, the eye position detection from the initial state should be performed. Because it is necessary. Step two
As for the waiting time t1 at 410, it is considered that a bad light environment or a driver's rolling condition may last for a relatively long time.
It seems preferable to set it in 2 minutes or more.

Step 2401 or Step 2411
After setting the appearance waiting time of continuous data in step 24,
Based on the determination at 04, it is determined whether or not there is a stable appearance of two or more continuous data by passing through step 2403 within the waiting time without the waiting time setting being performed again. If there is no stable occurrence of two or more continuous data within this waiting time, the steps 2402 to 2
In step 407, the eye tracking basin, the detection area of the symmetrical data, and the determination waiting time are cleared, and the eye position detection is returned from the initial state.

Next, another embodiment will be described with reference to FIG.
5 and the flowchart of FIG. 36.

In the above-described embodiment, the reference position of the eye is learned by outputting the eye opening value multiple times and detecting the eye position. However, in the former method, when the output value of the eye opening itself contains noise, the continuous data that is not an eye is mistakenly recognized as an eye to set the reference position, and the driver who has extremely few blinks In this case, even if the continuous eye data is correctly captured, it may be difficult to learn the position as the reference position of the eye. Further, in the latter method, in order to learn the reference position of the eye, it is necessary to repeat the detection of the eye position from the initial state, so there is a problem that it takes time to learn the reference position of the eye first. is there.

Therefore, in the present embodiment, the left and right regions in which the left-right symmetric data is detected based on the eye position detected in the processing when the abnormality is recognized by the eye tracking and the determination of the opening value state. Is set, and the eye position is re-detected from the appearance state of the continuous data appearing in the area. The reference position of the eye is learned based on the eye position data detected at this time.

According to this method, as the information for determining whether or not the re-detected continuous data is an eye, in addition to the value corresponding to the opening value, the symmetrical condition of the continuous data, the eye is opened. Since it is possible to use a feature having a convex shape on the continuous data that can be recognized in this case, it is possible to improve the learning accuracy of the reference position of the eye. Further, since the area where the re-detection of the eyes is performed is limited to the detection area of the symmetrical data, the learning time can be shortened.

Details of the processing contents will be described below with reference to the flowcharts of FIGS. Note that the processing content in each step to be described below will be performed focusing on the points different from the above-described embodiment.

In step 3508 of FIG. 35, eye tracking and an abnormal state on both sides of the opening value are recognized. The determination of the eye tracking condition until the reference position of the eye is learned is performed based on the position where the eye position is detected. Step 3508
If no abnormality is confirmed in step 3512, the flow shifts to step 3512, and the flow of repeating the determination of the awakening degree from the output pattern of closed eyes is repeated as in the above-described embodiment.

If it is determined in step 3508 that the eye tracking and the degree of opening value are abnormal, the process proceeds to step 3509. In step 3509, it is determined whether or not the eye reference position has been learned. Immediately after the start of the process, the reference position of the eye has not been learned, so the step 3
Move to 510. In step 3510, the left and right regions for detecting the left-right symmetrical data are set. The area for which the left-right symmetric data is detected is based on the eye position detected in step 3505. Then, step 3510
Returning to, the flow enters into the determination process of the appearance state of continuous data that appears in the left and right regions where the left and right symmetrical data is detected, as in the above-described embodiment.

In step 3603 of FIG. 36, it is determined whether or not two or more continuous data are stably appearing in the detection area of the left-right symmetric data, and if the appearance is confirmed, the process proceeds to step 3608. The position of the eye is re-detected on the basis of the continuous data that has been transferred and appeared, and step 36
At 09, the reference position of the eye is learned. Subsequent step 3
At 610, it is determined whether learning of the eye reference position is OK. This OK determination also includes conditions such as whether there is little variation in the re-detected eye position, or whether the opening value for the continuous data has changed by a size corresponding to the eye.

If the above conditions are satisfied in step 3610, the right and left regions where the left-right symmetric data are detected are cleared in step 3611, and the flow returns to the awakening level determination.

In the flow of the awakening degree determination after the return, when the eye tracking and the opening value abnormality are confirmed again in step 3508 of FIG. 35, the reference position of the eye is learned in step 3509. It is determined that step 3515
Move to. In step 3515, it is determined whether or not the condition of the eye tracking and the opening degree abnormality is steady. If it is determined in step 3515 that the abnormality is not a steady abnormality, the process proceeds to step 3511, and the tracking area is returned to the reference position of the eye to enable backup in a short time as in the above-described embodiment.

If it is determined in step 3515 that the abnormality is steady, the process proceeds to step 3516 to clear the eye tracking area and the eye reference position so that the eye position can be detected in the initial state. Then, the process returns to step 3501.
This processing is prepared on the assumption that the driver is replaced while the flow of the awakening degree determination is running.

In step 3603 of FIG. 36, if two or more consecutive data are not stably confirmed to appear in the detection area of the left-right symmetric data, the process proceeds to step 3604 and the same as the above-mentioned embodiment. A waiting time for determining the appearance of continuous data is set, and confirmation of the appearance of two or more continuous data that appears stably is continued within the waiting time range.

If the waiting time is exceeded in step 3602, the process moves to step 3607 to clear the eye tracking area and the left and right areas for detecting left-right symmetry data, and the judgment waiting time to detect the eye position in the initial state. Then, the process returns to step 3501.

It should be noted that the flow of this flow means that the operation is performed when the position of the eye initially detected in step 3505 of FIG. 35 is incorrect and when the reference position of the eye is learned in the left and right regions where the symmetrical data is detected. There is a case where the person is replaced.

As is clear from the above description, the present invention recognizes abnormality processing by monitoring the eye tracking state and the opening value state, and when there is abnormality, the eye reference position learned in advance is used. By returning the eye tracking area, highly accurate re-detection of the eye can be performed in the minimum necessary time even if there is an eye tracking error due to a fast movement of the face or a temporary deterioration of the light environment. Therefore, since the processing time ratio of the open / closed eye determination in the entire processing time is increased, the accuracy of the drowsiness detection can be improved.

[Brief description of drawings]

FIG. 1 is a functional block diagram of an embodiment of the present invention.

FIG. 2 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 3 is a flowchart showing the overall operation of one embodiment.

FIG. 4 is a flowchart showing the overall operation of one embodiment.

FIG. 5 is a flowchart showing an operation of eye position detection.

FIG. 6 is an explanatory diagram related to eye position detection.

FIG. 7 is an explanatory diagram related to eye position detection.

FIG. 8 is an explanatory diagram related to eye position detection.

FIG. 9 is an explanatory diagram related to eye position detection.

FIG. 10 is a chart for explaining eye position detection.

FIG. 11 is an explanatory diagram related to eye position detection.

FIG. 12 is a flowchart showing a method of calculating a binarization threshold value for obtaining an eye opening value.

FIG. 13 is an explanatory diagram related to a method of obtaining a binarization threshold value.

FIG. 14 is an explanatory diagram related to a method of obtaining a binarization threshold value.

FIG. 15 is an explanatory diagram related to a method of outputting an eye opening value.

FIG. 16 is an explanatory diagram regarding an eye tracking method.

FIG. 17 is an explanatory diagram regarding a method of returning the eye tracking region.

FIG. 18 is an explanatory diagram related to a method of returning the eye tracking region.

FIG. 19 is an explanatory diagram related to a method of returning an eye tracking region.

FIG. 20 is a diagram illustrating a problem of one embodiment.

FIG. 21 is a functional block diagram showing a configuration of another embodiment.

FIG. 22 is a flowchart showing the overall operation of another embodiment.

FIG. 23 is a flowchart showing the overall operation of another embodiment.

FIG. 24 is a flowchart showing the overall operation of another embodiment.

FIG. 25 is an explanatory diagram regarding a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 26 is an explanatory diagram regarding a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 27 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 28 is an explanatory diagram related to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 29 is an explanatory diagram related to a method of determining the appearance state of continuous data within a detection area of left-right symmetric data.

FIG. 30 is an explanatory diagram regarding a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 31 is an explanatory diagram regarding a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 32 is an explanatory diagram related to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 33 is an explanatory diagram related to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 34 is an explanatory diagram relating to a method of determining the appearance state of continuous data in a detection region of left-right symmetric data.

FIG. 35 is a flowchart showing the overall operation of still another embodiment.

FIG. 36 is a flowchart showing the overall operation of the same embodiment.

[Explanation of symbols]

CL1 image input means CL2 Face-direction vertical pixel row density detection means CL3 point extraction means CL4 Continuous data extraction means CL5 Eye position detection means CL6 Point extraction means within the tracking area of the eye CL7 Eye opening detection means CL8 Eye reference position determination means CL9 Eye tracking state determination means CL10 Eye tracking area resetting means CL11 Awakening level determination means CL12 alarm means 21 TV camera 22 A-D converter 23 Image memory 24 Image data operation circuit 25 eye position detection circuit 26 Open / closed eye detection circuit 27 Backup circuit 28 Awakening level judgment circuit 29 Alarm device

─────────────────────────────────────────────────── ─── Continuation of the front page (56) Reference JP-A-10-143669 (JP, A) JP-A-10-63850 (JP, A) JP-A-10-236181 (JP, A) JP-A-9- 270010 (JP, A) JP-A-7-208927 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) A61B 5/18 B60K 28/06 G06T 1/00

Claims (15)

(57) [Claims] 1. A face image input unit for inputting a face image, and a pixel is defined and an extraction point is determined for each local increase in density in a vertical pixel array of face image data input by the face image input unit. And an eye position specifying means for specifying the eye position based on continuous data in which the extraction points of the pixel rows adjacent to the extraction point are adjacent and extend in the lateral direction of the face, and a predetermined position including the specified eye position. Predetermined region setting means for setting a region, and the eye position is specified based on the continuous data within the predetermined region set by the predetermined region setting means, and the eye opening is detected from the density change state of the continuous data. In an eye condition detection device including an eye opening detection means, a first eye reference area determination means that determines and stores a reference area of an eye that can be predicted to exist based on the continuous data, Eye continuous data within the predetermined area A tracking condition of the eye and a tracking condition determining means for determining whether the tracking condition is normal based on the condition of the continuous data, and an eye tracking condition determining means for determining an eye tracking abnormality, An eye condition detection device, comprising: a first returning unit that returns the predetermined region to the reference region. 2. A face image input means for inputting a face image, and a pixel is defined and an extraction point is determined for each local increase in density in a vertical pixel array of the face image data input by the face image input means. And an eye position specifying means for specifying the eye position based on continuous data in which the extraction points of the pixel rows adjacent to the extraction point are adjacent and extend in the lateral direction of the face, and a predetermined position including the specified eye position. Predetermined region setting means for setting a region, and the eye position is specified based on the continuous data within the predetermined region set by the predetermined region setting means, and the eye opening is detected from the density change state of the continuous data. In an eye condition detection device including an eye opening detection means, a first eye reference area determination means that determines and stores a reference area of an eye that can be predicted to exist based on the continuous data, Eye continuous data within the predetermined area A tracking condition of the eye and a tracking condition determining means for determining whether the tracking condition is normal based on the condition of the continuous data, and an eye tracking condition determining means for determining an eye tracking abnormality, A left / right area setting unit that sets left and right areas that can detect continuous data of the left and right eyes based on the reference area, and a continuous data appearance state determination unit that determines whether continuous data appears in the left and right areas And re-detection means for causing the eye opening detection means to perform the detection again based on the continuous data when it is determined that continuous data appears in the left and right regions. Eye condition detection device. 3. A face image input means for inputting a face image, and a pixel is defined and an extraction point is determined for each local increase in density in a vertical pixel row of the face image data input by the face image input means. And an eye position specifying means for specifying the eye position based on continuous data in which the extraction points of the pixel rows adjacent to the extraction point are adjacent and extend in the lateral direction of the face, and a predetermined position including the specified eye position. Predetermined region setting means for setting a region, and the eye position is specified based on the continuous data within the predetermined region set by the predetermined region setting means, and the eye opening is detected from the density change state of the continuous data. In an eye state detection device having an eye opening detection means, tracking eye continuous data in the predetermined region, the eye to determine whether the tracking state is normal based on the state of the continuous data And a tracking condition of the eye. When it is determined that the eye tracking abnormality by the state determination unit, the left and right region setting unit that sets the left and right regions that can detect the continuous data of the left and right eyes based on the reference region, and the continuous data appears in the left and right regions. Appearance determination means for continuous data to determine whether or not, and when it is determined that the continuous data has appeared, re-determining the eye position specifying means to perform the detection again based on the continuous data. An identifying unit, a second eye reference region determining unit that determines and stores an eye reference region that can be predicted to exist based on the eye position information identified by the re-identifying unit, and the eye tracking A second method of returning the predetermined region to the reference region when the state determination unit determines that the eye tracking is abnormal.
And a return means for the eye. 4. The eye state detection device according to claim 1, wherein the first or second reference area determination unit repeats eye position detection processing to determine an eye reference area. An eye condition detection device characterized by learning. 5. The eye state detecting device according to claim 1, 2, or 3, wherein the first or second reference region determining means determines a time-series change state of an eye opening value. An eye condition detecting device characterized by learning a reference region of the eye. 6. The eye state detection device according to claim 1, wherein the first or second reference region determination means is a stable eye by a plurality of eye position detection processes. An eye condition detection device, which determines a reference region of the eye when the position of is output. 7. The eye condition detection device according to claim 2, wherein the re-detection unit has two consecutive data that can be stably detected laterally at predetermined intervals in the left and right regions. An eye condition detecting device characterized by performing re-detection. 8. The eye condition detecting apparatus according to claim 3, wherein the re-identifying unit has two consecutive data that can be stably detected at predetermined intervals in the lateral direction in the left and right regions. An eye condition detecting device characterized by performing re-detection. 9. The eye condition detection device according to claim 2, 3, 7, or 8, wherein the determination extension means extends the determination for a predetermined time when continuous data in the left and right regions cannot be stably detected. An eye condition detecting device comprising: 10. The eye state detection device according to claim 2, further comprising a determination extension unit that extends the determination for a predetermined time when continuous data in the left and right regions cannot be detected. An eye condition detection device characterized by the above. 11. The eye condition detecting device according to claim 2, 3, 7, or 8, wherein continuous data in the left and right regions cannot be stably detected, and continuous data in the left and right regions are detected. When it is not possible, a judgment extension means for extending the judgment for a predetermined time is provided, and the judgment extension means sets the extension time when the continuous data cannot be detected longer than when it cannot be detected stably. Detection device. 12. The eye state detection device according to claim 1, wherein the tracking state determination means tracks the eye when the continuous data in the predetermined area moves and stays at that position for a predetermined time. An eye condition detection device characterized by determining abnormality. 13. The eye state detection device according to claim 1, wherein the tracking state determination means determines that eye tracking is abnormal when continuous data in the predetermined region cannot be detected for a predetermined time. A characteristic eye condition detection device. 14. The eye state detection device according to claim 1, wherein the tracking state determination means tracks the eye when the eye opening value exceeds a predetermined range and is constantly detected. An eye condition detection device characterized by determining abnormality. 15. The eye state detection device according to claim 1, wherein the eye opening state output by the eye opening detecting means is determined, and the awakening degree is determined from the change in the open / closed state. A drowsy driving warning device, comprising: an awakening level determining means for determining; and an alarming means for issuing an alarm when the awakening level determining means determines that the awakening level is low.