JP4451195B2 - Gaze detection device - Google Patents

Description

  The present invention relates to a line-of-sight detection device.

Conventionally, for example, a method for detecting a line of sight using a camera that can shoot in the visible light region or a camera that can shoot in the infrared region is known (see, for example, Non-Patent Document 1). In the method using a camera that can be photographed, eye feature points (eye corners, eyelids, etc.) and iris are detected from an eyeball image photographed by the camera, and the line of sight is detected based on the relative position of the iris with respect to the feature point. Yes. However, this method has a problem that it is difficult to improve the detection accuracy because the change in the position of the iris with respect to the change in the line-of-sight direction is relatively small.
To solve this problem, the method using a camera capable of photographing in the infrared region irradiates the eyeball with infrared rays from an infrared point light source, and the reflected image (Purkinje image) at the pupil center and corneal surface detected from the photographed image. ) Is detected based on the relative position of (). With this method, it is possible to detect the line of sight with relatively high accuracy.
Takeo Ohno, “Interface Using Eyes,” Information Processing, Information Processing Society of Japan, July 2003, Vol. 44, No. 7, p. 726-732

By the way, in the above-described conventional technique, in the method using the camera capable of photographing in the infrared region, if the driver's face direction is relatively greatly deviated from the irradiation direction of the infrared point light source, the infrared ray is appropriately applied to the corneal surface. In some cases, the reflected image of the corneal surface cannot be clearly captured without being irradiated to a proper position. In this case, there is a possibility that the detection accuracy of the gaze detection is lowered or the gaze detection cannot be performed.
The present invention has been made in view of the above circumstances, and an object of the present invention is to provide a gaze detection device capable of detecting a gaze regardless of the orientation of the driver's face.

In order to solve the above-described problems and achieve the object, the line-of-sight detection device of the present invention according to claim 1 includes a plurality of infrared irradiation means for irradiating the eyeball of the driver of the own vehicle with infrared rays from different irradiation directions. (For example, the infrared light emitting diodes (infrared LEDs) 23 ₁ ,..., 23 _n in the embodiment) and the respective irradiation states of the plurality of infrared irradiation means are controlled, and the irradiation control is performed by switching the irradiation direction of the infrared rays. Means (light emission control unit 32 in the embodiment), infrared photographing means (for example, infrared camera 22 in the embodiment) for photographing the eyeball of the driver of the own vehicle in the infrared region, and the driver of the own vehicle Face direction detecting means for detecting the face direction of the vehicle, and line-of-sight detecting means for detecting the driver's line of sight based on an infrared image obtained by photographing by the infrared photographing means (for example, the infrared line-of-sight calculation unit in the embodiment) 34) and switching direction setting means for setting the irradiation direction of the infrared rays in accordance with the driver's face direction detected by the face direction detection means .

According to the visual line detection device of the above configuration, for example, even when the direction of the driver's face or the eyeball changes, to infrared radiation emitted by any of the infrared irradiation means of the plurality of infrared ray irradiation means A reflected image (Purkinje image) on the corneal surface can be detected, and the occurrence of a state in which the line of sight cannot be detected can be suppressed. And since the irradiation direction of infrared rays is changed according to a driver | operator's face direction , an accurate eyes | visual_axis detection can be performed rapidly.

Furthermore, in the line-of-sight detection device of the present invention according to claim 2 , the irradiation control means controls each operating state of the plurality of infrared irradiation means, and performs irradiation by sequentially switching the irradiation direction of infrared rays. Yes.
Further, in the line-of-sight detection device according to the third aspect of the present invention, the switching direction setting means sets the switching order of the irradiation directions.

  According to the line-of-sight detection device having the above configuration, the driver's line of sight can be accurately detected based on an infrared image in which a reflected image (Purkinje image) on the cornea surface is most clearly detected by sequentially switching the irradiation direction of infrared rays. can do.

According to the visual line detection device of the present invention, for example, even when the direction of the driver's face or the eyeball changes, it is possible to suppress the state where the visual axis detection becomes impossible occurring. And since the irradiation direction of infrared rays is changed according to a driver | operator's face direction , an accurate eyes | visual_axis detection can be performed rapidly.

Hereinafter, a gaze detection apparatus according to an embodiment of the present invention will be described with reference to the accompanying drawings.
The line-of-sight detection device 10 according to the present embodiment is provided in, for example, a vehicle travel safety device, and is configured by an electronic circuit including a CPU and the like to detect the line of sight of the driver of the host vehicle as shown in FIG. A detection control device 11 and a line-of-sight sensor 12 are provided.

For example, as shown in FIG. 2, the line-of-sight sensor 12 is provided on an instrument panel, an upper part of a dashboard, or the like in a vehicle interior, and a visible light camera 21 capable of photographing in a visible light region such as a CCD camera or a C-MOS camera. And an infrared camera 22 capable of photographing in the infrared region, and a plurality of infrared light emitting diodes (infrared LEDs) 23 ₁ ,..., 23 _n (n is an arbitrary natural number) that irradiates infrared rays from different directions toward the driver's eyeball. In this case, for example, n = 3), a visible light image processing unit 24, and an infrared image processing unit 25 are provided.
The visible light camera 21 captures visible light reflected from the driver's face and eyeball.
Infrared camera 22, the infrared LED 23 1 by the control of the detection control device 11, _..., to shoot an infrared reflection of irradiated toward the eye of the driver from 23 _n.
The image processing units 24 and 25 perform predetermined image processing such as filtering and binarization processing on the visible light image and the infrared image obtained by capturing with the cameras 21 and 22, respectively, Image data composed of pixels is generated and output to the detection control device 11.

Detection control device 11, the face direction detecting section 31 where the image data of the visible light image is input from the line-of-sight sensor 12, the infrared LED 23 1, _..., a light emission control unit 32 for controlling the light emission state of the 23 _n, line-of-sight sensor An infrared reflection determination unit 33 and an infrared line-of-sight calculation unit 34 to which image data of an infrared image is input from 12 are configured.

The face orientation detection unit 31 recognizes the image data of the visible light image input from the line-of-sight sensor 12 by performing recognition processing such as feature amount calculation and shape determination using the driver's face as a detection target, for example. Based on the detection target, the driver's face orientation is calculated and output to the light emission control unit 32. In the recognition processing by the face orientation detection unit 31, for example, as shown in FIG. 3, the driver's eye corner positions E1, E1 and eye corner positions E2, E2 are detected in the visible light image S, and further, the eye corners E1, E1. The distance L1 between the eye corners and the distance L4 between the eyes E2 and E2 are detected. Then, the driver's face orientation is calculated based on the detected distance between the corners L1 and the distance L4 between the eyes. That is, the detected distance L1 between the corners of the eyes and the distance L4 between the eyes changes in accordance with the direction P of the driver's face. For example, the angle formed by the direction P of the driver's face and the shooting direction of the visible light camera 21 As the distance increases, the distance L1 between the corners of the eyes changes in a decreasing trend. Therefore, the angle formed by the direction P of the driver's face and the shooting direction of the visible light camera 21 is calculated based on the distance L1 between the corners of the eyes. be able to. Further, based on the relative magnitude relationship between the distance L2 between the outer corner E1 and the eye E2 with respect to the right eye of the driver and the distance L3 between the outer eye E1 and the eye E2 with respect to the left eye, It can be determined which direction P is the left or right direction.
In the image data recognition processing, in the feature amount calculation processing, for example, extraction and labeling of detection objects based on pixel continuity is performed on the image data after binarization processing, and the extracted detection objects The center of gravity and area of the image, the aspect ratio of the circumscribed rectangle, and the like are calculated. In the shape determination process, for example, a search on image data is performed based on a predetermined pattern (for example, an outline) stored in advance, and a detection target is extracted according to the similarity to the predetermined pattern.

The light emission control unit 32 has a plurality of infrared light emitting diodes (infrared LEDs) 23 ₁ ,..., 23 according to the driver's face direction detected by the face direction detection unit 31 and the determination result by the infrared reflection determination unit 33. The light emission state of _n is controlled, and irradiation is performed by sequentially switching the irradiation direction of infrared rays. At this time, the order in which the infrared light emitting diodes (infrared LEDs) 23 ₁ ,..., 23 _n emit light is such that a reflection image (Purkinje image) on the cornea surface of the driver's eyeball is clearly detected according to the driver's face orientation. The higher the probability that the driver is wearing spectacles, the higher the ranking, the lower the degree to which the infrared rays irradiated toward the driver's eyeballs are reflected by the spectacles. Set to.

The infrared reflection determination unit 33 recognizes, for example, feature amount calculation and shape determination using, as an object to be detected, an area having brightness and size greater than a predetermined value for image data of an infrared image input from the line-of-sight sensor 12. Processing is performed to determine whether or not the infrared rays emitted from the respective infrared LEDs 23 ₁ ,..., 23 _n toward the driver's eyeballs are reflected by the driver's glasses, and this determination result is used as the infrared line-of-sight calculation unit 34. Output to. In the recognition processing by the infrared reflection determination unit 33, for example, as shown in FIG. 4A, an area having luminance and size greater than or equal to a predetermined value with respect to image data of an infrared image obtained by photographing a driver's eyeball. By extracting, for example, infrared reflected images Q1, Q2, and Q3 as shown in FIG. 4B are detected.
When the infrared line-of-sight calculation unit 34 determines that the infrared rays irradiated from the respective infrared LEDs 23 ₁ ,..., 23 _n toward the driver's eyeball are not reflected by the driver's glasses. In addition, the image data of the infrared image input from the line-of-sight sensor 12 is subjected to recognition processing such as feature amount calculation and shape determination using the driver's eyeball as a detection target, and based on the recognized detection target, A driver's line-of-sight vector (line-of-sight direction) is calculated, and a target position of the line of sight is calculated based on the line-of-sight vector. In the recognition processing by the infrared line-of-sight calculation unit 34, for example, the line-of-sight vector is calculated based on the relative distance between the center position of the reflected image (Purkinje image) on the cornea surface and the center position of the pupil.

  The line-of-sight detection apparatus 10 according to the present embodiment has the above-described configuration, and the operation of the line-of-sight detection apparatus 10 will be described next.

First, for example, in step S01 shown in FIG. 5 , a visible light image is acquired.
Next, in step S02, recognition processing such as feature amount calculation and shape determination using, for example, a face as a detection target is performed on the acquired visible light image image data to detect the face orientation.
Then, in step S03, the infrared LED 23 ₁ in accordance with the detected face direction, ..., sets the light emission order of 23 _n.
In step S04, the infrared LEDs 23 ₁ ,..., 23 _n are sequentially caused to emit light according to the set light emission order.
In step S05, an infrared image is acquired.
In step S06, for example, recognition processing such as feature amount calculation and shape determination using a region having a luminance and size greater than or equal to a predetermined value as a detection target is performed, and infrared rays emitted toward the driver's eyeball are emitted to the driver. It is determined whether it is reflected by the glasses.
When the determination result is "YES", the operation proceeds to step S07, the infrared LED 23 _1, ..., change the emission order of 23 _n, the process proceeds to step S04 described above.
On the other hand, if this determination is “NO”, the flow proceeds to step S08.
In step S08, recognition processing such as feature amount calculation and shape determination using the eyeball of the driver as a detection target is performed on the image data of the infrared image input from the line-of-sight sensor 12, and the detected detection target is recognized. Then, the driver's line-of-sight vector (line-of-sight direction) is calculated, the target position of the line-of-sight is calculated based on the line-of-sight vector, and the series of processing ends.

As described above, according to the line-of-sight detection device 10 according to the present embodiment, for example, even when the driver wears spectacles or when the driver's face or eyeball orientation changes, a plurality of infrared rays LED 23 1, _..., a reflection image at the corneal surface to infrared radiation emitted by any of the 23 _n can be detected (Purkinje image) suppresses that a state in which visual axis detection becomes impossible to generate be able to.
Moreover, since the direction of infrared irradiation is changed according to the driver's face direction detected by the face direction detection unit 31 and the determination result of the infrared reflection determination unit 33, accurate line-of-sight detection is quickly performed. be able to.

In the above-described embodiment, the infrared irradiation direction is changed in accordance with the driver's face direction detected by the face direction detection unit 31. However, the present invention is not limited to this. Omission may be performed simply by sequentially switching the irradiation direction of infrared rays according to a predetermined light emission order. In this case, the infrared reflection judgment unit 33, a plurality of infrared LED 23 _1, ..., according to the irradiation by each 23 _n, for example, as shown in FIG. 6 (a) ~ (c) , the image data of a plurality of infrared images The image data is acquired, image data that is highly likely to be clearly detected on the corneal surface of the driver's eyeball (Purkinje image) is selected and output to the infrared line-of-sight calculation unit 34. For example, in FIGS. 6 (a) to 6 (c), reflected images R1, R2, and R3 in which the infrared rays irradiated toward the driver's eyeball are reflected by the glasses are detected. For the image data, image data corresponding to FIG. 6B in which an infrared reflected image is not detected is selected.

It is a lineblock diagram of a gaze detection device concerning one embodiment of the present invention. It is a perspective view which shows the vehicle carrying the gaze sensor shown in FIG. It is a figure which shows an example of the relative position of a driver | operator's face direction and a visible light camera. It is a figure which shows an example of the infrared reflected image reflected with a driver | operator's spectacles. It is a flowchart which shows operation | movement of the gaze detection apparatus shown in FIG. It is a figure which shows an example of the infrared rays reflected with a driver | operator's spectacles.

Explanation of symbols

10 Gaze detection device 22 Infrared camera (infrared imaging means)
23 ₁ ,..., 23 _n Infrared light emitting diode (infrared irradiation means)
32 Light emission control unit (irradiation control means)
34 Infrared line-of-sight calculation unit (line-of-sight detection means)

Claims (3)

A plurality of infrared irradiation means for irradiating the eyeball of the driver of the own vehicle with infrared rays from different irradiation directions;
Irradiation control means for controlling each operating state of the plurality of infrared irradiation means, and switching and irradiating the infrared irradiation direction;
An infrared photographing means for photographing an eyeball of a driver of the own vehicle in an infrared region;
Face orientation detection means for detecting the face orientation of the driver of the host vehicle;
Line-of-sight detection means for detecting the driver's line of sight based on an infrared image obtained by photographing by the infrared photographing means;
A line-of-sight detection device comprising: a switching direction setting unit that sets the infrared irradiation direction in accordance with a driver's face direction detected by the face direction detection unit. The irradiation control means, line-of-sight detecting device according to claim 1, wherein controlling the respective operating states of the plurality of infrared ray irradiation means, and irradiating sequentially switches the irradiation direction of the infrared rays. The line- of- sight detection device according to claim 2, wherein the switching direction setting means sets the switching order of the irradiation directions .

Images