JP7638541B2 - Pupil detection device - Google Patents

Description

The embodiment relates to a pupil detection device that detects pupils from an image of a person.

In recent years, devices that detect the position of a person's pupil from images obtained using a light source such as a near-infrared light source and a video camera for purposes such as gaze detection have become popular (see Patent Document 1 below). In this device, a light that tends to make the pupil relatively bright is irradiated onto the subject's face to obtain an image (bright pupil image), and a light that tends to make the pupil relatively dark is irradiated onto the subject's face to obtain an image (dark pupil image). These images are then used to calculate a difference image to detect the subject's pupil.

In such methods of pupil detection based on differential images, there is generally a time lag between the acquisition of the bright pupil image and the dark pupil image, so if the subject's head moves between the acquisition of the two images, the pupil also moves, causing a positional shift between the pupil image in the bright pupil image and the pupil image in the dark pupil image, limiting the accuracy of pupil detection. To solve this problem, a method is used in which a corneal reflection image is detected from the bright pupil image and the dark pupil image, and a differential image is generated after position correction based on the position of the corneal reflection image (see Patent Document 2 below). A method of generating a differential image using the position of the nostrils is also used (see Patent Document 3 below).

JP 2005-185431 A JP 2008-29702 A JP 2007-268026 A

In the conventional methods described above, when the subject's eyeball rotates rapidly, the shape of the pupil itself changes as seen in the image. Even if the bright pupil image and the dark pupil image are aligned, the pupil images do not match between the two images, and the accuracy of pupil detection when using a difference image is insufficient. It is possible to use a high-speed camera to minimize the time difference between the bright pupil image and the dark pupil image, but introducing a high-speed camera can be difficult in terms of cost.

This embodiment has been made in consideration of the above-mentioned problems, and aims to provide a pupil detection device that can easily improve the accuracy of pupil detection regardless of the subject's condition.

In order to solve the above problem, a pupil detection device according to one embodiment of the present disclosure includes a camera that captures an image of a subject's eye by capturing an image of the subject's eye, a first light source that is provided on the outside or inside of the opening of the camera and illuminates the subject's pupil relative to the camera with light of a first central wavelength, a second light source that is provided on the outside or inside of the opening of the camera and illuminates the subject's pupil relative to the camera with a second central wavelength different from the first central wavelength, and a computing device that processes the eye image, and the camera includes an image sensor in which a polarizer that transmits linearly polarized light of at least two different angles is attached to each of adjacent pixels, and a first dividing element that combines a bandpass filter that transmits light of the first central wavelength and a first polarizer that transmits linearly polarized light of a predetermined angle. and a second dividing element combining a bandpass filter that passes light of the second central wavelength and a second polarizer that transmits linearly polarized light at an angle different from the predetermined angle, and an optical element divided and provided along the opening between the opening and the image sensor, wherein the arithmetic device calculates a luminance corresponding to the light of the first central wavelength based on the luminance of adjacent pixels in the eye image, combines the calculated luminances to obtain a bright pupil image in which the subject's pupil is relatively bright, calculates a luminance corresponding to the light of the second central wavelength based on the luminance of adjacent pixels in the eye image, combines the calculated luminances to obtain a dark pupil image in which the subject's pupil is relatively dark, and calculates the position of the subject's pupil image based on a comparison image obtained by comparing the bright pupil image with the dark pupil image.

Note that the "camera opening" here refers to the part that captures the light of an image from outside the camera to the image sensor inside the camera, and is not necessarily limited to the circular lens part of the camera barrel, but refers to the opening when the lens part of the camera is covered with a cover member having an opening narrower than the lens part. In addition, the shape of the opening is not limited to a circle, and may be various shapes such as a polygon such as a rectangle, an ellipse, etc.

According to the above-mentioned embodiment of the pupil detection device, the light from the pupil of the subject illuminated by the light of the first central wavelength from the first light source is converted into linearly polarized light at a predetermined angle by passing through the first dividing element of the optical element, and the brightness is detected by adjacent pixels of the image sensor to which two types of polarizers are attached. At the same time, the light from the pupil of the subject illuminated by the light of the second central wavelength from the second light source is converted into linearly polarized light at an angle different from the predetermined angle by passing through the second dividing element of the optical element, and the brightness is detected by adjacent pixels of the image sensor to which two types of polarizers are attached. Here, the image of the pupil illuminated by the light of the first central wavelength is relatively brighter than the image of the pupil illuminated by the light of the second central wavelength. The computing device then calculates a luminance corresponding to the light of the first central wavelength from the luminance of adjacent pixels detected by the image sensor and combines the luminances to obtain a bright pupil image, calculates a luminance corresponding to the light of the second central wavelength from the luminance of adjacent pixels detected by the image sensor and combines the luminances to obtain a dark pupil image, and calculates the position of the pupil image based on the bright pupil image and the dark pupil image. This makes it possible to easily improve the detection accuracy of the pupil image regardless of the state of the subject (for example, even if the subject's eyeball rotates), because there is no difference in the timing at which the bright pupil image and the dark pupil image are obtained.

According to the embodiment, the accuracy of pupil detection can be easily improved regardless of the subject's condition.

1 is a diagram showing a schematic configuration of a pupil detection device according to a first embodiment. FIG. 2 is a plan view of the lighting device of FIG. 1 as viewed from outside the housing. 2 is a plan view of the optical element of FIG. 1 as viewed from the outside of the housing. 2 is a plan view of the optical element of FIG. 1 as viewed from the outside of the housing. FIG. 2 is a plan view of the imaging element of FIG. 1 . FIG. 2 is a block diagram showing a hardware configuration of the control device shown in FIG. 1 . 2 is a block diagram showing a functional configuration of the pupil detection device shown in FIG. 1 . 1 is a graph showing the relationship between pupil area A and pupil luminance _LB in a bright pupil image and pupil luminance _LD in a dark pupil image. 11 is a graph showing the relationship between pupil area A and pupil luminance L _S in a difference image. 1 is a graph showing the relationship between the distance from the center of the camera aperture to the light source and pupil luminance. 13 is a plan view showing the configuration of a light source and an optical element according to a modified example. FIG. 13 is a plan view showing the configuration of a light source and an optical element according to a modified example. FIG.

Hereinafter, a preferred embodiment of the pupil detection device according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are given the same reference numerals, and duplicated explanations will be omitted.

(First embodiment)

As shown in FIG. 1, the pupil detection device 1 according to the first embodiment of the present invention includes a camera 2, an illumination device (light source) 3, and a control device (arithmetic device) 4. The camera 2 includes a housing 5, an image sensor (image sensor) 6 such as a CCD or CMOS housed in the housing 5, and an objective lens 7 and an optical element 9 housed in the housing 5. The camera 2 may be a high-speed camera in which the acquisition time interval of one frame of an image is very short, or may be a so-called medium-speed camera, a low-speed camera, or a camera having a frame rate of about 60 Hz or 30 Hz. The housing 5 has a circular opening 8 formed on a surface facing the eyeball A of the subject of observation. The objective lens 7 and the optical element 9 are disposed between the opening 8 inside the housing 5 and the image sensor 6. The optical axis L0 of the objective lens 7 coincides with the central axis of the opening 8. The image sensor 6 is fixed so that its light receiving surface perpendicularly intersects with the optical axis L0 of the objective lens 7. The optical element 9 is disposed inside the objective lens 7 (on the imaging element 6 side) on the optical axis L0 of the objective lens 7. The imaging element 6 captures an image of the eyeball A of the subject to be observed, thereby generating eye image data and outputting the data to the control device 4. The control device 4 controls the light emission intensity, lighting timing, and lighting period (light emission period) of the illumination device 3, as well as the imaging timing and imaging period of the camera 2. The control device 4 also executes image generation processing, comparison processing, pupil detection processing, and corneal reflex detection processing based on the eye image data output from the imaging element 6. That is, the control device 4 also functions as a pupil detection means and a corneal reflex detection means.

The diameter of the opening 8 is smaller than the diameter of the objective lens 7, and is approximately the same as the effective diameter of the objective lens 7. The optical element 9 has an approximately circular shape overall, and has a diameter approximately the same as the effective diameter of the objective lens 7. With this configuration, an image near the subject's eyeball A is introduced through the opening 8 toward the objective lens 7 and the image sensor 6 in the camera 2, and then imaged so as to converge on the light receiving surface of the image sensor 6 by the optical system including the objective lens 7 and the optical element 9 in the camera 2.

The lighting device 3 emits illumination light toward the face of the person being observed. As shown in FIG. 2, the lighting device 3 has a casing 10 and light sources 3A and 3B embedded in the casing 10. The casing 10 is attached to the housing 5 so as to cover the outside of the opening 8 along the edge of the opening 8. The light sources 3A and 3B are both provided on the casing 10 so as to emit illumination light along the optical axis L0 of the objective lens 7, and are configured to be point symmetrical with respect to the center of the opening 8.

The light source (first light source) 3A is a light source for illuminating the eye (pupil) of the subject of observation with illumination light (first illumination light) for obtaining a bright pupil image. A bright pupil image refers to an image in which the pupil of the subject of observation is relatively brighter than a dark pupil image described later. The light source 3A is, for example, composed of a plurality of semiconductor light-emitting elements (LEDs) whose output light has a central wavelength (first central wavelength) in the near-infrared region, and is disposed at a first distance D1 that is relatively close from the center of the opening 8. Specifically, the light-emitting elements constituting the light source 3A are arranged in a ring shape on the casing 10 at equal intervals along the edge of the opening 8 outside the opening 8. It is preferable that the light source 3A is provided as close as possible to the edge of the opening 8. As a result, as described later, in the image of the subject of observation illuminated by the light source 3A, the pupil is projected brighter, making it easier to detect even a small pupil.

The light source (second light source) 3B is a light source for illuminating the eye (pupil) of the subject with illumination light (second illumination light) for obtaining a dark pupil image. The dark pupil image refers to an image in which the pupil of the subject is relatively dark compared to the above-mentioned bright pupil image. The light source 3B is, for example, composed of a plurality of semiconductor light-emitting elements (LEDs) in the near-infrared region whose output light has a central wavelength (second central wavelength) longer than the first central wavelength, and is arranged at a relatively long second distance D2 from the center of the opening 8. This second distance D2 is greater than the first distance D1. Specifically, the light-emitting elements constituting the light source 3B are arranged in a continuous ring shape on the casing 10 at equal intervals from the light source 3A to the outside of the opening 8.

When illumination light is emitted from the light source 3A onto the eyeball A of the subject to be observed, and the pupil illuminated by the illumination light is captured by the camera 2, a bright pupil image is obtained. When illumination light is emitted from the light source 3B onto the eyeball A of the subject to be observed, and the pupil illuminated by the illumination light is captured by the camera 2, a dark pupil image is obtained. This is due to the following property. In other words, when illumination light onto eyeball A is incident from a position relatively distant from the optical axis L0 of camera 2, the illumination light that enters through the pupil of eyeball A, is reflected inside the eyeball, and passes through the pupil again is less likely to reach camera 2, so the pupil appears relatively dark.

Here, the first central wavelength of the output light of the light source 3A is set to, for example, 850 nm, and the second central wavelength of the output light of the light source 3B is set to, for example, 940 nm, which is longer than the first central wavelength. However, the light source 3A, which is a light source for acquiring a bright pupil image, may be a light source with a wavelength of output light shorter than around 900 nm in that the luminance of the light reflected back from the retina is strong. Similarly, the light source 3B, which is a light source for acquiring a dark pupil, may be a light source with a wavelength of output light longer than around 900 nm in that the luminance of the light reflected back from the retina is weak. On the other hand, since the light emission power of a light source with a long wavelength is weak and the sensitivity of the camera generally decreases as the wavelength becomes longer, it is preferable that the light source 3B has about twice as many light-emitting elements as the light source 3A. If there is no room to arrange a large number of light-emitting elements, the light source 3B may have a structure in which light-emitting elements are arranged in a double ring shape.

The light emission intensities (light emission powers) of the light sources 3A and 3B are preset so that the illuminance on the face of the subject being photographed is approximately the same when the light sources 3A and 3B are illuminated for the same lighting period. For this purpose, for example, the current value or power value supplied from the control device 4 to the light sources 3A and 3B is preset.

Figure 3 shows the structure of the optical element 9 arranged inside the opening 8. The optical element 9 has a disk-shaped structure and is arranged so that the optical axis L0 of the objective lens 7 passes near its center and is approximately perpendicular to the optical axis L0. This optical element 9 is composed of semicircular plate-shaped splitting elements 9A and 9B split into two at the center. The splitting element 9A is an optical element in which a bandpass filter that passes the first illumination light of a first center wavelength and a first polarizer that transmits linearly polarized light with a polarization direction of 0 degrees based on the vertical direction of the opening 8 are combined in two layers. The splitting element 9B is an optical element in which a bandpass filter that passes the second illumination light of a second center wavelength and a second polarizer that transmits linearly polarized light with a polarization direction of 90 degrees based on the vertical direction are combined in two layers. That is, the polarization direction of the linearly polarized light transmitted by the first polarizer and the polarization direction of the linearly polarized light transmitted by the second polarizer are approximately perpendicular to each other. The optical element 9 having such a structure is configured so that the splitting elements 9A, 9B are arranged on both sides of a central axis (line) that passes horizontally through the center of the opening 8 along the opening surface of the opening 8, more specifically, so as to be linearly symmetrical with respect to the central axis. The light sources 3A, 3B of the illumination device 3 are configured in a similar manner. Note that the shape of the optical element 9 combining the splitting elements 9A, 9B is not limited to a disk shape, and may be any other shape, such as a rectangle, as long as it corresponds to the shape of the edge of the opening 8.

On the other hand, the optical element 9 may have a structure as shown in FIG. 4. The optical element 9 shown in FIG. 4 is composed of splitting elements 9A and 9B having a continuous shape corresponding to the shape of the edge of the opening 8, and these splitting elements 9A and 9B have a shape divided into two by a boundary line along the edge of the opening 8, and the splitting element 9B is arranged inside the splitting element 9A. That is, the splitting element 9B has a disk-like shape (circular shape) whose center is located near the optical axis L0, and the splitting element 9A has a ring-like shape located outside the splitting element 9B. The optical element 9 having such a structure is configured so that the splitting elements 9A and 9B are point-symmetric with respect to the center of the opening 8, similar to the illumination device 3. The light sources 3A and 3B of the illumination device 3 are also configured in the same way. Note that the shape of the optical element 9 combining the splitting elements 9A and 9B is not limited to a disk shape, and may be other shapes such as a rectangular shape as long as it corresponds to the shape of the edge of the opening 8.

Figure 5 shows the structure of the image sensor 6 arranged inside the opening 8 as seen from the opening 8 side. The image sensor 6 has a light receiving surface 20 arranged to be approximately perpendicular to the optical axis L0, a plurality of pixels 21 arranged in a two-dimensional array on the light receiving surface 20 with a light receiving element (not shown) formed inside, and four types of polarizers 22A, 22B, 22C, and 22D arranged on the surfaces of the plurality of pixels 21 so as to cover the pixels 21. That is, different types of polarizers 22A, 22B, 22C, and 22D are attached to the surfaces of four adjacent pixels 21 (hereinafter also referred to as a four-pixel group). The polarizer 22A is an optical element that transmits linearly polarized light with a polarization direction of 0 degrees relative to the vertical direction of the opening 8. The polarizer 22B is an optical element that transmits linearly polarized light with a polarization direction rotated 45 degrees counterclockwise relative to the vertical direction of the opening 8. The polarizer 22C is an optical element that transmits linearly polarized light having a polarization direction rotated 90 degrees counterclockwise with respect to the up-down direction of the opening 8. The polarizer 22D is an optical element that transmits linearly polarized light having a polarization direction rotated 135 degrees counterclockwise with respect to the up-down direction of the opening 8. The imaging element 6 has a structure in which such structures are repeated two-dimensionally (in the left-right and up-down directions in FIG. 5 ).

The range of the four pixel groups on the light receiving surface 20 of the image sensor 6 is sufficiently small compared to the size of the objective lens 7. Therefore, the optical element 9 and the image sensor 6 are configured so that the first illumination light passing through the splitting element 9A of the optical element 9 is incident on the four pixels constituting the four pixel groups with the same intensity, and the second illumination light passing through the splitting element 9B of the optical element 9 is incident on the four pixels constituting the four pixel groups with the same intensity.

Next, with reference to Figures 6 and 7, the configuration of the control device 4 included in the pupil detection device 1 will be explained.

The control device 4 may be a computer that controls the image sensor 6 and the light sources 3A, 3B, and processes the eye image data of the subject to be observed to detect the pupil and corneal reflex. The control device 4 may be constructed by a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the control device 4 may be constructed by combining multiple computers of any type. When multiple computers are used, these computers are connected via a communication network such as the Internet or an intranet.

As shown in FIG. 6, the control device 4 comprises a CPU (processor) 101, a main memory unit 102, an auxiliary memory unit 103, a communication control unit 104, an input device 105, and an output device 106. The CPU 101 executes an operating system, application programs, etc. The main memory unit 102 is composed of ROM and RAM. The auxiliary memory unit 103 is composed of a hard disk, flash memory, etc. The communication control unit 104 is composed of a network card or a wireless communication module. The input device 105 includes a keyboard, a mouse, etc. The output device 106 includes a display, a printer, etc.

Each functional element of the control device 4, which will be described later, is realized by loading predetermined software onto the CPU 101 or main memory unit 102, operating the communication control unit 104, input device 105, output device 106, etc. under the control of the CPU 101, and reading and writing data in the main memory unit 102 or auxiliary memory unit 103. Data and databases required for processing are stored in the main memory unit 102 or auxiliary memory unit 103.

As shown in FIG. 7, the control device 4 has an image sensor driving unit 11, a lighting control unit 12, and a detection unit 13 as functional components. The image sensor driving unit 11 is a functional element that controls the timing of shooting the image sensor 6. Specifically, the image sensor 6 is repeatedly imaged at a predetermined frame rate and a predetermined exposure time, and is controlled to continuously acquire eye image data representing an image of the eyeball A of the observation subject. The lighting control unit 12 is a functional element that controls the lighting timing so that the light sources 3A and 3B are simultaneously turned on in accordance with the shooting of the image sensor 6, and controls the amount of light emitted by the light sources 3A and 3B. At this time, in order to improve the accuracy of the detection position of the pupil or corneal reflex when the head of the observation subject moves, it is preferable that the lighting control unit 12 turns on the light sources 3A and 3B for the same period. The detection unit 13 is a functional element that uses the eye image data output from the image sensor 6 to detect the pupil and corneal reflex in the eye image data. There is no limitation on the output destination of the information on the detected pupil and corneal reflex. For example, the control device 4 may display the results as images, graphics, or text on a monitor, store them in a storage device such as a memory or database, or transmit them to another computer system via a communications network.

The detection unit 13 has, as functional components, an image acquisition unit 14, an image calculation unit 16, a difference image generation unit 17, a pupil detection unit 18, and a corneal reflection detection unit 19. The image acquisition unit 14 acquires eye image data that is continuously captured (acquired) at a predetermined frame rate from the image sensor 6. The image calculation unit 16 calculates the luminance value (light intensity) of each pixel of the bright pupil image based on the eye image data at each imaging timing, and acquires the bright pupil image by combining these luminance values. The image calculation unit 16 also calculates the luminance value (light intensity) of each pixel of the dark pupil image based on the eye image data at each imaging timing, and acquires the dark pupil image by combining these luminance values. The difference image generation unit 17 generates a difference image as a kind of comparison image comparing the bright pupil image and the dark pupil image. Specifically, the difference image generation unit 17 generates a difference image comparing both images by calculating the difference in luminance between corresponding pixels of the bright pupil image and the dark pupil image. The pupil detection unit 18 is a functional element that calculates the position of the pupil image using the difference image. The corneal reflection detection unit 19 is a functional element that calculates the position of the corneal reflection image using the bright pupil image or the dark pupil image. An example of the processing performed by the pupil detection unit 18 and the corneal reflection detection unit 19 is as follows. First, the pupil detection unit 18 binarizes the difference image based on a pupil threshold, removes isolated points, removes noise by morphological processing, and performs labeling. Then, the pupil detection unit 18 detects the pixel group having the shape most similar to a pupil as the pupil. At this time, even if the pupil is hidden by the eyelids or eyelashes, the boundary between the eyelids or eyelashes and the pupil is excluded as a false pupil contour, and only the true pupil contour is fitted to an ellipse to detect the position of the true pupil contour on the difference image, and the center position of the pupil image is calculated from the equation of the ellipse obtained by the ellipse fitting. The corneal reflection detection unit 19 also binarizes the bright pupil image near the pupil with a corneal reflection threshold value higher than the pupil luminance, and determines the center of the corneal reflection image as the center of gravity taking the luminance into consideration. The pupil luminance is given by the average luminance of the pixels constituting the pupil obtained by binarization, rather than the area of the ellipse obtained as a result of ellipse fitting. The corneal reflection detection unit 19 may calculate the position of the corneal reflection image using the dark pupil image as the target.

Next, the functional configuration of the image calculation unit 16 will be explained in detail.

上記式中、偏光子２２Ａ，２２Ｂ，２２Ｃ，２２Ｄの透過する光の偏光方向の角度をそれぞれ、θ_ｓｉで表わし、第１の分割素子９Ａ及び第２の分割素子９Ｂを透過する光の偏光方向の角度をそれぞれθ_ａ１、θ_ａ２と表している。 The principle of acquisition of bright pupil images and dark pupil images by the image calculation unit 16 is as follows: When acquiring eye image data, light transmitted through the first and second dividing elements 9A and 9B is incident with the same light intensity on four pixel groups each consisting of four adjacent pixels 21. If the light intensities of the incident light are I(λ _a1 ) and I(λ _a2 ), the light intensities O i ( _i =1 to 4) detected by the pixels 21 to which the polarizers 22A, 22B, 22C, and 22D are attached among the four pixel groups are expressed by the following formula:

In the above formula, the angle of the polarization direction of the light transmitted through the polarizers 22A, 22B, 22C, and 22D is represented as θ _si , and the angle of the polarization direction of the light transmitted through the first dividing element 9A and the second dividing element 9B is represented as θ _a1 and θ _a2 , respectively.

に変形される。このような関係から、第１の分割素子９Ａ及び第２の分割素子９Ｂのそれぞれを透過した第１及び第２の中心波長の光の光強度Ｉ（λ_ａ１），Ｉ（λ_ａ２）は、２行４列の擬似逆行列［Ｍ_ｉｊ］^＋を用いて、下記式；

によって計算される。 The above formula can be expressed as a determinant to obtain the following formula:

From this relationship, the light intensities I(λ a1 ) and I(λ _a2 ) of the light having the first and second central wavelengths transmitted through the first and second dividing elements 9A and 9B, respectively, are expressed by the following equations using a ₂ -row, 4-column pseudo-inverse matrix [M _ij ] ⁺ :

It is calculated by:

Using the above formula, the image calculation unit 16 reads the luminance value [ _Oi ] of each of the four pixel groups from the eye image data, and calculates the luminance value I( _λa1 ) corresponding to the light of the first central wavelength incident on the four pixel groups and the luminance value I( _λa2 ) corresponding to the light of the second central wavelength incident on the four pixel groups using the luminance ^value [ _Oi ] and a preset pseudo-inverse matrix [Mij _] +. The image calculation unit 16 then combines the luminance values I( _λa1 ) calculated for each of the four pixel groups on the light receiving surface 20 to generate two-dimensional image data to obtain a bright pupil image, and combines the luminance values I( _λa2 ) calculated for each of the four pixel groups to generate two-dimensional image data to obtain a dark pupil image.

The action and effect of the pupil detection device 1 of the first embodiment will be explained.

In the pupil detection device 1 of the above-mentioned embodiment, the light from the pupil of the subject illuminated by the light source 3A with the light of the first central wavelength is transmitted through the first division element 9A of the optical element 9 and converted into linearly polarized light at a predetermined angle, and then its luminance value is detected by four adjacent pixels to which the four types of polarizers 22A, 22B, 22C, and 22D of the image sensor 6 are attached. At the same time, the light from the pupil of the subject illuminated by the light source 3B with the light of the second central wavelength is transmitted through the second division element 9B of the optical element 9 and converted into linearly polarized light at an angle different from the predetermined angle, and then its luminance value is detected by four adjacent pixels to which the four types of polarizers 22A, 22B, 22C, and 22D of the image sensor 6 are attached. Here, the image of the pupil illuminated with the light of the first central wavelength is relatively brighter than the image of the pupil illuminated with the light of the second central wavelength. Then, control device 4 calculates a luminance value corresponding to the light of the first central wavelength from the luminance values of four adjacent pixels detected by image sensor 6 and combines the luminance values to obtain a bright pupil image, calculates a luminance value corresponding to the light of the second central wavelength from the luminance values of adjacent pixels detected by image sensor 6 and combines the luminance values to obtain a dark pupil image, and calculates the position of the pupil image based on the bright pupil image and the dark pupil image. This causes no difference in the timing at which the bright pupil image and the dark pupil image are obtained, making it possible to easily improve the detection accuracy of the pupil image regardless of the state of the subject being observed (for example, even if the subject being observed rotates their eyeball).

In particular, in this embodiment, the polarization direction of the light passing through the first splitting element 9A is approximately orthogonal to the polarization direction of the light passing through the second splitting element 9B. In this case, the polarization directions of the light of the first central wavelength and the light of the second central wavelength incident on each pixel of the image sensor 6 are clearly separated, improving the accuracy of the luminance calculation of the bright pupil image and the dark pupil image, and as a result, improving the accuracy of detecting the position of the pupil image.

In addition, the image sensor 6 has polarizers 22A, 22B, 22C, and 22D attached to every four adjacent pixels, which transmit linearly polarized light at four different angles, and the control device 4 calculates the luminance values corresponding to the light of the first and second central wavelengths based on the luminance values of the four adjacent pixels in the eye image data. In this case, it is possible to equalize the balance of the arrangement of multiple pixels on the light receiving surface 20 that detect various polarization direction components of the light of the first and second central wavelengths. This makes it possible to equalize the images of the bright pupil image and the dark pupil image, and stabilize the accuracy of the position of the detected pupil image.

In addition, the distance of light source 3A from the center of aperture 8 of camera 2 is made smaller than the distance of light source 3B from the center of aperture 8. This structure makes it possible to increase the luminance difference between the image of the pupil illuminated with light of the first central wavelength and the image of the pupil illuminated with light of the second central wavelength. As a result, the detection accuracy of the pupil image position using the comparison image can be further improved.

The optical element 9 is configured so that the first dividing element 9A and the second dividing element 9B are point-symmetric or line-symmetric about the opening 8, and the light source 3A and the light source 3B are configured so that they are point-symmetric or line-symmetric about the opening 8. This makes it possible to prevent uneven brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, makes it possible to make only the pupil image stand out in the comparison image, and improves the accuracy of detecting the pupil image position using the comparison image.

In particular, when optical element 9 has a structure as shown in FIG. 3, it is possible to prevent unevenness in the brightness of the pupil image and its peripheral areas in the bright pupil image and dark pupil image in the vertical direction, and to improve the accuracy of detecting the pupil image position using the comparison image.
Furthermore, when optical element 9 has a structure as shown in FIG. 4, it is possible to prevent two-dimensional unevenness in the luminance of the pupil image and its peripheral areas in the bright pupil image and the dark pupil image, and further improve the accuracy of detecting the pupil image position using the comparison image.

Furthermore, the control device 4 calculates the position of the corneal reflection of the subject caused by the illumination of the light source 3A or the light source 3B based on the bright pupil image or the dark pupil image. With this configuration, the position of the corneal reflection can be stably detected.

The advantages of the configuration of the lighting device 3 and the optical element 9 in this embodiment are explained.

In pupil detection based on a comparison image such as a difference image, the greater the luminance difference in the pupil image between the bright pupil image and the dark pupil image, the higher the accuracy of pupil detection. In addition, if the pupil position does not change relative to the optical system (camera 2 and illumination device 3), the line of sight direction relative to the optical system does not change, and further, the light amount of the illumination device 3 and the sensitivity of the camera 2 do not change (referred to as "pupil detection conditions"), the larger the pupil, the higher the luminance of the pupil in the bright pupil image and the pupil in the dark pupil image. Figure 8 is a graph showing the relationship between pupil area A and pupil luminance L _B in the bright pupil image and pupil luminance L _D in the dark pupil image, and Figure 9 is a graph showing the relationship between pupil area A and pupil luminance L _S in the difference image. Thus, ideally, the following relational expression holds.
L _B = k _B · A,
L _D = k _D · A,
L _S =k _S・A=(k _B -k _D )・A
In the above relational expression, k _B , k _D , and k _S are proportional constants, and are constant when the above "pupil detection condition" is satisfied. As can be seen from this relational expression, when the pupil is large, the pupil luminance of the bright pupil image is high, and the pupil luminance is also high in the difference image. Therefore, since the luminance of the area other than the pupil is close to zero, it is easy to detect the pupil from the difference image. On the other hand, when the pupil is small, the light incident on the pupil is also reduced according to the pupil area, so the pupil luminance is low. As a result, when the pupil is small (for example, about 2.5 mm in diameter), it is necessary to irradiate the face with stronger light. In order to increase the light amount of the light source, it is necessary to increase the current supplied to the light source or increase the number of light sources, which increases power consumption or cost. In addition, in order to increase the pupil luminance in the bright pupil image, it is necessary to place the light source as close as possible to the opening of the camera, but in that case, there is a possibility that a problem such as a lack of physical space to place the light source may occur.

FIG. 10 is a graph showing the relationship between the distance from the center of the aperture of the camera to the light source (the difference in angle between the straight line connecting the eye of the observation subject to the light source and the center of the aperture) and the pupil luminance, where graphs G _B show the relationship when the pupil is large, and graphs G _S show the relationship when the pupil is small. Thus, the pupil luminance increases rapidly as the distance between the aperture and the light source decreases from a large value. As described above, in order to increase the difference between the pupil luminance of the bright pupil image and the pupil luminance of the dark pupil image, it is preferable to place the light source 3A as close as possible to the aperture and place the light source 3B away from the aperture. In this case, when the pupil is large, the difference in luminance between the bright pupil image and the dark pupil image is likely to be large, so problems are unlikely to occur. Problems are likely to occur when the pupil becomes extremely small. Even in such a case, it is desirable to increase the pupil luminance of the bright pupil image in order to more reliably detect the pupil from the difference image. This is because, when the pupil is small, as shown in graph G _S , moving light source 3B a short distance from the opening sufficiently reduces the pupil luminance (distance D _A ), and moving it further than necessary barely reduces the pupil luminance at all. In contrast, moving light source 3A closer to the opening (for example, from distance D _B to distance D _C ) causes the pupil luminance of the bright pupil image to increase rapidly, and the accuracy of pupil detection from the difference image increases.

On the other hand, when the pupil is large, as shown in graph G _B , the pupil tends to be bright until the light source 3B is quite far from the aperture. Since the pupil luminance of the bright pupil image is much higher than that of the dark pupil image, there is a luminance difference between the bright and dark pupil images, so long as the luminance value does not reach a saturation level, there is no problem with pupil detection. If it is necessary to reduce the amount of light from the light source for some reason, such as to reduce power consumption, it is also possible to reduce the amount of light from the light sources 3A and 3B in order to keep the pupil luminance constant when the pupil is large and when it is small. As a result, in graphs G _B and G _S , the luminance level when the pupil is large approaches the luminance level when the pupil is small.

Based on the above findings, this embodiment employs a configuration of the illumination device 3 and optical element 9 as shown in Figures 2 and 3. By employing such a configuration, it is possible to position the light source 3A as close as possible to the opening 8, and it is possible to guide the light components of the first and second central wavelengths of the pupil image from the subject to the imaging element 6 in the opening 8 in a well-balanced manner.

In addition, in this embodiment, when the configuration of the illumination device 3 and the optical element 9 shown in Figures 2 and 4 is adopted, the light components of the first and second central wavelengths of the pupil image from the subject of observation can be guided to the image sensor 6 in the opening 8 in a well-balanced manner in the two-dimensional direction. In addition, this configuration allows the light source 3A to be arranged close to the opening 8. Since the splitting element 9A has a ring-shaped shape, if an arbitrary point on the splitting element 9A is used as a reference, some of the light sources 3A will be far away, but if there is even one light source 3A close to that point, the effect of brightening the pupil is very high, so the light sources 3A as a whole have a sufficient effect of brightening the pupil. In addition, the light source 3B can be sufficiently separated from the splitting element 9B in the opening 8, and the pupil luminance of the dark pupil image can be sufficiently low.

The present invention is not limited to the above-described embodiment. The configuration of the above embodiment may be modified in various ways.

In the above-described embodiment, the difference image generating unit 17 generates a difference image as a comparison image by comparing the bright pupil image and the dark pupil image, but as described in JP 2008-246004 A, a division image may be used as the comparison image. Furthermore, an image may be generated as the comparison image by comparing the bright pupil image and the dark pupil image by other calculations.

In the above-described embodiment, the control device 4 may detect the gaze direction and gaze point of the subject of observation based on the position of the pupil image detected by the pupil detection unit 18 based on the difference image and the position of the corneal reflection image detected by the corneal reflection detection unit 19 for each of the dark pupil image and the bright pupil image. The method for calculating the gaze direction and gaze point may employ a method developed by the present inventors (see International Publication WO2012/020760). By using the position of the pupil image detected by the control device 4 of this embodiment, the gaze direction and gaze point of the subject of observation can be stably detected.

In addition, the configuration of each light source in the lighting device 3 in the above-mentioned embodiment and the configuration of the optical element 9 may be modified in various ways.

Figure 11 shows a modified example of the arrangement of the light sources 3A, 3B and the splitting elements 9A, 9B. In this example, the light sources 3A, 3B and the splitting elements 9A, 9B are arranged on both sides of the central axis (line) that passes through the center of the opening 8 from top to bottom along the opening surface of the opening 8, more specifically, arranged so as to be linearly symmetrical with respect to the central axis. The splitting element 9A has two divided shapes, which are arched shapes on the left and right, corresponding to the shape of the edge of the opening 8, provided on both sides of the central axis that passes through the center of the opening 8 from top to bottom. The splitting element 9A may have two or more divided shapes (for example, four). The splitting element 9B is arranged at the center of the opening 8, sandwiched between the arched shapes. The light source 3A is arranged outside the splitting element 9A near the opening 8, separated left and right, and the light source 3B is arranged outside the light source 3A, separated left and right. This configuration also has a left-right symmetrical configuration, so the luminance difference in the face region in the bright pupil image and the dark pupil image is kept small, and this is an effective configuration when it is desired to reduce the vertical size of the illumination device 3.

Figure 12 shows another modified example of the arrangement of the light sources 3A, 3B and the splitting elements 9A, 9B. The configuration of the splitting elements 9A, 9B is the same as that of Figure 3, and the light source 3A is arranged only near the opening 8 on the splitting element 9A side (left side), and the light source 3B is also arranged separately on the left side of the opening 8. In this configuration, the distance between the opening 8 on the splitting element 9A side and the light source 3A is close, and there is no light source 3A far from the opening 8 on the splitting element 9A side, so the pupil luminance in the bright pupil image is kept high. Furthermore, the distance between the light source 3B and the light source 3A and the center of the opening 8 can be reduced while maintaining a certain distance from the opening 8 on the splitting element 9B side. As a result, the size of the entire optical system can be reduced. Even if a luminance gradient occurs in the face or within the pupil in the bright pupil image and the dark pupil image, the positions of the light source 3A and the light source 3B are close, so the tendency of the luminance gradient does not differ greatly between the two images, so parts other than the pupil are less noticeable in the difference image, and the detection accuracy of the pupil can be maintained.

11, the present invention is not limited to point-symmetric and line-symmetric configurations, and may have an arrangement and shape that deviates from point-symmetric or line-symmetric arrangement and shape with respect to the center or central axis of the opening 8. For example, the arrangement and shape may be adjusted from point-symmetric or line-symmetric arrangement and shape in order to adjust the luminance of the bright pupil image and the dark pupil image.

The optical element 9 may also be configured with a plurality of division sections divided into a plurality of parts of any shape. For example, it may be configured with a plurality of rectangular division sections. In such a configuration, the division elements 9A and 9B are each assigned appropriately (for example, alternately) to the plurality of division sections.

In addition, the optical element 9 does not need to be disposed inside the objective lens 7 (on the imaging element 6 side), but may be disposed outside the objective lens 7 (on the side of the subject of observation). Alternatively, if the camera lens provided in the camera 2 is configured to include multiple optical lenses, the optical element 9 may be disposed between any two of the multiple optical lenses, or may be disposed in front of or behind the entire camera lens (on the side of the subject of observation or on the imaging element 6 side).

As described in Japanese Patent No. 4,528,980, the light sources 3A and 3B may be disposed inside the opening 8 of the camera 2. Alternatively, the light source 3A may be disposed inside the opening 8, and the light source 3B may be disposed outside the opening 8.

In the above embodiment, the image sensor 6 has a structure in which four types of polarizers are provided for every four pixel groups, but it is sufficient that the image sensor 6 has a structure in which at least two types of polarizers are provided. For example, two types of polarizers may be provided for every two adjacent pixels 21, or three types of polarizers may be provided for every three adjacent pixels 21. Even in such a modified example, the control device 4 can calculate the luminance value of the light component of the first central wavelength and the luminance value of the light component of the second central wavelength based on the pixel values of two adjacent pixels or three adjacent pixels.

In addition, in the above embodiment, the control device 4 calculates the luminance value of the light component of the first central wavelength and the luminance value of the light component of the second central wavelength based on the pixel values of four adjacent pixels, but it is also possible to calculate the luminance value of the light component of the first central wavelength and the luminance value of the light component of the second central wavelength based on the pixel values of at least two pixels (two pixels, or three pixels) among the four adjacent pixels.

Here, in the above embodiment, the polarization direction of the light passing through the first polarizer and the polarization direction of the light passing through the second polarizer may be approximately orthogonal. In this case, the polarization directions of the light of the first central wavelength and the light of the second central wavelength incident on each pixel of the image sensor are clearly separated, improving the accuracy of the luminance calculation of the bright pupil image and the dark pupil image, and as a result, improving the accuracy of the detection of the position of the pupil image.

In addition, in the image sensor, a polarizer that transmits linearly polarized light at four different angles may be attached to every four adjacent pixels, and the computing device may calculate the luminance corresponding to each of the light of the first and second central wavelengths based on the luminance of at least two of the four adjacent pixels in the eye image. In this case, it is possible to equalize the arrangement balance of multiple pixels that detect various polarization direction components of light of the first and second central wavelengths on the image sensor. This makes it possible to equalize the images of the bright pupil image and the dark pupil image, and stabilize the accuracy of the position of the detected pupil image.

The first light source may be arranged at a first distance from the center of the camera's aperture, and the second light source may be arranged at a second distance from the center of the aperture that is greater than the first distance. In this case, the luminance difference between the image of the pupil illuminated with light of the first central wavelength and the image of the pupil illuminated with light of the second central wavelength can be made larger. As a result, the accuracy of detecting the position of the pupil image using the comparison image can be further improved.

The second splitting element may be disposed inside the first splitting element. This prevents uneven brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, makes only the pupil image stand out in the comparison image, and improves the accuracy of detecting the pupil image position using the comparison image.

The second splitting element may have a circular shape located at the center of the opening, and the first splitting element may have a ring shape located outside the second splitting element. In this case, it is possible to prevent unevenness in the brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to highlight the brightness difference in the pupil image. As a result, it is possible to make only the pupil image stand out more in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

The first and second splitting elements may be arranged on both sides of a line on the optical element, and the first and second light sources may be arranged to sandwich the opening from both sides. In this case, unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image can be prevented, only the pupil image can be made to stand out in the comparison image, and the detection accuracy of the pupil image position using the comparison image can be improved.

Furthermore, the first dividing element may have two or more dividing shapes of a predetermined shape provided on both sides of a line on the optical element, and the second dividing element may be arranged sandwiched between the two or more dividing shapes. In this case as well, it is possible to prevent unevenness in the brightness of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to highlight the brightness difference in the pupil image. As a result, it is possible to make only the pupil image stand out more in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

The optical element may also have a plurality of division sections that are divided into a plurality of parts in a predetermined shape, and the first division element and the second division element may be assigned to the plurality of division sections. In this case as well, it is possible to prevent unevenness in the luminance of the pupil image and its surroundings in the bright pupil image and the dark pupil image, and to accentuate the luminance difference in the pupil image. As a result, it is possible to make only the pupil image more conspicuous in the comparison image, and to further improve the detection accuracy of the pupil image position using the comparison image.

Furthermore, the computing device may calculate the position of the subject's corneal reflection image caused by turning on the first light source or the second light source based on the bright pupil image or the dark pupil image. With this configuration, the position of the corneal reflection image can be detected stably.

Furthermore, the computing device may detect the subject's gaze direction based on the position of the pupil image calculated based on the comparison image and the position of the corneal reflection image calculated based on the bright pupil image or the dark pupil image. In this way, the subject's gaze direction can be detected stably.

One aspect of the present disclosure is the use of a pupil detection device that detects pupils from images of a person, and can easily improve the accuracy of pupil detection regardless of the subject's condition.

A...eyeball, 3A, 3B...light source, 1...pupil detection device, 2...camera, 3...illumination device, 4...control device (computing device), 6...imaging element (image sensor), 8...opening, 9...optical element, 9A, 9B...splitter element, 21...pixel, 22A, 22B, 22C, 22D...polarizer.

Claims (11)

A camera that captures an image of a subject's eye to obtain an eye image;
a first light source provided outside or inside an opening of the camera and configured to illuminate the pupil of the subject with light of a first central wavelength with respect to the camera;
a second light source provided outside or inside the opening of the camera and illuminating the pupil of the subject with a second central wavelength different from the first central wavelength with respect to the camera;
a computing device for processing the eye image,
The camera includes:
An image sensor in which a polarizer that transmits linearly polarized light having at least two different angles is attached to each of adjacent pixels;
an optical element in which a first dividing element is combined with a bandpass filter that passes light of the first central wavelength and a first polarizer that transmits linearly polarized light at a predetermined angle, and a second dividing element is combined with a bandpass filter that passes light of the second central wavelength and a second polarizer that transmits linearly polarized light at an angle different from the predetermined angle, are provided along the opening between the opening and the image sensor;
having
The computing device includes:
calculating a luminance corresponding to light of the first central wavelength based on the luminance of the adjacent pixels in the eye image, and combining the calculated luminances to obtain a bright pupil image in which the pupil of the subject is relatively bright;
calculating a luminance corresponding to light of the second central wavelength based on the luminance of the adjacent pixels in the eye image, and combining the calculated luminances to obtain a dark pupil image in which the pupil of the subject is relatively dark;
calculating a position of the pupil image of the subject based on a comparison image obtained by comparing the bright pupil image with the dark pupil image;
Pupil detection device. The polarization direction of light transmitted through the first polarizer and the polarization direction of light transmitted through the second polarizer are substantially perpendicular to each other.
2. The pupil detection device according to claim 1. In the image sensor, a polarizer that transmits linearly polarized light having four different angles is attached to every four adjacent pixels,
The computing device includes:
calculating luminances corresponding to the light of the first and second central wavelengths based on luminances of at least two pixels among the four adjacent pixels in the eye image;
3. The pupil detection device according to claim 1 or 2. the first light source is disposed at a first distance from a center of an aperture of the camera;
The second light source is disposed at a second distance from the center of the opening that is greater than the first distance.
The pupil detection device according to any one of claims 1 to 3. The second dividing element is disposed inside the first dividing element.
The pupil detection device according to any one of claims 1 to 4. the second dividing element has a circular shape disposed at the center of the opening;
The first divided element has a ring shape disposed outside the second divided element.
6. The pupil detection device according to claim 5. the first and second dividing elements are configured to be disposed on opposite sides of a line on the optical element;
The first light source and the second light source are arranged to sandwich the opening from both sides.
The pupil detection device according to any one of claims 1 to 4. the first dividing element has two or more dividing shapes of a predetermined shape provided on both sides of a line on the optical element;
The second divided element is disposed between the two or more divided shapes.
8. A pupil detection device according to claim 7. The optical element has a plurality of divided portions that are divided into a plurality of portions in a predetermined shape,
The first division element and the second division element are assigned to the plurality of division parts.
The pupil detection device according to any one of claims 1 to 4. The arithmetic device calculates a position of a corneal reflection image of the subject caused by turning on the first light source or the second light source based on the bright pupil image or the dark pupil image.
The pupil detection device according to any one of claims 1 to 9. the arithmetic device detects the gaze direction of the subject based on a position of the pupil image calculated based on the comparison image and a position of the corneal reflection image calculated based on the bright pupil image or the dark pupil image.
The pupil detection device according to claim 10.

Images