JP3219425B2 - Eye gaze detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detecting device for detecting a line of sight (fixation point) of an observer using an optical device such as a camera.

[0002]

2. Description of the Related Art Conventionally, a so-called line-of-sight detection device for detecting which position on an observation surface an observer of an optical device observes has been disclosed in Japanese Patent Application Laid-Open No. 61-172552 and Japanese Patent Application Laid-Open No.
Japanese Patent Application Laid-Open No. 2415111, Japanese Patent Application Laid-Open No. 2-5, and the like.

For example, in Japanese Patent Application Laid-Open No. Sho 61-172552, a parallel light beam from a light source is projected to the anterior segment of an observer's eyeball, and a corneal reflection image due to light reflected from the cornea and an image forming position of a pupil are used. And seek the visual axis.

FIGS. 15A and 15B are views for explaining the principle of the line-of-sight detection method. FIG. 15A is a schematic diagram of a line-of-sight detection optical system, and FIG. FIG.

In FIG. 1, reference numeral 5 denotes a light source such as a light emitting diode which emits infrared light insensitive to an observer, and is disposed on a focal plane of the light projecting lens 3.

The infrared light emitted from the light source 5 is
The light becomes parallel light and is reflected by the half mirror 2 so that the eyeball 20
The first cornea 21 is illuminated. At this time, a corneal reflection image d due to a part of the infrared light reflected on the surface of the cornea 21 is transmitted through the half mirror 2, condensed by the light receiving lens 4, and re-images at a position Zd ′ on the photoelectric element array 6.

Further, the ends (pupil edge) a, b of the iris 23
Forms an image of the ends a and b at positions Za 'and Zb' on the photoelectric element array 6 via the half mirror 2 and the light receiving lens 4. When the rotation angle θ of the optical axis a of the eyeball with respect to the optical axis (optical axis a) of the light receiving lens 4 is small, the Z coordinates of the ends a and b of the iris 23 are Za and Zb, and the center position c of the iris 23 is The coordinates Zc are expressed as Zc ≒ (Za + Zb) / 2.

Further, the Z coordinate of the position d where the corneal reflection image is generated is Z
d, assuming that the distance between the center of curvature O of the cornea 21 and the center C of the pupil 24 is OC, the rotation angle θ of the optical axis a of the eyeball is OC * SI
Nθ ≒ Zc−Zd (1) is substantially satisfied.

Here, the Z coordinate Zd of the position d of the corneal reflection image coincides with the Z coordinate Z _{0 of the} center of curvature O of the cornea 21.
Therefore, the arithmetic means 9 detects the position of each singular point (corneal reflection image d and the end portions a and b of the iris) projected on the surface of the photoelectric element array 6 as shown in FIG. The rotation angle θ of the axis b can be obtained. At this time, equation (1) is

[0010]

[Outside 1] Can be changed. Here, β is the position d where the corneal reflection image is generated.
The magnification is determined by the distance L1 between the light-receiving lens 4 and the light-receiving lens 4 and the distance L0 between the light-receiving lens 4 and the photoelectric element array 6, and is usually substantially constant.

[0011] This is because, for example, in a camera automatic focus detection device, when distance measuring points are provided not only at the center of the screen but also at a plurality of places in the screen, the observer selects one of the distance measuring points and performs automatic focus detection. When trying to do so, it is effective to omit the trouble of selecting and inputting one of them, regard the point observed by the observer as a ranging point, and automatically select the ranging point to perform automatic focus detection. .

[0012]

By the way, in the conventional gaze detecting device, the eyeball image is processed in the horizontal direction,
From the position of the pupil edge on the horizontal line (or a plurality of lines above and below it) corresponding to the diameter of the pupil circle (the circle forming the boundary between the pupil portion and the iris portion-referred to as the "pupil edge"), the horizontal center position of the pupil And the direction of the line of sight is detected.

As a pupil edge extraction method, since the reflectance of the pupil is the lowest among the eyeballs, it is appropriate to use the portion having the lowest luminance value in the eyeball image as the pupil determination condition.

However, considering the case where the time-series eyeball image signal output from the sensor is sequentially processed to extract the pupil edge due to the limitation of the storage capacity of the processing device, the lowest value among all the signals is considered. Is not determined until all image signals have been processed, so that the signal output may be judged as a pupil under inappropriate conditions in the middle edge extraction processing, which is extremely inconvenient for gaze detection.

[0015]

According to the first aspect of the present invention, a plurality of pixels are provided to solve the above problem .
Et consisting line are arranged side by side a plurality, more than a light receiving unit, the photoelectric conversion signals from a predetermined number of lines have been output by receiving the eyeball image to output a photoelectric conversion signal of each pixel in each line
Every time a photoelectric conversion signal is output for each line,
A minimum value holding unit that holds a minimum value among the photoelectric conversion signals output from a predetermined number of lines, and a photoelectric conversion signal that looks like a pupil edge of an eyeball using the minimum value held by the minimum value holding unit. A first extraction unit for extracting a pixel to be output, and a pixel for outputting a photoelectric conversion signal like a pupil edge using a final lowest value held in the lowest value holding unit after all the photoelectric conversion signals are output. A second extraction unit that extracts from the pixels extracted by the first extraction unit; a Purkinje image detection unit that detects a pixel position at which a signal relating to a Purkinje image is output from the photoelectric conversion signal; A detecting means for detecting a line-of-sight direction based on the position of the pixel extracted by the second extracting means and the pixel position detected by the Purkinje image detecting means.

According to a second aspect of the present invention, in order to solve the above problem, a plurality of lines composed of a plurality of pixels are arranged side by side.
Are arranged, a light receiving means for outputting a photoelectric conversion signal of each pixel for each line by receiving eyeball image, a predetermined number of lines
After the photoelectric conversion signal is output, the photoelectric
Each time the conversion signal is output, it is output from a predetermined number of lines.
The lowest luminance value holding means for holding the lowest luminance value among the photoelectric conversion signals to be output, and a pixel for outputting a photoelectric conversion signal like a pupil edge of an eyeball using the lowest luminance value held in the lowest luminance value holding means. First extraction means for extracting, storage means for storing the brightness and position of the pixel extracted by the first extraction means, and the lowest brightness value holding means after all the photoelectric conversion signals are output. A second extraction unit for extracting a pixel which outputs a pupil edge-like photoelectric conversion signal by using the final lowest luminance value among the pixels stored in the storage unit; A Purkinje image detecting means for detecting a pixel position for outputting a signal relating to an image, and a pixel position detected by the second extracting means and a pixel position detected by the Purkinje image detecting means. Detection means for detecting a gaze direction Te,
Is to have.

[0017]

[0018]

FIG. 1 is a schematic view of a main part of an optical system according to an embodiment when the present invention is applied to a single-lens reflex camera, and FIG. 2 is an explanatory diagram of a focus detection unit in FIG. FIG. 3 is a perspective view of a main part of the visual line detection system of FIG.

In FIG. 1, reference numeral 1 denotes an eyepiece, in which a dichroic mirror 1a for transmitting visible light and reflecting infrared light is obliquely provided, and also serves as an optical path splitter.

4 is a light receiving lens, 5 (5a, 5b, 5c,
5d) is an illuminating means, which comprises, for example, a light emitting diode. Of these, 5a and 5b are infrared light emitting diodes which are arranged below the eyepiece and generate a corneal reflection image. The light emitting diodes have a small apparent chip size and a spot shape so that the corneal reflection image can be detected with as high a resolution as possible. Is used.

Reference numerals 5c and 5d denote infrared light emitting diodes provided on the upper sides of both sides of the eyepiece, which have a diffusive property so as to illuminate the entire eyeball, or have surface emitting characteristics. Has an illumination range enough to cover even if it deviates from the center of the optical axis, and is arranged such that a corneal reflection image is not formed on the area sensor 7 by the light receiving lens 4.

The area sensor 7 for detecting iris information is constituted by a plurality of photoelectric element arrays.

The light receiving lens 4 and the photoelectric element array 6 constitute one element of the light receiving means.

The elements 1, 4, 5, 6, 7 constitute an eye gaze detection system.

Reference numeral 101 denotes a photographing lens, 102 denotes a quick return (QR) mirror, 103 denotes a display element, 104 denotes a focus plate, 105 denotes a condenser lens, 106 denotes a penta roof prism, 107 denotes a sub mirror, and 108 denotes a multipoint focus detection device. In addition, focus detection is performed by selecting a plurality of regions in a shooting screen.

The description of the multi-point focus detecting device is not necessary for understanding the present invention, and therefore will be briefly described.

That is, in the present embodiment, as shown in FIG. 2, a field mask 110 having a plurality of slits which are arranged in the vicinity of a predetermined image forming plane of the photographing lens 101 and each of which determines a distance measurement area, and an image in each slit are formed. A lens member 111 that functions as a field lens is arranged close to the lens element, and a set 112 of re-imaging lenses and a set 113 of photoelectric element arrays are arranged in order according to the number of slits. The slit 110, the field lens 111, the set 112 of the re-imaging lens, and the set 113 of the photoelectric element array constitute a well-known focus detection system.

In this embodiment, a part of the subject light transmitted through the photographing lens 101 is reflected by the QR mirror 102 to form a subject image near the focus plate 104. Focus plate 10
The object light diffused by the diffusing surface 4 is the condenser lens 1
05, through the penta roof prism 106 and the eyepiece 1 to the eye point E.

The display element 103 is, for example, a two-layer type guest-host type liquid crystal element that does not use a polarizing plate, and displays a distance measurement area (focus detection position) within a finder visual field.

A part of the subject light transmitted through the taking lens 101 is transmitted through the QR mirror 102 and
The light is reflected at 7 and guided to the above-mentioned multipoint focus detection device 108 arranged at the bottom of the camera body. Further, based on the focus detection information of the position on the object plane selected by the multipoint focus detection device 108, the taking lens 101 is extended or retracted by a not-shown taking lens driving device, and the focus is adjusted.

The line-of-sight detection signal processing circuit 109 is a so-called one-chip microcomputer (MCU), and has a ROM, a RAM, an A / D converter and the like built therein. The MCU 109 executes a series of gaze detection operations according to a program stored in a built-in ROM.

The procedure of the visual line detection will be described briefly. First, the infrared generating diode 5 is turned on. The emitted infrared light enters the eyepiece 1 from above in the figure, is reflected by the dichroic mirror 1a, and illuminates the eyeball 201 of the observer located near the eye point E. The infrared light reflected by the eyeball 201 is reflected again by the dichroic mirror 1a and forms an image on the area sensor 6 by the light receiving lens 4. The MCU 109 performs signal processing on the eyeball image photoelectrically converted by the area sensor 6 and detects gazing point information (line of sight).

The detected point-of-regard information is used for a multipoint focus detection operation and a display operation. That is, the focus is adjusted based on the focus detection result of the distance measuring point closest to the gazing point, and at the same time, the location gazed by the observer is displayed in the viewfinder of the camera by the display element 103, and the gazing point is confirmed. Can be done.

FIGS. 4A and 4B are principle diagrams showing the principle for detecting the line of sight. In FIG. 4A, the luminous fluxes from the infrared light emitting diodes 5a and 5b separately arranged in the lateral direction (X-axis direction) of the photoelectric element array 6 are reflected at the corneal reflection images e and d at positions separated in the X-axis direction. Are formed respectively. At this time, the X coordinate of the middle point of the corneal reflection images e and d matches the X coordinate of the center of curvature o of the cornea 21. Since the interval between the corneal reflection images e and d changes in accordance with the distance between the infrared light emitting diode and the eyeball of the observer, the positions e ′ and d ′ of the corneal reflection images re-imaged on the photoelectric element array 6. , It is possible to determine the imaging magnification β of the reflected image from the eyeball. Infrared light emitting diodes 5c and 5d for detecting iris information
Is arranged to illuminate the eyeball from the side direction of the finder, but to prevent the corneal reflection image from being re-imaged on the photoelectric element array 6 by the light receiving lens 4. This prevents the flare reflection image due to the unnecessary light from entering the boundary positions a and b between the iris and the pupil on the photoelectric element array 6, and the boundary positions a and b.
The detection accuracy of b is prevented from lowering.

FIG. 5A is an explanatory diagram showing a reflected image from an eyeball projected on an area sensor 7 including a plurality of photoelectric element arrays 6 in this embodiment. In the figure, the corneal reflection images e 'and d' are re-imaged on the photoelectric element row Yp '. As an example, FIG. 5B shows an output signal of the light intensity distribution obtained from the photoelectric element row Yp 'at this time.

As described above, the MCU 109 detects the center position of the pupil and the positions of the corneal reflection images e 'and d' from the sensor signals shown in FIGS. 5 (A) and 5 (B). The gaze point (line of sight) is detected based on the positional relationship.

FIG. 6 shows a flowchart of the microcomputer MCU 109 which is a signal processing device according to the embodiment of the present invention.

FIG. 6 is a main flow chart of gaze detection. When the MCU 109 starts the line-of-sight detection operation, the data is initialized in step (# 001) through step (# 000).

The variable EYEMIN is a variable for storing the lowest luminance value in the photoelectric conversion signal of the eyeball reflection image, and is an A / D built in the microcomputer (MCU 109).
Assuming that the resolution of the converter is 8 bits, the lowest value is sequentially compared and updated as the image signal is read. The initial value stores 255 representing the maximum value in 8 bits.

The variable EDGCNT is a variable for counting the number of edges extracted from the boundary between the iris and the pupil as an edge.

Variables IP1, IP2, JP1, and JP2 are variables representing the position of a corneal reflection image (hereinafter, referred to as a "Purkinje image; P image") of the light emitting diodes 5a and 5b, and a range IP1 in the horizontal direction (X axis). ~ IP2, within the area of the eyeball reflection image surrounded by the range JP1 to JP2 in the vertical direction (Y axis).
There are P images.

Now, the number of pixels of the area sensor 7 is 1 in the horizontal direction.
It is assumed that the size is 50 pixels and the size is 100 pixels in the vertical direction. For IP1, IP2, JP1, and JP2, the position (75, 50) at the center of the whole is stored as an initial value.

Following the data initialization, a step (# 00)
Go to 2).

In step (# 002), the light emitting diodes 5a and 5b for the P image and the light emitting diode 5 for the eyeball illumination are used.
Lights c and 5d. In the next step (# 003)
The accumulation operation of the area sensor 7 is started. Since the control of the sensor does not directly relate to the present invention, a detailed description thereof will be omitted. However, in the embodiment of the present invention, it is assumed that the driving is controlled by a sensor interface circuit (not shown).

In step (# 004), the process waits until the accumulation of the area sensor is completed.

When the predetermined charge accumulation is completed, the light emitting diode is turned off in the next step (# 005).

From the next step (# 006) onward, reading of the photoelectric conversion signal of the area sensor is started.

The step (# 006) sets the loop variable J to 0.
This represents a so-called "loop process" in which the process in the frame is executed while counting up from "99" to "99".

In the loop processing in step (# 006), first, in step (# 007), a photoelectric conversion signal of one line in the horizontal direction (X axis) of the area sensor is read. The reading of one line is in a subroutine format, and FIG. 7 shows a flowchart of the subroutine "1 line reading".

When the subroutine "1 line read" is called, the next step (# 101) is executed via the step (# 100) in FIG. Step (# 10
1) and the step (# 102) in that frame represent the same loop processing as the above-described step (# 006). In step (# 101), while the variable K is counted up from 0 to 3, In step (# 102), the process in each frame is executed while the variable I is counted up from 0 to 149. Therefore, step (# 101) and step (# 102) represent a so-called "nested" loop process of the variables K and I.

In the step (# 103) in the loop processing of the step (# 102), the work of restoring the array variables IM (i, k) is performed.

In this embodiment, the microcomputer MC
Although U109 performs signal processing, the storage capacity of a built-in RAM (random access memory) of a microcomputer is generally not large enough to store all pixel information of the area sensor at one time. Therefore, in the present embodiment, while reading out the image signals output from the area sensor one by one, only the latest image signals corresponding to five lines in the horizontal direction (X axis) are stored in the built-in RAM of the microcomputer, and 1 The processing for gaze detection is executed every time a line is read.

From step (# 101) to step (# 1)
The content executed in the double loop processing of 03) is an operation of updating the stored image signal data of the past five lines in order to read a new image signal of one line. That is, among the array variables IM (i, k), IM (i, 0) [i
= 0 to 149] is the oldest and IM (i, 4) [i
= 0 to 149] represents the most recent one-line image data, and the data is updated as follows to obtain a new one-line image signal as IM (i, 4) “i = 0 to 149]. Prepare to be stored in.

IM (i, 0) ← IM (i, 1) IM (i, 1) ← IM (i, 2) IM (i, 2) ← IM (i, 3) IM (i, 3) ← IM (I, 4) [i = 0 to 149]

When the rope processing for updating the data in steps (# 101) to (# 103) is completed, the loop processing in the next step (# 104) is executed.

In the loop processing of step (# 104),
The image signals for one line (150 pixels) in the horizontal direction (X-axis) of the area sensor are sequentially stored in the RAM while performing A / D conversion, and the minimum value of the image signal is detected.

In the first step (# 105) in the loop of step (# 104), the microcomputer MC
The digital value ADC obtained by A / D converting the image signal is taken out from the built-in A / D converter of U109, and the value is temporarily stored in the conversion EYEDT. Then, in the next step (# 106), the value of EYEDT is stored in the array variable IM.
(I, 4). The variable I is counted up from 0 to 149 in the outer loop processing step (# 104).

Steps (# 107) and (# 108) are processing for detecting the minimum value of the image signal. The variable EYEMIN is a variable that holds the minimum value of the image signal, and is set in step (# 10).
In 7), if EYEDT is smaller than EYEMIN, the process branches to step (# 108), and EYEMIN
Is updated with this small EYEDT value.

When the loop processing of steps (# 104) to (# 108) is completed and the storage of a new one-line image signal and the detection of the minimum value are completed, the next step (# 109)
Returns the subroutine "read one line".

Returning to the flowchart of FIG. 6, when the subroutine "read one line" of step (# 007) is completed, the process proceeds to the next step (# 008), and the loop of the outer loop processing step (# 006) Variable J is 5
Check to see if this is the case.

The loop variable J is set in the vertical direction (Y
Axis), and in this embodiment, the number of pixels of the area sensor is (150 × 100).
J is counted up from 0 to 99.

If the loop variable J is 5 or more in step (# 008), the flow branches to step (# 009). This is because when the number of lines of the read image signal becomes 5 or more,
This is because processing in the vertical direction (Y axis) of the area sensor can be performed.

In the step (# 009) ahead of the branch, the subroutine "P image detection" is executed.

The subroutine "detection of P image" is a process for detecting the position of the corneal reflection image (P image) described above, and is executed every time one line in the horizontal direction (X axis) of the area sensor is read. . The flowchart is shown in FIG.

When the subroutine "P image detection" is called, the process proceeds to step (# 20) through step (# 200).
The loop processing of 1) is executed. In the loop processing, the position of the P image in the image data (stored in the array variable IM (i, k)) is searched, and if found, the position on the area sensor is stored. In this embodiment, since two P images are generated,
Two pieces of position information are stored.

In the first step (# 202) in the loop, it is determined whether or not the image data at a predetermined position satisfies the condition as a P image. The conditions are as follows.

"P image condition" in step (# 202) IM (1,2)> C1 and IM (I, 1)> C2 and IM (I, 3)> C2 and IM (I-1,2)> C2 and IM (I + 1,2)> C2

Here, C1 and C2 are threshold constants, and C1
≧ C2, for example, C1 = 230, C2 =
200. The variable I is a loop variable of the loop processing, and represents the position of the area sensor in the horizontal direction (X axis).

The above conditions are based on the fact that the P image is like a spot image as described with reference to FIG.
It is defined in both directions of the vertical direction (X / Y axis). When this condition is satisfied, it is considered that a P image exists at the position (I, 2).

As described above, the array variable IM (i, k) is updated every time one line is read in the horizontal direction (X axis) of the area sensor, and the vertical (Y axis) position J line is set to IM.
(I, 4) Stored in “i = 0 to 149.” Therefore, the address (1, 2) for the variable IM is the position (I, J−2) on the area sensor.

In step (# 202), if there is image data that satisfies the condition of the P image, step (# 203)
Branching to the following, if not, the outer loop variable I is counted up.

The process after step (# 203) is a process for determining the existing range of two P images (the range [IP1 to IP2] in the X-axis direction and the range [JP1 to JP2] in the Y-axis direction).

First, in step (# 203), a variable I and a variable IP representing the position of the area sensor in the horizontal direction (X axis) are set.
1 and if I <IP1, step (# 204)
Branch to That is, if the position of the variable I is on the left side of the P image position IP1 on the left side in the horizontal direction in the existing range of the P image, IP1 is to be rewritten.

In step (# 204), the value of the variable I is stored in the variable IP1, and the vertical position (J-
2) is stored in the variable JP1.

In steps (# 205) and (# 206), the P image position IP2 on the right side in the horizontal direction in the P image existence range and the update of JP2 representing the vertical position are determined.

As described above, step (# 201)
In the loop processing, the position I in the horizontal direction (X axis) is changed from 0 to 14
When the processing of one line up to 9 is completed, the process proceeds to the next step (# 207).

In step (# 207), the variables XP1, XP2, YP1, and YP2 to be referred to in the subsequent processing are calculated as shown in the figure.

The meanings of these variables will be described in detail in the description of FIG. 12, but in brief, when detecting the center of the pupil, false pupil edge information generated around the P image position is excluded. Is what you use for

When the processing in step (# 207) is completed, the subroutine "P image detection" is returned in the next step (# 208).

Returning to the flowchart of FIG.

The subroutine "P" of step (# 009)
When the "image detection" is completed, a subroutine "pupil edge detection" is executed in the next step (# 010).

"Pupil edge detection" is a subroutine for detecting the position of the pupil edge (boundary between the iris and the pupil) in the eyeball reflection image, and its flowchart is shown in FIG.

When the subroutine "pupil edge detection" is called, a loop process of the next step (# 301) is executed via step (# 300). Step (# 301) is the same as step (# 201) in FIG.
This is a loop process in which the variable I representing the position of the area sensor in the horizontal direction (X axis) is used as a loop variable.

In the loop process of step (# 301), it is searched whether or not the image data has a feature representing the edge of the pupil, and if there is, the position information is stored.
The pupil edge position information is represented by an array variable EDGDT (m, n).
Is stored in

The data format of the array variable EDGDT (m, n) is set as follows.

EDGDT (m, 1)... Luminance of m-th edge point EDGDT (m, 2). X-axis coordinate of m-th edge point EDGDT (m, 3)... Y-axis coordinate of m-th edge point

M is the order of the edge points found in the course of the sequential processing of pupil edge detection. Therefore, if M edges are detected, the capacity of the array variable EDGDT is [M × 3].
You need to buy. In the flowchart, the number of detected edges is counted by the variable EDGCNT.

The first step in the loop (# 30
In 2), it is determined whether there is an edge point detected in the past near the image data IM (I, 2). The following is a more detailed explanation.

The loop variable I of the outer loop processing represents the position of the area sensor in the horizontal direction (X axis), and the address (I, 2) for the array variable IM (i, k) storing the image data is This is the point (coordinates of the pixel) for which it is to be tested whether or not it is a pupil edge. This (I,
An attempt is made to check whether or not each point adjacent to the point 2) is determined as a pupil edge in the process of the past sequential processing from the array variable EDGDT (m, n) storing the edge position information. is there.

The conditions for determination in step (# 302) are specifically described as follows.

"Judgment condition" of step (# 302) {EDGDT (m, 2), EDGDT (m, 3)} =
{(I-1), (J-2)} or = {(I-1),
(J-3)} or = {(I), (J-3)} or = {(I + 1), (J-3)} where {EDGDT (m, 2), EDGDT (m, 3)}
Exists.

However, m = 0 to (EDGCNT-1)

Since the coordinates to be tested are {(I), (J-2)}, the coordinates represent the positions of the left, upper left, upper, and upper right adjacent to the current coordinates in order. I have.

Further, EDGDT (m, 2), EDGDT
(M, 3) are the X-axis coordinates of the m-th edge point, and Y
Since the coordinates represent the axis coordinates, the above condition is to determine whether there is an edge point at the position on the left, next to the upper left, next to the upper, and next to the upper right of the current coordinates.

In step (# 302), if it is determined that there is an edge point near the coordinates (I, J-2), go to step (# 304); otherwise, go to step (# 304).
The process branches to step (# 303), and the pupil edge is determined using different conditions.

The case where there is no edge point in the vicinity will be described first.

In the step (# 303), the image data of the coordinates (I, J-2) to be examined is the pupil edge condition (the judgment condition in the step (# 303) is called "edge condition 1"). It is determined whether or not is satisfied.
The image data of the coordinates (I, J-2) is an array variable IM (I,
Note that it is stored in 2).

The judgment conditions are as follows.

“Edge condition 1” of step (♯303) 1, {IM (I-1,2) -IM (I, 2)}> C3 and ｛IM (I-2,2) -IM (I- 1,2)｝> C3 and IM (I, 2) <a 2,｛IM(I+1,2)-IM(I,2)｝> C3 and ｛IM (I + 2,2) -IM ( I + 1,2)｝> C3 and IM (I, 2) <a 3,｛IM(I,1)-IM(I,2)｝> C3 and ｛IM (I, 0) -IM (I, 1)｝> C3 and IM (I, 2) <a 4,｛IM(I,3)-IM(I,2)｝> C3 and ｛IM (I, 4) -IM (I, 3)｝> C3 and IM (I, 2) <a

If any of the above 1 to 4 is satisfied, the coordinates (I, J-2) are regarded as an edge point. Where a = EYEMIN + C4
EYEMIN is the lowest luminance value in the image data up to the current sequential processing.

The threshold values C3 and C4 are, for example, C3 =
3, C4 = 20.

The above condition is characterized in that the pupil edge (the boundary between the iris and the pupil) continuously has a predetermined luminance difference, and at the same time, the pupil has the lowest luminance in the eyeball reflection image. . The conditions 1 and 2 extract the edge in the horizontal direction (X axis) of the area sensor, and the conditions 3 and 4 extract the edge in the vertical direction (Y axis).

If the coordinates (I, J-2) are extracted as pupil edge points, the process branches from step (# 303) to step (# 305), and stores the luminance values and coordinates of the edge points.

In step (# 305), information is stored in the array variable EDGDT (m, k) for storing edge position information as follows.

EDGDT (EDGCNT, 1) ← IM
(I, 2) EDGDT (EDGCNT, 2) ← I EDGDT (EDGCNT, 3) ← J-2

IM (I, 2) is the luminance of the edge point detected at the EDGCNT-th, I is the same X coordinate, and (J-2) is the same Y coordinate.

Then, the variable EDGCNT for counting the number of detected edge points is incremented by one.

When the processing of step (# 305) is completed, the loop variable I (representing the coordinates in the horizontal direction and the X axis) of the outer loop processing is counted up, and the processing returns to step (# 3).
02) The following flowchart is executed.

The case where it is determined in step (# 302) that there is an edge point near the current coordinates (I, J-2) will be described.

In this case, the flow branches to step (# 304), and the image data of the coordinates (I, J-2) to be examined is determined by the pupil edge condition (step (# 303)) in the same manner as in step (# 303). It is determined whether the determination condition in 304) is referred to as “edge condition 2”).

Here, the "edge condition 2" is set to a condition that is less strict than the "edge condition 1". In the present embodiment, the threshold values C3 and C4 are C3 'and C4', respectively, as in the conditional expression, and are changed as follows.

C3 '= 2, C4' = 30 By setting as above, the rate of determination as an edge is higher than in "edge condition 1".

The reason for preparing two types of edge conditions in this way is that the edge points are not isolated in the first place, but continuous, and if a certain point is an edge point, its neighborhood It is also based on the viewpoint that it is likely to be an edge point.

"Edge condition 2" in step (# 304)
If it is determined in step (# 30) that the point is an edge point,
The process branches to 5) to store information on the coordinates.

As described above, when the loop variable I is 149
The loop processing of step (# 301) is executed until the processing of step (# 301) is completed, and when the processing of detecting the edge of one line in the horizontal direction (X-axis) of the area sensor is completed, the process proceeds to step (# 306) and the subroutine "pupil" Edge detection "is returned.

Returning to the description of FIG.

When the subroutine "pupil edge detection" in step (# 010) is completed, the loop variable J (representing the vertical and Y-axis coordinates of the area sensor) in the outer loop processing step (# 006) is counted up. And J is 9
Until the processing reaches 9, the processing after step (# 007) is executed again.

When the loop variable J becomes 99 and reading and processing of all pixels of the area sensor are completed, the step (#
006) to step (# 011).

In steps (# 011) to (# 013), the center coordinates of the pupil and the line of sight are detected from the P image position and the pupil edge information detected in the loop processing in step (# 006).

First, in step (# 011), a subroutine "setting of estimated pupil range" is called.

The pupil edge points detected in the subroutine "pupil edge detection" in step (# 010) include those other than the edge points that actually represent the pupil circle (the circle formed by the boundary between the iris and the pupil). Also, false edge points generated by various noises are included.

"Setting the estimated pupil range" is a subroutine for limiting the coordinate range of a probable edge point based on the P image position information in order to eliminate the false edge point. It is shown in FIG.

When the subroutine "set pupil estimation range" is called, step (# 401) is executed after step (# 400).

In step (# 401), the P image position range described in the "P image detection" subroutine, ie, IP1 to IP2 in the horizontal direction (X axis) and JP in the vertical direction (Y axis).
Using the information of 1 to JP2, the pupil circle coordinate range IS1,
IS2, JS1, and JS2 are calculated according to the following equations.

IS1 ← IP1-20 IS2 ← IP2 + 20 JS1 ← (JP1 + JP2) / 2-20 JS2 ← (JP1 + JP2) / 2 + 40

The pupil edge points that are likely to be in the range IS1 to IS2 in the horizontal direction (X axis) of the area sensor and in the vertical direction (Y
It is set that the point exists in the range JS1 to JS2 of (axis).

In the optical system of this embodiment, as shown in FIG. 5A, the two P images always exist in the upper part of the pupil circle. I do.

After the calculation in step (# 401), the process proceeds to step (# 402), and returns to the subroutine "set pupil estimation range".

Returning to FIG. 6, the next step (# 012)
Subroutine "detection of pupil center" is called.

“Detection of pupil center” is based on the coordinates of the pupil circle (center coordinates and size) based on the coordinates of the likely pupil edge points.
Is shown in FIGS. 11 to 13.

To estimate the shape of the pupil circle, the "least square method"
Is used. The concept is described above.

As is well known, the formula of the circle is (x−a) ² + (y−) where (a, b) is the central coordinate and c is the radius.
b) ² = c ² (10)

A plurality of observation points (x1, y1), (x2, y
2) Consider determining a, b, and c from (xn, yn) such that the error amount ER of the following equation is minimized.

ER = Σ [(xi−a) ² + (yi−b) ²
−c ² ] ² (11) ER is the sum of squares of the distance (error) in the normal direction of each observation point and the circle determined by a, b, and c, and minimizes this.

ER is partially differentiated with respect to a, b, and c, and is set to 0.

[0136]

[Outside 2] However, i = 1 to n.

From equation (14),

[0138]

[Outside 3] Equation (15) Equation (13), is substituted into (14), wherein, X1 = Σxi, X2 = Σxi 2, X3 = Σxi 3 ... (16) ~ (18) Y1 = Σyi, Y2 = Σyi 2, ^{Y3 = Σyi 3 ... (19)} ~ (21) Z1 = Σxiyi, Z2 = Σxi 2 yi, Z3 = Σxiyi 2 ... (22) ~ (2 4) Distant ^{further, V1 = X2-X1 2 /} n ... ( 25) V2 = Y2-Y1 ² / n (26) W1 = X3 + Z3 (27) W2 = Y3 + Z3 (28) W3 = (X2 + Y2) / n (29) W4 = Z1-X1Y1 / n (30) W5 = (Z1-2 · X1Y1 / n) Z1 (31) W6 = X1Y2 (32) W7 = X2Y1 (33) By rearranging, the center coordinates a and b of the circle are

[0139]

[Outside 4] Is calculated.

Although there is no direct relation to the calculation of the line of sight (point of gaze), the radius c is c = [W3-2 (aX1 + bY
1) / n + a ² + b ² ] ^1/2 (36)

In the embodiment of the present invention, the error amount ER is used for determining the reliability of pupil center detection, and ER is given by the following equation.

ER = X4-4aX3 + 2 (2a ² + d)
X2-4adX1 + Y4-4bY3 + 2 (2b ² + d)
Y2-4bdY1 + 2 (Z4-2aZ3-2bZ2 + 4
abZ1) + d ² n (37) where X4 = {xi ⁴ ...
(38) Y4 = Σyi ⁴ (39) Z4 = Σxi ² yi ² (40) d = a ² + b ² -c ² (41)

Now, the flowcharts of FIGS. 11 to 13 will be described according to the above numerical calculations.

When the subroutine "pupil center detection" is called, the process proceeds to step (# 500), and then to step (# 500).
The subroutine 501) "Estimation of least square of circle" is called.

The “least-squares estimation of a circle” is given by the following equation.
This is a subroutine for calculating the center coordinates (a, b) of the pupil circle and the error amount ER, and its flowchart is shown in FIG. In this subroutine, the minimum luminance value is further reviewed, and a false pupil edge is eliminated from the P image.

When the subroutine "least-squares estimation of circle" is called, the process proceeds to step (# 601) via step (# 600).

In step (# 601), the minimum 2
The work variables of the power estimation formula are being initialized.

The next step (# 602) is a loop process in which the variable L is used as a loop variable, and is a part for performing the first half of the calculation of the least squares method based on the stored pupil edge information.

Now, as a pupil edge point, (EDGCN
T-1) pieces of information are stored in the array variable EDGDT. The loop variable L indicates the order of storage.

The first step in the loop processing (# 60)
In 3), the luminance value EDGDT (L,
1) is compared with (EYEMIN + C5), and if the luminance value is larger, the process branches to terminate the processing of the current loop variable L.

In this embodiment, the sequential processing is performed while reading the photoelectric conversion signal of the area sensor.
The lowest luminance value used in the edge point detection part is also the lowest luminance value up to that point. Therefore, the point detected as an edge point is not actually determined based on the true lowest luminance value, and may include a point that is not actually suitable as an edge point. So, the purpose of this step is based on the final determined minimum brightness value,
The determination of the lowest luminance is performed again to eliminate points that are not suitable as pupil edges.

As the threshold value C5, for example, C5 = 2
0.

If it is determined in step (# 603) that the luminance value is small, the process proceeds to step (# 604), in which the horizontal (X-axis) coordinate and the vertical (Y-axis) coordinate are each set as a variable X. , Y temporarily.

In the next step (# 605), the horizontal coordinate X of the L-th edge point falls within the horizontal range IS1-IS.
2 is determined. IS1 and IS2 are values obtained in the subroutine "set pupil estimation range". Edge points that do not fall within this range are branched so as not to be recognized as pupil edge points, and the processing of the current loop variable L ends. I do.

In the next step (# 606), the same determination is made in the vertical direction.

If the L-th edge point exists in the pupil estimation range, the flow shifts to step (# 607).

Steps (# 607) and (# 608)
It is determined whether the coordinates of the L-th edge point are near the P image.

XP1, XP2, YP1, and YP2 are values determined in the subroutine "P image detection", and the coordinates of the edge point are in the horizontal range XP1 to XP2 and the vertical range Y
If it is in P1 to YP2, the process branches to end the processing of the current loop variable L. This is because, in the optical system of the present embodiment, two P images are present in the upper part of the pupil circle, so that the "tail" portion of the P image having a spot-like shape is formed. This is because it is easy to conform to the above-described pupil edge condition, and it is excluded that the pupil edge is detected as a false pupil edge.

The above steps (# 603) to (# 60)
The coordinate information of the edge point that has passed the determination of 8) is used for the calculation of the least squares method in step (# 609).

In the calculation in step (# 609), the above-described equations (16) to (24) and (38) to (40) are executed, and the number N of edge points used in the calculation is counted up.

In the loop processing of step (# 602),
When the processing for all the stored edge points (EDGCNT-1) is completed, the process proceeds to step (# 610).

In step (# 610), equations (25) to (25)
(35), (37) to (41) are calculated, and the center coordinates (a, b) of the pupil circle and the error amount ER are obtained.

Then, the flow shifts to the next step (# 611) to return the subroutine "least-squares estimation of circle".

Returning to FIG. 11, when the subroutine "least-squares estimation of circle" of step (# 501) is completed, the process proceeds to the next step (# 502).

In step (# 502), the number N of data used for estimating a circle is compared with a threshold value NTHR, and N <
In the case of NTHR, since the number of data is small, it is considered that the reliability of the result is low, and the flow branches to step (# 512) to determine that the detection has failed.

As NTHR, for example, NTHR = 3
0.

In step (# 502), N ≧ NTHR
Then, in the next step (# 503), the error amount ER is compared with the threshold value ERTHR.

If ER <ERTHR, the error is small,
Assuming that the detection result is sufficiently reliable, the flow branches to step (# 514), and it is assumed that the detection is successful.

The threshold value ERTHR is, for example, E
RTHR = 10000.

In step (# 503), ER ≧ E
In the case of RTHR, it is determined that the error is too large although the number of data is sufficient, and the recalculation after step (# 504) is performed. It is considered that the cause of the increase in the error is that false edge points other than the pupil circle were included in the calculation, and among the coordinates of each edge point, the edge point at the coordinates of the end in the vertical / horizontal direction was calculated. And check whether the error decreases.

In the step (# 504), a subroutine "least squares estimation recalculation 1 of circle" is called.

The "least-squares estimation of circle recalculation 1" means that among the edge points used in the calculation of the least-square estimation, the edge point (one-fifth of the whole) at the upper part in the vertical direction of the area sensor is determined. This is a subroutine for excluding and calculating the least squares estimation again, and its flowchart is shown in FIG.

When the subroutine “least-squares estimation of circle, recalculation 1” is called, the program goes through step (# 700)
In the next step (# 701), variables are stored as shown in the figure.

The variables XS1 to ZS4 are stored in the step (# 50
The value of the corresponding work variable when all the edge points calculated in 1) are used is stored. Then, the number of edge points to be excluded is stored in the variable M as 1/5 of the number N of all edge points.

In the next step (# 702), the work for calculation is initialized as in step (# 601), and the routine goes to step (# 703).

Step (# 703) is replaced with step (# 60)
This is a loop process similar to 2), in which the least squares method of an edge point to be excluded in this loop is calculated.

In the embodiment of the present invention, since the area sensor is read from the upper part in the vertical direction, the array variables EDGDT (m, k) storing the edge information are sequentially stored from the upper edge in the vertical direction. Is stored. Therefore, E
If m of DGDT (m, k) is counted up from 0, it can be extracted from an edge point in the vertical direction.

In the first step (# 704) in the loop of step (# 703), it is determined whether or not the edge point (X, Y) is valid as a pupil edge. ) To (# 608).

If it is determined that the pupil edge point is valid, the process proceeds to step (# 705), and the same calculation as in step (# 609) is executed.

Then, in the next step (# 706),
The newly calculated number N of the edge points is compared with the number M of the edge points to be excluded. When the calculation of the M points is completed, the process branches and the loop processing of the outer step (# 703) is stopped. M
If the number has not been reached, the loop variable L is counted up, and the process of step (# 704) is continued again.

When the M calculations are completed, step (# 70)
8), the center (a, b) of the pupil circle and the error E
Recalculate R '. The recalculation formula is as follows.

X1 = X1S-X1 (16 ') X2 = X2S-X2 (17') X3 = X3S-X3 (18 ') Y1 = Y1S-Y1 (19') Y2 = Y2S-Y2 ( 20 ') Y3 = Y3S-Y3 ... (21') Z1 = Z1S-Z1 ... (22 ') Z2 = Z2S-Z2 ... (23') Z3 = Z3S-Z3 ... (24 ') X4 = X4S-X4 ... ( 38 ') Y4 = Y4S-Y4 ... (39') Z4 = Z4S-Z4 ... (40 ')

The equations (25) to (35) and (37)
Recalculating (41) gives a new pupil center (a, b)
And the error amount ER 'can be obtained. Expressions (16) to (4)
Since (0) is originally a sequential format, it is not necessary to recalculate all data again, but it is sufficient to calculate addition (or power addition) of data to be excluded and subtract from the original value.

After the recalculation is completed, step (# 70)
The process proceeds to 9) and returns the subroutine "least squares estimation recalculation of circle 1".

Returning to FIG. 11, when step (# 504) is completed, the process proceeds to step (# 505), where the recalculated error amount ER 'is compared with the threshold value ERTHR. E
If R 'is small, it is determined that the exclusion operation has been effective, and the flow branches to step (# 514) to determine that the detection was successful.

If the error amount ER 'is still large, the flow shifts to step (# 506) to call another subroutine "recalculation of least squares of circle, recalculation 2".

The “least-squares estimation of circle, recalculation 2” means that among the edge points used in the calculation of the least-square estimation, the edge point (one-fifth of the whole) located below the area sensor in the vertical direction is determined. This is a subroutine for excluding and calculating the least squares estimation again, and its flowchart is shown in FIG.

“Recalculation 2” is almost the same as “Recalculation 1”. However, unlike “Recalculation 1”, step (# 71)
In 2), the loop variable L is counted down from (EDGCNT-1). The rest is exactly the same as “recalculation 1”, and thus the description is omitted.

Returning to FIG. 11, the description will be continued.

Upon completion of the subroutine "recalculation of least squares of circle, recalculation 2" in step (# 506), the flow advances to step (# 507) to compare recalculated error amount ER 'with threshold value ERYHR. If ER 'is small, it is determined that the exclusion operation has been effective, and step (# 51)
The process branches to 4), and the detection is regarded as successful.

If the error amount ER 'is still large, the flow shifts to step (# 508), and calls another subroutine "least squares estimation recalculation 3 of circle".

In the “least-squares estimation of circle, recalculation 3”, of the edge points used in the calculation of the least-square estimation, the edge point located at the left side in the horizontal direction of the area sensor (5/5 of the whole) 1)
Is a subroutine for calculating the least squares estimation again, excluding the subroutine, and its flowchart is shown in FIG.

When the subroutine "recalculation 3" is called, the process proceeds to step (# 720) and then to step (# 72).
In 1), the array variable EDGD storing the edge information
Rearrange T (m, k).

As described above, EDGDT (m,
k) is stored in order from the vertical edge points of the area sensor, so that the data stored in the EDGDT needs to be rearranged in order to perform processing while paying attention to the horizontal direction.

Since a value in the horizontal direction (X-axis coordinate) of the edge point is stored in EDGDT (m, 2), if a known "sort operation" is performed on this value, the EDGDT will have a horizontal value. Edge information in the order from the left in the direction can be stored again.

When the rearrangement is executed, the step (# 70)
By branching to 2) and performing the same processing as “recalculation 1”, recalculation can be performed excluding the left and right edge points of the area sensor in the horizontal direction.

Returning again to FIG. 11, step (# 50
When the subroutine of (8) "Least square estimation of circle, recalculation 3" is completed, the process proceeds to step (# 509) to compare the recalculated error amount ER 'with the threshold value ERTHR. E
If R 'is small, it is determined that the exclusion operation has been effective, and the flow branches to step (# 514) to determine that the detection has been successful.

If the error amount ER 'is still large, the flow shifts to step (# 510) to call another subroutine "least squares estimation recalculation 4 of circle".

In the “least-squares estimation of circle, recalculation 4”, among the edge points used in the calculation of the least-square estimation, the edge point located on the right side in the horizontal direction of the area sensor (5/5 of the whole) 1)
Is a subroutine for calculating the least-squares estimation again, and its flowchart is shown in FIG.

Now, the array variables EDGDT (m, k) include:
Since the edge points in the horizontal direction are stored from the left, if the edge points are to be excluded in order from the right, EDGDT
(M, k) may be handled in the same manner as “recalculation 2”. Therefore, if the subroutine "recalculation 4" is called, the process immediately branches to step (# 711), and the same processing as "recalculation 2" is performed.

Returning to FIG. 11, the description will be continued.

When the subroutine "least-squares estimation of circle, recalculation 4" in step (# 510) is completed, the process proceeds to step (# 511) to compare the recalculated error amount ER 'with the threshold value ERTHR. If ER 'is small, it is determined that the exclusion operation has been effective, and step (# 51)
The process branches to 4), and the detection is regarded as successful.

If the error amount ER 'is still large, the flow shifts to step (# 512), assuming that the above-mentioned operation did not work effectively, and shifts to step (# 512), and the detection fails.

Step (# 512) or (# 51)
When the pupil center detection is finally determined in 4), the subroutine "pupil center detection" is returned in step (# 513) or (# 515).

FIG. 14 shows an example of the least squares estimation according to the embodiment of the present invention.

In the figure, ● represents one edge point, and the pupil circle is estimated based on these edge points.

Returning to the description of FIG.

When the “detection of the pupil center” in step (# 012) is completed, the process proceeds to step (# 013), and calls a subroutine “detection of line of sight”.

"Detection of line of sight" is a subroutine for detecting a line of sight (point of gaze) from the position of the P image and the center position of the pupil circle detected in the processing so far.

Basically, the rotation angle θ of the optical axis of the eyeball may be calculated according to the equation (2) in the same manner as in the above-mentioned known example.

In the embodiment of the present invention, the center of the pupil is detected two-dimensionally in the horizontal direction (X-axis) and in the vertical direction (Y-axis). The direction of the line of sight can be detected in the same way as the detection in the lateral direction.

When the detection of the line of sight is completed, step (# 0)
The process proceeds to 14), and a series of processing ends.

[0213]

As described above, by adopting the configuration of the present invention, the pupil edge points are reviewed based on the finally held minimum value, and the pupil edge points that have been reviewed are used as a basis. Thus, since the gaze is detected, an accurate pupil edge can be detected, and an accurate gaze direction can be detected.

[Brief description of the drawings]

FIG. 1 is an AF equipped with a gaze detection device according to the present invention.
(Auto focus) Schematic diagram of a camera.

FIG. 2 is a perspective view of a focus detection system.

FIG. 3 is a perspective view of a visual line detection system according to the present invention.

FIG. 4 is a principle diagram for explaining a principle for detecting a line of sight according to the present invention.

FIG. 5 is a diagram showing a light intensity distribution of an area sensor according to the present invention.

FIG. 6 is a flowchart for detecting a line of sight according to the present invention.

FIG. 7 is a flowchart showing a reading operation of one line (X-axis direction) of the area sensor.

FIG. 8 is a flowchart for detecting a P image.

FIG. 9 is a flowchart for detecting a pupil edge point.

FIG. 10 is a flowchart for setting an estimated range of a pupil circle.

FIG. 11 is a flowchart for detecting a pupil circle.

FIG. 12 is a flowchart for selecting an appropriate point for deriving a pupil circle from a plurality of points of a detected pupil edge.

FIG. 13 is a flowchart for eliminating pupil edge points considered to be inappropriate in calculating a pupil circle.

FIG. 14 is a diagram showing a plurality of edge points and a pupil circle calculated from the edge points according to the present invention.

FIG. 15 is a view for explaining the principle for detecting a line of sight.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Eyepiece connection part 4 Light receiving lens 5a, 5b Infrared light emitting diode 5c, 5d as illuminating means for generating a corneal reflection image Infrared light emitting diode as illuminating means for detecting iris information 6 Photoelectric element array (area sensor) 21 Cornea Reference Signs List 22 eyeball 23 iris 24 pupil 101 photographing lens 102 flip-up mirror 103 display element 104 focus plate 105 condenser lens 106 pentatach prism

Claims (2)

(57) [Claims] 1. A plurality of lines composed of a plurality of pixels are arranged side by side.
Are arranged, a light receiving means for outputting a photoelectric conversion signal of each pixel for each line by receiving eyeball image, a predetermined number of lines
After the photoelectric conversion signal is output, the photoelectric
Each time the conversion signal is output, it is output from a predetermined number of lines.
First extracting the minimum value holding means for holding a minimum value in the photoelectric conversion signals, a pixel that outputs the pupil edge Rashiki photoelectric conversion signal of the eye by using the minimum value held in said minimum value holding means And a pixel outputting a pupil edge-like photoelectric conversion signal using the final lowest value held in the lowest value holding means after all the photoelectric conversion signals are output, by the first extraction means. A second extraction unit that extracts from the pixels extracted in step 2, a Purkinje image detection unit that detects a pixel position at which a signal relating to a Purkinje image is output from the photoelectric conversion signal, and a second extraction unit. A gaze detection device comprising: a detection unit configured to detect a gaze direction based on a position of a pixel to be extracted and a pixel position detected by the Purkinje image detection unit. 2. A plurality of lines composed of a plurality of pixels are arranged side by side.
Are arranged, a light receiving means for outputting a photoelectric conversion signal of each pixel for each line by receiving eyeball image, a predetermined number of lines
After the photoelectric conversion signal is output, the photoelectric
Each time the conversion signal is output, it is output from a predetermined number of lines.
The lowest luminance value holding means for holding the lowest luminance value of the photoelectric conversion signals to be output, and a pixel for outputting a photoelectric conversion signal like a pupil edge of an eyeball using the lowest luminance value held in the lowest luminance value holding means. First extraction means for extracting, storage means for storing the brightness and position of the pixel extracted by the first extraction means, and the lowest brightness value holding means after all the photoelectric conversion signals are output. A second extraction means for extracting a pixel which outputs a pupil edge-like photoelectric conversion signal by using the final lowest luminance value among the pixels stored in the storage means; A Purkinje image detecting means for detecting a pixel position for outputting a signal relating to an image; a pixel position extracted by the second extracting means; and a pixel position detected by the Purkinje image detecting means. Visual axis detecting apparatus characterized by having a detecting means for detecting a sight line direction are.