JP2952072B2 - Optical line-of-sight detecting device and optical apparatus having line-of-sight detecting means - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight detecting device and an optical apparatus having a line-of-sight detecting means.

[0002]

2. Description of the Related Art In a video camera or the like that converts an optical image into an electric image signal using an area sensor, the degree of focus of an optical system and the amount of exposure can be known from the image signal.
The video signal of a partial area of the shooting screen is extracted according to the movement of the subject, and automatic focus adjustment and automatic exposure adjustment are performed. A so-called tracking AF / AE function is presented. FIG. 6 shows the outline.

In FIG. 6, reference numeral 16 denotes a photographing lens, 17 denotes an aperture, 15 denotes an image sensor having a two-dimensional image sensor such as a CCD image sensor, 14 denotes a gate circuit, and 18 denotes an A / D.
M. P. It has an interface function with U. 1 is a microprocessing unit (M
PU), 11 is a lens drive circuit, and 13 is an aperture drive circuit.

When a photographer places a subject to be photographed at the center of the screen and switches to tracking AF, a distance measuring zone corresponding to the depth of field appears and moves in accordance with the movement of the subject. The movement mechanism divides the screen into 16 × 16 = 256 fields (areas) (this field is a basic area for calculating an ES signal), and converts the field in the ranging frame where the ES signal has a peak to M. . P. U1 detects the arithmetic processing and moves the distance measurement frame around the position to track the movement of the subject. That is, as shown in the flowchart of FIG. 7, when the mode is switched to tracking AF (step 1),
M. P. U1 sets a ranging zone corresponding to the depth of field (the center is initially the center of the screen), outputs the information to the gate 14, and outputs the information to the A / D converter 19 from the gate. A range (zone) is set (step 2). A / D converter 19 converts the signal output from the inside of the distance measuring frame of the image sensor output through the gate into a digital signal. P. Read into U1 (step 3). M. P. U uses this signal to perform AF calculation, AE
Calculation is performed to calculate the lens drive amount and the aperture drive amount, and output the values to the lens drive circuit 11 and the aperture drive circuit 13. Both drive circuits drive the lens aperture based on the value (step 4).

At the same time, M. P. U1 calculates an ES signal (a signal indicating the sharpness and sharpness of the edge of the subject) of a 256-divided field in the distance measurement frame for each field, and a point at which the value is maximum, that is, a distance measurement Find the point with the highest contrast in the frame. Then, the distance measuring frame is moved so that the point becomes the center, the information is output to the gate, and the range of the signal output from the gate is set (step 5). Hereinafter, this operation is repeated if the tracking operation switch is ON (step 6), and the tracking operation is performed.

A silver halide still camera has been proposed which has a plurality of distance measuring points and has an AF function for tracking a moving subject.

[0007]

However, in the above-mentioned conventional example, since the portion having the highest contrast among the subjects in the zone is tracked, "when the background has the highest luminance", Is dark, "if multiple subjects with the same brightness are moving randomly in the screen,""if the contrast of the subjects is low," or "the subject being tracked crosses another subject at the same distance." (When another subject has higher brightness) "and the like, there is a drawback that tracking cannot be performed well.

[0008]

According to a first aspect of the present invention, there is provided a line-of-sight detecting apparatus, wherein a line-of-sight detecting means for continuously detecting a line-of-sight position, and a first line-of-sight position detected by the line-of-sight detecting means. Calculating means for calculating an amount of change from the second gaze position detected thereafter, and determining whether the amount of change exceeds a predetermined reference value set in consideration of the phenomenon of eye movement fixation And determining the first line-of-sight position as an effective line-of-sight position when the amount of change does not exceed the predetermined reference value, and when the amount of change exceeds the predetermined reference value, Processing means for processing the second gaze position as an effective gaze position. The invention described in claim 2 includes a line-of-sight detection unit that continuously detects a line-of-sight position, and selects one of a plurality of areas set in the screen based on an output of the line-of-sight detection unit. Calculating means for calculating an amount of change between a first line-of-sight position detected by the line-of-sight detection unit and a second line-of-sight position detected thereafter, and wherein the amount of change is fixation of the eyeball. Judging means for judging whether or not a predetermined reference value set in consideration of the fine movement phenomenon, and selecting an area based on a first line-of-sight position when the amount of change does not exceed the predetermined reference value When the change amount exceeds the predetermined reference value, there is provided a selecting means for selecting an area based on a second line-of-sight position.

[0009]

FIG. 1 schematically shows the structure of an embodiment of the present invention.

FIG. 1 is a block diagram showing an embodiment of the present invention.
Is M. P. U and 2 are memories, 3 is an interface circuit having an A / D conversion function, and 4 is a sensor unit including an image sensor for observing a photographer's eyeball and its driving circuit. 7 is an infrared light emitting diode as a light emitting element, 5 is a light emission control circuit, 6 is a position sensor for detecting the vertical and horizontal positions of the camera, 10 is an AF sensor, 11 is a lens drive unit, 12 is a photometric sensor, 13 is An aperture drive unit.

In this embodiment, the line of sight of the photographer is obtained by calculating the image signal from the image sensor. The principle is as follows. When the photographer's eye is irradiated with parallel light (or divergent light), this light is reflected on the front surface of the cornea, and a virtual image of the light emitting diode is generated. This virtual image is called a Purkinje image, and its generation position coincides with the center of the pupil when the rotation angle of the eyeball is zero, and as the eyeball rotates, the distance between the Purkinje image and the center of the pupil becomes
It spreads in a form almost proportional to the sine of the rotation angle. Therefore, if the position of the Purkinje image, the position of the center of the pupil, and the distance between the positions are calculated from the image signal on the image sensor, the rotation angle of the eyeball and the viewpoint of the photographer can be known. (Due to the optical characteristics of the camera's viewfinder, the viewpoint on the focus plate is the same if the eyeball's rotation angle is the same, even if the head moves with respect to the camera.) Next, the above principle was used. The operation of this embodiment will be described with reference to the flowchart of FIG.

When the start of line-of-sight detection is instructed by turning on a main switch (not shown) of the camera or the like, M.P.
P. U1 transfers control to a line-of-sight detection routine (step 10).

When the routine for the line of sight detection is entered, an initialization process is performed first, and the values of all variables relating to the line of sight detection are set to zero (step 11). Then, information on the position of the camera (vertical position or horizontal position) at that time is received from the position sensor 6, and the light emission control circuit 5 receives an infrared light emitting diode (iRED).
The user sets which of the groups 7 emits light. At the same time, M.
P. U <b> 1 supplies an integration signal to the image sensor driving circuit 4 via the interface circuit 3 and a light emission control signal synchronized with the integration signal to the light emission control circuit 5. This allows
The infrared light emitting diode corresponding to the position of the camera at that time emits light in synchronization with the accumulation of the image sensor (step 12). Next, the image of the anterior segment of the eyeball on which the Purkinje image formed on the image sensor 4 has occurred is read via the interface circuit 3 (step 13). By processing this image, the position P of the Purkinje image and at least three pupil limbs (so-called pupil edges)
The positions D ₁ , D ₂ and D ₃ are detected (step 14). From the detected amounts, the horizontal and vertical rotation angles θ _H and θ _V of the eyeball are calculated. When the rotation angle of the eyeball is calculated, individual differences such as visual axis correction are corrected, and the viewpoint on the focus plate of the photographer is obtained, and the gazing point is obtained based on the dwell time or the like (step 15). Further, when an AF instruction is given from this point of gaze, an area for focus detection, a so-called AF area, and an AE area for exposure control are determined (step 16).

In general, a camera using a silver halide film requires the same number of distance measuring units as the number of AF points is increased. Therefore, the number of AF points is limited due to cost and space disadvantages.
Therefore, there is a possibility that there is no AF point corresponding to the viewpoint position on the focus plate. Therefore, interpolation processing is performed as follows.

The first method is a method in which, when there is no AF area corresponding to the viewpoint on the focus board, the AF point closest to the gazing point on the focus board is set as the AF point of the viewpoint. For example, when the AF point is set as shown in FIG. 3A, the coordinates (x,
y) and the center coordinates of the AF point 7 point _{A~G (x A, y A)} (x B, y B) (x C, y C) (x D, y D)
The distance 1 of (x _E , y _E ) (x _F , y _F ) (x _G , y _G ) is obtained by the following equation, and the distance that minimizes the value is determined by the gaze position A
F point.

[0016]

[Outside 1]

The second method is to set an AF point and an area for selecting the AF point in advance. For example, as shown in FIG. 3B, AF points A to G and an area for selecting the AF points are set.

The same can be said for the determination of the AE area. However, since the photometric sensor is often divided into areas, the second method is mainly used. That is, a photometric device that performs split photometry is provided so as to correspond to each of the seven areas in FIG.

By the way, the actual viewpoint of the photographer is not always at the subject, but fluctuates to some extent or looks at a display outside the screen. Therefore, when the viewpoint is outside the screen, it is necessary to perform processing such as excluding the object of the tracking operation or extracting the gazing point of the photographer by a known method.

Further, in this embodiment, in order to effectively perform the tracking operation, if the moving amount of the viewpoint is smaller than a certain value, the moving of the AF / AE area is not performed. The operation procedure is shown in FIG. When entering the routine for determining the AF / AE area from the gaze position, first, the information on the AF / AE area at the previous gaze position stored in the memory is read (step 21). If it is the initialized value, that is, if it is the first area determination operation (step 22), the above-described interpolation processing is performed based on the position of the gazing point at that time. To determine an area (step 25) and store the information in a memory (step 26). Then, in the case of the second time or later, the previous A
The information is compared with the information on the F / AE area, and it is determined whether the information is sufficiently separated. This is because if the coordinates of the gazing point position used for the determination are stored as the AF / AE area information, the coordinates of the viewpoint (x
_i , y _i ) and the coordinates (x _i-1 , y _i-1 ) of the previous viewpoint position

[0021]

[Outside 2] (Step 23), and this value is a certain value lcon
If was st more (step 24) currently obtained viewpoint coordinates (x _i, y _i) with, and further including performing such interpolation process described above to determine the area of a new gaze position,
The information is stored in a memory. Conversely, Δl is lcons
If it is less than t, the area information obtained last time is used without updating.

Performing such processing minimizes a detection error caused by the characteristics of human eye movements, and complicates AF /
A malfunction or inconvenience caused by switching the AE area (for example, when the viewpoint moves slightly in the middle of the two AF areas, one of the two AF areas is the main subject, and the other is the background, the lens is between the two AF areas. Inconvenience of round-trip) It is intended for deterrence.

The characteristic of the eye movement concerned here is fixation fine movement. Fixation tremor is irregular micromotion that occurs involuntarily in order to capture the image of the target object in the center of the eyeball and maintain that state. This tremor constantly fluctuates the light stimulus to the photoreceptor cells, which is the central nuisance, and has the role of not reducing the signal generation efficiency.

Therefore, when the photographer gazes at an object, this fixation slight movement occurs. The constant lconst is set so as to cover the range of the fixation fine movement (generally, 0.5 mm or less on the focus plate) so that the AF / AE area does not change due to the generated fixation fine movement.

When the photometry / switch SW1 is turned on (step 17), the AF operation and photometry are performed (steps 18 and 19). The AF operation is first performed by the M.A. P. U reads the signal from the AF sensor in the area corresponding to the gazing point and calculates the signal to obtain the lens driving amount. After that, the lens drive unit is controlled to perform focus adjustment. M. P. U1 obtains a photometric value based on the signals from the divided photometric sensors of all areas in which the area corresponding to the point of interest is weighted, and determines an exposure constant (shutter speed / aperture value...) In accordance with the designated photographing mode.

When the release button is requested by performing the second-stage operation of the release button (step 20), the aperture is driven to the calculated aperture value, the mirror is operated, the shutter is opened and closed, the film is wound, and the like. Performs a series of operations related to the release.

In the description so far, the AF operation (signal reading → calculation → lens driving) and photometry are performed almost simultaneously, but in an actual camera, the mode is set according to the set mode. That is, in the camera mode setting, photometry may be performed immediately before the release, and the exposure value may be determined based on the photometric value.

In the AF operation, only the lens drive may be performed after the release request.

As described above, this embodiment detects the movement of the line of sight of the photographer and performs AF / AF in the area where the line of sight exists.
Perform AE. This is to perform so-called automatic tracking of the subject.

(Other Embodiment) FIG. 4 shows a block diagram of the second embodiment.

1 is M. P. U, 2 are memories, 3 is an interface circuit, 4 is an image sensor for detecting a line of sight and its driving circuit, 5 is a light emission control circuit, 7 is an infrared light emitting diode, 11 is a lens driving unit, 13 is an aperture driving unit, 15 Denotes an imaging element for photographing, 16 denotes a photographing lens, 17 denotes an aperture unit, and 14 denotes a gate.

The first embodiment is a system suitable for a silver halide still camera, while the second embodiment is a system suitable for a video camcorder or the like. The principle of gaze detection uses the Purkinje image and the center of the pupil, as in the first embodiment.

FIG. 5 shows the operation procedure of this embodiment.

When the start of line-of-sight detection is instructed by turning on the line-of-sight detection request switch (for example, an automatic tracking switch) SW3 of the camera or the like, M.P. P. U1 transfers control to a line-of-sight detection routine (step 30).

When the routine enters the line-of-sight detection routine, an initialization process is first performed, and all variables relating to line-of-sight detection are set to zero (step 31). Subsequent M.P. P. U1 supplies an integration signal to the image sensor and the driving circuit 4 and a light emission control signal synchronized with the integration signal to the light emission control circuit 5 through the interface circuit 3, respectively. Thus, the infrared light emitting diode 7 emits light in synchronization with the accumulation of the image sensor 4 (step 32). Then, the image of the anterior segment of the eyeball where the Purkinje image formed on the image sensor 4 is generated is read via the interface circuit 3 (step 33).
Then, by processing this image, the horizontal and vertical rotation angles of the eyeball, and by correcting individual differences, the point of gaze on the photographer's screen is obtained (step 3).
5). Then, an area (approximated to FIG. 3B) for performing AF / AE is determined based on the gazing point (step 36),
AF / AE etc. are performed based on the information of this area (step 37).

The operations of steps 32 to 37 are repeated as long as the line-of-sight detection request continues (step 38).

In this case, for the same reason as in the first embodiment, if the viewpoint is out of the screen, it is excluded from the tracking operation.
According to Japanese Patent No. 220790, processing such as extraction of the point of gaze of the photographer is required.

In this embodiment, in order to effectively perform the tracking operation, the movement of the AF / AE area is not performed when the amount of movement of the viewpoint is smaller than a certain value as in the first embodiment. ing. FIG. 9 shows the operation procedure. When entering the routine for determining the AF / AE area from the viewpoint position, first, the viewpoint position information used for the previous AF / AE area determination stored in the memory is read (step 40). If it is an initialized value (step 4
1), the process proceeds viewpoint position at that time to the routine of the viewpoint position (x _o y _o) as an area setting after stored in the memory for the area determination (step 44). Further, in the case of the second time or later, comparison with the viewpoint position for the previous area determination is performed, and if they are sufficiently separated, that is, the current viewpoint position (x _i , y _i ) and the previous viewpoint position (x _i-1 y _i-1 )

[0039]

[Outside 3] (Step 42), and this value is a certain value lcon
If it is not less than st, the current viewpoint position (x _i , y _i ) is stored in the memory as the viewpoint position (x y _{o o} ) for area determination, and then the routine proceeds to the next routine (step 43).

[0040] The determination of the AF · AE area, using the viewpoint position for the area determine the (x _o y _o), it is determined in the following manner. First, the depth of field is determined from the focal length and aperture value of the photographing lens at that time (step 45), and the size of the area is determined according to the value (step 46). This is set to be small when the depth of field is shallow, and to be large when the depth of field is deep. Then, an area having a size corresponding to the depth of field is determined around the viewpoint position (x _o , y _o ) for area determination obtained in the previous stage, and this area is defined as an AF / AE area (step 47).

While the tracking operation is being performed, as described above,
The AF / AE area is determined, and moves in accordance with the movement of the subject (the movement of the viewpoint on the screen of the photographer). During tracking operation, M. P. U1 is AF · A determined as described above.
The E area information is output to the gate 14. As a result, the range of the video signal output from the gate 14 to the interface circuit 3 having the A / D conversion function (the area for performing AF / AE) is set.

The signal in the area of the image sensor 15 output through the gate 14 is converted into a digital signal by the interface circuit 3 and then converted to a digital signal by the M.P. P. Read into U1. M. P. U1 performs an AF operation and an AE operation using these signals, calculates a lens drive amount and an aperture drive amount, and outputs the values to the lens drive circuit 11 and the aperture drive circuit 13. Both drive circuits drive the lens diaphragm based on the value. Then, the routine enters a line-of-sight detection routine again, sets a new AF / AE area using the calculated viewpoint and depth of field on the photographer's screen, and uses the signal from the area to set a new AF / AE area. And so on.

The tracking operation is performed by repeating the above operation.

In a video camcorder, an AF operation is performed using an image signal output from an image sensor.
Since the AE calculation is performed, distance and photometry can be performed basically in any area of the entire screen, and interpolation processing of AF points and AE points as in a silver halide camera is basically unnecessary.

In the above-described embodiment, the area is not changed in the case of a small amount of movement of the line of sight such as the fixation fine movement. Conversely, the amount of movement is too large, and the eyes are moved out of the viewfinder. A similar effect can be obtained by not changing the area in the same manner even when the player is turned. In that case, it can be performed by adding a step of “Δl ≦ lmax” in addition to step 23 of FIG. 8 and step 43 of FIG. 9 and setting “lmax” to a large value here.

[0046]

As described above, in the eye gaze detecting device, the eye gaze detecting means for continuously detecting the eye gaze position;
Calculating means for calculating an amount of change between a first line-of-sight position detected by the line-of-sight detecting means and a second line-of-sight position detected after the first line-of-sight position; Determining means for determining whether the change exceeds a predetermined reference value, and processing the first line-of-sight position as a valid line-of-sight position if the amount of change does not exceed the predetermined reference value; When the amount exceeds the predetermined reference value, by having a processing unit that processes the second line-of-sight position as an effective line-of-sight position, it is possible to reduce the influence of unconscious eye movement, An accurate gaze position according to the will can be obtained. According to a second aspect of the present invention, there is provided a line-of-sight detecting means for continuously detecting a line-of-sight position, and based on an output of the line-of-sight detecting means, any one of a plurality of areas set in a screen. The first gaze position detected by the gaze detection means and the second gaze position detected thereafter
Calculating means for calculating the amount of change with respect to the line of sight position; determining means for determining whether or not the amount of change exceeds a predetermined reference value set in consideration of the phenomenon of eye fixation and fine movement; If the predetermined reference value is not exceeded, an area is selected based on the first line-of-sight position. If the amount of change exceeds the predetermined reference value, an area is selected based on the second line-of-sight position. With this configuration, it is possible to solve the problem of selecting an area based on unconscious eye movement, and to perform area selection that properly reflects the user's intention.

In this embodiment, an area for obtaining information when performing AF / AE or the like is determined based on the output of the line-of-sight detection function (the viewpoint on the screen of the photographer). By performing the movement in accordance with the output of the conventional example, it is possible to achieve a more accurate tracking operation, overcoming the drawbacks of the conventional example.

At the same time, a portion corresponding to the determined area of the AF / AE sensor or the image sensor is read out and subjected to arithmetic processing, thereby enabling a high-speed tracking operation.

[Brief description of the drawings]

FIG. 1 is a block diagram of a first embodiment of the present invention.

FIG. 2 is a flowchart of the first embodiment.

FIG. 3 is a diagram showing an AF / AE area.

FIG. 4 is a block diagram of a second embodiment.

FIG. 5 is a flowchart of a second embodiment.

FIG. 6 is a block diagram of a conventional example.

FIG. 7 is a flowchart of a conventional example.

FIG. 8 is a flowchart of the first embodiment.

FIG. 9 is a flowchart of an operation for determining an area according to the second embodiment.

[Explanation of symbols]

1 Micro Processing Unit (MP.
U) 4 Image sensor for gaze detection and its driving circuit 5 Light emission control circuit 6 Position sensor 7 Infrared light emitting diode 10 AF sensor 12 Photometry sensor 15 Image sensor for photographing 16 Photographing lens

Claims (2)

(57) [Claims] 1. A gaze detection means for continuously detecting a gaze position, and a change amount between a first gaze position detected by the gaze detection means and a second gaze position detected after the first gaze position is calculated. Calculating means for determining whether or not the amount of change exceeds a predetermined reference value set in consideration of the phenomenon of fixation of the eyeball; if the amount of change does not exceed the predetermined reference value The first
And processing means for processing the second line-of-sight position as a valid line-of-sight position when the amount of change exceeds the predetermined reference value. Eye gaze detection device. 2. An optical apparatus comprising a line-of-sight detecting means for continuously detecting a line-of-sight position, and selecting one of a plurality of areas set in a screen based on an output of the line-of-sight detecting means. Calculating means for calculating an amount of change between a first line-of-sight position detected by the line-of-sight detecting means and a second line-of-sight position detected after the first line-of-sight position; Determining means for determining whether the change exceeds a predetermined reference value, and a first means if the change amount does not exceed the predetermined reference value.
Selecting means for selecting an area based on the position of the line of sight, and selecting the area based on the position of the second line of sight when the amount of change exceeds the predetermined reference value. Optical device with means.