JP7693397B2 - Line-of-sight information acquisition device, imaging device, line-of-sight information acquisition method, program, and storage medium - Google Patents

Description

The present invention relates to a gaze information acquisition device, an imaging device, a gaze information acquisition method, a program, and a storage medium.

In recent years, imaging devices have become more automated and intelligent, and devices have been proposed that can select a position in space in the direction the photographer is looking or a position on the screen based on information about the photographer's line of sight (gaze information) as he or she looks through the viewfinder, without the need to manually input a position.

One of the causes of errors in gaze detection is physiological eye movement in humans. For example, even when the human eye is intended to be looking at one point, it will involuntarily shake slightly. This type of involuntary movement is called fixational eye movement.

Patent Document 1 proposes a configuration that determines whether the movement of the operator's eyeball is fixational eye movement or voluntary movement/jumping movement based on the distance the pupil center position moves within a specified time. Furthermore, Patent Document 1 proposes a configuration that changes the number of data points for the pupil center position based on the result of that determination, calculates the average value of the pupil center position, and detects the gaze direction based on this calculation result.

Japanese Patent Application Publication No. 11-282617

However, in the configuration of Patent Document 1, when the size or amount of movement of an object that the operator is looking at changes dynamically, the accuracy of gaze detection decreases due to fixational eye movement, resulting in unintended changes in gaze information.

The present invention aims to provide a technology that can effectively reduce changes in gaze information caused by fixational eye movement.

A first aspect of the present invention is a gaze information acquisition device having a first acquisition means for acquiring gaze information, which is information on the line of sight of the eye looking at an captured image, and a reduction means for reducing changes in the gaze information due to fixational eye movement, wherein the reduction means reduces changes in the gaze information due to fixational eye movement to a greater degree of reduction the shorter the focal length when capturing the image.

A second aspect of the present invention is an imaging device comprising an imaging means and the above-mentioned gaze information acquisition device, wherein the gaze information acquisition device controls the image captured by the imaging means to be displayed on a display unit and detects the gaze of the eye viewing the image.

A third aspect of the present invention is a gaze information acquisition method comprising an acquisition step of acquiring gaze information, which is information on the line of sight of the eye viewing an image captured, and a reduction step of reducing changes in the gaze information due to fixational eye movement, wherein in the reduction step, changes in the gaze information due to fixational eye movement are reduced to a greater degree the shorter the focal length when capturing the image.

A fourth aspect of the present invention is a program for causing a computer to function as each of the means of the above-mentioned gaze information acquisition device. A fifth aspect of the present invention is a computer-readable storage medium storing a program for causing a computer to function as each of the means of the above-mentioned gaze information acquisition device.

The present invention makes it possible to effectively reduce changes in gaze information caused by fixational eye movements.

FIG. FIG. FIG. 2 is a block diagram of the camera. FIG. 1 is a diagram for explaining the principle of a gaze detection method. FIG. 2 shows an eye image. 13 is a flowchart of a gaze detection operation. 11 is a graph showing a change over time in a detected viewpoint. 1 is a graph showing the relationship between focal length and cutoff frequency. The posterior estimated value is a diagram showing the relationship between the prior estimated value and the observed value. 13 is a flowchart showing control of fixation eye movement reduction processing. 1 is a graph showing the relationship between focal length, object distance, and cutoff frequency. FIG. 13 is an external view of another electronic device.

<<Embodiment 1>>
Hereinafter, a first embodiment of the present invention will be described with reference to the accompanying drawings.

<Description of configuration>
1(A) and 1(B) show the appearance of a camera 1 (digital still camera; interchangeable lens camera) according to the first embodiment. FIG. 1(A) is a front perspective view, and FIG. 1(B) is a rear perspective view. As shown in FIG. 1(A), the camera 1 has a photographing lens unit 1A and a camera housing 1B. A release button 5, which is an operating member that accepts photographing operations from a user (photographer), is disposed on the camera housing 1B. As shown in FIG. 1(B), an eyepiece lens 12 (eyepiece optical system) is disposed on the rear face of the camera housing 1B, through which the user peers into a display device 10 (display panel) (described below) contained within the camera housing 1B. Note that,
The eyepiece optical system may include multiple lenses. Operation members 41 to 43 that accept various operations from the user are also arranged on the back surface of the camera housing 1B. For example, the operation member 41 is a touch panel that accepts touch operations, the operation member 42 is an operation lever that can be pushed down in each direction, and the operation member 43 is a four-way key that can be pressed in each of four directions. The operation member 41 (touch panel) is equipped with a display panel such as a liquid crystal panel, and has a function of displaying an image on the display panel.

Figure 2 is a cross-sectional view of camera 1 cut along the YZ plane defined by the Y and Z axes shown in Figure 1 (A), and shows the general internal configuration of camera 1.

The photographing lens unit 1A includes two lenses 101 and 102, an aperture 111, an aperture drive unit 112, a lens drive motor 113, a lens drive member 114, a photocoupler 115, a pulse plate 116, a mount contact 117, and a focus adjustment circuit 118. The lens drive member 114 is made up of a drive gear and the like, and the photocoupler 115 detects the rotation of the pulse plate 116, which is linked to the lens drive member 114, and transmits this to the focus adjustment circuit 118. The focus adjustment circuit 118 drives the lens drive motor 113 based on information from the photocoupler 115 and information from the camera housing 1B (lens drive amount information), and moves the lens 101 to change the focus position. The mount contact 117 is an interface between the photographing lens unit 1A and the camera housing 1B. For simplicity, two lenses 101 and 102 are shown, but in reality, more than two lenses are included in the photographing lens unit 1A.

The camera housing 1B contains an image sensor 2, a CPU 3, a memory section 4, a display device 10, a display device drive circuit 11, etc. The image sensor 2 is disposed at the intended imaging surface of the photographing lens unit 1A. The CPU 3 is the central processing section of the microcomputer, and controls the entire camera 1. The memory section 4 stores images captured by the image sensor 2, etc. The display device 10 is composed of a liquid crystal or the like, and displays the captured image (subject image) etc. on the display surface of the display device 10. The display device drive circuit 11 drives the display device 10. The user can view the image (such as the image captured by the image sensor 2) displayed on the display surface of the display device 10 through the eyepiece 12.

The camera housing 1B also includes light sources 13a and 13b, a light splitter 15, a light receiving lens 16, and an ocular imaging element 17. The light sources 13a and 13b are light sources for illuminating the user's eyeball 14. The light sources 13a and 13b have been used in single-lens reflex cameras and the like to detect the line of sight (the direction of the line of sight; the direction in which the user is looking) from the relationship between the reflected image (corneal reflection image; Purkinje image) of light reflected by the cornea and the pupil. Specifically, the light sources 13a and 13b are infrared light-emitting diodes that emit infrared light that is insensitive to the user, and are arranged around the eyepiece 12. The optical image of the illuminated eyeball 14 (eyeball image; an image produced by the light emitted from the light sources 13a and 13b and reflected by the eyeball 14) passes through the eyepiece 12 and is reflected by the light splitter 15. The eyeball image is then formed by the light receiving lens 16 on the ocular imaging element 17, which is a two-dimensional array of photoelectric elements such as CCD or CMOS. The light receiving lens 16 positions the pupil of the eyeball 14 and the ocular imaging element 17 in a conjugate imaging relationship. Using a specific algorithm described below, the line of sight of the eyeball 14 is detected from the position of the corneal reflection image in the eyeball image formed on the ocular imaging element 17.

3 is a block diagram showing the electrical configuration in the camera 1. The CPU 3 is connected to a line of sight detection circuit 201, a photometry circuit 202, an autofocus detection circuit 203, a signal input circuit 204, a display device drive circuit 11, a light source drive circuit 205, and the like. The CPU 3 also transmits signals to a focus adjustment circuit 118 disposed in the photographing lens unit 1A and an aperture control circuit 206 included in an aperture drive section 112 in the photographing lens unit 1A via a mount contact 117. The memory section 4 associated with the CPU 3 has a function of storing image pickup signals from the image pickup element 2 and the eye image pickup element 17, and a function of storing line of sight correction parameters for correcting individual differences in the line of sight, which will be described later.

The gaze detection circuit 201 A/D converts the output of the eye imaging element 17 (eye image captured of the eye) when an eyeball image is formed on the eye imaging element 17, and transmits the result to the CPU 3. The CPU 3 extracts feature points required for gaze detection from the eye image according to a predetermined algorithm described below, and calculates the user's viewpoint (gaze position; position where the gaze is directed; position where the user is looking) on the display surface of the display device 10 from the positions of the feature points.

The photometry circuit 202 amplifies, logarithmically compresses, and A/D converts the signal obtained from the image sensor 2, which also functions as a photometry sensor; specifically, the luminance signal corresponds to the brightness of the field, and sends the result to the CPU 3 as field luminance information.

The auto focus detection circuit 203 A/D converts the signal voltage from multiple detection elements (multiple pixels) used for phase difference detection included in the image sensor 2, and sends it to the CPU 3. The CPU 3 calculates the distance to the subject corresponding to each focus detection point from the signals of the multiple detection elements. This is a well-known technique known as image plane phase difference AF. In the first embodiment, as an example, it is assumed that there are focus detection points in the field of view within the viewfinder (the field of view when looking through the viewfinder) in FIG. 4(A), specifically, at each of the 180 locations on the image plane corresponding to the 180 locations shown on the display surface of the display device 10.

Switches SW1 and SW2 are connected to the signal input circuit 204. Switch SW1 is turned on by the first stroke of the release button 5, and is a switch for starting the photometry, distance measurement, line of sight detection, and other operations of the camera 1. Switch SW2 is turned on by the second stroke of the release button 5, and is a switch for starting the shooting operation. The ON signals from switches SW1 and SW2 are input to the signal input circuit 204 and transmitted to the CPU 3.

Figure 4 (A) is a diagram showing the field of view in the viewfinder, and shows the state in which the display device 10 is operating (the state in which an image is displayed). As shown in Figure 4 (A), the field of view in the viewfinder includes a focus detection area 400, 180 ranging point indices 401, a field of view mask 402, and the like. Each of the 180 ranging point indices 401 is displayed superimposed on the through image (live view image) displayed on the display device 10 so as to be displayed at a position corresponding to the focus detection point on the imaging surface. Also, of the 180 ranging point indices 401, the ranging point indices 401 that corresponds to the current viewpoint A (estimated position) is displayed highlighted with a frame or the like.

Although an example has been described in which a frame highlighting the ranging point indicator corresponding to the current viewpoint (estimated position) is displayed as an item indicating the current viewpoint (estimated position), this is not limiting. For example, as shown in FIG. 4(D), the item indicating the current viewpoint may be an item indicating only the viewpoint (gaze pointer) rather than an item that identifiably indicates the ranging point indicator corresponding to the viewpoint. In that case, both the item indicating the current viewpoint and the ranging point indicator and various buttons corresponding to the viewpoint may be displayed with emphasis. The display position of the item indicating the current viewpoint may change continuously or in stages.

<Description of Gaze Detection Operation>
The gaze detection method will be described with reference to Figs. 5, 6(A), 6(B), and 7. Fig. 5 is a diagram for explaining the principle of the gaze detection method, and is a schematic diagram of an optical system for performing gaze detection. As shown in Fig. 5, the light sources 13a and 13b are arranged approximately symmetrically with respect to the optical axis of the light receiving lens 16, and illuminate the user's eyeball 14. A part of the light emitted from the light sources 13a and 13b and reflected by the eyeball 14 is collected on the eye imaging element 17 by the light receiving lens 16. Fig. 6(A) is a schematic diagram of an eye image captured by the eye imaging element 17 (eyeball image projected on the eye imaging element 17), and Fig. 6(B) is a diagram showing the output intensity of the CMOS in the eye imaging element 17. Fig. 7 is a flowchart of the gaze detection operation.

When the gaze detection operation starts, in step S701 in FIG. 7, the light sources 13a and 13b emit infrared light toward the user's eyeball 14. An image of the user's eyeball illuminated by the infrared light is formed on the eye image sensor 17 through the light receiving lens 16 and is photoelectrically converted by the eye image sensor 17. This provides an electrical signal of the eye image that can be processed.

In step S702, the gaze detection circuit 201 sends the eye image (eye image signal; electrical signal of the eye image) obtained from the eye imaging element 17 to the CPU 3.

In step S703, CPU 3 obtains the coordinates of the points corresponding to the corneal reflection images Pd and Pe of light sources 13a and 13b and the pupil center c from the eye image obtained in step S702.

The infrared light emitted from the light sources 13a and 13b illuminates the cornea 142 of the user's eyeball 14. At this time, corneal reflection images Pd and Pe formed by a portion of the infrared light reflected from the surface of the cornea 142 are collected by the light receiving lens 16 and focused on the ocular imaging element 17 to become corneal reflection images Pd' and Pe' in the eye image. Similarly, light beams from the ends a and b of the pupil 141 are also focused on the ocular imaging element 17 to become pupil end images a' and b' in the eye image.

Figure 6 (B) shows the luminance information (luminance distribution) of the region α in the eye image of Figure 6 (A). In Figure 6 (B), the horizontal direction of the eye image is the X-axis direction, and the vertical direction is the Y-axis direction, and the luminance distribution in the X-axis direction is shown. In the first embodiment, the coordinates of the corneal reflection images Pd', Pe' in the X-axis direction (horizontal direction) are Xd, Xe, and the coordinates of the pupil edge images a', b' in the X-axis direction are Xa, Xb. As shown in Figure 6 (B), an extremely high level of luminance is obtained at the X-coordinates Xd, Xe of the corneal reflection images Pd', Pe'. In the region from the X-coordinate Xa to the X-coordinate Xb, which corresponds to the region of the pupil 141 (the region of the pupil image obtained by focusing the light beam from the pupil 141 on the ocular imaging element 17), an extremely low level of luminance is obtained except for the X-coordinates Xd, Xe. In the region of the iris 143 outside the pupil 141 (the region of the iris image outside the pupil image obtained by focusing the light beam from the iris 143), a luminance intermediate between the two types of luminance described above is obtained. Specifically, a luminance intermediate between the two types of luminance described above is obtained in a region where the X coordinate (coordinate in the X-axis direction) is smaller than the X coordinate Xa, and a region where the X coordinate is larger than the X coordinate Xb.

From the luminance distribution as shown in FIG. 6B, the X coordinates Xd, Xe of the corneal reflection images Pd', Pe' and the X coordinates Xa, Xb of the pupil edge images a', b' can be obtained. Specifically, the coordinates of the corneal reflection images Pd', Pe' can be obtained as the coordinates of the corneal reflection images Pd', Pe', and the coordinates of the pupil edge images a', b' can be obtained as the coordinates of the pupil edge images a', b'. In addition, when the rotation angle θx of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 is small, the X coordinate Xc of the pupil center image c' (center of the pupil image) obtained by focusing the light beam from the pupil center c on the ocular imaging element 17 can be expressed as Xc ≒ (Xa + Xb) / 2. In other words, the X coordinate Xc of the pupil center image c' can be calculated from the X coordinates Xa, Xb of the pupil edge images a', b'. In this way, the coordinates of the corneal reflection images Pd', Pe' and the coordinates of the pupil center image c' can be estimated.

In step S704, the CPU 3 calculates the imaging magnification β of the eyeball image. The imaging magnification β is determined by the position of the eyeball 14 relative to the light receiving lens 16, and can be calculated using a function of the distance (Xd-Xe) between the corneal reflection images Pd', Pe'.

In step S705, the CPU 3 calculates the rotation angle of the optical axis of the eyeball 14 relative to the optical axis of the light receiving lens 16 as information on the line of sight of the eyeball 14. The X coordinate of the midpoint between the corneal reflection images Pd and Pe and the X coordinate of the center of curvature O of the cornea 142 are almost the same.
If the standard distance from the center of curvature O of 42 to the center c of the pupil 141 is Oc, the rotation angle θx of the eyeball 14 in the Z-X plane (plane perpendicular to the Y axis) can be calculated by the following formula 1. The rotation angle θy of the eyeball 14 in the Z-Y plane (plane perpendicular to the X axis) can also be calculated in a similar manner to the method for calculating the rotation angle θx.

β×Oc×SINθx≒{(Xd+Xe)/2}−Xc...(Formula 1)

In step S706, the CPU 3 uses the rotation angles θx, θy calculated in step S705 to determine (estimate) the user's viewpoint on the display surface of the display device 10. If the coordinates (Hx, Hy) of the viewpoint are coordinates corresponding to the pupil center c, the coordinates (Hx, Hy) of the viewpoint can be calculated by the following formulas 2 and 3.

Hx=m×(Ax×θx+Bx) (Formula 2)
Hy=m×(Ay×θy+By) (Formula 3)

The parameter m in Equations 2 and 3 is a constant determined by the configuration of the viewfinder optical system (light receiving lens 16, etc.) of the camera 1, and is a conversion coefficient that converts the rotation angles θx and θy into coordinates corresponding to the pupil center c on the display surface of the display device 10. The parameter m is determined in advance and stored in the memory unit 4. The parameters Ax, Bx, Ay, and By are gaze correction parameters that correct individual differences in the gaze, and are obtained by performing a calibration operation described below. The parameters Ax, Bx, Ay, and By are stored in the memory unit 4 before the gaze detection operation begins.

In step S707, the CPU 3 stores the coordinates (Hx, Hy) of the viewpoint in the memory unit 4 and ends the gaze detection operation.

Note that, although an example using a corneal reflection image has been described, the method of gaze detection is not limited to this, and any method of detecting the gaze based on an eye image may be used. Also, although an example has been described in which the coordinates of the viewpoint (Hx, Hy) are obtained as the result (final result) of gaze detection, any information related to the gaze (gaze information), such as rotation angles θx, θy, may be obtained as the result of gaze detection.

<Calibration procedure explanation>
Although the gaze can be estimated by the above-mentioned gaze detection operation, the gaze may not be estimated with high accuracy due to factors such as individual differences in the shape of the human eyeball. Specifically, unless the gaze correction parameters Ax, Ay, Bx, and By are adjusted to values suitable for the user, a deviation occurs between the actual viewpoint B and the estimated viewpoint C, as shown in Fig. 4(B). In Fig. 4(B), the user is gazing at a person, but the camera 1 erroneously estimates that the user is gazing at the background, resulting in a state in which appropriate focus detection and adjustment cannot be performed.

Therefore, before camera 1 captures an image, it is necessary to perform a calibration process to obtain gaze correction parameters suitable for the user and store them in camera 1.

Conventionally, calibration work has been performed by highlighting multiple indices at different positions on the display surface of the display device 10 before imaging, as shown in Fig. 4(C), and having the user look at the indices. A known technique is then to perform a gaze detection operation when the user gazes at each index, and to calculate gaze correction parameters suitable for the user from the calculated multiple viewpoints (estimated positions) and the coordinates of each index. If the position where the user should look is suggested,
The method of displaying the indicator is not particularly limited, and the indicator may be displayed as a graphic, or the indicator may be displayed by changing the brightness or color of an image (such as a captured image).

<Explanation of human physiological eye movements>
One of the causes of errors in gaze detection is physiological eye movement. For example, even when a person intends to look at one point, the human eye moves involuntarily and slightly. This type of involuntary movement is called fixation eye movement. Since changes in gaze caused by fixation eye movement are not intended by the user, fixation eye movement is a cause of errors in gaze detection.

Therefore, in the first embodiment, changes in gaze information (detected gaze) due to fixational eye movement are reduced (fixational eye movement reduction process). The fixational eye movement reduction process is performed, for example, by the CPU 3. There are no particular limitations on the fixational eye movement reduction process, but in the first embodiment, it is a low-pass filter process that attenuates components of a specific frequency band (a frequency band higher than the cutoff frequency) in the detected gaze change.

Figures 8(A) and 8(B) are graphs showing the change over time in the X-coordinate of the detected viewpoint. The vertical axis of Figures 8(A) and 8(B) indicates the X-coordinate of the detected viewpoint, and the horizontal axis of Figures 8(A) and 8(B) indicates time. Figure 8(A) shows the state before low-pass filter processing, which is a fixational eye movement reduction process, and Figure 8(B) shows the state after low-pass filter processing. Figure 8(A) shows that the X-coordinate of the detected viewpoint fluctuates drastically (finely) before low-pass filter processing. As shown in Figure 8(B), by performing low-pass filter processing, the high-frequency components of the detected viewpoint fluctuation (fluctuation in Figure 8(A)) are attenuated, and the fluctuation of the detected viewpoint X-coordinate is reduced.

However, changes in gaze information (detected gaze) due to fixational eye movement do not always need to be reduced to the same degree. In other words, fixational eye movement reduction processing does not always need to be performed with the same processing strength. Fixational eye movement is not the only physiological eye movement in humans. When tracking a slow-moving object with the eyes, humans tend to perform eye movements (a type of voluntary movement) that move the gaze smoothly, and when tracking a fast-moving object with the eyes, they tend to perform jumping movements that change the gaze suddenly. When fixational eye movement reduction processing is performed, changes in the detected gaze (gaze information) become slower compared to changes in the actual gaze (eye movement). In other words, the ability of the detected gaze to track the object becomes poor. Therefore, if the processing strength of the fixational eye movement reduction processing is strong (if the degree of reduction by the fixational eye movement reduction processing is large), the gaze information will not accurately represent voluntary movements, jumping movements, etc.

Therefore, in the first embodiment, the CPU 3 changes the strength of the fixation eye movement reduction process (the degree of reduction by the fixation eye movement reduction process) so as to suitably reduce changes in gaze information due to fixation eye movement. The details will be described later. The gaze pointer and the like are displayed to indicate the viewpoint after the fixation eye movement reduction process.

<Description of Control of Fixation Eye Movement Reduction Processing>
In the first embodiment, the CPU 3 changes the strength of the fixation eye movement reduction process (the degree of reduction by the fixation eye movement reduction process) according to the focal length (the zoom magnification of the camera 1). When the focal length is changed by changing the photographing lens unit 1A, information on the change in focal length and information (value) of the focal length are transmitted from the focus adjustment circuit 118 in the photographing lens unit 1A to the CPU 3 in the camera housing 1B. However, this is not limited to this, and for example, when the correspondence between the photographing lens unit 1A and the focal length is one-to-one, the memory unit 4 in the camera housing 1B may store the correspondence between the identification information (such as a model number) of the photographing lens unit 1A and the focal length. In that case, the identification information of the photographing lens unit 1A may be transmitted from the photographing lens unit 1A to the CPU 3, and the CPU 3 may determine the current focal length information from the received identification information and the information stored in the memory unit 4 (the correspondence between the identification information and the focal length).

When the focal length is short (when the zoom ratio is low), the angle of view (imaging range) is wide, so the size of the object (subject) relative to the angle of view tends to be small. In other words, the object tends to be small in the captured image. And when the object is small in the captured image, the relative magnitude (amount of shaking) of fixational eye movement to the size of the object tends to be large.

From the above perspective, when the focal length is short, the intensity of the fixation eye movement reduction process is increased. In other words, the degree of reduction achieved by the fixation eye movement reduction process is increased. Specifically, the cutoff frequency of the low-pass filter process, which is the fixation eye movement reduction process, is lowered (shifted to the lower frequency side). This makes it possible to stabilize the gaze information (detected gaze).

This makes the change in the detected gaze (gaze information) slower compared to the actual change in gaze (eye movement). However, if the object is small in the captured image, the amount of movement of the object relative to the angle of view tends to be small, and the object can continue to be captured by the detected gaze, so this is not a problem.

On the other hand, when the focal length is long (when the zoom ratio is high), the angle of view is narrow, so the size of the object (subject) tends to be large relative to the angle of view. In other words, the object tends to be large in the captured image. And when the object is large in the captured image, the amount of movement of the object relative to the angle of view tends to be large.

From this perspective, when the focal length is long, the strength of the fixation eye movement reduction process is weakened. In other words, the degree of reduction achieved by the fixation eye movement reduction process is reduced. Specifically, the cutoff frequency of the low-pass filter process, which is the fixation eye movement reduction process, is increased (shifted to the higher frequency side). This increases the responsiveness of changes in the detected gaze (gaze information) to changes in the actual gaze (eye movement), improving the tracking of the detected gaze to an object.

Doing so reduces the stability of the gaze information (detected gaze). However, if the object is large in the captured image, this is not a problem because the object can continue to be captured by the gaze (the detected viewpoint can be kept within the area of the object) even if the detected gaze is slightly blurred.

In the first embodiment, in order to realize the above-mentioned control, information (such as a function or a table) indicating the correspondence between focal length and cutoff frequency is stored in advance in the memory unit 4. FIG. 9 is a graph showing the correspondence between focal length and cutoff frequency. The vertical axis of FIG. 9 indicates the cutoff frequency, and the horizontal axis of FIG. 9 indicates the focal length. According to the correspondence in FIG. 9, the cutoff frequency of the low-pass filter processing, which is the fixational eye movement reduction processing, becomes lower as the focal length becomes shorter. Therefore, the degree of reduction by the fixational eye movement reduction processing becomes greater as the focal length becomes shorter.

In FIG. 9, the cutoff frequency increases linearly with increasing focal length, but the cutoff frequency may increase nonlinearly with increasing focal length. The cutoff frequency may increase stepwise (discontinuously) with a continuous increase in focal length. Similarly, the degree of reduction by the fixational eye movement reduction process may decrease linearly or nonlinearly with increasing focal length. The degree of reduction by the fixational eye movement reduction process may decrease stepwise (discontinuously) with a continuous increase in focal length.

In addition, the fixational eye movement reduction process is not limited to low-pass filter processing. For example, the fixational eye movement reduction process may be another filter process that attenuates components of a specific frequency band, such as band-pass filter processing. The fixational eye movement reduction process may be Kalman filter processing, which is a sequential Bayes filter process. For example, in the Kalman filter process, the posterior estimate is calculated using the following formula 4. The posterior estimate corresponds to the gaze information after the current fixational eye movement reduction process, the prior estimate corresponds to the gaze information after the previous fixational eye movement reduction process, and the observed value corresponds to the gaze information before the current fixational eye movement reduction process.

As shown in FIG. 10, the posterior estimate corresponds to the division point between the prior estimate and the observed value, with the internal division ratio being k:1-k (0<k<1). Increasing the uncertainty of the observation, that is, decreasing k, makes the posterior estimate approach the prior estimate. As described above, the prior estimate corresponds to the gaze information after the previous fixation eye movement reduction process. Therefore, by increasing the uncertainty of the observation, it is possible to significantly reduce changes in the gaze information (detected gaze) due to fixation eye movement. For this reason, by increasing the uncertainty of the observation when the focal length is short, it is possible to stabilize the gaze information so that small objects in the captured image can be captured by the gaze.

On the other hand, when the prediction uncertainty is increased, that is, when k is increased, the posterior estimate approaches the observed value. As described above, the observed value corresponds to the gaze information before the current fixation eye movement reduction process. Therefore, by increasing the prediction uncertainty, it is not possible to significantly reduce changes in gaze information (detected gaze) due to fixation eye movement, but it is possible to increase the responsiveness of changes in the detected gaze to changes in the actual gaze (eye movement). For this reason, by increasing the prediction uncertainty when the focal length is long, it is possible to improve the tracking ability of the detected gaze for objects with large amounts of movement in the captured image.

<Summary>
As described above, in the first embodiment, the degree of reduction by the fixation eye movement reduction process is changed depending on the focal length, thereby making it possible to suitably reduce changes in gaze information due to fixation eye movement.

<<Embodiment 2>>
A second embodiment of the present invention will be described below. Note that, in the following, a description of the same points (such as configuration and processing) as in the first embodiment will be omitted, and only points different from the first embodiment will be described. In the second embodiment, in addition to the focal length, the fixation eye movement reduction process is controlled by taking into account the depth to the object (subject) in the captured image (object distance; subject distance). Note that, although an example in which both the focal length and the object distance are taken into account will be described, only the object distance may be taken into account.

Figure 11 is a flowchart showing the control of embodiment 2.

In step S1101, the CPU 3 in the camera housing 1B obtains focal length information (value) from the focus adjustment circuit 118 in the photographing lens unit 1A.

In step S1102, the CPU 3 acquires information (value) of the object distance, which is the distance from the camera 1 (imaging position) to the object. For example, the CPU 3 detects a phase difference or contrast using the image sensor 2 to obtain a focal position, and obtains the object distance based on the focal position. Note that the method of acquiring the object distance information is not limited to this, and for example, the object distance information may be acquired using a distance measuring sensor.

In step S1103, CPU 3 determines the cutoff frequency based on the information (focal length and object distance) acquired in steps S1101 and S1102.

In step S1104, CPU 3 changes the cutoff frequency of the low-pass filter process, which is the fixational eye movement reduction process, to the cutoff frequency determined in step S1103.

If conditions other than the object distance (for example, focal length) are constant, the size of the object in the captured image will be larger when the object distance is short than when the object distance is long. Taking this point of view into further consideration, when the focal length is constant, the degree of reduction by the fixation eye movement reduction process is increased if the object distance is long, and decreased if the object distance is short.

In the second embodiment, in order to realize the above-mentioned control, information (such as a function or a table) indicating the correspondence between focal length, object distance, and cutoff frequency is stored in advance in the memory unit 4. FIG. 12 is a graph showing the correspondence between focal length, object distance, and cutoff frequency. The vertical axis of FIG. 12 indicates the cutoff frequency, and the horizontal axis of FIG. 12 indicates the object distance. FIG. 12 shows the correspondence between object distance and cutoff frequency for each focal length. For each correspondence, the cutoff frequency decreases as the object distance increases. Therefore, when the focal length is constant, the degree of reduction by the fixational eye movement reduction process increases as the object distance increases.

In FIG. 12, the cutoff frequency decreases linearly with increasing object distance, but the cutoff frequency may decrease nonlinearly with increasing object distance. The cutoff frequency may decrease stepwise (discontinuously) with a continuous increase in object distance. Similarly, the degree of reduction by the fixational eye movement reduction process may decrease linearly or nonlinearly with increasing object distance. The degree of reduction by the fixational eye movement reduction process may decrease stepwise (discontinuously) with a continuous increase in object distance.

In the second embodiment, the fixational eye movement reduction process is not limited to low-pass filter processing, and may be band-pass filter processing or Kalman filter processing. In the case of Kalman filter processing, when the focal length is constant, if the object distance is short, the prediction uncertainty is increased, and when the object distance is long, the observation uncertainty is increased. In other words, when the focal length is constant, if the object distance is short, k is increased, and when the object distance is long, k is decreased.

As described above, in the second embodiment, the degree of reduction by the fixational eye movement reduction process is changed taking into account the object distance. This makes it possible to suitably reduce changes in gaze information due to fixational eye movement. Furthermore, by taking into account both the focal length and the object distance, it is possible to suitably reduce changes in gaze information due to fixational eye movement compared to when only one of them is considered.

Note that the first and second embodiments are merely examples, and configurations obtained by appropriately modifying or changing the configurations of the first and second embodiments within the scope of the present invention are also included in the present invention. Configurations obtained by appropriately combining the configurations of the first and second embodiments are also included in the present invention.

For example, the present invention is applied to an imaging device (camera), but is not limited thereto. That is, the gaze information acquisition device of the present invention may be a device separate from the imaging device having an imaging unit. The display unit that displays the image seen by the user may be a device separate from the gaze information acquisition device, and the control unit (CPU) of the gaze information acquisition device may control the display unit, which is an external device, to display images, items, etc. The image seen by the user may be an image different from the captured image, such as a three-dimensional computer graphic. The gaze information acquisition device of the present invention may have at least a function of acquiring gaze information, a function of performing fixation eye movement reduction processing, and a function of changing the degree of reduction by the fixation eye movement reduction processing. In addition, the fixation eye movement reduction processing is a process of correcting (adjusting; changing) the acquired gaze information so that the change in the gaze information due to the fixation eye movement is reduced, but is not limited thereto. For example, the fixation eye movement reduction processing may be a process of changing the method of acquiring gaze information so that the change in the gaze information due to the fixation eye movement is reduced.

<<Applications to other electronic devices>>
13A is an external view of a notebook type personal computer 1310 (notebook PC) to which the present invention is applied. In FIG. 13A, an imaging unit 1315 for imaging a user looking at a display unit 1311 of the notebook PC 1310 is connected to the notebook PC 1310, and the notebook PC 1310 acquires an imaging result from the imaging unit 1315. The notebook PC 1310 detects the user's line of sight based on the imaging result, and changes the degree of reduction by the fixation eye movement reduction process according to the focal length and object distance corresponding to the image displayed on the display unit 1311. Note that the electronic device to which the present invention is applied only needs to have an interface that accepts the result of the line of sight detection (line of sight information) as a line of sight input, and the line of sight detection may be performed by an external device. In other words, the imaging unit 1315 may perform line of sight detection, and the notebook PC 1310 may acquire the result of the line of sight detection from the imaging unit 1315.

Fig. 13(B) is an external view of a smartphone 1320 to which the present invention is applied. In Fig. 13(B), the smartphone 1320 detects the user's line of sight based on the image captured by the in-camera 1321 (front camera), and changes the degree of reduction by the fixation eye movement reduction process according to the focal length and object distance corresponding to the image displayed on the display unit 1322.

Figure 13 (C) is an external view of a game machine 1330 to which the present invention is applied. In Figure 13 (C), a head mounted display 1335 (HMD) that displays a VR (Virtual Reality) image of a game is connected to the game machine 1330. The HMD 1335 has a camera 1337 that captures the eyes of a user wearing the HMD 1335, and the game machine 1330 acquires the captured image from the HMD 1335. The game machine 1330 then detects the user's line of sight based on the captured image, and changes the degree of reduction by the fixation eye movement reduction process according to the depth (object distance) to an object in the VR image displayed on the display unit 1336 of the HMD 1335. Here, the object in the VR image is, for example, an object that exists in the direction closest to the user's line of sight. Just as the present invention can be applied to viewing VR images displayed on an HMD, the present invention can also be applied to viewing AR (Augmented Reality) images displayed on the lens portion of a glasses-type wearable device. Just as the present invention can be applied to VR technology and AR technology, the present invention can also be applied to other xR technologies such as MR (Mixed Reality) technology and SR (Substitutional Reality) technology.

The imaging unit 1315 in FIG. 13(A) and the HMD 1335 in FIG. 13(C) may change the degree of reduction by the fixation eye movement reduction process and perform the fixation eye movement reduction process. In other words, the present invention is also applicable to the imaging unit 1315 in FIG. 13(A) and the HMD 1335 in FIG. 13(C).

<<Other embodiments>>
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

1: Camera 3: CPU 201: Line-of-sight detection circuit 1310: Notebook personal computer 1320: Smartphone 1330: Game machine 1335: Head-mounted display

Claims (12)

A first acquisition means for acquiring line-of-sight information which is information on a line of sight of an eye viewing a captured image;
A reduction means for reducing a change in the line of sight information due to fixational eye movement,
The reduction means reduces the change in the gaze information caused by the fixational eye movement to a greater degree as the focal length when capturing the image is shorter. 2. The line-of-sight information acquisition device according to claim 1 , further comprising a second acquisition means for acquiring the information on the focal length. The gaze information acquisition device of claim 1 or 2, characterized in that when the focal length is constant, the reduction means reduces the change in the gaze information due to the fixational eye movement to a greater degree when the object distance, which is the depth to the object in the image, is long than when the object distance is short. 4. The line-of-sight information acquisition device according to claim 3 , further comprising a third acquisition means for acquiring information about the object distance. The gaze information acquisition device according to any one of claims 1 to 4 , characterized in that the process of reducing the change in the gaze information due to the fixational eye movement is a process of attenuating components of a specific frequency band in the change in the gaze. the process of reducing the change in the gaze information due to the fixational eye movement is a low-pass filter process of attenuating a component of the specific frequency band higher than a cutoff frequency in the change in the gaze,
6. The line-of-sight information acquisition device according to claim 5 , wherein the reduction means increases the degree of reduction by lowering the cutoff frequency. 7. The line-of-sight information acquiring device according to claim 1, further comprising a display unit for displaying the image. The gaze information acquisition device according to any one of claims 1 to 7, further comprising a control means for controlling the display unit displaying the image to display an item indicating the position of the gaze after the change due to the fixational eye movement has been reduced. An imaging means;
The gaze information acquisition device according to any one of claims 1 to 8 ,
The line-of-sight information acquisition device is an imaging device characterized in that the image captured by the imaging means is displayed on a display unit, and the imaging device detects the line of sight of an eye viewing the image. An acquisition step of acquiring line-of-sight information which is information on the line of sight of an eye viewing a captured image;
A reduction step of reducing a change in the gaze information due to fixational eye movement,
A method for acquiring gaze information, characterized in that in the reduction step, the change in the gaze information due to the fixational eye movement is reduced to a greater degree as the focal length when capturing the image is shorter. A program for causing a computer to function as each of the means of the line-of-sight information acquisition device according to any one of claims 1 to 8 . A computer-readable storage medium storing a program for causing a computer to function as each of the means of the line-of-sight information acquisition device according to any one of claims 1 to 8 .

Images