JPH0446570B2 - - Google Patents

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to an eye movement analysis device, and more particularly to an eye movement analysis device that can detect eye movements and head movements of a subject and display a viewpoint on a display. Regarding.

[Prior Art] An eye camera is a device that detects eye movements of a subject and displays a gaze point on a display or film. It is. Various methods have been considered for detecting the subject's eye movements, including a method using corneal reflected light, a method using a contact lens, and an EOG method.

FIG. 10 is a diagram for explaining the principle of detecting eye movement using the scleral reflection method, which utilizes the difference in reflectance between the white and iris of the corneal surface. Referring to FIG. 10, sensors 1 and 2 each having three elements are arranged in left and right eyes E1 and E2. sensor 1,
A light emitting element 3 is disposed in the center of the light emitting element 2, and a light emitting diode having relatively wide directivity and emitting infrared radiation of approximately ±21 degrees is used as the light emitting element 3. Light receiving elements 4 are provided on both sides of the light emitting element 3. As the light receiving element 4, a photodiode with sharp directivity of about ±10° is used. The light projected from the light emitting element 3 to the eyeball has a different reflectance between the black eye and the white of the eye, and by amplifying this difference in reflectance and taking the difference, it becomes a horizontal (left and right) output, and when the sum is taken, it becomes a vertical (up and down) output. The output is

In addition, the positions of sensors 1 and 2 facing the eyeball are different between the horizontal direction and the vertical direction, with the horizontal sensor 1 detecting reflected light at the center of the eyeball above and below, and the vertical sensor 2 detecting the reflected light at the center of the eyeball. They are each arranged so as to detect reflected light below. Then, a horizontal sensor 1 is placed on one eye E1, and a vertical sensor 2 is placed on the other eye E2, and if these are used simultaneously, two-dimensional eye movement can be detected.
The viewpoint can be displayed on the display.
For example, from this method, the movement speed of the eyeball,
Japanese Patent Laid-Open No. 126140/1983 discloses a method for obtaining information such as movement direction, movement distance, and gaze time.

When analyzing eye movements using such a device, the subject's head is fixed using a chin rest, and
Record only eye movements. Alternatively, a small camera is mounted on goggles or eyeglass frames equipped with an eye movement detection sensor, and images corresponding to the subject's visual field are captured, allowing for free movement of the head while detecting eye movement. It is used. In addition, with conventional eye movement analyzers, the measured values are determined whether the head is fixed as described here, or even when the head is not fixed, the head movement is not detected or detected. However, it is not independently detected and converted into a line of sight.

[Problems to be solved by the invention] As mentioned above, in the conventional eye movement analysis device,
Because head movement is not taken into consideration, eye movement analysis is easy with methods that measure head movement by fixing it with a jaw rest, but in reality, gaze is performed using head movement in conjunction with the method. There was a problem in that it was not possible to realize the natural observational behavior of humans.

Even with a method that allows free head movement, it does not take into account the movement of the line of sight achieved by head movement, so if you move your head violently and change your line of sight, such as while driving a car, the eye movement will be affected. Because it has not occurred,
Due to the inconvenience of being displayed as a point of fixation, and the action of the posture control system when walking towards someone far away while gazing at their face, eye movements occur in the exact opposite direction to head movements, and in reality There was an inconvenience that even though I was looking at the other person's face, my eye movements were recorded and displayed.

Therefore, the main purpose of this invention is to simultaneously analyze the gaze movement achieved by head movement and the gaze movement caused by eyeball movement by detecting head movement and converting it into gaze movement. An object of the present invention is to provide an eye movement analysis device that can perform the following functions.

[Means for Solving the Problems] The present invention is an eye movement analysis device for analyzing eye movements of an observer, and includes an eye movement detecting means for detecting eye movements, and a head movement detecting means for detecting head movements. Gaze point processing that separates the gaze point from the gaze movement by calculating the observer's gaze movement according to the detection means and the detected eyeball movement and head movement, and defining the gaze point. means, an image processing means for displaying on the image gaze information consisting of the movement of the gaze moving between the gaze points, and a superimposing means for superimposing the gaze information on the video image of the observer. It is composed of:

[Function] The eye movement analysis device according to the present invention detects the movement of the eyeballs and the movement of the head, and calculates the movement of the observer's line of sight according to the movement of the eyeballs and the head. By defining the gaze point, the gaze point is separated from the gaze movement, and the gaze information consisting of the gaze movement moving between the gaze points is superimposed on the video image that the observer is watching. Display.

[Embodiment of the Invention] An embodiment of the invention will be described in detail below with reference to the drawings.

FIG. 1 is a schematic block diagram of one embodiment of the present invention. Referring to FIG. 1, an eye movement detection section 5 detects the movement of the observer's eyeballs, and provides the detection output to a gaze point processing section 7. The head motion detection unit 6 detects the movement of the observer's head, and provides the detection output to the gaze point processing unit 7. The gaze point processing unit 7 calculates the movement of the observer's line of sight according to the eye movement detected by the eye movement detection unit 5 and the head movement detected by the head movement detection unit 6, and determines the gaze point. By defining , we separate the movement component between the gaze points from the gaze movement. The line of sight information separated by the gaze point processing section 7 is given to the image processing section 8 and converted into an image signal. This image signal is applied to a superimposer 9, where it is superimposed on a video signal applied from the outside to become a video output, which is displayed on a display (not shown).

FIG. 2 is a diagram for explaining the movement of the line of sight due to head movement. Next, with reference to FIG. 2, the movement of the line of sight due to head movement will be explained. In general, moving a desired visual object can be achieved by rotating the eyeballs toward the visual object, or by moving the head without moving the eyeballs. Usually, both movements are used together. Head movements include parallel movements caused by the orientation of the feet and spine, and rotational movements realized by the neck, spine, hips, legs, etc. This head movement is converted into an eye rotation angle using a method described later to obtain a head movement corrected eye rotation angle α shown in FIG. Movement of the point of gaze is executed by a combination of this head movement corrected eyeball rotation angle α and the eyeball rotation angle θ. Here, the sum of these two rotation angles α and θ is collectively defined as the line of sight.

FIG. 3 is a diagram for explaining the method of defining the gaze point from the aforementioned line of sight. Even when we are fixating or gazing during eye movements, we make minute movements called fixation micromovements. In addition, the vestibulo-ocular reflex to prevent deviation of the line of sight due to posture fluctuations, and the head movement correction eyeball rotation angle α due to posture vibrations, occur during fixation. That is,
There are almost no moments when the line of sight is completely stationary in space. Therefore, it is necessary to separate the gaze point, which occurs when the subject gazes with the intention of receiving information from the object of view, from among the constantly moving gaze movements. For this purpose, it is necessary to set a certain standard and define the gaze point. Regarding this gaze point, one based on eye movement speed has already been reported. (Unexamined Japanese patent publication 1986
−126140: Journal of the Communication Society “Definition of gaze point in images and its application to image analysis” 1986, 9, Vol. J69
-D, No. 9, pp. 1335-1342).

One of the features of this invention is that the definition of the gaze point, which is determined based on the eye movement speed, is expanded to include the gaze movement speed including head movement. Specifically, as shown in Figure 3, when the next sample falls within the range Dth of the gaze point determined by the threshold velocity Vth and the sampling period Ts from the current gaze point, the current gaze point is It was set as the same point of fixation as the viewpoint, and when it exceeded this range, it was set as a new point of fixation. As an example of the threshold speed Vth, we set it as 5°/second based on the nature of follow-up movement during eye movement.
It is put into practical use. For example, when the sampling period is 10msec, Dth=5゜/sec×0.01sec=0.05
=3′.

FIG. 4 is a diagram for explaining the correspondence between the coordinates of an image viewed by an observer and the coordinates realized by an embodiment of the present invention. As shown in Figure 4a, the vertical Dv is a distance D from the subject's position,
When observing a horizontal Dh image, the image coordinates are (1,
1)...Represented by (Px, Py). The viewing angle of the presented image as seen from the subject is as shown in Figure 4b: a=180/π・tan ^-1 Dh/(2・D) b=180/π・tan ^-1 Dv/(2・D ) The formula for converting the gaze movement angle (Xv, Yv) into screen coordinates (Xp, Yp) is: Xp=f(Xv)=p+(r-p)・Xv/(a-
Xv) …(1) Yp=g(Yv)=q+(s-q)・Yv/(b-
Yv) ...(2)

FIG. 5 is a diagram showing an example in which the eye movement detecting section 5 and the head movement detecting section 6 shown in FIG. 1 are attached to goggles. The eye movement detecting section 5 is provided with a light emitting diode in the center in the same way as the sensor 1 shown in FIG. . A magnetic sensor is used as the head movement detection section 6.

FIG. 6 is a diagram for explaining the principle of the magnetic sensor used in the head motion detection section 6 shown in FIG. 5. Referring to FIG. 6, the head motion detection unit 6 includes an orthogonal coil that serves as a source 61 and an orthogonal coil that serves as a sensor 62. In response to commands from computer 64, drive circuit 65 drives orthogonal coils of source 61 to generate a magnetic field. When the subject wearing the head movement detection unit 6 moves, the sensor 62
A voltage is induced in the head, the detection circuit 63 detects this voltage, and the computer 64 calculates the detection output, thereby outputting data corresponding to the movement of the head.

FIG. 7 is a diagram showing the principle of a head coordinate system centered on the subject. Next, the head coordinate system detected by the head motion detection section 6 will be explained with reference to FIG. The head coordinate system consists of an XYZ coordinate system realized by parallel movement of the subject with respect to the observation target, as shown in Figure 7a, and a coordinate system as shown in Figure 7b.
There are two possible polar coordinate systems based on the rotational movement of the head shown in FIG. The amount of head movement in each coordinate system is defined as (Hx, Hy, Hz) and (Hφ, Hθ, Hψ).
Here, as an example, the direction of approaching the observation target is taken as the Y axis, the horizontal movement direction is taken as the X axis, and the vertical movement direction is taken as the Z axis. Hφ indicates the rotation point of the X axis, that is, the movement of nodding the head up and down, and Hθ indicates the rotation of the Y axis, that is, the movement of tilting the head once from the left shoulder to the right shoulder. Hψ is the rotation on the Z axis, which is the movement of rotating the neck from side to side.

The line of sight changes due to horizontal movement of the head (Hx, Hy, Hz), and when this is converted into an eyeball rotation angle (Ex, Ey), the following equation is obtained.

Ex=180/π・tan ^-1 Hx/(D+Hy) ...(3) Ey=180/π・tan ^-1 Hz/(D+Hy) ...(4) When you tilt your head Hθ toward the left or right shoulder, the eyeball The coordinates of the moving system rotate. Therefore,
It is necessary to convert the eye movement coordinate system (Xe, Ye) tilted by Hθ to a coordinate system (Xe', Ye') orthogonal to the original observation target.

Xe′=Xe・cosHθ+Ye・sinHθ…(5) Ye′=−Xe・sinHθ+Ye・cosHθ…(6) Gaze movement realized by head movement (Xh,
Yh) is expressed by the following equation from equations (3) and (4).

Xh＝Ex＋Hψ …(7) Yh＝Ey＋Hφ …(8) Therefore, from equations (5) to (8), the movement of the line of sight (Xv, Yv) considering the movement of the head is expressed by the following ( It is expressed by Equation 9) and Equation (10).

Xv=Xe′+Xh…(9) Yv=Ye′+Yh…(10) By using equations (9) and (10) above, head movement and eye movement are performed in combination. It is possible to reproduce normal eye movements.

FIG. 8 is a flowchart illustrating the overall operation of an embodiment of the present invention, and FIG. 9 is a diagram showing an example of a display screen according to the embodiment of the invention.

Next, referring to FIGS. 1 to 9, the specific operation of one embodiment of the present invention will be described using FIG. 8. First, in the initial settings, the coordinates where the observer is looking and the position data of the line of sight obtained by an embodiment of the present invention are set correctly and the coordinates are set to superimpose on the image the observer is looking at. Configure conversion coefficients and formulas. According to the coordinate conversion method shown in FIG. 4, the line-of-sight movement angle (Xv, Yv) is converted into image coordinates (Xp, Yp) using equations (1) and (2). Next, the eye movement data is calibrated using the eye movement detected by the eye movement detection section 5 and the head movement detected by the head movement detection section 6. In other words, the eye movement data and head movement data are converted into gaze movements using equations (9) and (10) above, and similarly, equations (1)
It is converted to screen coordinates using Equation and Equation (2).
The movement of the line of sight obtained by these transformations is transformed into screen coordinates, but the transformation is relative. In order to reproduce the movement of the subject's line of sight on the screen that captures the visual target that the subject is actually looking at, it is necessary to absolutely match the screen coordinates with the subject's line of sight. Eye movement calibration performs this task.

After the calibration is completed, the start of measurement is commanded by, for example, a key, and the measurement is repeated until the end of the measurement is commanded by key input again. In image display, the true movement of the line of sight is calculated from the eye movement data and head movement data using equations (1) to (10), and is displayed on the image that the subject is viewing. At this time, the movement of the line of sight is separated into movement components between the gaze points by the definition of the gaze point, an example of which has already been disclosed. Then, the superimposer 9 superimposes the gaze point and the movement trajectory between the gaze points on the image that the observer is viewing. An example of the display at that time is shown in FIG. 9th
The example shown in the figure is an example of the movement of the subject's gaze when looking at a girl. Points of interest are color-coded according to their gaze time, movement trajectories between the points of interest are color-coded according to their speed, and superimposed at corresponding positions in the image. Data such as eye movement, head movement, and number of samples obtained by these methods are recorded on a disk (not shown) or the like.

[Effects of the Invention] As described above, according to the present invention, the movement of the eyeballs and the movement of the head are detected, the movement of the observer's line of sight is calculated according to these detection outputs, and the point of gaze is defined. By doing this, the gaze point is separated from the gaze movement, and the gaze information is superimposed on the moving image that the observer is looking at, so the gaze movement due to eyeball movement and head movement are It is possible to simultaneously analyze the movement of the line of sight.

[Brief explanation of the drawing]

FIG. 1 is a schematic block diagram of one embodiment of the present invention. FIG. 2 is a diagram for explaining the movement of the line of sight due to head movement. FIG. 3 is a diagram for explaining the definition of a gaze point. FIG. 4 is a diagram for explaining the correspondence between the coordinates of an image viewed by an observer and the coordinates of a system realized by an embodiment of the present invention. FIG. 5 is a diagram showing an example in which an eye movement detection sensor and a head movement detection sensor are attached to goggles. FIG. 6 is a diagram for explaining the principle of the magnetic sensor used in the head motion detection section shown in FIG. 5. FIG. 7 is a diagram showing the principle of a head coordinate system centered on the subject. FIG. 8 is a flowchart for explaining the specific operation of one embodiment of the present invention. FIG. 9 is a diagram showing an example of a display screen. FIG. 10 is a diagram for explaining the principle of detecting eye movement using the scleral reflex method. In the figure, 5 is an eye movement detection unit, 6 is a head movement detection unit, 7 is a gaze point processing unit, 8 is an image processing unit,
9 is a superimposer, 61 is a source, 62 is a sensor, 63 is a detection circuit, 64 is a computer,
65 indicates a drive circuit.

Claims (1)

[Scope of Claims] 1. An eye movement analysis device that analyzes eye movements of an observer, comprising: eye movement detection means for detecting eye movement; head movement detection means for detecting head movement; and said eye movement. calculating the movement of the observer's line of sight according to the movement of the eyeballs detected by the detection means and the movement of the head detected by the head movement detection means, and defining a gaze point; a gaze point processing means for separating a gaze point from a gaze movement by a gaze point, an image processing means for displaying gaze information on an image consisting of gaze movement moving between gaze points, and said gaze point. An eye movement analysis device comprising superimposing means for superimposing information onto a video image being viewed by said observer.