JPH0583249B2 - - Google Patents

Description

[Detailed description of the invention]

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a line-of-sight information control device, and particularly to a line-of-sight information control device that controls the direction in which a person looks, that is, the direction of the line of sight, and an interlocking system using this direction.

[0002]

[Prior Art] In technical fields that require so-called telexistence, in which information existing in a remote location is photographed by remotely controlling a man-machine interface or a camera, information on the direction of a person's line of sight is used in various control systems. One of the important challenges is to design a line-of-sight detector that transmits data to and interlocks with the direction in which it is viewed. In particular, it is strongly desired to realize stable and sensitive line-of-sight linkage. In other words, there is a need for a gaze information control algorithm that extracts movement information corresponding to the intention from the trajectory of the eyeball movement, which moves continuously and involuntarily.

[0003] FIG. 5 is a diagram illustrating an example of a device that detects the line of sight direction using a conventional method. Referring to FIG. 5, a self-propelled robot 1 includes a camera 2 and a manipulator 3 having six degrees of freedom: left and right, up and down, and left and right rotation.
and is remotely guided by commands from the command system 4. In the command system 4, movements 5 of a person's eyes and head are detected by a head orientation detection unit 7 and an eyeball orientation detection unit 8 provided on glasses 6, and hand movements are detected by an operation unit 9. and the respective detection outputs are provided to the transmission section 10. These detection outputs are transmitted from the transmission section 10 to the self-propelled robot 1 by wireless or the like. The self-propelled robot 1 moves according to the detection output of the operation unit 9, the manipulator 3 operates, and the camera 2 captures an image in the direction detected by the detection units 7 and 8.
It is transmitted to the command system 4. The captured image is displayed on the monitor 11. Ideally, a stable continuous landscape is reproduced as shown in 12.

[0004] FIG. 6 is a diagram for explaining a state in which a linked image becomes blurred when a user gazes in a certain direction while operating a conventional line-of-sight direction detection device. Figure 5
In the monitor 11 shown in FIG. The image will be displayed as an image. However, due to the involuntary coordination involved in eye movements and their unstable behavior, scenes that are unrelated to our intentions may be reproduced even though we are staring intently in a specific direction. Therefore, by considerably lowering the detection sensitivity of the head direction detection section 7 and the eyeball direction detection section 8, a stable image can be reproduced. Become aware of delays.

[0005] FIG. 7 is a diagram showing an algorithm of a conventional eyeball detection system. With reference to FIG. 7, the detection unit 21
is to detect the eyeball, and the eyeball coordinate Ex,
Output Ey. In order to remove noise contained in this component, the direction of the line of sight is calculated according to the displacement of the eyeball, which is passed through a filter unit 22 consisting of a low-pass filter and filtered by a line-of-sight direction calculation unit 23. 24 to the remote system.

[0006] FIG. 8 is a diagram showing an example of a visual experiment result when a fixed visual target is observed, and FIG. 9 is a diagram showing an example of a visual experiment result when a specific image is observed.

[0007] The experimental results shown in FIG. 8 are for a case where the subject was instructed to gaze at a stationary optotype 31 on the screen for a specific period of time. It can be seen that the line of sight 32 is constantly moving around due to the addition of the incidental movement of the eyeballs.

[0008] The experimental results shown in FIG.
3 is presented to the subject and the subject is made to observe for a specific period of time. Gaze behavior can be broadly divided into a moving process and a gazing process. In the case of movement, the trajectory is mostly non-linear, and rectangular waveforms are partially included. Also, when the user is looking closely, a trajectory 34 is shown that turns around within a certain narrow range. One of the characteristics is that the moving speed is very slow and the rate of change in the orbital direction, that is, the change in curvature, is large.

[0009] However, even if the moving speed is slow,
This includes cases where the user is gazing even when the change in curvature is small. Therefore, in order to determine whether the user is gazing, it is desirable to use at least the movement speed of the line of sight and changes in curvature as criteria for determination. Therefore, in order to realize stable control, it is also necessary to set a representative value from the coordinate values in the gaze area and send it to the transmission system. Controlling line-of-sight information means processing information that matches the behavior after filtering to the position of what is required for operation. For example, when an observer wants to closely observe a certain area, it is necessary to use the method described above. Furthermore, even when moving the line of sight, it is necessary to form a rectangular waveform because landscapes are smoothly captured in daily life. Satisfying these conditions cannot be achieved simply by applying filtering.

[0010]

[Problems to be Solved by the Invention] As mentioned above, the conventional algorithm for detecting the gaze point is
It was not intended to control gaze information. Furthermore, there are no examples of its application to interlocking systems, and simply extracting the gaze point has problems with stabilization and the direction of trajectory formation when moving.

[0011] Therefore, the main purpose of the present invention is to provide a line-of-sight information control device that is equipped with a real-time gaze point determination algorithm and an algorithm that stably determines a control value to drive an interlocking system. .

[0012]

[Means for Solving the Problems] The present invention is a line-of-sight information control device, which includes an eye movement detecting means for detecting an image input from a camera or an amount of movement of an eyeball detected by a detector, and an eye movement detecting means. A filter means for smoothing the amount of eye movement detected by the filter means, and an evaluation function is calculated from the speed of the line of sight and the change value of the direction of movement of the line of sight according to the output of the filter means, and threshold processing is performed. and a transmitting means that transmits line-of-sight information controlled in accordance with the output of the gaze area detecting means.

[0013]

[Operation] The line-of-sight information control device according to the present invention removes the rectangular waveform component and filters the smooth component when the line-of-sight is moving based on unstable line-of-sight information, especially when you want to gaze. By detecting constant displacement information, the system controls information according to the operator's arbitrary line of sight direction in real time.

[0014]

Embodiments of the Invention FIG. 1 is a diagram schematically showing the movement of the line of sight, and FIG. 2 is a diagram for explaining the movement speed and curvature change of the line of sight in the gaze area.

[0015] In visual experiments, even when the operator intentionally moves the line of sight smoothly in the horizontal direction, the detection system detects rectangular eye movements due to involuntary movements.
1 is detected. Therefore, in the present invention, such a rectangular eye movement 41 is filtered to obtain a smooth trajectory 42 of change. A Kalman filter is used as the filter.

[0016] In the gaze area 43 shown in FIG.
Characteristic trajectories are shown by six points K, K1...K5. In this invention, the target point 44 is
is calculated from the center of gravity of three points (K1, K2, K3) to stabilize the line of sight information. When the continuous point sequence of the gaze area 43 is regarded as a wire, it is characterized by a relatively large change in curvature θ within a short period of time. Further, there are cases where the speed is slow and the change in curvature φ is small, or there are cases where the speed is fast and the curvature is large. Therefore, the present invention calculates the movement speed and curvature change of the line of sight to determine whether the user is gazing.

FIG. 3 is a schematic block diagram of an embodiment of the present invention. First, the eye movement detection unit 51 detects two-dimensional behavior of eye movement. The detection output of the eyeball interlocking detection section 51 is given to the filtering section 52. The filtering section 52 includes a Kalman filter, and forms a smooth waveform from the rectangular waveform shown in FIG. The output of the filtering section 52 is given to the gaze area detection section 53. The gaze area detection unit 53 determines whether or not the user is gazing based on the speed and curvature of the line of sight based on a certain judgment technique. The transmitting unit 54 transmits continuous values of the trajectory to the interlocking system when the line of sight is moving, and transmits discrete values to the interlocking system when the line of sight is gazing, according to the detection output of the gaze area detection unit 53. do.

FIG. 4 is a detailed block diagram of the gaze area detection section shown in FIG. 3. The gaze area detection unit 53 will be described in more detail with reference to FIG. 4. The speed calculation unit 531 calculates the speed V from the time-series change in the line of sight, and the curvature calculation unit 532 calculates the curvature K from the time-series change in the line of sight. The calculated velocity V and curvature K are given to the potential calculation unit 533.
The potential calculation unit 533 calculates a potential amount J (evaluation function) from these velocity V and curvature K in order to specify the gaze area. Potential amount J
is expressed as a linear sum of the square of the reciprocal of the velocity V and the square of the curvature K using appropriate weighting constants a and b. In other words, the slower the speed or the larger the curvature, the larger the potential amount becomes. In addition, constants a, b
is determined by the amount of visual stimulation.

[0019] The threshold processing unit 534 compares the potential amount J and a constant potential value Jth, and determines that the viewer is gazing if the potential amount J is larger than the potential value Jth. If it is determined that the user is gazing, the representative point selection unit 535 detects consecutive potentials, and then the representative point selection unit 535 detects three successive past points (the potential value J exceeding the threshold value Jth has started to be calculated). ), the center of gravity of the coordinate values is calculated, and a constant value is transmitted to the transmitter 54. If the potential value J does not exceed the threshold Jth, it is determined that the line of sight is moving, and the filtered coordinate values are transmitted as they are to the transmitter 54 without selecting a representative point.

【0020】

[Effects of the Invention] As described above, according to the present invention, the amount of movement of the eyeball is detected, the detected amount of movement of the eyeball is made into a smooth amount of displacement, and then the speed of the line of sight and the change in the direction of movement of the line of sight are determined. By calculating the evaluation function from Information can be obtained through real-time processing. In this way, by controlling the line-of-sight information, it is possible to obtain a landscape according to the line-of-sight direction that meets the operator's or observer's intention.

[Brief explanation of drawings]

FIG. 1 is a diagram schematically showing movement of the line of sight.

FIG. 2 is a diagram for explaining the movement speed and curvature change of the line of sight in the gaze area.

FIG. 3 is a schematic block diagram of an embodiment of the invention.

FIG. 4 is a more detailed block diagram of the gaze area detection section shown in FIG. 3;

FIG. 5 is a diagram showing an example of a conventional remote control device using line-of-sight direction detection information.

FIG. 6 is a diagram for explaining a state in which a linked image is blurred when an operator gazes in a certain direction using a conventional line-of-sight direction detection device.

FIG. 7 is a diagram showing an algorithm of a conventional eyeball detection system.

FIG. 8 is a diagram showing an example of the results of a visual experiment when observing a fixed line of sight.

FIG. 9 is a diagram showing an example of the results of a line-of-sight experiment when observing a specific image.

[Explanation of symbols]

51 Eye movement detection unit 52 Filtering section 53 Gaze area detection unit 54 Transmitter 531 Speed calculation section 532 Curvature calculation unit 533 Potential calculation section 534 Threshold processing unit 535 Representative point selection section.

Claims (1)

[Claims] 1. Eye movement detection means for detecting the amount of movement of the eyeball detected by an image input from a camera or a detector, the amount of movement of the eyeball detected by the eye movement detection means being a smooth displacement amount a filter means for calculating an evaluation function from the speed of the line of sight and the change value of the moving direction of the line of sight according to the output of the filter means;
A line-of-sight information control device, comprising: a gaze area detecting unit that determines a gaze point by threshold processing; and a transmitting unit that transmits controlled visual line information according to an output of the gaze area detecting unit.