JP6884123B2 - Gaze point estimation device, program and method for estimating the user's gaze point - Google Patents

Description

The present invention relates to a technique for estimating a gaze point when a user points the pointer at the gaze point.

Conventionally, there is a technique in which a user operates a target device by a hand gesture without having a device dedicated to pointing by himself (see, for example, Patent Document 1). As a result, an object that is out of reach of the user can be specified by a natural motion (gesture), and for example, remote control of the object is possible.
According to this technique, for example, a distance image in which a user is reflected is acquired by using a TOF (Time of Flight) depth camera that measures the depth based on the time of arrival and departure of infrared rays. Based on the distance image, the foreground including the user and the background other than the foreground are separated to learn the three-dimensional model. The background subtraction method for the three-dimensional model detects the user in the foreground and the device to be operated in the background. In addition, the user's hand part in the foreground is specified by graph matching with a known human body model. Then, the control command for the operation target device is estimated from the shape of the hand (for example, the direction of the extension line of the fingertip) and the state change in the time series.
According to Patent Document 1, an object that intersects a half straight line toward the gazing point estimated by the shape and posture of the hand portion is the operation target.

Japanese Unexamined Patent Publication No. 2013-205983

Cao, Zhe and Simon, Tomas and Wei, Shih-En and Sheikh, Yaser, “Realtime Multi-Person 2D Pose Optimization using Part Affinity Fields”, CVPR, vol.1, no.2, pp.7291-7299, 2017. , [Online], [Search on May 7, 2018], Internet <URL: https://arxiv.org/abs/1611.08050> Yonemoto, Satoshi and Matsumoto, Asuka and Arita, Daisaku and Taniguchi, Rin-ichiro, “A Real-time Motion Capture System with Multiple Camera Fusion”, Image Analysis and Processing, 1999, pp.600-605., [Online], [Search on May 7, 2018], Internet <URL: https://kyushu-u.pure.elsevier.com/en/publications/a-real-time-motion-capture-system-with-multiple-camera -fusion ＞ Marin, Giulio and Dominio, Fabio and Zanuttigh, Pietro, “Hand gesture recognition with leap motion and kinect devices”, IEEE International Conference on Image Processing (ICIP), pp.1565-1569, 2014., [online], [2018 Search on May 7, 2014], Internet <URL: https://scholar.google.it/citations?user=t9UpeEAAAAAJ&hl=en> A. Laurentini, “The Visual Hull Concept for Silhouette Based Image Understanding”, IEEE Trans. Pattern Analysis and Machine Intelligence, vol.16, no.2, pp.150-162, 1994., [online], [2018 Search on May 7], Internet <URL: https://ieeexplore.ieee.org/document/273735/> DGLowe, Distinctive image features from scale-invariant key points, Proc. Of Int. Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004), [online], [May 7, 2018 Day search], Internet <URL: https://www.cs.ubc.ca/~lowe/papers/ijcv04.pdf> H.Bay, T.Tuytelaars, and LVGool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006), [online], [Search May 7, 2018], Internet < URL: https://www.researchgate.net/publication/275057952_SURF_Speeded_up_robust_features> Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571., [Online], [Search May 7, 2018], Internet <URL : http://www.willowgarage.com/sites/default/files/orb_final.pdf ＞ Liu, Wei and Anguelov, Dragomir and Erhan, Dumitru and Szegedy, Christian and Reed, Scott and Fu, Cheng-Yang and Berg, Alexander C, “SSD: Single Shot MultiBox Detector”, European conference on computer vision, pp.21- 37, 2016., [online], [Search on May 7, 2018], Internet <URL: https://uncch.pure.elsevier.com/en/publications/ssd-single-shot-multibox-detector>

According to the technique described in Patent Document 1, since the designated object is not as displayed on the display, the position pointed to by the user is not fed back to the user himself / herself. Therefore, there is an inconvenience that the user cannot clearly specify it.

Therefore, the present invention provides a gaze point estimation device, a program, and a method for estimating a gaze point from an image in which a user, a pointer, and a gaze object are reflected when a user points the pointer at the gaze point. The purpose.

According to the present invention, the gaze point estimation device estimates the gaze point from an image in which the user, the pointer, and the gaze point are reflected when the user points the pointer at the gaze point.
A coordinate point detecting means for detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer from the image.
When the user moves the pointer to the first position that points to the gaze point seen from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected, and the user left eye. A gaze vector that detects the left eye gaze vector from the left eye to the gaze point via the pointer when the pointer is moved to a second position different from the first position, which points to the gaze point seen from Detection means and
It is characterized by having a gaze point estimation means for estimating the intersection of the right eye gaze vector and the left eye gaze vector as a gaze point as three-dimensional coordinates of the world coordinate system.

According to another embodiment of the gaze point estimator of the present invention.
A pointer is a predetermined human body part or indicator,
It is also preferable that the coordinate point detecting means detects a predetermined human body part or an indicator as a pointer from the image.

According to another embodiment of the gaze point estimator of the present invention.
The predetermined human body part may be a fingertip.

According to another embodiment of the gaze point estimator of the present invention.
For the user, attach markers to the right eye part, left eye part and pointer part,
It is also preferable that the coordinate point detecting means detects the marker by chroma key processing on the image.

According to another embodiment of the gaze point estimator of the present invention.
When the pointer is a predetermined human body part, the predetermined human body part is set as a predetermined joint.
It is also preferable that the coordinate point detecting means detects a predetermined joint of the human body by skeleton analysis from the image.

According to another embodiment of the gaze point estimator of the present invention.
It is also preferable that the coordinate point detecting means detects a predetermined human body part or an indicator by an object recognition process from the image.

According to another embodiment of the gaze point estimator of the present invention.
The image is a three-dimensional image taken by a camera,
It is also preferable that the camera is a single depth camera capable of capturing RGB values and depth (depth distance) values, or a plurality of image cameras capable of capturing RGB values and arranged at different positions.

According to another embodiment of the gaze point estimator of the present invention.
The pointer has one reference point and
It is also preferable that the gaze vector detecting means detects the right eye gaze vector from the reference point of the right eye, which can be seen by the user's monocular vision, and detects the left eye gaze vector from the reference point of the left eye.

According to another embodiment of the gaze point estimator of the present invention.
In order to detect the right eye gaze vector and the left eye gaze vector separately, it further has a gesture recognition means for recognizing a predetermined gesture of the user from the image.
It is also preferable that the gaze vector detecting means detects either the right eye gaze vector or the left eye gaze vector according to the gesture recognized by the gesture recognition means.

According to another embodiment of the gaze point estimator of the present invention.
It is also preferable that the gesture recognition means determines the right eye gaze vector detection, the left eye gaze vector detection, or the gaze vector switching detection by a predetermined gesture of fingertip operation.

According to another embodiment of the gaze point estimator of the present invention.
The gesture recognition means detects the opening and closing of the user's right eye and left eye as a predetermined gesture, and when the right eye is closed and the left eye is open, it is determined that the left eye gaze vector detection is performed and the left eye is closed. It is also preferable to determine that the right eye gaze vector is detected when the right eye is open.

According to another embodiment of the gaze point estimator of the present invention.
The pointer has two reference points and
When the user moves the first reference point to the first position pointing to the gaze point viewed from the right eye, the gaze vector detecting means moves from the right eye to the gaze point via the first reference point. When the user moves the second reference point from the left eye to a second position different from the first position, which points to the gazing point seen from the left eye at the same time as detecting the gaze vector of the right eye to be directed. It is also preferable to detect the left eye gaze vector toward the gaze point via the second reference point.

According to another embodiment of the gaze point estimator of the present invention.
The gaze vector detecting means includes a first reference point for right eye vision, a second reference point for right eye vision, a first reference point for left eye vision, and a left eye, which are visible by the user's binocular diplopia. With respect to the second reference point for vision, the two central reference points (the first reference point for right-eye vision and the second reference point for left-eye vision) are overlapped at a position that points to the gazing point seen by the user. It is also preferable to detect both the right eye gaze vector and the left eye gaze vector at the same time when they are moved.

According to another embodiment of the gaze point estimator of the present invention.
It is also preferable that the gaze vector detecting means functions only when the pointer is included in a predetermined range from the coordinate points of the user's right eye and the coordinate points of the left eye.

According to another embodiment of the gaze point estimator of the present invention.
The gaze point estimation means are a first cone having a central angle θ with the right eye as the apex and the right eye gaze vector as the axis, and a second cone having a central angle θ with the left eye as the apex and the left eye gaze vector as the axis. It is also preferable to estimate the volume region in which the above intersects and estimate the center of gravity of the volume region as the gazing point.

According to another embodiment of the gaze point estimator of the present invention.
An object recognition means that detects object identification information from an image from the local feature area of the object reflected in the image,
The gaze point estimated by the gaze point estimation means is compared with the local feature area detected by the object recognition means, and the object identification information associated with the local feature area including the gaze point is estimated as the object gazed by the user. It is also preferable to further have a gaze object estimation means.

According to another embodiment of the gaze point estimator of the present invention.
The gaze point estimation means are a first cone having a central angle θ with the right eye as the apex and the right eye gaze vector as the axis, and a second cone having a central angle θ with the left eye as the apex and the left eye gaze vector as the axis. Estimate the volume area where
It is also preferable that the gaze object estimating means estimates the object identification information associated with the local feature region as the object gazed by the user only when the volume region includes the local feature region of one object.

According to the present invention, when the user points the pointer at the gazing point, the computer functions so as to estimate the gazing point from the image in which the user, the pointer, and the gazing point are reflected.
A coordinate point detecting means for detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer from the image.
When the user moves the pointer to the first position that points to the gaze point seen from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected, and the user left eye. A gaze vector that detects the left eye gaze vector from the left eye to the gaze point via the pointer when the pointer is moved to a second position different from the first position, which points to the gaze point seen from Detection means and
It is characterized in that a computer functions as a gaze point estimation means for estimating the intersection of the right eye gaze vector and the left eye gaze vector as a gaze point as three-dimensional coordinates of the world coordinate system.

According to the present invention, the gaze point estimation method is executed by a device that estimates the gaze point from an image in which the user, the pointer, and the gaze point are reflected when the user points the pointer at the gaze point.
The device is
The first step of detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer from the image,
When the user moves the pointer to the first position that points to the gaze point seen from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected, and the user left eye. When the pointer is moved to a second position different from the first position, which points to the gazing point viewed from the left eye, the second eye gazing vector from the left eye toward the gazing point via the pointer is detected. Steps and
It is characterized by executing a third step of estimating the intersection of the right eye gaze vector and the left eye gaze vector as the gaze point as the three-dimensional coordinates of the world coordinate system.

According to the gaze point estimation device, the program, and the method of the present invention, when the user points the pointer at the gaze point, the gaze point can be estimated from the image in which the user, the pointer, and the gaze object are reflected.

It is the first external view which the user pointed at the fingertip toward the gaze point. It is a second external view which the user pointed at the fingertip toward the gaze point. It is explanatory drawing which shows the diplopia caused by one reference point when a user focuses both eyes on a gaze point. It is explanatory drawing which shows the monocular vision which occurs when the user focuses one eye on the gaze point. It is a functional block diagram of the gaze point estimation apparatus in this invention. It is explanatory drawing which shows the coordinate point detection using an optical marker. It is explanatory drawing which shows the coordinate point detection based on a skeleton analysis or an object recognition process. FIG. 5 is a first explanatory view showing diplopia caused by two reference points when the user focuses both eyes on the gazing point. FIG. 2 is a second explanatory view showing diplopia caused by two reference points when the user focuses both eyes on the gazing point. It is explanatory drawing which detects a gaze vector when a pointer is positioned in a predetermined range from both eyes of a user. It is explanatory drawing which shows the intersection of the right eye gaze vector and the left eye gaze vector.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

FIG. 1 is a first external view in which a user points a fingertip toward a gazing point.

According to FIG. 1, the camera 10 captures a three-dimensional image in which the user, the pointer, and the gaze object are reflected. The three-dimensional image is input to the gazing point estimation device 1 of the present invention.
The camera 10 may be a single depth camera capable of capturing RGB values and depth (depth distance) values. For example, as in Kinect (registered trademark), the depth distance to an object in the shooting environment can be measured by the TOF method using the infrared traffic time. In this case, the three-dimensional image is, for example, a point cloud in the three-dimensional space based on the depth value.

Further, the camera 10 can capture RGB (Red Green Blue) values, and may be a plurality of image cameras arranged at different positions. A three-dimensional image is created from two-dimensional images taken from different viewpoints.
In this case, it is necessary to calculate the coordinate conversion matrix to the fixed world coordinate system set in advance and the coordinate system photographed by the camera 10 by a well-known calibration in image processing. That is, it is necessary to acquire the geometrical positional relationship between the coordinate system of the two-dimensional image and the world coordinate system.

When the user points the fingertip toward the gazing point, the gazing object exists at the tip of the gazing vector.
According to FIG. 1, a part of the gaze object (object C) is hidden by other objects A and B from the user's field of view, and for example, when pointing with only a monocular, the gaze direction is specified. However, it is difficult to identify the gaze object because the depth distance to the gaze object is unknown.

FIG. 2 is a second external view in which the user points the fingertip toward the gazing point.

According to FIG. 2, the gaze object is visible from the user's field of view, for example, through glass (semi-transparent object). At this time, even if the image taken by the camera 10 is analyzed, if the user points the fingertip toward the gaze point using only a monocular, for example, the gaze object is glass or glass. It is difficult to identify whether it is a gaze object (object C) that can be seen through.

FIG. 3 is an explanatory diagram showing diplopia caused by one reference point when the user focuses both eyes on the gazing point.
According to FIG. 3, the user is looking at the gaze point of the gaze object with both eyes open. At this time, the pointer (for example, the fingertip) is seen by the user's eyes as a double left and right illusion due to diplopia.

FIG. 4 is an explanatory diagram showing monovision that occurs when the user focuses on the gazing point one eye at a time.
According to FIG. 4, the user points the pointer at the gaze object when viewed from the right eye. The user then points the pointer at the gaze object as viewed from the left eye. At this time, the position of the pointer seen from the right eye and the position of the pointer seen from the left eye do not match.

FIG. 5 is a functional configuration diagram of the gaze point estimation device according to the present invention.

According to FIG. 5, the gaze point estimation device 1 includes a coordinate point detection unit 11, a gaze vector detection unit 12, a gaze estimation unit 13, a gesture recognition unit 14, an object recognition unit 15, and a gaze object estimation unit. It has 16. These functional components are realized by a program that causes the computer mounted on the gaze point estimation device 1 to function. In addition, the processing flow of these functional components can be understood as a method of estimating the gaze point of the device.

[Coordinate point detection unit 11]
The coordinate point detection unit 11 detects the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer from the three-dimensional image captured by the camera 10.
Here, the "pointer" may be a predetermined human body part such as a fingertip, or may be an indicator such as the tip of a pointer.
That is, the coordinate point detection unit 11 detects the position of the predetermined human body part or the indicator detected from the three-dimensional image as the coordinate point of the “pointer”.

For example, there are the following methods for detecting each coordinate point.
<Coordinate point detection using optical markers>
<Detection of coordinate points by skeleton analysis>
<Coordinate point detection by object recognition processing>

<Coordinate point detection using optical markers>
FIG. 6 is an explanatory diagram showing coordinate point detection using an optical marker.

According to FIG. 6, markers are attached to the right eye portion and the left eye portion of the user and the pointer portion. The marker may be, for example, an optical marker that can be detected by chroma key processing on the image. As a result, the coordinate point detection unit 11 can detect the coordinate points in the right eye portion, the left eye portion, and the pointer portion for the user. According to the chroma key processing, the pointer can be detected by using an optical motion capture technique (see, for example, Non-Patent Document 2) that tracks an optical marker in an image.

The optical marker of the user's eye portion may be, for example, one worn on eyeglasses.
If the pointer is a fingertip, an optical marker may be attached to the fingertip. Further, if the pointer is an indicator (for example, a pointer), an optical marker may be attached to the tip of the indicator.

<Coordinate point detection based on skeleton analysis>
FIG. 7 is an explanatory diagram showing coordinate point detection based on skeleton analysis or object recognition processing.
According to FIG. 7A, when the pointer is a predetermined human body part, the coordinate point detection unit 11 detects the predetermined joint of the human body by skeleton analysis from the image with the predetermined human body part (for example, a fingertip) as the predetermined joint. Then, those coordinate points are specified.

Specifically, a skeleton model such as OpenPose (registered trademark) is used to extract feature points of human joints. OpenPose is software that can detect key points of body / hand / face between multiple people in real time from images, and is published by GitHub.
OpenPose is learned from a large amount of human images and skeletal posture information associated with them as teacher data, using characteristic parts based on the user's eyes and the skeleton and joints of the human body as reference points. By inputting an image, a person can be specified from the image and the skeletal posture information can be specified (see, for example, Non-Patent Document 1).

<Coordinate point detection based on object recognition processing>
According to FIG. 7B, the coordinate point detection unit 11 detects a predetermined human body part and an indicator by an object recognition process from a three-dimensional image (a two-dimensional image is also possible). Then, those coordinate points are specified.
Feature vector extraction algorithms used for object recognition are robust to rotation and scale changes, such as SIFT (Scale-Invariant Feature Transform) (see, for example, Non-Patent Document 5) and SURF (Speeded Up Robust Features) (for example, non-patent). Reference 6) is used. For example, in the case of SIFT, a set of 128-dimensional feature vectors is extracted from one image. SIFT is a technique for analyzing a characteristic local feature region using a scale space and describing a feature vector that is invariant to the scale change and rotation. On the other hand, in the case of SURF, faster processing than SIFT is possible, and a set of 64-dimensional feature vectors is extracted from one image. While SIFT has a high processing cost and difficult real-time matching, SURF speeds up processing by using an integrated image.

The object recognition processing unit is a well-known machine learning engine based on deep learning, and includes feature information based on images of a predetermined human body part to be detected and an indicator, and identification information (object name, identification number, etc.). (For example, see Non-Patent Document 8).
The amount of local feature regions such as SIFT or SURF is extracted from the image taken by the camera 10. The amount of local feature region has properties that are invariant to any combination of rotation, scaling, and projective transformation, and is calculated based on the relative brightness gradient in the local feature region of the image. A well-known ORB (Oriented FAST and Rotated BRIEF) feature having the same properties may be used (see, for example, Non-Patent Document 7).

[Gaze vector detection unit 12]
The gaze vector detection unit 12 detects the right eye gaze vector and the left eye gaze vector as follows.
When the user moves the pointer (for example, the fingertip) to the position pointing to the gazing point viewed from the right eye, the "right eye gazing vector" from the right eye to the gazing point via the pointer is detected.
When the user moves the pointer to the position pointing to the gazing point viewed from the left eye, the "left eye gazing vector" from the left eye to the gazing point via the pointer is detected.

The reference point of the pointer may be one case or two cases.
<One reference point of the pointer>
As shown in FIG. 4, when the pointer has one reference point, the gaze vector detection unit 12 sees the right-eye gaze vector from the right-eye reference point and the left-eye view, which can be seen by the user's monocular vision. The left eye gaze vector from the reference point of is detected separately.

<Two reference points for pointers>
FIG. 8 is a first explanatory view showing diplopia caused by two reference points when the user focuses both eyes on the gazing point.
For example, the fingertip of the right hand may be the first reference point of the pointer, and the fingertip of the left hand may be the second reference point of the pointer. Further, the first finger of the right hand may be used as the first reference point of the pointer, and the second finger of the same right hand may be used as the second reference point of the pointer.

According to FIG. 8, the user opens both eyes to focus the eyes on the designated object, and in this case, the two reference points that differ depending on the diplopia appear as a total of four illusions, each of which is double left and right. It becomes a state. For each reference point, the left side is the image of the right eye and the right side is the image of the left eye.

When the user moves the first reference point to a position pointing to the gaze point viewed from the right eye, the gaze vector detection unit 12 heads for the gaze point from the right eye via the first reference point. "Right eye gaze vector" is detected. At the same time, when the user moves the second reference point to the position pointing to the gaze point seen from the left eye, the "left eye gaze vector" from the left eye to the gaze point via the second reference point. Is detected.

FIG. 9 is a second explanatory view showing the diplopia caused by the two reference points when the user focuses both eyes on the gazing point.

According to FIG. 9, with respect to the reference points (fingertips) of the four phantom images arranged side by side, the user points the overlapping reference points toward the gazing point in a state where the two reference points located in the middle overlap.
Specifically, the first reference point for right-eye vision, the second reference point for right-eye vision, the first reference point for left-eye vision, and the left-eye vision that can be seen by the user's binocular diplopia. With respect to the second reference point of the above, the two central reference points (the first reference point of the right eye and the second reference point of the left eye) are overlapped at the position indicating the gazing point seen from the user. When moved, both the "right eye gaze vector" and the "left eye gaze vector" are detected at the same time.

FIG. 10 is an explanatory diagram for detecting a gaze vector when the pointer is located within a predetermined range from both eyes of the user.

According to FIG. 10, the gaze vector detection unit 12 functions only when the pointer is included in a predetermined range from the coordinate points of the user's right eye and the coordinate points of the left eye. For example, assuming a fan shape having a predetermined radius and a predetermined angle passing through the coordinate points of the right eye and the coordinate points of the left eye, the gaze vector is detected only when the pointer (fingertip) is detected in the range. As a result, it is possible to reduce the processing load for detecting the gaze vector even though the pointer is not in front of the user's face.

[Gaze point estimation unit 13]
The gaze point estimation unit 13 estimates the intersection of the right eye gaze vector and the left eye gaze vector as the gaze point (see FIGS. 3 and 8).

FIG. 11 is an explanatory diagram showing the intersection of the right eye gaze vector and the left eye gaze vector.

According to FIG. 11, the gaze point estimation unit 13 has a first cone having a central angle θ with the right eye as the apex and the right eye gaze vector as the axis, and a center with the left eye as the apex and the left eye gaze vector as the axis. Estimate the volume region where the second cone of angle θ intersects. For this, it is preferable to use the visual volume crossing method (see, for example, Non-Patent Document 4). Then, the gaze point estimation unit 13 estimates the center of gravity of the volume region as the gaze point.

[Gesture recognition unit 14]
The gesture recognition unit 14 recognizes a predetermined gesture of the user from the image in order to detect the right eye gaze vector and the left eye gaze vector separately (see, for example, Non-Patent Document 3).
On the other hand, the gaze vector detection unit 12 detects either the right eye gaze vector or the left eye gaze vector according to the gesture recognized by the gesture recognition unit 14.

Specifically, the gesture recognition unit 14 determines by two methods as follows, for example.
(1) The gesture recognition unit 14 determines the right eye gaze vector detection, the left eye gaze vector detection, or the gaze vector switching detection by a predetermined gesture of fingertip operation.
(2) The gesture recognition unit 14 detects the opening and closing of the user's right eye and left eye as predetermined gestures, and determines that the left eye gaze vector detection is performed when the right eye is closed and the left eye is open. When the left eye is closed and the right eye is open, it is determined that the right eye gaze vector is detected.

[Object recognition unit 15]
The object recognition unit 15 detects the object identification information from the local feature region of the object reflected in the image from the image taken by the camera 10. Local feature regions of one or more objects and their respective object identification information are detected from the image.
The object recognition unit 15 has exactly the same method as the object recognition process of the coordinate point detection unit 11 described above. The coordinate point detection unit 11 detects a predetermined human body part or an indicator, while the object recognition unit 15 detects a local feature region of an object and its object identification information.

[Gaze object estimation unit 16]
The gaze object estimation unit 16 compares the gaze point estimated by the gaze point estimation unit 13 with the local feature area detected by the object recognition unit 15, and object identification information associated with the local feature area including the gaze point. Is estimated as an object that the user gazes at.
This makes it possible to identify the object that the user has gazed at.

As described in detail above, according to the gaze point estimation device, the program, and the method of the present invention, when the user points the pointer at the gaze point, the image of the user, the pointer, and the gaze object is reflected. The gaze point can be estimated.

With respect to the various embodiments of the present invention described above, various changes, modifications and omissions in the technical idea and the scope of the present invention can be easily made by those skilled in the art. The above explanation is just an example and does not attempt to restrict anything. The present invention is limited only to the scope of claims and their equivalents.

1 Gaze point estimation device 10 Camera 11 Coordinate point detection unit 12 Gaze vector detection unit 13 Gaze point estimation unit 14 Gesture recognition unit 15 Object recognition unit 16 Gaze object estimation unit

Claims (19)

A gaze point estimation device that estimates the gaze point from an image in which the user, the pointer, and the gaze point are reflected when the user points the pointer at the gaze point.
From the image, the coordinate point detecting means for detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer,
When the user moves the pointer to the first position pointing to the gaze point viewed from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected. When the user moves the pointer to a second position different from the first position, which points to the gazing point viewed from the left eye, the left eye moves to the gazing point via the pointer. A gaze vector detecting means for detecting the gaze vector of the heading left eye,
A gaze point estimation device comprising a gaze point estimation means for estimating the intersection of the right eye gaze vector and the left eye gaze vector as the gaze point as the three-dimensional coordinates of the world coordinate system. The pointer is a predetermined human body part or an indicator, and is
The gaze point estimation device according to claim 1, wherein the coordinate point detecting means detects a predetermined human body part or an indicator as the pointer from the image. The gaze point estimation device according to claim 2, wherein the predetermined human body part is a fingertip. For the user, markers are attached to the right eye portion, the left eye portion, and the pointer portion.
The gaze point estimation device according to claim 2 or 3, wherein the coordinate point detecting means detects the marker by chroma key processing on the image. When the pointer is a predetermined human body part, the predetermined human body part is set as a predetermined joint.
The gaze point estimation device according to claim 2 or 3, wherein the coordinate point detecting means detects a predetermined joint of a human body by skeleton analysis from the image. The gaze point estimation device according to claim 2 or 3, wherein the coordinate point detecting means detects a predetermined human body part or an indicator from the image by an object recognition process. The image is a three-dimensional image taken by a camera.
The camera is characterized by being one depth camera capable of photographing RGB values and depth (depth distance) values, or a plurality of image cameras capable of photographing RGB values and arranged at different positions. The gaze point estimation device according to any one of claims 1 to 6. The pointer has one reference point and
The gaze vector detecting means is characterized in that it detects the right eye gaze vector from the reference point of the right eye, which is visible by the user's monocular vision, and detects the left eye gaze vector from the reference point of the left eye. The gaze point estimation device according to any one of Items 1 to 7. In order to detect the right eye gaze vector and the left eye gaze vector separately, a gesture recognition means for recognizing a predetermined gesture of the user from the image is further provided.
The gaze point estimation according to claim 8, wherein the gaze vector detecting means detects either the right eye gaze vector or the left eye gaze vector according to the gesture recognized by the gesture recognition means. apparatus. The gaze point estimation device according to claim 9, wherein the gesture recognition means determines the right eye gaze vector detection, the left eye gaze vector detection, or the gaze vector switching detection by a predetermined gesture of fingertip operation. The gesture recognition means detects the opening and closing of the right eye and the left eye of the user as a predetermined gesture, and determines that the left eye gaze vector detection is performed when the right eye is closed and the left eye is open, and the left eye. The gaze point estimation device according to claim 9, wherein when the right eye is closed and the right eye is open, it is determined that the right eye gaze vector is detected. The pointer has two reference points.
When the user moves the first reference point to the first position pointing to the gaze point viewed from the right eye, the gaze vector detecting means passes through the first reference point from the right eye. When the right eye gaze vector toward the gaze point is detected and at the same time, the second reference point is moved to a second position different from the first position where the user points to the gaze point seen from the left eye. The gaze point estimation device according to any one of claims 1 to 7, further comprising detecting a left eye gaze vector from the left eye toward the gaze point via a second reference point. The gaze vector detecting means includes a first reference point for right eye vision, a second reference point for right eye vision, and a first reference point for left eye vision, which can be seen by the user's binocular diplopia. With respect to the second reference point for the left eye, the two central reference points (the first reference point for the right eye and the second reference point for the left eye) are overlapped to point to the gazing point seen by the user. The gaze point estimation device according to claim 12, wherein both the right eye gaze vector and the left eye gaze vector are detected at the same time when the eye is moved to the above position. Any one of claims 1 to 13, wherein the gaze vector detecting means functions only when the pointer is included in a predetermined range from the coordinate points of the right eye and the coordinate points of the left eye of the user. The gaze point estimation device described in the section. The gaze point estimation means has a first cone having a central angle θ with the right eye as the apex and an axis of the right eye gaze vector, and a second cone having a central angle θ with the left eye as the apex and the left eye gaze vector as the axis. The gaze point estimation device according to any one of claims 1 to 14, wherein a volume region where the cone intersects is estimated, and the center of gravity of the volume region is estimated as the gaze point. An object recognition means for detecting object identification information from a local feature region of an object reflected in the image from the image, and an object recognition means.
The user compares the gazing point estimated by the gazing point estimating means with the local feature area detected by the object recognizing means, and the user obtains object identification information associated with the local feature area including the gazing point. The gaze point estimation device according to any one of claims 1 to 15, further comprising a gaze object estimating means for estimating as a gaze object. The gaze point estimation means has a first cone having a central angle θ with the right eye as the apex and an axis of the right eye gaze vector, and a second cone having a central angle θ with the left eye as the apex and the left eye gaze vector as the axis. Estimate the volume area where the cone intersects and
The gaze object estimating means is characterized in that the object identification information associated with the local feature region is estimated as the object gazed by the user only when the volume region includes the local feature region of one object. 16. The gaze point estimation device according to claim 16. A program that causes a computer to function to estimate the gaze point from the image in which the user, the pointer, and the gaze point are reflected when the user points the pointer at the gaze point.
From the image, the coordinate point detecting means for detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer,
When the user moves the pointer to the first position pointing to the gaze point viewed from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected. When the user moves the pointer to a second position different from the first position, which points to the gazing point viewed from the left eye, the left eye moves to the gazing point via the pointer. A gaze vector detecting means for detecting the gaze vector of the heading left eye,
A program characterized in that a computer functions as a gaze point estimation means for estimating the intersection of the right eye gaze vector and the left eye gaze vector as the gaze point as the three-dimensional coordinates of the world coordinate system. A gaze point estimation method performed by a device that estimates the gaze point from an image in which the user, the pointer, and the gaze point are reflected when the user points the pointer at the gaze point.
The device is
From the image, the first step of detecting the coordinate points of the world coordinate system for each of the user's right eye, left eye, and pointer,
When the user moves the pointer to the first position pointing to the gaze point viewed from the right eye, the right eye gaze vector from the right eye to the gaze point via the pointer is detected. When the user moves the pointer to a second position different from the first position, which points to the gazing point viewed from the left eye, the left eye moves to the gazing point via the pointer. The second step of detecting the heading left eye gaze vector,
A gaze point estimation method comprising executing a third step of estimating the intersection of the right eye gaze vector and the left eye gaze vector as the gaze point as the three-dimensional coordinates of the world coordinate system.

Images