JPH0351407B2 - - Google Patents

Description

[Detailed Description of the Invention] [Industrial Application Field] The present invention relates to a line of sight detection method, and in particular to detecting a line of sight and a gaze point in a non-contact manner by image processing. In the field of intelligent coding image communication where communication computer graphics are used, the results of gaze detection can be used to generate images. This invention relates to a line-of-sight detection method that can be used for display methods that generate high-quality images at high speed by providing feedback to the method.

[Background Art] Due to advances in computers in recent years, they have become capable of realizing complex and rich functions, and the scope of their application is increasingly expanding. Meanwhile, the range of users is expanding from experts to non-experts. This is where human interface technology becomes important, making it easier to use increasingly complex systems. In communication between humans, smooth dialogue is carried out not only by words but also by inferring the other person's intentions from facial expressions and gestures, and eye movements in particular play an important role in communication.

In other words, it is thought that a person's intention is largely reflected in the movement of the viewpoint. In computers and communication system interfaces, users face some kind of display device for long periods of time, so we believe that if we can constantly detect the movement of the viewpoint toward the display screen, it will be a great clue to extracting the user's intentions. It will be done. Simply put, where are users looking, what are they interested in, and what are their concerns?

Therefore, the inventors of the present application aimed to realize an interface function that understands the user's operating intention and returns a flexible response, and furthermore, in the field of intelligent communication, the purpose of We have pointed out the importance of an efficient method for detecting line of sight, with the aim of realizing receiver-centered image communication with a sense of reality by extracting the visual target and providing feedback to the communication side.

An eye camera has been known as a device for detecting the line of sight as described above. however,
Eye cameras require glasses to be worn, and in order to determine the viewpoint in the coordinate system of the display the user is facing, there are limitations such as the need to fix the head. It is not necessarily suitable for use in image communications.
In other words, without having the person wear anything special,
In order to detect line of sight without contact, a detection method using image processing is more advantageous.

FIG. 15 is a diagram showing an example of a possible structure of a non-contact line of sight detection method. Referring to FIG. 15, cameras 2 and 3 and lighting devices 4 and 5 are provided on both sides of display 1. In such a non-contact line of sight detection device, the first problem is to convert the image of the user illuminated by the lighting devices 4 and 5 into the camera 2,
3, and extracting a plurality of feature points necessary for line-of-sight detection from the photographed image. Second
The challenge is to measure the spatial positions of these feature points at high speed and with high precision. The third task is to determine the direction of the line of sight and the position of the point of view on the display from the position of the feature point.

[Problem to be Solved by the Invention] The first problem described above can be solved by using the "image photographing device" (Japanese Patent Laid-Open No. 138673/1999) proposed by the inventor of the present application, which can be expected to efficiently extract feature points. However, even if feature points can be extracted in this way,
Regarding the next second and third issues, namely, a method for detecting the user's line of sight without contact while allowing movement of the user's head, and a method for detecting the gaze point that the user is actually looking at, There are no practical presentation examples.

Therefore, the main object of the present invention is to detect the movement of the user's line of sight in a non-contact manner based on an image of a photographic subject photographed by an image photographing device, without requiring the user to wear any special equipment. An object of the present invention is to provide a line of sight detection method that can detect the line of sight.

Furthermore, another object of the present invention is to provide a line of sight detection method that can be applied as an input tool for a man-machine interface.

[Means for Solving the Problems] The present invention extracts feature points from a face image photographed by an image photographing device, detects the spatial position of the feature points in a desired coordinate system, and extracts feature points from a face image photographed by an image photographing device. This is a gaze detection method for detecting gaze, which includes a facial coordinate system generation step in which the spatial position and direction of the head are determined from a facial image in a desired coordinate system, and a facial coordinate system is created; an eyeball center estimation step for estimating the rotation center of the face; a detection step for detecting the spatial position of the eyeball center by converting the eyeball center expressed in the face coordinate system into a desired coordinate system; and a detection step for detecting the spatial position of the eyeball center from the face image. a step of detecting the spatial position of the eyeball feature point and determining the spatial position of the center point in a desired coordinate system; and a step of finding a line segment connecting the spatial position of the center of the eyeball and the spatial position of the eyeball feature point; The system is configured to detect a line segment or a line segment having a predetermined inclination with the line segment as a line of sight.

[Operation] The present invention is a line of sight detection method, in which the position and direction of the head are determined in a desired coordinate system, and by determining the position and direction, a face that can be uniquely converted to the desired coordinate system is obtained. A coordinate system can be provided. Once the face coordinate system is set, the center of rotation of the eyeball can be considered to be approximately a fixed point within the face coordinate system, and therefore the eyeball center can be treated as a constant vector within the face coordinate system. The constant vector can be determined by finding the three components x, y, and z in the face coordinate system.
There are various methods for determining this, but it can be determined by presenting the index to the user in advance and using the spatial position detection result of the feature point as described later. Since there are three unknowns, such calibration can be performed relatively easily.

Since the facial coordinate system and the desired coordinate system can be uniquely converted as described above, the position of the center of the eyeball expressed in the facial coordinate system can be converted into the desired coordinate system. When the position and direction of the face are detected in this manner, the position of the center of the eyeball is determined using a desired coordinate system.

On the other hand, when the spatial position of the center point of the pupil or iris (the eyeball feature point) is determined in a desired coordinate system, as described later, the line of sight is determined as a line segment connecting the spatial position of the eyeball center and the spatial position of the eyeball feature point. It can be obtained approximately. However, since this line segment includes an error as a line of sight, it is desirable to use a line segment that has a constant slope with the line segment as the line of sight.

In order to determine the position and direction of a face, it is sufficient if the face has at least three feature points with little movement. Once the spatial positions of the three points are determined in a desired coordinate system, a plane passing through the three points can be defined in that coordinate system. Once the surface is determined, the direction is determined. Further, the origin of the face coordinate system may be, for example, the center of gravity of three points. In this way, the face coordinate system can be determined from the spatial positions of the three points.

If it is difficult to find three fixed points on the face, three marks may be placed on the glasses. By wearing these glasses, you can easily model your face using three points.

[Embodiments of the Invention] First, before explaining the line of sight detection method, the line of sight will be explained.

FIG. 2 is a diagram schematically showing the structure of the eyeball. Referring to FIG. 2, the transparent membrane covering the front surface of the eye is the cornea 161, and most of the light is focused on the retina 162 by the cornea 161, but fine adjustment is required to align the viewpoint with the index. (Adjustment action) is performed by changing the thickness of the crystalline lens 163. cornea 1
61, retina 162 on the extension of the optical axis of crystalline lens 163
There is a part called the fovea 164 that has the highest resolution. When people gaze at an object,
The eye is moved so that the image is formed on this fovea 164. Therefore, the line of sight can be considered to be approximately on the imaging optical axis. In addition, the center of the pupil 165,
Since the center of rotation of the eyeball is also near this optical axis, the line of sight can basically be detected by determining the center position of the pupil and the center position of the eyeball.

Hereinafter, a line connecting the center of the eyeball and the center of the pupil will be approximated as a line of sight, and the detection algorithm will be described. In addition, correction when the direction of line of sight is strictly different is as follows.
I will explain later.

FIG. 1 is a diagram for explaining one embodiment of the present invention. Next, referring to FIG. 1, an outline of the line of sight detection method of the present invention will be explained. 172 is the center point of the pupil or the iris of the eye, and P1, P2, and P3 are feature points of the face. Pupils or iris, as well as feature points of the face, can be directly photographed with a camera, so they can be determined in a desired coordinate system (for example, world coordinate system) X, Y, Z by stereo measurement, which will be described later. e is the head coordinate system x, y, z as the face coordinate system
This is the eyeball center vector expressed by . Although the position of the center of the eyeball cannot be directly measured with a camera, since it is a fixed point in the facial coordinate system, it can be determined through calibration, which will be described later. Once the eyeball center vector e is determined, the position 171 of the center of rotation of the eyeball can be expressed in the world coordinate system as described later. Therefore, the line of sight is determined as a line segment connecting the rotation center position 171 of the eyeball and the center point 172 of the pupil or iris. If the Y- and Z-axis planes of the world coordinate system are the display screen surface facing the user, the point where the line of sight intersects with that surface can be determined as the point of gaze.

The details of the line of sight detection method will be explained below using equations.

In Figure 1, the head coordinate system is a triangle (P1,
The origin is at the center of gravity P _G (X _G , Y _G , Z _G ) of P2, P3),
The direction of the line segment connecting P1 and P2 is the x-axis, the direction orthogonal to the x-axis is the y-axis, and the normal direction of the triangle is the z-axis. The eyeball center position vector e is currently unknown, but is a constant vector. The position from the center of gravity _PG can be measured in some way. This vector can be determined by calibration, as described below. Therefore, in the following explanation, it is assumed that it can be given as input data. Therefore, the input data includes the three feature points P _i (X _i , Y _i , Z _i ) (i=1 to 3) on the face, and the eyeball center coordinates in the head coordinate system (x, y, z). e(x _e , y _e ,
z _e ). The output data is world coordinates C (X _C , Y _C , Z _C ) centered on the eyeball. Once this result is known, the direction of the line of sight is D(X _D , Y _D ,
Z _D )−C(X _C , Y _C , Z _C ).

The above proposition can be said to convert the head coordinate system (x, y, z) to the world coordinate system (X, Y, Z).

FIG. 3 is a diagram for explaining coordinate transformation involving movement of the origin and rotation of coordinate axes. Referring to FIG. 3, coordinate transformation involving movement of the origin and rotation of the coordinate axes is generally possible using the following equation (1).

x y z=RX-X _G Y-Y _G Z-Z _G (1) In addition, the inverse transformation is expressed by the following equation (2).

X Y Z=R ^-1 x y z+X _G Y _G Z _G …(2) Also, R ^-1 rotates the world coordinate system β around the Y axis, then α rotates it around the When rotated by γ in the direction to match the head coordinate system,
It is expressed by the following equation (3).

R ^-1 sinα・sinβ・sinγ sinα・sinβ・c
osγ cosα・sinβ＋cosβ・cosγ−cosβ・sinγ cosα・sinγ cosα・cosγ sinα・cosβ・sinγ sinα・cosβ・cosγ cosα・cosβ−sinβ・cosγ+sinβ・sinγ …(3) P _G (X _G , Y _G , Z _G ) is measured, so sinα,
All you need to know is sinβ, sinγ, cosα, cosβ, and cosγ.
The method for determining this will be explained below.

Figure 4 shows the feature point P1 in the face in Figure 1.
This is a diagram in which the origin of the world coordinate system has been moved. It is assumed that the z direction of the head coordinate system is equal to the normal vector N of the triangle formed by the three points on the face. The direction cosine of the normal vector of a triangle in the world coordinate system is given by the following equation (4).

n _x = cosθ n _y = cosφ n _z = cosφ (4) This value can be obtained as follows from the position coordinates of the three points on the face measured in the world coordinate system. In Figure 3, the equation of the plane is expressed as (5)
It is shown by the formula.

A・X+B・Y+C・Z+D=0...(5) X, Y, and Z are 3 points P _i (X _i , Y _i , Z _i ) (i=1~
_{3 ) , so X X 1 X 2} Since the equation has a solution other than A=B=C=D=0, the determinant of the coefficient is 0, and the plane equation is the following equation (7).

_{X X 1 X 2 _ _ _ _ _ _ _ _ _ _} Y ₃ Z ₃ 1 = Y ₁ (Z ₂ - Z ₃ ) - Z ₁ (Y ₂ - Y ₃ ) + (Y ₂ Z ₃ - _{Y 3}
Z ₂ )...(8) B=-X ₁ Z ₁ 1 X ₂ Z ₂ 1 X ₂ Z ₂ 1 X ₃ Z ₃ 1=-X ₁ (Z ₂ -Z ₃ )+Z ₁ (X ₂ -X ₃ ) −(X ₂ Z ₃ −X
₃ Z ₂ )… ₍ 9) C= X ₁ Y ₁ 1 X _{2 Y 2 1} X ₂ Y _{2 1 X 3} Y _{3 1} = +(X ₂ Y ₃ −X ₃
Y ₂ )…(10) D=−X ₁ Y ₁ Z ₁ X ₂ Y ₂ Z ₂ X ₂ Y ₂ Z ₂ X ₃ Y ₃ Z ₃ =−X ₁ (Y ₂ Z ₃ −Y ₃ Z ₂ )+Y ₁ (X ₂ Z ₃ −X ₃ −Z ₂ )−
Z ₁ (X ₂ Y ₃ −X ₃ Y ₂ )...(11) Therefore, the normal vector N of this plane 170
becomes as follows.

Also, the direction cosine of the normal vector of the triangle (n _x ,
n _y , n _z ) means that the world coordinate system is β around the Y axis.
It can be similarly determined from the conditions of rotating, then α rotating around the X axis, and aligning the Z axis with the Z time of the head coordinate system.

In FIG. 5, indicates that the world coordinate system is rotated by β around the Y axis. At this time, (X,
Y, Z) become (X', Y', Z') shown by the broken line.
Then, the world coordinate system is α around X′.
It shows that it rotates. With this operation,
The Z′ axis coincides with the z axis. Therefore, from FIG. 5, the direction cosine (n _x , n _y , n _z ) is as follows: n _x = cos α·sin β _ny = −sin α n _z = cos α·sin β (13). In this way, the direction of the z-axis is determined, and from equations (12) and (13), sinα,
1 to find sinβ, sinγ, cosα, cosβ, cosγ
Two conditions can be created.

Next, in Fig. 5, the world coordinate system is rotated by γ around the z-axis (i.e., N) of the head coordinate system as shown in , so that the X'-axis overlaps the x-axis of the head coordinate system, i.e. The X-axis and Y-axis should also be parallel to the head coordinate system.

If the angle between P1, P2 and the XZ plane is δ, then the fifth
As you can see from the figure, It is. Letting this be n, the rotation angles α, β, and γ are expressed by the following equation (15) from equations (12), (13), and (14).

sinα=−n _y , cosα=√1− ² _y sinβ=n×/√1− ² _y , cosβ=n _z √1− ² _y siniγ=m/√1− ² _y , cosγ=√1− ² _y −m/√1− ² _y …(15) If we substitute the value of equation (15) above into equation (3), we get
Find R ^-1 . According to equation (10), points in the head coordinate system are transformed to the world coordinate system. Therefore, the position of the center of the eyeball can be found in the world coordinate system.

Next, a method of calculating the line of sight will be explained. In the above-mentioned FIG. 1, the world coordinate value C at the center of the eyeball
Once (X _C , Y _C , Z _C ) is found, this and the pupil coordinate D
The line connecting (X _D , Y _D , Z _D ) is the line of sight. The vector of any point on the line of sight can be expressed by the following equation (17).

A=C+t・(D−C) (17) Here, t is a parameter indicating the distance to the visual target, and can be determined by various methods. If the display determined in the world coordinate system is a plane (for example, the YZ plane is the display surface in Figure 1), the origin and direction of the line-of-sight vector (D-C) have been detected, so this vector is The point that hits the display is uniquely determined by the display coordinates.

Further, in this invention, since the line-of-sight direction of the left and right eyes can be detected independently, detection is possible even if the gaze point is not on a plane. In other words, spatial points of interest can also be extracted. In other words, if the vectors to arbitrary points on the left and right lines of sight are A1 and A2, respectively, then the (17th)
From the equation, the following equation (18) is obtained.

A ₁ = C1 + t1・(D1−C1) A ₂ =C2+t2・(D2−C2) Here, C1, C2: Vectors at the center of the left and right eyeballs (D1−C1), (D2−C2); Left and right line of sight vectors t1 , t2: Parameter (t ₁ , t ₂ > 1) (18) If we use t1 and t2 as parameters and find the point where A1 and A2 are closest, that is the point of interest. In this way, in the present invention, the convergence angle of both eyes can be detected at the same time as the point of gaze.

Next, estimation of the eyeball center coordinates will be explained.
The position of the center of the eyeball is assumed to be fixed in the head coordinate system, and is used as input data in the above explanation. However, it is difficult to actually give the coordinates of the center of the eyeball accurately, so the following method is used to estimate it.

Now, let the subject know M points L _i with known world coordinates.
(X _i , Y _i , Z _i ), and the world coordinates of the pupil at that time D _i (X _Di , Y _Di , Z _Di ) and the world coordinates of the three feature points on the face that form the head coordinates.
P _1i (X _1i , Y _1i , Z _1i ), P _2i (X _2i , Y _2i , Z _2i ), P ₃ (X _3
i ,
Y _3i , Z _3i ). Using these measured values, the eyeball center coordinate e (X _e ,
Estimate the most probable value of Y _e , Z _e ) using the least squares method.

In Figure 6, the vector at the center of the eyeball in the head coordinate system is defined as e, and the center of the pupil when staring at point L _i , the center of gravity of the three feature points on the face, and the vector to point L _i are respectively expressed in the world coordinate system. In D _i , P _Gi , L _i
shall be. When the vector e is expressed in the world coordinate system, it is R _i ^-1 ·e, so as is clear from FIG. 6, the following equation (19) holds true.

L _i +t _i (D _i −L _i )=R _i ^-1 e+P _Gi (i=1 to M)...(19) t _i is a parameter (t _i >1), that is, shown in the following equation (20) It becomes a simultaneous equation.

Here, r _11i , r _12i . . . r _33i are each element of the rotation matrix R _i ^-1 , and are obtained by substituting equation (16) into equation (9). By eliminating t _i from equation (18), x _e , y _e ,
By rearranging z _e , the following equation (21) is obtained.

U _1i x _e +U _2i y _e +U _3i z _e +U _0i = 0 V _1i x _e +V _2i y _e +V _3i z _e +V _0i = 0 (i = 1 to N) ... (21) Here, U _1i , U _2i , U _3i , U _0i , V _1i , V _2i , V _3i ,
V _0i is obtained as measurement data. Therefore, regarding the solutions to the unknowns x _e , y _e , and z _e , the right-hand sides of equation (19) are set as δ _i and ε _i , respectively, and the sum of squared errors Σ (δ _i ² + ε _i ² is the minimum All you have to do is find it.

In the above description, the only parameter that must be determined through calibration is the eyeball center position in head coordinates. Therefore, in order to calibrate,
Only a small number of indicators need to be presented to the subject. Furthermore, since calibration is simple, there are fewer errors due to variations between subjects, and accuracy is increased. In addition, in the above description, the line of sight has been explained as connecting the pupil and the center of the eyeball, but there are cases where the line of sight connects the pupil and the center of the eyeball.

FIG. 7 is a diagram showing a case where there is no eyeball center on a line segment connecting the pupil and the fovea. Referring to Figure 7,
If the center of the eyeball is not on the line segment connecting the pupil and the fovea, assume the center of the eyeball and find a vector from this point to the fovea in some way, or use a constant vector to find it in the calibration method. Good too. In any case, it is clear that determining the center point of the eyeball with little change in position, regardless of the rotation of the eyeball, is effective for viewpoint detection, and the above-mentioned problem can be solved in the improvement of this invention. It is something. Further, if the subject wears glasses, correction is required taking into account the influence of the glasses lenses, but the same applies to this case, and the improvement of the present invention allows for calibration.

The method using pupil and corneal reflection images uses only feature points inside the eye. When the position of the light source is known and the structure of the eyeball is constant, the corneal reflection image is uniquely determined by the position of the center of the eyeball and the position of the pupil (corresponding to the rotation of the eyeball). Therefore, the position of the eyeball center can be determined from the corneal reflection image and the pupil position using the eyeball structure parameters. The line of sight can be determined from the center of the eyeball and the position of the pupil. The eyeball structure parameters are the radius of the eyeball, the radius of curvature of the cornea, and the distance between the center of the eyeball and the center of curvature of the cornea. These can be measured in advance or calibrated by having the user look at a specific part of the display during use.

Next, how to select feature points on the face will be explained. Users who face the display often do not change their facial expressions much. Therefore, there is less movement of the skin. Also, even if it does move, it often returns to normal immediately. For this reason, there are parts with little movement within a range that does not interfere with line-of-sight detection.
Specific examples include the inner corner of the eye, the tip of the nose, and the middle part of the upper lip. There is little change in these aspects unless the facial expression changes significantly. Additionally, since the color is different from the surrounding area, it is easy to extract features through image processing. That is, feature point parts can be identified by edge extraction and binarization processing. Note that in order to improve the accuracy of line-of-sight detection, it is better to greatly enlarge the area around the eyes when photographing, so it is desirable to select facial feature points from around the eyes. There are many spots like this around the eyes. Furthermore, if it is desired to detect the line of sight at high speed through simple image processing, markers may be pasted at such points.

FIG. 8 is a diagram showing an example of a model of a head skeleton using eyeglasses with markers. If it is difficult to put a marker on the face, use glasses 18 as shown in Figure 8.
Marker 182 at 3 locations around frame frame 1
By attaching the glasses 181 and wearing the glasses 181, the head skeleton is modeled using the markers 182. Note that the method for extracting the marker 182 will be described later, as well as the method for extracting the pupil. In the example shown in FIG. 8, the case where modeling is performed using three points on the face has been described, but modeling may be performed using even more points.

FIG. 9 is a diagram modeled using the fourth point. In this example, when the first triangle 183 cannot be found due to failure in feature point extraction, the first triangle 183
Instead, the center of the eyeball is determined using the second triangle 184. In fact, because a person's head moves in a variety of ways, each feature point may not always be captured by the camera. Therefore, in practice, by preparing a plurality of triangles and switching them at any time, it is possible to continuously detect the line of sight.

Furthermore, there are not many points on the face where the skin moves little. It may be difficult to extract such points. Therefore, it is conceivable to capture four or more feature points on the face as much as possible, recalibrate three or more virtual points with little change from these points, and model the head skeleton using these points.

FIGS. 10 and 11 are diagrams showing an example of determining an immobile virtual feature point from two facial feature points. The positions of the outer and inner corners of the eyes change when the eyes are squinted as shown in FIG. 11 compared to when the eyes are open as shown in FIG. 10. But if you squint on both eyes at the same time (which is what people usually do),
The positions of the centers of the two eye corner feature points hardly change. The same can be said about the corners of the nose, eyelashes, etc. The skin is moved by facial muscles, so when a certain area stretches, there is always a part that contracts.
In this way, by preparing facial expressions and skin movements as a database, it is possible to virtually obtain feature points with little skin movement. By using virtual feature points in this way, the accuracy of line of sight detection is further improved. In the future, it would be desirable to capture facial expressions and gaze at the same time.

FIG. 12 is a diagram for explaining the stereo image measurement method. Referring to Figure 12, (X^, Y^, Z^)
is the world coordinate system 161, and it is assumed that it overlaps with the display screen. The mirror 164 is inserted at a predetermined position in the Z-axis direction of the world coordinate system 161 parallel to the XY plane only when calibrating the stereo measurement system. (X, Y, Z) is the world coordinate system 16 obtained using the mirror 164
It is a coordinate system that is surface symmetrical to 1, and the symmetrical coordinate system 162
It is called. (x, y, z) is the coordinate system 163 of the photographing camera, and the origin o is the viewpoint of the lens.
An imaging plane 165 is located at a focal length f from the viewpoint o.

The object photographed by the camera is an object in a symmetrical coordinate system 162, and the coordinates of this coordinate system 162 need to be transformed into the world coordinate system (X^, Y^, Z^). Although details will be described later, this coordinate transformation matrix can be obtained by mirror-symmetric scanning using the mirror 164 introduced as described above. Furthermore, the symmetrical coordinate system 162 is a coordinate system relative to the display, and in an interface type environment, the user is assumed to be located approximately in this coordinate system o.

In this invention, the final purpose is to utilize a line-of-sight interface, so it is necessary to detect the line-of-sight in the display coordinate system, and it is necessary to accurately find the feature points of the person in the display coordinate system. In general, calibration is required to convert the positions of feature points captured in the camera coordinate system 163 into a coordinate system different from the camera coordinate system 163. To perform calibration, a method is used in which a point whose position is known in advance in the coordinate system to be converted is prepared, captured by this camera, and the conversion parameters are determined. For the purpose of this invention, we want to measure using the display coordinate system, so basically we only need to display a clear point on the display and capture this point with a camera for calibration. However, in this case, the camera naturally faces the direction of the display, and cannot capture the user facing the display.

Therefore, as shown in FIG. 12, a mirror 164 is provided in front of the world coordinate system 161 during calibration, and the virtual image on the display reflected on this mirror 164 is used as data for calibration. By capturing this virtual image with a camera and using the above-described algorithm, the spatial position in the world coordinate system 161 can be accurately determined. That is, in FIG. 12, point P^ of world coordinate system 161 is reflected by mirror 164, and its mirror image appears at point P of symmetric coordinate system 162. The light is reflected by the mirror 164 as shown by the arrow in FIG. 1, but the camera coordinate system 16
When viewed from 3, the symmetrical coordinate system 162 as shown by the arrow
It is measured as light from .

Next, the mirror 164 is set in the world coordinate system 161
, so that the position and direction are known in advance. In this way, the transformation between the world coordinate system 161 and the symmetric coordinate system 162 becomes easy.
For example, as described above, if a mirror 164 is placed in which the world coordinate system 161 is orthogonal to the Z-axis, the spatial position coordinates in the symmetrical coordinate system 162 can be converted to the world coordinate system 161 by simply reversing the sign in the Z^-axis direction. I can do it. In other words, the world coordinate system 161
If it is easy to convert between It becomes possible.

Therefore, in the following description, a method will be described in which the spatial position of the feature point P in the symmetrical coordinate system 162 in FIG. 12 is determined based on the two-dimensional position data of the feature point captured by a camera.

First, consider a point P in space. This point P is (ξ, η) in camera imaging plane coordinates and camera coordinates (X,
The relationship between Y, Z) and symmetrical coordinates (X, Y, Z) will be described. As is clear from Fig. 12, x y z = - (ξ/f) z - (η/f) z z ... (22) f is the distance between the lens principal point and the imaging surface (=focal length) It is. In other words, when the coordinates of the image on the imaging plane are known (although they are known by measurement), only that direction of the subject is known, and the direction vector of point P in the camera coordinate system 163 is expressed as x→ in FIG. =oP→=r→・tr r→=−(ξ/f) −(η/f) 1 (23) (t is a parameter). Let the coordinates of point P in the symmetric coordinate system 162 be P
(X, Y, Z), vector oP→=X→=X Y Z...(24) The coordinates in the camera coordinate system 163 are P(x,
y, z), vector Op→=x→=x y z...(25) If the coordinates of the image of point P on the film are p'(ξ, η), then from equation (23), x→= r→・t=R・(X−X ₀ ) That is, x y z=−ξ/f −η/f 1・t=R・X−X ₀ Y−Y ₀ Z−Z ₀ …(26) t is a parameter The inverse transformation is from equations (23 ₎ and (24): X→= ^{R -1} x→+X _{→ 0} Z ₀ ... (27) In the above equation (27), ξ, η, X, Y,
Given Z as input, it is a positional parameter
Consider obtaining R ^-1 , f, X ₀ , Y ₀ , and Z ₀ as outputs. That is, the symmetric coordinate system 162 (X,
A method for determining the tilt of the camera position and the focal length in Y, Z) will be described below.

N points P _i (X _i , Y _i , Z _i ) (i
= 1 to N), and the position coordinates (ξ _i ,
By measuring η _i ) (i = 1 to N), the camera position (X ₀ , Y ₀ , Z ₀ ) and the rotation angle (α, β, γ ) and focal length f.

Expanding the above equation (26), we get the following equation (28)
The formula is obtained.

−ξ _i /f・t _i =R ₁₁ (X _i −X ₀ )R ₁₂ (Y _i −Y
₀ ) + R ₁₃ (Z _i −Z ₀ ) −η _i /f・t _i =R ₂₁ (X _i −X ₀ )R ₂₂ (Y _i −Y
₀ ) + R ₂₃ (Z _i - Z ₀ ) t _i = R ₃₁ (X _i - X ₀ ) + R ₃₂ (Y _i - Y ₀ ) + R ₃₃
(Z _i −Z ₀ ) (i=1 to N) (28) R _ij (i, j=1 to 3) are each element of the rotation matrix, and are as follows.

R ₁₁ = cosβ・cosγ R ₁₂ = sinα・sinβ・cosγ・cosα・sinγ R ₁₃ = −cosα・sinβ・cosγ・sinα・sinγ R ₂₁ = −cosβ・sinγ R ₂₂ = −sinα・sinβ・sinγ＋cosα・cosγ R ₂₃ = cosα・sinβ・sinγ+sinα・cosγ R ₃₁ = sinβ R ₃₂ = −sinα・cosβ R ₃₃ = −cosα・cosβ …(29) The third equation of the above equation (28) is transformed into the first equation and By substituting into the second stage equation and eliminating t _i , the following equation (30) is obtained.

R ₁₁ +ξ _i /f・R ₃₁ ) (X _i −X ₀ ) + (R ₁₂ +ξ _i /f・R ₃₂
)(Y _i −Y ₀ )+(R ₁₃ +ξ _i /f・R ₃₃ )(Z _i −Z ₀ )=0 R ₂₁ +ξ _i /f・R ₃₁ )(X _i −X ₀ )+(R ₂₂ +ξ _i /f・R ₃₂
)(Y _i −Y ₀ )+(R ₂₃ +ξ _i /f・R ₃₃ )(Z _i −Z ₀ )=0 (i=1~N) …(30) Here, the unknown quantity to be found is (X ₀ , Y ₀ , Z ₀ ),
(α, β, γ) and f. Therefore, N≧4 must be satisfied. Actually, as shown in the solution method below, the most probable value is determined by the least squares method, so it is desirable that N be large to some extent. The solution method will be explained here. Due to measurement errors, etc., the left side of equation (30) does not actually become 0, causing errors σ _i and ε _i . Therefore, the sum of squares of the error Σ
A solution can be found by minimizing (σ _i ² +ε _i ² ).

In this way, by photographing multiple points whose spatial positions are known in advance with a camera, the camera coordinates, camera direction, focal length, etc. can be set in a symmetric coordinate system.
It is found by 62. Since the symmetrical coordinate system 162 can be easily converted to the world coordinate system 161 as described above, all the parameters of the measurement system can be determined with respect to the world coordinate system 161 after all.

Next, three-dimensional position measurement will be explained.

FIGS. 13 and 14 are diagrams for explaining a method for determining the three-dimensional position of the world coordinate system using images taken by two arbitrarily set cameras.

Referring to FIG. 13, the two cameras 165 and 166 in the world coordinate system are based on the coordinates of the camera lens principal point (X^ _0i , Y^ _0i , Z^ _0i ), the optical axis direction, and the camera 1
Rotation angles (α _i , β _i , γ _i ) that determine the inclination of 65,166
and focal length f _i (i=1 to 2) are known.
Let the coordinates of images of P (X^, Y^, Z^) of the world coordinate system 161 taken by two cameras 165 and 166 be (ξ ₁ , ε ₁ ) and (ξ ₂ , ε ₂ ), respectively. Image (ξ _i
,
Assuming that the direction vectors from η _i ) to the lens principal point are A ₁ and A ₂ respectively, P ₁ = A ₁ t ₁ + X ₀₁ P ₂ = A ₂ t ₁ + X ₀₂ (t ₁ and t ₂ are parameters)...( 31) A ₁ and A ₂ are obtained from equation (27) as A ₁ = R ₁ ^-1 −ξ ₁ /f ₁ −η ₁ /f ₁ 1A ₂ = R ₂ ^-1 −ξ ₂ /f ₂ −η ₂ / f ₂ 1 …(32). A ₁ , A ₂ , X ₀₁ , and X ₀₂ are known values determined by the above-mentioned calibration. Therefore, if t ₁ and t ₂ are determined, P ₁ and P ₂ are determined. If there is no error, P ₁ , P ₂
match, and equation (31) can be solved by setting P ₁ = P ₂ . From equation (31), t ₁ and t ₂ can be determined. However, in reality, P ₁ and P ₂ do not match due to measurement errors, so the closest point is found.
In this way, any spatial position can be measured in the world coordinate system 161 using the two images.

[Effects of the Invention] As described above, according to the present invention, the spatial position and direction of the head are detected from a face image, the rotation center of the eyeball is estimated, and the eyeball center is converted to a desired coordinate system. Detect the spatial position of the center of the eyeball, extract the pupil or iris, find the spatial position of the center point in a desired coordinate system, and extract the line segment or line segment connecting the spatial position of the center of the eyeball and the spatial position of the eyeball feature point. By detecting a line segment with a predetermined inclination as a line of sight, approximately one constant vector is required as a necessary parameter, and calibration is easy. By determining the spatial position of the eyeball center and pupil, it is possible to The direction of the user's line of sight can be determined regardless of the movement of the user's head. Moreover, the line of sight and point of gaze can be detected in the camera coordinate system or the final coordinate system, and the human
It can be used in various ways as a line-of-sight input device in the field of machine interfaces.

[Brief explanation of drawings]

FIG. 1 is a diagram for explaining one embodiment of the line of sight detection method according to the present invention. FIG. 2 is a diagram schematically showing the structure of the eyeball. FIG. 3 is a diagram for explaining coordinate transformation involving movement of the origin and rotation of the coordinate axes. FIG. 4 is a diagram showing a state in which the origin of the world coordinate system, which is the feature point _P1 in the face, has been moved in FIG. FIG. 5 is a diagram showing a state in which the world coordinate system is rotated. FIG. 6 is a diagram for explaining a vector centered on the eyeball in the head coordinate system. FIG. 7 is a diagram showing a case where the center of the eyeball is not on the line segment connecting the pupil and the fovea. 8th
The figure shows an example of a model of a head skeleton using eyeglasses with markers. FIG. 9 is a diagram showing an example in which the head skeleton is modeled at four points. Figure 10 and 1
1 is a diagram showing an example of determining an immovable virtual feature point from two feature points. FIG. 12 is a diagram for explaining the stereo image measurement method. Figure 13 and 1
FIG. 4 is a diagram for explaining a method for determining a three-dimensional position using images taken by two cameras arbitrarily set with respect to the world coordinate system. FIG. 15 shows an example of a conventional non-contact line of sight detection device. In the figure, 161 is the world coordinate system, 170
is a plane, 171 is the eyeball center point, 181 is the glasses, 1
82 is a marker, and _PG indicates the center.

Claims (1)

[Claims] 1. A line of sight detection method for detecting line of sight in a desired coordinate system based on a face image photographed by an image capturing device, the method comprising: determining the spatial position and direction of the head from the face image; a face coordinate system generation step of calculating in the desired coordinate system and creating a face coordinate system; an eyeball center estimation step of estimating the center of rotation of the eyeball in the face coordinate system generated in the face coordinate system generation step; a detection step of detecting the spatial position of the eyeball center by converting the eyeball center expressed in the coordinate system to the desired coordinate system; a step of detecting the spatial position of the eyeball feature point to be determined in a desired coordinate system; and a step of determining a line segment connecting the spatial position of the center of the eyeball and the spatial position of the eyeball feature point, the step of determining the line segment or a predetermined inclination thereof. A line-of-sight detection method characterized by detecting a line segment with as a line-of-sight. 2. The face coordinate system generation step extracts at least three feature points with little movement of the skin on the face from the face image, determines the spatial position of the extracted points in the desired coordinate system, and generates the face coordinate system. The line of sight detection method according to claim 1, characterized in that: 3 In addition, at least 3
3. The line-of-sight detection method according to claim 2, characterized in that glasses having marks for easy feature extraction are used for extracting feature points of the face.