JP4839432B2 - Pointing device and method based on pupil position detection - Google Patents

Description

  The present invention relates to a pointing device that can replace a mouse as an input device in a computer.

Conventional method and problems:
Various pointing methods based on the viewpoint (user's line of sight) have been proposed so far (for example, http://www.ericainc.com/ (Human-computer interaction using eye-gaze input.pdf), http: //www.eyegaze.com/ etc.)
However, when pointing depending on the viewpoint, the viewpoint must be moved to the point on the display device where you want to point, but the cursor always appears at the position of the viewpoint while searching for the position to be viewed. There is a problem that it is difficult to move the cursor to the desired position. In addition, when a cursor is always presented at a viewpoint position, it becomes a visual stimulus, and there is a problem that it is difficult to see an object to be seen.

According to the movement of the head, the position where the cursor appears is different from the one you are looking at, and the cursor is at the target position while looking at the cursor in peripheral vision while looking at the necessary place on the front display device Can be moved.
Example of conventional pointing system with head movement:
・ Eagle eye (refer to Non-Patent Document 1 and Web address http://www.cs.bc.edu/~gips/CM/)
Eagle eye installs a video camera on the side of the computer, captures a face image, tracks part of the face specified in advance by template processing by image processing, and on the computer screen according to the movement This is a method of moving the cursor. As the tracking target, the tip of the nose, the tip of the chin, the black eye, etc. are tried. However, even if the template is updated, there is a possibility that it may be difficult to cope with a sudden decrease in ambient brightness, and problems such as a gradual shift in the target to be tracked are likely to occur.
・ Origin Instruments Corporation (see http://www.orin.com/access/)
The Origin Instruments method is a kind of stereo camera that detects the movement of a marker attached to the head and moves the cursor accordingly. The problem with this is that the markers must be attached one by one. For this reason, it is generally difficult to use.

-Attach a marker made of a sticker to the face of the face, etc., illuminate the face with an infrared light source installed next to the video camera, detect the marker, detect the two-dimensional position of the marker that moves with the movement of the head, There is a method to move the cursor on the PC screen. The problem with this is that the marker must be attached to the face.
-Method using a magnetic sensor: This is inconvenient because it is necessary to attach a magnetic sensor to the head.
・ Method using ultrasonic waves: It is necessary to attach an ultrasonic sensor to the head.
Further, these prior arts cannot perform an operation equivalent to a click with a mouse, and it is necessary to use another auxiliary input device for the click operation.
"The Camera Mouse: Visual Tracking of Body Features to Provide Computer Access for People With Severe Disabilities", (Margrit Betke, James Gips, and Peter Fleming), IEEE Transactions on Neural Systems and Rehabilitation Engineering, Vol. 10, No. 1, March 2002

Method summary:
The present invention illuminates the face with an invisible infrared light source, shoots with a video camera at a magnification that makes the entire width of the face easier, and directly detects two or one pupil in the image taken by the video camera It is a method. The cursor on the monitor such as a personal computer is moved according to the movement of the pupil in the image caused by the movement of the head, the menu displayed on the monitor is selected, and characters are input using the keyboard software. Or switch between multiple windows. Further, the device can generate a signal corresponding to pressing of the left and right buttons of the mouse, and there is no need for another input operation device for click operation.

In pupil detection, a light source installed near the optical axis of the camera and a light source installed far away are turned on alternately for each field, and a bright image and a dark image of the pupil are obtained, and image difference calculation is performed. This facilitates pupil detection.
The principle of obtaining bright and dark images of the pupil is based on the phenomenon that the light reflected from the lens of the eyeball is reflected by the retina and returns to the original light source. This is based on the knowledge that the reflected light cannot be recognized even when observed at a deviated position.

System configuration:
FIG. 1 shows the configuration of the entire system.
The camera 1 is a general-purpose CCD video camera sensitive to near-infrared rays, one light source 2 near the opening of the camera 1 (portion through which light passes), and another one away from the opening of the camera. Two light sources 3 are provided. The camera 1, the light source 2 and the light source 3 are connected to the control device 4. The control device 4 is connected to a personal computer (hereinafter referred to as “personal computer”) 5 having a monitor 6. The connection interface is preferably a PCI bus interface when the control device is inserted into the PCI slot, and is preferably a USB interface when an external adapter is used.
Furthermore, it is also possible to use the personal computer itself as a control device, and in that case, it is conceivable to provide a light source irradiation (flashing) control part separately.
It is possible to divide the control functions of the control device as desired, and to distribute functions as appropriate, such as a part executed by software, a part executed by hardware, a part executed by the personal computer itself, and a part executed by the PCI board. Will be understood by those skilled in the art.

The target person (user) 7 is caused to alternately emit the light source 2 in the vicinity of the camera and the light source 3 away from the camera, and a reflected image from the pupil is captured by the camera 1. The controller 4 calculates the difference between the image from the light source 2 and the image from the light source 3, and transfers the obtained change in the pupil position to the personal computer 5 through the USB interface. In the personal computer 5, the signal from the control device 4 is regarded as equivalent to a mouse which is a kind of pointing device, and the position of the cursor 8 displayed on the monitor 6 is controlled.
Further, the control device 4 detects an operation of closing only one eye and generates a signal corresponding to pressing of the left and right buttons.
Here, the control device controls the light emission of the light source, but there is no problem in operation as long as it can capture an image signal in synchronization with light emission / extinction of the light source, for example, in synchronization with the commercial frequency. A flashing light source and a control device that captures an image in synchronization with the commercial frequency can be used.

A system constituted by the camera 1, the light source 2 in the vicinity, the light source 3 away from the camera, and the control device 4 has the same function as a normal personal computer mouse. Therefore, it is preferable to output signals corresponding to the direction and speed of movement, two buttons, etc., in exactly the same format as in an ordinary mouse. Note that this pupil mouse and normal mouse can be used in combination. Connect this system to another personal computer via a USB cable. Therefore, it looks like a normal mouse for a personal computer. If you connect a normal mouse to the same computer at the same time, you can use this system and the mouse together.
Here, although the connection interface is exemplified as USB, PS / 2, RS232C, a parallel port, etc. can be used without being limited to this.

In order to input characters with this system, a software keyboard is used as described below.
A keyboard input tool as shown in FIG. 2 is displayed on the display screen of the personal computer by software. Move the cursor to the desired character and click to enter the character into general application software such as Word (trademark) (Web address http://orin.com/access/softype/index. see htm). If you move the cursor to a place other than this keyboard input tool, you can do the same as a normal mouse operation.

System configuration for pupil position detection:
At present, a general-purpose CCD video camera having sensitivity to near infrared rays is used as the camera 1. As shown in FIG. 1, one light source 2 is provided near the opening (portion through which light passes) of the camera 1 and another light source 3 is provided away from the opening of the camera.
Two types of actual light sources are shown in FIG. In FIG. 3A, a light emitting diode (LED) 11 is concentrically arranged in the nearest region 12 of the opening (lens) 10 of the camera, and the light emitting diode 11 is also arranged concentrically in the outer peripheral region 13. An example is shown.

  In FIG. 3B, the light emitting diodes 11 are arranged concentrically in a region 14 near the opening 10 of the camera, and the light emitting diodes 11 are arranged by providing a circular region 15 at a position slightly separated from the camera opening 10. An example is shown. As described below, the positions of these light emitting diodes are illumination for obtaining reflected light from the pupil and illumination for obtaining an entire face image necessary for performing a difference calculation. It is not preferable that there is a difference in the face image due to the above, and since illumination from a distant position is not for obtaining reflected light from the pupil, it is desirable to arrange the face image at an appropriate position.

An image obtained by actually capturing the subject 7 is shown in FIG.
As shown in FIG. 4, the image of the camera 1 is basically taken to have a size that allows two pupils to sufficiently enter the frame. In this state, when a light source close to the opening of the camera is illuminated, the pupil appears bright as shown in FIG. 4 left (a) (bright pupil). On the other hand, when a light source far from the aperture of the camera is illuminated, the pupil appears dark as shown in FIG. 4B (dark pupil). However, a small point in the pupil seen in these two images is called corneal reflection in which each light source is reflected on the corneal surface. In this method, corneal reflection is not used.

  Now, the above-mentioned two light sources are synchronized with the odd and even fields of the video signal of the camera, the two light sources are alternately illuminated, and images of the bright pupil and the dark pupil are alternately obtained. When these images are differentiated, an image as shown in the right part (c) of FIG. 4 can be obtained. In this difference image, the pupil appears to protrude from the other part (background), and two pupils can be detected relatively easily.

Response to eye opening and closing information and mouse signals:
The output of the pointing device by moving the pupil corresponds to a normal mouse signal, so a signal indicating the amount of mouse movement, whether the right button is pressed, and whether the left button is pressed are output. The
The recipient's computer determines whether the user left-clicked or dragged. In addition, since the click time etc. can be adjusted in the receiver's personal computer, it is not necessary to consider this point in this apparatus.

  Basically, when the left eye or right eye wink is quickly winked, it corresponds to a left click or right click of the mouse, respectively, and when the head is moved while the left eye is closed, it corresponds to dragging. This is equivalent to simply moving the mouse when moving the head with both eyes open. However, you must be careful when you close both eyes in a blink. Fortunately, with a two-button mouse, it is not necessary to press both buttons. Therefore, when both eyes are closed, it is determined that the eye is blinking, and in that case, it is ignored.

To be more specific, it can be summarized as follows.
・ When the pupils of both eyes (both pupils) are detected, the average coordinate of each pupil center coordinate is calculated, and a signal corresponding to the change from the same coordinate in the immediately preceding frame is output.
・ When the pupil of the left eye (left pupil) is detected and the pupil of the right eye (right pupil) is not detected, this is equivalent to pressing the right button. The change in the left pupil coordinate between frames is output as a signal corresponding to the cursor movement.
・ When the right eye pupil (right pupil) is detected and the left eye pupil (left pupil) is not detected, this corresponds to the left button being pressed, and the corresponding signal is output at the same time as the previous frame. The change in the right pupil coordinate between frames is output as a signal corresponding to the cursor movement.
-When both pupils are not detected, if a blink occurs and a signal indicating that the coordinate change signal between the previous frame and the current frame is zero is output as a signal corresponding to zero cursor movement At the same time, it outputs a signal indicating that both buttons are not pressed, and waits for both eyes to appear.

This is illustrated in FIG. FIG. 5A shows the movement of the cursor when both pupils are detected. The left pupil 20 and the right pupil 22 exist as initial positions, and the movement is detected at the positions of the left pupil 21 and the right pupil 23 by the movement of the face. In the control device 4, the average coordinates of the initial positions of both pupils are obtained, and the average coordinates after movement are further obtained. A cursor movement signal is generated based on the difference between the two obtained average coordinates. The personal computer 5 moves the cursor 24 on the monitor to the cursor 25 which is a new position based on the cursor movement signal.
In the case of simplification, the cursor position may be controlled only by detecting the position of one pupil instead of obtaining the average coordinates of both pupils. This is because there is no practical problem because there is no drag operation by pressing the right button on the mouse.

FIG. 5B shows a case where one pupil (left pupil) disappears and movement of the other pupil (right pupil) is detected. The left pupil 26 disappears at this position, and the right pupil 22 is moved to the right pupil 23 and a new position is detected. In this case, it is determined that the subject (user) has instructed dragging with the cursor, and the control device 4 drags from the position of the cursor 24 to the position of the cursor 25 as a movement while pressing the left button of the mouse. Create a signal as.
Here, the disappearance of the left pupil is equivalent to pressing the left button of the mouse, and the disappearance of the right pupil is equivalent to pressing the right button of the mouse, but this correspondence can be arbitrarily set, and is limited to this. is not.

Detection of two pupils in the image and detection of closed eyes:
First, the luminance information is used to find the approximate location of the pupil. Next, a window is given around it, and a luminance histogram in the window is created. In the luminance histogram, two peaks (mountains) representing a pupil part and a background part appear. Therefore, binarization is performed using an appropriate value as a threshold value between two peaks. This threshold determination is performed automatically. From this, the number of pixels in the pupil is counted. The number of pupil pixels is used corresponding to the pupil area.

  Now, if the eyes are closed and there are no pupils in the image, it is difficult to determine that the pupils are certainly gone. Therefore, in order to ensure the detection of the pupil, if two pupils are detected when both eyes are open, a prediction model such as a Kalman filter is used to predict the pupil position of the next frame. A window is given near the pupil position expected from. Then, the threshold value is automatically determined only in the window as described above, the pupil part is judged, and the pupil center is determined.

  However, the closed eye is determined using the fact that when the eye is closed, the luminance histogram in the window only shows one peak of the background. If only one eye is closed, only the pupil window that is closed is released. In the next frame, the window given to the open pupil in the previous frame is maintained, and pupil detection is attempted from the entire other image. Then, a window is given to the position listed as a candidate for the pupil, and it is analyzed in the same manner in the window. When it is determined that the pupil part exists, a new window is given there, and the eyes that have been closed are opened. It is judged. However, if it is determined that the pupil portion does not exist, it is assumed that the eye is still closed, and the pupil window that is still open is maintained.

If both eyes are closed, both windows are released. Also in the next frame, two candidate pupil positions are searched from the entire image. However, in each window, the presence or absence of the pupil is determined by the same method.
If it is determined that neither pupil is present (both eyes closed), the same operation is repeated in the next frame. However, when both eyes are closed, it is determined that the blink is a mere blink, and it is assumed that there is no pupil movement during the period until the next both pupils appear.
If it is determined that there is one pupil, the next frame will be given a window at the predicted position, and another pupil will be searched. If there is no eye, it is determined that one eye is open, and if both eyes are open, In the next frame, a window is given to both pupils.

According to the method described above, the opening and closing of the two eyes are distinguished based on the presence or absence of each of the two pupils. If it is determined that a pupil is present, the center coordinates of each pupil are calculated by the centroid method or the like. To do.
The above processing can be easily created by software. It can also be realized as hardware.
In addition, the movement of the cursor has been described as if it were proportional to the movement of the pupil so far, but the movement of the pupil due to the movement of the head is particularly small in the vertical direction, and the deviation of the pupil position from the reference point should be integrated. There is a case where it is preferable to determine the cursor position by means of, so this embodiment is not denied.

The following can be considered as uses of this apparatus.
1. It is used by people with disabilities who cannot move from the neck down to operate ordinary computers or to control nursing equipment via the computer.
2. The driver can control the navigation system presented at the front in the car by turning the head while looking forward.
3. It is used to speed up and improve the efficiency of personal computer operations, such as moving the mouse cursor, changing the active window, and moving while performing key operations in normal personal computer operations.
4). When there is not much back and forth movement of the head, the virtual reality that needs to detect head movement using the movement of the pupil in the image to detect the approximate movement of the head left and right and up and down In particular, it is used to change the visual information displayed on the display according to the movement of the head.

System configuration diagram Display example of keyboard on monitor by software A diagram showing an arrangement example of illumination light Image example showing reflected image and difference image from pupil Schematic diagram showing the movement of the cursor corresponding to the movement and disappearance of the pupil

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Camera 2, 3 Light source 4 Control apparatus 5 Personal computer 6 Monitor 7 Target person (user)
8 Cursor 10 Camera opening (lens)
11 Light-emitting diodes 12, 13, 14, 15 Regions 20 to 23 where light-emitting diodes are arranged Pupils 24 and 25 Cursor 26 Vanished pupil

Claims (3)

A camera that captures the face of the subject;
A first light source disposed near the camera and irradiating near infrared light;
A second light source disposed nearer to the camera than the first light source and irradiating near infrared light;
The illumination of the first and second light sources is controlled, and the difference between the image obtained by the first light source and the image obtained by the second light source obtained by the camera is calculated to calculate the pupil position is directly detected, the pointing device according to the pupil position detection comprising a control device for output as a signal corresponding to the cursor movement amount on the monitor change in direct coordinates of the pupil aperture. A camera that captures the face of the subject;
A first light source disposed near the camera and irradiating near infrared light;
A second light source disposed nearer to the camera than the first light source and irradiating near infrared light;
The illumination of the first and second light sources is controlled, and the difference between the image obtained by the first light source and the image obtained by the second light source obtained by the camera is calculated to calculate the pupil The position is directly detected, and the presence or absence of the pupil is detected from the time-series change of the image obtained by the camera, and the change in the position coordinate of the pupil is directly used as a signal corresponding to the cursor movement amount on the monitor. A pointing device based on pupil position detection, comprising: a control device that outputs the presence or absence of the pupil to the outside as a cursor-related button press signal. Irradiating the face of the subject with a first light source disposed in the vicinity of the camera to obtain a first face image;
Irradiating the face of the subject with a second light source disposed at a position farther from the camera than the first light source to obtain a second face image;
Calculating the difference between the first face image and the second face image to directly detect the position of the pupil;
A pointing method by pupil position detection comprising the step of displaying the cursor on the monitor by controlling the amount of movement of the cursor on the monitor in direct correspondence to the change in the position coordinates of the pupil.

Images