JP3673834B2 - Gaze input communication method using eye movement - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To construct a visual line input communication system using eyeball motion for enabling stay-home work to be carried out by selecting an intended item on a display screen, by obtaining a face image of a patient by using a video camera to detect a visual line direction of the patient by image processing in a contactless manner, by inputting the visual line only by an eyeball function by eyeball motion and the opening-closing action of an eyelid, and by operation by a virtual board with a switching operation, by setting, as an issue, development of the visual line input communication system using eyeball motion. <P>SOLUTION: This method is so structured that a differential image is obtained, by making an examinee open and close his or her eyes, from an image formed by capturing the entire face of the examinee; then, templates of eyes and eyebrows are registered; and the visual line direction is detected by comparing data obtained in calibration with data obtained from the input image. <P>COPYRIGHT: (C)2005,JPO&amp;NCIPI

Description

  The present invention relates to a gaze input communication system using eye movement that supports communication between a severe amyotrophic side sclerosis patient (hereinafter referred to as ALS) and the like, a caregiver, a family, and the like.

  Research on estimating the direction of gaze from images captured from a video camera has a long history, and conventional research includes methods that use near-infrared rays and methods that detect the direction of gaze by projecting a fluorescent lamp in the pupil. .

  First, when an ALS patient or the like is using a communication device, the situation where the gaze position of the eyeball cannot be detected due to the movement of the face is eliminated, and the communication device is prevented from being erroneously operated. Patents that are excellent in economic efficiency are disclosed (for example, Patent Document 1). The apparatus is an apparatus that can be easily and accurately controlled without erroneous operation when using the communication device using the line of sight. The apparatus was small and economical. Specifically, the method used when reading the test subject's face as data was changed from infrared rays to a CCD camera. In addition, the amount of face image data (not only the position of the eyeball but also the position, orientation, and line-of-sight direction of the face) was increased. In addition, by dividing the monitor in advance and recognizing what part the subject was looking at, it was converted into data (dictionary) as an image, and when it was actually used, it was erroneously operated by selecting image data from the dictionary to see which key the subject was looking at To achieve miniaturization and excellent economic efficiency. The specific procedure for inputting and controlling the instruction transmission device easily and accurately by visual line is as follows. Dividing the monitor of the communication device in advance and having the subject look at each of the divided parts, each time the subject's image (face position, orientation, line-of-sight direction) is read and recognized by the CCD camera and converted into data ( Dictionary). When using the communication device, the user's image data is read by a CCD camera, and the image obtained by viewing the keys on the monitor when the subject actually operates is compared with a dictionary that has been converted into image data in advance. Similar image data is selected from the above, and the speech and operation procedure of the conversation recorded at the location corresponding to the selected image data is executed. However, the one described in Patent Document 1 has a problem that, as in the present invention, an ALS patient or the like is easy to operate with only an eyeball function and general-purpose PC software cannot be used.

  Next, a conventional technique for detecting the line-of-sight direction from an image captured from a video camera will be described. Some are determined by the image processing apparatus from the images of the operator's face captured by two small cameras mounted on the display screen. The gaze direction calculation device is determined in advance. When looking at a reference point on the display screen, several points are stored as initial values. Further, the direction of the face and the eyeball is determined from the point determined by the image processing apparatus and the point stored as the initial value, and the direction of the line of sight is calculated based on these. The calculated line-of-sight direction is given to the arithmetic processing unit. Further, the distance to the operator is determined, and the result is given to the arithmetic processing unit. The arithmetic processing unit outputs a cursor on the display screen only when the operator is closer than a predetermined distance. Moreover, the cursor is moved according to the direction of the line of sight (for example, Patent Document 2). However, the one described in Patent Document 2 has a problem that, as in the present invention, an ALS patient or the like is easy to operate with only an eyeball function and general-purpose PC software cannot be used.

  In addition, the cursor is controlled according to the user's intention by interrupting and resuming the function of gaze detection based on at least one information of the user's gesture, voice, operation, gaze, and blink. A line-of-sight detection means for detecting the line-of-sight direction, a cursor management means for managing a cursor tracking mode for determining whether or not to move the cursor to the detected line-of-sight position, and a cursor control unit for moving the cursor to the detected line-of-sight position Some are provided (for example, Patent Document 3). However, similar to Patent Document 2, the one described in Patent Document 3 has a problem that ALS patients and the like are easy to operate with only the eyeball function and general-purpose PC software cannot be used.

TNCornsweet et al. Are made by irradiating the eyes with near infrared rays.
Focusing on the Purkinje image, the first Purkinje image (reflected image on the cornea surface) and the fourth Purkinje image (reflected image on the back of the crystalline lens) offset the effects of head movement. High-precision line-of-sight detection is made possible by simply fixing the headrest (Non-Patent Document 1). However, there is a problem that an optical system for separating and detecting a signal from the fourth Purkinje image that is about 1/500 times as large as the first Purkinje image becomes complicated and large.

  Iida and Banno have proposed a method for detecting a point of interest on the display screen regardless of the movement of the user's head by using an eye camera using a corneal scleral reflection method and a three-dimensional magnetic sensor. (Non-patent document 2). However, the eye camera uses an LBM type eye camera that uses the difference in light reflectance between the cornea (black eye) and the sclera (white eye), and needs to be worn on the head. The result of the accuracy evaluation was 0.89 deg. As an average error between the detected gaze point and the index. Further, it has been confirmed that the use of the line of sight and the mouse is advantageous when the movement distance of the index is large compared to the instruction input by the mouse alone. However, problems such as experiments in a special case of displaying an index or a cursor in a single color background remain. Therefore, an instruction input experiment that is as close as possible to the case of instructing a display screen on an actual workstation is performed, and the effectiveness is confirmed by a method that suits practical use (Non-Patent Document 3).

  Paying attention to the fact that the camera reflects the light that is reflected and emitted from the pupil when near-infrared rays are applied to the eyes, Banno uses two types of illumination with different arrangement conditions. Thus, a method for extracting the pupil has been proposed (Non-Patent Document 4). The eyeball was approximated by a model in which two spheres overlap, and the lighting arrangement conditions under which the entire pupil was photographed with the same brightness and the illumination arrangement conditions under which the pupil was photographed darkly were obtained by ray tracing. Under these two arrangement conditions, an image with a bright pupil and a dark image are taken with the same camera, and the pupil is extracted by taking the difference between them.

  Banno and Kishino obtain three spatial positions of the face and pupil by stereo image measurement, measure the center of the eyeball from the position information of the feature points, and perform gaze detection (Non-patent Document 5). By wearing eyeglasses with three marks on the edge and without lenses, three feature points with little movement are created on the face. Each of the two camera systems can detect the relative three-dimensional positions of the feature points with an accuracy of about 0.1 to 0.15 mm by capturing four feature points on the entire photographing screen.

  Mukai et al. Performs gaze direction detection by extracting feature patterns using grayscale images (Non-Patent Document 6). A general fluorescent lamp is used for illumination, and a special illumination lamp such as infrared light is not used. The 100-inch display screen is divided into 3 × 3, and line-of-sight identification in 9 directions is performed. In the input image, the area above the nose of the face is used, and the position of the eyes is detected using a Sobel filter. An eye periphery image is extracted from the obtained eye position information. Since the skin color becomes noise and it is difficult to obtain the position of the black eye, the feature pattern is extracted by removing the color close to the skin color and making it darker. The line-of-sight direction is identified by matching with the standard pattern. The standard pattern is created from learning image data, and three horizontal and vertical patterns are prepared for each eye. As a result of the experiment, the correct answer rate was 71.4% in 9-direction discrimination. Considering the size of the display screen of 100 inches, it is considered impractical.

  Aoyama et al. Obtains the line-of-sight direction by adding the face direction to the rotation angle of the eyeball (Non-Patent Document 7). Psychological experiments have been conducted and it has been confirmed that gaze direction can be estimated from information on both eyes and mouth. The input image is the entire upper body on the chest, and a face image area is extracted from the input image using edges and a mosaic pattern. Template matching is used to extract both eyes and mouth. Matching is performed in advance using both eyes and mouth images cut out from the front image of the person himself, and ten candidate areas are determined for each. From the obtained candidate regions, a combination at an appropriate distance as both eyes and mouth is selected. Both ends of the eyes and face are also detected, and the direction of the face is estimated. By using the texture mapping image, the direction is corrected by comparing the projected image with the input image. As a result of the evaluation experiment, 113 out of 126 sheets and 89.7% succeeded in feature extraction. The average error in the horizontal direction was 12.9 degrees for the cylindrical model and 10.2 degrees for the planar model. In the vertical direction, this is a future issue.

  Horiba et al. Binarize an image in which the area around the eyes is binarized to extract eyebrow end points and iris centers, and estimate the line-of-sight direction based on the displacement of the relative distance between the two points (Non-patent Document 8). . By providing a reference point on the face image, the movement of the head is allowed, but the image around the eyes is enlarged, and it is sufficient that the eyes and eyebrows protrude from the camera image when the head shakes. Conceivable.

  Nishiuchi et al. Detect the face direction by extracting facial feature points without the need for markers, and detect the line of sight by adding the position of the center of the black eye (Non-Patent Document 9). From the face image converted to only white and black by binarization, the midpoint of the closest positions of both eyes and the two nostrils is extracted as a feature point of the nose. The distance between the feature points of each person's face is measured in advance, and the coordinates in the three-dimensional space are calculated from this distance to estimate the face direction. Instead of using infrared rays, a fluorescent lamp is placed below the CRT, and there is no other illumination. The center of the black eye is obtained from the reflection of the fluorescent light projected in the iris area and the end point of the black eye.

  Takegami and Goto have estimated the gaze direction based on the relative relationship between the cornea reflection image and the iris region (Non-patent Document 10). It is noted that the position of the reflected image of the light source in the cornea relatively changes in the iris region with a change in the viewing direction. In addition, by using the cornea reflection image and the edge portion of the iris region as feature points, it is possible to perform relatively high-precision measurement with a single camera without fixing the head or wearing an index or the like. Although it remains a problem whether a corneal reflection image can be stably produced with a light source, it has been confirmed by experiments that it can be detected with an accuracy (± 0.5 to 0.9 deg.) Substantially corresponding to fixation fine movement.

JP 2001-350578 A JP-A-5-298015 JP 2001-100903 A

T.N.Cornsweet and H.D.Crane, “Accurate two-dimensional eye tracker using first and fourth Purkinje images,” Journal Opt. Soc. Am., Vol. 62, No. 8, pp. 921-928, 1973. Iida, Banno, “A gaze detection device that allows head movement and its application to instruction input,” IEICE Transactions, D-II, No. 4, pp. 520-527, 1991. Banno, Tetsuya, Kishino, “Evaluation of Instruction Input Method Using Gaze and Mouse,” IEICE Transactions, D-II, No.6, pp.867-875, 1993. Banno, “Design method of pupil imaging optical system for eye-gaze detection,” IEICE Transactions, D-II, No. 6, 1991. Banno, Kishino, “Gaze point detection algorithm based on 3D position measurement of face and pupil,” IEICE Transactions, D-II, No.5, pp.861-872,1992. Mukai, Mitani, Tokawa, “Gaze direction detection by image processing,” Proceedings of the 2nd Image Sensing Symposium, pp.135-138, 1996. Aoyama, Yamamura, “Estimation of Face and Gaze Direction Using One Camera,” IEICE Technical Report, PRU.95-233, pp.131-136, 1996. HORIBA, Lee, Inoue, “Gaze detection method using image processing and its application,” 40th Annual Conference of the Institute of Systems, Control and Information Engineers, pp.187-188, 1996. Nishiuchi, Shibata, Takada, "Study on non-contact gaze detection by image processing," Transactions of the Japan Society of Mechanical Engineers (C), Vol. 64, 620, pp 121-127, 1998. Takegami, Goto, “Gaze detection method using corneal reflection image and iris contour information,” IEICE Transactions, D-I, vol.J82, pp.1295-1303, 1999.

  The present invention has an object of developing a gaze input communication system using eye movements, acquires a patient's face image using a video camera, and performs non-contact detection of the patient's gaze direction by image processing. Gaze input communication using eye movement that selects intended items on the screen and inputs only by eye function by eye movement and eyelid opening and closing operation and working from home by virtual board operation by switching operation The purpose is to build a system.

In order to achieve the above object, in the first invention, a difference between values of each pixel of an image in which the subject's eyes are open and an image in which the subject's eyes are closed is calculated from an image obtained by capturing the entire face of the subject. When a new image having a difference as a pixel value is created to obtain a difference image, and then the coordinates of the center of the eye are obtained from the difference image, the eye and eyebrow templates are registered from the coordinates , and the calibration is performed. The computer screen is divided into several regions, and the subject has registered an image according to the direction in which the subject has seen each divided region, the position of the black eye is obtained by an image processing method, and the direction-specific reference for gaze direction detection The relative distance between the black eyes and the eyebrows was calculated . In the second invention based on the first invention, the calibration and the line-of-sight direction are obtained using an image in which the periphery of the eye is captured by the zoom-in function of the camera while continuously tracking the eye position using high-speed template matching . To get .

In the third invention based on the first invention, after the calibration, a line-of-sight pointer is input on the personal computer screen in several divided areas instead of the mouse with the function of opening and closing the eyeball and eyelid, Switching operation enables virtual keyboard operation.

In the fourth invention based on the first invention, the area of the section that has been watched is enlarged by gazing at the sectioned screen display including the application to be activated for 2 seconds or more, and the subject's line of sight When the screen is directed in the same direction for 3 seconds or more, the screen scroll speed is increased . After moving the application to be activated by the high speed scroll to the vicinity of the center of the display screen, By doing so, the line-of-sight pointer was moved to be located near the center of the display screen .

  By developing a gaze input communication system using eye movement of the present invention, a face image of a patient is acquired using a video camera, and the gaze direction of the patient is detected in a non-contact manner by image processing. The communication system that can select the intended item has been constructed. Furthermore, it is possible to input by only the eyeball function, and to perform operation by a virtual board by switching using general-purpose PC software, and it is possible to work at home for an ALS patient or the like.

  Next, an embodiment of a line-of-sight input communication system using eye movement according to the present invention will be described in detail with reference to FIGS.

  1 is a hardware configuration diagram of a gaze input communication system using eye movement according to the present invention, FIG. 2 is a schematic diagram of a gaze input type communication system, and FIG. 3 is a nine-division screen as an example of a communication screen (initial screen). FIG. 4 shows an example of the communication screen when “TV” is selected in the communication screen of FIG. 3, and FIG. 4A shows an example of the communication screen. Is a diagram showing an example of a communication screen when FIG. 4B is selected (FIG. 4B), FIG. 5 is a flowchart showing a processing procedure of the system, FIG. 6 is a flowchart showing a personal identification procedure, and FIG. 7 is an input image. (A) shows the entire face, (b) is an enlarged view showing the eye peripheral area, FIG. 8 is a view showing a difference image, FIG. 9 is a view showing eye position detection, FIG. Is a diagram showing template images by size, FIG. 11 is a diagram showing template matching by image size, FIG. 12 is a diagram showing examples of registered template images, FIG. 13 is a diagram showing matching results, and FIG. FIG. 15 is a diagram showing an example of a template image according to nine directions in the correlation method (method I), FIG. 15 is a diagram showing an example of black eye tracking by high-speed template matching in the black pixel region detection method (method II), and FIG. 16 is edge feature point detection. FIG. 17 is a diagram showing an example of preprocessing in the method (method III), FIG. 17 is a diagram showing an example of edge detection by a Sobel filter in the edge feature point detection method (method III), and FIG. 18 is an edge feature point detection method (method III). ) Is a diagram showing an example of detection of four contact points in FIG. 19, FIG. 19 is a diagram showing an experiment screen in a gaze direction detection experiment for a subject, and FIG. 20 is a subject It illustrates an example of a nine directions visual axis detection result of the gaze direction detection experiments against, FIG. 21 shows an example of 12 directions visual axis detection result of the visual line detection experiments on subjects.

Furthermore, FIG. 22 is a flowchart from the acquisition of the gaze direction to the use of the virtual keyword or the application, FIG. 23 is a screen diagram in which the screen area near the gaze pointer is enlarged and displayed in the center of the screen, and FIG. Figure 25 shows the screen for moving to the vicinity of the application to be activated. Figure 25 shows the screen for confirming the position in the window at the center of the display screen and moving the line-of-sight pointer over the application. Figure 26 shows that the eyelid is intentionally closed for 3 seconds or more. FIG. 27 is a screen diagram for displaying a selection determination area for selecting an application to be activated at the center of the display screen. FIG. 28 is a screen diagram for displaying an application to be activated. Figure 29 shows a screen for scrolling the display screen so that it is located nearby. A selection decision area is always fixedly displayed in the center, and a screen diagram for moving an application to be activated into the selection decision area. FIG. 30 shows that the application in the area is activated when the bag is consciously closed for 3 seconds or more. 31 is an initial screen, FIG. 32 is a screen diagram for gazing and selecting a partition including an application, FIG. 33 is a screen diagram in which the selected partition area is enlarged, and FIG. 34 is a screen for directing the application. Screen for scrolling, FIG. 35 is a screen for displaying a pointer with a conscious blink, FIG. 36 is a screen for moving and starting the pointer on the application, and FIG. FIG. 38 is a result diagram (1-5th average measurement value), FIG. 38 is a screen scroll method measurement result diagram (1-5th average measurement value), and FIG. 39 is divided. FIG. 40 is a measurement result diagram by the area enlargement method (average measurement values for the first to fifth times), FIG. 40 is a measurement result diagram by the pointer vicinity region expansion method (average measurement values for the sixth to tenth times), and FIG. 41 is a measurement result diagram by the screen scroll method. (6th to 10th average measurement value), FIG. 42 is a measurement diagram (6th to 10th average measurement value) by the divided region enlargement method.
It is.

  First, ALS (Amyotrophic Lateral Sclerosis) is a progressive neurological disease designated as a specific disease in the country. The prevalence is about 5 out of 100,000 people, 90% of which have developed since middle age. The gender ratio or gender ratio is 1: 1.5, which is slightly higher for men. There are about 4500 patients in Japan. Since it was first defined in 1874 by Dr. Charcot in France, it is said that there is no cure or medical treatment that can keep it going. As the symptom progresses, motor nerves are affected, muscle weakness and atrophy of limb muscles, swallowing muscles, and respiratory muscles progress, and complete limb paralysis occurs in 4 to 5 years from the onset of symptoms, not only the limbs but also the body The whole muscle is paralyzed and cannot speak. Eventually, the muscles that breathe are also violated, and without a ventilator, they cannot survive. However, intelligence, sensation, and eye movement are normal, and intelligent creative activities are possible. In the United States, major league baseball player Lou Gehrig was affected, and so it is called Gaelic disease. The famous British astrophysicist Dr. Hawking has also been a patient for 30 years.

  The ALS patient 20 needs care because of paralysis of the limbs, but if it becomes impossible to speak language as the medical condition progresses, it becomes difficult to communicate with medical staff, caregivers, and families. . Therefore, there are some medical institutions that cannot communicate smoothly with the ALS patient 20 and refuse to be hospitalized due to a large amount of care. For this reason, development of the communication apparatus for improving a patient's QOL (Quality of Life) is desired. Equipment and devices that help people with disabilities to communicate intentions and information using the remaining functions to communicate more smoothly are called communication aids. The types of communication aids used by people with physical disabilities range from simple tools such as dials to high-tech devices that apply various engineering technologies.

  In the present invention, such an ALS patient aimed to develop a gaze input communication system using eye movement. A face image of the patient is acquired using a video camera, and the eye direction of the patient is detected in a non-contact manner using image processing. The purpose of the present invention is to construct a gaze input communication system that uses the detected gaze direction to identify which position in the display the patient is looking at and to select an intended item.

  FIG. 1 is a hardware configuration diagram of a line-of-sight input communication system using eye movement according to the present invention. In FIG. 1, an ALS patient (user or subject) captured by a video camera 12 attached to the top of a display screen 11 of a personal computer (hereinafter referred to as a PC, including the functions of both the display screen 11 and the arithmetic processing unit 18). ) The position of the eyes of the ALS patient 20 is determined by the arithmetic processing unit 18 from the image of the 20 face. In the calculation of the line-of-sight direction, several points when the reference points on the PC determined in advance are viewed are stored as initial values. The direction of the eyeball is determined from the eye position determined by the arithmetic processing unit 18 and the eye position stored as the initial value, and the direction of the line of sight is calculated based on these. The calculated line-of-sight direction is given to the arithmetic processing unit 18. When a positional deviation of the patient's head is detected from the image of the small video camera 12 obtained via the image capturing device 14, the arithmetic processing unit 18 corrects the deviation of the head by using the video camera 12. Are sent to the camera control device 16, and a movement corresponding to the correction values is given to the video camera 12. The arithmetic processing unit 18 moves the cursor displayed on the PC in a direction corresponding to the line-of-sight direction. Moreover, by blinking or gazing consciously, it is possible to send a command by clicking with the same function as clicking with a normal PC mouse. In addition, the region viewed by the ALS patient 20 can be estimated from the calculation result obtained by the image capturing device 14, and the color of the region can be changed and displayed (output). The indoor lighting conditions are sufficient only with the brightness of a normal fluorescent lamp, and there is no need to install infrared rays or special lighting.

FIG. 2 is an explanatory diagram of a line-of-sight input communication system. This system is a non-contact communication system mainly composed of a single PC and a video camera 12. This system is intended for ALS patients 20 and the like, and is assumed to be used on a bed. Commercially available products are used for both the PC and the video camera 12, and a relatively inexpensive system is realized. An ALS patient or the like can select an intended item by gazing at the partitioned area on the display screen 11 of the PC (see FIG. 2). The display screen 11 may be a liquid crystal display that is easy to install. Moreover, it is necessary to perform calibration at the start of use of the system. In the calibration (registration of the direction reference image) , the sections of the display screen 11 are sequentially blinked (changed to a color different from the others), and the ALS patient 20 follows it. The position of the eyeball at that time is recorded and used as a reference for determining the direction.

  FIG. 3 is a front view showing a nine-divided screen as an example of a communication screen (initial screen). The present invention is a communication system intended for particularly severe ALS patients 20. The display screen 11 is divided into nine or twelve as shown in FIG. 3, and expressions that are important for the ALS patient 20 are presented in each divided area. This screen is hereinafter referred to as a communication screen. The presented intention items are determined with reference to the results of a questionnaire conducted on the family of ALS patients 20 and hospitals. The ALS patient 20 selects and selects the intended item with his / her eyes. When the gaze for 2 seconds or more is performed, the PC recognizes that the gaze is being performed, and selects the item. The selected item is read aloud by voice registered in advance. In 9 divisions or 12 divisions, it is not possible to present all the expressions required for the patient, but more expressions can be presented by preparing spare items below the items. “Yes” and “No”, which are considered to be frequently used, are displayed on the initial screen.

  FIG. 4 shows an example of the communication screen when “TV” is selected in the communication screen of FIG. 3, and FIG. 4A shows a case where “change channel” is selected on the screen of FIG. 4A. For example, if the item “TV” is selected on the initial screen, the next screen shows items related to TV as shown in FIG. 4A. Is displayed. Note that the two items “return” and “go to menu” are always displayed on the screen other than the initial screen. When “change channel” is selected here, FIG. 4B is displayed. Normally, the TV channel is often used up to 12 channels, so when “Next” is selected, 7 to 12 are displayed. When the patient selects a desired number (5 in this case), the voice is read out as “Please change to channel 5.”

FIG. 5 is a flowchart showing the processing procedure of the system. First, the ALS patient 20 (or subject) opens and closes the eyes from the image that captures the entire face, and the position of the eyes is detected (100) from the difference image. Next, when a plurality of ALS patients 20 share one system, personal authentication (102) using template matching (image correlation ) is performed, and necessary settings can be selected for each individual. After detecting the position of the eyes, the camera zooms in (104) function to acquire an image that captures a large area around the eyes. Next, eye and eyebrow templates are registered (106), and calibration (108) is performed. At this time, an image is registered for each direction, and the position of the black eye is obtained by the proposed image processing method, and a reference value (relative distance between the black eye and the eyebrows) for each direction for detecting the gaze direction is obtained. Keep it. After calibration, line-of-sight detection (110) is performed on the input image by the proposed method. The line-of-sight direction is detected by comparing the data (black eye position etc.) obtained during calibration with the data obtained from the input image. As a result, the ALS patient 20 can make a selection by staring at an intended item in the screen display 11. The region viewed by the ALS patient 20 can be confirmed by changing the color and outputting. When selecting an intended item, it is possible to watch the area of the item for 2 seconds or longer. If a spare item exists under the selected item (112), the screen is switched and the spare item is displayed. For the finally selected item, the content of the item is read out by voice (114).

  Using the image processing method described above, personal authentication (102) is performed. The purpose of performing personal authentication (102) is that there are individual differences in line-of-sight detection, so that each individual's data is registered, personal authentication is performed for each ALS patient 20, and the subject 20's data is instantly extracted. is there. In the field of security, personal authentication (102) based on facial images will play an important role in the future.

  FIG. 6 shows a procedure for personal authentication. Subjects shall be able to open and close their eyes. First, the eyes are opened and closed, and eye position detection (120) is performed based on the difference image. Here, matching (122) is performed using a template image registered in advance, and personal identification (124) is performed. A password is determined for each individual, and the password (126) is input by opening and closing the eyes. The personal identification number referred to here is the order of opening and closing the eyes. For example, the eyes are opened and closed alternately like “right, left, left, right”. If the personal identification number is correctly input, the personal recognition is completed (128).

  When face image recognition is performed using template matching, if the region included in the image includes hair or the like, it is likely to change with time, and face image recognition becomes difficult. Therefore, the regions included in the template image are four patterns of “eyes + brows”, “eyes + brows + nose”, “eyes + brows + nose + mouth”, “eyes + brows + nose + mouth + cheek contour”. (FIG. 10) was prepared and matched. In template matching including “eye + brow” and “eye + brow + nose region”, it was confirmed that the correlation value for the person was a high value exceeding 0.99 (FIG. 11). For template matching including “eye + eyebrow + nose + mouth” and “eye + eyebrow + nose + mouth + cheek contour”, the correlation value is the highest and useful, but the remaining 2 The correlation value was lower than the one. Therefore, template matching including “eyes + eyebrows” or “eyes + eyebrows + nose area” is considered optimal.

Using a template (standard image) including “eye + eyebrow + nose area”, 10 template images (80 × 60) per person are prepared for 10 subjects 20, and the template image and each individual As a result of matching with each input image and identifying the individual from the correlation value, it was found that the individual can be identified. It was possible to authenticate individuals with a high probability by template matching and input of a personal identification number by opening and closing eyes.

  In the present invention, the line-of-sight direction of the ALS patient 20 is detected by processing the image captured from the video camera 12 in the PC and used for the operation of the communication system. In particular, the goal is to establish a communication system for severe ALS patients 20, and it is appropriate to detect the direction of gaze assuming that the head of the ALS patient 20 does not move greatly due to symptoms of ALS. . It is also intended for use in hospitals and homes, and it is important to operate properly under normal lighting conditions such as fluorescent lamps. In the present invention, it is desirable that the sight line detection can be performed only with the light from a normal indoor fluorescent lamp, and the use of infrared rays or a special increase in the light source such as a fluorescent lamp is not performed. Although there is a report (Non-Patent Document 6) that does not set the illumination condition, there remains a problem in the low detection accuracy.

  The visual line direction detection accuracy of the present invention is intended to divide the display screen 11 of the PC into several parts and to select each region correctly. We examined three methods: eye-direction image correlation method using template matching, black pixel region detection method, and edge feature point detection method focusing on the edges of the iris region, as the gaze direction detection methods for realizing these conditions. . It is known that the eyeball has a fine movement called fixation fixation micromotion, so that even when one point is being watched, the line of sight is shifted by about 0.3 degrees from that direction (Yamada, Fukuda, “Notes in the image”). Definition of viewpoint and application to image analysis, “The Transactions of the IEICE, D-II, No. 9, pp. 1335-1342, 1986.” However, in this system, it is unlikely that a shift of 0.3 degrees will cause an error in the detection of the gaze direction when gazing in the designated area, so it is not considered. As the gaze direction detection method, three methods were examined: a direction-specific image correlation method (method I), a black pixel region detection method (method II), and an edge feature point detection method (method III).

  In the case of the image correlation method for each direction (method I), the eye peripheral image is registered as a template for each direction at the time of calibration (FIG. 14). The eye position was detected by high-speed template matching using a pyramid structure. Matching was performed on the detected eye position using the registered template for each direction, and the line-of-sight direction was determined from the direction image giving the highest correlation.

  In the black pixel region detection method (method II), an iris region including a pupil (hereinafter abbreviated as “iris region”) is registered as a template image (60 × 60 pixels), and an image with an enlarged eye position (input image) is registered. Template matching was performed to determine the position of the iris region (FIG. 15).

  In the edge feature point detection method (method III), attention is paid to changes in luminance between the iris region (black eye), the white eye region, and the eyelid in the eyeball, and the line-of-sight direction is detected using edge detection. In order to facilitate edge detection, image enhancement and smoothing with a median filter were performed, and edges were detected with a Sobel filter.

  By correcting the position of the head, the detection accuracy in method II and method III was greatly improved (FIGS. 20 and 21). In Method I, since the gaze direction is detected based on the correlation value of matching, it is difficult to correct the head position, but the average correct answer rate exceeds 80% in both the 9th and 12th directions.

  Severe ALS patient 20 cannot move hands, mouths, etc. freely and falls into communication with third parties. An object of the present invention is to construct a communication support system using eye movement, which is one of the remaining functions of the ALS patient 20. The present invention also aims to develop a non-contact communication system that has the least burden on the ALS patient 20. It is also an object to construct an inexpensive system by using only a commercially available video camera 12 and a PC.

  In Example 1, a face image was taken with a video camera 12 as a subject 20 instead of an ALS patient 20 and taken into a PC for various image processing. The size of the input image is 320 × 240 pixels, which is an RGB image with 256 gradations (FIG. 7). The video camera 12 has a zoom function, and the enlargement ratio of the captured image can be freely set.

  The method of eye position detection is shown below. The position of the eyes in the input image was determined by focusing on the opening and closing of the eyelids and by finding the place where the luminance change was significant between successive images. Next, the recorded image and the image recorded in the past were compared. That is, a difference in values at each pixel is calculated, and a new image having the difference value as a pixel value is generated. Hereinafter, this is referred to as a difference image.

An eye position detection procedure using this difference image will be described.
(1) The image is Y-divided in the vertical direction. The m-th (1 ≦ m ≦ Y) divided region from the top is expressed as a horizontal divided region Hm (FIG. 8).
(2) The pixel values on the horizontal division area Hm are summed and set as Hm # SUM.
(3) The mth value of the horizontal division region that gives the largest Hm # SUM is adopted as the vertical position of the eye.
(4) In the adopted horizontal division area Hm, two consecutive areas having large values are detected. At this time, it is conceivable to use a continuous area whose value is not 0, but in order to avoid the influence of noise of the video camera 12 or the like at the time of detection, a continuous area having a certain value or more was adopted.
(5) The center or center of each area is defined as n1, n2, and this is adopted as the horizontal position of the eye. Through the above steps, the eyes of the subject 20 are present at the obtained positions (n1, m) and (n2, m) (FIG. 9).

  After the eye position was found by the difference image, the eye position was continuously tracked using high-speed template matching. This operation was performed using high-speed template matching using a pyramid structure, which is image processing necessary for eye position tracking. The binarization method was used for the determination of eye opening / closing.

  Personal authentication was performed using the image processing method described above. The purpose of performing personal authentication is to register each individual's data because there is an individual difference in line-of-sight detection, to perform individual authentication for each subject 20, and to instantly retrieve the data of the subject 20. In the field of security, personal authentication using facial images will play an important role in the future.

  As described above, the personal authentication procedure according to FIG. 6 is shown. First, the eyes were opened and closed, and eye position detection (120) was performed based on the difference image. Here, matching (122) was performed using a template image registered in advance, and personal identification was performed. A password is determined for each individual, and the password (126) is input by opening and closing the eyes. The security code here refers to the order of opening and closing the left and right eyes, and is input by alternately opening and closing the eyes such as “right, left, left, right”. If the personal identification number is correctly input, personal recognition is completed.

  When performing face image recognition using template matching, if the region included in the image includes hair or the like, it is likely that the region is likely to change over time and recognition becomes difficult. Therefore, the regions included in the template image are four patterns of “eyes + brows”, “eyes + brows + nose”, “eyes + brows + nose + mouth”, “eyes + brows + nose + mouth + cheek contour”. Prepared and matched. An example of a template image by size is shown in FIG. There were six test subjects 20, and matching was performed on images acquired in advance using four patterns of size-specific template images. FIG. 11 shows the matching result.

  In the template matching including the “eye + brow” and “eye + brow + nose” regions, it was confirmed that the correlation value for the person exceeded 0.99. For template matching including “eye + eyebrow + nose + mouth” and “eye + eyebrow + nose + mouth + cheek contour”, the correlation value is the highest and useful for the person himself. It was found that the correlation value was low compared to the template matching including the “+ brow” and “eye + brow + nose” regions. Therefore, template matching including the “eye + brow” or “eye + brow + nose” region is considered optimal.

  Experiments were conducted to determine whether an individual can be identified using a template including the “eye + brow + nose” region, which was optimized in the template matching recognition experiment for each image size. Ten template images (80 × 60) were prepared for 10 subjects 20 per person, matching was performed on input images, and individual identification was performed based on correlation values. FIG. 12 shows an example of a registered template image.

  FIG. 13 shows an example of an individual difference comparison result obtained by matching the template image (80 × 60) and the input image (320 × 240) as described above. The vertical axis indicates the correlation value between the input image and the template image. The closer the value is to 1, the more similar the two images are. The horizontal axis represents 100 prepared template images (10 sheets / person, 10 persons). For each individual registered in advance, 10 template images were matched with the input image for each individual. As can be seen from the result of FIG. 13, it was possible to recognize the person himself in the template matching this time. The result is that it is possible to authenticate individuals with a higher probability by template matching and the input of a personal identification number by opening and closing one eye.

  In the second embodiment, the image captured from the video camera 12 is processed in the PC, and the gaze direction is detected as the subject 20 by the student instead of the ALS patient 20 as in the first embodiment, and used for the operation of the communication system. did. In particular, the goal is to establish a communication system for severe ALS patients 20, and it is appropriate to detect the direction of gaze on the assumption that the patient's 20 head does not move due to symptoms of ALS. It is also intended for use in hospitals and homes, and it is important to operate properly under normal lighting conditions such as fluorescent lamps.

  The visual line direction detection accuracy of the present invention is intended to divide the display screen 11 of the PC into several parts and to select each region correctly. We examined three methods: eye-direction image correlation method using template matching, black pixel region detection method, and edge feature point detection method focusing on the edges of the iris region, as the gaze direction detection methods for realizing these conditions. . It is known that the eyeball has a fine movement called fixational tremor, so even when one point is being watched, the line of sight is shifted by about 0.3 degrees from that direction (Yamada, Fukuda, “Notes in the image”). Definition of viewpoint and application to image analysis, “The Transactions of the IEICE, D-II, no.9, pp.1335-1342, 1986.” However, in this system, it is unlikely that a shift of 0.3 degrees will cause an error in the detection of the gaze direction when gazing in the designated area, so it is not considered.

  In the case of the image correlation method for each direction (method I), the eye peripheral image is registered as a template for each direction at the time of calibration (FIG. 14). The eye position was detected by high-speed template matching using a pyramid structure. Matching was performed on the detected eye position using the registered template for each direction, and the line-of-sight direction was determined from the direction image giving the highest correlation.

  In the black pixel region detection method (method II), an iris region including a pupil (hereinafter abbreviated as “iris region”) is registered as a template image (60 × 60 pixels), and an image with an enlarged eye position (input image) is registered. Template matching was performed to determine the position of the iris region (FIG. 15). In order to shorten the calculation time, template matching using a pyramid structure (Takagi, Shimoda, “Image Analysis Handbook,” The University of Tokyo Press, 1978.) was performed. At this time, the position of the iris region changes due to a slight movement of the head, and an error occurs in the detection of the gaze direction. Therefore, the reference point was set at the upper left of the eyebrows with little change with respect to the eye movement. Conventionally, since the position of the reference point is for the whole including the eyes and eyebrows, the problem remains that the reference point moves in accordance with the movement of the eyeball. With this improvement, the line-of-sight direction could be detected by the change in the relative distance between the reference point and the iris.

In FIG. 15 (b), the upper left corner coordinates of the template including the black eye are defined as (X _e , Y _e ). Let the upper left corner coordinates of the template taken near the eyebrows be (X _b , Y _b ). When the distance between the two points is (L _x , L _y ), the following equation is obtained.

Compared to L _x ⁽ⁿ⁾ , L _y ⁽ⁿ⁾ (0 <n ≦ N: N is the number of divisions) registered for each direction and registered at the time of calibration, the n-th that gives the smallest weighted Euclidean distance The item was adopted as the gaze direction. However, when we follow something with eyes, it seems that we often follow the object by moving the head, not the eye movement. Therefore, when following a partitioned area on the display screen 11, the head tends to move naturally. In the present invention, it is intended for severely disabled people, and it is not necessary to take into account that the eyes and eyebrows protrude from the input image due to a large movement of the head, but it is possible to detect the line of sight by subtle movement of the head. It is quite possible that an error will occur. Therefore, the movement amount of the head is offset by adding the following correction terms to L _x and L _y . The correction term was obtained as follows.

Here, Xc and Yc are constants and are obtained experimentally. Note (Xbo, YBO) is the initial coordinates of the eyebrows upper left reference point in FIG. 15 (a), (Xb, Yb) is the eyebrow upper left coordinates of the input image at that time.

  In the edge feature point detection method (method III), attention is paid to changes in luminance between the iris region (black eye), the white eye region, and the eyelid in the eyeball, and the line-of-sight direction is detected using edge detection. In order to facilitate edge detection, image enhancement and smoothing with a median filter were performed, and edges were detected with a Sobel filter. As a technique for removing or reducing noise superimposed in the process of imaging, there is smoothing. Noise is an undesirably steep change in density value in a portion that naturally changes gradually, and smoothing that smoothly converts the sudden change has an effect of reducing noise. Several methods have been proposed for smoothing, but this time a median filter was used. Median filter is more effective than local averaging (Hasegawa, “Basic Techniques of Image Processing <Technical Introduction>,” Technical Review, 1986.), (2) Smoothing small fluctuations. There are advantages such as (3) the degree of edge blurring is small.

An image processing method for edge detection on images, such as Roberts and Prewitt (Tsuchiya, Fukada, “Image Processing,” Corona,
1990.) can also be used, but this time we used the Sobel filter to detect the tangent between the iris area and the white and eyelids. Since the minimum interval is 1 in a digital image, a difference is used instead of differentiation, and the absolute value of the difference is a numerical value of edge strength, in other words, edge likeness.

  From the input image (320 × 240 pixels) including the eyes and eyebrows, an image of the peripheral area of the eyes was extracted by template matching. The image size at this time is 160 × 80 pixels (FIG. 16A). First, in order to facilitate edge detection, the image is emphasized as a pre-process, and then the edge information is smoothed using a 3 × 3 pixel median filter, and the brightness of the iris region, the white-eye region, and the eyelids is increased. The difference was clarified (FIG. 16 (b)). If edge detection is performed at this stage, a white boundary is created by reflection of light between the black eye and the lower part of the eyelid, making detection difficult. Therefore, the skin color and the white-eye area were all converted to white by converting the contrast of each pixel to 3 times (FIG. 16 (c)). In addition, by converting the skin color region to white, the boundary between the iris and the eyelid is more emphasized, and the edge detection becomes easy.

Next, the Sobel filter was used separately in the horizontal direction and the vertical direction to detect the left and right ends of the iris and the tangent side of the iris and eyelid. At this time, when the output value of the Sobel filter is positive (white to black), it is output in blue, and when the output value is negative (white to black), it is output in green (FIG. 17). As a result, the left and right sides of the iris and the top and bottom of the tangent side of the iris and the eyelid are distinguished. From FIG. 17 (a), the coordinates giving the highest value of the total of about 10 pixels adjacent in the oblique direction are adopted as the left and right ends X _R and X _L of the iris. Then with respect to FIG. 17 (b), focusing on the previously obtained X _R, the midpoint of X _L, on the vertical direction of the middle point, the maximum value sum of 10pixel about pixels neighboring in the horizontal direction The points to give are adopted as the contacts Y _U and Y _D with the heel (FIG. 18). Here, as with the black pixel region detection method, high-speed template matching is performed with the eyebrows as a reference, and distances L _xR , L _xL , L _YT , and L _YB with the obtained X _b and Y _b are obtained, and the line-of-sight direction Was detected.

  A gaze direction detection experiment was performed on four subjects 20. The breakdown of the four is three men (one wearing glasses) and one woman. The distance between the eyes of the subject 20 and the display screen 11 was 75 cm, and the distance from the video camera 12 was 85 cm. The indoor lighting conditions were the brightness of the fluorescent lamps installed on the normal ceiling, and there was no significant change during the experiment. Is the display screen 11 divided into 9 and 12 divisions (FIG. 19), and based on the movement of the eyeball when looking at each area, can the gaze direction be detected correctly with the three proposed methods? Was confirmed. First, the center of the display screen 11 was watched and a reference image was recorded. Next, as calibration, the sections in the display screen 11 were sequentially blinked (changed to a different color), followed by eyes, and an image serving as a reference for determining the direction was recorded. Thereafter, the task of blinking the compartments randomly about 50 times and repeating the eye tracking was repeated, and the face image, eye coordinates, and line-of-sight direction at that time were recorded. For the image obtained in the experiment, we tried to detect the gaze direction using the three methods proposed earlier. In order to ensure reproducibility with respect to the three proposed methods, the same image was used.

  It can be seen from FIGS. 20 and 21 that the detection accuracy is greatly improved in method II and method III by correcting the position of the head. In Method I, since the gaze direction is detected based on the correlation value of matching, it is difficult to correct the head position, but the average correct answer rate exceeds 80% in both the 9th and 12th directions. Looking at test subject T, the average accuracy rate exceeded 90% in all three methods without adding correction terms in both the 9 and 12 directions. Moreover, since it was confirmed that the line-of-sight detection can be performed correctly by correcting the position of the head, it may be possible to expand the scope of the system in the future. Next, we considered each proposed method. As shown in FIGS. 20 (a) and 21 (a), the direction-specific image correlation method (method I) was found to have slightly better gaze detection accuracy in 12 directions than in 9 directions. Looking at misrecognition, in most cases, one of the left, right, upper and lower is recognized incorrectly. The gaze direction is detected with an accuracy of 85% or more in both the 9 and 12 directions without correcting the position of the head. Compared with Method II and III, since the head position correction is difficult, the correct answer rate after the head position correction is not so good, but the correct answer rate is 85% or higher without correction. It can be seen that the black pixel area detection method (method II) is improved by correcting the head position in both the 9 and 12 directions to a correct answer rate of 90% or more. Most of the misrecognitions in this method are in the vertical direction. Compared to the horizontal direction, the vertical direction has more regions where the eyeball is hidden behind the eyelids, and it is considered difficult to follow the black pixel region (black eyes) by template matching. The edge feature point detection method (method III) is almost the same as the black pixel region detection method.

  Next, based on Example 3, it demonstrates with reference to FIGS. 5 to 18 have been described above. In the third embodiment, the system processing procedure from the eye position detection based on the difference image to the determination of the line-of-sight direction is briefly described again, and then the line-of-sight is described. After the direction is determined, an eyeball function (only eye movement and eyelid opening / closing operation) is input instead of the mouse with the line-of-sight pointer on the display screen 11, and a switching operation enables a so-called virtual keyboard operation. Therefore, the present invention that enables telecommuting using the eyeball function of a severely handicapped person such as an ALS patient will be described.

  In particular, since the ALS patient 20 has an obstacle in the extremities, the ALS patient 20 does not have means for using an input device for operating the PC. Since it is difficult to speak, PC operation by voice input is also impossible. Therefore, in the present invention, the open / close movement of the eyeball and eyelid, which is the remaining function of the ALS patient 20, is applied to the input of the PC operation, thereby realizing a free PC operation environment using general-purpose PC software. It became possible to work at home for people with severe physical disabilities such as In addition, although the ALS patient 20 was described here, it is not necessary to limit to the ALS patient 20, and the same correspondence is also possible for a severely physically handicapped person similar to the ALS patient 20.

  As described above, the present invention focuses on working from home only by using the eyeball function of the ALS patient 20. In the present invention, an input signal generally sent to a PC using a mouse is generated and transmitted by opening and closing movements of the eyeball and eyelid. As a result, an eye function (only eye movement and eyelid opening / closing operation) is input to the line-of-sight pointer on the display screen 11 instead of the mouse, and a so-called virtual keyboard can be operated by a switching operation. That is, if the severe ALS patient 20 can only move and blink, only the periphery of the eyeball changes in the images (input images) sequentially captured by the video camera 12. Therefore, the line-of-sight direction is determined from the relative distance between the reference coordinates appropriately set in the input image and the coordinates that change due to eye movement, and the density error between the input image and the direction-specific registered image. The processing procedure and the detailed description thereof are the same as those described in detail with reference to FIGS. 5 to 21, but the outline will be briefly described.

  First, only the eye region is extracted from the face image of the severe ALS patient 20 including the background and enlarged. Next, an area of 120 × 80 pixels including black eyes is registered as a template. After template registration, the eye position can be automatically tracked. A severe ALS patient 20 is allowed to open and close the eyelid for 5 seconds, and a threshold value for opening / closing discrimination is registered. After opening / closing threshold registration, it is possible to determine the open / closed state of the bag. Subsequently, after calibration, the input image is subjected to image processing, the result obtained after analysis is compared with the data registered at the time of calibration, and the line-of-sight direction is detected. The eyelid direction was input using the eyelid opening / closing operation. If the eyelid is closed for more than a certain time, it is determined that the eye is consciously blinked, and the line-of-sight direction immediately before closing the eyelid is adopted.

  Subsequent operations will be described with reference to FIG. FIG. 22 is a flowchart from the acquisition of the line-of-sight direction to the use of a virtual keyboard or the use of an application (hereinafter, the application refers to various software installed in the PC).

  As shown in FIG. 22, after the line-of-sight direction determination (200) is performed, the line-of-sight pointer is moved (202) to determine whether or not the target has been reached (204). Then, the line-of-sight pointer is moved again (202). If it is determined that the target has been reached, a conscious blink of eyes (206) is performed, and the state is transmitted as a command (208). The transmitted command can be used for either the use of the virtual keyboard (210) or the use of the application (212), or a combination of both.

  An example of using a virtual keyboard is briefly described. First, the line-of-sight pointer is moved to the target character area. Then, a conscious blink is performed to input the target character. The target character is then input to the target input location via the virtual keyboard.

  Next, an application usage example will be briefly described. As an example, a procedure for starting up Microsoft Internet Explorer (hereinafter referred to as IE) on a desktop is shown. First, the line-of-sight pointer is moved over the IE icon. Then, a conscious blink is performed to activate the IE. Then, the IE icon is activated by receiving the activation command via the line-of-sight pointer.

  First, a method for using an application on the display screen 11 using a line-of-sight pointer will be described. The methods proposed in the present invention are three methods: (1) pointer vicinity region enlargement method, (2) screen scroll method, and (3) divided region enlargement method. These three methods are mainly intended to overcome the disadvantage that the operability in the vicinity of the application to be activated and the time required for selection are inferior to those of the mouse pointer.

First, the pointer vicinity area expansion method will be described. In the case of the pointer vicinity area enlargement method, an area in the vicinity of the pointer is displayed in an area ahead of the line of sight where the pointer does not move (screen central area) (FIG. 23). The pointer can be operated (FIGS. 24 and 25) while confirming the position of the application to be executed. Thereby, the fine position adjustment similar to the operation of operating the pointer with a normal mouse can be realized. Furthermore, the position of the line-of-sight pointer and the application became clear visually by enlarging the pointer vicinity area and displaying it in the center of the screen. Then, after the line-of-sight pointer reaches the application to be activated, the application is activated by performing a conscious blink for 3 seconds or more (FIG. 26).
The movement time of the line-of-sight pointer increases in proportion to the movement distance, and the difference becomes more noticeable compared to mouse operation. Therefore, when the ALS patient 20 turns the line of sight in the same direction for 3 seconds or more, the movement speed of the line-of-sight pointer is increased. As a result, the time required to move to a far away position can be shortened.

Next, in the screen scroll method, to the center of the screen where the line-of-sight pointer is not moved, the application to be started is moved and selected instead of the line-of-sight pointer, as opposed to the cursor area expansion method. An area (selection determination area) for selecting an application to be activated is provided in the center of the display screen 11 (FIG. 27), and the application is moved within the area (FIG. 28). Further, the display screen 11 is a scroll type and the entire screen is always displayed on the monitor (FIG. 29), and the usable screen area is wider than that of the cursor area enlargement method. Then, after the application to be activated reaches the selection area, the application is activated by performing a conscious blink for 3 seconds or more (FIG. 30).
Furthermore, in this method, since the application to be started is moved to the center of the display screen 11, the moving time does not matter so much. However, considering the current situation that applications are usually placed on the far side of the screen, the screen scrolling speed is high when the user's line of sight is directed in the same direction for 3 seconds or more, as with the pointer vicinity area expansion method. Adopted a method to convert.

  Finally, in the divided region enlargement method, the initial screen is displayed in nine divisions in advance (FIG. 31), and the section including the application that the ALS patient 20 wants to activate is watched and selected (FIG. 32). In order to solve the movement time problem of the line-of-sight pointer, only the application vicinity area to be activated is extracted and enlarged and displayed on the entire monitor (FIG. 33), and the application to be activated by high-speed scrolling of the display screen 11 is displayed near the center of the display screen 11 (Fig. 34). Regarding the pointer operation in the vicinity area of the application to be activated, the operation is performed by displaying the pointer on the display screen 11 in which the application is located substantially in the center of the display screen 11 and the vicinity area is enlarged nine times. Movement operation is possible (FIGS. 35 and 36).

The processing procedure of the divided area expansion method is shown below.
(1) An initial screen divided into nine sections is displayed.
(2) Watch the section that contains the application you want to launch. If the gaze time is 2 seconds or more, the area in the section where the gaze is focused is enlarged.
(3) When paying attention to the application to be started, the display screen 11 scrolls so that the application is positioned near the center of the monitor.
(4) After the application to be activated has moved to the vicinity of the center of the monitor, when a conscious blink is performed, a line-of-sight pointer is displayed at the center of the display screen 11.
(5) Move the line-of-sight pointer to the application that you want to activate and consciously close the eyelid for 3 seconds or more and less than 5 seconds to select and activate the target application. However, when returning to the display screen before enlargement, close it intentionally for 5 seconds or more.

  In order to verify the practicality of the line-of-sight pointer, the following experiment was performed on seven healthy persons. Each subject 20 activates the IE on the screen by a line-of-sight pointer to which each method is applied. Each test subject 20 was performed 10 times by each method, and the success or failure of IE activation and the measurement of elapsed time until IE selection were investigated (FIGS. 37, 38, and 39). When we usually use PCs, applications that are frequently started often have their shortcuts created and displayed on the left side of the screen. Therefore, if only the application activation work is considered, the movement range of the line-of-sight pointer can be limited to only the left side of the screen. Therefore, the initial display position of the line-of-sight pointer was set near the center of the display screen 11 and the IE display position was set near the upper left corner of the display screen 11 for investigation. The purpose of this survey is to determine whether or not the proposed method has improved the loss of eye pointer movement time and the low operability in the vicinity of the application to be activated, which are problems with the eye pointer.

  Table 1 shows success or failure of IE activation by the line-of-sight pointer. From Table 1, it was confirmed that IE can be activated with a line-of-sight pointer using the three methods proposed in the present invention. Considering that many of the test subjects 20 are using the gaze pointer for the first time, it can be said that the proposed gaze pointer is highly practical. The operability that was the subject of the line-of-sight pointer was high in accuracy and intuitive and easy to understand because the subject 20 moved in the intended direction without causing malfunction. In the operation in the vicinity of the IE, the line-of-sight pointer is smoothly moved as the number of uses increases. This means that the average visual pointer movement time (FIGS. 41, 42, and 43) for the 6th to 10th experiments is based on the average visual pointer movement time (FIGS. 38, 39, and 40) for the 1st to 5th experiments. It can also be seen from the fact that it is about 5 seconds shorter.

  Regarding the movement time of the line-of-sight pointer, all subjects have an average of about 10 seconds for all subjects, and it cannot be denied that it is slower than normal mouse operation, but it can be said that there is no problem in selecting an application. Further, if the amount of movement of the line-of-sight pointer in one line-of-sight detection is increased, the movement speed is inevitably increased. In this survey, because there were many subjects 20 who were using the gaze pointer for the first time, the movement amount was reduced and the movement speed was suppressed. However, the movement speed was increased according to the use of the gaze pointer, and the optimum speed for each subject 20 was set. By setting, more comfortable operation can be realized. In that case, it is necessary to set the movement amount in consideration of the operability in the vicinity of the application to be selected.

It is a hardware block diagram of the gaze input communication system using the eye movement which concerns on this invention. It is a schematic diagram of a gaze input type communication system. It is a figure which shows 9 division | segmentation screen as an example of a communication screen (initial screen). An example of the communication screen when “TV” is selected in the communication screen of FIG. 3 is shown in FIG. 4A, and further when “Change channel” is selected on the screen of FIG. 4A. It is the figure which showed an example (FIG.4 (b)). It is a flowchart which shows the process sequence of a system. It is a flowchart which shows a personal identification procedure. (A) in the input image shows the entire face, and (b) is an enlarged view showing the eye peripheral region. It is a figure which shows a difference image. It is a figure which shows the position detection of an eye. It is a figure which shows the template image classified by size. It is a figure which shows the template matching according to image size. It is a figure which shows the example of the registered template image. It is a figure which shows a matching result. It is a figure which shows an example of the template image classified by 9 directions in the image correlation method classified by direction (method I). It is a figure which shows an example of the black eye tracking by the high-speed template matching in a black pixel area | region detection method (method II). It is a figure which shows an example of the pre-processing in an edge feature point detection method (method III). It is a figure which shows an example of the edge detection by a Sobel filter in the edge feature point detection method (method III). It is a figure which shows an example of the detection of four contact points in the edge feature point detection method (method III). It is a figure which shows the experiment screen in the gaze direction detection experiment with respect to a test subject. It is a figure which shows an example of the 9 direction gaze detection result in the gaze direction detection experiment with respect to a test subject. It is a figure which shows an example of the 12 direction gaze detection result in the gaze direction detection experiment with respect to a test subject. It is a flowchart from acquisition of a gaze direction to use of a virtual keyword or use of an application. FIG. 10 is a screen diagram in which a display screen area near the line-of-sight pointer is enlarged and displayed at the center of the screen. It is a screen figure which moves a gaze pointer to the application vicinity which wants to start. It is the screen figure which confirmed the position in the window of the center of a display screen, and moved the gaze pointer on the application. It is a screen figure in which the selected application moves when the bag is consciously closed for 3 seconds or more. It is a screen figure which displays the selection determination area | region for selecting the application which wants to be started at the center of a display screen. FIG. 6 is a screen diagram that scrolls a display screen so that when an eye is directed to an application to be activated, the application is positioned near the center of the monitor. FIG. 5 is a screen diagram for constantly moving a selection determination area in the center of the monitor and moving an application to be activated into the selection determination area. It is a screen figure for starting the application in an area | region when consciously closing a bag for 3 seconds or more. This is the initial screen. It is a display screen figure for paying attention and selecting a section containing an application. It is the screen figure to which the selected division area was expanded. It is a screen figure for turning an application and scrolling a screen. It is a screen figure which performs a conscious blink and displays a pointer. It is a screen figure for moving and moving a pointer on an application. It is a measurement result figure (1st-5th average measurement value) by the pointer neighborhood area expansion method. It is a measurement result figure (1st-5th average measurement value) by a screen scroll method. It is a measurement figure (average measurement value of the 1st-5th time) by a division field expansion method. It is a measurement result figure (6th-10th average measured value) by a gaze pointer neighborhood field expansion method. It is a measurement result figure (6th-10th average measurement value) by the screen scroll method. It is a measurement figure (6th-10th average measured value) by a division field expansion method.

Explanation of symbols

11 Display Screen 12 Video Camera 14 Image Capture Device 16 Camera Control Device 18 Arithmetic Processing Device 20 ALS Patient (Subject)

Claims (4)

The difference image is calculated by calculating the difference in the value of each pixel between the image in which the subject's eyes are open and the image in the closed state from the image that captures the entire face of the subject, and creating a new image with this difference as the pixel value. Then , the coordinates of the center of the eye are obtained from the difference image, the eye and eyebrow template is registered from the coordinates , and when the calibration is performed, the subject divides the area obtained by dividing the personal computer screen into several parts. The image for each direction in which the area was viewed was registered, the position of the black eye was obtained by the image processing method, and the relative distance between the black eye and the eyebrows that became the reference for each direction for detecting the direction of the eye was obtained. Eye-gaze input communication method using eye movement characterized by this. 2. The calibration and the line-of-sight direction are acquired with an image obtained by capturing a large area around the eyes by a zoom-in function of the camera while continuously tracking the eye position using high-speed template matching . A gaze input communication method using the described eye movement. After the calibration, on the personal computer screen in several divided areas, a gaze pointer is input instead of a mouse with the function of opening and closing the eyeballs and eyelids, and a virtual keyboard can be operated by switching operation. The eye-gaze input communication method using eye movement according to claim 1. By gazing at the partitioned screen display including the application to be activated for 2 seconds or more, the area of the section that was being watched is enlarged, and the screen of the subject is directed in the same direction for 3 seconds or more. By adopting a method of increasing the scrolling speed, moving the application to be activated by the high-speed scrolling to near the center of the display screen, and then performing a conscious blink, the gaze pointer is moved to the vicinity of the center of the display screen. eye control communication method using the eye movement as claimed in claim 1, wherein it has moved to be positioned.

Images