JP7842405B2 - Eye-tracking device and program for eye-tracking device - Google Patents

Description

Applicable to Article 30, Paragraph 2 of the Patent Act. Disclosed on August 6, 2021, by Double Giken Co., Ltd. (920-7 Kurihara, Zama City, Kanagawa Prefecture) and Midori no Makiba Kikaku Co., Ltd. (4-38-28-613 Honcho, Funabashi City, Chiba Prefecture).

Applicable under Article 30, Paragraph 2 of the Patent Act. Published on August 18, 2021 at https://orangearch-labo.com/.

Applicable to Article 30, Paragraph 2 of the Patent Act. Publicly disclosed on August 24, 2021, at Double Giken Co., Ltd. (920-7 Kurihara, Zama City, Kanagawa Prefecture).

This invention relates to an eye-tracking input device that allows users to select and input characters displayed on a screen using their gaze, and to a program for the eye-tracking input device.

In recent years, eye-tracking input devices have been developed to support communication for patients who are unable to adequately move their limbs or mouths due to severe paralysis or other conditions. These devices display multiple characters on a screen, allowing the user to select the desired character using their gaze, thereby enabling communication.

As an example of such an eye-tracking input device, Patent Document 1 discloses a device that displays a virtual character board on a screen, consisting of multiple character panels, such as hiragana characters, and allows the user to select a desired character panel by eye to input characters. This eye-tracking input device detects the user's gaze using a camera or the like, and when the user's gaze remains on one character panel on the character board, it accepts input for the character indicated by that panel and displays the character in a predetermined area on the screen. This allows the user to create a message by selecting character panels one by one using their gaze, thereby communicating their intentions to others.

Japanese Patent Publication No. 2019-149031

However, with conventional eye-tracking input devices, if an unintended character is accidentally entered, the user must shift their gaze to a delete button located in a specific position on the keyboard and continue to focus on it in order to undo the input. This can be stressful for the user.

This invention was made in view of the above problems, and its primary objective is to provide an eye-tracking input device that allows users to input characters by selecting from multiple character panels displayed on a screen using their gaze, while also enabling easy cancellation of input operations.

In other words, the eye-tracking input device according to the present invention displays a plurality of character panels on a screen, allowing the user to select a character panel using their gaze to input characters. It is characterized by comprising: a gaze point estimation unit that estimates the position of the user's gaze point within the screen; a character input unit that accepts input of the character shown in one of the character panels when the estimated gaze point remains within that panel; and, if the gaze point continues to remain within the character panel, switching the character panel to display a cancel button for canceling the accepted character input.

With this configuration, if the user continues to look at the character panel, it will switch to a cancel button for undoing character input. This allows the user to easily undo an unintended character input without significantly shifting their gaze (i.e., without moving their eyes away from the character panel).

Preferably, the character input unit decides to cancel the received character input if the point of focus remains on the cancel button.
This way, you can simply keep your eyes fixed on the text panel and then cancel text input without moving your gaze significantly, making the cancel operation much easier.

When the display control unit decides to cancel the input of the characters, it is preferable to switch the cancel button on the character panel and display it.
This way, even if a user accidentally undoes a character input, they can immediately start typing again without having to move their eyes.

The display control unit preferably displays the cancel button in the same size and position as the character panel.
This method allows the character panel to switch to a cancel button at the same size and position, making it easier to undo input operations. Furthermore, this design ensures the cancel button doesn't overlap with surrounding character panels, allowing users to quickly select another character panel after canceling.

Preferably, the eye-tracking input device has a display control unit that, when the gaze point remains within the character panel, switches the character displayed on the character panel to another pre-associated character, and the character input unit accepts character input each time the character displayed on the character panel changes.
With this configuration, as the user continues to look at a designated character panel, related characters will be displayed in a switching manner. This allows the user to select the desired related character without significantly shifting their gaze, thus reducing eye strain.
On the other hand, with this configuration, the characters change as the user continues to look at the character panel, and character input is accepted each time, making it easy for the user to input unintended characters. Therefore, the effect of the present invention is particularly noticeable when there is a character panel in which the characters change when the user's gaze remains fixed on it.

Furthermore, in the eye-tracking input device, it is preferable that the character input unit confirms the input of the received character when the point of focus moves outside the character panel.
With this configuration, users can easily input related characters by keeping their eyes on the character panel, selecting the related character they want to input, and then confirming the input by looking away from the character panel.
On the other hand, if character input is accepted while continuously looking at the character panel, the input will be confirmed as soon as the user looks away from the character panel, making it easy for the user to input unintended characters. Therefore, the effects of the present invention become even more pronounced in such a configuration.

Furthermore, a specific embodiment of the display control unit is one in which the characters shown in the character panel are switched to all pre-associated characters and displayed, and then the character panel is switched to the cancel button and displayed.
In this way, even if the user continues to look at the character panel and the desired related character does not appear, they can cancel the input operation that was accepted by continuing to look at the character panel.

A specific embodiment of the eye-tracking input device is one in which the character initially displayed on the character panel is a kana character with an unvoiced consonant, and the other characters displayed after switching are one or more selected from kana characters with voiced consonants, kana characters with semi-voiced consonants, kana characters with diphthongs, and small-sized kana characters.

Furthermore, the present invention relates to a program for an eye-tracking input device that displays a plurality of character panels on a screen and allows the user to input characters by selecting one of the character panels with their gaze. The program is characterized by causing a computer to perform the functions of a display control unit, which includes a gaze point estimation unit that estimates the position of the user's gaze point on the screen, a character input unit that accepts input of the character shown by one of the character panels when the estimated gaze point remains within that character panel, and a display control unit that switches the character panel to display as a cancel button for canceling the accepted character input when the gaze point continues to remain within that character panel.
Such a program for an eye-tracking device can achieve the same effects and advantages as the eye-tracking device of the present invention described above.

According to the present invention as described above, in an eye-tracking input device that allows users to input characters by selecting from multiple character panels displayed on a screen using their gaze, it is possible to easily cancel input operations.

A schematic diagram showing the overall configuration of the eye-tracking input device of this embodiment. A configuration diagram showing the equipment configuration of the eye-tracking input device according to the same embodiment. A functional block diagram showing the equipment configuration of the eye-tracking input device according to the same embodiment. This figure shows a portion of the display screen of the eye-tracking device according to the same embodiment. A screen display diagram showing the operation of the eye-tracking input device in the same embodiment. A screen display diagram showing the operation of the eye-tracking input device in the same embodiment. A diagram partially showing the display screen of an eye-tracking device in another embodiment.

An embodiment of the eye-tracking device of the present invention will be described below with reference to the drawings.

<Configuration of the eye-tracking input device 100>
The eye-tracking input device 100 of this embodiment is intended to support communication for people who have difficulty moving their hands, feet, or mouth, and allows users to select and input characters on a virtual keyboard (also called a character board) B displayed on a screen 31 of a personal computer or the like using their gaze.

Specifically, as shown in Figure 1, this eye-tracking input device 100 comprises an eye-tracking detection means 1 for detecting the user's gaze, and a notebook computer C that integrates a computer body 2 and a display 3. The screen 31 of the display 3 shows a virtual character board (hereinafter simply referred to as the character board) B, on which multiple characters such as hiragana are arranged, and a message panel M on which the entered characters are displayed. The user can input characters by continuously focusing on the desired character on the character board B, and these characters can be displayed on the message panel M. By inputting multiple characters in this way, a message can be displayed on the message panel M, allowing the user to communicate their intentions.

The gaze detection means 1 detects the direction of the user's gaze directed towards the screen 31 of the display 3 and outputs the detection result as gaze data. Specifically, this gaze detection means 1 utilizes the corneal reflection method and comprises a light source (not shown) that irradiates the user's eyeball with near-infrared light, a camera (not shown) that images the user's eyeball, and a gaze data output unit (not shown) that processes the image data output from the camera to detect the direction of the gaze and outputs it as gaze data. In this embodiment, the gaze detection means 1 is a so-called mounting type, and the light source and camera are mounted on the lower part of the display 3 so that they face the user's face.

The computer unit 2, as shown in Figure 2, is structurally equipped with a CPU 201, memory 202, input/output interface 203, etc. Based on various application software (hereinafter referred to as "programs") stored in the memory 202, the CPU 201 and its peripheral devices work together to perform at least the functions of a gaze point estimation unit 21, a character data storage unit 22, a character display control unit 23, a character input unit 24, and an input character display control unit 25, as shown in Figure 3. Each part will be described below.

The gaze point estimation unit 21 estimates the position of the user's gaze point G on the screen 31 based on the detection results of the gaze detection means 1. Specifically, it receives a gaze signal from the gaze detection means 1 and estimates the position of the user's gaze point G on the screen 31 based on the gaze signal. It then outputs gaze point data indicating the estimated position of gaze point G to the character input unit 24 and the character display control unit 23. In this embodiment, as shown in Figure 1, a pointer is displayed at the estimated position of gaze point G.

The character data storage unit 22 is set in a predetermined area of the memory 202 and stores character data indicating the codes of various characters to be displayed on the character dial B. In this embodiment, "characters" refer to, for example, hiragana, katakana, kanji, alphanumeric characters, Roman letters, symbols, emojis, emoticons, etc.

The character display control unit 23 creates character data representing character B, which has multiple characters arranged on it, by referring to the character data storage unit 22, and outputs this data to the character input unit 24 and the display 3. This character data includes information regarding the display mode of the characters and their display position on the screen.

As shown in Figure 1, the character display control unit 23 fixes and displays the entire character panel B at a predetermined position on the screen 31. The character panel B displays multiple square character panels P arranged in a grid (matrix) pattern, with each character panel P displaying one character (in this case, a voiceless hiragana character). Here, adjacent character panels P are displayed without gaps, but gaps may exist between character panels P. Furthermore, in each character panel P, the center position (centroid) of each character coincides with the center position of the character panel P. In this embodiment, a border line indicating the area of each character panel P is displayed on the screen 31, but this border line may be hidden.

Furthermore, the dial display control unit 23 is configured to switch the display of the character shown by a predetermined character panel P within the dial B to a pre-associated character (also called a related character) if the point of focus G remains continuously within that panel.

These related characters refer to characters that are conversion candidates for the parent character. For example, in this embodiment, the character data storage unit 22 stores data indicating a "unvoiced hiragana character" as the parent character, and stores one or more character data selected from its "voiced consonant character," "semi-voiced consonant character," "diphthong character," and "small character," etc., as related characters. For example, the character panel P indicating the unvoiced hiragana character "tsu" as the parent character has the voiced consonant character "zu" and the small character "tsu" associated with it.

The dial display control unit 23, when the point of focus G remains within a character panel P to which related characters are associated, switches and displays the related characters one by one in a predetermined order, replacing the parent character shown in the character panel P. After displaying all the related characters, it then displays the parent character again within the character panel P. In other words, the dial display control unit 23 is configured to repeatedly display the parent character and its associated characters within the character panel P. However, the dial display control unit 23 does not switch the character shown in the character panel P if the point of focus G remains within a character panel P to which no related characters are associated.

Each related character is displayed sequentially within the character panel P at approximately the same position as the parent character. That is, within the character panel P, the related characters are displayed sequentially so that their center positions coincide with the center positions of the parent character.

The display control unit 23 switches between the parent character and related characters at regular time intervals T1 (e.g., 1 second). Specifically, after the time interval T1 has elapsed since the point of focus G remained within the character panel P, the first related character is displayed in place of the parent character, and thereafter, related characters are successively switched and displayed at regular time intervals T1.

The character input unit 24 accepts character input based on the estimated position of the gaze point G and outputs this as input character data to the input character display control unit 25. Specifically, when the gaze point G remains within a character panel P, the character input unit 24 measures the time the gaze point G remains within that character panel P (i.e., the gaze time). When the measured gaze time reaches a predetermined time interval T2 (e.g., 1 second), the character input unit 24 accepts input for the character displayed on the character panel P. In other words, the character input unit 24 accepts input operations for the character displayed on the character panel P by the gaze point remaining within the character panel P (or, in other words, by the user continuing to gaze at the character panel P). The time interval T2 for determining whether to accept input is the same as the time interval T1 for the character within the character panel P to switch.

Furthermore, if the gaze point G remains within the character panel P to which the related characters are associated, the character input unit 24 accepts character input each time the character indicated by the character panel P changes, and if there is a character that has already been accepted for input, it updates (changes) the character that is currently in the input acceptance state. More specifically, if the gaze point G remains within the character panel P indicating the parent character for a predetermined time interval T1, and the character indicated by the character panel P switches to the first related character (first related character), the character input unit 24 accepts input of the parent character that was displayed before the switch. Then, if the gaze point G remains within the character panel P indicating the first related character for a predetermined time interval T1, and the character indicated by the character panel P switches to the next related character (second related character), the character input unit 24 updates (changes) the character that has been accepted for input from the parent character to the first character. In other words, when the point of focus G remains within the character panel P and the characters displayed on the character panel P change sequentially, the character that was displayed immediately before the change is considered the character for which input is being accepted.

The character input unit 24, while accepting character input, confirms the character input when the gaze point G moves outside the character panel P (or to another character panel P) and outputs an input confirmation signal to the input character display control unit 25 to indicate that the input has been confirmed.

The input character display control unit 25 displays the message panel M on the screen 31 and sequentially displays the entered characters within the message panel M. The message panel M displays not only characters that have been confirmed, but also characters that are still being accepted for input (i.e., characters before confirmation). The input character display control unit 25 immediately displays the character on the message panel M as soon as the character input unit 24 accepts input. Furthermore, the input character display control unit 25 immediately updates the display of the character being accepted for input within the message panel M as soon as the character input unit 24 updates the character being accepted for input. The input character display control unit 25 may display the characters after confirmation and the characters being accepted for input differently, or they may be displayed the same.

However, in this eye-tracking input device 100, to allow the user to easily cancel an input operation if they accidentally input an unintended character, the character display control unit 23 switches the character panel P to display a cancel button Q for canceling the input of the received character if the gaze point G remains fixed on the character panel P.

Specifically, the dial display control unit 23, when the point of focus G remains within the character panel P, switches the character panel P to display the cancel button Q after the input of the character shown on the character panel P has been received but before the input of the character has been confirmed. As shown in Figure 4, the dial display control unit 23 displays the cancel button Q at the same size and position as the character panel P.

If the point of focus G remains within the character panel P to which related characters are associated, the character display control unit 23 switches the characters within the character panel P to all pre-associated related characters and displays them, then switches the character panel P to the cancel button Q. More specifically, after a predetermined time interval T1 has elapsed since the last switched related character was displayed, and input of that related character is accepted (i.e., the character being input is updated), the character panel P is switched to the cancel button Q and displayed immediately.

On the other hand, if the point of focus G remains within a character panel P that is not associated with any related characters, the character display control unit 23 switches the character panel P to the cancel button Q and displays it simultaneously with or immediately after a predetermined time interval T2 has elapsed since the point of focus G began to remain and character input is received.

The character input unit 24 then decides to cancel the most recently received character input if the gaze point G remains on the cancel button Q for a predetermined time interval T3 (e.g., 1 s). This predetermined time interval T3 is the same as the time interval T1 during which characters in the character panel P switch. On the other hand, if the gaze point G moves away from the cancel button Q (i.e., moves to another character panel P) before the predetermined time interval T3 has elapsed, the accepted character input is confirmed. The input character display control unit 25 then deletes the character currently being input and displayed on the message panel M simultaneously or immediately after the character input unit 24 decides to cancel the character input. The character display control unit 23 also switches the cancel button Q back to the original character panel P and displays it simultaneously or immediately after the character input unit 24 decides to cancel the character input.

<Operation of the eye-tracking device 100>
Next, the operation of the eye-tracking input device 100 with this configuration will be explained using Figures 5 and 6.

(1) Input and Cancel Input Operations for Character Panels Not Associated with Related Characters As shown in Figure 5(a), when the point of focus G remains within a character panel P that is not associated with related characters (in this case, the character panel for "na"), the time that the point of focus G remains within the character panel P (gaze time) is measured. At this time, an indicator showing the elapsed time may be displayed within the character panel P.

When the measured gaze time reaches a predetermined time interval T2, the input of the character "な" is accepted, and the character "な" is displayed on the message panel M as shown in Figure 5(b). Simultaneously, the character panel P that displayed "な" switches to the cancel button Q. As soon as the cancel button Q is displayed, measurement of the time the gaze point G remains on the cancel button Q (gaze time) begins. At this time, an indicator showing the elapsed time may be displayed on the cancel button Q.

As shown in Figure 5(c), if the gaze point G moves away from the cancel button Q and to another character panel P (for example, the "ni" character panel) before the predetermined time interval T3 has elapsed since the cancel button Q was displayed, the input of the character "na" being received is confirmed, and at the same time, the cancel button Q switches back to the original character panel P (the character panel showing "na").

On the other hand, as shown in Figure 5(d), if the gaze point G remains on the cancel button Q for a predetermined time interval T3, the input of the character "な" that was being received is canceled, and the character "な" being received is deleted from the message panel M. Simultaneously, the cancel button Q switches back to the original character panel P (the character panel displaying "な").

(2) Input and Cancel Input Operations of Character Panels As shown in Figure 6(a), when the point of focus G remains within the character panel P to which the related character is associated (in this case, the character panel for "ha"), the time that the point of focus G remains within the character panel P (gaze time) is measured. At this time, an indicator showing the elapsed time may be displayed within the character panel P.

When the measured gaze time reaches a predetermined time interval T1, the input of the parent character "は" is accepted, and the character "は" is displayed on the message panel M as shown in Figure 6(b). Simultaneously, the character displayed on the character panel P switches from the parent character "は" to the first related character "ば," and the measurement of gaze time for the character panel P resumes.

As shown in Figure 6(c), if the gaze point G moves to another character panel P (for example, the "hi" character panel) before a predetermined time interval T1 has elapsed since switching to the first related character "ba," the input of the character "ha" being received is confirmed, and simultaneously, the character indicated by character panel P switches to the parent character "ha."

On the other hand, as shown in Figure 6(d), when the gaze time on the character panel P reaches a predetermined time interval T1, the character being input is updated (or changed) from "ha" to "ba," and the character "ha" displayed on the message panel M switches to the updated character "ba." Simultaneously, the character displayed on the character panel P switches from the first related character "ba" to the second related character "pa," and the measurement of gaze time on the character panel P restarts.

As shown in Figure 6(e), if the gaze point G moves to another character panel P before the predetermined time interval T1 has elapsed after switching to the second related character "pa", the input of the character "ba" being received is confirmed, and at the same time, the character indicated by character panel P switches to the parent character "ha".

On the other hand, as shown in Figure 6(f), when the gaze time on the character panel P reaches a predetermined time interval T1, the character being input is updated from "ba" to "pa," and the character "ba" displayed on the message panel M switches to the updated character "pa." Simultaneously, the character panel P, which was displaying "pa," switches to the cancel button Q, and the measurement of the time the gaze point G remains on the cancel button Q (gaze time) begins.

As shown in Figure 6(g), if the point of focus G moves away from the cancel button Q and to another character panel P before the predetermined time interval T3 has elapsed since the cancel button Q was displayed, the input of the character "pa" being received is confirmed, and at the same time, the cancel button Q switches back to the original character panel P (the character panel showing "ha") which represents the parent character.

On the other hand, as shown in Figure 6(h), if the gaze point G remains on the cancel button Q for a predetermined time interval T3, the input of the character "pa" that was being received is canceled, and the character "pa" being received is deleted from the message panel M. Simultaneously, the cancel button Q switches back to the original character panel P (the character panel showing "ha").

<Effects of the eye-tracking input device 100 of this embodiment>
With the eye-tracking input device 100 of this embodiment configured in this way, when the user continues to look at the character panel P, the character panel P switches to a cancel button Q for canceling character input. Therefore, when the user accidentally inputs an unintended character, they can easily cancel the erroneous input operation without having to move their gaze significantly (i.e., without moving their gaze away from the character panel P).

<Other Embodiments>
However, the present invention is not limited to the embodiments described above.
For example, the eye-tracking input device 100 in the above embodiment fixed and displayed the entire character dial B within the screen 31, but is not limited to this. In other embodiments of the eye-tracking input device 100, the character dial B may move within the screen 31 according to the user's gaze. In this eye-tracking input device 100, as shown in Figure 7, the user can input characters by moving their gaze to move the character dial B within the screen 31 and aligning the desired character panel with a selection position S set at a predetermined position within the screen 31. In this embodiment, the character dial display control unit 23 moves the character dial B within the screen 31 based on gaze point data acquired from the gaze point estimation unit 21. Specifically, as shown in Figure 7(a), the character dial display control unit 23 moves the character dial B so that the character panel P closest to the gaze point G moves toward the selection position S within the screen 31. Then, as shown in Figure 7(b), when the center of the character panel P where the gaze point G is located coincides with the selection position S, the movement of the character dial B stops. In this state, the gaze time on the character panel P begins, and thereafter, character input and input cancellation operations can be performed in the same manner as with the eye-tracking input device 100 of the above embodiment.

In the embodiments described above, multiple character panels P were arranged in a grid pattern at equal intervals within the character board B, but this is not limited to this arrangement. For example, the multiple character panels P may be arranged radially or in a single row.

Furthermore, in the above embodiment, the cancel button Q was displayed at the same size and position as the character panel P, but this is not limited to this. In other embodiments, the cancel button Q may be displayed with a larger area than the character panel P before the switch.

In the embodiments described above, the gaze detection means 1 was of the mountable type, but it is not limited to this. In other embodiments, the gaze detection means 1 may be integrated with the display 3 or the computer body 2, or it may be a wearable type attached to the user's body. It may also be attached to any location, such as the top of the display 3.

In the embodiments described above, the computer body 2 implemented its functions as a gaze-point estimation unit 21, etc., based on a program stored in the memory 202, but it is not limited to this. In other embodiments, the computer body 2 may access a cloud via a network such as the Internet, and the CPU 201 and its peripheral devices may work together to implement its functions as a gaze-point estimation unit 21, etc., based on a program stored on the cloud.

In the embodiments described above, the computer body 2 and the display 3 were integrated into a notebook computer, but this is not limited to that. In other embodiments, they may be separate components. The display 3 may also consist of a projector and a projection screen. Furthermore, it is not limited to a notebook computer; it may be a mobile device such as a smartphone or tablet, or a wearable device such as a head-mounted display.

Furthermore, the eye-tracking input device 100 of other embodiments may not be limited to applications that support communication for people who have difficulty moving their limbs or mouths, but may also be used to support text input by able-bodied individuals using their eyes.

Furthermore, the present invention is not limited to the embodiments described above, and it goes without saying that various modifications are possible without departing from its spirit.

100... Eye-tracking input device 1... Eye-tracking detection means 21... Point of focus estimation unit 23... Character display control unit 24... Character input unit 31... Screen G... Point of focus position P... Character panel

Claims (9)

An eye-tracking input device that displays multiple character panels on a screen and allows the user to input characters by selecting from these multiple character panels using their gaze,
A gaze point estimation unit that estimates the position of the user's gaze point within the aforementioned screen,
When the estimated point of focus remains within one of the character panels, a character input unit accepts input of the character indicated by that character panel,
A gaze input device comprising: a display control unit that, when the gaze point remains within the character panel, switches the character panel to display a cancel button for canceling the received character input. The eye-tracking input device according to claim 1, wherein the character input unit determines to cancel the received character input if the gaze point remains on the cancel button. The eye-tracking input device according to claim 2, wherein the display control unit, upon determining whether to cancel the character input, switches the cancel button on the character panel for display. The eye-tracking input device according to claim 1, wherein the display control unit displays the cancel button in the same size and position as the character panel. If the point of focus remains within the character panel, the display control unit switches the character displayed by the character panel to another pre-associated character , and then switches the character panel to the cancel button .
The eye-tracking input device according to claim 1, wherein the character input unit updates the character being input each time the character displayed on the character panel changes. The eye-tracking input device according to claim 5, wherein the character input unit confirms the input of the character being received when the gaze point moves outside the character panel. The eye-tracking input device according to claim 5 or 6, wherein the display control unit switches the characters shown in the character panel to all pre-associated characters and displays them, and then switches the character panel to display the cancel button. The eye-tracking input device according to claim 5, wherein the character initially displayed on the character panel is a kana character with a clear consonant, and the other characters displayed after switching to this character are one or more selected from kana characters with voiced consonants, kana characters with semi-voiced consonants, kana characters with diphthongs, and small-lettered kana characters. A program for an eye-tracking input device that displays multiple character panels on a screen and allows the user to input characters by selecting from these multiple character panels using their gaze,
A gaze point estimation unit that estimates the position of the user's gaze point within the aforementioned screen,
When the estimated point of focus remains within one of the character panels, a character input unit accepts input of the character indicated by that character panel,
A program for an eye-tracking input device that causes a computer to function as a display control unit, which, if the point of focus remains within the character panel, switches the character panel to display a cancel button for canceling the received character input.