JP7644478B2 - Character input device, character input system, character input method, and character input program - Google Patents

Description

The present invention relates to a character input device, a character input system, a character input method, and a character input program.

Previously, eye-gaze input devices have been proposed that allow efficient character input by displaying characters of sufficient size (for example, Patent Document 1). In this technology, a portion of a group of multiple types of characters is displayed in the character area. A character in the character area is selected and input using a gaze cursor that indicates the position of the observer's gaze. When a character that is not displayed in the character area is to be input, the gaze cursor is moved to the position of a character that is partially displayed on the edge of the character area. When the gaze cursor is moved to this position, the group of characters in the character area is scrolled and the new character is displayed in the character area.

JP 2000-20196 A JP 2014-158144 A JP 2002-215288 A Special Publication No. 2009-545802 JP 2011-59998 A Special Publication No. 2011-529597 JP 2020-5114 A

When entering characters by eye gaze, the distance the eyes must move on the screen is generally large, causing eye fatigue. In addition, the more characters displayed on the screen, the more likely input errors will occur. On the other hand, when scrolling through character options on the screen, there is a problem of waiting time until the desired option appears, reducing input efficiency.

The present invention aims to improve the ease of entering text using eye gaze.

The character input device has a gaze detection unit that switches between a first screen including a plurality of regions related to consonants and a second screen including a plurality of regions related to vowels and displays them on the display device, and that detects the gaze of a user facing the display device, acquires information indicating the gaze point from a gaze detection device that outputs the gaze point of the user, and determines that a predetermined area has been selected when the gaze point indicated by the acquired information is included in the predetermined area for a predetermined gaze time or more, and a character input unit that accepts input of a character according to a combination of the area selected on the first screen and the area selected on the second screen based on the judgment of the gaze detection unit. In addition, the multiple areas included in the first screen and the multiple areas included in the second screen are displayed at a distance of a predetermined distance or more from other areas included in the same screen.

When inputting characters by the user's line of sight, there tends to be a large number of gaze areas as options, but by inputting options related to consonants and options related to vowels on two separate screens, it is possible to prevent erroneous inputs such as inputting unintended characters. In addition, erroneous inputs can also be prevented by displaying the multiple areas as options at a predetermined distance or more from other areas included in the same screen.

The gaze detection unit may also provide a waiting time during which it does not determine whether the gaze point is included in a predetermined area for a predetermined gaze time or longer after switching the display of the display device between the first screen and the second screen. This waiting time can prevent erroneous input based on the user's gaze immediately after the screen is switched.

The multiple regions included in the first screen and the multiple regions included in the second screen may each be arranged on an arc. In this way, the distance the line of sight moves to each region is approximately the same, reducing user fatigue.

The first screen may further include a plurality of areas for the user to express a predetermined intention, with the plurality of areas relating to consonants being arranged on the outer arc of the double circle, and the plurality of areas for expressing a predetermined intention being arranged on the inner arc of the double circle. For example, being able to express an intention of affirmation or negation in parallel while inputting characters is useful when a question is suddenly asked, for example.

The gaze detection unit may also be configured to allocate a longer gaze time to multiple areas on the first screen for expressing a predetermined intention than to multiple areas corresponding to rows to determine that an area has been selected. The gaze often passes through the areas located on the inside, and they are used relatively infrequently, so by setting a difference in gaze time, it is possible to prevent a decrease in operability while also reducing the occurrence of erroneous input.

The first screen may further include a temporary text box that temporarily displays the character string that has been input, and a decision area for outputting the character strings displayed in the temporary text box to another specified text box or outputting them as sound, and when the gaze detection unit determines that the decision area has been selected, the character input unit may output the character strings displayed in the temporary text box to another specified text box or output them as sound, and when the gaze detection unit determines that the area for making a specified expression of intent has been selected, the character input unit may output according to the selected area for making a specified expression of intent prior to outputting the character string displayed in the temporary text box. For example, this may allow a user to express an affirmative or negative intent in parallel with the character input, which is useful when a user is suddenly asked a question.

The first screen may further include a rest button area for stopping the process by the gaze detection unit that determines that the area has been selected, and when the gaze detection unit determines that the rest button area has been selected, it may stop accepting operations other than those to cancel the stop. This can reduce user fatigue caused by continuous operations.

In addition, each of the multiple areas included in the second screen may be displayed at a position on the display device that does not overlap with the position where the selected area is displayed on the first screen. In this way, it is possible to prevent erroneous input based on the user's line of sight immediately after the screen is switched.

The first screen may also include a change area for changing the gaze time or waiting time, and the gaze detection unit may increase or decrease the gaze time or waiting time when it is determined that the change area has been selected. Because the gaze time or waiting time can be changed depending on the user's level of proficiency, operability may be improved.

In addition, the specified distance may be the radius r of the centrally visible range calculated by the following formula, where l is the expected distance from the user's eyeball to the point of gaze on the display device, and θ is the angle between a straight line connecting the eyeball and the point of gaze and a straight line connecting the eyeball and the edge of the centrally visible range on the display device.
r = l tan θ
In addition, θ may be 1 to 2 degrees. Specifically, by adopting such a size, it is generally possible to clearly recognize an object by central vision, and to suppress unintended control by the user due to microsaccades that occur during fixation.

The contents of the means for solving the above problems can be combined as much as possible without departing from the objectives and technical ideas of the present invention. The above means may also be realized as a method executed by a computer, or as a program for causing a computer to execute the above means. The program may be provided by recording it on a computer-readable recording medium.

The present invention can improve the ease of entering characters using eye gaze.

FIG. 1 is a diagram showing an example of a character input system according to the present embodiment. FIG. 2 is a functional block diagram showing an example of a character input system. FIG. 3 is a diagram showing an example of the first screen. FIG. 4 is a diagram showing an example of the second screen. FIG. 5 is a diagram for explaining the position and size of the operation area set on the display device. FIG. 6 is a diagram showing the relationship between the first screen shown in FIG. 3 and the above-mentioned radius r. FIG. 7 is a diagram showing the relationship between the second screen shown in FIG. 4 and the above-mentioned radius r. FIG. 8 is a process flow diagram showing an example of a process performed by the display device. FIG. 9 is a process flow diagram showing an example of a process performed by the gaze detection device. FIG. 10 is a process flow diagram illustrating an example of a process performed by the control device. FIG. 11 is a process flow diagram illustrating an example of a process performed by the control device. FIG. 12 is a process flow diagram illustrating an example of a process performed by the control device. FIG. 13 is a process flow diagram illustrating an example of a process performed by the control device. FIG. 14 is a diagram showing an example of the second screen according to the modified example. FIG. 15 is a diagram showing an example of the second screen according to another modified example. FIG. 16 is a diagram showing an example of the first screen according to another modified example. FIG. 17 is a diagram showing an example of the second screen according to another modified example. FIG. 18 is a diagram showing an example of the second screen according to another modified example. FIG. 19 is a functional block diagram showing an example of a character input device.

The character input system according to an embodiment of the present invention will be described below with reference to the drawings.

<Device Configuration>
FIG. 1 is a configuration diagram showing an example of a character input system according to the present embodiment. The character input system 1 shown in FIG. 1 includes a control device 2, a display device 3, and a gaze detection device 4. The character input system 1 is a system for inputting characters based on the user's gaze. The gaze detection device 4 includes, for example, a camera that captures infrared light, and detects the gaze based on the movement of the user's eyes. The control device 2 displays an interface for inputting characters on the display device 3, and acquires information indicating a gaze point, which is a part on the display device 3 where the user is gazing (or fixating), from the gaze detection device 4, and accepts input of a character selected according to the position of the gaze point. The inputted characters may be outputted as voice from a speaker, for example, and used for conversation with other users. The inputted characters may also be outputted to a text box of other software for performing text chat, for example. The control device 2 corresponds to the "character input device" in the present invention.

<Functional configuration>
2 is a functional block diagram showing an example of a character input system 1. The character input system 1 in FIG.

The control device 2 is a general computer, such as a PC (Personal Computer), a smartphone, or a tablet. The control device 2 also includes a communication interface (I/F) 21, a storage device 22, an input/output device 23, and a processor 24, which are connected via a bus 25. The communication I/F 21 is a wireless module for communicating using Wi-Fi (registered trademark), Bluetooth (registered trademark), or other RF (Radio Frequency) signals, a wired network card, a parallel port, a serial port, or the like, and communicates with other devices based on a predetermined protocol. The storage device 22 is a main storage device such as a RAM (Random Access Memory) or a ROM (Read Only Memory), and an auxiliary storage device (secondary storage device) such as a HDD (Hard-disk Drive), an SSD (Solid State Drive), or a flash memory. The main storage device temporarily stores a program read by the processor 24 and secures a working area for the processor 24. The auxiliary storage device stores programs executed by the processor 24, etc. The input/output device 23 is a user interface such as a keyboard, a mouse, or a touch panel. The input/output device 23 may also function as the display device 3. The processor 24 is an arithmetic processing device such as a CPU (Central Processing Unit), and executes a program to perform each process described in this embodiment. In addition, in the example of FIG. 2, functional blocks are shown in the processor 24. Specifically, the processor 24 includes an I/F output unit 241, a gaze detection unit 242, a character input unit 243, and a setting change unit 244. The I/F output unit 241 switches between a first screen including a plurality of areas related to consonants and a second screen including a plurality of areas related to vowels to display them on the display device. The gaze detection unit 242 acquires information on the movement of the user's gaze from the gaze detection device 4 via the communication I/F 21, and detects the selection of a character by the user according to the position of the gaze point. The character input unit 243 accepts input of a character selected by the user according to the position of the gaze point. The setting change unit 244 accepts a change in the setting regarding the operation of the present system according to the position of the gaze point.

The display device 3 is an image display device that acquires and displays a video signal of an image (still image or video) output by a computer or the like. In FIG. 2, functional blocks are also shown within the display device 3. The display device 3 includes a communication I/F 31 and a display unit 32. The communication I/F 31 is an interface for receiving input of a video signal, such as an HDMI (High-Definition Multimedia Interface, registered trademark) terminal, a DVI (Digital Visual Interface) terminal, a VGA terminal, a wireless communication module, etc. The display unit 32 is a liquid crystal display (LCD), a plasma display (PDP), an organic electroluminescence display, a cathode ray tube (CRT), etc., and outputs an image according to the video signal input via the communication I/F 31.

The gaze detection device 4 is a device that detects the movement of the user's gaze and identifies the gaze point. In Fig. 2, functional blocks are also shown in the gaze detection device 4. The gaze detection device 4 includes a communication I/F 41, an infrared transmission/reception unit 42, and a gaze detection unit 43. The communication I/F 41 is an interface such as a USB (Universal Serial Bus) and outputs gaze point information to the control device 2. The infrared transmission/reception unit 42 irradiates infrared rays to a user facing the gaze detection device 4 and captures an image of the reflected light. The gaze detection unit 43 detects the gaze direction based on the position of the pupil in the eye from an image of the user and calculates the position of the gaze point on the display device.

<User Interface>
In this embodiment, character input is accepted based on a combination of areas on the first screen and the second screen that the user gazes upon. The first screen is a user interface including an area related to input of consonants, and the second screen is a user interface including an area related to input of vowels. For example, the first screen may include a plurality of areas corresponding to rows in the Japanese syllabary, and the second screen may include a plurality of areas corresponding to columns.

Figure 3 is a diagram showing an example of the first screen. The circular area in Figure 3 is an operation area that accepts some operation by the user gazing at it for a predetermined period of time. In the example of Figure 3, a plurality of operation areas arranged on double arcs are displayed near the center of the display unit 32. On the outer arc, operation areas 101 to 112 are arranged clockwise from the left (the 9 o'clock direction when viewed from the center), displaying "a", "ka", "sa", "ta", "na", "ha", "ma", "ya", "ra", "wa/-", "゛゜small", and "x". On the inner semicircular arc, operation areas 113 to 116 are arranged clockwise from the left, displaying an upward arrow, "Yes", "No", and "Confirm". In addition, operation areas 120 to 124 are arranged on the right side of the display unit 32, displaying "First", "Speaking", an upward triangle, a downward triangle, and "Rest". In addition, the rectangular area in Figure 3 is a text box. Near the center of the display unit 32, there are arranged a text box 117 in which the input character string is temporarily displayed, a first predictive conversion field 118 in which the predictive conversion result of the character string being input is displayed, and a second predictive conversion field 119. In addition, on the left side of the display unit 32, there is arranged a text box 125 in which the input character string is displayed.

The operation areas 101 to 110, in which "a", "ka", "sa", "ta", "na", "ha", "ma", "ya", "ra" and "wa/-" are displayed, are areas related to inputting consonants. For example, "a" is an area for selecting the A row (no consonant input), "ka" is an area for selecting the Ka row (/k/), and "sa" is an area for selecting the Sa row (/s/). The areas related to inputting consonants are used to input characters in combination with the selection of the area related to inputting vowels on the second screen, which will be described later. The operation area 111, in which "゛゜小" is displayed, is used to add a voiced or semi-voiced mark to the character input immediately before, or to change it to a kana represented by small characters (cut kana). The operation area 112, in which "x" is displayed, is used to cancel the character input immediately before.

Figure 4 is a diagram showing an example of the second screen. The circular area in Figure 4 is also an operation area that accepts some operation by the user gazing at it for a predetermined period of time. In the example of Figure 4, a plurality of operation areas arranged on an arc are displayed near the center of the display unit 32. On the arc, operation areas 131 to 136 are arranged clockwise from the upper left (the 10 o'clock direction when viewed from the center), displaying "a", "i", "u", "e", "o" and "x", respectively. The rectangular area in Figure 4 is also a text box. Text boxes 117 and 125 are the same as the corresponding components in Figure 3. Note that after the first screen and the second screen are switched, a waiting time is set during which input operations by line of sight are not accepted, to suppress erroneous input (input contrary to the user's intention).

The operation areas 131 to 135 in which "a", "i", "u", "e" and "o" are displayed are areas related to the input of vowels. For example, "a" is an area for selecting the a-row (/a/), "i" is an area for selecting the i-row (/i/), and "u" is an area for selecting the u-row (/u/). The areas related to the input of vowels are used to input characters in combination with the selection of the area related to the input of consonants on the first screen. Therefore, for example, when the ka row is selected on the first screen, "ka", "ki", "ku", "ke", and "ko" are displayed in the operation areas 131 to 135, and when the sa row is selected on the first screen, "sa", "shi", "su", "se", and "so" are displayed in the operation areas 131 to 135. Also, when the ya row is selected on the first screen, "ya", "yu", "yo", "," (comma), and "." (period) may be displayed in the operation areas 131 to 135. When the wa row is selected on the first screen, "wa", "wo", "n", "-" (macro note), and "_" (underscore) may be displayed in operation areas 131 to 135. Also, operation area 136 on the second screen displaying "x" is used to cancel an operation being input and return to the first screen.

Returning to the explanation of FIG. 3, the input characters are displayed in the text box 117. Furthermore, according to the character string being input, the first predictive conversion field 118 and the second predictive conversion field 119 display the predicted results. Note that character strings that are frequently used can be stored in advance and displayed as predicted results. The operation area 113 with the upward arrow displayed is an area for inputting the predicted result displayed in the first predictive conversion field 118 into the text box 117. At this time, the predicted result displayed in the second predictive conversion field 119 is displayed in the first predictive conversion field 118, and a new predicted result candidate is displayed in the second predictive conversion field 119. The user can select the next candidate by gazing at the operation area 113 with the upward arrow displayed.

The input character string is first added to the text box 117, and is moved to the text box 125 in the order of the oldest character. The operation area 116 in which "OK" is displayed is an area for outputting the character strings input in the text boxes 117 and 125 by voice or to a text box of another software. That is, the text boxes 117 and 125 correspond to the "temporary text box" of the present invention, and the text box of the other software corresponds to the "other specified text box" of the present invention. The operation areas 114 and 115 in which "Yes" and "No" are displayed are used to indicate affirmative and negative intentions. When "Yes" and "No" are selected, the character strings displayed in the text boxes 117 and 125 are not changed, and "Yes" and "No" are output by voice or to a text box of another software, respectively. That is, the process of outputting the affirmative or negative intention is interrupted. Such an output is useful because it can be used to answer another question while inputting characters.

The operation area 120 displaying "First" is an area for changing the setting to a predetermined beginner mode. In the beginner mode, the waiting time after the first screen and the second screen are switched is set to, for example, two seconds. The operation area 121 displaying "Speak" is an operation area for turning on/off (enabling or disabling) the audio output when outputting a character string. The operation areas 122 and 123 displaying an upward triangle and a downward triangle are areas for increasing and decreasing, respectively, the gaze time, which is the time that gaze on the operation area is required to determine that the operation area has been selected. The operation area 124 displaying "Break" is an area for temporarily stopping the acceptance of operations by gaze in this system. If the acceptance of operations is temporarily stopped, the processing of this system will resume when the operation area 124 is gazed on again.

3 and 4, the user inputs the next character by moving his/her gaze mainly to each operation area arranged on the arc while checking the character input in the central text box 117. Since each operation area is approximately equidistant from the center, the degree of fatigue does not vary depending on the character to be input. In addition, the arrangement of the operation areas is relatively easy to remember, and even if the arrangement is not remembered, the user can find the desired operation area by searching clockwise on the arc on which the operation areas are arranged.

The gaze time required to determine that an operation area has been selected on the first screen may be different between the inside and outside of the double circle. For example, the gaze time for the operation area on the inner arc may be 1.0 second, and the gaze time for the operation area on the outer arc may be 0.4 seconds. In other words, when generally inputting text, the inner operation area is used less frequently than the outer operation area. Also, when a user checks the characters input in the central text box 117 and moves their gaze to the outer operation area to select the next character to input, their gaze passes through the inner operation area, which may result in erroneous input. By setting a longer gaze time on the inner side, such erroneous input can be suppressed.

Figure 5 is a diagram for explaining the position and size of the operation area set on the display device. In this embodiment, the operation area is set based on the human visual field and eye movement. Figure 5 shows the relationship between the user's viewpoint (eyeball) and the gaze point on the display unit 32. Generally, for a human to identify the shape of an object with high resolution, central vision is required, which projects a retinal image at the central part of the retina with a visual angle of about 1 to 2 degrees. In this embodiment, the size of the operation area is set to at least the range that can be captured by central vision.

When a person continues to fixate (fixate) an object with central vision, an eye movement called a microsaccade (fixational eye movement) occurs. A microsaccade is an involuntary movement of the gaze with an amplitude of about 1 to 2 degrees. Although there is some variation, the average speed is about 10 degrees/second and the frequency is about 1 to 3 Hz. When the gaze detection device 4 is capable of detecting changes in the gaze caused by microsaccades, depending on the arrangement or size of the operation area, the microsaccades may cause the user to unconsciously perform an erroneous operation. Therefore, in this embodiment, the operation area is set with an arrangement and size that ensures that the user does not move away from the area he or she is gazing at due to microsaccades.

The radius r of the circular operation area shown in Figures 3 and 4 can be calculated by the following formula (1), where l is the distance from the eyeball to the gaze point, and θ is the angle between the line connecting the eyeball and the gaze point and the line connecting the eyeball and the edge of the centrally visible range on the image displayed on the display unit 32. That is, r is the radius of the centrally visible range. Also, θ can be said to be the angle at which the line of sight moves due to microsaccades.
r=ltanθ...(1)
For example, if l is 550 mm and θ is 2 degrees, then r is 19.2 mm.

By setting the radius of the operation area to r, when the user gazes at the center of the operation area, even if the user is affected by microsaccades, the detected gaze point can be prevented from straying from the operation area. FIG. 6 is a diagram showing the relationship between the first screen shown in FIG. 3 and the above-mentioned radius r. The outer circle on which the centers of the operation areas 101 to 110 for inputting consonants are located on the first screen has a radius of 6r. On the outer circle, operation areas with a radius of r are arranged at intervals of 30 degrees from the center. On the inner circle on which the centers of the operation areas 114, 115 for inputting intentions are arranged on the first screen has a radius of 3r. On the inner circle, operation areas with a radius of r are arranged at intervals of 60 degrees from the center. Therefore, the distance between the operation areas 113 and 114, between the operation areas 114 and 115, and between the operation areas 115 and 116 arranged on the inner circle is also r. In this way, each operation area is placed at least r apart, so if the user is looking at the edge of an operation area, even if there is an effect of microsaccades, the detected gaze point is prevented from being determined to be within another operation area.

FIG. 7 is a diagram showing the relationship between the second screen shown in FIG. 4 and the above-mentioned radius r. The circle on which the centers of the operation areas 131 to 135 for inputting vowels are arranged on the second screen has a radius of 6r. On the circle, the operation areas of radius r are arranged at intervals of 60 degrees from the center. Since the operation areas are also arranged at intervals of r or more on the second screen, if the user looks at the edge of the operation area, even if the user is affected by microsaccades, it is possible to prevent the detected gaze point from being determined to be within another operation area. As described above, for example, with the arrangement of the operation areas shown in FIG. 6 and FIG. 7, it is possible to suppress unconscious erroneous operation (erroneous detection) due to the influence of microsaccades.

<Character input processing>
FIG. 8 is a process flow diagram showing an example of the process performed by the display device. The display device 3 receives a video signal for displaying the above-mentioned first screen and second screen from the control device 2 via the communication I/F 31 (FIG. 8: S1). After that, the display unit 32 of the display device 3 displays the images of the first screen and the second screen (S2). The display unit 32 also judges whether to continue the display operation (S3). For example, when a user operates to end the operation, when a video signal is not received for a predetermined time or more, the display unit 32 judges to end the operation. When the operation is continued (S3: YES), the process returns to S1 and is repeated. On the other hand, when the operation is not continued (S3: NO), the process is ended.

9 is a process flow diagram showing an example of the process performed by the gaze detection device. When the gaze detection device 4 is installed on the display device 3, the infrared transmission/reception unit 42 transmits infrared rays in front of the display device 3 and receives the reflected light (FIG. 9: S11). After that, the gaze detection unit 43 of the gaze detection device 4 judges whether the gaze of the user has been detected from the reflected light (S12). In this step, the gaze detection unit 43 of the gaze detection device 4 identifies the position of the pupil in the eye from, for example, the image of the user, and judges that the gaze has been detected if it has been identified. If the gaze has been detected (S12: YES), the gaze detection unit 43 calculates the gaze point based on, for example, the position of the pupil (S13). In this step, an existing gaze detection technology can be used, and the position of the gaze is calculated based on the position of the pupil relative to some reference, such as the reflected light of infrared rays from the inner corner of the eye or the cornea. The gaze point is calculated, for example, as a coordinate on the display device 3 or on the image displayed on the display device 3. Furthermore, the gaze detection unit 43 transmits information indicating the position of the gaze point (gazing point data) to the control device 2 via the communication I/F 41 (S14). After S14, or if the gaze cannot be detected in S12 (S12: NO), the gaze detection device 4 determines whether to continue the gaze detection operation (S15). In this step, it determines to end the operation, for example, when an operation to end the operation is received from the user. If the operation is to be continued (S15: YES), the process returns to S11 and is repeated. On the other hand, if the operation is not to be continued (S15: NO), the process is ended.

Figures 10 to 13 are process flow diagrams showing an example of the process performed by the control device. The I/F output unit 241 of the control device 2 outputs a first screen including an area related to inputting consonants to the display device 3 (Figure 10: S21). In this step, an image such as that shown in Figure 3 is displayed on the display device 3. The gaze detection unit 242 of the control device 2 also acquires information indicating the position of the user's gaze point on the display unit 32 of the display device 3 (on the image displayed on the display unit 32) from the gaze detection device 4 (S22). The gaze detection unit 242 also determines whether a consonant or the like has been selected based on the acquired information indicating the position of the gaze point (S23). In this step, it is determined whether any of the operation areas 101 to 111 of the first screen has been selected.

If it is determined that a consonant or the like has been selected (S23: YES), the I/F output unit 241 of the control device 2 causes the display device 3 to output the second screen including an area for inputting a vowel (S24). In this step, an image as shown in FIG. 4 is displayed on the display device 3. After switching to the second screen, it is assumed that no input by gaze is accepted for a predetermined waiting time (S25). In addition, the gaze detection unit 242 of the control device 2 acquires information indicating the position of the user's gaze point on the display unit 32 of the display device 3 from the gaze detection device 4 (S26). Then, the gaze detection unit 242 determines whether a vowel or the like has been selected based on the acquired information indicating the position of the gaze point (S27). In this step, it is determined whether any of the operation areas 131 to 135 of the second screen has been selected.

If it is determined that a vowel or the like has been selected (S27: YES), the character input unit 243 accepts character input based on the combination of the consonant determined to have been selected in S23 and the vowel determined to have been selected in S27 (S28). In this step, the character input unit 243 adds the character specified by the selected consonant and vowel to the text box 117.

In addition, in S28, the character input unit 243 extracts the results of predictive conversion from pre-registered fixed phrases based on the character string displayed in the text box 117, and displays them in the first predictive conversion field 118 and the second predictive conversion field 119. The predicted results are extracted from the character strings stored in the storage device 22 as fixed phrases in advance that match the character string being input. In addition, for example, a character string that has been used many times in input during a recent predetermined period may be preferentially extracted. In this way, the fixed phrases with higher priority based on a predetermined criterion are displayed in the first predictive conversion field 118 and the second predictive conversion field 119. On the other hand, if a vowel is not selected in S27 (S27: NO), such as when the area 136 of the second screen is selected, or after S28, the process proceeds to the process of FIG. 11. Then, the I/F output unit 241 outputs the first screen including the area related to the input of consonants to the display device 3 (FIG. 11: S29). In this step, an image as shown in FIG. 3 is displayed on the display device 3. After switching to the first screen, eye-gaze input is not accepted for a predetermined waiting time (S30). Then, the process returns to S22 and is repeated.

Also, if it is determined in S23 of FIG. 10 that a character other than a consonant has been selected, the process proceeds to the process of FIG. 12. The gaze detection unit 242 determines whether a change to the most recently input character has been instructed based on the acquired information indicating the position of the gaze point (FIG. 12: S31). In this step, it is determined whether the user's gaze point is located in the operation area 111 with a voiced consonant, a semi-voiced consonant, or a small character (cut kana) for a predetermined time or more. If the operation area 111 is selected (S31: YES), if the most recently input character displayed in the text box 117 is a character to be changed, a voiced consonant or a semi-voiced consonant is added, or the character is changed to a small character (S32). In this step, for example, if the most recently input character is a character in the ka row, sa row, ta row, or ha row, the character input unit 243 changes it to a voiced consonant. If the most recently input character is a character in the a row or ya row, the character is changed to a small character. Also, if the immediately preceding character is a character in the ba row or "zu", it is changed to a semi-voiced sound or "っ" (short consonant), respectively. Note that if the immediately preceding character is a character in the ga row, za row, a character in the da row or pa row other than "zu", or "っ", it may be changed back to an unvoiced sound character. Then, the process returns to S21 in FIG. 10.

In addition, if it is determined in S31 that a change in character has not been instructed (S31: NO), the gaze detection unit 242 determines whether input of a fixed phrase (i.e., a result of predictive conversion) has been selected based on the acquired information indicating the position of the gaze point (S33). In this step, it is determined whether the user's gaze point is located in the operation area 113 with an upward arrow for a predetermined period of time or more. If the operation area 113 is selected (S33: YES), input of a fixed phrase is accepted (S34). In this step, the character input unit 243 inputs the prediction result displayed in the first predictive conversion field 118 into the text box 117, displays the prediction result displayed in the second predictive conversion field 119 in the first predictive conversion field 118, and displays new prediction result candidates extracted based on a predetermined priority in the second predictive conversion field 119. Then, the process returns to S21 in FIG. 10.

Also, if it is determined in S33 that a fixed phrase has not been selected (S33: NO), the gaze detection unit 242 determines whether a predetermined expression of intent has been selected based on the acquired information indicating the position of the gaze point (S35). In this step, it is determined whether the user's gaze point is located in the operation area 114 marked with "Yes" or in the operation area 115 marked with "No" for a predetermined period of time or more. If the operation area 114 or 115 is selected (S35: YES), the input of the expression of intent is accepted by interrupt (S36). In this step, the character input unit 243 outputs "Yes" or "No" by voice or in a text box of another software according to the selected operation area. Then, the process returns to S21 in FIG. 10.

If it is determined in S35 that a predetermined intention expression has not been selected, the gaze detection unit 242 determines whether a setting change of the system has been instructed based on the acquired information indicating the position of the gaze point (S37). In this step, it is determined whether the user's gaze point is located for a predetermined time or more in the operation area 120 marked with "first", the operation area 121 marked with "utterance", the operation area 122 marked with an upward triangle, or the operation area 123 marked with a downward triangle. If these operation areas are selected (S37: YES), a predetermined setting change is made (S38). If the operation area 120, 122, or 123 is selected, the waiting time, which is a period during which no input is accepted after the first screen and the second screen are switched, is changed. Specifically, if the operation area 120 is selected, the setting change unit 244 changes the waiting time to a predetermined setting value for beginners. The setting value for beginners is, for example, 2 seconds. In this embodiment, a predetermined waiting time is provided after the screen is switched, but a user who is not familiar with the operation may take time to find the desired operation area, and in the process, an unintended operation area may be determined to have been selected. Therefore, by setting a longer waiting time after the screen is switched in the beginner mode, it is expected that the user will be able to operate with ease. In addition, the setting change unit 244 increases the gaze time when the operation area 122 is selected, and decreases the gaze time when the operation area 123 is selected. The gaze time may be increased or decreased by switching the setting in a predetermined step such as 0.4 seconds, 0.7 seconds, 1.0 seconds, and 1.3 seconds. In addition, the gaze time may be changed so that the gaze time for the operation area arranged on the inner circle on the first screen is longer than the gaze time for the operation area arranged on the outer circle. When the operation area 121 is selected, the activation/inactivation of the voice output is switched when the input character string is output. After S38, the process returns to S21 in FIG. 10.

If it is determined in S37 that no instruction to change the settings has been given, the gaze detection unit 242 determines whether the operation area 116 marked with "OK" has been selected based on the acquired information indicating the position of the gaze point (S39). If the operation area 116 has been selected (S39: YES), the character strings displayed in the text boxes 125 and 117 are output (S40). In this step, the character input unit 243 outputs the character strings displayed in the text boxes 125 and 117, for example, as voice, or outputs them to a text box of other software executed by the control device 2. At this time, the presence or absence of voice output may be switched depending on the on/off setting that can be changed by the operation area 121 marked with "Speak". After S40, the process proceeds to the process of FIG. 10.

Also, if it is determined in S39 that the output of the character string has not been instructed, the gaze detection unit 242 determines whether the operation area 124 marked with "break" has been selected based on the acquired information indicating the position of the gaze point (FIG. 13: S41). If the operation area 124 has been selected (S41: YES), the system temporarily stops accepting operations by gaze until an instruction to cancel the break is given. Here, the gaze detection unit 242 acquires information indicating the position of the user's gaze point on the display unit 32 of the display device 3 from the gaze detection device 4 (S42). Then, the gaze detection unit 242 determines whether the operation area 124 marked with "break" has been selected again based on the acquired information indicating the position of the gaze point (S43). If the operation area 124 has not been selected (S43: NO), the process returns to S42 and repeats. On the other hand, if the operation area 124 has been selected (S43: YES), the process returns to S21 in FIG. 10. Also, if it is determined in S41 that the operation area 124 has not been selected, the gaze detection unit 242 determines whether the operation area 112 marked with "x" has been selected based on the acquired information indicating the position of the gaze point (S44). If the operation area 112 has been selected (S44: YES), the character input unit 243 cancels the input of the character input immediately before (S45). That is, the last character of the character string displayed in the text box 117 is deleted. If no character is displayed in the text box 117, the last character of the character string displayed in the text box 125 may be deleted. Then, after S45, the process returns to S21 in FIG. 10. Also, if it is determined based on the acquired information indicating the position of the gaze point that none of the operation areas shown in FIG. 3 has been gazed on for a predetermined gaze time or more (S44: NO), the process returns to S21 in S10 and repeats the process. If an operation to end the operation is received from the user at any timing in FIGS. 10 to 13, the process may be ended.

<Modifications of the layout of the second screen>
Among the operation areas on the outer arc in the first screen shown in Fig. 3, the operation areas 102, 104, 106, 108, 110, and 112 are located at the same position on the display unit 32 of the display device 3 as the operation areas 131 to 136 in the second screen shown in Fig. 4. Therefore, although a waiting time is provided after switching between the first screen and the second screen, if it is determined that the user's line of sight remains in the same position, there is a risk of erroneous input occurring. Therefore, the arrangement of the operation areas in the second screen after the operation areas 102, 104, 106, 108, 110, and 112 are selected may be different.

Figure 14 is a diagram showing an example of a second screen according to a modified example. In the modified example, the same reference numerals are used for corresponding components, and the description is omitted. In the example of Figure 14, a plurality of operation areas arranged on an arc are displayed near the center of the display unit 32. On the arc, operation areas 131 to 136 are arranged clockwise from the left (the 9 o'clock direction when viewed from the center), displaying "ka", "ki", "ku", "ke", "ko" and "x". In addition, text boxes 117 and 125 are the same as the corresponding components in Figure 4. In this way, the operation areas 131 to 136 of the second screen may be arranged at positions that do not overlap with the operation areas 102, 104, 106, 108, 110 and 112 of the first screen, thereby suppressing erroneous input. In this case, the waiting time after switching between the first screen and the second screen may be shortened or eliminated.

FIG. 15 is a diagram showing an example of a second screen according to another modified example. In the example of FIG. 15, a plurality of operation areas arranged on a circle with a radius of 3r are displayed near the center of the display unit 32. On the arc, operation areas 131 to 136 are arranged clockwise from the left (the 9 o'clock direction when viewed from the center), with "a", "i", "u", "e", "o" and "x" respectively displayed. Each of the operation areas 131 to 136 is also a circle with a radius of r, and is arranged at intervals of 60 degrees from the center. That is, each of the operation areas 131 to 136 is also arranged at a distance of r or more from each other. The text boxes 117 and 125 are the same as the corresponding components in FIG. 4. Even in this way, the operation areas 131 to 136 of the second screen can be arranged at positions that do not overlap with the operation areas 101 to 112 of the first screen, and erroneous input can be suppressed. In this case, the waiting time after switching between the first screen and the second screen may be shortened or eliminated. In the example shown in Fig. 14, when the operation areas 101, 103, 105, 107, 109, and 111 are selected on the first screen, the layout of the second screen is different from that when the operation areas 102, 104, 106, 108, 110, and 112 are selected, so the user may be confused about the operation. In the example shown in Fig. 15, the layout of the second screen is unified, so the user can easily remember the layout of the operation areas. On the other hand, the gaze time of the operation areas 113 to 116 on the first screen is relatively long, so it is unlikely that the user's gaze will be determined to remain in the same position as the operation areas 131 to 134 on the second screen after switching from the second screen to the first screen.

<Modifications of User Interface>
The characters displayed in the operation areas of the first screen and the second screen are not limited to those shown in Fig. 3, Fig. 4, etc. For example, it may be possible to input characters in Roman letters. That is, "a", "k", "s", "t", "n", "h", "m", "y", "r", and "w" are displayed in the operation areas 101 to 110 of the first screen, and "a", "i", "u", "e", and "o" are displayed in the operation areas 131 to 135 of the second screen, and characters may be input by combining the operation areas selected in the same manner as in the above-mentioned embodiment. In addition, it may be applied to languages other than Japanese in which characters can be determined by a combination of a vowel and a consonant.

Figure 16 is a diagram showing an example of a first screen according to another modified example. The size of operation areas 101-111 and the distance between them and the surrounding operation areas can be determined based on the expected distance from the user to the screen and the radius r calculated using the above-mentioned formula (1). The first screen shown in Figure 3 may also be used, but the example in Figure 16 uses an arrangement that users are accustomed to using on a typical smartphone.

Figure 17 is a diagram showing an example of a second screen according to another modified example. On the second screen as well, the operation areas are spaced apart to take microsaccades into consideration. Furthermore, the operation areas 131 to 135 on the second screen are positioned at the vertices of a polygon (a pentagon in Figure 17) or on a circle so as not to overlap with the operation areas on the first screen, preventing erroneous input.

Figure 18 is a diagram showing an example of a second screen according to another modified example. In the example of Figure 17, the operation areas 131-135 of the second screen are arranged at the vertices of a polygon or at positions on a circle centered on a selected operation area from among the operation areas 101-112 of the first screen. These operation areas 131-135 of the second screen are also arranged so as not to overlap with the operation areas of the first screen, preventing erroneous input. Also, compared to the example of Figure 17, the distance of eye movement required for input operations is smaller.

In the example of FIG. 16, the operation area for inputting a predetermined intention and the predictive conversion field are omitted, but for example, operation areas for inputting standard phrases such as "yes," "no," "thank you," "hello," "goodbye," and "pleased to meet you" may be arranged in the positions of operation areas 101 to 111 shown in FIG. 16, making it possible to easily input character strings that are used relatively frequently. In this way, the efficiency of character input can be improved.

<Modifications of the gaze detection device>
The gaze detection device 4 is not limited to a device that detects the position of the user's pupil using infrared rays. For example, it may be a device that identifies the direction of the gaze by detecting the position of an iris using an image captured with visible light. That is, a camera provided in a smartphone or tablet may be used as the gaze detection device 4, and a touch panel may be used as the display device 3. The gaze detection device 4 may also be a user contact type device that uses a predetermined potential difference.

FIG. 19 is a functional block diagram showing an example of a character input device. In this modification, the control device 2 shown in FIG. 2 is, for example, a smartphone, the input/output device 23, which is, for example, a touch panel, also serves as the display unit 32 of the display device 3, and the input/output device 23, which is, for example, a front camera mounted on the touch panel side, also serves as a part of the gaze detection device 4. The processor 24 is connected to the input/output device 23 via a signal line 25 and causes the input/output device 23 to display a predetermined user interface. The processor 24 also analyzes the user's face image obtained from the front camera and detects the position of the gaze on the touch panel based on, for example, the positions of the pupils and irises in the eyes. That is, the camera and the processor 24 function as a gaze detection device.

＜Other＞
The above-mentioned configuration is an example, and the present invention is not limited to the illustrated configuration. For example, the above-mentioned items can be appropriately combined within the scope of the problem and technical idea of the present invention. The processing flow can also be changed in order or performed in parallel within the scope of the result. After S32, S34, S36, S38, and S40 in FIG. 12 and S43 and S45 in FIG. 13, the screen does not change, so the processing may be returned to S22 in FIG. 10. The above-mentioned processing steps may also be realized as a method in which a computer executes the processing by software, a method in which a device executes the processing by hardware such as a PLD (Programmable Logic Device), or a program to be executed by a computer. The program may be provided by recording it on a computer-readable recording medium. A computer-readable recording medium is a recording medium that accumulates information by electrical, magnetic, optical, mechanical, or chemical action and can be read by a computer. Among such recording media, those that can be removed from a computer include, for example, optical disks, magneto-optical disks, flexible disks, magnetic tapes, and memory cards. In addition, a hard disk drive (HDD) is used as a fixed recording medium in a computer.
), SSD (Solid State Drive), ROM (Read Only Memory), etc.

Furthermore, the system of the present invention may be connected to the Internet and used as a device for communicating with people in remote locations, such as overseas, using only the eyes. Also, a plurality of the present systems may be connected to the Internet and used as a device for communicating with a plurality of people using only the eyes. That is, the character input system 1 shown in FIG. 2 or the character input device 2 shown in FIG. 19 is connected to one or more other character input systems 1 or character input devices 2 via the communication I/F 31 and the network. Then, the character input system 1 or the character input device 2 accepts character input from a user using the above-mentioned first screen and second screen, and performs processing to output the character input to the other character input system 1 or character input device 2. The output may be to display a character string, or to read aloud using voice synthesis. In addition, software for performing such processing may be provided, or the system may operate in cooperation with an existing online calling service or web conference system. When cooperating with an existing service or system, a character string is output to a specified text box of the other software, or a voice read aloud via virtual audio input/output is output to an audio input of the other software.

1: Character input system 2: Control device 21: Communication I/F
22: Storage device 23: Input/output device 24: Processor 241: Output unit 242: Line-of-sight detection unit 243: Character input unit 244: Setting change unit 25: Bus 3: Display device 31: Communication I/F
32: Display unit

4: Line-of-sight detection device 41: Communication I/F
42: Infrared transmission/reception unit 43: Line-of-sight detection unit 101 to 116, 120 to 124, 131 to 136: Operation area 117, 125: Text box 118, 119: Predictive conversion field

Claims (14)

a gaze detection unit that switches between a first screen including a plurality of regions related to consonants and a second screen including a plurality of regions related to vowels, which is displayed when it is determined that any one of the plurality of regions related to consonants has been selected on the first screen, detects the gaze of a user facing the display device, acquires information indicating the gaze point from a gaze detection device that outputs the gaze point of the user, and determines that the region has been selected when the gaze point indicated by the acquired information is included in the predetermined region for a predetermined gaze time or longer;
a character input unit that accepts an input of a character according to a combination of one of a plurality of areas related to the consonant selected on the first screen and one of a plurality of areas related to the vowel selected on the second screen based on a determination of the gaze detection unit;
Equipped with
a plurality of areas included in the first screen and a plurality of areas included in the second screen are displayed at a distance equal to or greater than a predetermined distance from other areas included in the same screen ,
Each of the plurality of areas included in the second screen is displayed at a position on the display device that does not overlap with a position at which the selected area is displayed on the first screen.
Character input device. a gaze detection unit that switches between a first screen including a plurality of regions related to consonants and a second screen including a plurality of regions related to vowels, which is displayed when it is determined that any one of the plurality of regions related to consonants has been selected on the first screen, detects the gaze of a user facing the display device, acquires information indicating the gaze point from a gaze detection device that outputs the gaze point of the user, and determines that the region has been selected when the gaze point indicated by the acquired information is included in the predetermined region for a predetermined gaze time or longer;
a character input unit that accepts an input of a character according to a combination of one of a plurality of areas related to the consonant selected on the first screen and one of a plurality of areas related to the vowel selected on the second screen based on a determination of the gaze detection unit;
Equipped with
a plurality of areas included in the first screen and a plurality of areas included in the second screen are displayed at a distance equal to or greater than a predetermined distance from other areas included in the same screen,
The plurality of areas included in the first screen and the plurality of areas included in the second screen are
Each is arranged on an arc,
the first screen further includes a plurality of areas for a user to indicate a predetermined intention, the plurality of areas relating to consonants being arranged on an outer arc of a double circle, and the plurality of areas for indicating the predetermined intention being arranged on an inner arc of the double circle,
The gaze detection unit takes the gaze time required to determine that a plurality of areas for expressing the predetermined intention have been selected longer than the gaze time required to determine that a plurality of areas related to the consonants have been selected on the first screen.
Character input device. The character input device according to claim 1 , wherein the plurality of regions included in the first screen and the plurality of regions included in the second screen are each arranged on an arc. 4. The character input device according to claim 3, wherein the first screen further includes a plurality of areas for a user to indicate a predetermined intention, the plurality of areas relating to consonants being arranged on an outer arc of a double circle, and the plurality of areas for indicating the predetermined intention being arranged on an inner arc of a double circle. The plurality of regions relating to the vowels on the second screen are arranged on the same arc as the outer arc of the double circles on the first screen.
5. The character input device according to claim 2 or 4. 6. The character input device according to claim 4, wherein the gaze detection unit takes the gaze time required to determine that a plurality of areas for expressing the predetermined intention on the first screen have been selected longer than the gaze time required to determine that a plurality of areas related to the consonants have been selected. the first screen further includes a temporary text box for temporarily displaying a character string that has been input, and a decision area for outputting the character strings displayed in the temporary text box to another predetermined text box or outputting the character strings as voice,
When the gaze detection unit determines that the decision area has been selected, the character input unit outputs the character strings displayed in the temporary text box to the predetermined text box or outputs them as voice,
7. The character input device according to claim 2, 4, 5 or 6, wherein, when the gaze detection unit determines that any one of a plurality of areas for making the predetermined expression of intention has been selected, the character input unit performs output according to the selected area for making the predetermined expression of intention prior to outputting the character string displayed in the temporary text box. the first screen further includes a rest button area for stopping a process of determining that the area has been selected by the line-of-sight detection unit,
The character input device according to claim 1 , wherein the gaze detection unit stops accepting operations other than an operation to release the pause when it determines that the rest button area has been selected. 9. The character input device according to claim 1, wherein the gaze detection unit provides a waiting time during which the gaze detection unit does not determine whether the gaze point is included in a predetermined area for a predetermined gaze time or longer after switching the display of the display device between the first screen and the second screen . the first screen includes a change area for changing the gaze time or the waiting time,
The character input device according to claim 9 , wherein the gaze detection unit increases or decreases the gaze time or the waiting time when it is determined that the change area is selected. The predetermined distance is a radius r of the centrally visible range calculated by the following formula, where l is an expected distance from the eyeball of the user to the gaze point on the display device, and θ is an angle between a straight line connecting the eyeball and the gaze point and a straight line connecting the eyeball and an edge of the centrally visible range on the display device: r = l tan θ
The character input device according to claim 1 . The display device;
The gaze detection device;
The character input device according to claim 1 ,
A character input system comprising: a display step of switching between a first screen including a plurality of areas related to consonants and a second screen including a plurality of areas related to vowels, which is displayed when it is determined that any one of the plurality of areas related to consonants is selected on the first screen, and displaying the first screen and the second screen on the display device;
a gaze detection step of detecting a gaze of a user facing the display device, acquiring information indicating the gaze point from a gaze detection device that outputs a gaze point of the user, and determining that a predetermined area has been selected when the gaze point indicated by the acquired information is included in the predetermined area for a predetermined gaze time or longer;
a character input step of accepting an input of a character corresponding to a combination of any one of a plurality of areas related to the consonant selected on the first screen and any one of a plurality of areas related to the vowel selected on the second screen based on a determination made in the line of sight detection step;
The computer executes
a plurality of areas included in the first screen and a plurality of areas included in the second screen are displayed at a distance equal to or greater than a predetermined distance from other areas included in the same screen ,
Each of the plurality of areas included in the second screen is displayed at a position on the display device that does not overlap with a position at which the selected area is displayed on the first screen.
How to enter text. a display step of switching between a first screen including a plurality of areas related to consonants and a second screen including a plurality of areas related to vowels, which is displayed when it is determined that any one of the plurality of areas related to consonants is selected on the first screen, and displaying the first screen and the second screen on the display device;
a gaze detection step of detecting a gaze of a user facing the display device, acquiring information indicating the gaze point from a gaze detection device that outputs a gaze point of the user, and determining that a predetermined area has been selected when the gaze point indicated by the acquired information is included in the predetermined area for a predetermined gaze time or longer;
a character input step of accepting an input of a character according to a combination of any one of a plurality of areas related to the consonant selected on the first screen and any one of a plurality of areas related to the vowel selected on the second screen based on a determination made in the line of sight detection step;
on the computer,
a plurality of areas included in the first screen and a plurality of areas included in the second screen are displayed at a distance equal to or greater than a predetermined distance from other areas included in the same screen ,
Each of the plurality of areas included in the second screen is displayed at a position on the display device that does not overlap with a position at which the selected area is displayed on the first screen.
Text input program.

Images