JP7675526B2 - Text input method and text input device - Google Patents

Description

The present invention relates to a text input method and a text input device.

A technology has been proposed that uses event-related potentials contained in the user's electroencephalogram signal to infer the user's intentions (Patent Document 1). In this technology, a menu screen displays multiple options for a question, and each option is sequentially blinked on and off. The event-related potentials corresponding to each option, which are contained in the electroencephalogram signal after a certain time has passed since each blink, are used to infer the option that the user intended to choose.

Patent No. 4856791

When the technology of Patent Document 1 is used to input a sentence that the user intends, the user can recognize which word in a choice is blinking on a screen that displays all the choice words that make up the sentence, and then react to the recognized word. The user's reaction to each word in a choice appears in the blinking event-related potential included in the electroencephalogram signal a certain time after the user recognizes the blinking word.

The present invention has been made in consideration of the above circumstances, and an object of the present invention is to reduce the time required for inputting words that constitute a sentence intended by the user by utilizing event-related potentials contained in the user's electroencephalogram signal.

In order to solve the above-mentioned problems, a sentence input method according to one aspect of the present invention estimates a sentence that a user intends to input and proposes a plurality of sentence options. The constituent words constituting each of the plurality of sentence options are classified into the order in each sentence option, and the plurality of constituent words classified into the same order are sequentially presented to the user one by one. An event-related potential starting from the presentation of the constituent words, which is included in an electroencephalogram signal measured from the user, is analyzed for each of the sequentially presented constituent words, and the constituent words in the same order that the user intends to input are determined. The sequential presentation of the plurality of constituent words and the determination of the constituent words that the user intends to input are repeated for all orders to determine the sentence that the user intends to input.

According to the present invention, by utilizing the event-related potential contained in the user's electroencephalogram signal, it is possible to reduce the time required for inputting words that constitute the user's intended sentence.

FIG. 1 is a block diagram of a text input device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram for comparing the relationship between the presentation pattern of constituent words and the user's reaction between the text input device of FIG. 1 and the conventional device. FIG. 3A is an explanatory diagram of an example of a sentence candidate proposed by the selected sentence proposing unit. FIG. 3B is an explanatory diagram of an example of a group of word options generated by the word presentation control unit. FIG. 4 is an explanatory diagram of an example of a presentation pattern of constituent words by the word presentation control unit. FIG. 5 is a flowchart of an example of a procedure for determining the presentation order of constituent words. FIG. 6A is a graph showing an example of electroencephalographic potential changes in response to words presented by the presentation unit. FIG. 6B is a graph showing an example of electroencephalographic potential changes in response to words presented by the presentation unit. FIG. 7A is a graph of an electroencephalogram analyzed by the desired word recognition signal analysis unit. FIG. 7B is a graph of an electroencephalogram analyzed by the desired word recognition signal analysis unit. FIG. 8 is an explanatory diagram showing the analysis result of the desired word recognition signal analysis unit in FIG. FIG. 9A is a graph showing the relationship between the presentation interval of constituent words and the accuracy rate of judgment. FIG. 9B is a graph showing an example of the relationship between the number of sequential presentations of constituent words, the number of repetitions, and the accuracy rate of judgment. FIG. 10A is a graph showing another example of the relationship between the presentation interval of constituent words and the accuracy rate of the judgment. FIG. 10B is a graph showing an example of the relationship between the number of sequential presentations of constituent words, the number of repetitions, and the accuracy rate of judgment. FIG. 11 is a flowchart of an example of a procedure for the selection process. FIG. 12 is a flowchart of an example of a learning process procedure. FIG. 13A is an explanatory diagram showing a classification boundary between targets and non-targets using a retrained classifier. FIG. 13B is an explanatory diagram showing the analysis result of the desired word recognition signal analysis unit in which both a determination error due to misclassification by the classifier and a determination error due to misrecognition by the user exist. FIG. 14A is an explanatory diagram showing a boundary line for distinguishing between targets and non-targets by a classifier when event-related potential data erroneously recognized by a user is excluded from data used for re-learning of the classifier. FIG. 14B is an explanatory diagram showing the analysis result of the desired word recognition signal analysis unit in which only a determination error due to misclassification by the classifier exists. FIG. 15 is an explanatory diagram showing an example of a schedule for executing optimization of the control parameters of the word presentation control unit and re-learning of the classifier of the signal determination unit.

Embodiments of the present invention will be described below with reference to the drawings. In the description of the drawings, identical parts are given the same reference numerals and the description will be omitted.

(Embodiment)
A configuration of a text input device 100 will be described with reference to Fig. 1. As shown in Fig. 1, the text input device 100 according to the present embodiment executes a text input method according to an embodiment of the present invention.

The text input device 100 presents a selection of constituent words of a sentence that the user intends to input, and uses the event-related potential contained in the user's electroencephalogram signal to determine which of the presented options the user intends to input.

For example, when the technology of Patent Document 1 described in the Prior Art section is used to present candidate constituent words, the user visually watches the constituent words of each option blinking in sequence on a screen on which all options for the constituent words of a sentence are presented all at once. The user's reaction to the constituent words of each option that he or she has viewed appears as an event-related potential in the EEG signal after the "eye movement time" and "presentation time (blinking time)" have elapsed from the start of the display of all options on the screen, as shown by the bar graph of all at once presentation in Figure 2. The "eye movement time" is the time it takes the user to move their viewpoint on the screen to find the blinking part. The "presentation time (blinking time)" is the time it takes for the user's reaction to the constituent words of the blinking option to appear in the EEG signal.

In the sentence input device 100 of this embodiment, options for constituent words of a sentence are presented to the user one by one in sequence. By presenting the options in sequence, as shown by the bar graph of sequential presentation in FIG. 2, the user recognizes the constituent words presented one by one without having to move the viewpoint within the screen of the presentation unit 120 to search for them. Since there is no need to search for the presented constituent words within the screen of the presentation unit 120, in the sentence input device 100 of this embodiment, there is no need to spend "eye movement time" until the user's reaction to the constituent words of the presented options appears as an event-related potential in the electroencephalogram signal.

The text input device 100 of this embodiment has a controller 110, a presentation unit 120, and a biosignal detection unit 130.

The text input device 100 can be used as a man-machine interface for a device to which a user needs to input text. Examples of devices to which a user needs to input text include personal computers, tablet terminals, and smartphones. When the text input device 100 is mounted in a vehicle, examples of devices to which a user needs to input text include an in-vehicle multimedia player. These devices may be devices that incorporate the text input device 100, or may be external devices connected to the text input device 100.

In the following embodiment and its modified examples, a description will be given of a case where the text input device 100 is mounted on a vehicle, but it goes without saying that the text input device 100 does not have to be mounted on a vehicle. The user of the text input device 100 mounted on a vehicle is an occupant of the vehicle. The occupant of the vehicle may be either the driver or a passenger.

The presentation unit 120 sequentially displays options for words that make up a sentence one by one. The contents of the words displayed by the presentation unit 120 are determined by the controller 110 executing the process described below. The biosignal detection unit 130 detects the user's brain waves measured by the brain wave sensor unit 200 worn by the user as an brain wave signal.

The user generates brain waves from the head. Brain waves are generated from specific brain activity areas of the user's head. The brain activity areas change depending on the user's mental state, emotions, vision, smell, taste, motor control, and other behaviors. For example, when a user responds to a change in visual information, the occipital lobe of the brain, which is related to visual information, becomes active. When the visual information is words or sentences containing letters, the parietal lobe of the brain, which is related to language comprehension, becomes active.

In this embodiment, the EEG sensor unit 200 can measure EEG in the parietal lobe and occipital lobe of the user's head. For example, when using the international standard 10-20 electrode placement method (10-20 method), the EEG sensor unit 200 can have at least an electrode placed at Cz (central midline). When using the 10-10 electrode placement method (10% electrode placement method), which is suitable for recording high-resolution EEG, the EEG sensor unit 200 can have at least an electrode placed at Cpz.

In this embodiment, the text input device 100 is mounted in a vehicle, so the brainwave sensor unit 200 can be placed, for example, in the headrest of the seat where the user is seated.

The biosignal detection unit 130 detects the user's EEG signal measured by the EEG sensor unit 200. The user's EEG signal contains an event-related potential that occurs due to the user's reaction to an event. The user's reaction appears in the EEG signal a certain time after the occurrence of the event. The biosignal detection unit 130 can extract features of the event-related potential.

The feature of the event-related potential can be, for example, a component called the P300 component (also called the P3 component) of the event-related potential. The P300 component of the event-related potential is the positive maximum value of the event-related potential at a certain time after the occurrence of an event. The certain time can be, for example, around 300 ms. The certain time may increase or decrease depending on the content of the event (the active area of the user's brain that reacts to the event).

The controller 110 can be realized using a microcomputer equipped with a CPU (Central Processing Unit), memory, and input/output units. The controller 110 installs and executes a computer program in the microcomputer for causing the microcomputer to function as the controller 110. This allows the microcomputer to function as the multiple information processing units (101 to 106) equipped in the controller 110.

Here, an example of implementing an information notification device using software is shown, but it is of course possible to configure the controller 110 by providing dedicated hardware for executing each information process. Dedicated hardware includes devices such as application specific integrated circuits (ASICs) and conventional circuit components arranged to perform the functions described in the embodiments.

Although the controller 110 has been described as an independent component, the controller 110 may of course be configured together with other devices as a single device. Alternatively, the multiple information processing units (101-106) may be divided and configured using two or more different devices. Furthermore, all or part of the multiple information processing units (101-106) may be configured using an ECU (Electronic Control Unit) mounted on the vehicle.

The controller 110 includes a number of information processing units (101 to 106), including a selected sentence suggestion unit 101, a word presentation control unit 102, a desired word recognition signal analysis unit 103, a desired word determination unit 104, a selected word output unit 105, and a signal determination unit 106.

The selected sentence suggestion unit 101 can estimate the sentence that the user intends to input and propose (generate) constituent words of multiple sentence options. For example, as shown in FIG. 3A, the selected sentence suggestion unit 101 generates sentence candidates 1 to 3 as multiple sentence options. Sentence candidate 1 is a sentence in which three constituent words A to C are arranged in the order A, B, C. Sentence candidate 2 is a sentence in which three constituent words D to F are arranged in the order D, E, F. Sentence candidate 3 is a sentence in which three constituent words G to I are arranged in the order G, H, I.

The selected sentence suggestion unit 101 (sentence suggestion unit) may estimate sentence options to be suggested from the environment surrounding the user, the communication status of the device connected to the sentence input device 100, and the like. The environment surrounding the user can be obtained, for example, from traffic congestion information of a car navigation system. For example, if the traffic congestion information indicates that a traffic jam has occurred at the current location of the vehicle, the environment surrounding the user can be determined as "traffic congestion." The communication status can be, for example, the operating status of a device that requests an operation by inputting a sentence, or the status of sending and receiving e-mails and the like using the sentence input to the sentence input device 100 by a device that incorporates the sentence input device 100 or a connected device.

For example, when an email is being sent or received from a vehicle traveling through a traffic jam, the selected sentence suggestion unit 101 can estimate, as a sentence option, a sentence that the user is expected to intend to input as a sentence in an email notifying the user of the traffic jam. The selected sentence suggestion unit 101 can use, for example, AI (artificial intelligence) to estimate the sentence option.

The word presentation control unit 102 can classify the constituent words that make up each of the multiple sentence options proposed by the selected sentence suggestion unit 101 according to the order in which they appear in each sentence option.

For example, as shown in FIG. 3B, the word presentation control unit 102 classifies the constituent words A-C, D-F, and G-I of each of the sentence candidates 1-3 into three word option groups 1-3. The constituent words A, D, and G in the first order are classified into word option group 1. The constituent words B, E, and H in the second order are classified into word option group 2. The constituent words C, F, and I in the third order are classified into word option group 3.

The word presentation control unit 102 can present the multiple constituent words, each classified in the same order, one by one to the user in the presentation unit 120.

For example, the word presentation control unit 102 sequentially presents the first constituent words A, D, and G of each of the sentence candidates 1 to 3 classified into the word option group 1 to the user. The word presentation control unit 102 can repeat the constituent words A, D, and G multiple times and sequentially present them to the user.

The word presentation control unit 102 presents the constituent words A, D, and G of the word option group 1 to the user in a random order in each word presentation. In the example shown in FIG. 4, in the first word presentation, constituent words A, D, and G are presented to the user in that order, and in the second word presentation, constituent word D is presented to the user first. The presentation interval of constituent words A, D, and G can be, for example, 200 msec.

The word presentation control unit 102 can also sequentially present the second constituent words B, E, and H of each of the sentence candidates 1 to 3 classified into word option group 2 to the user one by one in the presentation unit 120 in the same manner as the constituent words A, D, and G of word option group 1. The word presentation control unit 102 can also sequentially present the third constituent words C, F, and I of each of the sentence candidates 1 to 3 classified into word option group 3 to the user one by one in the presentation unit 120 in the same manner as the constituent words A, D, and G of word option group 1.

When the constituent words of each of the word option groups 1 to 3 are presented one by one in sequence, the user's reaction to the presentation of the same constituent word will be different when two constituent words with high similarity are presented in succession than when two constituent words with low similarity are presented in succession.

For example, when a target word that the user intends to input is presented next to a non-target word that has low similarity to the target word, the preceding and following words are easily distinguished, and the user's reaction to the target word is pronounced. However, when a target word is presented next to a non-target word that has high similarity to the target word, the preceding and following words are not easily distinguished, and the user's reaction to the target word is dull.

If the user responds slowly to the target constituent words, it becomes difficult to determine the constituent words that the user intended to input from the event-related potentials of the user's EEG signal.

The word presentation control unit 102 presents the constituent words of each of the word option groups 1 to 3 to the user in a random order, in principle. However, if there is a combination of constituent words in each of the word option groups 1 to 3 that is evaluated as having a high similarity based on the distance value between two words calculated by the distance function, the word presentation control unit 102 makes the presentation interval between the constituent words in that combination longer than usual.

To determine the order of sequential presentation of constituent words taking similarity into consideration, the word presentation control unit 102 executes the process shown in the flowchart of FIG. 5. For the constituent words classified into each of the word option groups 1 to 3, the word presentation control unit 102 calculates the edit distance of each word to other words (step S101), and compares the calculated edit distance with a threshold value that is used as a criterion for judging similarity (step S103).

The word presentation control unit 102 determines the similarity of each word to other words based on the result of comparing the edit distance with a threshold (step S105), and records the similarity determination result in the memory of the controller 110 (step S107). Based on the record in the memory, the word presentation control unit 102 generates a presentation order for constituent words in which highly similar words are not consecutive (step S109), and ends the series of processes.

The presentation order of the constituent words generated by the word presentation control unit 102 in step S109 can be, for example, an order in which two highly similar constituent words are sandwiched between other constituent words that have low similarity to the two highly similar constituent words. Note that, instead of calculating the edit distance in step S101, the distance value between two words may be calculated using a distance function such as the Hamming distance.

In the process performed by the word presentation control unit 102, the parameters of the number of sequential presentations of constituent words in one word presentation, the interval between sequential presentations of constituent words, and the number of repetitions of word presentation are each set to appropriate values.

When highly similar combinations of constituent words are included in each of the word option groups 1 to 3, increasing the number of constituent words presented and the presentation interval increases the probability that the presentation interval between highly similar constituent words will be longer.When highly similar combinations of constituent words are included in each of the word option groups 1 to 3, increasing the number of repetitions of word presentation increases the number of times the target constituent word is presented, and the user's response to the target constituent word will gradually become stronger.By increasing the number of constituent words presented, the presentation interval, and the number of repetitions of word presentation, the accuracy of determining the target constituent word can be improved.

However, increasing the number of constituent words presented, the presentation interval, or the number of times the words are presented will increase the time required to present all of the constituent words classified into one of the word option groups 1 to 3, and will delay the arrival of the point at which the constituent words of the target can be identified.

For this reason, the parameters of the number of constituent words presented, the presentation interval, and the number of repetitions of word presentation should be set by taking into consideration the balance between the accuracy of determining the constituent words of the target and the time at which the constituent words of the target can be determined.

The desired word recognition signal analysis unit 103 (signal analysis unit) extracts the portion of the user's electroencephalogram signal detected by the biosignal detection unit 130 from the 0 point on the horizontal axis when the constituent word is presented to the presentation unit 120 to the time when the P300 component of the event-related potential appears. When the target constituent word is presented to the user, the P300 component appears approximately 500 msec after the 0 point on the horizontal axis when the constituent word is presented to the presentation unit 120.

In the example of the user's EEG signal when non-target constituent words are presented to the user shown in FIG. 6A, only frequency fluctuations corresponding to the period (100-200 msec) at which the constituent words are presented one by one sequentially by the presentation unit 120 are observed, and no event-related potentials are observed. In the example of the user's EEG signal when target constituent words are presented to the user shown in FIG. 6B, in addition to frequency fluctuations corresponding to the presentation period of the constituent words, event-related potentials due to the user's reaction to the presentation of the target constituent words are observed.

The desired word recognition signal analysis unit 103 inputs the extracted EEG signal portion to the signal determination unit 106. The signal determination unit 106 (signal analysis unit) has, for example, a classifier that classifies the extracted portion of the EEG signal input from the desired word recognition signal analysis unit 103 according to whether or not it is a constituent word of the target that the user intends to input.

The classifier can be configured, for example, using a neural network that receives as input the extracted portion of the EEG signal input from the desired word recognition signal analysis unit 103 and outputs a judgment value indicating whether or not the word is a constituent word of the target.

The signal determination unit 106 extracts, for example, the portion of the EEG signal shown in Figures 7A and 7B from the presentation of the constituent words in the presentation unit 120 to the time when the event-related potential appears, and inputs the feature value of the EEG signal of the extracted portion. The feature value input to the classifier can be, for example, the slope of the waveform portion leading to the maximum value of the P300 component of the event-related potential or the waveform portion that has passed the maximum value, or the maximum value (peak value), etc.

Based on the input feature values, the classifier can output a judgment value indicating whether or not the constituent words presented to the presentation unit 120 are constituent words of the target. FIG. 7A shows a case where non-target constituent words are presented to the user, and FIG. 7B shows a case where target constituent words are presented to the user.

For example, when a non-target word is presented to a user, the user's reaction to the non-target word appears in the portion extracted from the EEG signal. A classifier to which the feature value of the EEG signal of the extracted portion containing this reaction is input outputs a judgment value of "0" indicating that it is a non-target.

When the constituent words of the target are presented to the user, the user's reaction to the constituent words of the target (the P300 component of the event-related potential) appears in the portion extracted from the EEG signal. A classifier to which the feature value of the EEG signal of the extracted portion containing this reaction is input outputs a judgment value of "1" indicating that it is a target.

The feature values input to the classifier and the judgment values output by the classifier are associated with each other and stored in the learning database (DB) of the controller 110. The stored data in the learning DB can be used when retraining the classifier.

The desired word recognition signal analysis unit 103 outputs the judgment values output by the classifier of the signal judgment unit 106 for all classified constituent words for each of the word option groups 1 to 3 as the analysis results of the P300 component of the event-related potential contained in the user's electroencephalogram signal corresponding to each constituent word.

The desired word determination unit 104 (word determination unit, sentence determination unit) compares the analysis results of the P300 component of the event-related potential corresponding to each constituent word output by the desired word recognition signal analysis unit 103 for all constituent words classified into each of the word option groups 1 to 3. The desired word determination unit 104 compares the analysis results for each of the word option groups 1 to 3. The desired word determination unit 104 executes a process of determining the constituent words of the target (desired words) according to the analysis results of the desired word recognition signal analysis unit 103 shown in FIG. 8.

In Figure 8, the judgment values (judgment results) of the P300 component of the event-related potential corresponding to all constituent words (option words) of eight constituent words A to H classified into a certain word option group are arranged by the number of times the words were presented. In the example shown in Figure 8, constituent word A is the target constituent word, and constituent words B to H are non-target constituent words. The parts of the judgment values in Figure 8 surrounded by a thick line frame are where a target constituent word has been classified as a non-target, or a non-target constituent word has been classified as a target, due to misclassification by the classifier.

The rightmost column of the table in FIG. 8 shows the average value of the judgment value obtained after five repetitions (presentations). The desired word judgment unit 104 judges the constituent word with the highest average judgment value to be the constituent word of the target. In the example shown in FIG. 8, the desired word judgment unit 104 judges the constituent word A with the highest average judgment value to be the constituent word of the target.

The selected word output unit 105 determines that a sentence in which the constituent words determined by the desired word determination unit 104 as targets for each of the word option groups 1 to 3 are connected in order is the sentence that the user intends to input. The selected word output unit 105 presents the determined sentence to the user at the presentation unit 120. The selected word output unit 105 can also output the text data of the determined sentence as a sentence input by the user to a device where the user requires input of a sentence.

In this embodiment, the options for the constituent words of a sentence are sorted by order, and the constituent words of each ordered option (word option group) are presented one by one to the presentation unit 120. The user recognizes the constituent words presented one by one without moving the viewpoint on the screen of the presentation unit 120 to search for them. Therefore, the user's EEG signal shows a reaction according to whether the presented constituent word is a target or a non-target, without spending the "time to move the eyes" shown in the bar graph of the collective presentation in FIG. 2. Therefore, by utilizing the P300 component of the event-related potential contained in the user's EEG signal, it is possible to shorten the time required for inputting the words that constitute the intended sentence.

In addition, in this embodiment, when each of the word option groups 1 to 3 contains a combination of constituent words that are evaluated as being highly similar based on the distance value between the two words calculated by the distance function, the presentation interval between the constituent words in that combination is made longer than usual. This allows the user to react more prominently to the constituent words of the target by making the user's reaction to the next target's constituent words appear after the user's reaction to the highly similar non-target constituent words has subsided.

Furthermore, in this embodiment, the constituent words are presented sequentially in the order of two highly similar constituent words sandwiched between other constituent words with low similarity. This allows the user to easily distinguish between the preceding and following constituent words, and allows the user to react more prominently to the target constituent word.

In addition, in this embodiment, the word presentation control unit 102 repeats each of the constituent words A, D, and G classified into the word option group in the same order multiple times to sequentially present them to the user in random order. Therefore, the constituent words of the target are intermittently and repeatedly presented to the user, and the user's reaction to the constituent words of the target increases as the number of times they are presented increases.

(First Modification)
The accuracy rate when determining constituent words of a sentence intended by a user using the P300 component of the event-related potential varies depending on the presentation time for which the constituent words are presented to the user by the presentation unit 120. The presentation time refers to the interval at which the constituent words are presented sequentially in the word presentation performed by the word presentation control unit 102 for each of the word option groups 1 to 3.

When the presentation interval (presentation time) of constituent words becomes longer, the accuracy rate of the judgment by the desired word determination unit 104 increases after a certain length of time, as shown in FIG. 9A, for example. The longer the presentation interval (presentation time) of constituent words, the higher the probability that the presentation interval between highly similar constituent words will be long, and also the easier it is for the user to recognize whether or not a constituent word is the one they intended while it is being presented.

The accuracy rate of the judgment by the desired word judgment unit 104 decreases when the number of sequentially presented constituent words in one word presentation performed by the word presentation control unit 102 for each of the word option groups 1 to 3 is small, and increases when the number of presentations is large. The accuracy rate of the judgment by the desired word judgment unit 104 decreases when the number of repetitions of word presentation performed by the word presentation control unit 102 for each of the word option groups 1 to 3 is small, and increases when the number of repetitions is large. The graph in Figure 9B shows the relationship between the accuracy rate of the judgment by the desired word judgment unit 104, the number of constituent words presented by the word presentation control unit 102 (number of presented words), and the number of repetitions of word presentation (number of word presentations).

Users have different reaction sensitivities to the presentation of constituent words. For this reason, the relationship between the accuracy rate of the judgment by the desired word judgment unit 104 and the presentation interval (presentation time) of constituent words by the word presentation control unit 102 shown in FIG. 9A may differ for each user. For the same reason, the relationship between the accuracy rate of the judgment by the desired word judgment unit 104, the number of constituent words presented by the word presentation control unit 102 (number of presented words), and the number of times the word presentation is repeated (number of times the word is presented) shown in FIG. 9B may also differ for each user.

As shown in FIG. 10A, for a user other than the user shown in FIG. 9A, the accuracy rate of the judgment by the desired word judgment unit 104 increases when the interval (presentation time) between constituent words is longer than that of the user shown in FIG. 9A. Also, as shown in FIG. 10B, for this user, the accuracy rate of the judgment by the desired word judgment unit 104 is lower than that of the user shown in FIG. 9A for the same number of constituent words presented (number of presented words) and the same number of repetitions of word presentation (number of word presentations) as the user shown in FIG. 9A.

Therefore, when the word presentation control unit 102 sequentially presents the constituent words, the number of constituent words presented in one word presentation, the interval between sequential presentations of the constituent words, and the number of repetitions of word presentation may be set as control parameters, and optimized using the classification results by the classifier of the signal determination unit 106. To optimize each parameter, the word presentation control unit 102 selects an optimal combination of the number of constituent words presented (number of presented words) and the number of repetitions of word presentation (number of presentations).

The optimal combination is the combination of the number of constituent words presented (number of presented words) and the number of times word presentation is repeated (number of presentations) that results in the correct answer rate for the constituent words that the user intends to input exceeding a predetermined threshold and also minimizes the word selection time (word decision time). The optimal combination is premised on a constant presentation interval (presentation time). The word selection time is the product of three parameters: the number of repetitions of word presentation, the number of constituent words presented in one word presentation, and the interval between successive presentations of the constituent words (number of word presentations x number of word presentations x presentation time = word selection time).

If there are multiple optimal combinations of the number of constituent words presented (number of presented words) and the number of times the words are presented repeatedly (number of presentations), the optimal combination is, for example, the combination with the fewest number of times the words are presented repeatedly (number of presentations).

To select the optimal combination of the number of constituent words presented (number of presented words) and the number of times word presentation is repeated (number of presentations), the word presentation control unit 102 executes the selection process shown in the flowchart of FIG. 11. While sequentially changing the value of each parameter, the word presentation control unit 102 causes the controller 110 to determine the constituent words of the sentence intended by the user using the P300 component of the event-related potential. The word presentation control unit 102 acquires the numerical values of the number of constituent words presented (number of presented words) at the time of judgment, the number of times word presentation is repeated (number of presentations), and the accuracy rate of the determined constituent words. The word presentation control unit 102 then stores data indicating each acquired numerical value in the optimization database (DB) of the controller 110 (step S201).

The word presentation control unit 102 analyzes the data stored in the optimization DB and selects the optimal combination of the number of constituent words presented (number of presented words) and the number of repetitions of word presentation (number of presentations) that will maximize the correct answer rate (step S203). The word presentation control unit 102 also analyzes the data stored in the optimization DB and selects the optimal combination of the number of constituent words presented (number of presented words) and the number of repetitions of word presentation (number of presentations) that will minimize the word selection time (step S205). The word presentation control unit 102 also analyzes the data stored in the optimization DB and selects the optimal combination of the number of constituent words presented (number of presented words) and the number of repetitions of word presentation (number of presentations) that will minimize the number of repetitions of word presentation (number of presentations) (step S207). This ends the selection process.

The optimal interval (shortest time) for presenting the constituent words will be determined separately.

If there are multiple optimal combinations selected by the selection process of FIG. 11, the word presentation control unit 102 selects, as the optimal combination, for example, the combination with the fewest number of repetitions of word presentation (number of presentations), as described above.

By performing the above selection process for each user, the parameters of the number of constituent words presented in one word presentation, the interval between successive presentations of the constituent words, and the number of repetitions of word presentation when the constituent words are presented successively in the presentation unit 120 can be optimized for each user. By optimizing the parameters for each user, the accuracy rate of the judgment by the desired word judgment unit 104 can be increased for each user. In addition, by optimizing each parameter, the time required to judge the constituent words of a sentence that the user intends to input can be further shortened.

(Second Modification)
The classifier in the signal determination unit 106 can improve the accuracy of the determination value indicating whether or not a word is a constituent word of the target by, for example, adjusting the bias and gain in the neural network through re-learning. By improving the accuracy of the determination value of the classifier, it is possible to further reduce the time required to determine the constituent words of the sentence that the user intends to input.

To retrain the classifier, the signal determination unit 106 executes the learning process shown in the flowchart of FIG. 12. Each time the classifier classifies the input feature values and outputs a judgment value, the signal determination unit 106 accumulates the combination of the feature value and the judgment value as event-related potential data in the learning DB of the controller 110 (step S301).

The signal determination unit 106 selects usable data of target data with a judgment value of "1" from the user's past event-related potential data stored in the learning DB (step S303). The signal determination unit 106 also selects usable data of non-target data with a judgment value of "0" from the data stored in the learning DB (step S305).

The signal determination unit 106 performs machine learning of the classifier using the utilization data of the selected target data and non-target data (step S307), and ends the learning process. The machine learning of the classifier can be, for example, a content that optimizes (adjusts) the bias and gain in the neural network of the classifier to suit the user by re-learning using the backpropagation method.

When selecting the target data and non-target data to be used, the signal determination unit 106 excludes event-related potential data that has been misrecognized by the user from the data to be used for re-learning the classifier. If event-related potential data that has been misrecognized by the user is used for re-learning the classifier, as shown in FIG. 13A, the classification boundary line of the data classified (discriminated) by the classifier may be biased toward either the target side or the non-target side.

Figure 13B shows the analysis results of the desired word recognition signal analysis unit 103, which includes a mixture of judgment errors due to misclassification by the classifier and judgment errors due to user misrecognition. The bold judgment value parts in Figure 13B are where the constituent words of the target are classified as non-targets, or the constituent words of the non-target are classified as targets, due to misclassification by the classifier. The parts of the judgment value in Figure 13B surrounded by a bold frame are where the constituent words of the target are classified as non-targets, or the constituent words of the non-target are classified as targets, due to user misrecognition.

In the learning process performed by the signal determination unit 106, if only data misclassified by the classifier is excluded as the target data and non-target data to be used, and data misrecognized by the user is used, the bias and gain of the classifier will be inappropriately adjusted. As a result, the classifier after re-learning will perform classification with a biased separation boundary between target data and non-target data, as shown in FIG. 13A.

Therefore, in the example of Figure 13A, the four constituent words A, B, E, and H have the highest average classifier judgment values obtained after five iterations, and the constituent words of the target cannot be determined.

For this reason, when the signal determination unit 106 selects the target data and non-target data to be utilized during the learning process, it excludes not only data that has been misclassified by the classifier, but also data that has been misrecognized by the user.

This allows the bias and gain of the classifier to be appropriately adjusted after re-learning, reliably improving the accuracy of the judgment value through re-learning the classifier, and reliably shortening the time it takes to determine the constituent words of a sentence that the user intends to input.

The utilization data of the target data and non-target data can be selected, for example, by comparing the peak area of the portion marked with diagonal lines in Figures 7A and 7B from the presentation of the constituent words in the presentation unit 120 to the time when the event-related potential appears with a threshold value.

For example, in the case of target data, when the peak area is equal to or greater than the threshold value corresponding to the minimum area, it can be selected as target data. In the case of non-target data, when the peak area is equal to or less than the threshold value corresponding to the maximum area, it can be selected as non-target data.

By determining the data to be used for re-learning the classifier by comparing the peak area with a threshold value, it is possible to accurately select the data to be used for re-learning for each user, even if the peak shapes of the event-related potentials vary from user to user. In addition, by determining the data to be used for re-learning the classifier by comparing the peak area with a threshold value, it is possible to prevent data that has resulted in a judgment error due to a user's misrecognition from being used in re-learning the classifier.

When both the optimization of the control parameters of the word presentation control unit 102 in the first modified example and the re-learning of the classifier in the second modified example are performed, the execution schedule for each can be, for example, as shown in FIG. 15. First, when the sentence input device 100 is first purchased (when operation begins), a general-purpose classifier with bias and gain set to initial values is used as the classifier for the signal determination unit 106.

While increasing or decreasing the control parameters of the word presentation control unit 102, the user is fixed and the controller 110 is made to determine the constituent words of the sentence intended by the user using the P300 component of the event-related potential. This allows data for optimizing the control parameters according to the user to be stored in the optimization DB of the controller 110.

Once sufficient data has been accumulated in the optimization DB, the data is analyzed, and the optimal combination of the number of constituent words (number of presented words) and the number of repetitions of word presentation (number of presentations) that minimizes the number of repetitions of word presentation (number of presentations) is selected, and the control parameters are optimized. At this point, the classifier remains a general-purpose classifier.

After optimizing the control parameters of the word presentation control unit 102, event-related potential data that combines the feature values input to the classifier and the judgment values output by the classifier is stored in the learning DB of the controller 110 as data for re-learning the classifier.

When sufficient data has been accumulated in the learning DB, data to be used for re-learning is selected from the accumulated data, and the selected data is used to perform machine learning of the classifier, updating the classifier to an optimized classifier. At this point, the control parameters of the word presentation control unit 102 are optimized parameters, and the classifier becomes an optimized classifier. Thereafter, every time sufficient new data is accumulated in the learning DB, or periodically, machine learning of the classifier is performed to update the classifier.

In this way, by first optimizing the control parameters of the word presentation control unit 102 and then re-learning the classifier, the data in the learning DB used for re-learning will be data that will further improve the classification accuracy of the classifier through re-learning. Therefore, it is possible to further reduce the time required to input the constituent words of the sentence that the user intends to input.

In this embodiment, the text input device 100 is described as being mounted on a vehicle, but the text input device 100 can also be used without being mounted on a vehicle. In addition, in this embodiment, the brain wave sensor unit 200 is used to detect brain activity in the biosignal detection unit 130, but instead, various known means for detecting electromagnetic waves generated due to brain nerve activity can be applied, such as magnetic field detection using a non-contact biomagnetic field sensor.

The above-mentioned embodiment and modified examples are merely examples of the present invention. Therefore, the present invention is not limited to the above-mentioned embodiment and modified examples, and it goes without saying that various modifications can be made to the design, etc., even in forms other than the above-mentioned embodiment, as long as they do not deviate from the technical concept of the present invention.

100 Sentence input device (sentence suggestion unit)
101: Selected sentence suggestion unit 102: Word presentation control unit 103: Desired word recognition signal analysis unit 104: Desired word determination unit (word determination unit, sentence determination unit)
106 Signal judgment unit (classifier)
120 Presentation unit 200 Brain wave sensor unit

Claims (8)

The system predicts the sentence the user intends to input and suggests multiple options for the sentence.
classifying constituent words constituting each of the plurality of sentence options into an order in which each of the sentence options is arranged;
presenting the plurality of constituent words, each classified into the same sequence, one by one to the user;
analyzing an event-related potential, which is included in an electroencephalogram signal measured from the user and is generated based on the presentation of the constituent words, for each of the constituent words presented one by one in sequence to determine the constituent words that the user intends to input in the same order;
repeating the sequential presentation of the plurality of constituent words and the determination of the constituent words that the user intends to input for all of the permutations to determine a sentence that the user intends to input ;
the sequential presentation of the plurality of constituent words in the same order is repeated, and the event-related potential is analyzed for each of the constituent words at each repetition; at least one of the parameters of the number of constituent words presented and the presentation interval in one sequential presentation and the number of repetitions of the sequential presentation is optimized based on the result of the judgment of the sentence that the user intends to input; and each of the parameters of the number of presentations and the number of repetitions is optimized to a combination of the number of presentations and the number of repetitions such that, when the presentation interval is constant, the accuracy rate in the judgment of the constituent words that the user intends to input exceeds a predetermined threshold value and the word judgment time from the start of the sequential presentation of the constituent words in the same order to the judgment of the constituent words that the user intends to input is the shortest .
How to enter text. The system predicts the sentence the user intends to input and suggests multiple options for the sentence.
classifying constituent words constituting each of the plurality of sentence options into an order in which each of the sentence options is arranged;
presenting the plurality of constituent words, each classified into the same sequence, one by one to the user;
analyzing an event-related potential, which is included in an electroencephalogram signal measured from the user and is generated based on the presentation of the constituent words, for each of the constituent words presented one by one in sequence to determine the constituent words that the user intends to input in the same order;
repeating the sequential presentation of the plurality of constituent words and the determination of the constituent words that the user intends to input for all of the permutations to determine a sentence that the user intends to input;
A sentence input method in which, when a combination of constituent words that is evaluated to have a high similarity based on a distance value between the two words calculated by a distance function is present among the plurality of constituent words classified in the same order, the two constituent words included in the combination are presented at an interval longer than the sequential presentation interval of the plurality of constituent words. The sentence input method according to claim 2, wherein the constituent words not included in the combination are presented in an order sandwiched between two of the constituent words included in the combination. 4. The sentence input method according to claim 2 or 3, wherein the sequential presentation of the plurality of constituent words is repeated in the same order, the event-related potential is analyzed for each of the constituent words at each repetition, and at least one of the parameters of the number of presentations and the presentation interval of the constituent words in one session of the sequential presentation and the number of repetitions of the sequential presentation is optimized based on a result of a judgment of a sentence that the user intends to input, and each of the parameters of the number of presentations and the number of repetitions is optimized to a combination of the number of presentations and the number of repetitions such that, when the presentation interval is constant, a correct answer rate in the judgment of the constituent words that the user intends to input exceeds a predetermined threshold value and the word judgment time from the start of the sequential presentation of the constituent words in the same order to the judgment of the constituent words that the user intends to input is the shortest. The text input method according to claim 4, wherein, when there are multiple combinations, the parameters of the number of presentations and the number of repetitions are optimized for the combination with the smallest number of repetitions. The text input method according to any one of claims 1 to 5, wherein in the analysis of the event-related potential for each of the constituent words, a classifier that classifies the event-related potential according to whether or not the constituent word is one that the user intends to input is optimized for the user by machine learning based on past analysis results of the event-related potential for the user. The text input method according to claim 6, wherein the past analysis results of the event-related potential are separated based on the peak area of the waveform portion of the event-related potential in the electroencephalogram signal, and a part of the separated past analysis results of the event-related potential are used for the machine learning. Inferring a plurality of sentence options to be proposed as sentences that the user intends to input, based on the environment surrounding the user and a communication state of a device connected to a sentence input device used as a man-machine interface of a device to which the user needs to input sentences ;
The text input method according to claim 1 .

Images