JP6908902B2 - Information input device based on fluctuations in pupil diameter - Google Patents

Description

The present invention relates to an information input device based on fluctuations in pupil diameter, and more particularly to an information input device that determines an option that a user desires to input from a plurality of options by measuring fluctuations in pupil diameter.

As an information input device for a computer, a mobile terminal, a game device, or the like, a keyboard or a game controller that can be input by a finger is generally known. An information input device that uses the line of sight of the user without using a finger is also known (for example, Patent Document 1 and Patent Document 2). All of the input devices using such a line of sight are selected from a plurality of options by detecting the physical direction (line of sight direction) of the user's eyeball by the electro-oculography method, the corneal reflex method, the video image method, or the like. When the user gazes at the option desired to be input (turns his / her eyes), this option is determined as the option desired to be input.

Further, although it is not an information input device, Patent Document 3 by the same applicant as the applicant of the present application uses the relationship between the brightness distribution of the image in the field of view of the user and the change in the size of the pupil of the user. A caution position estimation device that estimates a position that a user is paying attention to is disclosed. This uses the low brightness position in the luminance distribution when the pupil size changes to the larger side, and the high luminance position in the luminance distribution when the pupil size changes to the smaller side. It is presumed to be the position that the person is paying attention to. That is, it is possible to estimate the user's attention position without looking at it. This is a technique of detecting a change in the size of the pupil and estimating the attention position of the user by using the luminance distribution, and does not change the diameter of the pupil by an external stimulus.

Japanese Unexamined Patent Publication No. 2012-22148 JP-A-2017-507400 Japanese Unexamined Patent Publication No. 2013-240469

The devices of Patent Document 1 and Patent Document 2 are not a problem if the user can freely look at the options, but should be used for a person with a disability who cannot move the eyeball. I couldn't. Further, in the case of using the line-of-sight direction as in Patent Document 1 and Patent Document 2, since it is necessary to match the measured eyeball position with the actual line-of-sight, calibration is required before use. In addition, since the line of sight shifts as the head moves, it is necessary to fix the head position or introduce an algorithm that corrects the head position. Further, the line-of-sight direction varies greatly among individuals, and in some cases, sufficient accuracy cannot be obtained.

In view of such circumstances, the present invention is an information input device based on fluctuations in pupil diameter, which enables the user to accurately determine an option for input with high accuracy even if the line of sight cannot be moved. Is intended to provide.

In order to achieve the above-described object of the present invention, the information input device based on the fluctuation of the pupil diameter according to the present invention is displayed on the display unit and the display unit for displaying a plurality of options to be input to the user. Relationship between the brightness fluctuation unit that applies unique brightness fluctuations to each of the plurality of options, the measurement unit that measures the fluctuation of the user's pupil diameter, and the brightness fluctuation by the brightness fluctuation unit and the pupil diameter fluctuation by the measurement unit. It is provided with an input determination unit that determines an option desired to be input by the user consciously from a plurality of options by using.

Here, the luminance fluctuation unit may be any one that changes the luminance by any of frequency modulation, amplitude modulation, phase modulation, and duty ratio modulation.

Further, the input determination unit may determine the option by using the rise time or fall time of the pupil diameter fluctuation, or the rise angle or fall angle.

Further, the input determination unit may determine the option after removing the variation as noise when the variation that is not in the luminance variation applied by the luminance variation unit appears in the pupil diameter variation.

In addition, the measurement unit further measures the user's respiration and / or heartbeat, and the input determination unit determines the option after removing the fluctuation synchronized with the respiration and / or heartbeat measured by the measurement unit as noise. It may be something to do.

Further, the display unit may be configured so that a plurality of options can be displayed hierarchically by the user being aware of the options for a predetermined time.

The information input device based on the fluctuation of the pupil diameter of the present invention has an advantage that even if the line of sight cannot be moved, it is possible to determine the option desired to be input with high accuracy only by the user's consciousness. be.

FIG. 1 is a schematic block diagram for explaining the configuration of the information input device of the present invention. FIG. 2 is a fluctuation graph for explaining the relationship between luminance fluctuation and pupil diameter fluctuation in the information input device of the present invention. FIG. 3 is a pupil diameter fluctuation graph for explaining noise to be removed in the input determination unit of the information input device of the present invention.

The information input device of the present invention inputs predetermined information to an information processing device or the like based on a change in the pupil diameter of the user. Here, the information processing device may be a computer, a mobile terminal, a game device, or the like. Further, the information input device is a so-called human interface, and is used to appropriately input characters, symbols, predetermined options, and the like into the information processing device.

Hereinafter, embodiments for carrying out the present invention will be described with reference to the illustrated examples. FIG. 1 is a schematic block diagram for explaining the configuration of the information input device of the present invention. As shown in the figure, the information input device of the present invention based on the fluctuation of the pupil diameter of the user 1 is mainly composed of a display unit 10, a brightness fluctuation unit 20, a measurement unit 30, and an input determination unit 40. ..

The display unit 10 displays a plurality of options 11 to be input to the user 1. The display unit 10 may be, for example, a monitor of a computer or a game device or a screen of a mobile terminal. As a plurality of options 11 to be input, for example, 10 keys are shown in the illustrated example. However, the present invention is not limited to this, and any option such as a QWERTY key or a Yes / No key can be displayed.

The brightness fluctuation unit 20 applies unique brightness fluctuations to each of the plurality of options 11 displayed on the display unit 10. Specifically, the luminance fluctuation unit 20 may perform the luminance variation with respect to the option 11, for example, by any of frequency modulation, amplitude modulation, phase modulation, and duty ratio modulation. That is, for example, in the case of frequency modulation, the rhythm (frequency) of the luminance fluctuation is displayed differently for each option 11. Further, in the case of amplitude modulation, the brightness (intensity) of the peak of the luminance fluctuation is displayed differently for each option 11. Further, in the case of phase modulation, the peak positions (phases) of the luminance fluctuations are displayed differently. Further, in the case of duty ratio modulation, the peak time (duty ratio) of the luminance fluctuation is displayed differently. Further, the variable waveform may be a waveform such as a sine wave, a triangular wave, or a square wave that can change the brightness of the option 11 displayed on the display unit 10.

The measuring unit 30 measures the fluctuation of the pupil diameter of the user 1. It is preferable that the measuring unit 30 can measure the pupil diameter of the user 1 in real time. Specifically, for example, the measuring unit 30 includes an imaging unit that captures the eyeball of the user 1 and a pupil measuring unit that measures the size of the pupil from the captured eyeball image. As for the measurement of the pupil diameter, any method such as an eye tracker or an eye mark recorder that can measure the change in the pupil diameter may be used. The method for measuring the pupil diameter is not limited to a specific method, and it may be measured by any conventional method or a method to be developed in the future.

The input determination unit 40 determines an option for which input is desired from among the plurality of options 11 by the user 1 being aware of (attention). This is determined by using the relationship between the luminance fluctuation by the luminance fluctuation unit 20 and the pupil diameter variation by the measurement unit 30. Here, the relationship between the luminance fluctuation and the pupil diameter fluctuation will be described. It is generally known that when the line of sight is directed toward the brighter side, the pupil contracts and becomes smaller, and when the line of sight is directed toward the darker side, the pupil expands and becomes larger. .. That is, according to this phenomenon, the pupil diameter fluctuates in conjunction with the luminance fluctuation by the luminance fluctuation unit 20. However, the present invention does not use such a pupil diameter variation linked to the luminance variation at the tip of the line of sight. In the present invention, the pupil diameter fluctuation linked to the brightness fluctuation at the point where the consciousness is directed is used. This uses a phenomenon in which the pupil diameter fluctuates according to the brightness fluctuation of the option to which the consciousness is directed by focusing the consciousness on the option desired to be input without moving the line of sight. This phenomenon was discovered by the inventor of the present application. That is, even if the user 1 is looking at a specific option, if the conscious option is another option, the pupil diameter variation will occur in accordance with the brightness variation of the other option. Therefore, even if the head is not fixed and the line of sight moves, the pupil diameter measurement is not affected by the movement of the line of sight, and a specific conscious option can be determined as the option desired to be input.

Here, the relationship between the luminance fluctuation and the pupil diameter fluctuation will be specifically described. If the frequency of the luminance fluctuation is too high or too low, the pupil diameter fluctuation may not follow it. Therefore, the frequency range of the luminance fluctuation may be determined within the range in which the pupil diameter fluctuation can be followed. As specific numerical values, for example, the relationship between luminance fluctuation and pupil diameter fluctuation was investigated under the following conditions. As options 11 to be displayed on the display unit 10, 12 types having a diameter of about 6.5 deg were prepared. The background color was constant at ^{16.4 cd / m 2.} Then, the brightness fluctuation unit 20 performed different frequency modulation for each option 11 with a brightness fluctuation amplitude in the range of the minimum brightness of 0.37 cd / m ² to the maximum brightness of 108.3 cd / m ^2. As a result, it was found that, for example, on the low frequency side, the followability of the pupil diameter fluctuation becomes high with respect to the luminance fluctuation frequency of about 0.50 Hz or more, more preferably 0.82 Hz or more. Further, on the high frequency side, sufficient followability was observed up to about 2.00 Hz. Then, if there is a frequency difference of about 0.12 Hz or more with respect to each option 11, the followability of the pupil diameter variation is observed to the extent that the plurality of options 11 can be sufficiently distinguished from each other. Note that these numerical values are merely examples, and the information input device of the present invention is not limited to these numerical values.

The relationship between the luminance variation and the pupil diameter variation will be described in detail with reference to FIG. 2 using a variation graph. FIG. 2 is a fluctuation graph for explaining the relationship between luminance fluctuation and pupil diameter fluctuation in the information input device of the present invention. In the figure, the horizontal axis is time and the vertical axis is brightness or pupil diameter. Further, FIG. 2A is a fluctuation graph in the case of frequency modulation, FIG. 2B is amplitude modulation, FIG. 2C is phase modulation, and FIG. 2D is duty ratio modulation. In addition, FIG. 2 (a), FIG. 2 (b), and FIG. 2 (c) show the brightness changed by a sine wave, and FIG. 2 (d) shows the one changed by a square wave. .. However, in any modulation method, it is possible to change the brightness not only by a sine wave or a square wave but also by a triangular wave or the like. The upper part of each figure is a luminance fluctuation graph by the luminance fluctuation unit 20, and the lower row is a pupil diameter fluctuation graph by the measurement unit 30. Then, as an example, the fluctuations of the two options with different modulations are shown by the black line and the gray line. It should be noted that these are schematically shown for the sake of explanation, and do not show actual measurement data. As shown in the figure, it can be seen that the luminance fluctuation and the pupil diameter fluctuation are synchronized. That is, even when the display unit 10 is subjected to different luminance fluctuations for the plurality of options 11, the pupil diameter variation synchronized with the option 11 to which the user 1 is conscious can be obtained. I understand. Specifically, as shown in FIG. 2A, it can be seen that if the frequency of the luminance fluctuation changes, the frequency of the pupil diameter fluctuation also changes. Further, as shown in FIG. 2B, it can be seen that if the amplitude of the luminance fluctuation changes, the amplitude of the pupil diameter fluctuation also changes. Similarly, as shown in FIG. 2C, it can be seen that if the phase of the luminance fluctuation changes, the phase of the pupil diameter fluctuation also changes. Further, as shown in FIG. 2D, it can be seen that if the duty ratio of the luminance fluctuation changes, the duty ratio of the pupil diameter fluctuation also changes.

In the information input device of the present invention, using such a phenomenon, the input determination unit 40 determines an option desired to be input from the plurality of options 11 by the user 1 being aware of it. Although there are individual differences in the fluctuation rate of the pupil diameter, since the pupil diameter fluctuations are synchronized with each other with a predetermined relationship, the error due to the individual differences is smaller than that using the line of sight.

Here, the method of determining the options by the input determination unit will be described more specifically. As described above, the input determination unit 40 determines the option for which input is desired by using the relationship between the luminance fluctuation and the pupil diameter fluctuation. At this time, the observation of the amplitude of the fluctuation for a predetermined time, that is, a plurality of times is observed. Just wait and decide on your options. However, the information input device of the present invention is not limited to this. For example, the input determination unit 40 may determine the option by using the rise time or the fall time of the pupil diameter fluctuation. That is, when the user 1 is aware of the option desired to be input, the pupil diameter fluctuation starts according to the brightness fluctuation of the option. Therefore, if the rise time and the fall time of the pupil diameter fluctuation are measured, the brightness fluctuation It is possible to determine a specific option synchronized with the rise time and fall time of. This may be not only the rising time and the falling time, but also the rising angle or the falling angle of the pupil diameter fluctuation. As a result, it is possible to realize an information input device having a faster reaction speed without observing the amplitude of fluctuations for a plurality of times. Specifically, for example, it is possible to determine the options in less than 1 second.

Next, noise removal will be described with reference to FIG. FIG. 3 is a pupil diameter fluctuation graph for explaining noise to be removed in the input determination unit of the information input device of the present invention. In the figure, the horizontal axis is time and the vertical axis is the pupil diameter. The results of pupil diameter fluctuation when frequency modulation is used as the brightness fluctuation are shown. In the absence of noise, in the case of frequency modulation, there should be constant variation at a given frequency, given the particular choice of input desired. However, as shown in the figure, fluctuations that are not included in the brightness fluctuations performed in the brightness fluctuation portion may appear in the pupil diameter fluctuations. That is, there may be fluctuations in frequencies that deviate from a certain frequency. At this time, the fluctuation may be removed as noise by the input determination unit 40. For example, first, the peak value of each amplitude is detected. Then, noise is removed by excluding points where the interval between peak values is significantly different from other intervals. Then, the average of the intervals between the other peak values is calculated. After performing such noise removal, it may be configured so as to determine an option in which the luminance fluctuation is performed in synchronization with the pupil diameter fluctuation.

As for noise, pupil diameter fluctuations may be affected by, for example, respiration and heartbeat. To remove this noise, the configuration may be as follows. First, the measuring unit 30 further measures, for example, the respiration and / or heartbeat of the user in addition to the pupil diameter. Then, the input determination unit 40 may determine the option after removing the fluctuation synchronized with the respiration and / or the heartbeat measured by the measurement unit 30 as noise. Regarding blinking, a defect will occur in the pupil diameter fluctuation graph, but there is no particular effect due to this. That is, if the fluctuation is measured to the extent that the frequency, phase, etc. can be detected, it is possible to determine the synchronous luminance fluctuation option without any problem even if there is a defect.

In addition, the measuring unit 30 may further measure the congested eye movement, the following eye movement, the electroencephalogram fluctuation, and the like, and may use it as auxiliary information when determining the options of the input determination unit 40, if necessary.

In the above illustrated example, an example in which a plurality of options are displayed on one screen of the display unit 10 is shown, but the present invention is not limited thereto. For example, the user may be aware of one option for a predetermined time, so that a plurality of options may be displayed hierarchically. That is, one option and a plurality of options associated with the option may be configured to be hierarchically displayable. As a result, the function of displaying another option by a so-called long press can be realized by the information input device of the present invention. Specifically, for example, in the case of the Japanese syllabary, only the "A" column is displayed as an option on the first screen, such as "A", "Ka", "Sa", "Ta", and "Na". Then, when the user 1 is conscious of, for example, "ka" as an option for which he / she wants to input for a predetermined time, as a plurality of new options, "ka", "ki", "ku", "ku" in the "ka" line, It is configured so that "ke" and "ko" are displayed. As a result, efficient input is possible even when the display screen of the display unit 10 is small.

As described above, according to the information input device of the present invention, it is possible to determine an option for which input is desired without looking at the line of sight. Therefore, even if the eyeball cannot be moved and the line of sight cannot be directed, the option can be determined with high accuracy only by the user being aware of the option desired to be input.

Further, in the information input device of the present invention, it is possible to use not only characters but also figures and the like as a plurality of options to be input to be displayed on the display unit. Further, for the luminance variation with respect to the options by the luminance fluctuation unit, any modulation method of frequency modulation, amplitude modulation, phase modulation, and duty ratio modulation can be selected. Here, if the optimum brightness variation exists depending on the options, the brightness variation unit may dynamically switch the optimum modulation method for each option. Further, the brightness variation with respect to the options by the brightness variation unit does not have to uniformly change the entire one option, and the brightness variation may be applied to a part of one option. Further, the modulation state of each option may be changed with time, and the option may be determined by using the pupil diameter variation synchronized with the change of the modulation state. Specifically, for one option, the luminance fluctuation frequency is dynamically changed from, for example, 1 Hz to 2 Hz, so that the pupil diameter fluctuation synchronized with this becomes a characteristic movement. It is possible to improve the accuracy. Further, the relationship of the modulation state for each option may be changed with time. Specifically, at a certain time, the luminance fluctuation frequency is modulated at, for example, 1 Hz for option A, the luminance fluctuation frequency is modulated at, for example, 2 Hz for the adjacent option B, and at the next time, option A is modulated at 2 Hz. It may be changed with time, such as modulation with and modulation of option B at 1 Hz. In this way, it is also possible to improve the identification accuracy by changing the modulation state with time between one option or a plurality of options.

The information input device based on the fluctuation of the pupil diameter of the present invention is not limited to the above-mentioned illustrated examples, and it goes without saying that various changes can be made without departing from the gist of the present invention.

1 User 10 Display unit 11 Choices 20 Luminance fluctuation unit 30 Measurement unit 40 Input determination unit

Claims (5)

An information input device based on fluctuations in the pupil diameter of the user, and the information input device is
A display unit that displays multiple options to be input to the user,
A luminance fluctuation unit that applies unique brightness fluctuations to each of the plurality of options displayed on the display unit, and a luminance fluctuation unit.
A measuring unit that measures fluctuations in the pupil diameter of the user,
By using the relationship between the brightness fluctuation due to the brightness fluctuation unit and the rise time or fall time of the pupil diameter fluctuation by the measurement unit, or the rise angle or fall angle , the user can be aware of the plurality of options. An input decision unit that determines the options you want to enter, and
An information input device comprising. An information input device based on fluctuations in the pupil diameter of the user, and the information input device is
A display unit that displays multiple options to be input to the user,
A luminance fluctuation unit that applies unique brightness fluctuations to each of the plurality of options displayed on the display unit, and a luminance fluctuation unit.
A measuring unit that measures fluctuations in the pupil diameter of the user,
When fluctuations that are not included in the brightness fluctuations performed by the luminance fluctuation unit appear in the pupil diameter fluctuations by the measurement unit, the fluctuations are removed as noise, and then the relationship between the luminance fluctuations by the luminance fluctuation unit and the pupil diameter fluctuations by the measurement unit. An input determination unit that determines the option that the user wants to input from a plurality of options by being aware of it.
An information input device comprising. The information input device according to claim 1 or 2 , wherein the luminance fluctuation unit performs luminance fluctuation by any one of frequency modulation, amplitude modulation, phase modulation, and duty ratio modulation. In the information input device according to any one of claims 1 to 3 , the measuring unit further measures the user's respiration and / or heartbeat.
The input determination unit determines the option after removing the fluctuation synchronized with the respiration and / or the heartbeat measured by the measurement unit as noise.
An information input device characterized by the fact that. In the information input device according to any one of claims 1 to 4 , the display unit is configured to be able to display a plurality of options hierarchically by the user being aware of the options for a predetermined time. An information input device characterized by.

Images