JP6448051B2 - Brain machine interface device - Google Patents

Description

  The present invention relates to a brain machine interface (BMI) device, and more particularly, a steady state visual evoked potential (SSVEP) or a steady visual stimulus evoked magnetoencephalogram (SSVEF). : Steady State Visual Evoked Field).

  In a human cranial nerve network, an electroencephalogram (EEG) in a different form (change in frequency and intensity) is generated in accordance with a stimulus such as vision and hearing. Therefore, conventionally, research and development of a BMI that measures the brain waves generated by applying various visual and auditory stimuli to humans and controls the operation of any machine / device using the measured brain waves as input information. Is underway. According to such a control system using BMI, since the operation control of the machine / device can be achieved regardless of the operation by the human body movement, the driver operating the vehicle operates the operation of the machine / device. It is possible to control the operation of machines and devices even in situations where physical movement is restricted.

  As an example of the technology related to the BMI device as described above, for example, in Patent Document 1 by a part of the applicant of the present application, blinking light is emitted in a human brain wave that is gazing at blinking light at a certain frequency. One embodiment has been disclosed for improving a BMI device that utilizes a steady visual stimulus evoked potential (SSVEP) that appears in synchronism with the frequency of. In the configuration described in the publication, first, in order to reduce the glare and flickering feeling of the flashing light received by the user, the flashing frequency of the light being watched is the critical fusion frequency (flashing but continuously lighting up). (Frequency: CFF: Critical Fusion Frequency, usually approximately 34 Hz) or higher. However, the signal amplitude of SSVEP due to flashing light of CFF or higher is weak, and as it is, it is difficult to detect a signal with a high S / N ratio. Therefore, paying attention to the fact that the frequency of the SSVEP signal matches the frequency of the flashing light, the cross spectrum value (cross-correlation function value) between the driving signal of the flashing light source and the electroencephalogram signal is calculated, and the cross spectrum value is high ( That is, a configuration has been proposed in which it is determined that the user is gazing at the light source having the frequency of the flashing light (when the driving signal of the flashing light source and the electroencephalogram signal are synchronized).

  Further, in Patent Document 2 by a part of the applicant of the present application, as one form for practical application of a BMI device, a BMI device using SSVEP is an arbitrary one that a driver performs while driving a vehicle such as an automobile. When applied as an input means for operations such as wipers, air conditioners, headlamps, audio, etc., the brightness of the light emitted by the flashing light source can be adjusted by the ambient brightness, body temperature, etc. in order to improve operability A BMI control apparatus configured as described above has been proposed. In Patent Document 3 by a part of the applicant of the present application, in the BMI device using SSVEP, stochastic resonance caused by causing the user to visually recognize background light blinking at a random frequency simultaneously with the blinking target light. A configuration has been proposed in which SSVEP is amplified by the phenomenon to improve the detection accuracy or S / N ratio of the electroencephalogram and shorten the detection time.

  The change in the electroencephalogram that generates SSVEP, that is, the change in the electrical activity in the brain is also detected in the magnetoencephalogram obtained by measuring the magnetic field around the human brain, and the flashing light corresponding to SSVEP. The change in the magnetic field that appears in synchronization with the frequency of is called the steady visual stimulus-induced magnetoencephalogram (SSVEF). Similar to the case of a BMI device using SSVEP, a BMI device using SSVEF can also be realized.

JP2011-15788 JP2012-128512 JP 2014-71825 A

  As described in Patent Documents 1 to 3 above, when a user gazes at a light source that blinks at a certain blinking frequency, the blinking that appears in the electroencephalogram or magnetoencephalogram from the visual cortex of the user's brain In a BMI device that detects the presence or absence of SSVEP or SSVEF synchronized with the frequency and determines whether or not the user is gazing at the blinking light source, the intensity of SSVEP or SSVEF depends on the blinking frequency of the blinking light source. Change. In general, when the blinking frequency is lower than the critical fusion frequency or about 40 Hz or less, the intensity of SSVEP or SSVEF is relatively strong and easy to detect. However, when the blinking frequency of the blinking light source is lower than the critical fusion frequency, i.e., when the user can recognize the blinking of the light source, as already mentioned, the user will be aware of the blinking of the light source, i.e., the flickering of light. You may feel annoyance. In addition, when the blinking of a light source is in a state that can be recognized by a person, it is easier for people around the light source to be careful of the light source (because there is only light that can be recognized blinking in the peripheral vision, pay attention to the light. The flashing of the light source may be felt uncomfortable not only by the user of the device but also by the people around it. However, if the blinking frequency of the light source is made higher than the critical fusion frequency, as already mentioned, the intensity of SSVEP or SSVEF becomes weak as it is, and the detection accuracy can be reduced. Thus, in this field, the problem is how to increase the strength of SSVEP or SSVEF without making the user or the people around them feel “no bother” with the flickering of light.

  Thus, one object of the present invention is to provide an intensity of SSVEP or SSVEF by giving a pattern on the light source so that the user can unconsciously concentrate on the blinking light source in the BMI device using SSVEP or SSVEF as described above. Is to increase.

  In this regard, in the research of the inventors of the present invention, when a visually recognizable pattern is provided on the light source so that the user can unconsciously concentrate on the flashing light source, the intensity of SSVEP or SSVEF is: It was found to be increased over that otherwise. As will be described in more detail later in the section of the embodiment, according to the experiments by the inventors of the present invention, specifically, in the blinking light source, the shape of the human face, that is, the eyes, When a pattern having a shape such as a mouth or contour is displayed, it has been found that the strength of SSVEP or SSVEF is significantly increased as compared with the case where there is no such pattern. In the present invention, the above knowledge is utilized.

  According to the present invention, the above-mentioned problem is, in one aspect, a brain machine interface device using a steady visual stimulus evoked potential (SSVEP) or a steady visual stimulus evoked magnetoencephalogram (SSVEF). A light emitter that emits flashing light that is disposed at a position that can be visually recognized by the user, a brain wave or magnetoencephalogram detection means that detects the brain wave or magnetoencephalogram of the user, and the user emits the light in the electroencephalogram or magnetoencephalogram Detection means for detecting a steady visual stimulus evoked potential or a steady visual stimulus evoked magnetoencephalogram generated by visually recognizing blinking light emitted by the sensor, and detection of the presence of a steady visual stimulus evoked potential or a steady visual stimulus evoked magnetoencephalogram And a control means for controlling the operation of a control object associated with the light emitter in response, and achieved by a device on which a pattern of the shape of a human facial element is displayed on the light emitter. .

  In the above-described BMI device according to the present invention, basically, in the same manner as the BMI device using the conventional SSVEP, the user can watch the light emitting device that emits the flashing light, so that the user's visual field can be seen. Detection of SSVEP or SSVEF occurring in the system is performed. The blinking frequency of the blinking light is preferably set to CFF or higher so as to reduce glare and flickering feeling of the blinking light received by the user. The electroencephalogram or magnetoencephalogram detection means may be any device for detecting an electroencephalogram or magnetoencephalogram normally used in this field. In the case of detecting an electroencephalogram, typically, a plurality of electrodes fixed on the surface of the head around the user's visual cortex, and the potential of each electrode are sequentially measured, and signals from the electrodes are measured in an arbitrary manner. And a signal processing device for processing. In the case of detection of magnetoencephalogram, it consists of a magnetic sensor fixed around the head around the user's visual cortex, and a signal processing device that measures the output of each magnetic sensor and processes it in an arbitrary manner May be. As understood by those skilled in the art, the electroencephalogram and magnetoencephalogram differ only in whether the electrical activity in the brain is detected as a potential or as a magnetic field, and substantially the same in the brain. Electrical activity is the detection target.

  In the BMI device as described above, in the case of the present invention, generally speaking, a pattern on which a significant effect of increasing the intensity of SSVEP or SSVEF is obtained is displayed on the light emitter. One aspect of such a pattern is a pattern that has the effect of inducing the user to focus closely on a light emitting device that emits flashing light, i.e. the user unconsciously directs attention to the light source. An example of such a pattern is a pattern of the shape of a human face element, for example, a pattern of eyes, mouth, contour, etc. on the light emitting surface on the light emitter. The pattern of the shape of the elements of the human face may be a schematic figure.

  According to the experiment, when the pattern on the light emitter emits light outside the contour pattern of the face, the SSVEP or SSVEF does not emit light when the pattern outside the face contour does not emit light. It has been found that a further increase in strength is obtained.

  As described above, the BMI device of the present invention may be used to execute operation control of an arbitrary machine / device. In that case, specifically, it is determined whether or not the user has watched (or is) the blinking light by detecting the presence or absence of SSVEP, and based on the determination result, any machine / device is determined. Operation control such as operation start or stop is executed. In the BMI device, a plurality of light emitters that blink at different frequencies may be provided. Then, the SSVEP or SSVEF induced by each flashing light is synchronized with the flashing of each light, and therefore, in the electroencephalogram / magnetogram, SSVEP having a frequency substantially the same as the flashing frequency of each flashing light. Alternatively, by detecting whether or not the SSVEF is present, it is possible to determine whether or not the user is gazing at any one of the light emitters. When it is determined that a certain light emitter is being watched, operation control of a control object such as a machine / device associated with the light emitter is executed.

  Thus, according to the above configuration, in the detection of SSVEP or SSVEF, a pattern that induces the user to unconsciously concentrate on the light emitter as a stimulus for inducing SSVEP or SSVEF, particularly on the light emitter. By drawing the pattern of the shape of the element of the human face on the light emitter, the intensity of SSVEP or SSVEF can be increased. In general, it is known that an increase in the intensity of SSVEP or SSVEF can be achieved, for example, by increasing the intensity of flashing light, but such an increase in the intensity of flashing light is bothersome for the user because the light is too strong. You may feel it. On the other hand, it should be noted that in the present invention, an increase in the intensity of SSVEP or SSVEF can be achieved without necessarily increasing the intensity of the flashing light. Of course, it should be understood that higher SSVEP or SSVEF intensity may be obtained in combination with some increase in the intensity of the flashing light. Further, the method of the present invention may be used together with other methods for increasing the strength of SSVEP or SSVEF using, for example, the stochastic resonance phenomenon of Patent Document 3, in which case the strength of SSVEP or SSVEF is increased. An increase is expected.

  Other objects and advantages of the present invention will become apparent from the following description of preferred embodiments of the present invention.

FIG. 1 is a diagram for explaining the configuration of a BMI apparatus to which the present invention is applied. FIG. 2 is a diagram for explaining the generation of SSVEP or SSVEF used in the BMI apparatus of the present invention. (A) is a case where light blinking at a frequency higher than CFF is used as the stimulation light, and (B) is a case where light blinking at a frequency lower than CFF is used as the stimulation light. (C) is a case where the light source by which the pattern was drawn in the light emission site | part is used as stimulation light by this invention. In the figure, the black region is a light emitting region, and the white region is a region that does not emit light. FIG. 3 shows an example of a pattern drawn on the surface of the light emitting portion of the light emitter used in the present invention. In the figure, a black region is a light emitting region, and a white region is a region that does not emit light. In FIG. 3A, a to d are examples of patterns in which elements of a human face are schematically drawn so that a user's gaze point is guided at the center of the light emitting part. In FIG. 3B, a is the same pattern as in FIG. 3A, and a ′ is a pattern in which a light emitting region is provided outside the face contour in FIG. is there. FIG. 4A shows an average value and a standard value of the SSVEF characteristic value in the magnetoencephalogram measurement experiment when the user visually recognizes the flashing light from the light emitter on which the pattern of FIG. The deviation value is shown. For comparison, a case (S) in which a light emitter without a pattern is used is also shown. FIG. 4B shows the SSVEF characteristic value in the magnetoencephalogram measurement experiment when the user visually recognizes the flashing light from the light emitters with the patterns a and a ′ shown in FIG. 3B. Mean values and standard deviation values are shown.

DESCRIPTION OF SYMBOLS 100 ... Light emitter 110 ... Flashing light light emitter Br ... Human brain el ... Electrode apparatus

  The present invention will now be described in detail with reference to a few preferred embodiments with reference to the accompanying drawings. In the figure, the same reference numerals indicate the same parts.

Configuration of Device Referring to FIG. 1, a BMI device of the present invention has a basic configuration in which a light source device 100 carrying a light emitter 110 that emits flashing light to be visually recognized by a user, and the light emission. The electrical activity of the visual cortex of the user who visually recognizes the device is measured as an electroencephalogram or magnetoencephalogram to perform signal processing, and the operation of any device is controlled based on the detected electroencephalogram or magnetoencephalogram signal. It is comprised from the processing apparatuses 200-500.

  In the above device configuration, the light emitter 110 typically has a CFF (flashing but continuously to reduce glare and flickering for the user viewing the flashing light. It may be configured to emit light that blinks at an arbitrary frequency equal to or higher than the blinking frequency (usually approximately 34 Hz) that appears to be lit, for example, 34 to 40 Hz. Further, one or more light emitters 110 (three in the figure) may be provided on the light emitter 100. In that case, light emitted from each light emitter 110 is blinked at a different frequency. . Therefore, when the user focuses on one of the light emitters 110 with both eyes focused, SSVEP or SSVEF is generated in the user's visual cortex, which is synchronized with the blinking frequency of the observed light. . Thus, each of the plurality of light emitters 110 is associated with a device to be operated and controlled, and the user gazes at one of the light emitters 110 and there is the presence of SSVEP or SSVEF that is synchronized with the flashing light that has been watched. When detected, the operation control for the device associated with the light is executed.

  The electroencephalogram or magnetoencephalogram measurement may be performed in the normal manner in this field. In the electroencephalogram measurement, the electrode device el that senses a change in the potential of the visual cortex VF may be configured or arranged so as to be able to measure the potential at a measurement point according to the International 10-20 method, for example. Then, the potential signal sensed by the electrode device el is input to the processing device, where it is first amplified and A / D converted in a normal manner, and then digitized potential signal, ie, Then, FFT conversion is performed on the electroencephalogram signal to generate a power spectrum of the electroencephalogram signal (200). When the user gazes at the flashing light of the light emitter 110, and the SSVEP is generated, the frequency band substantially matches the flashing frequency of the light being watched by the SSVEP in the generated power spectrum. In this power spectrum, in the frequency band including the flashing frequency of the light emitting device 110, the presence or absence of the maximum value is inspected, and the flashing frequency is substantially reduced. This results in the detection of SSVEP when a maximum value is detected at a frequency equal to (300). When there are a plurality of light emitters 110 as shown in FIG. 1, the power spectrum in the frequency band including the flashing frequency of each flashing light is calculated, and the presence or absence of the maximum value at each flashing frequency. Is inspected. In magnetoencephalography measurement, the object to be measured is a magnetoencephalogram, a magnetic sensor is used as a sensor, a magnetic field signal sensed by the magnetic sensor is input to the processing device, and in the same manner as described above. An SSVEF detection process or the like is executed. Further, depending on the user, the intensity of the electroencephalogram or magnetoencephalogram may be high even in a state where SSVEP or SSVEF is not generated. Accordingly, the electroencephalogram or magnetoencephalogram measurement of the region where SSVEP or SSVEF does not occur is performed simultaneously, and the ratio of the intensity or power value of the electroencephalogram or magnetoencephalogram in the region where SSVEP or SSVEF can occur and the region where SSVEP or SSVEF does not occur. When the intensity ratio increases, it may be determined that SSVEP or SSVEF has occurred.

  Thus, when SSVEP or SSVEF corresponding to the light of the light emitter 110 is detected, the processing device generates a control command to the device to be controlled associated with the target light (command generation: 400), In order to execute the control of the device in accordance with such a control command, the operation of the related drive device or the like is executed (device control: 500). The device to be controlled may be a wiper, an air conditioner, a headlamp, an audio, or the like when the BMI device of the present invention is used as a device operating means by a driver of a vehicle such as an automobile. Further, it may be used for powering on / off an arbitrary device, channel operation or volume adjustment of a television or radio.

  In the BMI apparatus as described above, in particular, in the case of the present invention, as described in the “Summary of Invention” section, the user unconsciously concentrates on the flashing light source on the surface of the light emitting portion of the light emitter 110. Pattern, specifically, a pattern of the shape of a human face element, for example, a pattern such as an eye, a mouth, a contour, etc., thereby increasing the strength of SSVEP or SSVEF and thereby improving the detection accuracy Will be achieved.

As described above for the pattern of the flashing light source of the light emitter, when a person watches the light source that flashes at a certain flashing frequency, that is, the light source whose light intensity periodically increases or decreases, or the light source that repeatedly turns on and off Since SSVEP or SSVEF synchronized with the flashing frequency appears in the electroencephalogram or magnetoencephalogram from the visual cortex of the person's brain, the person watches the flashing light source by checking for the presence of SSVEP or SSVEF. It can be determined whether or not. In that case, in order to prevent the user who is gazing at the light source from feeling “inconvenience” in the flickering of the light, the blinking frequency has been set as shown in FIG. When flashing light higher than the critical fusion frequency is used, generally, the higher the flashing frequency, the more the intensity of SSVEP tends to decrease. Therefore, as it is, only SSVEP having a relatively low intensity is detected. On the other hand, as shown in FIG. 2B, when the blinking frequency is lower than the critical fusion frequency or about 40 Hz or less, the intensity of SSVEP is relatively strong, but the user who watches the light source blinks the light source. Because it can be recognized, the flickering of the light may feel “annoying”, and for those around it, it is easy to be careful of blinking, and it may feel uncomfortable. As a light source to be watched by a user of the apparatus, a light source having a blinking frequency of not more than a critical fusion frequency or not more than about 40 Hz has been supposed to be not very advantageous in practice.

  In this regard, as described in the “Summary of the Invention” section, as schematically illustrated in FIG. 2C, the pattern of the shape of the human face element, for example, the eyes, mouth, contour As described above, it has been found that the intensity of SSVEP or SSVEF generated when a person gazes at the light is increased. Thus, in the BMI device of the present invention, as described above, the pattern of the shape of the human face element, which is one aspect of the pattern in which the user's gazing point is guided to the flashing light source, is drawn as the flashing light source. Is used.

  Specifically, referring to FIG. 3 (A), as a pattern of the shape of a human face element, as shown in FIG. 3 (A) a, a pattern in which the outlines of eyes, mouth and face are schematically drawn. It may be. In addition, similar to this, as shown in FIG. 3 (A) b, a pattern including facial elements having characteristics different from the human face, such as a large head, large eyes, etc. 3) A pattern imagining an angry face as shown in FIG. 3A (the virtual extension of the eyes and mouth is the intersection, leading to the point of gaze), or As shown in FIG. 3 (A) d, it may be a profile of a profile (because it has elements of wrinkles and ambiguous figures of profile, it attracts more attention and induces more attention). As shown in a later experimental example, the pattern of the shape of the human face element is shown in FIG. It has been found that the increase in the intensity of SSVEP or SSVEF is greater than the pattern with no light emitting sites on the outside. That is, in order to increase the intensity of SSVEP or SSVEF, it can be said that a pattern having a light emitting part on the outside of a human contour is more preferable.

Effect Confirmation Experiment The effectiveness of the present invention described above was confirmed by the following experiment. It should be understood that the following experiments illustrate the effectiveness of the present invention and do not limit the scope of the present invention.

In this experiment, a magnetoencephalogram was measured as an electrical activity in the subject's brain. As a measuring device, a 306 channel brain magnetic field measuring device (MEG) manufactured by ELEKTA-Neuromag was used. The measuring device was installed in a dark room. As a light source to be visually recognized by the subject, a ground glass was placed in front of the subject in the dark room, and the subject was visually recognized by projecting flashing light including the pattern depicted in FIG. . The position and dimensions of the light source are respectively set such that the distance (viewing distance) from the subject is 1000 mm, the vertical viewing angle is 3 deg (52 mm), and the brightness of the flashing light (in the flashing state) is 200 cd / m ² . The frequency was 40 Hz (duty ratio lighting: extinguishing = 2: 1). In the measurement of the magnetoencephalogram, 306 sensors are arranged around the head of the subject, and 19 of them are located in the occipital lobe V1 to V5 area of the brain and the temporal region where SSVEF occurs. The leaf TE was distributed (target sensor group). The number of subjects was 10. In the analysis of the measured value, FFT analysis is performed on the sensor output value for each sensor, a power value in the range of 39.9329 to 40.554 Hz is calculated, and all sensor groups (306 are used as an index of SSVEF intensity. The ratio (SSVEF characteristic value) of the root mean square of the power values of the target sensors and the root mean square of the power values of the target sensor group (19) was referred to. According to the SSVEF characteristic value, it is possible to detect a relative change between a state in which SSVEP or SSVEF does not occur and a state in which SSVEP or SSVEF has occurred. (The SSVEF characteristic value is used as an index of the strength of the SSVEF. As already mentioned, there is an individual difference in the magnetoencephalogram strength when the SSVEF is not generated depending on the user, and the power value of the frequency of the SSVEF generated is This is because it is difficult to compare the strength of SSVEF with only the absolute value.)

  FIG. 4A shows SSVEF characteristic values (for 10 subjects) measured in the case of the light source with the pattern a illustrated in FIG. 3A and the case of the plain light source S. Average value (bar graph) and standard deviation (error bar) are shown. With reference to the figure, the average value of the SSVEF characteristic value of the light source a on which the pattern was drawn increased as compared with the case of the plain light source S. According to the t-test of the result of the SSVEF characteristic value of the light source on which the pattern is drawn with respect to the average value of the SSVEF characteristic value of the plain light source S, the average of the SSVEF characteristic values of the light source on which the pattern is drawn with 5% significance It was shown that the value is higher than the average value of the SSVEF characteristic values of the plain light source S. The reason why the intensity of SSVEP or SSVEF is increased is that when a blinking light is viewed, only V1 (primary visual cortex) reacts in the case of a plain light source, but V2 (2 It is considered that this is because the next visual cortex, V3 (third visual cortex), V4 (fourth visual cortex), V5 (fifth visual cortex), and temporal lobe TE cortex also react. Thus, according to this result, it was confirmed that the intensity of SSVEP or SSVEF was increased in the case of a light source with a pattern drawn.

  FIG. 4B shows an average value (bar graph) and standard deviation of SSVEF characteristic values (measured by 10 subjects) in measurement using the patterns a and a ′ illustrated in FIG. 3B. (Error bar). Referring to the figure, the average value of a ′ is larger in the case of the pattern having the light emitting part outside the contour of the person than the average value of the pattern a having no light emitting part outside the contour of the person. . It should be noted that according to the t-test of the result of the SSVEF characteristic value of a ′ in the case of a pattern having a light emitting part outside the human contour relative to the average value of the SSVEF characteristic value of the pattern a having no light emitting part outside the human contour, It was shown that the average value of the SSVEF characteristic value of a ′ was high in the case of a pattern having a light emitting part on the outside of the outline of the person with 5% significance. Thus, according to this result, it was confirmed that the intensity of SSVEP or SSVEF is further increased in the case where the light emitting part is also present outside the outline of the person.

  Thus, according to the present invention described above, in the detection of SSVEP or SSVEF, by using a light source having a light-emitting part in the shape of a human facial element as a stimulus for inducing SSVEP or SSVEF, SSVEP or SSVEF is used. An increase in strength is achieved. In the case of the present invention, this can be achieved only by adding a pattern to the light source. This is advantageous in that the S / N ratio and detection accuracy of SSVEP detection can be improved easily and inexpensively without complicating other configurations. is there.

  Although the above description has been made in relation to the embodiment of the present invention, many modifications and changes can be easily made by those skilled in the art, and the present invention is limited to the embodiment exemplified above. It will be apparent that the invention is not limited and applies to various devices without departing from the inventive concept.

Claims (1)

A brain machine interface device using a steady visual stimulus evoked potential or a steady visual stimulus evoked magnetoencephalogram,
A light emitter that emits flashing light that is placed at a position that is visible to the user;
An electroencephalogram or magnetoencephalogram detection means for detecting the electroencephalogram or magnetoencephalogram of the user;
Detecting means for detecting a steady visual stimulus evoked potential or a steady visual stimulus evoked magnetoencephalogram generated by the user visually recognizing the blinking light emitted by the light emitter in the electroencephalogram or magnetoencephalogram;
Control means for controlling the operation of a control object associated with the light emitter in response to detection of the presence of the steady visual stimulus evoked potential or the steady visual stimulus evoked magnetoencephalogram,
An apparatus for displaying, using the flashing light, a pattern in which the inner side and the outer side of the outline emit light including the shape of the outline of a human eye and face on the light emitter .

Images