JP7844007B2 - Processing method and program - Google Patents

Description

This invention relates to a processing method and program for more quantitatively estimating the state of brain activity.

Non-invasive methods for measuring brain activity include, most notably, electroencephalography (EEG), which observes the brain's electrical activity, magnetoencephalography (MEG), and functional magnetic resonance imaging (fMRI), which observes the state of blood flow in the brain.

Neurofeedback (NF) training methods have been developed and proposed that observe the neural activity of a subject based on information indicating brain activity measured from the subject, and guide the subject in a more favorable direction according to the observed neural activity (see, for example, Non-Patent Documents 1 and 2).

As a visualization method for presenting information indicating brain activity, for example, Non-Patent Document 3 discloses an example of representing the activity of a specific brain region (EEG) in the form of a level meter. Non-Patent Document 4 also discloses an example of representing the activity of a specific brain region (fMRI) by the size of a flame. However, the technologies disclosed in Non-Patent Documents 3 and 4 are not for NF training, but rather for healthy individuals to understand the state of their own brain activity.

Furthermore, Non-Patent Document 5 discloses a technique for making specific waveforms audible in order to detect abnormal brain waves (e.g., stroke). The technique disclosed in Non-Patent Document 5 is for monitoring the brain state of patients, not for NF training.

Furthermore, Non-Patent Document 6 discloses a technique for mapping and presenting electroencephalogram (EEG) spectra in a multimodal manner. The technique disclosed in Non-Patent Document 6 is intended for use as a relaxation tool.

Anastasiia Belinskaia et al, "Short-delay neuralfeedback facilitates training of the parietal alpha rhythm," Journal of Neural Engineering, Volume 17, Number 6, 16 December 2020 Laura Diaz Hernandez et al, "Towards Using Microstate-Neurofeedback for the Treatment of Psychotic Symptoms in Schizophrenia," A Feasibility Study in Healthy Participants, Brain Topogr 29, 308-321 (2016). https://doi.org/10.1007/s10548-015-0460-4, 19 November 2015 Madita Stirner et al, "An Investigation of Awareness and Metacognition in Neurofeedback with the Amygdala Electrical Fingerprint," Consciousness and Cognition, Volume 98, February 2022, 103264. https://doi.org/10.1016/j.concog.2021.103264 R. Christopher deCharms et al, "Control over brain activation and pain learned by using real-time functional MRI," PNAS, December 20, 2005, vol.102, no.51. https://www.pnas.org/doi/pdf/10.1073/pnas.0505210102 Gerold Baier et al, "Event-based sonification of EEG rhythms in real time," Clinical Neurophysiology 118 (2007) 1377-1386, Thilo Hinterberger et al, "The Sensorium: Psychophysiological Evaluation of Responses to a Multimodal Neurofeedback Environment," Appl Psychophysiol Biofeedback 41, 315-329 (2016). https://doi.org/10.1007/s10484-016-9332-2

For example, when considering applications such as neuromuscular training, there is a need for technology that can evaluate the state of brain activity corresponding to cognitive function (whole-brain state) in real time. The background technologies disclosed in the aforementioned prior art documents cannot meet this requirement.

This invention provides a novel solution for more quantitatively estimating the state of brain activity.

A processing method according to one embodiment of the present invention includes the steps of: acquiring multiple brain activity patterns indicating the state of a subject's brain activity over a predetermined period of time; generating a spatial correlation matrix by calculating the spatial correlation for each combination of the multiple brain activity patterns; converting the spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating coordinates of a predetermined number of dimensions for each of the multiple brain activity patterns by singular value decomposition of the distance matrix; and determining the brain state corresponding to each brain activity pattern based on the magnitude of the spatial correlation between each of the multiple brain activity patterns and a plurality of templates each representing a predetermined number of brain states.

The processing method may further include the step of representing the coordinates corresponding to each of multiple brain activity patterns, along with the corresponding brain states, on the same coordinate system.

Multiple templates may include multiple pairs of templates with reversed polarity. For each pair of templates, the average value of the coordinates corresponding to one template and the coordinates corresponding to the other template may be determined as the center of the coordinates corresponding to multiple brain activity patterns.

The processing method may further include the step of determining the coordinates corresponding to a new template based on the coordinates corresponding to a first template among multiple templates and the coordinates corresponding to a second template among multiple templates.

A processing method according to another embodiment of the present invention includes the steps of: obtaining a set of standard state patterns including a plurality of brain activity patterns indicating the brain activity states of one or more subjects and a plurality of templates each indicating a predetermined set of brain states; generating a first spatial correlation matrix by calculating the spatial correlation for each combination included in the set of standard state patterns; converting the first spatial correlation matrix into a distance matrix using multidimensional scaling and calculating a first set of coordinates corresponding to the elements included in the first spatial correlation matrix by singular value decomposition of the distance matrix; and indicating the brain activity states of the subjects. The method includes the steps of: obtaining a first brain activity pattern; generating a second spatial correlation matrix by calculating the spatial correlation between the standard state pattern group and each combination included in the first brain activity pattern; converting the second spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating a second set of coordinates corresponding to the elements included in the second spatial correlation matrix by performing singular value decomposition on the said distance matrix; and estimating the coordinates corresponding to the first brain activity pattern in the first coordinate set based on the relative positional relationships of the coordinates corresponding to the first brain activity pattern in the second coordinate set.

The processing method may further include the step of displaying the coordinates corresponding to the first brain activity pattern in a two-dimensional or three-dimensional coordinate system.

The processing method may further include repeating the steps of acquiring a first brain activity pattern, generating a second spatial correlation matrix, calculating a second set of coordinates, and estimating coordinates. The display step may include displaying the sequentially estimated coordinates as a trajectory.

The display step may include a step of displaying the estimated coordinates using a spherical projector.

The processing method may further include the steps of: acquiring multiple brain activity patterns representing the brain activity states of multiple subjects; generating a third spatial correlation matrix by calculating the spatial correlation for each combination included in the acquired multiple brain activity patterns; converting the third spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating a third set of coordinates corresponding to the elements included in the third spatial correlation matrix by singular value decomposition of the said distance matrix; and extracting brain activity patterns corresponding to a predetermined number of coordinates extracted from the third set of coordinates as multiple brain activity patterns of a standard state pattern group.

A predetermined number of coordinates may be extracted from a third group of coordinates according to a predetermined rule.

According to another embodiment of the present invention, a program is provided for causing a computer to execute the above-described processing method.

According to one embodiment of the present invention, a means for more quantitatively estimating the state of brain activity can be realized.

This is a schematic diagram showing an example configuration of a neurofeedback system according to this embodiment. This is a schematic diagram showing the device configuration of a processing unit included in a neurofeedback system according to this embodiment. This figure illustrates an example of processing for estimating a brain state according to this embodiment. This figure shows an example of a vector representing a brain state according to this embodiment. This figure illustrates brain activity patterns and templates for estimating brain states according to this embodiment. This is a schematic diagram showing an example of a user interface screen provided by a processing device according to this embodiment. This figure shows an example of the evaluation results for the intrinsic dimensions included in the EEG. This figure shows an example of a brain activity pattern used to determine the state space in the brain state estimation process according to this embodiment. This figure shows an example of a spatial correlation matrix used to determine the state space in the brain state estimation process according to this embodiment. This figure shows an example of a state space calculated from a predetermined number of brain activity patterns in the brain state estimation process according to this embodiment. This figure shows an example of a neural manifold that appears in the state space determined by the brain state estimation process according to this embodiment. This figure shows another example of a neural manifold appearing in the state space determined by the brain state estimation process according to this embodiment. This figure shows an example of hyperalignment processing according to this embodiment. This is a schematic diagram illustrating the hyperalignment processing procedure according to this embodiment. This is a schematic diagram showing an example of a brain state according to this embodiment mapped onto a standard space. This is a schematic diagram showing another example of mapping brain states according to this embodiment onto a standard space. This is a schematic diagram showing an example of a discrete mapping of brain states according to this embodiment onto a standard space. This flowchart shows an example of a processing procedure for generating a standard space according to this embodiment. This flowchart shows an example of a hyperalignment processing procedure according to this embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings. Parts identical or corresponding to those shown in the drawings are denoted by the same reference numerals, and their descriptions will not be repeated.

[A. Brain state]
In this embodiment, a "brain state" is estimated as an example of a state of brain activity. The estimated "brain state" refers to the state of brain activity determined by evaluating the entire brain of the subject. The "brain state" is estimated based on information obtained by measuring brain activity such as EEG, MEG, and fMRI.

Multiple brain states can be defined in advance. Below, we will use "microstates," as reported in previous studies, as possible brain states. A "microstate" refers to the smallest unit representing a state of brain activity observed regardless of the subject. Microstates represent the overall state dynamics of the brain.

Previous studies have repeatedly reported that there are four microstates, regardless of the number of channels, preprocessing procedures, or subjects. Therefore, in this embodiment, the brain state is assumed to be one of these four microstates. These four microstates included in the brain state are sometimes referred to as msA, msB, msC, and msD. However, the number of states included in the brain state is not limited to four; any number of states can be set.

Each microstate can be considered a range of stable states (attractors). By considering the transition of "brain states" between microstates, it becomes possible to observe the transitions of brain states labeled by microstates and calculate the duration of each labeled brain state. Based on these features, applications as biomarkers for mental disorders are anticipated.

For the sake of explanation, the following will primarily describe an example of a process for estimating brain state using EEG. However, the technical scope of the present invention is not limited to processes for estimating brain state using EEG, but includes processes for estimating brain state using any information obtained by measuring brain activity.

[B. System Configuration Example]
Next, as an example of an application of the brain state estimation process according to this embodiment, we will describe an example configuration of a neurofeedback system for performing NF training.

Figure 1 is a schematic diagram showing an example configuration of a neurofeedback system 1 according to this embodiment. Referring to Figure 1, the neurofeedback system 1 estimates the brain state of subject 2 in real time using EEG measured via a cap 10 worn by subject 2. The neurofeedback system 1 guides subject 2 in an ideal direction by providing visual and/or auditory feedback to subject 2 according to the estimated brain state.

The neurofeedback system 1 includes, as its main components, a measurement circuit 40, a processing unit 100, a display 20, and a speaker 30.

The measurement circuit 40 corresponds to the measurement unit that measures the brainwaves of subject 2. More specifically, the measurement circuit 40 measures multiple channel components (electrical signals corresponding to EEG) as the brainwaves of subject 2 via a cap 10 containing multiple electrodes 12 attached to subject 2. The measurement circuit 40 includes, for example, a multiplexer 42, a noise filter 44, and an A/D (Analog to Digital) converter 46.

The multiplexer 42 sequentially selects multiple channels (electrodes 12) output from the cap 10 and electrically connects them to the noise filter 44. The noise filter 44 is a filter that allows only specific frequency components to pass through, removing noise components contained in the signal (electrical signal) appearing in the selected channel. For example, the passband can be set to 0.016 to 250 Hz.

The A/D converter 46 samples the electrical signal (analog signal) output from the noise filter 44 at predetermined intervals and outputs it as a digital signal.

The processing unit 100 estimates the brain state of subject 2 and outputs feedback corresponding to the estimated brain state.

The display 20 provides visual feedback to subject 2. The speaker 30 provides auditory feedback to subject 2.

Figure 2 is a schematic diagram showing the device configuration of the processing unit 100 included in the neurofeedback system 1 according to this embodiment. The processing unit 100 can typically employ a computer following a general-purpose architecture. Referring to Figure 2, the processing unit 100 includes, as its main components, a processor 102, a memory 104, an input unit 106, a network controller 108, a measurement interface 110, a display controller 112, an audio controller 114, and storage 120.

The processor 102 consists of arithmetic processing circuits such as a CPU (Central Processing Unit) and a GPU (Graphical Processing Unit), and executes the code contained in various programs stored in the storage 120 in a specified order to realize the various functions described later. The memory 104 consists of DRAM (Dynamic Random Access Memory) and SRAM (Static Random Access Memory), and holds the code of the programs executed by the processor 102 and various work data necessary for program execution.

The input unit 106 typically consists of a mouse or keyboard, and accepts user input.

The network controller 108 exchanges data with external devices (e.g., a data server on the cloud). The network controller 108 is composed of various communication components such as wired LAN (Local Area Network), wireless LAN, USB (Universal Serial Bus), and Bluetooth®.

The measurement interface 110 receives the EEG (digital signal) output from the measurement circuit 40.

The display controller 112 displays an image on the display 20 according to the processing result of the processor 102. The audio controller 114 outputs sound from the speaker 30 according to the processing result of the processor 102. Note that a controller that outputs both image and sound simultaneously may also be used.

The storage 120 typically consists of a hard disk or SSD (Solid State Drive) and holds various programs executed by the processor 102, various data necessary for processing, and setting values. More specifically, the storage 120 includes the OS (Operating System) 122, the measurement program 124, and the NF program 126.

The measurement program 124 includes code to cause the processor 102 to perform processing and visualization processing for estimating the brain state from the subject 2's EEG. The measurement program 124 may also include code to cause the processor 102 to perform processing for providing visual and/or auditory feedback to subject 2.

[C. Estimation of brain state]
Next, we will explain an example of processing for estimating brain state.

Figure 3 illustrates an example of processing for estimating brain state according to this embodiment. Figure 3(A) shows an example of the raw EEG signal 50 (time waveform) measured through the cap 10 worn by subject 2. Figure 3(A) shows, as an example, an example of the raw signal 50 (32 channels) measured at each of the 32 electrodes 12. That is, the cap 10 includes 32 electrodes 12 corresponding to regions of subject 2's head.

Figure 3(B) shows an example of vectorizing the set of time waveforms (hereinafter also referred to as "epoch 52") that occur in the interval (100 ms in this example) set in the raw EEG signal 50, in the spatial direction. The channel space 60 shown in Figure 3(B) has each channel of the measured EEG as its dimension. The vector 62 shown in Figure 3(B) corresponds to the epoch 52 shown in Figure 6(A). The vector 62 represents the brain state set with respect to the spatiotemporal dimension. Each of the vectors 62 represents the brain state at the moment corresponding to epoch 52.

The components of each dimension constituting vector 62 may be the average value at epoch 52, or they may be measured values at any point in time within epoch 52.

Figure 4 shows an example of a vector 62 representing a brain state according to this embodiment. Referring to Figure 4, the vector 62 generated from each epoch 52 of the EEG can be defined on the channel space 60.

By clustering a large number of vectors 62 defined on the channel space 60, one or more clusters 64 can be determined. Each of the clusters 64 determined using a sufficient number of vectors 62 represents a specific brain state.

In this embodiment, four microstates, msA, msB, msC, and msD, are adopted as brain states, allowing for the determination of four corresponding clusters, 64A, 64B, 64C, and 64D. The centroid coordinates of the clusters 64A, 64B, 64C, and 64D, corresponding to each of the microstates msA, msB, msC, and msD, represent 32-dimensional vectors. Known clustering methods such as modified k-means clustering or hierarchical clustering can be used.

Vectors 62 belonging to the same cluster 64 have similar characteristics. These common characteristics among vectors 62 belonging to the same cluster 64 are also called eigenspatial patterns. As will be described later, the eigenspatial pattern 74 corresponds to a mapping of the signal intensity of each channel of the EEG according to the position of the corresponding electrode 12 (i.e., the brain region where the information is collected).

Figure 5 illustrates the brain activity patterns and templates for estimating brain states according to this embodiment. Referring to Figure 5, the brain activity pattern 70 is a matrix representing the state of brain activity for each epoch 52. In the example shown in Figure 5, a 7x7 matrix is used, where the intensity of each matrix element indicates the magnitude of the set value (within a range of ±5).

The measured values of each channel of the EEG are mapped to specific positions (row x column) within a matrix representing the brain activity pattern 70, depending on the position of the corresponding electrode 12. For example, the electrodes 12 of the cap 10 are assumed to be positioned at the following locations: Fp1, Fp2, Fz, F3, F4, F7, F8, F9, F10, FC1, FC2, FC5, FC6, Cz, C3, C4, T7, T8, CP1, CP2, CP5, CP6, Pz, P3, P4, P7, P8, P9, P10, Oz, O1, and O2. The reference electrode and ground electrode may also be positioned at the FCz and Fpz locations, respectively.

Since the number of channels in the measured EEG is 32, there will be matrix elements that are not mapped to channels. For matrix elements that are not mapped to channels, any value (for example, "0") may be set.

Brain activity patterns 70 may be generated at intervals of epoch 52 (e.g., 100 ms). Spatial correlations (similarity) are calculated between the generated brain activity patterns 70 and each of the pre-prepared templates 72A, 72B, 72C, and 72D (hereinafter sometimes collectively referred to as "templates 72"). The microstate corresponding to the template 72 with the largest calculated spatial correlation is determined as the brain state represented by the brain activity pattern 70.

Template 72 is a type of spatial filter that displays a peak dataset of GFP (global field power) representing the strength of the brain's electric field at each instant. Templates 72A, 72B, 72C, and 72D may be determined, for example, based on the centroid coordinates (32-dimensional vectors) of clusters 64A, 64B, 64C, and 64D defined on the channel space 60. That is, template 72 can be determined by mapping the centroid coordinates of cluster 64 to a matrix in a similar manner to that used for brain activity patterns 70.

Figure 5 shows the eigenspatial patterns 74A, 74B, 74C, and 74D, corresponding to templates 72A, 72B, 72C, and 72D. These eigenspatial patterns 74A, 74B, 74C, and 74D were generated by spatial interpolation of templates 72A, 72B, 72C, and 72D, and represent the continuous distribution of brain activity in each microstate.

[D. Presentation of Brain State]
Next, we will describe an example that presents an estimated brain state.

Figure 6 is a schematic diagram showing an example of a user interface screen provided by the processing unit 100 according to this embodiment.

Referring to Figure 6(A), the processing unit 100 displays a brain state object 200 containing four brain states (microstates) and a current state object 210 indicating the estimated current brain state on the display 20. As shown in Figure 6(A), the processing unit 100 may present the subject 2 with a display (brain state object 200) containing multiple regions corresponding to multiple brain states. By providing a user interface screen like the one shown in Figure 6(A), the subject 2 can recognize at a glance which state their current brain state is.

Referring to Figure 6(B), the processing unit 100 may further display a target object 220 indicating the target brain state. As shown in Figure 6(B), the processing unit 100 may display the estimated brain state of subject 2 (current brain state) in a first display mode (e.g., a circle as shown in Figure 6(B)) in association with multiple regions indicated by the brain state object 200, and display the target brain state in a second display mode (e.g., a rectangle as shown in Figure 6(B)). By providing a user interface screen like the one shown in Figure 6(B), subject 2 can recognize at a glance not only what their current brain state is, but also what brain state they should be in.

Referring to Figure 6(C), if the processing unit 100 cannot determine the current brain state, the current state object 210 will not be displayed.

Furthermore, the fact that the brain state cannot be determined may be visually communicated to Subject 2 using text or objects, or auditorily communicated to Subject 2 using voice messages or sound effects.

In this case as well, subject 2 attempts to focus their attention on achieving the target brain state.

[E. State Space and Neuronal Manifolds]
While the microstate method described above is very effective when comparing subjects, it is preferable to have a mechanism that can track the transitions in the overall state dynamics of the brain in order to observe the brain state of a specific subject.

Below, we will describe a state space that can represent the continuous transitions in the state dynamics of the entire brain.

In the state space according to this embodiment, instead of jumping between brain states, a smooth trajectory of state dynamics can be represented. Such a state space capable of representing a smooth trajectory is sometimes referred to as a neural manifold.

The inventors of this application investigated the eigendary dimensions contained in the EEG in channel space 60. More specifically, they used principal component analysis (PCA) to investigate the eigendary dimensions contained in the 32-channel EEG.

Figure 7 shows an example of the evaluation results for the intrinsic dimensions included in the EEG. Figure 7(A) shows the evaluation results when using the raw EEG signal, and Figure 7(B) shows the evaluation results when normalized on a channel-by-channel basis.

Figures 7(A) and 7(B) suggest that a dimension of "3" is sufficient to adequately represent the dynamics of the EEG. In other words, using a three-dimensional space allows for a sufficiently intuitive representation of global state dynamics. It is also possible to use a two-dimensional space, obtained by projecting the three-dimensional space onto a specific plane, as a representation.

In the brain state estimation process according to this embodiment, a dimensionality reduction method is applied to compress the channel space 60 of the EEG (32 dimensions if there are 32 channels) into a sufficiently low-dimensional space. Any method such as principal component analysis (PCA) or independent component analysis (ICA) can be used as the dimensionality reduction method, but in this embodiment, multidimensional scaling (MDS) is employed. According to multidimensional scaling, the space after dimensionality reduction (typically a simple sphere) becomes a smooth surface, and in this space, the state becomes geometrically differentiable.

Figure 8 shows an example of a brain activity pattern 76 used to determine the state space in the brain state estimation process according to this embodiment. Figure 9 shows an example of a spatial correlation matrix 80 used to determine the state space in the brain state estimation process according to this embodiment.

Referring to Figure 8, multiple brain activity patterns 76, which represent the state of the subject's brain activity, are acquired over a predetermined period of time. Each brain activity pattern 76 corresponds to a sampling point for the raw EEG signal 50, and represents the brain state at each sampling point. Similar to brain activity pattern 70, brain activity patterns 76 are maps of the measured values of each channel of the EEG to their corresponding positions. The predetermined number can be set, for example, to 500 points (1 second at a sampling rate of 500 Hz).

For example, the processing unit 100 samples the measured values of each channel of the EEG and generates a brain activity pattern 76 by mapping the measured values of each channel obtained through sampling.

Next, a spatial correlation matrix 80 is generated by calculating the spatial correlation for each combination of the multiple brain activity patterns 76. That is, spatial correlations are calculated for all combinations of a predetermined number of brain activity patterns 76. By placing the calculated spatial correlations in their corresponding positions, a spatial correlation matrix 80, as shown in Figure 8, can be calculated. The spatial correlation matrix 80 indicates the spatial similarity between the brain activity patterns 76 (each value ranges from -1 to 1).

By focusing on features included in the spatial correlation matrix 80 (e.g., nonlinear waveforms and ripples), the time-series dynamics as trajectories can determine the space as a neural manifold. Specifically, the spatial correlation matrix 80 is transformed into a distance matrix using multidimensional scaling, and this distance matrix is subjected to singular value decomposition (SVD) to calculate coordinates of a predetermined number of dimensions for each of the multiple brain activity patterns 76. As an example, the calculated coordinates may be three-dimensional (XYZ coordinates).

The sets of coordinates corresponding to each of the multiple brain activity patterns 76 form the state space 90 and the neural manifold 92 that appear in the state space 90.

Figure 10 shows an example of a state space calculated from a predetermined number of brain activity patterns 76 in the brain state estimation process according to this embodiment. Referring to Figure 10, the state space 90 plots coordinates (500 points) corresponding to each of the brain activity patterns 76, and can be considered temporally continuous.

Thus, the state space 90 according to this embodiment can also be defined as a "spatial similarity space," and in the state space 90, trajectories that are continuous in time are also continuous in space and time.

By plotting as many brain activity patterns 76 as possible onto the state space 90, a spherical neural manifold emerges. That is, the brain activity patterns 76 can be considered to exist at one of the coordinates on the sphere determined by the state space 90, and they move along the sphere in time.

Figure 11 shows an example of a neural manifold 92 appearing in the state space 90 determined by the brain state estimation process according to this embodiment. Referring to Figure 11, the brain activity patterns 76 within the neural manifold 92 are distributed such that their norm is approximately 1. Therefore, a sphere (or spherical surface) with radius 1 can be determined as the neural manifold 92. That is, the centroid (center) of the entire set of coordinates constituting the neural manifold 92 corresponds to the origin of the state space 90.

Furthermore, by determining the brain state (microstate) corresponding to the template 72 that shows the greatest spatial correlation with the brain activity pattern 76 as the brain state in that brain activity pattern 76 (a winner-take-all method), the regions corresponding to microstates msA, msB, msC, and msD can be determined for the neural manifold 92.

In other words, the brain state (microstate) corresponding to each brain activity pattern is determined based on the magnitude of the spatial correlation between each of the multiple brain activity patterns 76 and the multiple templates 72 that each represent a predetermined set of brain states. At this time, for a single brain activity pattern 76, the spatial correlation (i.e., four values) for the microstates msA, msB, msC, and msD is calculated, and the brain state (microstate) with the largest spatial correlation may be determined as the brain state for that brain activity pattern 76.

Furthermore, as shown in Figure 11, the coordinates (plots) corresponding to multiple brain activity patterns and the corresponding brain states (microstates) may be represented on the same coordinate system.

The template 72 can be determined based on the vector corresponding to the centroid coordinates of the cluster 64. For example, since polarity is ignored in the modified k-means method, it is preferable to prepare two sets of templates 72 with reversed polarity. For instance, one set of templates 72 with the frontal region as positive and another set of templates 72 with the frontal region as negative can be prepared. When determining regions in the neural manifold 92, two sets of templates 72 are used. That is, the template 72 includes multiple pairs of templates with reversed polarity.

Here, we will explain the center point of the neural manifold 92. Considering a set of microstates msA and msA' with opposite polarities, the coordinates corresponding to microstate msA and microstate msA' are arranged point-symmetrically with respect to the center point of the neural manifold. In other words, the average value of the coordinates corresponding to microstate msA and microstate msA' represents the center point of the neural manifold 92. The same applies to other microstates.

In this way, for each of a pair of templates with opposite polarities, the average value of the coordinates corresponding to one template and the coordinates corresponding to the other template is determined as the center of the neural manifold 92.

Figure 12 shows another example of a neural manifold 92 appearing in the state space 90 determined by the brain state estimation process according to this embodiment. Figures 12(A) to (C) show the state space 90 drawn from different viewpoints.

The coordinates corresponding to the templates representing each microstate exist on the spherical neural manifold 92. Figures 12(A) to (C) show the state space 90 containing the coordinates corresponding to the four types of microstates.

Here, by utilizing the property that the centroid (center) of the entire set of coordinates constituting the neural manifold 92 corresponds to the origin of the state space 90, it is also possible to heuristically determine or estimate new microstates. More specifically, by focusing on any combination of templates, the center (average) of the coordinates corresponding to each template can become a candidate for the coordinates corresponding to the new microstate.

As an example, Figure 12(C) shows a case where a landmark 78, which could be a candidate for a new microstate, is placed at the center of the coordinates corresponding to microstate msA and microstate msB. In this case, the paired landmark 78 (i.e., the coordinates corresponding to the template with reversed polarity) is placed at a position symmetrical with respect to the origin of the state space 90.

Thus, the coordinates corresponding to a new microstate may be determined from the coordinates corresponding to any two pre-defined microstates. That is, the coordinates corresponding to a new template may be determined based on the coordinates corresponding to one template and the coordinates corresponding to another template among multiple templates.

By heuristically determining new templates, the number of pre-defined templates can be increased as needed. In other words, the required number of microstates can be created retrospectively.

As described above, by using the state space 90 according to this embodiment, the brain activity pattern 76 calculated from the EEG at each moment moves continuously on the spherical neural manifold 92, which becomes the trajectory of the state dynamics. Note that if the sampling rate is sufficiently high (the sampling period is short), a continuous trajectory will be observed. However, if the sampling rate is low, a jumping trajectory may occur; therefore, it is preferable to set the sampling rate appropriately.

[F. Standard Space and Hyperalignment]
The result of mapping the EEG measured by a subject to state space is unique to that subject. Therefore, when statistically evaluating state dynamics, it is necessary to determine the coordinates when mapped to a space standardized across subjects (hereinafter also referred to as the "standard space"). The process of arranging the result of mapping the EEG measured by a subject to state space into the standard space will be referred to as "hyperalignment" below.

Figure 13 shows an example of hyperalignment processing according to this embodiment. Referring to Figure 13, if the EEG is mapped to the state space 90 for each subject, even the same brain state may be plotted at different coordinates.

Therefore, hyperalignment is used to realize a mapping to a standard space 94 that standardizes the unique characteristics of each subject.

Figure 14 is a schematic diagram showing the hyperalignment processing procedure according to this embodiment.

Referring to Figure 14(A), first, a basic state space (neural manifold 92) used for hyperalignment is generated. In generating the neural manifold 92, multiple brain activity patterns 76 representing the brain activity states of multiple subjects are acquired. The brain activity patterns 76 are obtained using sampled EEG data measured from the subjects.

Generating the neural manifold 92 requires several minutes of EEG (resting state) data from several subjects. For example, measuring five subjects for one minute at a sampling rate of 100 Hz can collect a total of 30,000 brain activity patterns 76.

Then, a spatial correlation matrix is generated by calculating the spatial correlation for each combination included in the multiple brain activity patterns 76 obtained. Furthermore, the spatial correlation matrix is transformed into a distance matrix using multidimensional scaling, and the coordinate sets corresponding to the elements included in the spatial correlation matrix are calculated by singular value decomposition of the distance matrix. The calculated coordinate sets constitute the neural manifold 92. In this way, a neural manifold 92 is generated from the numerous brain activity patterns 76 collected using multidimensional scaling and singular value decomposition. The neural manifold 92 is a set of coordinates plotted on a sphere with radius 1.

Next, a predetermined number of grids 96 are extracted (sampled) from the generated neural manifold 92 according to a predetermined rule. That is, brain activity patterns 76 corresponding to the predetermined number of coordinates (grids 96) extracted from the calculated coordinate group (neural manifold 92) are extracted as part of the standard state pattern group.

The sampled grid 96 serves as a reference (landmark) for determining coordinates in hyperalignment. While a larger number of grid 96 increases the accuracy of hyperalignment, it also requires more processing time. Therefore, the number of sampled grid 96 is determined based on a balance between accuracy and processing time. For example, 50 grid 96 points may be sampled. It is preferable that the grid 96 be sampled spatially evenly across the neural manifold 92 (sphere).

Next, for the brain activity patterns 76 (e.g., 50 points) corresponding to the sampled grid 96 and the templates 72 (e.g., a set of two templates 72 with reversed polarity: 8 points), each point is plotted in the state space using multidimensional scaling and singular value decomposition. That is, a spatial correlation matrix is generated by calculating the spatial correlation for each combination included in the standard state pattern group. Furthermore, the spatial correlation matrix is transformed into a distance matrix using multidimensional scaling, and the coordinate sets corresponding to the elements included in the spatial correlation matrix are calculated by singular value decomposition of this distance matrix. The calculated coordinate sets constitute the standard space 94.

In standard space 94, the coordinates corresponding to grid 96 are not necessarily placed at the same coordinates as the original. However, the center point of the neural manifold 92, determined from each coordinate corresponding to template 72, is considered to remain substantially unchanged. Therefore, the center point of the neural manifold can be estimated based on each coordinate corresponding to template 72.

The brain activity patterns 76 and templates 72 corresponding to the grid 96 described above, plotted in state space, are used as the standard space 94. The set of brain activity patterns 76 and templates 72 used to generate the standard space 94 (for example, a total of 58 points) corresponds to a standard state pattern group representing a standard state. That is, the standard state pattern group includes multiple brain activity patterns 76 representing the brain activity states of one or more subjects, and multiple templates 72 representing predetermined brain states (microstates).

Through the above process, a standard space 94 is prepared, consisting of a set of standard state patterns and a set of coordinates calculated from the standard state patterns. Using the standard space 94 and the standard state patterns, the coordinates in the state space (standard space 94) corresponding to the EEG measured by the subject are sequentially determined.

Referring to Figure 14(B), a new brain activity pattern 98, sampled from EEG data measured from the subject, is added to the standard state pattern group. That is, a brain activity pattern 98 representing the subject's brain activity state is obtained and added to the standard state pattern group.

Next, for the set to which the brain activity patterns 98 have been added, each point is plotted in the state space using multidimensional scaling and singular value decomposition to generate a modified standard space 94A. More specifically, a spatial correlation matrix is generated by calculating the spatial correlation between each combination included in the standard state pattern group and the brain activity patterns 98 of the target subjects. Furthermore, the spatial correlation matrix is transformed into a distance matrix using multidimensional scaling, and the coordinate sets corresponding to the elements included in the spatial correlation matrix are calculated by performing singular value decomposition on the distance matrix. The calculated coordinate sets constitute the modified standard space 94A.

Here, if the number of standard state patterns is sufficiently large compared to the number of additional brain activity patterns 98 (e.g., one), the modified standard space 94A will only shift slightly from the standard space 94. That is, the relative positional relationships (e.g., distances between coordinates) of the coordinate groups corresponding to the standard state pattern groups are substantially maintained.

Therefore, a precise transformation can be performed based on the positional relationship between the standard space 94 and the modified standard space 94A to which the brain activity pattern 98 has been added. That is, the coordinates corresponding to the brain activity pattern 98 in the coordinate group constituting the standard space 94 are estimated based on the relative positional relationship of the coordinates corresponding to the brain activity pattern 98 in the coordinate group constituting the modified standard space 94A. In this way, the coordinates in the modified standard space 94A to which the brain activity pattern 98 has been added can be converted back to the coordinates in the original standard space 94.

As an example, in the modified standard space 94A, multiple coordinates near the coordinate corresponding to the brain activity pattern 98 are extracted. Alternatively, all coordinates within the modified standard space 94A may be extracted. The relative positional relationship between the extracted neighboring coordinates and the coordinate corresponding to the brain activity pattern 98 is determined. Then, in the standard space 94, the coordinate corresponding to each of the extracted neighboring coordinates is identified, and by applying the relative positional relationship to the identified coordinates, the coordinate corresponding to the brain activity pattern 98 in the standard space 94 is determined.

As another example, a mapping transformation formula is determined for the standard state pattern set between the standard space 94 and the modified standard space 94A. Then, using the determined mapping transformation formula, the coordinates corresponding to the brain activity pattern 98 in the modified standard space 94A are mapped to the standard space 94, thereby determining the coordinates corresponding to the brain activity pattern 98 in the standard space 94.

In standard space 94, the coordinates corresponding to the brain activity pattern 98 represent the standardized coordinates of the brain activity pattern 98.

Each time a brain activity pattern 98 is acquired by sampling EEG measured from the subject, the process described above is repeatedly executed. That is, by repeatedly performing a series of processes—acquiring the brain activity pattern 98, calculating the modified standard space 94A, and estimating the coordinates corresponding to the brain activity pattern 98—the trajectory of the subject's state dynamics can be represented in the standard space 94, as shown in Figure 14(C). In other words, the sequentially estimated coordinates are displayed as the trajectory.

According to this embodiment of hyperalignment, the trajectory of state dynamics can be represented on a common standard space 94 among subjects, allowing for more accurate comparisons and statistical processing between subjects.

Furthermore, according to this embodiment of hyperalignment, the coordinates in the standard space 94 can be determined each time a new brain activity pattern 98 (EEG sampling data) is acquired, allowing for the real-time representation of the subject's brain state. Presenting this brain state to the subject in real time can also make NF training more efficient.

Furthermore, according to this embodiment of hyperalignment, the trajectory of state dynamics indicating the subject's brain state can be confirmed, making it easy to verify the effects of NF training on the subject.

[G. Presentation to the user]
Next, we will illustrate several methods for presenting the coordinates and trajectory calculated by this embodiment to the user.

As described above, hyperalignment according to this embodiment allows the EEG measured from the subject to be represented as coordinates and trajectories (temporal changes in coordinates) in standard space 94. Standard space 94 becomes a spherical neural manifold, and the state dynamics move sequentially along a sphere with radius 1. Therefore, the coordinates in standard space 94 corresponding to the subject's brain activity pattern 98 may be displayed in a two-dimensional or three-dimensional coordinate system.

Figure 15 is a schematic diagram showing an example of mapping brain states to a standard space according to this embodiment. Referring to Figure 15, the movement of state dynamics may be represented by sequentially displaying the coordinates corresponding to each sampling point on a sphere representing the standard space.

Furthermore, the viewing direction of the sphere representing standard space may be arbitrarily changed. While the example shown in Figure 15 is represented in three dimensions, a two-dimensional representation is also possible by positioning the viewpoint on a specific axis.

Figure 16 is a schematic diagram showing another example of mapping brain states according to this embodiment onto a standard space.

Figure 16(A) shows an example of a three-dimensional representation. As shown in Figure 16(A), a sphere representing the standard space may be represented as a mesh, and the coordinates corresponding to the subject's brain state and template may be represented in any way.

A spherical projector may be used to visualize the three-dimensional representation shown in Figure 16(A). That is, the estimated coordinates may be displayed using a spherical projector. A spherical projector is a spherical light-emitting body that can arbitrarily change the color of any region on its surface. Using such a spherical projector allows for a clearer presentation of the state dynamics.

Figure 16(B) shows an example of a two-dimensional representation. As shown in Figure 16(B), by projecting the standard space onto a specific two-dimensional coordinate system, the movement of brain states in the two-dimensional coordinate system of interest can be represented. The regions of each microstate may also be displayed.

Figure 17 is a schematic diagram showing an example of a presentation in which brain states according to this embodiment are discretely mapped onto a standard space. Referring to Figure 17, state dynamics moving continuously across the standard space may be represented, or the regions corresponding to brain states may be discretely and appropriately selected, and transitions between regions may be represented. Adopting a discrete representation as shown in Figure 17 is suitable for NF training and other applications where a target brain state is specified.

The presentation examples shown in Figures 15-17 are not the only ways to represent the overall state dynamics and brain states; they can be expressed in any method and form.

[H. Processing Procedure]
Next, an example of the processing procedure for estimating brain states according to this embodiment will be described.

Figure 18 is a flowchart showing an example of a processing procedure for generating a standard space according to this embodiment. Figure 19 is a flowchart showing an example of a hyperalignment processing procedure according to this embodiment. Each step shown in Figures 18 and 19 may be implemented, for example, by the processor 102 of the processing unit 100 executing the measurement program 124 and the NF program 126.

Referring to Figure 18, the processing unit 100 acquires raw EEG signals measured from multiple subjects (step S100). Then, the processing unit 100 generates multiple brain activity patterns from each raw EEG signal at a predetermined sampling rate (step S102). Furthermore, the processing unit 100 calculates the spatial correlation for each combination included in the generated brain activity patterns (step S104), and generates a spatial correlation matrix from the calculated spatial correlations (step S106). The processing unit 100 calculates a set of coordinates corresponding to the elements included in the generated spatial correlation matrix using multidimensional scaling and singular value decomposition (step S108). The calculated set of coordinates becomes the neural manifold 92.

Next, the processing unit 100 samples a predetermined number of grids 96 from the generated neural manifold 92 (step S110). Then, the processing unit 100 generates a set of standard state patterns consisting of brain activity patterns corresponding to the sampled grids 96 and templates 72 (step S112). That is, brain activity patterns corresponding to the predetermined number of sampled grids 96 are extracted as part of the set of standard state patterns.

The processing unit 100 then calculates the spatial correlation for each combination included in the standard state pattern group (step S114), and generates a spatial correlation matrix from the calculated spatial correlations (step S116). The processing unit 100 calculates the coordinate groups corresponding to the elements included in the generated spatial correlation matrix using multidimensional scaling and singular value decomposition (step S118). The calculated coordinate groups become the standard space 94. Finally, the processing unit 100 stores the standard space 94 (coordinate groups) and the standard state pattern group (step S120). The processing then ends.

Referring to Figure 19, the processing unit 100 acquires a new brain activity pattern 98 by sampling EEG measured from the subject (step S200), and adds the acquired brain activity pattern 98 to the standard state pattern group (step S202). Then, the processing unit 100 calculates the spatial correlation for each combination included in the added brain activity pattern (step S204), and generates a spatial correlation matrix from the calculated spatial correlations (step S206). The processing unit 100 calculates a set of coordinates corresponding to the elements included in the generated spatial correlation matrix using multidimensional scaling and singular value decomposition (step S208). The calculated set of coordinates becomes the modified standard space 94A.

Next, the processing unit 100 identifies the coordinates corresponding to the new brain activity pattern 98 in the modified standard space 94A and their positional relationship (relative positional relationship) with other coordinates (step S210). Then, based on the identified positional relationship, the processing unit 100 estimates the coordinates corresponding to the new brain activity pattern 98 in the standard space 94 (step S212).

The processing unit 100 then outputs the coordinates of the standard space 94 corresponding to the new brain activity pattern 98 (step S214). A specific example of the method for outputting coordinate information will be described later.

The processing unit 100 determines whether or not the observation of the subject is continuing (step S216). If the observation of the subject is continuing (YES in step S216), the processing from step S200 onward is repeated.

If the subject's observation is not continued (NO in step S216), the process terminates.

[I. Others]
In the explanation above, the case using EEG was illustrated as an example, but it is also applicable to MEG, fMRI, etc. Even when using MEG or fMRI, the same process can be performed by collecting data from multiple subjects and multiple samples and generating a spatial correlation matrix.

[J. Advantages]
According to this embodiment, by mapping the subject's brain activity pattern to a relatively low-dimensional state space and representing it as state dynamics, it is possible to visualize not discrete transitions between four brain states (microstates), but rather more intricate, continuous transitions and movements of brain states.

Furthermore, according to this embodiment, the brain activity patterns of subjects can be mapped onto a standard space. The standard space is a space standardized (normalized) so that no differences occur between subjects. By evaluating the coordinates projected onto the standard space and their movement (trajectory), evaluations can be performed on multiple subjects using common criteria.

Furthermore, according to this embodiment, coordinates projected onto standard space can be represented in various forms. This allows for observation and evaluation of the subject's brain state from various perspectives. It also enables the provision of easily understandable information to subjects in NF training and other applications.

The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is indicated by the claims rather than by the description of the embodiments above, and all modifications within the meaning and scope equivalent to the claims are intended to be included.

1 Neurofeedback system, 2 Subject, 10 Cap, 12 Electrodes, 20 Display, 30 Speaker, 40 Measurement circuit, 42 Multiplexer, 44 Noise filter, 46 A/D converter, 50 Raw signal, 52 Epoch, 60 Channel space, 62 Vector, 64, 64A, 64B, 64C, 64D Cluster, 70, 76, 98 Brain activity pattern, 72, 72A, 72B, 72C, 72D Template, 74, 74A, 74B, 74C, 74D Eigenspace pattern, 80 Spatial correlation matrix, 90 State space, 92 Neural manifold, 94 Standard space, 94A Modified standard space, 96 Grid, 100 Processing unit, 102 Processor, 104 Memory, 106 Input unit, 108 Network controller, 110 Measurement interface, 112 Display controller, 114 Voice controller, 120 storage, 124 measurement program, 126 program, 200 brain state objects, 210 current state object, 220 target object, msA, msB, msC, msD microstates.

Claims (11)

The steps include acquiring brain activity patterns indicating the state of the subject's brain activity multiple times at predetermined intervals over a predetermined period of time ,
The steps include: generating a spatial correlation matrix by calculating spatial correlations for all combinations of the multiple brain activity patterns obtained ;
The steps include: converting the spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating coordinates of a predetermined number of dimensions for each of the multiple brain activity patterns by singular value decomposition of the distance matrix;
A processing method comprising the step of determining a brain state corresponding to each brain activity pattern based on the magnitude of the spatial correlation between each of the aforementioned plurality of brain activity patterns and a plurality of templates that each represent a predetermined plurality of brain states. The processing method according to claim 1, further comprising the step of representing the coordinates corresponding to each of the multiple brain activity patterns and the corresponding brain states on the same coordinate system. The aforementioned multiple templates include multiple pairs of templates with reversed polarity,
The processing method according to claim 1, wherein for each of the pair of templates, the average value of the coordinates corresponding to one template and the coordinates corresponding to the other template is determined as the center of the coordinates corresponding to each of the plurality of brain activity patterns. The processing method according to claim 1, further comprising the step of determining coordinates corresponding to a new template based on the coordinates corresponding to a first template among the plurality of templates and the coordinates corresponding to a second template among the plurality of templates. The process includes a step of acquiring a set of standard state patterns, which include multiple brain activity patterns representing the brain activity states of one or more subjects, and multiple templates representing predetermined brain states, wherein each brain activity pattern is acquired at predetermined intervals .
The steps include generating a first spatial correlation matrix by calculating spatial correlations for all combinations included in the aforementioned standard state pattern group,
The steps include: converting the first spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating a first set of coordinates corresponding to the elements included in the first spatial correlation matrix by singular value decomposition of the distance matrix;
The method includes a step of acquiring a first brain activity pattern that indicates the state of the subject's brain activity, wherein each first brain activity pattern is acquired at predetermined intervals .
The steps include generating a second spatial correlation matrix by calculating spatial correlations for all combinations included in the standard state pattern group and the first brain activity pattern,
The steps include: converting the second spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating a second set of coordinates corresponding to the elements included in the second spatial correlation matrix by singular value decomposition of the distance matrix;
A processing method comprising the steps of: estimating coordinates corresponding to the first brain activity pattern in the first coordinate group based on the relative positional relationship of coordinates corresponding to the first brain activity pattern in the second coordinate group. The processing method according to claim 5, further comprising the step of displaying the coordinates corresponding to the first brain activity pattern in a two-dimensional or three-dimensional coordinate system. The method further comprises repeating the steps of acquiring the first brain activity pattern, generating the second spatial correlation matrix, calculating the second set of coordinates, and estimating the coordinates.
The processing method according to claim 6, wherein the display step includes the step of displaying the sequentially estimated coordinates as a trajectory. The processing method according to claim 7, wherein the display step includes the step of displaying the estimated coordinates using a spherical projector. The process includes a step of acquiring multiple brain activity patterns that show the state of brain activity of multiple subjects, wherein each brain activity pattern is acquired at predetermined intervals .
The steps include: generating a third spatial correlation matrix by calculating spatial correlations for all combinations included in the multiple brain activity patterns obtained;
The steps include: converting the third spatial correlation matrix into a distance matrix using multidimensional scaling, and calculating a third set of coordinates corresponding to the elements included in the third spatial correlation matrix by singular value decomposition of the distance matrix;
The processing method according to claim 5, further comprising the step of extracting brain activity patterns corresponding to a predetermined number of coordinates extracted from the third group of coordinates as the plurality of brain activity patterns of the standard state pattern group. The processing method according to claim 9, wherein the predetermined number of coordinates are extracted from the third group of coordinates according to a predetermined rule. A program for causing a computer to execute the processing method described in any one of claims 1 to 10.