JP7583373B2 - Biological information analysis device, biological information analysis system, and biological information analysis program - Google Patents

Description

Embodiments of the present invention relate to a biometric information analysis device, a biometric information analysis system, a biometric information analysis program, and a biometric information analysis method.

In psychological counseling, it is necessary to understand the mental state of the subject. For this reason, for example, each time a counselor holds a session (interview) with a subject, the subject is asked to answer a questionnaire about their mental state, and the answers are analyzed to infer the subject's mental state.

JP 2018-153244 A

Here, in the above-mentioned settings, there is a need to grasp the mental state more accurately due to factors such as an increase in the number of counseling sessions in recent years and the diversification of the attributes of those receiving counseling.

The present invention provides a biometric information analysis device, a biometric information analysis system, a biometric information analysis program, and a biometric information analysis method that can accurately grasp the mental state of a subject.

In order to solve the above-mentioned problems and achieve the objective, the present invention provides a bioinformation analysis device comprising an acquisition unit that acquires bioinformation including autonomic nerve data of a subject and behavioral record information in which multiple behaviors of the subject are recorded, and a calculation unit that calculates mental state information for each behavior of the subject based on the bioinformation and the behavioral record information.

The present invention also provides a bioinformation analysis system including a server device and a display terminal, wherein the server device includes an acquisition unit that acquires bioinformation including autonomic nerve data of a subject and behavioral record information in which multiple behaviors of the subject are recorded, a calculation unit that calculates mental state information for each behavior of the subject based on the bioinformation and the behavioral record information, and an output control unit that transmits the mental state information for each behavior of the subject to the display terminal, and the display terminal includes a display control unit that receives the mental state information for each behavior of the subject transmitted from the server device and displays the received mental state information for each behavior of the subject.

The present invention also relates to a bioinformation analysis program that acquires bioinformation including autonomic nerve data of a subject and behavioral record information in which multiple behaviors of the subject are recorded, and causes a computer to execute various processes to calculate mental state information for each behavior of the subject based on the bioinformation and the behavioral record information.

The present invention also relates to a bioinformation analysis method, which includes acquiring bioinformation including autonomic nerve data of a subject and behavioral record information in which multiple behaviors of the subject are recorded, and calculating mental state information for each behavior of the subject based on the bioinformation and the behavioral record information.

The present invention provides a biometric information analysis device, a biometric information analysis system, a biometric information analysis program, and a biometric information analysis method that can accurately grasp the mental state of a subject.

FIG. 1 is a diagram illustrating an example of a schematic configuration of an information processing apparatus according to an embodiment. FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing apparatus according to the embodiment. FIG. 3 is a flowchart illustrating an example of the operation of the information processing device according to the embodiment. FIG. 4 is a diagram for explaining the process of the acquisition unit. FIG. 5 is a diagram for explaining the process of the acquisition unit. FIG. 6 is a diagram for explaining the process of the acquisition unit. FIG. 7 is a diagram for explaining the process of the acquisition unit. FIG. 8 is a diagram for explaining the process of the acquisition unit. FIG. 9 is a diagram for explaining the process of the acquisition unit. FIG. 10 is a diagram for explaining the process of the calculation unit. FIG. 11 is a diagram for explaining the process of the generation unit. FIG. 12 is a diagram for explaining the process of the generation unit. FIG. 13 is a diagram for explaining the process of the generation unit. FIG. 14 is a diagram for explaining the process of the output control unit. FIG. 15 is a diagram for explaining the process of the output control unit. FIG. 16 is a diagram for explaining the process of the output control unit. FIG. 17 is a diagram for explaining the process of the output control unit. FIG. 18 is a diagram for explaining the process of the output control unit. FIG. 19 is a diagram for explaining the process of the output control unit. FIG. 20 is a diagram illustrating an example of a schematic configuration of an information processing apparatus according to the second embodiment. FIG. 21 is a diagram showing processes during learning and operation performed by the information processing apparatus according to the second embodiment. FIG. 22 is a flowchart illustrating an example of the operation of the information processing device according to the second embodiment. FIG. 23 is a diagram illustrating an example of a schematic configuration of a system according to an embodiment. FIG. 24 is a diagram for explaining trained models related to each example and comparative example.

Below, the biometric analysis device, biometric analysis system, biometric analysis program, and biometric analysis method according to the embodiments will be described with reference to the drawings. Note that the following embodiments are not limited to the following description. Furthermore, each embodiment can be combined with other embodiments or conventional technologies to the extent that no inconsistencies arise in the processing content.

In the following, a case where this embodiment is applied to psychological counseling will be described. That is, a case where an information processing device according to this embodiment is used by a person providing psychological counseling (e.g., a counselor, a doctor, etc.) to grasp the mental state of a subject involved in psychological counseling will be described. In addition, this embodiment is not limited to psychological counseling, and can be applied in various situations to grasp the mental state of a subject.

(First embodiment)
Fig. 1 is a diagram showing an example of a schematic configuration of an information processing device according to an embodiment. As shown in Fig. 1, an information processing device 20 according to an embodiment is capable of directly or indirectly communicating with a sensor 11 and a mobile terminal 12 via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). The information processing device 20 is an example of a bioinformation analysis device.

It should be noted that the sensor 11 and the mobile terminal 12 do not need to be constantly connected to the information processing device 20. For example, it is sufficient for them to be connected in order to exchange various information when a session with the subject is conducted. Furthermore, when exchanging information via an external recording medium or the like, the sensor 11 and the mobile terminal 12 do not need to be connected to the information processing device 20.

The sensor 11 is a sensor device attached to the subject. For example, the sensor 11 is a multi-sensor device equipped with multiple sensor functions. Specifically, the sensor 11 has the functions of an electrocardiograph, an optical pulse wave meter, a body motion meter, and a thermistor. The electrocardiograph measures time-series electrocardiogram waveform data. The optical pulse wave meter measures time-series pulse wave data. The body motion meter measures time-series acceleration data. The thermistor measures time-series temperature data. The sensor 11 records the various measured measurement data in a memory inside the device in association with the measurement start time. It should be noted that the electrocardiograph, optical pulse wave meter, body motion meter, and thermistor can be appropriately selected and applied from known technologies.

Here, the sensor 11 is basically attached to the subject at all times. When a session is held, the subject removes the sensor 11 and hands it over to an operator of the information processing device 20 (typically a counselor or doctor). The operator of the information processing device 20 transfers the measurement data recorded since the previous session from the sensor 11 received from the subject to the information processing device 20. Details of the process of acquiring bioinformation from the measurement data will be described later.

Note that the sensor 11 is not necessarily limited to a multi-sensor device, but may have at least the functionality of an electrocardiograph. The sensor 11 is not necessarily limited to being worn at all times, but may be removed as long as it does not affect the bioinformation analysis process described below. The timing of transfer of the measurement data recorded in the sensor 11 is not necessarily limited to the time of the session, but may be periodically (e.g., once a day) via a network, for example.

The mobile terminal 12 is, for example, a smartphone carried by the subject, and a recording application is introduced (installed) for collecting behavioral record information in which the subject's behavior is recorded. The behavioral record information is recorded in the internal memory of the mobile terminal 12 automatically or manually by the subject. The behavioral record information recorded in the mobile terminal 12 is transferred to the information processing device 20 at any time. Details of the process of acquiring the behavioral record information will be described later.

The mobile terminal 12 is not necessarily limited to a smartphone, but may be any information processing device, such as a tablet or personal computer, into which an application for collecting behavioral record information can be installed. However, due to the nature of the behavioral record information collected, it is preferable for the mobile terminal 12 to be a device (mobile terminal) that the subject can carry around with them on a daily basis.

The information processing device 20 is, for example, a computer such as a personal computer or a workstation. The information processing device 20 calculates mental state information indicating the mental state of the subject based on information acquired by the sensor 11 and the mobile terminal 12. The information processing device 20 includes an acquisition unit 201, a calculation unit 202, a generation unit 203, and an output control unit 204. Note that the functions of the information processing device 20 are not limited to the acquisition unit 201, the calculation unit 202, the generation unit 203, and the output control unit 204. The acquisition unit 201, the calculation unit 202, the generation unit 203, and the output control unit 204 will be described later.

Here, the hardware configuration of the information processing device 20 will be described with reference to FIG. 2. FIG. 2 is a diagram showing an example of the hardware configuration of the information processing device 20 according to an embodiment. As shown in FIG. 2, the information processing device 20 includes a CPU (Central Processing Unit) 21, a ROM (Read Only Memory) 22, a RAM (Random Access Memory) 23, an auxiliary storage device 24, an input device 25, a display device 26, and an external I/F (Interface) 27.

The CPU 21 is a processor (processing circuit) that executes a program to comprehensively control the operation of the information processing device 20 and realizes various functions of the information processing device 20. The various functions of the information processing device 20 will be described later.

The ROM 22 is a non-volatile memory that stores various data (information written during the manufacturing stage of the information processing device 20) including a program for starting the information processing device 20. The RAM 23 is a volatile memory that has a working area for the CPU 21. The auxiliary storage device 24 stores various data such as the program executed by the CPU 21. The auxiliary storage device 24 is composed of, for example, a hard disc drive (HDD), a solid state drive (SSD), etc.

The input device 25 is a device for an operator using the information processing device 20 to perform various operations. The input device 25 is composed of, for example, a mouse, a keyboard, a touch panel, or hardware keys. The operator corresponds to, for example, a medical professional such as a doctor.

The display device 26 displays various information. For example, the display device 26 displays image data, model data, a GUI (Graphical User Interface) for receiving various operations from an operator, medical images, etc. The display device 26 is configured, for example, as a liquid crystal display, an organic EL (Electro Luminescence) display, or a cathode ray tube display. The input device 25 and the display device 26 may be configured as one unit, for example in the form of a touch panel.

The external I/F 27 is an interface for connecting (communicating) with external devices such as the sensor 11 and the mobile terminal 12.

Using Figure 3, the process (biometric information analysis process) in which the information processing device 20 calculates mental state information of the subject will be described. Figure 3 is a flowchart showing an example of the operation of the information processing device 20 according to the embodiment. Figure 3 will be explained with reference to Figures 4 to 19. Note that the biometric information analysis process shown in Figure 3 can be executed at any timing, but is preferably executed every time a session is held for the subject.

3, the acquisition unit 201 acquires biometric information of the subject (step S101). For example, the acquisition unit 201 acquires the biometric information of the subject by generating the biometric information of the subject based on various measurement data collected by the sensor 11.

Here, the biological information includes, for example, autonomic nerve data, total power data, body movement data, body temperature data, pulse data, etc. The autonomic nerve data is time series information of activity indexes of the autonomic nerves, and includes sympathetic nerve data, parasympathetic nerve data, and total power data. The sympathetic nerve data is time series information of activity indexes of the sympathetic nerves (Sympathetic Nervous System: SNS level). The parasympathetic nerve data is time series information of activity indexes of the parasympathetic nerves (ParaSympathetic Nervous System: PSNS level). The total power data is data in which the sympathetic nerve data and the parasympathetic nerve data are integrated, and corresponds to, for example, the sum of the sympathetic nerve data and the parasympathetic nerve data. The body movement data is time series information indicating the intensity of body movement. The body temperature data is time series information of the body temperature on the body surface (the part where the sensor 11 is attached). The pulse data is time series information of the pulse.

An example of a process for acquiring biometric information of a subject will be described with reference to Figures 4 and 5. Figures 4 and 5 are diagrams for explaining the process of the acquisition unit 201. Figure 4 shows an example of electrocardiogram waveform data (electrocardiogram) measured by the sensor 11. Also, Figure 5 shows an example of autonomic nerve data generated by the acquisition unit 201.

For example, the acquisition unit 201 acquires the autonomic nerve data by generating the autonomic nerve data from electrocardiogram waveform data showing a time series of electrocardiogram waveforms. Specifically, the acquisition unit 201 detects the position of the R wave from the electrocardiogram waveform data shown in FIG. 4, and generates a heartbeat interval variation time series (RRI time series) in which the interval between the detected R waves (R-R interval: RRI) is plotted over time. The acquisition unit 201 then calculates a power spectrum from the RRI time series and accumulates the power in a predetermined frequency range. As an example, the acquisition unit 201 calculates the accumulated value of the power in the low frequency band (0.05 Hz to 0.15 Hz) as the SNS level (upper part of FIG. 5), and calculates the accumulated value of the power in the high frequency band (0.15 Hz to 0.40 Hz) as the PSNS level (lower part of FIG. 5). It is generally known that the SNS level is dominant in a stressed state, and the PSNS level is dominant in a relaxed state. In this manner, the acquisition unit 201 generates the sympathetic nerve data and the parasympathetic nerve data.

Furthermore, the acquisition unit 201 acquires total power data based on the sympathetic nerve data and the parasympathetic nerve data. For example, the acquisition unit 201 acquires total power data by adding up the SNS level and the PSNS level corresponding to each time.

In this way, the acquisition unit 201 acquires biological information of the subject based on various measurement data collected by the sensor 11. Note that the above-mentioned methods of acquiring sympathetic nerve data, parasympathetic nerve data, and total power data are merely examples, and are not limited to the above description. For example, known techniques can be appropriately selected and applied as methods of acquiring sympathetic nerve data, parasympathetic nerve data, and total power data.

In addition to the above data, the acquisition unit 201 can also acquire data based on various measurement data collected by the sensor 11. For example, the acquisition unit 201 can acquire time series data of heart rate and respiration rate from electrocardiogram waveform data. The acquisition unit 201 can also acquire time series data of percutaneous arterial oxygen saturation (SpO2) and pulse rate from pulse wave data. The acquisition unit 201 can also acquire physical movement data from acceleration data. The acquisition unit 201 can also acquire body temperature data from temperature data. As a method for acquiring these data, known techniques can be appropriately selected and applied.

Next, the acquisition unit 201 acquires the subject's behavior record information (step S102). For example, the acquisition unit 201 acquires the behavior record information from the subject's mobile terminal 12. Here, the behavior record information includes behavior data indicating the content of each behavior, time data indicating the time when each behavior was performed, location data indicating the location where each behavior was performed, medication history data indicating the medication history, and mood history data indicating the mood history at the time when each behavior was performed. In addition, the behavior data is represented by hierarchical classification information.

An example of a process for acquiring behavioral record information of a subject will be described with reference to Figures 6 and 7. Figures 6 and 7 are diagrams for explaining the process of the acquisition unit 201. Figure 6 shows an example of behavioral record information recorded in the mobile terminal 12. Also, Figure 7 shows an example of classification of behavioral data.

As shown in FIG. 6, the mobile terminal 12 collects behavior record information including behavior data, time data, location data, medication history data, and mood history data. For example, in the behavior record information shown in FIG. 6, the second record is associated with behavior data "breakfast meal", time data "1 hour 15 minutes 7:00", location data "home", medication history data "drug name: AAA medication time 7:40", and mood history data "expression mark (smile)". In other words, the second record indicates that the subject had breakfast at home for 1 hour 15 minutes from 7:00, and that the subject was in a good mood at that time. The mobile terminal 12 similarly collects behavior record information including behavior data, time data, location data, medication history data, and mood history data for the other records.

It is preferable that the behavioral data is represented by hierarchical classification information, as shown in FIG. 7. For example, the behavioral data is divided into major categories, medium categories, and minor categories. As an example, the major category "work" includes the medium categories "meetings" and "desk work". The medium category "desk work" includes the minor categories "preparing reports" and "organizing documents". It is preferable that the classification information of the behavioral data is predefined, but that the subject can add, delete, and edit the classification information.

Here, the behavior data, time data, and location data are automatically input by the mobile terminal 12 based on a schedule. For example, the mobile terminal 12 acquires the subject's schedule information from a schedule application installed in the mobile terminal 12. This schedule information includes information such as the subject's planned behavior, date and time, and location. The mobile terminal 12 then extracts the behavior data, time data, and location data from the schedule information and records the extracted information as behavior record information. The behavior data, time data, and location data automatically recorded by the mobile terminal 12 can be manually modified.

The location data may also be automatically corrected based on coordinate information acquired by a GPS (Global Positioning System) function of the mobile terminal 12. In this case, each piece of location data is associated in advance with the coordinate information of the location on the GPS function and registered in the recording application. If the location data corresponding to the coordinate information actually acquired by the GPS function differs from the location data based on the schedule information, the mobile terminal 12 discards the location data based on the schedule information. Note that if the coordinate information indicates movement, "movement" is input as the location data and the behavior data.

The medication history data and mood history data are also entered manually. For example, the subject launches a recording application each time they take medication and enters the name of the medication taken and the time they took it. After completing each action, the subject selects the mood history data corresponding to the mood they felt when performing that action. The medication history data and mood history data can also be manually modified. The mood history data is not limited to facial expression marks, and may be represented by text information expressing mood, such as "sleepy" or "relaxed," and numerical information expressing the degree of mood.

In this way, the mobile terminal 12 collects the subject's behavior record information. Then, the mobile terminal 12 transfers the collected behavior record information to the information processing device 20. The timing of transferring the behavior record information can be set arbitrarily. For example, the mobile terminal 12 may transfer the behavior record information periodically, or may transfer the behavior record information in response to a request from the subject or the operator of the information processing device 20. In this way, the acquisition unit 201 acquires the subject's behavior record information from the mobile terminal 12.

The subject's behavior record information can be corrected based on the physical movement data acquired from the sensor 11. For example, the acquisition unit 201 acquires the start and end times of the subject's sleep based on the intensity of the physical movement in the physical movement data. If the start and end times of sleep based on the physical movement data differ from the start and end times of sleep in the behavior record information, the start and end times of sleep in the behavior record information can be discarded and overwritten with the start and end times of sleep based on the physical movement data.

The acquisition unit 201 also acquires question answer information of the subject (step S103). For example, the acquisition unit 201 acquires, as the question answer information, mental state answer data regarding the mental state of the subject and session evaluation answer data regarding the evaluation of a session conducted with the subject.

Using Figures 8 and 9, an example of mental state answer data and session evaluation answer data will be described. Figures 8 and 9 are diagrams for explaining the processing of the acquisition unit 201. Figure 8 shows an example of mental state answer data. Figure 9 shows an example of session evaluation answer data.

For example, each time a session is held, the operator of the information processing device 20 distributes to the subject a questionnaire corresponding to each item of the mental state answer data (Figure 8) and the session evaluation answer data (Figure 9). Then, when the operator receives the questionnaire on which the subject has written their answers, he or she inputs the answers (question answer information) written on the received questionnaire into the information processing device 20. The question answer information may be input by image recognition of data scanned with a camera or the like, or may be input manually by the operator. As a result, the acquisition unit 201 acquires question answer information including the mental state answer data and the session evaluation answer data.

The calculation unit 202 then calculates the mental state information based on the biometric information, the behavioral record information, and the question answer information (step S104). For example, the calculation unit 202 identifies partial data from the time-series autonomic nerve data that corresponds to the period during which each behavior was performed, and calculates the statistical value of the identified partial data as the mental state information.

An example of mental state information will be described using Figure 10. Figure 10 is a diagram for explaining the processing of the calculation unit 202. Note that a numerical value indicating the mental state information calculated by the calculation unit 202 is input into "XXX" in Figure 10.

10, the calculation unit 202 calculates mental state information for each behavior based on various information acquired by the acquisition unit 201. Here, the mental state information is, for example, representative values of sympathetic nerve data, parasympathetic nerve data, and total power data for each behavior.

For example, the calculation unit 202 matches the measurement start time by the sensor 11 with the time data included in the behavior record information to associate the time series of the biometric information with the time series of the behavior record information. Then, the calculation unit 202 identifies partial data from the time series of autonomic nerve data that corresponds to the period during which each behavior was performed. Specifically, the calculation unit 202 identifies sympathetic nerve data from the time series of sympathetic nerve data for a period corresponding to "24:00 to 7:00" during which the behavior data "sleep" was performed. Then, the calculation unit 202 calculates the average value of the sympathetic nerve data for the identified period as the mental state information in FIG. 10. Note that the calculation unit 202 also calculates the parasympathetic nerve data and the total power data in the same manner as the sympathetic nerve data.

In this way, the calculation unit 202 calculates the mental state information for each behavior based on the biometric information and the behavior record information. Note that in the above explanation, the "average value" is calculated as a representative value of the sympathetic nerve data for each behavior, but this is not limited to the above, and any statistical value such as a peak value, a median value, or a standard deviation can be calculated.

Furthermore, the calculation unit 202 calculates the mental state score of the subject based on the mental state answer data. For example, the calculation unit 202 calculates the mental state score related to the mental state of the subject by inputting the scores for each item included in the mental state answer data into an arbitrary function.

The calculation unit 202 also calculates a session evaluation score for the session based on the session evaluation answer data. For example, the calculation unit 202 calculates a session evaluation score related to the evaluation of the session by inputting the scores for each item included in the session evaluation answer data into an arbitrary function.

The method of calculating the mental state score and session evaluation score can be changed as appropriate depending on the question items included in each question and answer information.

Then, the generation unit 203 generates a comment based on the mental state information (step S105). For example, the generation unit 203 generates at least one of a comment and an image that evaluates the mental state in each behavior based on the mental state information in each behavior.

An example of a process for generating comments will be described using Figures 11, 12, and 13. Figures 11, 12, and 13 are diagrams for explaining the process of the generation unit 203. Figure 11 shows an example of a template for each comment. Figure 12 shows an example of a comment generated by the generation unit 203. Figure 13 shows an example of an overall judgment generated by the generation unit 203.

As shown in FIG. 11, the auxiliary storage device 24 stores information for each of a plurality of comments, in which each comment template corresponds to the generation condition of each comment. For example, the auxiliary storage device 24 stores information in which the template "XX% of the time between XX:00 and XX:00 is good quality sleep" corresponds to the generation condition of sympathetic nerve data "less than threshold A", the generation condition of parasympathetic nerve data "threshold B or higher", and the generation condition of behavioral data "sleeping". This information indicates that when there is behavior in which the sympathetic nerve data is "less than threshold A", the parasympathetic nerve data is "threshold B or higher", and the behavioral data is "sleeping", a comment based on the template "XX% of the time between XX:00 and XX:00 is good quality sleep" is generated.

For example, the generation unit 203 determines whether or not each comment satisfies the generation condition for each behavior included in the behavior record information. If a comment is determined to satisfy the generation condition, the generation unit 203 reads out a template for that comment from the auxiliary storage device 24. The generation unit 203 then generates a comment for the behavior that satisfies the generation condition based on the read template.

For example, when there is an action in which the sympathetic nerve data is "less than threshold A", the parasympathetic nerve data is "above threshold B", and the action data is "sleep", the generation unit 203 reads out a comment template based on the template "xx% of the time from xx:xx to xx:xx was good quality sleep". The generation unit 203 then calculates the start time, end time, and percentage of good quality sleep for the "sleep" that satisfies the generation condition, and inputs the calculated information into the template to generate a comment such as "29% of the time from 11:05 p.m. to 6:00 a.m. was good quality sleep" (Figure 12). The percentage of good quality sleep is calculated, for example, as the percentage of the period during which the PSNS level was equal to or greater than a predetermined value during the sleep time.

Note that comments generated by the generation unit 203 are assigned to behavior that satisfies the conditions for generating the comment. For this reason, the comment "29% of the time between 11:05 p.m. and 6:00 a.m. was good quality sleep" is assigned to a sleep that ended at "6:00 a.m." on the date "2018/10/19."

In addition, the generating unit 203 generates an overall judgment of a common behavior based on mental state information for each of the multiple occurrences of a common behavior performed multiple times within a certain period of time. For example, if the behavior data "sleep" has been performed seven times between the previous session and the current session, the generating unit 203 performs an "overall judgment of sleep" based on the mental state information for the seven occurrences of sleep. Specifically, the generating unit 203 inputs the sympathetic nerve data and parasympathetic nerve data for the seven occurrences of sleep into an arbitrary function, and performs an overall judgment based on a four-level score of "0" to "3" shown in FIG. 13. Then, the generating unit 203 generates an overall judgment comment corresponding to the score of the overall judgment. The overall judgment comment is stored in the auxiliary storage device 24 in advance.

Then, the output control unit 204 outputs the mental state information and the comment (step S106). For example, the output control unit 204 outputs information that associates behavioral data indicating each behavior, a period during which each behavior was performed, and mental state information for each behavior.

An example of displaying autonomic nerve data will be described using Figures 14, 15, and 16. Figures 14, 15, and 16 are diagrams for explaining the processing of the output control unit 204. In Figures 14, 15, and 16, the vertical axis corresponds to the SNS level, and the horizontal axis corresponds to the PSNS level. Also, in Figures 14, 15, and 16, the size of the circle corresponds to the length of the period (time) during which each action was performed.

As shown in FIG. 14, for example, the output control unit 204 displays the autonomic nerve data for each behavior. In the example shown in FIG. 14, the output control unit 204 plots the four behaviors of work, travel, eating, and sleep. This allows the operator to easily grasp the difference in SNS level and PSNS level for each behavior. For example, the operator can see that the SNS level for work is higher than that for travel, eating, and sleep.

As shown in FIG. 15, the output control unit 204 simultaneously displays the latest autonomic nerve data and past autonomic nerve data for each action. In the example shown in FIG. 15, the output control unit 204 plots the autonomic nerve data for the current job, the autonomic nerve data for the previous job, and the autonomic nerve data for the job before last. This allows the operator to easily grasp the difference between the current mental state and past mental states for each action. For example, the operator can see that the SNS level at work is higher than before.

As shown in FIG. 16, the output control unit 204 also simultaneously displays standard autonomic nerve data for each behavior. In the example shown in FIG. 16, the output control unit 204 plots the autonomic nerve data for each behavior of the subject together with the autonomic nerve data of a standard person of the same age as the subject. This allows the operator to easily grasp the difference between the mental state of the subject and that of a standard person. For example, the operator can see that the SNS level of the subject is high in all behaviors of work, travel, eating, and sleep. The operator can also see that the subject's sleep time is shorter than that of a standard person.

In addition, the output control unit 204 displays the mental state score and the session evaluation score as mental state information.

Using Figures 17 and 18, we will explain examples of displaying the mental state score and session evaluation score. Figures 17 and 18 are figures for explaining the processing of the output control unit 204. In Figures 17 and 18, the vertical axis corresponds to the index of each piece of data, and the horizontal axis corresponds to the number of sessions. Note that the vertical axis is plotted with the index of each piece of data at the first session set to 100.

As shown in FIG. 17, the output control unit 204 displays the mental state score and the total power data. This total power data is a representative value of the total power data for a period (e.g., one week) separated by each session. This allows the operator to simultaneously view the objective mental state information detected by the sensor 11 along with the mental state score. For example, the mental state score may contain inaccuracies or lies depending on the mental state of the subject at the time of answering, but the operator can simultaneously view mental state information that objectively indicates the mental state that causes the inaccuracies or lies, which can be useful for giving advice for the next session.

As shown in FIG. 18, the output control unit 204 also displays the session evaluation score and the total power data. This total power data is a representative value of the total power data for the period (e.g., one week) divided by each session. This allows the operator to simultaneously view the objective mental state information detected by the sensor 11 along with the session evaluation score. For example, the mental state score may contain inaccuracies or lies depending on the mental state of the subject at the time of answering, but the operator can simultaneously view mental state information that objectively indicates the mental state that causes inaccuracies or lies, which can be useful when deciding the content of the next session.

In addition, the output control unit 204 displays comments regarding each action in correspondence with the action record information.

An example of comment display will be described using FIG. 19. FIG. 19 is a diagram for explaining the processing of the output control unit 204. FIG. 19 illustrates information in which bioinformation including arrhythmia, heart rate, amount of physical activity, sympathetic nerves, and parasympathetic nerves is associated with behavioral record information including behavioral history and mood history. In FIG. 19, the horizontal axis corresponds to time.

19, the output control unit 204 compares the start time of measurement by the sensor 11 with the time data included in the behavior record information, thereby displaying the time series of the biometric information and the time series of the behavior record information in association with each other. The output control unit 204 then displays comments regarding each behavior in association with the behavior record information.

Here, the comment generated by the generation unit 203 is assigned to the behavior that satisfies the condition for generating the comment. Therefore, the output control unit 204 displays the comment "29% of the time between 11:05 p.m. and 6:00 a.m. was good quality sleep" in association with "sleep" that ended at time "6:00" on date "2018/10/19". In addition, the output control unit 204 displays the comment "Feeling a bit excited while eating" in association with "dinner" that ended at time "20:00" on date "2018/10/18".

In this way, the output control unit 204 outputs the mental state information and the comments. Note that, although the case where the output control unit 204 displays the information on the display device 26 has been described here, the present invention is not limited to this. For example, the output control unit 204 may transmit the mental state information and the comments to an external device, or may store them in a memory or a recording medium.

Furthermore, Figures 14 to 19 are merely examples, and it is not required that all display forms be displayed simultaneously. For example, the operator can appropriately select and display the display forms exemplified in Figures 14 to 19 as necessary. Furthermore, the display items shown in Figures 14 to 19 can be changed at the operator's discretion. For example, the total power data shown in Figures 17 and 18 may be replaced with other mental state information, or multiple pieces of mental state information may be displayed. Furthermore, any statistical value such as the average value, peak value, median value, or standard deviation can be used as a representative value of the total power data.

As described above, in the information processing device 20 according to the embodiment, the acquisition unit 201 acquires biometric information including autonomic nerve data of the subject, and behavioral record information in which multiple behaviors of the subject are recorded. Furthermore, the calculation unit 202 calculates mental state information for each behavior of the subject based on the biometric information and the behavioral record information. This allows the operator to accurately grasp the mental state of the subject. For example, according to the information processing device 20 according to the embodiment, the mental state can be objectively quantified based on the electrocardiogram waveform data collected by the sensor 11, so that the mental state of the subject can be accurately grasped.

In addition, for example, the information processing device 20 quantifies the mental state of each behavior, making it easier to identify behaviors that cause stress. This allows the operator to provide more appropriate advice to the subject.

(Modification)
It is known that the magnitude of autonomic nerve data varies from person to person, so it is preferable to adjust the condition (threshold) for generating a comment in accordance with the total power data of the subject.

For example, the generation unit 203 adjusts the generation conditions for each comment stored in the auxiliary storage device 24 according to the total power data of the subject. For example, when the total power data of the subject is larger than the standard data, the generation unit 203 increases the threshold included in the generation conditions. Also, when the total power data of the subject is smaller than the standard data, the generation unit 203 decreases the threshold included in the generation conditions. Then, the generation unit 203 uses the adjusted generation conditions to determine whether or not the generation conditions for each comment are satisfied. This allows the information processing device 20 to appropriately assign comments according to the magnitude of the individual's autonomic nerve data.

Second Embodiment
In the first embodiment, a case has been described in which mental state information is calculated based on biometric information, behavioral record information, and question answer information, and a comment is generated based on the calculated mental state information, but the embodiment is not limited to this. For example, it is also possible to generate a mental state assessment result by inputting biometric information and behavioral record information into a trained model created by machine learning. Note that the mental state assessment result is data equivalent to the mental state answer data and the session evaluation answer data generated (determined) by the trained model.

That is, the information processing device 50 according to the second embodiment constructs a trained model for multiple subjects (second subjects) with a particular mental tendency, using biometric information including autonomic nerve data of each subject, behavioral record information in which multiple behaviors of each subject are recorded, and question answer information answered by each subject. The information processing device 50 then inputs the biometric information and behavioral record information of an arbitrary subject (first subject) into the constructed trained model, thereby generating a mental state determination result in which the mental state of the arbitrary subject is determined. Note that the first subject may be included in the second subject.

In the second embodiment, the "second subject" is, for example, a subject including a person with a specific mental tendency. The "specific mental tendency" includes mental disorders (mental illnesses) such as schizophrenia, mood disorders (depression, manic depression), panic disorder, eating disorder, developmental disorder, dementia, epilepsy, addiction, and higher brain dysfunction. In addition, among such persons with specific mental tendencies, if the second subject is a person with a mood disorder (depression, manic depression) and an eating disorder, the accuracy of the trained model (the accuracy of generating the mental state judgment result) is likely to improve, and in particular, if the second subject is a person with depression, the accuracy of the trained model is more likely to improve. However, the second subject is not limited to a person with a mental disorder, and may be a person classified by a mental tendency such as a preference. In addition, the second subject may be a person who is receiving psychological counseling.

Figure 20 is a diagram showing an example of a schematic configuration of an information processing device according to the second embodiment. As shown in Figure 20, the information processing device 50 according to the embodiment is capable of communicating directly or indirectly with the sensor 11 and the mobile terminal 12 via a network such as a LAN or WAN. The information processing device 50 is an example of a biometric information analysis device. The sensor 11 and the mobile terminal 12 are similar to the sensor 11 and the mobile terminal 12 shown in Figure 1, and therefore will not be described.

The information processing device 50 includes an acquisition unit 501, a determination unit 502, and an output control unit 503. Note that the acquisition unit 501 and the output control unit 503 are similar to the acquisition unit 201 and the output control unit 204 shown in FIG. 1, respectively, and therefore their explanations are omitted. In addition, the hardware configuration of the information processing device 50 is similar to the hardware configuration of the information processing device 20 shown in FIG. 2, and therefore their explanations are omitted. In addition, the determination unit 502 will be described later.

Using Figure 21, we will explain the processing during learning and operation performed by the information processing device 50 according to the second embodiment. Figure 21 is a diagram showing the processing during learning and operation performed by the information processing device 50 according to the second embodiment. Note that subject X shown in Figure 21 is an example of a first subject. Furthermore, subjects S-1 to S-N are examples of second subjects.

As shown in the upper part of Figure 21, during learning, the information processing device 50 performs machine learning using, for example, the biometric information, behavioral record information, and question answer information (mental state answer data and session evaluation answer data) of multiple subjects S-1 to S-N (e.g., multiple individuals prone to depression and multiple healthy individuals) as training data. Of the training data, the biometric information and behavioral record information are input data, and the question answer information is correct answer data. This machine learning constructs a trained model that outputs a mental state assessment result for the subject to be diagnosed by inputting the biometric information and behavioral record information of the subject to be diagnosed. This trained model is stored in a storage device (e.g., ROM 22, RAM 23, auxiliary storage device 24, etc.).

Then, as shown in the lower part of Figure 21, during operation, the information processing device 50 inputs the biometric information and behavioral record information of subject X, who is the subject of diagnosis, into the trained model constructed during learning, thereby causing the trained model to output a mental state assessment result for subject X. Here, subject X is, for example, a person suspected of having a tendency toward depression. Then, the information processing device 50 presents the mental state assessment result for subject X output from the trained model to the operator (or subject X).

The biometric information is the same as the biometric information described in the first embodiment. For example, the biometric information may include all of the autonomic nerve data, body temperature data, pulse data, and physical movement data, or may include only optional data. The target period for the biometric information can be set arbitrarily, such as several hours, several days, or several weeks.

The behavioral record information is similar to the behavioral record information described in the first embodiment. For example, the behavioral record information may include all of the behavioral data, time data, and mood history data, or may include only optional data. The target period of the behavioral record information can be set arbitrarily, but it is preferable that it be the same as the target period of the biometric information.

The question answer information is similar to the question answer information described in the first embodiment. For example, the question answer information may include both mental state answer data and session evaluation answer data, or may include only one of the data. The question answer information may be obtained once during any target period (preferably the same as the target period of the biometric information) as a trend for the entire period, or may be obtained any number of times at any timing, but it is preferable to obtain it once as a trend for the entire period.

The process of constructing the trained model is executed by, for example, the judgment unit 502. That is, the judgment unit 502 acquires, for a plurality of second subjects having a specific mental tendency, bioinformation including autonomic nerve data of the second subjects, behavior record information in which a plurality of behaviors of the second subjects are recorded, and question answer information answered by the second subjects. Here, the bioinformation, behavior record information, and question answer information of the second subjects are collected in advance and stored in a storage device (e.g., ROM 22, RAM 23, auxiliary storage device 24, etc.). For example, the judgment unit 502 acquires the bioinformation, behavior record information, and question answer information of the second subjects by reading the information from the storage device. Then, the judgment unit 502 constructs the trained model by performing machine learning using the acquired bioinformation, behavior record information, and question answer information. Note that the process of constructing the trained model is not limited to the judgment unit 502 and can be executed by any processing unit (processor).

Using FIG. 22, a process (mental state assessment process) in which the information processing device 50 outputs the mental state assessment result of the subject X during operation will be described. FIG. 22 is a flowchart showing an example of the operation of the information processing device 50 according to the second embodiment. Note that the mental state assessment process shown in FIG. 22 is a process for determining whether or not the subject X suspected of having a depressive tendency, etc., has a depressive tendency, etc. Furthermore, the mental state assessment process can be executed at any timing.

22, the acquisition unit 501 acquires biometric information of the subject X (step S201). For example, the acquisition unit 501 acquires the biometric information of the subject X by generating the biometric information of the subject X based on various measurement data collected by the sensor 11.

Next, the acquisition unit 501 acquires the behavior record information of the subject X (step S202). For example, the acquisition unit 501 acquires the behavior record information from the mobile terminal 12 of the subject X.

The judgment unit 502 then generates a mental state judgment result for the subject X by inputting the biometric information and behavioral record information of the subject X into the trained model. For example, the judgment unit 502 reads out the trained model stored in a storage device. The judgment unit 502 then inputs the biometric information and behavioral record information acquired by the acquisition unit 501 into the trained model that has been read out, thereby causing the trained model to output the mental state judgment result for the subject X. The judgment unit 502 then sends the mental state judgment result output from the trained model to the output control unit 503.

The mental state judgment result output from the trained model includes the same type of information as the question answer information used to construct the trained model. In other words, when the question answer information used in machine learning includes both mental state answer data and session evaluation answer data, the mental state judgment result output from the trained model includes both mental state answer data and session evaluation answer data. When the question answer information used in machine learning includes mental state answer data, the mental state judgment result output from the trained model includes mental state answer data. When the question answer information used in machine learning includes session evaluation answer data, the mental state judgment result output from the trained model includes session evaluation answer data.

The output control unit 503 outputs the mental state assessment result of the subject X. For example, the output control unit 503 displays the mental state assessment result on the display device 26. Note that the output destination of the mental state assessment result is not limited to the display device 26, and may be, for example, transmitted to an external device or stored in a memory or recording medium.

As described above, in the information processing device 50 according to the second embodiment, the acquisition unit 501 acquires biometric information including autonomic nerve data of the subject, and behavioral record information in which multiple behaviors of the subject are recorded. The determination unit 502 generates a mental state determination result for the subject by inputting the biometric information and behavioral record information of the subject to the trained model. This allows the operator to accurately grasp the mental state of the subject.

For example, according to the information processing device 50 of the second embodiment, the mental state assessment result corresponding to the mental state answer data and/or the session evaluation answer data can be automatically generated (determined) by the trained model, without the subject himself having to enter it. Therefore, the operator can easily understand whether or not the subject has a particular mental tendency by viewing the mental state assessment result output from the information processing device 50.

In the second embodiment, the case where the mental state assessment process is applied to a person suspected of having a tendency toward depression, etc. is described, but the present invention is not limited to this. For example, the mental state assessment process can also be applied to a person who is not suspected of having a tendency toward depression, etc. In other words, the subject X to be assessed (also called the first subject or assessment subject) does not need to be particularly limited and may be any subject. For example, the first subject may be any one of a plurality of second subjects.

In addition, in the second embodiment, a case has been described in which the trained model is constructed by the information processing device 50, but the embodiment is not limited to this. For example, the trained model may be constructed by an external device different from the information processing device 50. In this case, the information processing device 50 acquires the trained model constructed by the external device and executes the above-mentioned mental state assessment process.

(Modification 1 of the second embodiment)
In addition, the information processing device 50 described in the second embodiment is capable of not only presenting the mental state assessment result directly to the operator, but also calculating and presenting the “match rate” with the question answer information of persons prone to depression, etc.

For example, the judgment unit 502 compares the mental state judgment result of the subject X with representative question answer information of a person with a tendency towards depression, etc. Note that the representative question answer information of a person with a tendency towards depression, etc. may be any question answer information of a person with a tendency towards depression, etc. that is used as is, or the question answer information of a plurality of people with a tendency towards depression, etc. may be integrated (averaged) and used.

Specifically, the determination unit 502 compares each item included in the mental state assessment result with each item included in the question answer information, and calculates the number of matching items. The determination unit 502 then calculates the "match rate" by dividing the number of matching items by the total number of items. Note that the calculation method for the "match rate" described here is merely one example, and any calculation method can be applied.

Then, the output control unit 503 outputs the "match rate" calculated by the determination unit 502. This allows the information processing device 50 to support a diagnosis by an operator (a medical professional such as a doctor). For example, by viewing the match rate, the operator can accurately and easily diagnose whether or not a subject has a particular mental tendency.

In addition, when the matching rate is output, the mental state assessment result does not necessarily have to be output.

(Modification 2 of the second embodiment)
In addition, the information processing device 50 described in variant example 1 of the second embodiment is not only capable of presenting the matching rate directly to the operator, but also of presenting a suggestion as to whether or not the person is prone to depression, etc. (a suggestion of a possible mental tendency).

For example, the suggestion of a possible mental tendency may be information indicating the degree of possibility of a depressive tendency in stages. For example, the mental tendency judgment result may be information that classifies the possibility of a depressive tendency into three stages, "high," "medium," and "low," depending on the degree of match. When classifying into three stages, two threshold values are used.

In addition, when information indicating a possible mental tendency is output, the mental state assessment result or matching rate does not necessarily have to be output.

In addition, the information indicating possible mental tendencies that is presented to the operator is intended solely to assist in diagnosis and is not a diagnosis result itself. In other words, the processing of the information processing device 50 does not constitute a medical procedure.

(Modification 3 of the second embodiment)
The trained model (mental state assessment process) described in the second embodiment may be applied to the information processing device 20 shown in the first embodiment. This allows the information processing device 20 to output mental state information and comments even if the subject himself/herself does not enter question answer information.

Specifically, the information processing device 20 executes the process of step S203 in Fig. 22 instead of the process of step S103 in Fig. 3. That is, the information processing device 20 generates a mental state assessment result of the subject by inputting the biometric information and behavioral record information acquired in steps S101 and S102 in Fig. 3 to the trained model.

Then, in the processing of step S104 in Fig. 3, the information processing device 20 uses the mental state assessment result instead of the question answer information. That is, the information processing device 20 calculates the mental state information based on the biometric information, the behavioral record information, and the mental state assessment result. Note that the mental state assessment result is data equivalent to the mental state answer data and the session evaluation answer data, and therefore can be used in place of the question answer information.

Then, the information processing device 20 can output mental state information and comments by performing processing similar to steps S105 and S106 of FIG. 3.

Other Embodiments
In addition to the above-described embodiment, the present invention may be embodied in various different forms.

(Use of sensors that can be attached to the body surface)
In the above-described embodiment, it has been described that the sensor 11 can be applied by appropriately selecting a known technology. However, in order to collect biological information for a long period of time (several days or several weeks), it is preferable to apply a sensor that can be attached to the body surface. Such a sensor that can be attached to the body surface is preferably a wearable sensor having a circuit that does not break when bent or the like inside a flexible device so that it can flexibly follow the movement of the body. In addition, such a sensor that can be attached to the body surface is preferably a sensor that can be attached to the body surface via an adhesive such as an acrylic resin that does not cause discomfort to the body and has high medical quality characteristics and is compatible with biological safety tests.

For example, the acquisition unit 201 acquires biometric information based on data collected by a sensor 11 that can be attached to the subject's body surface. As a result, the acquisition unit 201 can collect both biometric information during sleep and biometric information during wakefulness without omission, so that continuous biometric information can be collected over a long period of time. The biometric information used in this embodiment does not necessarily have to be continuous data, but continuous data is preferable. Such long-term continuous data is preferably continuous data for 24 hours or more, more preferably continuous data for 72 hours or more, particularly preferably continuous data for 144 hours or more, and most preferably continuous data for 168 hours or more. In addition, such continuous data preferably includes continuous data on weekdays and continuous data on holidays. With such continuous data, a more accurate trained model can be constructed in the second embodiment.

(Biometrics Analysis System)
The functions of the information processing device 20 according to the above-described embodiment may be provided as a system.

Figure 23 is a diagram showing an example of a schematic configuration of a system according to an embodiment. As shown in Figure 23, system 1 according to an embodiment includes a server device 30 and a display terminal 40. The server device 30 and the display terminal 40 are capable of communicating with each other directly or indirectly via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network). System 1 is an example of a bioinformation analysis system.

In addition, the sensor 11 and the mobile terminal 12 are capable of communicating with the server device 30 directly or indirectly via a network, but do not need to be constantly connected to the server device 30. The sensor 11 and the mobile terminal 12 are basically configured in the same manner as the sensor 11 and the mobile terminal 12 shown in FIG. 1, and therefore a description thereof will be omitted.

The server device 30 is provided, for example, in a service center that provides biometric analysis processing as a cloud service. The server device 30 includes an acquisition unit 301, a calculation unit 302, a generation unit 303, and an output control unit 304. The processing contents of the acquisition unit 301, the calculation unit 302, the generation unit 303, and the output control unit 304 are basically the same as the processing contents of the acquisition unit 201, the calculation unit 202, the generation unit 203, and the output control unit 204 shown in FIG. 1, and therefore description thereof will be omitted.

The display terminal 40 is, for example, an information processing terminal used by a user of the cloud service, and is, for example, a terminal equipped with a display device, such as a personal computer, a workstation, a smartphone, or a tablet. Here, the user of the cloud service corresponds to a person who provides psychological counseling (for example, a counselor, a doctor, etc.).

Here, the output control unit 304 of the server device 30 transmits the mental state information for each behavior of the subject to the display terminal. The display control unit 401 of the display terminal 40 receives the mental state information for each behavior of the subject transmitted from the server device 30, and displays the received mental state information for each behavior of the subject. This allows the user to accurately grasp the mental state of the subject.

Furthermore, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU and a program analyzed and executed by the CPU, or can be realized as hardware using wired logic.

Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified.

The bioinformation analysis method described in the above embodiment can be realized by executing a prepared bioinformation analysis program on a computer such as a personal computer or a workstation. This bioinformation analysis program can be distributed via a network such as the Internet. This bioinformation analysis program can also be recorded on a computer-readable recording medium such as a hard disk, flexible disk (FD), CD-ROM, MO, or DVD, and executed by being read from the recording medium by a computer.

According to at least one of the embodiments described above, the mental state of the subject can be accurately grasped.

Although several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. These embodiments and their modifications are within the scope of the invention and its equivalents as set forth in the claims, as well as the scope and gist of the invention.

[Example]
The present invention will be described in more detail below based on examples, but the present invention is not limited to these examples.

(Examples 1 to 5, Comparative Example 1)
Using the method described in the second embodiment, a trained model was created that outputs mental state answer data and session evaluation answer data in response to input of biometric information and behavioral record information. Trained models were created for each of Examples 1 to 5 and Comparative Example 1 shown in Fig. 24 by varying the type of data used for machine learning. Fig. 24 is a diagram for explaining trained models relating to each Example and Comparative Example.

First, we will explain the input information (training data) used to create the trained model. In this example, the input information used was "biometric information," "behavioral record information," "mental state answer data," and "session evaluation answer data." In addition, 15 people prone to depression and 15 healthy individuals were used as subjects, and input information for each was collected for one week (168 hours including weekdays and holidays).

As "biometric information", autonomic nerve data, body temperature data, pulse data, and physical movement data were used. The autonomic nerve data includes sympathetic nerve data, parasympathetic nerve data, and total power data, each of which was generated from electrocardiogram waveform data. The body temperature data (temperature data) is time series information of the body temperature on the subject's body surface. The pulse data is time series information of the pulse. The physical movement data indicates the amount of physical activity, and was generated from acceleration data. Note that the electrocardiogram waveform data, body temperature data, pulse wave data, and acceleration data were collected over time using a sensor that can be attached to the subject's body surface (for example, a wearable device of the type described in JP2017-510390).

As the "behavioral record information," we used the behavioral data, time data, and mood history data shown in Figure 6. The behavioral record information was collected by having each subject record their behavior over the target period (the above-mentioned seven days).

The "mental state answer data" and "session evaluation answer data" used were the same data as those shown in Figures 8 and 9, respectively. The mental state answer data and session evaluation answer data were collected by having each subject record the data once at the end of the target period as a trend during each example and comparative example period.

Next, the creation of the trained models for each of the examples and comparative examples will be described. The trained models for each of the examples and comparative examples were created using various data corresponding to the "circles" in Figure 24 as training data. The trained models for each of the examples and comparative examples were created by machine learning using biometric information and behavioral record information as "input data" and mental state answer data and session evaluation answer data as "correct answer data", and in this example, Random Forest was used as the machine learning method.

The trained model for Example 1 was created using training data (input data and correct answer data) for 24 hours (one day) corresponding to weekdays (Monday to Friday). Note that in Example 1, machine learning was performed using autonomic nerve data as bioinformation, without using body temperature data, pulse data, or physical movement data.

The trained model for Example 2 was created using training data for 24 hours (one day) corresponding to holidays (Saturday and Sunday). Note that in Example 2, machine learning was performed using autonomic nerve data as biological information, without using body temperature data, pulse data, or physical movement data.

The trained model for Example 3 was created using 24 hours (one day) of training data corresponding to a weekday.

The trained model in Example 4 was created using 72 hours (three days) of training data corresponding to weekdays. Note that these three days of training data are, for example, a combination of training data for Monday, Wednesday, and Friday.

The trained model for Example 5 was created using 168 hours (one week) of training data corresponding to weekdays and holidays.

The trained model for the comparative example was created using training data for 24 hours (one day) corresponding to a weekday. In the comparative example, machine learning was performed using pulse data as biological information, without using autonomic nerve data, body temperature data, or physical movement data.

Next, a method for evaluating the trained models for each of the examples and comparative examples will be described. For one subject randomly selected from the subjects from whom the above training data was collected, input information for one week (168 hours including weekdays and holidays) was collected and used as input data for each trained model. The various collected information was cut out for a period corresponding to the "acquisition period" of the trained models for each of the examples and comparative examples, and used as input data.

Additionally, mental state answer data and session evaluation answer data for evaluation were collected by having the subjects record them once at the end of the target period. Note that mental state answer data and session evaluation answer data were collected once each target period had elapsed because it was thought that trends throughout each target period would be obtained. Furthermore, when training data from Mondays, Wednesdays, and Fridays was combined, the mental state answer data and session evaluation answer data collected at the end of data collection on Friday was used.

The collected mental state answer data and session evaluation answer data for evaluation were then compared with the output data (mental state answer data and session evaluation answer data) output from each trained model to determine the match rate. The match rate was determined according to the proportion of items that matched among the multiple items included in the mental state answer data and session evaluation answer data. Specifically, a match of 70% or more was determined as a "match." The match rate was also evaluated according to the number of times a match was determined out of 10 times (number of matches). Specifically, "AAA" indicates that the proportion of items that matched was 90% or more. "AA" indicates that the proportion of items that matched was 85% or more and less than 90%. "A" indicates that the proportion of items that matched was 80% or more and less than 85%. "B" indicates that the proportion of items that matched was 75% or more and less than 80%. "C" indicates that the proportion of items that matched was less than 75%.

As shown in FIG. 24, the degree of agreement of the trained model according to Example 1 was "A". The degree of agreement of the trained model according to Example 2 was "B". The degree of agreement of the trained model according to Example 3 was "AA". The degree of agreement of the trained model according to Example 4 was "AA". The degree of agreement of the trained model according to Example 5 was "AAA". The degree of agreement of the trained model according to the comparative example was "C".

The degree of agreement of the trained model in Example 5 was higher than the degrees of agreement of the other trained models. This result suggests that it is preferable for the input data during training and operation to be seven days or more (168 hours). In other words, it is preferable for the acquisition unit 201 to acquire seven days or more of biometric information and behavioral record information, and for the calculation unit 202 to calculate mental state information based on the seven days or more of biometric information and behavioral record information.

This result also suggests that it is preferable for input data during learning and operation to include weekdays and holidays. In other words, it is preferable for the acquisition unit 201 to acquire biometric information and behavioral record information for several days or more, including weekdays and holidays, and for the calculation unit 202 to calculate mental state information based on the biometric information and behavioral record information for several days or more, including weekdays and holidays.

In addition, the degree of agreement of the trained models in Examples 3 and 4 was higher than the degree of agreement of the trained models in Examples 1 and 2. This result suggests that it is preferable for the input data during training and operation to include body temperature data, pulse data, and physical movement data as biometric information.

In addition, the degree of agreement of the trained model in Example 1 was higher than the degree of agreement of the trained model in Example 2. This result suggests that weekday data is more suitable than holiday data for input data during training and operation.

The comparative example is an example in which only pulse rate data that can be collected by a relatively large number of wearable devices is used. The degree of match of the trained model in the comparative example was lower than the degrees of match of the trained models in Examples 1 to 5. This result suggests that it is preferable for the input data during learning and operation to include autonomic nerve data as bioinformation.

Claims (11)

an acquisition unit that acquires biometric information including autonomic nerve data including sympathetic nerve data, parasympathetic nerve data, and total power data obtained by integrating the sympathetic nerve data and the parasympathetic nerve data of the first subject based on data collected by a wearable sensor that can be attached to a body surface of the first subject, and physical movement data indicating the intensity of the first subject's physical movements, and also acquires behavior record information including behavior data indicating each behavior of the first subject and time data indicating the time when each behavior was performed, mental state answer data regarding the first subject's mental state, and a start time and end time of sleep based on the physical movement data, and corrects the behavior record information based on the start time and end time of sleep;
a calculation unit that identifies partial data corresponding to a period during which each behavior was performed among the time-series autonomic nerve data based on the biological information and the corrected behavior record information, calculates a statistical value of the identified partial data as mental state information for each behavior of the first subject, and calculates a mental state score of the first subject based on the mental state answer data;
a generation unit that generates at least one of a comment and an image that evaluates the mental state in each behavior based on the mental state information in each behavior;
a determination unit that generates a mental state determination result in which the mental state of a plurality of second subjects having a specific mental tendency is determined by inputting the biometric information and the behavior record information of the first subject into a trained model trained using the biometric information of the second subjects, the behavior record information of the second subjects, and question answer information answered by the second subjects; and
a control unit that outputs, for the first subject, behavioral data indicating each behavior, a period during which each behavior was performed, information associating the mental state information for each behavior with at least one of a comment and an image evaluating the mental state for each behavior, the mental state score, and the mental state assessment result. The trained model is a trained model trained using 72 hours or more of the biometric information of the second subject and 72 hours or more of the behavioral record information of the second subject,
The biological information analysis device according to claim 1 . The biometric information of the second subject and the behavior record information of the second subject include data on weekdays and holidays.
The biological information analysis device according to claim 2 . The biometric information of the second subject and the behavior record information of the second subject include continuous data for 24 hours or more.
The biological information analysis device according to claim 2 . The biometric information of the second subject and the behavioral record information of the second subject include continuous data for 144 hours or more.
The biological information analysis device according to claim 2 . The biometric information of the second subject and the behavioral record information of the second subject include continuous data for 168 hours or more.
The biological information analysis device according to claim 2 . The generation unit is
adjusting a generation condition of at least one of the comment and the image according to the total power data of the first subject;
generating at least one of the comment and the image using the adjusted generation conditions;
The biological information analyzing device according to any one of claims 1 to 6. The acquisition unit acquires session evaluation response data regarding an evaluation of a session conducted for the first subject,
the calculation unit calculates a session evaluation score of the session based on the session evaluation answer data.
The biological information analyzing device according to any one of claims 1 to 7. The acquisition unit acquires the biological information and the behavior record information for the first subject for 144 hours or more,
The calculation unit calculates the mental state information based on the biological information and the behavior record information of the first subject for 144 hours or more.
The biological information analyzing device according to any one of claims 1 to 8. A biometric information analysis system including a server device and a display terminal,
The server device includes:
an acquisition unit that acquires biometric information including autonomic nerve data including sympathetic nerve data, parasympathetic nerve data, and total power data obtained by integrating the sympathetic nerve data and the parasympathetic nerve data of the first subject based on data collected by a wearable sensor that can be attached to a body surface of the first subject, and physical movement data indicating the intensity of the first subject's physical movements, and also acquires behavior record information including behavior data indicating each behavior of the first subject and time data indicating the time when each behavior was performed, mental state answer data regarding the first subject's mental state, and a start time and end time of sleep based on the physical movement data, and corrects the behavior record information based on the start time and end time of sleep;
a calculation unit that identifies partial data corresponding to a period during which each behavior was performed among the time-series autonomic nerve data based on the biological information and the corrected behavior record information, calculates a statistical value of the identified partial data as mental state information for each behavior of the first subject, and calculates a mental state score of the first subject based on the mental state answer data;
a generation unit that generates at least one of a comment and an image that evaluates the mental state in each behavior based on the mental state information in each behavior;
a determination unit that generates a mental state determination result in which the mental state of a plurality of second subjects having a specific mental tendency is determined by inputting the biometric information and the behavior record information of the first subject into a trained model trained using the biometric information of the second subjects, the behavior record information of the second subjects, and question answer information answered by the second subjects; and
an output control unit that transmits, to the display terminal, behavioral data indicating each behavior of the first subject, a period during which each behavior was performed, the mental state information for each behavior, and information associating at least one of a comment and an image evaluating the mental state for each behavior, the mental state score, and the mental state assessment result;
The display terminal includes:
a display control unit that receives and displays behavioral data indicating each behavior of the first subject transmitted from the server device, information correlating the period during which each behavior was performed, the mental state information for each behavior, and at least one of a comment and an image evaluating the mental state for each behavior, the mental state score, and the mental state assessment result. Based on data collected by a wearable sensor that can be attached to the body surface of a first subject, obtain biometric information including autonomic nerve data including sympathetic nerve data, parasympathetic nerve data, and total power data obtained by integrating the sympathetic nerve data and the parasympathetic nerve data of the first subject, and physical movement data indicating the intensity of the first subject's physical movements, and obtain behavioral record information including behavioral data indicating each behavior of the first subject and time data indicating the time when each behavior was performed, mental state answer data regarding the mental state of the first subject, and sleep start time and end time based on the physical movement data, and correct the behavioral record information based on the sleep start time and end time;
Based on the biological information and the corrected behavior record information, partial data corresponding to a period during which each behavior was performed is identified from the time-series autonomic nerve data, and a statistical value of the identified partial data is calculated as mental state information for each behavior of the first subject, and a mental state score of the first subject is calculated based on the mental state answer data;
generating at least one of a comment and an image evaluating the mental state in each behavior based on the mental state information in each behavior;
generating a mental state determination result in which the mental state of a plurality of second subjects having a specific mental tendency is determined by inputting the biometric information and the behavioral record information of the first subject into a trained model trained using the biometric information of the second subjects, the behavioral record information of the second subjects, and question answer information answered by the second subjects;
A bioinformation analysis program that causes a computer to execute each process of outputting, for the first subject, behavioral data indicating each behavior, a period during which each behavior was performed, the mental state information for each behavior, and information associating at least one of a comment and an image evaluating the mental state for each behavior, the mental state score, and the mental state assessment result.

Images