JP5987238B2 - Multimodal tracking system and program - Google Patents

Description

  The present invention relates to a multimodal tracking system and a multimodal tracking program.

  In recent years, a mental health care service that leads mental health to a good state has attracted attention. Usually, the diagnosis target person visits the therapist, the therapist interviews the diagnosis target person, and the therapist diagnoses and conveys the result. However, this diagnosis result greatly depends on the knowledge and experience value of the therapist, and it takes time to make an accurate inquiry.

  Therefore, a diagnosis using a computer network has been proposed. For example, Patent Document 1 discloses a technique for improving a user's lifestyle using a computer connected to a network. According to this, advice is output from the computer based on information input by the user.

  By the way, regarding the input method to a computer, a multimodal interface which is a technique for inputting and outputting multimedia information such as audio information and image information has been developed (see Patent Document 2).

JP 2001-22837 A Japanese Patent No. 3844874

  However, in Patent Document 1, there is little daily information input from the user, and the lifestyle habit improvement support device cannot grasp much of the user's life content. On the other hand, Patent Document 2 does not show a specific application example using a multimodal interface.

  There are many problems in mental health care services, including those mentioned above, including the following. First, in the diagnosis by the inquiry, the diagnosis results vary depending on the knowledge and experience value of the person who makes the inquiry. In addition, since there is a difference in the degree of self-recognition and communication ability of the person to be diagnosed, it takes time to collect information until the person making the inquiry makes a diagnosis. Since there is a difference in the degree of self-recognition and communication ability of the person to be diagnosed, the diagnostic results of the person who is interviewed vary. When a person to be diagnosed has a mood disorder, self-recognition itself may adversely affect the physical condition.

  In addition, the person to be diagnosed may sometimes feel a psychological burden in that his / her privacy is recorded. The person to be diagnosed needs a large investment burden such as time, labor, and money to receive an inquiry from the therapist. Mood disorders are often unaware and unpredictably affected by the person being diagnosed (without awareness). Except in the case of hospitalization, the medical condition of the diagnosis subject is judged only at the time of an outpatient or counselor interview without grasping the actual situation of life

  The recovery status of the person being diagnosed is determined by the duration of positive transformation of self-recognition, but the person's declaration is the only way to ensure that sustainability. Understanding the facts of behavior is necessary to deal with problems that lower the mood of the person being diagnosed, but many are based on self-reporting. Problem coping for the purpose of improving the mood of the person being diagnosed (coping) depends on the experience of the interviewer, and if there is insufficient understanding of the behavior facts, there is a risk of lowering mood

  Moreover, although it is said that the mood state of a person to be diagnosed often takes a waveform in time series of ups and downs, it is not always measured. The recovery of the mood state of the person to be diagnosed is said to have a long period of waveform in the time series of ups and downs, but there is no means for measuring it. Since the person to be diagnosed has few contacts with a person, it is difficult to grasp the state.

  Also, in cognitive behavioral therapy used in the treatment of mood disorders, the therapist considers the thought pattern (automatic thinking) that the person takes on a daily basis without being aware of it. Arise. Personality disorder, phobia, etc. are revealed only by specific actions, but it takes time to identify the medical condition because the therapist cannot know all the information of the person's life.

  In addition, cognitive therapy corrects inappropriate cognition, but whether it can be practiced on a daily basis depends on the intermittent observation and guidance timing of the therapist, and it takes time to make an autonomous habit. In behavioral therapy, the content of behavior is changed step by step toward a major goal of shaping, but the behavior records are mainly self-reported, and there is variation in authenticity. One autonomous training method of behavioral therapy is exercise that controls respiration, pulse, and body temperature regulation by self-impliedness, but does not recognize the physical changes uniquely or in time series.

  In addition, behavioral analysis is performed during behavioral therapy, but the collection of behavior is mainly due to self-reports from the person, which may cause variation and result in misdiagnosis. Because the language ability is underdeveloped, such as preschool children who are diagnosed, it may be difficult to grasp changes in psychological state by language alone, or it may be difficult to find a disorder. When a person to be diagnosed is a minor, there may be a problem in the behavior when a supervisor such as a parent cannot observe the behavior, but there is no system for objectively recording the behavior.

  Also, if the person being diagnosed has the belief that it is bad to express emotional self-recognition (schema), the signs of psychological transformation must be grasped by other vital signs. When a person to be diagnosed lives alone, contact with a person is necessary to discover daily changes, but it is personal and depends on the social nature of the person. When a person to be diagnosed has a physical disability (such as a language disorder), it may be difficult to obtain information necessary for diagnosis.

  In addition, support for dependents is necessary while the subject is symptomatic, but support is limited in terms of time and conditions, and cannot be done when support is needed. If the person being diagnosed is a worker who is on leave, supervision of the supervisor of the affiliated company is required, but grasping the status is personal. Early episodes are thought to cause depression, but there is no way to measure it.

  In addition, domestic violence of preschool children is the cause of mental disorders such as PTSD, but the detection of signs depends on the social nature of the family and varies. Suicides caused by bullying at school depend on stakeholder's ability to collect psychological information to detect and deal with the signs. Alcoholism is a behavioral habit of drinking alcohol unconsciously, and the behavior records for its treatment are self-reported and vary, except for hospitalization. Bulimia, etc. is caused by the unconscious eating behavior of the person, but the behavior records are self-reported and vary, except for being hospitalized.

  To solve one or more of the above problems, the present invention provides a multimodal tracking system and a multimodal tracking program capable of outputting accurate information quickly without requiring knowledge or experience values. With the goal.

In order to achieve the above object, the multimodal tracking system of the present invention includes an acquisition unit that acquires a plurality of pieces of information among a subject's biological information, social environment information, and self-recognition information;
A mood state master that defines each coefficient used when obtaining a mood state level from information that can be acquired by the acquisition unit;
A response master that defines the mood state level and the response in association with each other;
A storage unit storing the mood state master and the response master;
Based on the information acquired by the acquisition unit, the mood state master, and the response master, a control unit that calculates a current mood state level and generates a response;
An output unit that is an output destination of the generated response.

  In the multimodal tracking system described above, the biological information is at least one of audio information, facial expression information, facial temperature information, eye movement information, pulse information, heart rate information, respiratory information, and blood pressure information. Is desirable.

  In the multimodal tracking system, it is preferable that the self-recognition information is at least one information of emotion information and personality tendency information.

  In the multimodal tracking system, the social environment information is preferably personal space information.

  In the multimodal tracking system, the information acquired by the acquisition unit is preferably a vital sign automatically detected by a vital sign detection device.

  In the multimodal tracking system, the information acquired by the acquisition unit is preferably information input to the input device by the target person.

In the multimodal tracking system, when one or more pieces of information acquired by the acquisition unit is a vital sign automatically detected by a vital sign detection device,
The controller is
Multiplying the obtained fluctuation value of each vital sign per predetermined time by the coefficient, the sum thereof is used as the first mood state level calculated value,
Multiplying the obtained fluctuation value of each vital sign at a certain time by the coefficient, the sum thereof is used as a second mood state level calculation value,
From the first calculated mood state level value and the second calculated mood state level value, it is determined whether the acquired vital sign is abnormal or stable or not, and is used as an element for generating the response. desirable.

In the multimodal tracking system, the control unit
Comparing the mood state level first calculated value with a first threshold, comparing the mood state level second calculated value with a second threshold;
If the mood state level first calculated value> the first threshold and the mood state level second calculated value ≧ the second threshold, it is determined to be abnormal,
If the mood state level first calculated value> the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value <the first threshold and the mood state level second calculated value> the second threshold, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value <the first threshold value and the mood state level second calculated value ≦ the second threshold value, it is determined to be abnormal,
If the mood state level first calculated value = the first threshold value and the mood state level second calculated value> the second threshold value, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value = the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a tendency to stabilize,
When the mood state level first calculated value = the first threshold value and the mood state level second calculated value = the second threshold value, it is desirable to determine that there is no change.

In the above-described multimodal tracking system, the voice information is scaled, and the scaler for calculating the number of syllables and the voice value;
A character string converting device that converts voice information into a character string;
One of the mood state masters, and a scale master that defines voice states and scale coefficients corresponding to each other;
An emotion master that is one of the mood state masters and defines the emotion state and the emotion coefficient in correspondence with each other,
When one of the information acquired by the acquisition unit is audio information,
The controller is
The voice state is determined from the scaled result, the sum of the scale coefficient corresponding to the corresponding voice state and a constant specific to the subject is calculated, and the product of the calculated value and the voice value is calculated by the number of syllables. Divided into mood state level 3rd calculated value,
The number of words representing emotional state and emotion is determined from the result of characterizing, and the product of the emotional coefficient corresponding to the relevant emotional state and the number of words is calculated from the number of syllables within the measurement time from 1 The total value up to the mood state level 4th calculated value,
A value obtained by dividing a total value within the measurement time for the product of the third mood state level calculated value and the fourth mood state level calculated value by the number of syllables within the measurement time is used as an emotion value, and the response It is desirable to be one element that generates

In the multimodal tracking system, when one of the information acquired by the acquisition unit is character information,
The controller is
The number of syllables, emotional states, and the number of words representing emotions are determined from the character information, and the product of the emotion coefficient corresponding to the relevant emotional state and the number of words is calculated from the syllable number from 1 to the syllable within the measurement time. It is desirable that a value summed up to a number be the mood state level fourth calculated value.

In the multimodal tracking system, the control unit
Comparing the emotion value with a third threshold;
If it is determined that the emotion value> the third threshold value and the stabilization tendency, the mood state level is determined to be praise, and the response classified as praise in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
When it is determined that the emotion value is the third threshold value and the stabilization tendency, the mood state level is determined to be a countermeasure, and the response classified as the countermeasure in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
If it is determined that the emotion value <the third threshold value and the stabilization tendency, the mood state level is determined to be throwing, and the response classified as throwing in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
If it is determined that the emotion value is greater than the third threshold and the change is not present, the mood state level is determined to be throwing, and the emotion is selected from the responses classified as throwing in the response master. Select and generate a response that corresponds to the score equivalent to the value,
When it is determined that the emotion value = the third threshold and the change is not present, the mood state level is determined to be consent, and the emotion is selected from among the responses classified as consent in the response master. Select and generate a response that corresponds to the score equivalent to the value,
If it is determined that the emotion value <the third threshold and the change is not present, the mood state level is determined to be agreement, and the emotion is selected from among responses classified as agreement in the response master. Select and generate a response that corresponds to the score equivalent to the value,
If it is determined that the emotion value is greater than the third threshold and the abnormality, the mood state level is determined to be consent, and the emotion value is selected from among responses classified as consent in the response master. Select and generate a response corresponding to a score equivalent to
If it is determined that the emotion value is the third threshold and the abnormality, the mood state level is determined to be consent, and the emotion value is selected from among the responses classified as consent in the response master. Select and generate a response corresponding to a score equivalent to
If it is determined that the emotion value <the third threshold value and the abnormality, the mood state level is determined to be throwing, and the emotion value is selected from the responses classified as throwing in the response master. It is desirable to select and generate a response corresponding to a score equivalent to.

In the multimodal tracking system, the storage unit includes a behavior characteristic information master that stores past personal data of the subject including the mood state level and the output response.
The control unit preferably generates a response in consideration of the information of the behavior characteristic information master.

  In the multimodal tracking system, the output unit is preferably a speaker.

  In the multimodal tracking system, the output unit is preferably mounted on a portable terminal device.

  The multimodal tracking program of the present invention operates any of the multimodal tracking systems described above.

  According to the present invention, it is possible to provide a multimodal tracking system and a multimodal tracking program capable of outputting accurate information quickly without requiring knowledge and experience values.

It is a block diagram which shows the structure of the multimodal type tracking system of this invention. It is a flowchart which shows operation | movement of the multimodal type tracking system of this invention. It is an example of the life log master of this invention. It is an example of the vital sign master of this invention. It is an example of the face temperature of this invention. It is an example of the heart rate of this invention. It is an example of the respiration frequency of this invention. It is an example of the pulse rate of this invention. It is an example of the facial expression of this invention. It is an example of the eye movement of the present invention. It is an example of the blood pressure of this invention. It is an example of the personal space of this invention. It is an example of the self-recognition level of this invention. It is an example of the scale master of the present invention. It is an example of the language master of this invention. It is an example of the emotion master of this invention. It is an example of the action characteristic information master of this invention. It is an example of the response master of this invention. It is an Example of this invention.

  FIG. 1 is a block diagram showing the configuration of the multimodal tracking system of the present invention. A multimodal tracking system 10 includes a central device 12 capable of storing and calculating data such as a server device in a communication network 11 such as the Internet, terminal devices 13 and 14 such as a personal computer, and a mobile terminal device such as a mobile phone. 15, 16, a scale conversion device 26, and a character string conversion device 27 are connected. Printing devices 17 to 19 such as printers are connected to the central device 12 and the terminal devices 13 and 14, respectively. Further, cameras 20 and 23, microphones 21 and 24, and speakers 22 and 25 are connected to the terminal devices 13 and 14.

  Each of the terminal devices 13 and 14 and the mobile terminal devices 15 and 16 includes input units 13a, 14a, 15a, and 16a including keys and a touch panel, and output units 13b and 14b including a liquid crystal display device that displays characters and images. 15b and 16b. The mobile terminal devices 15 and 16 also include cameras 15c and 16c, microphones 15d and 16d, and speakers 15e and 16e. Here, the camera and the microphone have a role as a vital sign detection device.

  The scale conversion device 26 is a device that converts the voice information into scales and calculates the number of syllables and the voice value. The character string converting device 27 is a device that converts voice information into a character string. It should be noted that conventionally developed techniques can be appropriately adopted as a technique for converting voice information into a scale, a technique for calculating the number of syllables and a voice value, and a technique for converting voice information into a character string.

  In the configuration of FIG. 1, a multimodal tracking program is stored in the central device 12, and the terminal devices 13, 14 and the mobile terminal devices 15, 16 transmit / receive information to / from the central device 12, and output units and speakers. Or information input from the input unit, camera, or microphone is transmitted to the central device 12.

  Note that the printing devices 17 to 19 are not necessarily required. The terminal devices 13 and 14 and the mobile terminal devices 15 and 16 may be any one or more, and the central device 12 includes, for example, an arithmetic device (control unit 12b), a database (storage unit 12a), and an acquisition unit 12c. A plurality of devices may be linked such as being separate. Further, the system of the present invention does not necessarily need to be configured by a plurality of devices connected by the communication network 11 as shown in FIG. 1, and may have a configuration in which the devices are directly connected without the communication network 11. All the configurations may be incorporated in one apparatus.

  Next, automatic generation of a response from stress tracking in the multimodal tracking system 10 will be described. A response suitable for the current situation is created for each individual (subject).

  FIG. 2 is a flowchart showing the operation of the multimodal tracking system. The broken lines in the figure indicate the data flow. Here, an example will be described in which a response is generated in accordance with the daily activity of Mr. N (subject) during rehabilitation.

  In step S 10, when Mr. N's activities start, for example, when Mr. N gets up, the camera 20 installed near the bed captures the movement of Mr. N, and the central device is connected via the terminal device 13. It is detected that the user has woken up by transmitting data to 12. In addition, for example, when Mr. N who gets up operates the input unit 15a of the mobile terminal device 15 and checks in the system, the data is transmitted to the central device 12 and is detected to be up. The position of the subject detected by GPS (Global Positioning System) can also be used for check-in. Such activity data of Mr. N is stored as target information in the life log master of the storage unit 12a together with the time information and life log ID.

  FIG. 3 shows an example of the life log master. The activity performed by the subject, such as getting up, falling asleep, breakfast, work, etc., is recorded as a life log along with the time (time information) and ID of each activity. The actual time corresponding to the time information is written in a correspondence table (not shown) stored in the storage unit 12a.

  Next, when the acquisition unit 12c acquires a plurality of pieces of information about Mr. N, the process proceeds to any of step S11, step S12, and step S13 according to the acquired information. Here, the information that can be acquired by the acquisition unit 12c is a plurality of pieces of information among the subject's biological information, social environment information, and self-recognition information.

  The biological information is, for example, at least one or more information among speech information, facial expression information, facial temperature information, eye movement information, pulse information, heart rate information, respiratory information, and blood pressure information based on words uttered by the subject. The self-recognition information is, for example, at least one or more information among emotion information and personality tendency information. The social environment information is, for example, personal space information.

  The information acquired by the acquisition unit 12c may be a vital sign automatically detected by a camera or a microphone, or may be information input to the input unit by the subject. For example, when the camera captures the subject, the central device 12 can analyze the subject's facial expression, facial temperature, eye movement, pulse, breathing, blood pressure, personal space, emotions, and so on. In addition, voice information can be generated by the central device 12 by picking up words spoken by the subject to speak or ask the system. Moreover, self-recognition information etc. can be produced | generated by the character information input into the input part, such as the act in which an object person transmits an email.

  And when the acquisition part 12c acquires a vital sign, it progresses to step S11 and memorize | stores the acquired vital sign in a vital sign master. In this embodiment, vital signs are appropriately stored in the vital sign master shown in FIG. 4, and specific numerical values of the vital signs are shown in FIG. 5 to FIG. ing. FIG. 5 shows the facial temperature (° C.), FIG. 6 shows the heart rate (bpm), FIG. 7 shows the number of breaths (times / minute), FIG. 8 shows the pulse rate (bpm), and FIG. Ratio), FIG. 10 shows the eye movement ((pupil diameter) mm), FIG. 11 shows the blood pressure (mmHg), FIG. 12 shows the personal space (cm), and FIG. 13 shows the self-recognition level.

  Next, it progresses to step S14 from step S11, and the control part 12b calculates each element for producing | generating a response from the acquired vital sign. Specifically, the obtained fluctuation value per predetermined time of each vital sign is multiplied by a predetermined coefficient, and the sum thereof is set as a mood state level first calculated value V1. Further, the obtained fluctuation value of each vital sign at a certain time is multiplied by a predetermined coefficient, and the sum thereof is used as a mood state level second calculated value V2. Further, an integrated value of the mood state level second calculated value V2 is set as a mood state level fifth calculated value V3.

V1 can be expressed by the following formula, for example.
_{V1 (t n) = a 0} + a 1 (FT (t n) -FT (t 0)) + a 2 (HB (t n) -HB (t 0)) + a 3 (BR (t n) -BR (t ₀ )) + a ₄ (PR (t _n ) −PR (t ₀ )) + a ₅ (EX (t _n ) −EX (t ₀ )) + a ₆ (EB (t _n ) −EB (t ₀ )) + a _{7 (BP (t n) -BP (} t 0)) + a 8 (PS (t n) -PS (t 0)) + a 9 (SR (t n) -SR (t 0))
In the equation, t _n is a time interval, t ₀ is a time when the subject uses the system for the first time, a _{0 to} a ₉ are predetermined coefficients predetermined for each type of vital sign, FT is a facial temperature, HB Is the heart rate, BR is the respiratory rate, PR is the pulse rate, EX is the facial expression, EB is the eye movement, BP is the blood pressure, PS is the personal space, and SR is the self-recognition level.

V2 can be expressed by the following equation, for example.
V2 (t _n ) = b ₀ + b ₁ dFT (t _n ) / dt + b ₂ dHB (t _n ) / dt + b ₃ dBR (t _n ) / dt + b ₄ dPR (t _n ) / dt + b ₅ dEX (t _n ) / dt + b ₆ dEB (t _n ) / dt + b ₇ dBP (t _n ) / dt + b ₈ dPS (t _n ) / dt + b ₉ dSR (t _n ) / dt
In the equation, b _{0 to} b ₉ are predetermined coefficients predetermined for each type of vital sign.

V2 can also be expressed by the following equation.
V2 (t _n ) = (V1 (t _n ) −V1 (t _n−1 )) / Δt = dV1 (t _n ) / dt

V3 can be expressed by the following equation, for example.
V3 (t _n ) = ∫V2 (t _n ) dt (n is an arbitrary measurement section)

  Subsequently, the process proceeds from step S14 to step S15, and the controller 12b determines whether the acquired vital sign is abnormal or has a stabilization tendency from the first mood state level calculated value V1 and the second mood state level calculated value V2. Is used as an element for generating a response. Here, a mood state master defining each coefficient used when obtaining a mood state level from information that can be acquired by the acquisition unit 12c is used. The mood state master is stored in the storage unit 12a and includes a vital sign master, a scale master, and an emotion master.

  Specifically, the control unit 12b compares the mood state level first calculated value V1 with a first threshold (for example, 0), and the mood state level second calculated value V2 and the second threshold (for example, 0). And whether there is an abnormality (for example, Z = -1), a stabilization tendency (for example, Z = 1), or no change (for example, Z = 0) Determine.

For example, when mood state level first calculated value> first threshold value and mood state level second calculated value ≧ second threshold value, it is determined that there is an abnormality. That is, when V1 (t _n )> 0 and V2 (t _n ) ≧ 0, it is determined that Z = −1.

Further, when the mood state level first calculated value> the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a stabilization tendency. That is, when V1 (t _n )> 0 and V2 (t _n ) <0, it is determined that Z = 1.

Further, when the mood state level first calculated value <the first threshold value and the mood state level second calculated value> the second threshold value, it is determined that there is a stabilization tendency. That is, when V1 (t _n ) <0 and V2 (t _n )> 0, it is determined that Z = 1.

Further, when the mood state level first calculated value <the first threshold value and the mood state level second calculated value ≦ the second threshold value, it is determined to be abnormal. That is, when V1 (t _n ) <0 and V2 (t _n ) ≦ 0, it is determined that Z = −1.

Further, when the mood state level first calculated value = the first threshold value and the mood state level second calculated value> the second threshold value, it is determined that there is a stabilization tendency. That is, when V1 (t _n ) = 0 and V2 (t _n )> 0, it is determined that Z = 1.

Further, when the mood state level first calculated value = the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a stabilization tendency. That is, when V1 (t _n ) = 0 and V2 (t _n ) <0, it is determined that Z = 1.

Further, when the mood state level first calculated value = the first threshold value and the mood state level second calculated value = the second threshold value, it is determined that there is no change. That is, when V1 (t _n ) = 0 and V2 (t _n ) = 0, it is determined that Z = 0.

  On the other hand, when the acquisition unit 12c acquires voice information, the process proceeds from step S10 to step S12, and the central device 12 transmits the voice information to the scale conversion device 26 and the character string conversion device 27. The voice information is scaled by the scaler 26 and the number of syllables and voice values are calculated and transmitted to the central unit 12. On the other hand, the voice information is converted into a character string by the character string converting device 27 and transmitted to the central device 12. The central device 12 receives the scaled data in step S16, and receives the characterized data in step S17.

  Proceeding from step S16 to step S18, the control unit 12b determines the voice state from the scaled result, calculates the sum of the scale coefficient corresponding to the corresponding voice state and the subject-specific constant, and calculates the calculation. The product of the calculated value and the voice value is divided by the number of syllables to obtain a mood state level third calculated value MN. Here, a scale master that is defined by associating a voice state with a scale coefficient is stored in the storage unit 12a as one of the mood state masters. In addition, constants specific to the subject are described in the behavior characteristic information master described later.

  FIG. 14 shows an example of the scale master. IDs and coefficients are written in correspondence with voice states such as excitement, calmness, and depression.

The mood state level third calculated value MN can be expressed by the following formula, for example.
MN (t _n ) = {Vo (t _n ) × (mn ₀ + k ₁ )} / VC
In the equation, Vo is a voice value, mn ₀ is a constant specific to the subject, k ₁ is a scale coefficient corresponding to the voice state, and VC is the number of syllables.

  On the other hand, the process proceeds from step S17 to step S13, where the control unit 12b determines the number of words representing the emotional state and the emotion from the result of characterizing, stores the word in the language master, and stores the life log in the lifelog master. Remember.

  FIG. 15 shows an example of the language master. An ID and time (time information) are written in correspondence with words such as tired or faint. The actual time corresponding to the time information is written in a correspondence table (not shown) stored in the storage unit 12a.

  Next, proceeding to step S19, the control unit 12b adds the product of the emotion coefficient corresponding to the corresponding emotion state and the number of words to the mood state level from the sum of the number of syllables from 1 to the number of syllables within the measurement time. A fourth calculated value e is assumed. Here, an emotion master defined by associating an emotion state with an emotion coefficient is stored in the storage unit 12a as one of the mood state masters.

  FIG. 16 shows an example of the emotion master. IDs and emotion coefficients are written in correspondence with emotional states such as fun, satisfaction, and anger.

The mood state level fourth calculated value e can be expressed by the following equation, for example.
e (t _n ) = Σ (W (t _n ) × k ₂ ) <n = 1 → WC>
In the equation, W is the number of words representing emotion, k ₂ is the emotion coefficient corresponding to the emotion state, and WC is the number of syllables within the measurement time.

  From step S18 and step S19, the process proceeds to step S20. In step S20, the control unit 12b divides the total value within the measurement time for the product of the mood state level third calculated value MN and the mood state level fourth calculated value e by the number of syllables within the measurement time. The value is an emotion value E, and is an element for generating a response.

The emotion value E can be expressed by the following equation, for example.
E (Δt) = Σ (e (t _n ) MN (t _n ) / WC (t _n )) <t _n-1 → t _n >

  On the other hand, when the acquisition unit 12c acquires character information, the process proceeds from step S10 to step S13. In step S13, similarly to the above, the control unit 12b determines the number of words representing syllable number, emotional state, and emotion from the character information, and calculates the product of the emotion coefficient corresponding to the emotional state and the number of words. The sum of the number of syllables from 1 to the number of syllables within the measurement time is defined as a mood state level fourth calculated value e. From step S13, the process proceeds to step S20 through step S19, and the same processing as described above is performed.

  Next, it progresses to step S21 from step S20, and the control part 12b compares the emotion value E with the content of the action characteristic information master. Here, the behavior characteristic information master is past personal data of the subject including the mood state level and the output response, and is stored in the storage unit 12a.

  FIG. 17 shows an example of the behavior characteristic information master. Corresponding to each ID, a past emotion value E and a response ID (response log) output at that time are described.

  Next, it progresses to step S22 from step S21, and response ID in case the emotion value E is the same value is extracted. This response ID is taken into account when generating a response below.

  Next, the process proceeds from step S22 to step S23, and a response is generated using the response master. The response master is defined by associating mood state levels with responses, and is stored in the storage unit 12a.

  FIG. 18 shows an example of the response master. Responses are classified into praise (positive), coping (negative), consent (neutral), and throwing (question), and ID, response (contents output to the target person), and score (emotion value E) Value).

  Specifically, the control unit 12b compares the emotion value E with a third threshold value (for example, 0), and determines whether the magnitude relationship between the emotion value E and the above-described abnormality (for example, Z = −1) is stable. Which of the praise (positive), coping (negative), consent (neutral), and throwing (questioning) from the combination of the tendency to change (for example, Z = 1) or no change (for example, Z = 0) It is determined whether it is a classification, and a score response with the same value as the emotion value E is selected and generated from the classification.

  For example, when it is determined that the emotion value> the third threshold value and the stabilization tendency, it is determined that the mood state level is praise, and the emotion value is selected from the responses classified as praise in the response master. A response corresponding to a score equivalent to E is selected and generated.

  If it is determined that the emotion value is the third threshold value and the tendency to stabilize, it is determined that the mood state level is the countermeasure, and the emotion value is selected from the responses classified as the countermeasure in the response master. A response corresponding to a score equivalent to E is selected and generated.

  Further, when it is determined that the emotion value <the third threshold value and the stabilization tendency, it is determined that the mood state level is the throwing, and the emotion value is selected from the responses classified as throwing in the response master. A response corresponding to a score equivalent to E is selected and generated.

  If it is determined that the emotion value> the third threshold and no change, the mood state level is determined to be throwing, and the emotion value E is selected from the responses classified as throwing in the response master. A response corresponding to the score equivalent to is selected and generated.

  If it is determined that the emotion value is equal to the third threshold value and there is no change, it is determined that the mood state level is consent, and the emotion value E is selected from the responses classified as consent in the response master. A response corresponding to the score equivalent to is selected and generated.

  If it is determined that the emotion value <the third threshold and there is no change, it is determined that the mood state level is consent, and the emotion value E is selected from the responses classified as consent in the response master. A response corresponding to the score equivalent to is selected and generated.

  If it is determined that the emotion value is greater than the third threshold value and abnormal, the mood state level is determined to be agreement, and the emotion value E and the response are classified as agreement in the response master. Select and generate a response corresponding to the score of the same value.

  If it is determined that the emotion value is equal to the third threshold value and abnormal, the mood state level is determined to be consent, and the emotion value E and the response classified as consent in the response master are determined. Select and generate a response corresponding to the score of the same value.

  Also, if it is determined that the emotion value <the third threshold and abnormal, the mood state level is determined to be throwing, and the emotion value E and the response classified as throwing in the response master are determined. Select and generate a response corresponding to the score of the same value.

  The generated response proceeds to one of steps S24 to S27 according to the classification, and is displayed on one of the output units 13b, 14b, 15b, and 16b. At this time, it is preferable to output to the output unit closest to the subject. The response output destination may be a speaker. That is, the response may be output from the speaker by voice. Further, both display and sound may be output. The character string to be output and conversion to speech can be performed using the character string converting device 27.

  Thus, by automatically outputting a response, it is possible to output accurate advice quickly without requiring knowledge or experience values.

  From step S24 to step S27, the process proceeds to step S28 to determine whether or not to continue for generation of the next response. When new information on the subject such as the next vital sign is acquired, it is determined that the information is to be continued, and the process returns to step S10.

On the other hand, if the acquisition unit 12c does not obtain the following information predetermined time, it takes a V3 (tn) by the end by equivalent time interval and the time of measurement, the difference between the measured time V3 (tn-1). If the difference is less than 0, it is considered to be stable, so it is determined not to continue, the process proceeds to step S29, the log of the process performed this time is recorded in the behavior characteristic information master, and the system is in a standby state. Migrate to On the other hand, if the difference is 0 or more, it is regarded as an abnormal state, and therefore the person to be diagnosed is inquired of whether or not to end. If the intention is to continue, the process returns to step S10, and if the intention is to end, the process proceeds to step S29. Note that the standby state may be the end of the program.

  FIG. 19 shows a list of specific examples of the above-described embodiment. This is an example, and the present invention is not limited to this.

  In the above embodiment, steps S21, S22, and S29 are not essential components. Moreover, what is necessary is just to be able to acquire several information from at least 1 type among vital signs, audio | voice information, and character information.

  Therefore, if the present invention has at least the following configuration, the intended effect of the present invention can be obtained. In other words, the biometric information, social environment information, and self-recognition information of the target person, the acquisition unit that acquires a plurality of information, and each coefficient used when obtaining the mood state level from the information that can be acquired by the acquisition unit A mood state master, a response master defined by associating the mood state level with a response, a storage unit storing the mood state master, the response master, and information acquired by the acquisition unit; Multimodal tracking comprising: a control unit that calculates a current mood state level based on the mood state master and the response master and generates a response; and an output unit that is an output destination of the generated response System.

  Note that if the vital signs and emotional abnormalities of the subject are not stable even after a certain period of time, the central device 12 may automatically send a contact to the emergency contact designated by the subject. In addition, when the target person is an elderly person, if the vital sign is not detected for a certain period of time, a notification may be automatically transmitted from the central device 12 to the emergency contact address. Alternatively, sleep apnea may be determined from the vital sign, and a warning sound may be emitted from a speaker when detected.

  In addition, the value of said various parameters is an example, and is not specifically limited. In addition, it is assumed that information of various masters including responses can be appropriately printed by a printing apparatus.

  The mental health care service mentioned above is a service that is performed during the period of studying for the introduction of treatment, such as when the patient is not feeling well, or when rehabilitation work is possible.

  The present invention is not limited to mental health care rehabilitation but can also be used for physical rehabilitation. Furthermore, it can be applied to orthopedic rehabilitation, gym training menu, diet menu, metabolic elimination menu, cram school assignment menu, workplace management support, and personnel evaluation system.

DESCRIPTION OF SYMBOLS 10 Multimodal type tracking system 12a Memory | storage part 12b Control part 12c Acquisition part 13a-16a Input part 13b-16b Output part 15, 16 Portable terminal device 15e, 16e, 22, 25 Speaker 26 Scale apparatus 27 Character string conversion apparatus

Claims (15)

An acquisition unit that acquires a plurality of pieces of information among the subject's biological information, social environment information, and self-recognition information;
A mood state master that defines each coefficient used when obtaining a mood state level from information that can be acquired by the acquisition unit;
A response master that defines the mood state level and the response in association with each other;
A storage unit storing the mood state master and the response master;
Based on the information acquired by the acquisition unit, the mood state master, and the response master, a control unit that calculates a current mood state level and generates a response;
An output unit that is an output destination of the generated response;
Equipped with a,
When one or more of the information acquired by the acquisition unit is a vital sign automatically detected by a vital sign detection device,
The controller is
The sum of values obtained by multiplying the fluctuation values from the start of acquisition of each acquired vital sign by the coefficient is the mood state level first calculated value,
The time differential value of the mood state level first calculated value is the mood state level second calculated value,
Multimodal as one element for determining whether or not the acquired vital sign is abnormal or stabilizing from the first mood state level calculated value and the second mood state level calculated value, and generating the response Type tracking system. The control unit further includes:
Calculating a time integral value of the mood state level second calculated value;
When the time interval at which the acquisition unit acquires the information exceeds a certain time, the first time integration value at the first first time point among the time points when the acquisition unit acquires the information, and the time point Calculating a difference from a second time integral value at a second time point immediately before the first time point, and if the difference is less than a predetermined threshold, the multimodal tracking system is waited or stopped, and the difference is 2. The multimodal tracking system according to claim 1, wherein the response is made to ask whether or not to wait or stop the multimodal tracking system if it is equal to or greater than the predetermined threshold value. The biological information, audio information, facial expression information, face temperature information, eye movement information, pulse information, heartbeat information, breathing information, according to claim 1 or 2, characterized in that at least one or more information in the blood pressure information The described multimodal tracking system. The multimodal tracking system according to any one of claims 1 to 3, wherein the self-recognition information is at least one or more of emotion information and personality tendency information. The social environment information, multimodal type tracking system according to any one of claims 1-4, characterized in that the personal space information. Multimodal type tracking system as claimed in any one of claims 1 to 5 information the acquisition unit acquires the which is a vital sign that is automatically detected by vital signs detection system. Multimodal type tracking system according to any one of claims 1 to 6 information the acquisition unit acquires the characterized in that the information input to the input device by the subject. The controller is
Comparing the mood state level first calculated value with a first threshold, comparing the mood state level second calculated value with a second threshold;
If the mood state level first calculated value> the first threshold and the mood state level second calculated value ≧ the second threshold, it is determined to be abnormal,
If the mood state level first calculated value> the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value <the first threshold and the mood state level second calculated value> the second threshold, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value <the first threshold value and the mood state level second calculated value ≦ the second threshold value, it is determined to be abnormal,
If the mood state level first calculated value = the first threshold value and the mood state level second calculated value> the second threshold value, it is determined that there is a tendency to stabilize,
If the mood state level first calculated value = the first threshold value and the mood state level second calculated value <the second threshold value, it is determined that there is a tendency to stabilize,
The multimodal tracking according to any one of claims 1 to 7, wherein when the mood state level first calculated value = the first threshold value and the mood state level second calculated value = the second threshold value, it is determined that there is no change. system. A scale conversion device for converting voice information into scales and calculating the number of syllables and voice values;
A character string converting device that converts voice information into a character string;
One of the mood state masters, and a scale master that defines voice states and scale coefficients corresponding to each other;
An emotion master that is one of the mood state masters and defines the emotion state and the emotion coefficient in correspondence with each other,
When one of the information acquired by the acquisition unit is audio information,
The controller is
The voice state is determined from the scaled result, the sum of the scale coefficient corresponding to the corresponding voice state and a constant specific to the subject is calculated, and the product of the calculated value and the voice value is calculated by the number of syllables. Divided into mood state level 3rd calculated value,
The number of words representing emotional state and emotion is determined from the result of characterizing, and the product of the emotional coefficient corresponding to the relevant emotional state and the number of words is calculated from the number of syllables within the measurement time from 1 The total value up to the mood state level 4th calculated value,
A value obtained by dividing a total value within the measurement time for the product of the third mood state level calculated value and the fourth mood state level calculated value by the number of syllables within the measurement time is used as an emotion value, and the response The multi-modal tracking system according to claim 8, wherein the multi-modal tracking system is an element that generates When one of the information acquired by the acquisition unit is character information,
The controller is
The number of syllables, emotional states, and the number of words representing emotions are determined from the character information, and the product of the emotion coefficient corresponding to the relevant emotional state and the number of words is calculated from the syllable number from 1 to the syllable within the measurement time. The multimodal tracking system according to claim 9, wherein a value summed up to a number is used as the fourth mood state level calculated value. The controller is
Comparing the emotion value with a third threshold;
If it is determined that the emotion value> the third threshold value and the stabilization tendency, the mood state level is determined to be praise, and the response classified as praise in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
When it is determined that the emotion value is the third threshold value and the stabilization tendency, the mood state level is determined to be a countermeasure, and the response classified as the countermeasure in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
If it is determined that the emotion value <the third threshold value and the stabilization tendency, the mood state level is determined to be throwing, and the response classified as throwing in the response master is Select and generate a response corresponding to the score equivalent to the emotion value,
If it is determined that the emotion value is greater than the third threshold and the change is not present, the mood state level is determined to be throwing, and the emotion is selected from the responses classified as throwing in the response master. Select and generate a response that corresponds to the score equivalent to the value,
When it is determined that the emotion value = the third threshold and the change is not present, the mood state level is determined to be consent, and the emotion is selected from among the responses classified as consent in the response master. Select and generate a response that corresponds to the score equivalent to the value,
If it is determined that the emotion value <the third threshold and the change is not present, the mood state level is determined to be agreement, and the emotion is selected from among responses classified as agreement in the response master. Select and generate a response that corresponds to the score equivalent to the value,
If it is determined that the emotion value is greater than the third threshold and the abnormality, the mood state level is determined to be consent, and the emotion value is selected from among responses classified as consent in the response master. Select and generate a response corresponding to a score equivalent to
If it is determined that the emotion value is the third threshold and the abnormality, the mood state level is determined to be consent, and the emotion value is selected from among the responses classified as consent in the response master. Select and generate a response corresponding to a score equivalent to
If it is determined that the emotion value <the third threshold value and the abnormality, the mood state level is determined to be throwing, and the emotion value is selected from the responses classified as throwing in the response master. 11. The multimodal tracking system according to claim 9 or 10, wherein a response corresponding to a score equivalent to is selected and generated. The storage unit includes a behavior characteristic information master that stores past personal data of the subject including the mood state level and the output response,
The multi-modal tracking system according to claim 1, wherein the control unit generates a response in consideration of information of the behavior characteristic information master.   The multimodal tracking system according to claim 1, wherein the output unit is a speaker.   The multimodal tracking system according to claim 1, wherein the output unit is mounted on a mobile terminal device.   A multimodal tracking program for operating the multimodal tracking system according to claim 1.