JP7715387B2 - Emotional movement estimation device, emotional movement estimation method, program, and recording medium - Google Patents

Description

The present invention relates to an emotional movement estimation device, an emotional movement estimation method, a program, and a recording medium.

When communicating face-to-face, it is possible to detect if the other person's behavior is different from usual from subtle differences in facial expressions and gestures. However, when communicating remotely via a camera, it is difficult to grasp the subtleties of the other person's facial expressions and emotions. Therefore, technology is known that estimates the emotions of the person communicating from their facial expressions, etc. (Patent Document 1).

Japanese Patent Application Laid-Open No. 2019-133447

However, there are individual differences in how emotions are expressed, and even if emotions are estimated using uniform standards, there is a problem in that it is not possible to determine whether they differ from the subject's usual state, i.e., how the subject's individual emotions have changed.

Therefore, the present invention aims to provide a device that can estimate the emotional changes of a subject.

In order to achieve the above object, the emotional movement estimation device of the present invention comprises:
The system includes a storage unit, an emotion information acquisition unit, a reference emotion vector calculation unit, a comparative emotion vector calculation unit, a differential emotion vector calculation unit, an estimation unit, and an output unit,
the storage unit includes vector transformation information;
the emotion information acquisition unit acquires emotion information of a subject;
the reference emotion vector calculation unit calculates a reference emotion vector for a first predetermined period based on the vector conversion information and the emotion information;
the comparative emotion vector calculation unit calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation unit calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation unit estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output unit outputs the emotional movement of the subject.

The emotional movement estimation method of the present invention includes:
The method includes an emotion information acquisition step, a reference emotion vector calculation step, a comparative emotion vector calculation step, a differential emotion vector calculation step, an estimation step, and an output step,
the emotion information acquiring step acquires emotion information of a subject,
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output step outputs the emotional movements of the subject.

The program of the present invention is a program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparative emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure,
the emotion information acquisition step acquires emotion information of a subject;
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
The differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output procedure is a program that outputs the emotional movements of the subject.

This invention makes it possible to estimate the emotional changes of a subject.

FIG. 1 is a block diagram showing an example of the configuration of a motion estimation device according to the first embodiment. FIG. 2 is a block diagram showing an example of the hardware configuration of the motion estimation device according to the first embodiment. FIG. 3 is a flowchart showing an example of processing in the motion estimation device according to the first embodiment. FIG. 4 is an explanatory diagram regarding estimation of emotional movements by the estimation unit in the movement estimation device according to the first embodiment. FIG. 5 is a block diagram showing an example of the configuration of a motion estimation device according to the second embodiment. FIG. 6 is a flowchart showing an example of processing in the motion estimation device of the second embodiment. FIG. 7 is a block diagram showing an example of the configuration of a detection unit in the motion estimation device according to the second embodiment. FIG. 8 is a block diagram showing an example of the hardware configuration of a detection unit in the motion estimation device according to the second embodiment. FIG. 9 is a flowchart showing an example of processing in the motion estimation device according to the second embodiment.

Embodiments of the present invention will be described using the drawings. The present invention is not limited to the following embodiments. In each of the following drawings, identical parts are designated by the same reference numerals. Furthermore, unless otherwise specified, the descriptions of each embodiment can be used interchangeably, and the configurations of each embodiment can be combined unless otherwise specified.

[Embodiment 1]
Figure 1 is a block diagram showing the configuration of an example of an emotional movement estimation device 10 according to this embodiment. As shown in Figure 1, this device 10 includes an emotion information acquisition unit 11, a reference emotion vector calculation unit 12, a comparison emotion vector calculation unit 13, a difference emotion vector calculation unit 14, an estimation unit 15, an output unit 16, and a memory unit 17.

The device 10 may be, for example, a single device including each of the above-mentioned components, or a device in which each of the above-mentioned components can be connected via a communication network. Furthermore, the device 10 can be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, and may be wired or wireless, for example. Examples of the communication network include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), and L5G (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (ad hoc communication), infrastructure communication, indirect communication via an access point, or the like. The device 10 may be incorporated into a server as a system. The device 10 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. The device 10 may also be in the form of cloud computing or edge computing, in which at least one of the components is located on a server and the other components are located on a terminal.

Figure 2 shows an example block diagram of the hardware configuration of the device 10. The device 10 includes, for example, a CPU 101, memory 102, a bus 103, a storage device 104, an input device 106, a display device 107, and a communication device 108. Each component of the device 10 is connected to each other via the bus 103 and their respective interfaces (I/F).

The CPU 101 operates in conjunction with other components via a controller (system controller, I/O controller, etc.) and is responsible for overall control of the device 10. In the device 10, the CPU 101 executes, for example, the program 105 of the present invention and other programs, and also reads and writes various types of information. Specifically, the CPU 101 functions as an emotion information acquisition unit 11, a reference emotion vector calculation unit 12, a comparative emotion vector calculation unit 13, a differential emotion vector calculation unit 14, an estimation unit 15, and an output unit 16. The device 10 includes a CPU as a computing device, but may also include other computing devices such as a GPU (Graphics Processing Unit) or APU (Accelerated Processing Unit), or may include a combination of a CPU and these.

The bus 103 can also be connected to, for example, external devices. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, and external imaging devices. The device 10 can be connected to an external network (the communication line network) via, for example, a communication device 108 connected to the bus 103, and can also be connected to other devices via the external network.

The memory 102 may be, for example, a main memory (primary storage device). When the CPU 101 performs processing, the memory 102 reads various operating programs, such as the program of the present invention, stored in the storage device 104 (described below), and the CPU 101 receives data from the memory 102 and executes the programs. The main memory may be, for example, a RAM (random access memory). The memory 102 may also be, for example, a ROM (read-only memory).

The storage device 104 is also referred to as an auxiliary storage device, in contrast to the main memory (primary storage device). As mentioned above, the storage device 104 stores an operating program including the program 105 of the present invention. The storage device 104 may be, for example, a combination of a recording medium and a drive that reads and writes from and to the recording medium. The recording medium is not particularly limited and may be internal or external, and examples include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, and memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or solid state drive (SSD) that integrates the recording medium and drive. The storage device 104 functions as the storage unit 17 and includes vector transformation information 171.

Vector conversion information 171 is information that serves as a reference for representing emotional information on two-dimensional coordinates. For example, on two-dimensional coordinates, the origin represents a state without emotional fluctuations (normal state), and the type of emotion is linked to the angle from the origin, and the intensity (score) of the emotion is linked to the distance from the origin. Using vector conversion information 171, the device 10 can convert the type of emotion of the subject into the angle from the origin, and the intensity (score) of the emotion into the distance from the origin. There are no particular restrictions on the link between the type of emotion and the angle from the origin in the two-dimensional coordinates, and any link can be used; however, it is preferable to use, for example, Russell's emotional circumplex model.

Russell's circumplex model of emotions is a model that depicts human emotions on a circle, with valence (pleasant-unpleasant) on one axis and arousal (arousal-non-arousal) on the other axis, perpendicular to the first. In Russell's circumplex model, for example, the direction of valence and arousal levels from the center (the intersection of the axes) represents the type of emotion, and the distance from the center represents the intensity of the emotion. According to Russell's circumplex model, emotions can be classified into alert, excited, energetic, happy, satisfied, cheerful, relaxed, calm, tired, lethargic, depressed, sad, anxious, stressed, nervous, and tense, for example, based on valence and arousal. Therefore, for example, by setting the intersection of the valence axis and the arousal axis in Russell's circumstantial model as the origin, and linking the type of emotion estimated by the emotion estimation engine described below with the type of emotion on Russell's circumstantial model, as well as the score of the emotion estimated by the emotion estimation engine and the distance from the origin, and storing these as vector conversion information 171, it becomes possible to easily represent the subject's emotions on two-dimensional coordinates.

In the present device 10, the memory 102 and storage device 104 can also store various information such as log information, information obtained from an external database (not shown) or external device, information generated by the present device 10, and information used by the present device 10 when executing processing. In this case, the memory 102 and storage device 104 may store at least one piece of information selected from the group consisting of, for example, emotional information of the subject, identification information of the subject, attribute information of the subject, score threshold, etc., as described below. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 102 and storage device 104, or may be stored in a distributed manner across multiple terminals using blockchain technology, etc.

The device 10 further includes, for example, an input device 106 and a display 107. Examples of the input device 106 include pointing devices such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; and audio input means such as a microphone. The display 107 includes, for example, a display device such as an LED display or LCD display. In the first embodiment, the input device 106 and the display 107 are configured separately, but the input device 106 and the display 107 may also be configured as an integrated device, such as a touch panel display. The device 10 may also include, for example, an imaging device such as a camera as the input device 106, and the CPU 101 may control the imaging device to acquire images.

Next, an example of the emotion movement estimation method of this embodiment will be described based on the flowchart in Figure 3. The movement estimation method of this embodiment is implemented as follows, for example, using the movement estimation device 10 of Figure 1 or Figure 2. Note that the movement estimation method of this embodiment is not limited to use with the movement estimation device 10 of Figure 1 or Figure 2.

First, the emotion information acquisition unit 11 acquires emotion information of the subject (S1, emotion information acquisition step). The emotion information includes, for example, the emotion of the subject and identification information of the subject. The identification information of the subject is information that can identify the subject, such as the subject's name; an image of the subject; the ID of the subject's device; or the username or ID used by the subject in a remote conference. The emotion information acquisition unit 11 may acquire the emotion information from the emotion estimation engine via a communication network, or may acquire the emotion information from an external database in which emotion information estimated by the emotion estimation engine is recorded. In the latter case, the emotion information acquisition unit 11 may, for example, acquire emotion-related information described below and estimate the emotion of the subject based on the emotion-related information to acquire the emotion information of the subject. In this case, the emotion information acquisition unit 11 may also be referred to as, for example, an emotion estimation unit.

The subject's emotion is, for example, information that estimates the subject's emotion based on emotion-related information about the subject. The emotion-related information is, for example, information about the state that appears in the subject in relation to the subject's emotion, and examples of such information include image information such as the subject's facial expression; vocal information such as voiced and unvoiced sounds; text information such as the content of speech; and vital data information such as blood pressure and heart rate. The emotion information is, for example, information estimated by a known emotion estimation engine based on the subject's emotion-related information. Specific examples of the emotion estimation engine include, but are not limited to, those that estimate emotions based on image information such as facial expressions, such as Affdex, Microsoft Azure Face API, Amazon Rekognition, Realeyes, and User Local facial expression estimation AI; those that estimate emotions based on audio information, such as emotion estimation software from STEmotion, Empath, and BeyondVerbal, IBM Watson Tone Analyzer, and User Local voice emotion recognition AI; those that estimate emotions using natural language processing based on text information, such as User Local text emotion recognition AI; and those that estimate emotions based on vital data, such as NEC emotion analysis solutions. The emotion estimation engine may be, for example, one type, or two or more types may be used in combination. The emotion estimation engine may be, for example, external to the device 10, or may be a component of the device 10.

The emotion information may include, for example, score information. The score information is information including, for example, a score indicating the degree of emotion for each type of emotion of the subject. The score may be, for example, a score calculated by the emotion estimation engine based on emotion-related information of the subject. The score may be, for example, qualitative information (for example, whether the emotion occurred or not), quantitative information (for example, the degree to which the emotion occurred), or information on the probability of occurrence of each emotion. Furthermore, the score may be, for example, an absolute evaluation of each type of emotion, or a relative evaluation with respect to other emotions (whether the emotion is stronger or weaker than other emotions).

The emotion information may also include, for example, other information. Examples of the other information include priority emotion information, attribute information of the subject, information on the date and time when the emotion was estimated, information such as the location where the emotion was estimated, and information on the situation in which the emotion was estimated. The priority emotion information is, for example, information linking the type of emotion with a priority. The priority is, for example, information indicating the weight of each type of emotion and is not particularly limited and can be set arbitrarily. Examples of the attribute information include information such as the gender, age, affiliation, and position in the organization of the subject. The attribute information of the subject may be linked to, for example, the identification information of the subject described above. Examples of the information on the date, time, and situation when the emotion was estimated include information on the date, time, and situation when the emotion-related information used as the basis for estimating the emotion was obtained. Specifically, when the emotion-related information is the estimation of the subject's emotion based on video information of a remote conference held by the subject, the date and time information is, for example, the time when the remote conference was held, the location information is, for example, information that identifies the remote conference, and the situation information is, for example, information about the participants of the remote conference, information about the agenda of the remote conference, etc. When the emotion information includes the other information, it is preferable that the emotion information and the other information are linked to each other, for example.

The emotion information acquisition unit 11 may acquire emotion information about one type of emotion of the subject, or emotion information about two or more types of emotions. In the latter case, the emotion information acquisition unit 11 may acquire emotion information about the subject for each type of emotion, for example. The type of emotion is not particularly limited, and examples include emotions that can be estimated by the emotion estimation engine. Specific examples of the emotion types include joy, anger, sadness, fear, disgust, contempt, and surprise. Furthermore, the emotion types may be, for example, multiple emotions shown in Russell's emotional circumplex model.

Next, the reference emotion vector calculation unit 12 calculates a reference emotion vector for a first predetermined period based on the vector conversion information and the emotion information (S2, reference emotion vector calculation step). The first predetermined period is, for example, a period considered to be the subject's usual state. The usual state is also referred to as, for example, a normal emotion. For example, the reference emotion vector calculation unit 12 converts the emotion type and emotion score for the first predetermined period, among the acquired emotion information, into an angle and distance from the origin based on the vector conversion information to calculate a provisional reference emotion vector for the first predetermined period. Then, the reference emotion vector calculation unit 12 combines the provisional reference emotion vectors for the first predetermined period to calculate a reference emotion vector. Alternatively, the reference emotion vector calculation unit 12 may combine the provisional reference emotion vectors to calculate a combined provisional reference emotion vector, and then divide the combined provisional reference emotion vector by the number of combined provisional reference emotion vectors to calculate the average of the combined provisional reference emotion vectors as the reference emotion vector. The first predetermined period is not particularly limited and may be, for example, a period longer than the second predetermined period described below, and may be measured in units of years, months, weeks, or days. Specific examples of the first predetermined period include 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, and 3 days. The reference emotion vector is also referred to as the subject's usual emotion vector, for example.

Furthermore, the reference emotion vector calculation unit 12 may, for example, calculate the reference emotion vector for all emotions included in the emotion information, or may extract emotions that satisfy a condition and calculate the reference emotion vector for the extracted emotion. The condition is not particularly limited and may be, for example, an emotion whose score per unit time during the first predetermined period exceeds a threshold, an emotion whose score per unit time during the first predetermined period is the highest, or an emotion whose priority per unit time during the first predetermined period is the highest. The unit time is not particularly limited and may be set to any unit, such as an hour or an event. If the unit time is in hours, examples include 10 minutes, 30 minutes, and 1 hour. If the unit time is in events, examples include the number of times an event occurs or a unit for each event segment (e.g., one class period (45 minutes) in an elementary school). The event is not particularly limited and examples include meetings such as web conferences, school classes, seminars, etc. The emotion extracted based on the condition is also referred to as a representative emotion.

The storage unit 17 may, for example, store the reference emotion vector in association with the subject's identification information. This makes it possible to create, for example, a database representing the reference emotion vector for each subject, i.e., the subject's usual emotions.

Next, the comparative emotion vector calculation unit 13 calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information (S3, comparative emotion vector calculation step). The second predetermined period is, for example, a period during which differences in the subject's emotions are examined (e.g., examined, investigated, tested, or confirmed). The comparative emotion vector calculation unit 13, for example, converts the acquired emotion information, including the emotion type and emotion score for the second predetermined period, into an angle and distance from the origin based on the vector conversion information, and calculates an emotion vector for the second predetermined period. Then, it combines the emotion vectors for the second predetermined period to calculate a comparative emotion vector. Alternatively, the comparative emotion vector calculation unit 13 may, for example, combine the provisional comparative emotion vectors to calculate a combined provisional comparative emotion vector, and then divide the combined provisional comparative emotion vector by the number of combined provisional comparative emotion vectors to calculate the average of the combined provisional comparative emotion vectors as the comparative emotion vector. The second predetermined period is not particularly limited and may be, for example, a period shorter than the first predetermined period, and may be measured in units of years, months, weeks, or days. Specific examples of the second predetermined period include 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, 1 day, and half a day. The comparative emotion vector is also referred to as the subject's current emotion vector, for example.

The comparative emotion vector calculation unit 13 may, for example, calculate the comparative emotion vector for all emotions included in the emotion information, or may extract emotions that satisfy a condition and calculate the comparative emotion vector for the extracted emotion. The condition is not particularly limited and may be, for example, an emotion whose score per unit time during the second specified period exceeds a threshold, an emotion whose score per unit time during the second specified period is the highest, or an emotion whose priority per unit time during the second specified period is the highest. The unit time is not particularly limited and can be set to any unit, such as an hour or an event. If the unit time is in hours, examples include 10 minutes, 30 minutes, and 1 hour. If the unit time is in events, examples include the number of times an event occurs or a unit for each event (e.g., one class period (45 minutes) in an elementary school). The event is not particularly limited and examples include meetings such as web conferences, school classes, seminars, etc. The emotion extracted based on the condition is also referred to as a representative emotion.

Next, the differential emotion vector calculation unit 14 calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector (S4, differential emotion vector calculation step). The differential emotion vector calculation unit 14 can use, for example, a normal vector calculation method to calculate the differential emotion vector. The differential emotion vector represents, for example, the difference between the subject's usual emotion and their current emotion, and is therefore also referred to as an emotion vector that represents a difference from usual.

Next, the estimation unit 15 estimates the emotional changes of the subject between the first and second predetermined periods based on the vector transformation information and the differential emotion vector (S5, emotion estimation process). Specifically, the estimation unit 15 can, for example, compare the direction of the differential emotion vector with the direction of the corresponding emotion on the two-dimensional coordinate system of the vector transformation information to estimate how the type of emotion of the subject has changed. Furthermore, since the length of the differential emotion vector corresponds to, for example, the degree (amount) of emotional change, the estimation unit 15 can estimate the amount of emotional change of the subject based on the length of the differential emotion vector. However, the processing by the estimation unit 15 is not limited to this. For example, the estimation unit 15 may estimate the emotional changes of the subject by creating an axis that connects two emotions located at opposite poles across the origin on the two-dimensional coordinate system and passes through the origin, and comparing the axis with the differential emotion vector. In this case, the estimation unit 15 can estimate the emotional movement of the subject from the direction of the differential emotion vector compared to the axis on the two-dimensional coordinate system and the length of the differential emotion vector on the two-dimensional coordinate system.

A specific example of the estimation of emotional movements by the estimation unit 15 will be described using Figure 4. Figure 4 is a diagram showing a specific example of vector transformation information 171, and shows examples of the reference emotion vector 121, the comparison emotion vector 131, and the difference emotion vector 141 on two-dimensional coordinates 1711 included in the vector transformation information 171. Note that the vector transformation information 171, the reference emotion vector 121, the comparison emotion vector 131, and the difference emotion vector 141 are not limited to the specific example shown in Figure 4. For example, as shown in Figure 4, the estimation unit 15 first connects the emotions of cheerfulness and depression, which are located at opposite poles, on the two-dimensional coordinates 1711, and creates an axis 1712 passing through the origin. Axis 1712 represents, for example, whether the subject's emotion is cheerful or depressed, and can therefore also be considered a vitality axis indicating the subject's level of vitality. When the differential emotion vector 141 has the direction and length shown in the specific example in FIG. 4, the estimation unit 15 can, for example, from the direction of the differential emotion vector 141 relative to the vitality axis 1712, estimate that the subject's emotion has changed in a depressed direction, i.e., that the subject has become less energetic, and from its length, estimate the extent of the change. For convenience of explanation, the axis 1712 has been described as an vitality axis connecting vitality and depression, but the processing by the estimation unit 15 is not limited to this. For example, the estimation unit 15 can create axes connecting any emotions located at opposite poles across the origin, such as excitement and lethargy, happiness and sadness, and compare each axis with the differential emotion vector to estimate the subject's emotional changes. Furthermore, the estimation unit 15 may, for example, estimate the subject's emotional changes based on one type of axis 1712, or may estimate the subject's emotional changes based on two or more types of axes 1712.

The output unit 16 then outputs the estimated emotional movement of the subject (S6, output step), and the process ends (END). The output unit 16 may, for example, output the reference emotion vector, comparison emotion vector, and differential emotion vector together.

Note that in this embodiment, S2 is performed before S3, but the emotional movement estimation method of the present invention is not limited to this. S2 and S3 may be performed, for example, in processes upstream of S4, and S2 may be performed after S3, or S2 and S3 may be performed simultaneously.

The movement estimation device 10 of this embodiment can estimate the emotional movement of a subject based on, for example, a differential emotion vector between a reference emotion vector and a comparison emotion vector. Therefore, the movement estimation device 10 of this embodiment can estimate how the emotional movement has changed, taking into account the individual's usual emotional state. This also makes it possible to estimate changes in the subject's state, particularly their mental state, such as how the subject's energy level has changed. For this reason, the emotional movement estimation device of the present invention can also be referred to as, for example, a subject's state estimation device.

[Embodiment 2]
This embodiment is similar to the estimation device 10 of embodiment 1, except that a detection unit is provided in addition to the configuration of the estimation device 10 of embodiment 1, and the description thereof can be cited. The estimation device of this embodiment includes a detection unit that detects differences in how the subject's emotions are manifested based on the emotion information, and the estimation unit estimates the emotional behavior of the subject when the detection unit detects differences in how the subject's emotions are manifested.

Figure 5 is a block diagram showing an example configuration of an estimation device 10A of this embodiment. As shown in Figure 5, the estimation device 10A includes a detection unit 17 in addition to the configuration of the estimation device 10 of embodiment 1. The hardware configuration of the estimation device 10A is the same as that of the estimation device 10 of Figure 2, except that the CPU 101 includes the configuration of the estimation device 10A of Figure 5 instead of the configuration of the estimation device 10 of Figure 1.

Next, the estimation method of this embodiment will be described using the flowchart in Figure 6. The estimation method of this embodiment can be implemented, for example, using the estimation device 10A of this embodiment shown in Figure 5. Note that the estimation method of the present invention is not limited to use with the estimation device 10A.

First, S1 is performed in the same manner as S1 in the difference estimation method of embodiment 1, and emotion information of the subject is obtained.

Next, the detection unit 21 detects differences in how the subject's emotions are manifested based on the emotion information (S11, detection step). The detection unit 21 can detect differences in how the subject's emotions are manifested by, for example, comparing emotion information from the first predetermined period with emotion information from a second predetermined period, and determining whether the emotion information from the second predetermined period includes an abnormal value. Note that the processing by the detection unit 21 is not limited to this, and differences may be detected by each step described in the specific examples below.

Next, if it is determined that the subject's current emotional expression is not different from usual (S11, No), the process ends (END). In this case, although not shown, for example, the output unit 16 may output that there is nothing abnormal in the subject's emotional expression. On the other hand, if it is determined that the subject's current emotional expression is different from usual (S11, YES), steps S2 to S6 are carried out in the same manner as steps S2 to S6 in the motion estimation method of embodiment 1.

With the estimation device of the present invention, if it is determined that the subject's current emotional state is different from usual, the estimation unit can estimate the subject's emotional movements.

[Embodiment 3]
Specifically, the detection unit 21 in the estimation device 10A of embodiment 2 will be described with reference to the accompanying drawings. FIG. 7 is a block diagram illustrating an example of the configuration of the detection unit 21. As illustrated in FIG. 7, the detection unit 21 includes, for example, an emotion information acquisition unit 211, a reference emotion counting unit 212, an emotion counting unit 213, an abnormal value detection unit 214, a determination unit 215, and an output unit 216. The detection unit 21 is also referred to as, for example, an emotion difference detection device 21, and the processing performed by the detection unit 21 is also referred to as, for example, an emotion difference detection method. The detection unit 21 may be, for example, a device included in the estimation device 10A, or may be a device separate from the estimation device 10A. In the latter case, the detection unit 21 (emotion difference detection device 21) and the estimation device 10A can communicate with each other. The communication may be, for example, wired or wireless.

The detection unit 21 (detection device 21) may be, for example, a single device including each of the above-mentioned components, or a device to which each of the above-mentioned components can be connected via a communication network. Furthermore, the detection device 21 can be connected to an external device (described later) via the communication network. The communication network is not particularly limited and any known network can be used, and may be wired or wireless, for example. Examples of the communication network include the Internet, WWW (World Wide Web), telephone lines, LAN (Local Area Network), SAN (Storage Area Network), DTN (Delay Tolerant Networking), LPWA (Low Power Wide Area), and L5G (Local 5G). Examples of wireless communication include Wi-Fi (registered trademark), Bluetooth (registered trademark), local 5G, and LPWA. The wireless communication may be a form in which each device communicates directly (ad hoc communication), infrastructure communication, indirect communication via an access point, or the like. The detection device 21 may be incorporated into a server as a system. The detection device 21 may also be, for example, a personal computer (PC, e.g., desktop or laptop) on which the program of the present invention is installed, a smartphone, a tablet terminal, or the like. The detection device 21 may also be, for example, a form of cloud computing or edge computing in which at least one of the above-mentioned components is located on a server and the other components are located on a terminal.

Figure 8 shows an example block diagram of the hardware configuration of the detection device 21. The detection device 21 includes, for example, a CPU 201, memory 202, bus 203, storage device 204, input device 206, display device 207, and communication device 208. Each component of the detection device 21 is connected to each other via the bus 203 and their respective interfaces (I/F).

The CPU 201 operates in cooperation with other components via a controller (system controller, I/O controller, etc.) and is responsible for overall control of the detection device 21. In the detection device 21, the CPU 201 executes, for example, the program 205 of the present invention and other programs, and also reads and writes various types of information. Specifically, the CPU 201 functions as, for example, an emotion information acquisition unit 211, a reference emotion counting unit 212, an emotion counting unit 213, an abnormal value detection unit 214, a judgment unit 215, and an output unit 216. The detection device 21 includes a CPU as a computing device, but may also include other computing devices such as a GPU (Graphics Processing Unit) or APU (Accelerated Processing Unit), or may include a combination of the CPU and these.

The bus 203 can also be connected to, for example, external devices. Examples of such external devices include external storage devices (external databases, etc.), printers, external input devices, external display devices, and external imaging devices. The detection device 21 can be connected to an external network (the communication line network) via, for example, a communication device 208 connected to the bus 203, and can also be connected to other devices via the external network.

The memory 202 may be, for example, a main memory (primary storage device). When the CPU 201 performs processing, the memory 202 reads various operating programs, such as the program of the present invention, stored in the storage device 204 (described below), and the CPU 201 receives data from the memory 202 and executes the programs. The main memory may be, for example, a RAM (random access memory). The memory 202 may also be, for example, a ROM (read-only memory).

The storage device 204 is also referred to as an auxiliary storage device, in contrast to the main memory (primary storage device). As mentioned above, the storage device 204 stores an operating program including the program 205 of the present invention. The storage device 204 may be, for example, a combination of a recording medium and a drive for reading and writing from and to the recording medium. The recording medium is not particularly limited and may be internal or external, and examples include HD (hard disk), CD-ROM, CD-R, CD-RW, MO, DVD, flash memory, and memory card. The storage device 104 may be, for example, a hard disk drive (HDD) or solid state drive (SSD) that integrates a recording medium and a drive. If the detection device 21 includes the storage unit, the storage device 204 functions as the storage unit, for example. The storage device 204 may store, for example, vector transformation information 171.

In the detection device 21, the memory 202 and storage device 204 can also store various information such as log information, information obtained from an external database (not shown) or external device, information generated by the detection device 21, and information used by the detection device 21 when executing processing. In this case, the memory 202 and storage device 204 may store at least one piece of information selected from the group consisting of, for example, the subject's emotion information, the subject's identification information, the subject's attribute information, the number of times a reference emotion has occurred, the number of times an emotion has occurred, and a score threshold, as described below. Note that at least a portion of the information may be stored, for example, in an external server other than the memory 202 and storage device 204, or may be stored in a distributed manner across multiple terminals using blockchain technology or the like.

The detection device 21 further includes, for example, an input device 206 and a display 207. Examples of the input device 206 include pointing devices such as a touch panel, trackpad, or mouse; a keyboard; imaging means such as a camera or scanner; a card reader such as an IC card reader or magnetic card reader; and audio input means such as a microphone. Examples of the display 207 include display devices such as an LED display or LCD display. In this embodiment, the input device 206 and the display 207 are configured separately, but the input device 206 and the display 207 may also be configured as an integrated unit, such as a touch panel display. Furthermore, the detection device 21 may include, for example, an imaging device such as a camera as the input device 206, and the CPU 201 may control the imaging device to acquire images.

Next, an example of the emotion movement estimation method of this embodiment will be described based on the flowchart in Figure 9. The movement estimation method of this embodiment is implemented as follows, for example, using a movement estimation device 10A including the difference detection device 21 of Figure 7 or Figure 8. Note that the movement estimation method of this embodiment is not limited to use of a movement estimation device 10A including the difference detection device 21 of Figure 7 or Figure 8.

First, the emotion information acquisition unit 211 acquires emotion information of the subject (S21, emotion information acquisition step). The emotion information is, for example, as described above. The emotion information acquisition unit 211 is, for example, similar to the emotion information acquisition unit 11 in the motion estimation device 10 of embodiment 1, and the description therein can be used. Alternatively, for example, the emotion information acquired in step S1 of embodiment 1 may be used.

Next, the reference emotion counting unit 212 counts the number of occurrences of a reference emotion based on the emotion information (S22, reference emotion counting step). The number of occurrences of the reference emotion is the number of times the emotion occurs per unit time of the subject during a first predetermined period. The first predetermined period is, for example, a period considered to be the subject's usual state. The usual state is also referred to as, for example, a normal emotion. The unit time is not particularly limited and can be set to any unit, such as an hour or an event. If the unit time is in hours, examples include 10 minutes, 30 minutes, and 1 hour. If the unit time is in events, examples include the number of times an event occurs, or a unit for each event segment (for example, one class period (45 minutes) in an elementary school). The event is not particularly limited and examples include meetings such as web conferences; school classes; seminars; etc. The reference emotion counting unit 212 may, for example, count all emotions that occurred per unit time during the first predetermined period among the emotion information of the subject acquired by the emotion information acquisition unit 211, or may count only emotions that satisfy a predetermined condition. The predetermined condition may be, for example, a case where the score for each emotion included in the emotion information exceeds an arbitrary threshold. In this case, the reference emotion counting unit 212, for example, determines whether the score exceeds a threshold for each type of emotion and counts the number of occurrences of emotions having a score exceeding the threshold as the reference emotion occurrence count. The threshold may, for example, be set to an arbitrary value for each type of emotion. By counting emotions having a score exceeding the threshold, for example, the influence of the accuracy of the emotion estimation engine used to estimate the emotion can be suppressed, and the reference emotion occurrence count can be more accurately counted. For example, the reference emotion counting unit 212 may use data obtained by removing outliers from the number of occurrences of emotions of the subject per unit time during the first predetermined period as the reference emotion occurrence count. The first predetermined period is not particularly limited and may be, for example, a period longer than the second predetermined period described below, and may be measured in units of, for example, years, months, weeks, or days. Specific examples of the first predetermined period include 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, and 3 days.

If the detection device 21 includes the memory unit, the memory unit may, for example, store the number of times the reference emotion occurred in association with the subject's identification information. This makes it possible to create, for example, a database representing the number of times the reference emotion occurred for each subject, i.e., the subject's usual emotions.

Next, the emotion counting unit 213 counts the number of emotion occurrences based on the emotion information (S23, emotion counting process). The number of emotion occurrences is the number of times the emotion occurred by the subject per unit time during a second predetermined period. The second predetermined period is, for example, a period during which differences in the subject's emotions are examined (e.g., examined, investigated, tested, or confirmed). The unit time is not particularly limited and can be set to any unit, such as an hour or an event. If the unit time is in hours, examples include 10 minutes, 30 minutes, and 1 hour. If the unit time is in events, examples include the number of times an event occurs, or a unit for each event segment (e.g., one class period (45 minutes) in an elementary school). The event is not particularly limited and examples include meetings such as web conferences; school classes; seminars; etc. Specifically, the emotion counting unit 213 may, for example, count all emotions occurring per unit time during the second predetermined period among the emotion information of the subject acquired by the emotion information acquisition unit 211, or may count only emotions that satisfy a predetermined condition. The predetermined condition may be, for example, when the score for each emotion included in the emotion information exceeds an arbitrary threshold. In this case, the emotion counting unit 213 determines, for example, for each type of emotion, whether the score exceeds a threshold, and counts the number of occurrences of emotions having a score exceeding the threshold as the number of emotion occurrences. The threshold may, for example, be set to an arbitrary value for each type of emotion. By counting emotions having a score exceeding the threshold, for example, the influence of the accuracy of the emotion estimation engine used to estimate the emotions can be suppressed, thereby enabling the number of emotion occurrences to be counted more accurately. The second predetermined period is not particularly limited and may, for example, be a period shorter than the first predetermined period, and the unit may be, for example, years, months, weeks, or days. Specific examples of the second predetermined period include 5 years, 3 years, 1 year, 6 months, 3 months, 1 month, 2 weeks, 3 days, 1 day, and half a day.

Next, the outlier detection unit 214 detects whether the number of emotion occurrences includes an outlier compared to the reference emotion occurrence count (S24, outlier detection step). The outlier detection unit 214 can detect outliers, for example, using a known outlier detection method based on a statistical technique in which the reference emotion occurrence count is used as training data and the emotion occurrence count is used as validation data. The outlier detection method may be, for example, an outlier detection method based on a statistical model, or an outlier detection method based on the distance between data. Specific examples of outlier detection methods include normal distribution, mixed normal distribution, Hotelling's theory, k-nearest neighbors, local outlier factor (LOF) method, principal component analysis (PCA), time series data, and one-class support vector machine (OCSVM), with detection using the LOF method or OCSVM being preferred.

Next, the determination unit 215 determines whether the number of emotion occurrences includes an abnormal value (S25 (S25A, S25B, S25C), determination step). The determination unit 215 determines whether the number of emotion occurrences includes an abnormal value (S25A). If the number of emotion occurrences includes an abnormal value, the determination unit 215 determines that the subject's current emotion is being manifested differently than usual (S25B). The determination unit 215 may, for example, determine that the subject's current emotion is being manifested differently than usual if the number of abnormal values included in the number of emotion occurrences exceeds a threshold, or may determine that the subject's current emotion is being manifested differently than usual if the proportion of abnormal values included in the number of emotion occurrences exceeds a threshold, or may determine that the subject's current emotion is being manifested differently than usual if the score of at least one abnormal value among the abnormal values included in the number of emotion occurrences exceeds a threshold.

If the determination unit 215 determines that the subject's current emotional expression is different from usual (S25A, YES), the output unit 216 outputs that the subject's emotional expression is different from usual (S26, output step). In S26, the output unit 216 may also output information such as the type of emotion for which an abnormal value was detected and the score of the emotion. Then, based on the emotional information acquired in S21, the reference emotional vector calculation unit 12, comparative emotional vector calculation unit 13, differential emotional vector calculation unit 14, estimation unit 15, and output unit 16 of the motion estimation device 10A perform S2 to S6 in the same manner as S2 to S6 in the motion estimation method of embodiment 1, and then the process ends (END).

If the number of emotion occurrences does not include an abnormal value, i.e., if the judgment unit 215 judges that the subject's current emotional expression is not different from usual (S25A, NO), the output unit 216 may output that there is nothing abnormal in the subject's emotional expression (S26, output step), and the processing may end (END).

Note that in this embodiment, S22 is performed before S23, but the motion estimation method of the present invention is not limited to this. S22 and S23 only need to be performed in processes upstream of S24. For example, S222 may be performed after S23, or S22 and S23 may be performed simultaneously.

The difference detection device of this embodiment can compare the subject's usual emotional expression with the subject's current emotional expression, for example, by comparing the number of times an emotion is expressed with a reference emotional expression count. Therefore, the difference detection device of this embodiment can determine whether the subject's emotional expression is different from usual. Therefore, a movement estimation device that includes the difference detection device of this embodiment as a difference detection unit can estimate the subject's emotional movement by the estimation unit when it is determined that the subject's current emotional expression is different from usual.

[Embodiment 4]
The program of this embodiment is a program for causing a computer to execute each of the steps of the emotional movement estimation method described above. Specifically, the program of this embodiment is a program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparison emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure.

the emotion information acquisition step acquires emotion information of a subject;
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the emotion information;
The differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the differential emotion vector;
The output step outputs the emotional movement of the subject.

The program of this embodiment can also be described as a program that causes a computer to function as an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparative emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure.

The program of this embodiment can use the descriptions of the motion estimation device and motion estimation method of the present invention. For each of the steps, for example, "step" can be read as "processing." The program of this embodiment may also be recorded on a computer-readable recording medium. The recording medium is, for example, a non-transitory computer-readable storage medium. The recording medium is not particularly limited, and examples include random access memory (RAM), read-only memory (ROM), hard disk (HD), optical disk, and floppy disk (FD).

The present invention has been described above with reference to embodiments, but the present invention is not limited to the above embodiments. Various modifications that would be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

<Additional Notes>
Some or all of the above embodiments can be described as, but are not limited to, the following supplementary notes.
(Appendix 1)
The system includes a storage unit, an emotion information acquisition unit, a reference emotion vector calculation unit, a comparative emotion vector calculation unit, a differential emotion vector calculation unit, an estimation unit, and an output unit,
the storage unit includes vector transformation information;
the emotion information acquisition unit acquires emotion information of a subject;
the reference emotion vector calculation unit calculates a reference emotion vector for a first predetermined period based on the vector conversion information and the emotion information;
the comparative emotion vector calculation unit calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation unit calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation unit estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output unit outputs the emotional movement of the subject.
(Appendix 2)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation unit
extracting emotions for which the scores exceed a threshold for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting emotions for which the scores exceed a threshold for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
2. The estimation device according to claim 1, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 3)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation unit
extracting the emotion with the highest score for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting the emotion with the highest score for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
3. The estimation device according to claim 1, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 4)
the emotion information includes priority emotion information,
The priority emotion information is information in which the type of emotion is associated with a priority level,
The reference emotion vector calculation unit
extracting an emotion with the highest priority for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting the emotion with the highest priority for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
4. The estimation device according to claim 1, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 5)
the emotion information includes identification information of the subject;
5. The estimation device according to claim 1, wherein the storage unit stores identification information of the subject and the reference emotion vector in association with each other.
(Appendix 6)
A detection unit is included,
the detection unit detects differences in how the subject's emotions are expressed based on the emotion information;
6. The estimation device according to claim 1, wherein the estimation unit estimates the emotional movement of the subject when the detection unit detects a difference in how the subject's emotions are expressed.
(Appendix 7)
the detection unit includes an emotion information acquisition unit, a reference emotion counting unit, an emotion counting unit, an abnormal value detection unit, a determination unit, and an output unit;
the emotion information acquisition unit acquires emotion information of a subject;
the emotion information includes an emotion of the subject and identification information of the subject;
the reference emotion counting unit counts the number of occurrences of a reference emotion based on the emotion information;
the reference emotion occurrence count is the number of times the emotion of the subject occurs per unit time in a first predetermined period,
the emotion counting unit counts the number of occurrences of emotions based on the emotion information;
the number of occurrences of an emotion is the number of occurrences of the emotion of the subject per unit time in a second predetermined period,
the abnormal value detection unit detects whether the number of emotion occurrences includes an abnormal value with respect to the reference number of emotion occurrences;
When the number of times of emotion occurrence includes an abnormal value, the determination unit determines that the subject's current emotion is being expressed differently than usual;
7. The emotion estimation device according to claim 6, wherein an output unit outputs the determination result.
(Appendix 8)
a storage unit,
the storage unit stores the identification information of the subject and the reference emotion occurrence count in association with each other;
8. The estimation device according to claim 7, wherein the determination unit determines whether there is a difference between the stored reference emotion occurrence count and the emotion occurrence count.
(Appendix 9)
the emotion information acquisition unit acquires emotion information of the subject for each emotion type;
the reference emotion counting unit counts the number of occurrences of the reference emotion for each type of emotion;
9. The estimation device according to claim 7, wherein the emotion counting unit counts the number of occurrences of the emotion for each type of emotion.
(Appendix 10)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
the reference emotion counting unit determines whether or not the score exceeds a threshold for each type of emotion, and counts the number of occurrences of emotions having scores exceeding the threshold as the number of occurrences of the reference emotion;
10. The estimation device according to any one of appendices 7 to 9, wherein the emotion counting unit determines, for each type of emotion, whether the score exceeds a threshold, and counts, as the number of occurrences of emotions, the number of occurrences of emotions having a score exceeding the threshold.
(Appendix 11)
11. The estimation device according to any one of appendices 7 to 10, wherein the determination unit determines that the subject's current emotion is being expressed differently than usual when the number of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 12)
11. The estimation device according to any one of appendices 7 to 10, wherein the determination unit determines that the subject's current emotional expression is different from usual when a proportion of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 13)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
11. The estimation device according to any one of appendices 7 to 10, wherein the determination unit determines that the subject's current emotion is being expressed differently than usual when a score of at least one abnormal value among the abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 14)
The method includes an emotion information acquisition step, a reference emotion vector calculation step, a comparative emotion vector calculation step, a differential emotion vector calculation step, an estimation step, and an output step,
the emotion information acquiring step acquires emotion information of a subject,
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output step outputs the emotional movement of the subject.
(Appendix 15)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation step includes:
extracting emotions for which the scores exceed a threshold for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation step includes:
extracting emotions for which the scores exceed a threshold for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
15. The estimation method according to claim 14, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 16)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation step includes:
extracting the emotion with the highest score for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation step includes:
extracting the emotion with the highest score for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
16. The estimation method according to claim 14, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 17)
the emotion information includes priority emotion information,
The priority emotion information is information in which the type of emotion is associated with a priority level,
The reference emotion vector calculation step includes:
extracting an emotion with the highest priority for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation step includes:
extracting the emotion with the highest priority for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
17. The estimation method according to any one of appendices 14 to 16, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 18)
a storing step,
the emotion information includes identification information of the subject;
18. The estimation method according to any one of appendices 14 to 17, wherein the storing step stores the identification information of the subject and the reference emotion vector in association with each other.
(Appendix 19)
a detection step,
the detecting step detects differences in how the subject's emotions are expressed based on the emotion information;
19. The estimation method according to any one of appendices 14 to 18, wherein the estimation step estimates the emotional movement of the subject when a difference in the way the subject's emotions are expressed is detected by the detection step.
(Appendix 20)
the detection step includes an emotion information acquisition step, a reference emotion counting step, an emotion counting step, an abnormal value detection step, a determination step, and an output step;
the emotion information acquiring step acquires emotion information of a subject,
the emotion information includes an emotion of the subject and identification information of the subject;
the reference emotion counting step counts the number of occurrences of a reference emotion based on the emotion information;
the reference emotion occurrence count is the number of times the emotion of the subject occurs per unit time in a first predetermined period,
the emotion counting step counts the number of occurrences of emotions based on the emotion information;
the emotion occurrence count is the number of times the emotion occurs per unit time of the subject during a second predetermined period;
the abnormal value detection step detects whether the number of occurrences of emotions includes an abnormal value with respect to the reference number of occurrences of emotions;
The determining step determines that the subject's current emotional expression is different from usual when the number of times the emotion is generated includes an abnormal value,
20. The emotion estimation method according to claim 19, wherein the output step outputs the determination result.
(Appendix 21)
a storing step,
the storing step stores the identification information of the subject and the number of occurrences of the reference emotion in association with each other;
21. The estimation method according to claim 20, wherein the determining step determines whether or not there is a difference between the stored reference emotion occurrence count and the emotion occurrence count.
(Appendix 22)
the emotion information acquiring step acquires emotion information of the subject for each emotion type;
the reference emotion counting step counts the number of occurrences of the reference emotion for each type of emotion;
22. The estimation method according to claim 20, wherein the emotion counting step counts the number of occurrences of the emotion for each type of emotion.
(Appendix 23)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
the reference emotion counting step determines whether or not the score exceeds a threshold for each type of emotion, and counts the number of occurrences of emotions having scores exceeding the threshold as the number of occurrences of the reference emotion;
23. The estimation method according to any one of Appendices 20 to 22, wherein the emotion counting step determines whether or not the score exceeds a threshold for each type of emotion, and counts the number of occurrences of emotions having a score exceeding the threshold as the number of occurrences of emotions.
(Appendix 24)
24. The estimation method according to any one of appendices 20 to 23, wherein the determination step determines that the subject's current emotional expression is different from usual when the number of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 25)
24. The estimation method according to any one of appendices 20 to 23, wherein the determination step determines that the subject's current emotional expression is different from usual when a proportion of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 26)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
24. The estimation method according to any one of Appendices 20 to 23, wherein the determination step determines that the subject's current emotional expression is different from usual when the score of at least one abnormal value among the abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 27)
A program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparative emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure,
the emotion information acquisition step acquires emotion information of a subject;
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
The differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output step outputs the emotional movements of the subject.
(Appendix 28)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation procedure includes:
extracting emotions for which the scores exceed a threshold for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation procedure includes:
extracting emotions for which the scores exceed a threshold for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
28. The program according to claim 27, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 29)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation procedure includes:
extracting the emotion with the highest score for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation procedure includes:
extracting the emotion with the highest score for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
29. The program according to claim 27, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 30)
the emotion information includes priority emotion information,
The priority emotion information is information in which the type of emotion is associated with a priority level,
The reference emotion vector calculation procedure includes:
extracting an emotion with the highest priority for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation procedure includes:
extracting the emotion with the highest priority for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
30. The program according to any one of appendices 27 to 29, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector.
(Appendix 31)
causing the computer to execute a memorized procedure;
the emotion information includes identification information of the subject;
31. The program according to any one of appendices 27 to 30, wherein the storage step stores the identification information of the subject and the reference emotion vector in association with each other.
(Appendix 32)
causing the computer to perform a detection procedure;
the detecting step detects differences in how the subject's emotions are expressed based on the emotion information;
32. The program described in any one of Appendices 27 to 31, wherein the estimation step estimates the emotional movement of the subject when the detection step detects a difference in how the subject's emotions are expressed.
(Appendix 33)
the detection procedure includes an emotion information acquisition procedure, a reference emotion counting procedure, an emotion counting procedure, an outlier detection procedure, a determination procedure, and an output procedure;
the emotion information acquisition step acquires emotion information of a subject;
the emotion information includes an emotion of the subject and identification information of the subject;
the reference emotion counting step counts the number of occurrences of a reference emotion based on the emotion information;
the reference emotion occurrence count is the number of times the emotion of the subject occurs per unit time in a first predetermined period,
the emotion counting step counts the number of occurrences of emotions based on the emotion information;
the emotion occurrence count is the number of times the emotion occurs per unit time of the subject during a second predetermined period;
the abnormal value detection step detects whether the number of occurrences of emotions includes an abnormal value with respect to the reference number of occurrences of emotions;
The determination step determines that the subject's current emotional expression is different from usual when the number of times the emotion occurs includes an abnormal value;
33. The emotion program according to claim 32, wherein the output step outputs the determination result.
(Appendix 34)
Includes memory procedures,
the storing step includes storing the identification information of the subject and the number of occurrences of the reference emotion in association with each other;
34. The program according to claim 33, wherein the determining step determines whether or not there is a difference between the stored reference emotion occurrence count and the emotion occurrence count.
(Appendix 35)
the emotion information acquisition step acquires emotion information of the subject for each emotion type;
the reference emotion counting step counts the number of occurrences of the reference emotions for each type of emotion;
35. The program according to claim 33, wherein the emotion counting step counts the number of occurrences of the emotion for each type of emotion.
(Appendix 36)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
the reference emotion counting step includes determining whether or not the score exceeds a threshold for each type of emotion, and counting the number of occurrences of emotions having scores exceeding the threshold as the reference emotion occurrence count;
36. The program according to any one of Appendices 33 to 35, wherein the emotion counting step determines, for each type of emotion, whether the score exceeds a threshold, and counts, as the number of occurrences of emotions, the number of occurrences of emotions having a score exceeding the threshold.
(Appendix 37)
37. The program described in any one of Appendices 33 to 36, wherein the determination procedure determines that the subject's current emotional expression is different from usual if the number of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 38)
37. The program described in any one of Appendices 33 to 36, wherein the determination procedure determines that the subject's current emotional expression is different from usual if the proportion of abnormal values included in the emotion occurrence count exceeds a threshold.
(Appendix 39)
the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
37. The program described in any one of Appendices 33 to 36, wherein the determination procedure determines that the subject's current emotional expression is different from usual if the score of at least one abnormal value included in the number of emotion occurrences exceeds a threshold.
(Appendix 40)
A computer-readable recording medium having recorded thereon a program according to any one of appendices 27 to 39.

According to the present invention, the emotional movements of a subject can be estimated based on the differential emotion vector between a reference emotion vector and a comparison emotion vector. Therefore, the estimation device of the present invention can estimate how an individual's emotional movements have changed, taking into account their usual emotional state. This also makes it possible to estimate changes in the subject's condition, particularly their mental state, such as how the subject's energy level has changed. Therefore, the present invention is particularly useful in fields such as human resource management and health management in remote work and remote classes.

10, 10A Motion estimation device 11 Emotion information acquisition unit 12 Reference emotion vector calculation unit 13 Emotion vector calculation unit 14 Difference emotion vector calculation unit 15 Estimation unit 16 Output unit 17 Storage unit 101 CPU
102 Memory 103 Bus 104 Storage device 105 Program 106 Input device 107 Display device 108 Communication device 21 Detection unit (difference detection device)
211 emotion information acquisition unit 212 reference emotion counting unit 213 emotion counting unit 214 abnormal value detection unit 215 determination unit 216 output unit 201 CPU
202 Memory 203 Bus 204 Storage device 205 Program 206 Input device 207 Display device 208 Communication device

Claims (9)

The system includes a storage unit, an emotion information acquisition unit, a reference emotion vector calculation unit, a comparative emotion vector calculation unit, a differential emotion vector calculation unit, an estimation unit, and an output unit,
the storage unit includes vector transformation information;
the emotion information acquisition unit acquires emotion information of a subject;
the reference emotion vector calculation unit calculates a reference emotion vector for a first predetermined period based on the vector conversion information and the emotion information;
the comparative emotion vector calculation unit calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation unit calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation unit estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output unit outputs the emotional movement of the subject. the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation unit
extracting emotions for which the scores exceed a threshold for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting emotions for which the scores exceed a threshold for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
The estimation device according to claim 1 , wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector. the emotion information includes score information,
the score information is information including a score indicating the degree of each emotion type,
The reference emotion vector calculation unit
extracting the emotion with the highest score for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting the emotion with the highest score for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
3. The estimation device according to claim 1, wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector. the emotion information includes priority emotion information,
The priority emotion information is information in which the type of emotion is associated with a priority level,
The reference emotion vector calculation unit
extracting an emotion with the highest priority for each unit time during the first predetermined period;
Calculating a provisional reference emotion vector for the extracted emotion based on the vector conversion information;
calculating the reference emotion vector based on the provisional reference emotion vector;
The comparative emotion vector calculation unit
extracting an emotion with the highest priority for each unit time during the second predetermined period;
Calculating a provisional comparison emotion vector for the extracted emotion based on the vector conversion information;
The estimation device according to claim 1 , wherein the comparative emotion vector is calculated based on the provisional comparative emotion vector. the emotion information includes identification information of the subject;
The estimation device according to claim 1 , wherein the storage unit stores the identification information of the subject and the reference emotion vector in association with each other. A detection unit is included,
the detection unit detects differences in how the subject's emotions are expressed based on the emotion information;
The estimation device according to claim 1 , wherein the estimation unit estimates the emotional behavior of the subject when the detection unit detects a difference in how the subject's emotions are expressed. The method includes an emotion information acquisition step, a reference emotion vector calculation step, a comparative emotion vector calculation step, a differential emotion vector calculation step, an estimation step, and an output step,
the emotion information acquiring step acquires emotion information of a subject,
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
the differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output step outputs the emotional movement of the subject. A program for causing a computer to execute an emotion information acquisition procedure, a reference emotion vector calculation procedure, a comparative emotion vector calculation procedure, a differential emotion vector calculation procedure, an estimation procedure, and an output procedure,
the emotion information acquisition step acquires emotion information of a subject;
the reference emotion vector calculation step calculates a reference emotion vector for a first predetermined period based on vector conversion information and the emotion information;
the comparative emotion vector calculation step calculates a comparative emotion vector for a second predetermined period based on the vector conversion information and the emotion information;
The differential emotion vector calculation step calculates a differential emotion vector based on the reference emotion vector and the comparison emotion vector;
the estimation step estimates a change in the subject's emotions between a first predetermined period and a second predetermined period based on the vector conversion information and the differential emotion vector;
The output step outputs the emotional movements of the subject. A computer-readable recording medium on which the program according to claim 8 is recorded.