JP7379788B2 - Emotion estimation device and emotion estimation system - Google Patents

Description

The present invention relates to an emotion estimation device and an emotion estimation system.

2. Description of the Related Art In recent years, techniques have been known for estimating the emotions a user feels, such as joy, anger, sadness, and normality, taking into consideration the user's situation. For example, in Patent Document 1, features corrected according to the user's situation are input into a learning model that has already learned the relationship between the features of the user's voice information and emotions. A technology for outputting emotions has been disclosed.

Japanese Patent Application Publication No. 2018-072876

However, in situations where the user's voice does not express much emotion, it is difficult to determine whether it is difficult for the user to express emotion in the voice, or whether the user is not expressing emotion in the first place. , it is difficult to accurately judge the emotions a user feels.

An emotion estimation device according to a preferred aspect of the present invention includes a generation unit that generates a feature amount for the user's voice based on the user's voice information, and a first emotion that the user feels based on the feature amount. a first evaluation unit that generates a first voice evaluation value indicating the intensity and a second voice evaluation value indicating the intensity with which the user feels a second emotion; and a first evaluation unit that indicates the content of the user's utterance based on the voice information. a recognition unit that generates a recognized character string; a first character evaluation value that indicates the strength with which the user feels the first emotion based on the recognition character string; and a first character evaluation value that indicates the strength with which the user feels the second emotion. a second evaluation unit that generates a two-character evaluation value; and a second evaluation unit that generates the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second voice evaluation value based on situation information indicating the user's situation. It includes a correction section that corrects the second character evaluation value, and an estimation section that estimates one or more emotions that the user feels based on the correction result of the correction section.

An emotion estimation system according to a preferred aspect of the present invention is an emotion estimation system comprising the above-described emotion estimation device and a terminal device capable of communicating with the emotion estimation device, wherein the terminal device receives the user's voice. a sound collection device that collects sound; a situation information generation unit that generates situation information indicating the user's situation; a communication device that receives, from the device, the recognized character string and emotional information indicating one or more emotions held by the user estimated by the estimation unit; and an output unit that outputs information obtained by executing the process on the column.

According to the present invention, even in a situation where the user's voice expresses little emotion, it is possible to accurately estimate the user's emotion.

FIG. 1 is a diagram showing an overview of a user device 1. FIG. FIG. 1 is a block diagram showing the configuration of a user device 1 according to a first embodiment. The figure which shows an example of the memory content of the dictionary information for analysis 31. FIG. 3 is a diagram showing an example of the stored content of emotion classification information 33; FIG. 3 is a diagram showing an example of stored contents of schedule information 35; FIG. 2 is a diagram showing an overview of the functions of the user device 1. FIG. FIG. 3 is a diagram showing an example of stored contents of situation related information 37; 2 is a flowchart showing the operation of the user device 1. FIG. 2 is a block diagram showing a user device 1a according to a second embodiment. FIG. 3 is a diagram showing an overview of the functions of a user device 1a in a second embodiment. FIG. 3 is a diagram illustrating an example of stored contents of character string related information 38; Flowchart showing the operation of user device 1a. The figure which shows the whole structure of emotion estimation system SYS. FIG. 2 is a block diagram showing the configuration of a user device 1b. 1 is a block diagram showing the configuration of a server device 10. FIG.

1. First Embodiment FIG. 1 is a diagram showing an overview of a user device 1. The user device 1 is assumed to be a smartphone. The user device 1 is an example of an "emotion estimation device." However, any information processing device can be used as the user device 1, and for example, it may be a terminal type information device such as a personal computer, or a portable type information device such as a notebook computer, wearable terminal, or tablet terminal. It may be an information terminal.

The user device 1 has a function of transmitting a recognized character string obtained by performing voice recognition processing on voice information indicating a sound including the voice of the user U who owns the user device 1 to a device used by another person. Alternatively, the user U has a function of emitting a sound indicating the recognized character string to be heard by others located near the user U. Furthermore, the user device 1 estimates the emotion that the user U has based on the voice of the user U, and adds a figure corresponding to the estimated emotion to the recognized character string, or adds a figure corresponding to the estimated emotion to the recognized character string. By emitting a sound indicating the recognized character string with an intonation corresponding to the character string, it is possible to add the emotional expression necessary for communication.
In the example of FIG. 1, the user U utters "Hello", and the user device 1 adds a graphic PI corresponding to the estimated emotion to the image representing the recognized character string.

FIG. 2 is a block diagram showing the configuration of the user device 1 according to the first embodiment. The user device 1 is realized by a computer system including a processing device 2, a storage device 3, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS (Global Positioning System) device 8. Each element of the user equipment 1 is interconnected by one or more buses 9 for communicating information. Note that the term "apparatus" in this specification may be replaced with other terms such as circuit, device, or unit. Further, each element of the user device 1 may be composed of one or more devices, and some elements of the user device 1 may be omitted.

The processing device 2 is a processor that controls the entire user device 1, and is composed of, for example, a single chip or a plurality of chips. The processing device 2 includes, for example, a central processing unit (CPU) that includes an interface with peripheral devices, an arithmetic unit, a register, and the like. Note that some or all of the functions of the processing device 2 may be realized by hardware such as a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). It's okay. The processing device 2 executes various processes in parallel or sequentially.

The storage device 3 is a recording medium readable by the processing device 2, and stores a plurality of programs including a control program PR executed by the processing device 2, analysis dictionary information 31, emotion classification information 33, schedule information 35, and situation related information. 37, and store the learning model LM. The storage device 3 is composed of one or more types of storage circuits, such as ROM (Read Only Memory), EPROM (Erasable Programmable ROM), EEPROM (Electrically Erasable Programmable ROM), and RAM (Random Access Memory).

FIG. 3 is a diagram showing an example of the stored contents of the analysis dictionary information 31. The analysis dictionary information 31 is information in which part of speech, subclassification of part of speech, and original form information are associated with each other for each morpheme. A morpheme is a character string that is the minimum unit of a meaningful expressive element. Parts of speech are types of words classified by grammatical properties, such as nouns, verbs, and adjectives. Part-of-speech subclassifications are items that further subdivide parts of speech. The original form information is a character string indicating the original form of the word when the relevant morpheme is a word to be used, and is the same character string as the relevant morpheme when the relevant morpheme is a word that is not used.

FIG. 4 is a diagram showing an example of the stored contents of the emotion classification information 33. The emotion classification information 33 is information in which a character string is classified as joy, anger, sadness, or normal. In the example of FIG. 4, the character string group 331 classified as joy includes "happy", "pass", "win", "win", and the like. Similarly, the character string group 332 classified as anger includes "irritated", "angry", and the like. Similarly, the character string group 333 classified as sad includes words such as "sad" and "defeat." Similarly, the character string group 334 classified as normal includes words such as "safety".

FIG. 5 is a diagram showing an example of the stored contents of the schedule information 35. Schedule information 35 indicates user U's schedule. The schedule information 35 shown in FIG. 5 has records 35_1 to 35_3. Record 35_1 indicates that user U's schedule from 10:00 to 11:00 on April 10, 2019 is a meeting with a client. Record 35_2 indicates that user U's schedule from 15:00 to 16:00 on April 12, 2019 is to attend an internal meeting. Record 35_3 indicates that user U's schedule from 18:00 to 20:00 on April 15, 2019 is to attend a class reunion.

The explanation returns to FIG. 2. The situational information 37 is used when estimating emotions. The learning model LM has already learned the relationship between the feature amount corresponding to human voice and the intensity for each of a plurality of emotions. Further, it is preferable that the learning model LM has already learned, for a plurality of humans, the relationship between feature amounts corresponding to human voices and intensities for each of a plurality of emotions.

The input device 4 is a device for inputting information used by the user device 1 to the processing device 2. Input device 4 includes a sound collection device 41 and a touch panel 43. The output device 5 is a device for outputting information. The output device 5 includes a display device 51 and a sound emitting device 53.

The sound collection device 41 includes, for example, a microphone and an AD converter, and collects sounds including the user's U voice under the control of the processing device 2 . A microphone converts the collected sound into an electrical signal. The AD converter performs AD conversion on the electrical signal converted by the microphone, and converts it into audio information VI shown in FIG. The sound indicated by the audio information VI may include noise emitted from around the speaker in addition to the speaker's voice. The touch panel 43 detects contact with the display surface of the display device 51. Note that the touch panel 43 may include a plurality of operators that can be operated by the user U.

The display device 51 displays various images under the control of the processing device 2. For example, various display panels such as a liquid crystal display panel and an organic EL (electro-luminescence) display panel are suitably used as the display device 51. The sound emitting device 53 is composed of, for example, a speaker, and emits sound under the control of the processing device 2.

The communication device 6 is hardware (transmission/reception device) for communicating with other devices via a network. The communication device 6 is also called, for example, a network device, a network controller, a network card, a communication module, or the like.

The inertial sensor 7 measures the inertial force applied to the user device 1 and outputs inertial information IFI shown in FIG. 6 obtained from the measurement results. For example, the inertial sensor 7 is one or both of an acceleration sensor and an angular velocity sensor.

The GPS device 8 receives radio waves from a plurality of satellites and generates position information PoI shown in FIG. 6. The position information PoI may be in any format as long as the position can be specified. The location information PoI indicates, for example, the latitude and longitude of the user device 1. Further, in this embodiment, the location information PoI is obtained from the GPS device 8, but the user device 1 may obtain the location information PoI by any method. For example, the user equipment 1 acquires a cell ID (IDentifier) assigned to a base station with which the user equipment 1 communicates as the location information PoI. The cell ID is identification information that uniquely identifies a base station. Furthermore, when the user device 1 communicates with an access point of a wireless LAN (Local Area Network), it also includes an identification address (MAC (Media Access Control) address) on the network assigned to the access point and the actual address (location). The location information PoI may be obtained by referring to a database that associates the information with the location information PoI.

1.1. Functions of the First Embodiment The processing device 2 functions as the acquisition section 21, the situation information generation section 23, the emotional information generation section 25, and the output section 27 by reading and executing the control program PR from the storage device 3.
The functions realized by the processing device 2 will be explained using FIG. 6.

FIG. 6 is a diagram showing an overview of the functions of the user device 1. The acquisition unit 21 acquires audio information VI output by the sound collection device 41.

The situation information generation unit 23 generates situation information SiI indicating the situation of the user U based on the position information PoI obtained from the GPS device 8, the inertial information IFI obtained from the inertial sensor 7, and the schedule information 35. More specifically, the situation information generation unit 23 identifies the location of the user U based on the location information PoI. The locations include, for example, the user U's home, the user U's office, and the commuting route from the user U's home to the office. Further, the situation information generation unit 23 determines whether the user U is stopped, walking, or moving in a vehicle based on the inertial information IFI. Then, when the identified location is a home, the situation information generation unit 23 generates identification information indicating a situation in which the user U is at home as the situation information SiI. Furthermore, if the identified location is within the commuting route and the user U is traveling in a vehicle, the situation information generation unit 23 indicates, as situation information SiI, the situation in which the user U is using the train. Generate identification information. The situation of using a train is an example of the "situation of using public transportation." Further, when the schedule information 35 indicates a meeting or a conference regarding the user U's schedule at the current time, the status information generation unit 23 indicates, as status information SiI, a status where the user U is in a conference. Generate identification information.

The emotion information generation unit 25 estimates one or more emotions held by the user U from among the plurality of emotions held by the user U. In the first embodiment, the description will be made assuming that the plurality of emotions that the user U has are four: joy, anger, sadness, and normalcy. Below, joy, anger, sadness, and normalcy are examples of a plurality of emotions.

The emotion information generation section 25 includes a feature generation section 251 , a first evaluation section 252 , a recognition section 254 , a second evaluation section 255 , a correction section 257 , and an estimation section 258 . The feature amount generation unit 251 is an example of a “generation unit”.

The feature generation unit 251 generates a feature for the user U's voice from the voice information VI. Features include, for example, 12 dimensions of MFCC (Mel-Frequency Cepstrum Coefficients), loudness, fundamental frequency (F0), voice probability, zero-crossing rate, HNR (Harmonics-to-Noise-Ratio), and their first derivatives, MFCC. and some or all of a total of 47 second-order differentials of loudness. Loudness is the loudness of a sound and indicates the intensity of the sound perceived by the human hearing. The voice probability indicates the probability that a voice is included in the sound indicated by the voice information VI. The zero crossing rate is the number of times the sound pressure becomes zero. Further, the feature amount generation unit 251 may perform a correction process on the audio information VI, and extract the feature amount from the corrected audio information obtained by performing the correction process. The correction process is, for example, one or both of a process of removing silent portion data from the audio information VI and a process of removing noise included in the sound indicated by the audio information VI.

The first evaluation unit 252 generates voice evaluation values EV1 to EV4 corresponding to each of the plurality of emotions based on the feature amount. More specifically, the first evaluation unit 252 evaluates the voice evaluation value EV1 corresponding to joy, the voice evaluation value EV2 corresponding to anger, the voice evaluation value EV3 corresponding to sadness, and the voice evaluation value EV4 corresponding to normality. generate. The voice evaluation value EV indicates the strength of the user U's feelings.
In the following description, when distinguishing between elements of the same type, reference numerals are used, such as voice evaluation value EV1 corresponding to joy and voice evaluation value EV2 corresponding to anger. On the other hand, when elements of the same type are not distinguished, only the common number among the reference symbols is used, as in the voice evaluation value EV.
Note that joy, anger, sadness, and normalcy are each examples of "first emotions." The "second emotion" is any one of joy, anger, sadness, and a normal emotion that is different from the "first emotion." The voice evaluation value EV1 corresponding to joy, the voice evaluation value EV2 corresponding to anger, the voice evaluation value EV3 corresponding to sadness, and the voice evaluation value EV4 corresponding to normality are examples of the "first voice evaluation value." A voice evaluation value corresponding to a second emotion different from the first emotion is an example of a "second voice evaluation value."
For example, the first evaluation unit 252 generates the audio evaluation values EV1 to EV4 in one of the following two ways.

In the first aspect, the first evaluation unit 252 generates voice evaluation values EV1 to EV4 by comparing the feature amount and a predetermined value. For example, when joy or anger is expressed in voice, the fundamental frequency tends to be higher and the loudness tends to be greater than when joy or anger is not expressed in voice. For example, when the fundamental frequency is greater than a predetermined value and the loudness is greater than the predetermined value, the first evaluation unit 252 compares the case where the fundamental frequency is smaller than the predetermined value and the loudness is smaller than the predetermined value. The voice evaluation value EV1 and the voice evaluation value EV2 are set to large values.

In the second aspect, the first evaluation unit 252 inputs the features generated by the feature generation unit 251 to the learning model LM in order to generate the voice evaluation values EV1 to EV4. is obtained from the learning model LM.

The recognition unit 254 generates a recognized character string RT indicating the content of the user U's utterance based on the voice information VI. More specifically, the recognition unit 254 executes a speech recognition process including a method of recognizing a character string from speech using, for example, an acoustic model and a language model prepared in advance, and outputs a recognized character string RT. .

The second evaluation unit 255 generates character evaluation values ET corresponding to each of the plurality of emotions based on the recognized character string RT. More specifically, the second evaluation unit 255 calculates a character evaluation value ET1 corresponding to joy, a character evaluation value ET2 corresponding to anger, a character evaluation value ET3 corresponding to sadness, and a character evaluation value ET4 corresponding to normality. generate. The character evaluation value ET indicates the intensity with which the user U feels.
Note that the character evaluation value ET1 corresponding to joy, the character evaluation value ET2 corresponding to anger, the character evaluation value ET3 corresponding to sadness, and the character evaluation value ET4 corresponding to normality are examples of "first character evaluation values". . A character evaluation value corresponding to a second emotion different from the first emotion is an example of a "second character evaluation value."

More specifically, the second evaluation section 255 includes an analysis section 2552 and a calculation section 2554. The analysis unit 2552 refers to the analysis dictionary information 31, performs morphological analysis processing on the recognized character string RT, and outputs a corrected recognized character string CRT. The morphological analysis process is a process of breaking down the recognized character string RT into morphemes. In the morphological analysis process, parts of speech and subclassifications of parts of speech in the analysis dictionary information 31 are used. The corrected recognized character string CRT is a character string excluding character strings such as fillers that are unnecessary for estimating the emotion felt by the user U. Fillers are words inserted between utterances, such as "um," "um," and "well."

The calculation unit 2554 calculates a character evaluation value ET for each emotion by comparing the character string included in the emotion classification information 33 and the corrected recognized character string CRT. Regarding the calculation of the character evaluation value ET of each emotion, when the corrected recognized character string CRT includes the character string included in the emotion classification information 33, the calculation unit 2554 calculates the character string included in the corrected recognized character string CRT. The character evaluation value ET of the emotion corresponding to is increased.
For example, if the corrected recognized character string CRT is "I won the game today", the calculation unit 2554 outputs character evaluation values ET for each emotion as follows.

Character evaluation value corresponding to joy ET1 1
Character evaluation value corresponding to anger ET2 0
Character evaluation value corresponding to sadness ET3 0
Character evaluation value ET4 0 corresponding to normal

In the above example, since the corrected recognized character string CRT includes "Kachi" included in the emotion classification information 33, the calculation unit 2554 increases the joy character evaluation value ET1 corresponding to "Kachi" by 1. let The amount of increase in the character evaluation value ET is not limited to 1, and may be different for each character string included in the emotion classification information 33. For example, the amount of increase in the character evaluation value ET for a character string that strongly indicates joy may be set to 2. Furthermore, when the corrected recognized character string CRT includes a character string included in the emotion classification information 33 and a character string that emphasizes the content, the calculation unit 2554 calculates that even if the amount of increase in the character evaluation value ET is increased, good. For example, if the corrected recognized character string CRT is "I'm very happy to win the game today," then the corrected recognized character string CRT includes "happy," which is included in the emotion classification information 33, and "very." Since the character string that emphasizes the content is included, the calculation unit 2554 increases the character evaluation value ET1 of joy by 2, for example. Which character string in the corrected recognized character string CRT is a character string that emphasizes the content can be determined based on the morphemes obtained by the morphological analysis process. In the following example, for ease of explanation, it is assumed that the amount of increase in the character evaluation value ET is 1.
Furthermore, if the corrected recognized character string CRT includes a character string included in the emotion classification information 33 and a character string that negates the content, the calculation unit 2554 calculates the character string included in the corrected recognized character string CRT. You may perform a process different from the process of increasing the character evaluation value ET corresponding to . For example, if the corrected recognized character string CRT is "I couldn't win the game today," the corrected recognized character string CRT includes "win," which is included in the emotion classification information 33. Since a character string that negates the content is included, the calculation unit 2554 increases the character evaluation value ET3 of sadness by 1, for example. Which character string in the corrected recognized character string CRT is a character string that negates the content can be determined based on the morphemes obtained by the morpheme analysis process. In this way, by the morphological analysis process, it is possible to estimate whether the corrected recognized character string CRT has positive content or negative content. In the following example, for ease of explanation, if the corrected recognized character string CRT includes a character string included in the emotion classification information 33, the character string corresponding to the corrected recognized character string CRT will be The explanation will be given assuming that the evaluation value ET is increased.

The correction unit 257 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the situation information SiI. More specifically, the correction unit 257 corrects the voice evaluation value EV1, the voice evaluation value EV2, the voice evaluation value EV3, and the voice evaluation value EV4 to obtain a corrected voice evaluation value CEV1, a corrected voice evaluation value CEV2, The corrected audio evaluation value CEV3 and the corrected audio evaluation value CEV4 are output. Similarly, the correction unit 257 corrects the character evaluation value ET1, character evaluation value ET2, character evaluation value ET3, and character evaluation value ET4, and corrects the character evaluation value CET1 after correction, the character evaluation value CET2 after correction, and the character evaluation value CET2 after correction. The character evaluation value CET3 and the corrected character evaluation value CET4 are output.

More specifically, the correction unit 257 outputs the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4 by executing the following calculations.

CEV1=kv1×EV1
CEV2=kv2×EV2
CEV3=kv3×EV3
CEV4=kv4×EV4
CET1=kt1×ET1
CET2=kt2×ET2
CET3=kt3×ET3
CET4=kt4×ET4

However, the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4 are real numbers of 0 or more and 1 or less. Furthermore, the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4 have the following conditions.

kv1+kt1=kv2+kt2=kv3+kt3=kv4+kt4

The correction unit 257 sets the weighting coefficients kv1 to kv4 and the weighting coefficients kt1 to kt4, for example, with reference to the situation related information 37.

FIG. 7 is a diagram showing an example of the stored contents of the situation related information 37. The situation relationship information 37 indicates the relationship between identification information indicating a situation that a human being can take, and each of a plurality of voice evaluation values EV and each of a plurality of character evaluation values ET set according to the situation.

Situations that a person can be in include a situation where the user is in a private space that is prohibited from entering without the user's permission, and a situation where the user is in a non-private space that can be entered without the user's permission. A private space is, for example, inside one's home. Non-private spaces are, for example, in public transport and in the workplace.

In the situation related information 37 shown in FIG. 7, "home", "train", and "meeting" are registered as identification information indicating situations that a person can be in. The identification information "home" indicates a situation where the user U is at home. The identification information "train" indicates that the user U is using a train. The identification information “meeting” indicates that the user U is in a meeting at work.

The situation related information 37 indicates that the weighting coefficient kv1 is larger than the weighting coefficient kt1 regarding the weighting coefficient related to the identification information "home". Similarly, the situation related information 37 indicates that the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4. Specifically, the weighting coefficients related to the identification information "home" shown in FIG. 7 are as follows.
Weighting coefficient kv1=0.7
Weighting coefficient kt1=0.3
Weighting coefficient kv2=0.8
Weighting coefficient kt2=0.2
Weighting coefficient kv3=0.7
Weighting coefficient kt3=0.3
Weighting coefficient kv4=0.8
Weighting coefficient kt4=0.2

Further, the situation related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 regarding the weighting coefficient related to the identification information "train". Similarly, the situation related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4. Specifically, the weighting coefficients related to the identification information "train" shown in FIG. 7 are as follows.
Weighting coefficient kv1 = Weighting coefficient kv2 = Weighting coefficient kv3 = Weighting coefficient kv4 = 0.1
Weighting coefficient kt1 = Weighting coefficient kt2 = Weighting coefficient kt3 = Weighting coefficient kt4 = 0.9

Further, the situation related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 regarding the weighting coefficient related to the identification information "meeting". Similarly, the situation related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4. Specifically, the weighting coefficients related to the identification information "meeting" shown in FIG. 7 are as follows.
Weighting coefficient kv1=0.4
Weighting coefficient kt1=0.6
Weighting coefficient kv2=0.2
Weighting coefficient kt2=0.8
Weighting coefficient kv3=0.3
Weighting coefficient kt3=0.7
Weighting coefficient kv4=0.2
Weighting coefficient kt4=0.8

The explanation returns to FIG. 6. The estimation unit 258 estimates one or more emotions held by the user U based on the correction results of the correction unit 257, that is, the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4. do. For example, the estimation unit 258 executes the following calculations to generate an evaluation value E1 corresponding to joy, an evaluation value E2 corresponding to anger, an evaluation value E3 corresponding to sadness, and an evaluation value E4 corresponding to normality. do.
E1=CEV1+CET1
E2=CEV2+CET2
E3=CEV3+CET3
E4=CEV4+CET4

Then, the estimating unit 258 estimates the emotion that the user U has, for example, according to one of the following two aspects. In the first aspect, the estimation unit 258 outputs emotion information EI indicating the emotion corresponding to the largest evaluation value E among the evaluation value E1, evaluation value E2, evaluation value E3, and evaluation value E4. In the second aspect, the estimation unit 258 compares each of the evaluation values E1 to E4 with a threshold value, and outputs emotion information EI indicating the emotion corresponding to the evaluation value E that exceeds the threshold value. In the second aspect, the emotional information EI may indicate multiple emotions, such as anger and sadness.
Emotion information EI has, for example, two aspects shown below. The first aspect of the emotion information EI is an identifier indicating one or more emotions that the user U has estimated. Identifiers indicating emotions include an identifier indicating joy, an identifier indicating anger, an identifier indicating sadness, and an identifier indicating normality. The second aspect of the emotion information EI is an identifier indicating one or more estimated emotions that the user U has, and an evaluation value E of the estimated emotion that the user U has.

The output unit 27 outputs information obtained by performing processing on the recognized character string RT obtained by the recognition unit 254 according to one or more emotions indicated by the emotion information EI. For example, there are two types of processing according to emotions as shown below.
The first aspect of emotion-based processing is a process of adding a figure that embodies the emotion to the recognized character string RT. Examples of graphics that embody emotions are pictographs that embody emotions, and emoticons that embody emotions. A pictogram is an image associated with a character code. The character code is, for example, Unicode. An emoticon is a character string that represents a face by combining symbols and characters. In the following explanation, a graphic embodying an emotion will be explained as a pictogram embodying an emotion. An emoticon embodying joy is, for example, an emoticon showing a smiling face. An emoticon embodying anger is, for example, an emoticon showing an angry face. An emoticon embodying sadness is, for example, an emoticon showing a crying face. Further, when the emotional information EI is in the second mode, the output unit 27 outputs a pictogram that embodies the emotion indicated by the emotional information EI and has an intensity corresponding to the evaluation value E included in the emotional information EI. may be determined as the pictogram to be added to the recognized character string RT. For example, when the emotion indicated by the emotion information EI is sadness and the evaluation value E included in the emotion information EI is less than or equal to a predetermined threshold, the output unit 27 recognizes a pictogram representing a tearful face as a character string RT Decided as the emoji to be added to. On the other hand, if the emotion indicated by the emotion information EI is sadness and the evaluation value E included in the emotion information EI is greater than the predetermined threshold, the output unit 27 adds an emoticon representing a crying face to the recognized character string RT. Decide as the emoji to be used. An emoji showing a crying face embodies a higher intensity of sadness than an emoji showing a tearful face.
The output unit 27 outputs a character string with pictograms obtained by adding pictograms to the recognized character string RT. For example, there are two positions for adding pictograms as shown below. The first position is the end of the recognized character string RT. The second position is next to the character string included in the emotion classification information 33 in the recognized character string RT. The display device 51 displays an image based on the character string with pictograms output by the output unit 27.

The second aspect of emotion-based processing is processing that generates synthesized speech that is read aloud with emotion-based intonation added. Intonation is, for example, speeding up or slowing down the reading speed, or increasing or decreasing the volume. Pleasure-based intonation, for example, increases reading speed. An example of anger-based intonation is an increase in volume. An intonation based on sadness is, for example, a reduction in volume. The output unit 27 outputs information indicating the synthesized speech read out with emotion-based intonation added. Then, the output unit 27 adds intonation based on emotion to the synthesized voice indicated by the generated data, and outputs information indicating the synthesized voice read out with the intonation based on emotion added. The sound emitting device 53 emits synthesized speech indicated by the data output by the output unit 27.

1.2. Operation of First Embodiment Next, the operation of the user device 1 will be described using FIG. 8.

FIG. 8 is a flowchart showing the operation of the user device 1. The acquisition unit 21 acquires audio information VI (step S1). The feature amount generation unit 251 generates feature amounts from the audio information VI (step S2). Then, the first evaluation unit 252 inputs the feature amount to the learning model LM and obtains the voice evaluation value EV of each emotion (step S3).

On the other hand, the recognition unit 254 outputs the recognized character string RT based on the voice information VI (step S4). Next, the analysis unit 2552 executes morphological analysis processing and outputs the corrected recognized character string CRT (step S5). Then, the calculation unit 2554 generates character evaluation values ET1 to ET4 for each emotion from the corrected recognized character string CRT (step S6).

The situation information generation unit 23 generates situation information SiI based on the position information PoI obtained from the GPS device 8, the inertial information IFI obtained from the inertial sensor 7, and the schedule information 35 (step S7). Then, the correction unit 257 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficients kv1 to kv4 and kt1 to kt4 according to the situation indicated by the situation information SiI (step S8). .

The estimation unit 258 estimates one or more emotions held by the user U based on the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4, and outputs emotional information EI ( Step S9). The output unit 27 outputs information obtained by performing processing on the recognized character string RT according to the emotion indicated by the emotion information EI (step S10). After completing the process in step S10, the user device 1 ends the series of processes shown in FIG.

1.3. Effects of the First Embodiment According to the above description, the user device 1 corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the situation information SiI. When the situation information SiI indicates a situation in which it is difficult to express emotions in voice, it can be said that the accuracy of the voice evaluation values EV1 to EV4 has decreased. Therefore, when the situation information SiI indicates a situation in which it is difficult to express emotions in voice, the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 are corrected by lowering the voice evaluation values EV1 to EV4. Accordingly, the emotions felt by the user U can be corrected with high accuracy.

Further, the user device 1 generates the feature amount generation unit 251 in the learning model LM that has already learned the relationship between the plurality of feature amounts corresponding to the human voice and the intensity of each of the plurality of emotions felt by the person who uttered the voice. The generated feature quantities are input, and voice evaluation values EV1 to EV4 are obtained from the learning model LM. By using the learning model LM, the user device 1 can obtain highly accurate voice evaluation values EV1 to EV4.

Furthermore, the learning model LM has already learned, for a plurality of humans, the relationship between a plurality of feature amounts corresponding to the human voice and the intensity of each of the plurality of emotions felt by the person who uttered the voice. In other words, the learning model LM is generated by teacher data based on voice information VI of a plurality of users. Therefore, the learning model LM is a general-purpose model that is not tailored to a specific individual. In the first embodiment, since a general-purpose learning model LM can be used, it is easier to introduce it than when using a learning model tailored for a specific individual.

Further, the correction unit 257 refers to the situation related information 37 and sets weighting coefficients for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 according to the situation indicated by the situation information SiI. By referring to the situation-related information 37, it is possible to accurately identify whether the user U is in a situation in which it is difficult to express emotions.

The situation related information 37 indicates that the weighting coefficient kv1 is larger than the weighting coefficient kt1 regarding the weighting coefficient related to the identification information indicating the situation where the user U is in the private space. Further, the situation related information 37 indicates that the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4.
When the user U is in a private space such as his or her home, he or she does not attract the attention of others, so it is easy for him or her to express his or her emotions through voice. Therefore, since the weighting coefficient kv1 is larger than the weighting coefficient kt1, the weighting coefficient kv2 is larger than the weighting coefficient kt2, the weighting coefficient kv3 is larger than the weighting coefficient kt3, and the weighting coefficient kv4 is larger than the weighting coefficient kt4, the user equipment 1 , joy, anger, sadness, and normality can be estimated with high accuracy.

The situation related information 37 indicates that the weighting coefficient kt1 is larger than the weighting coefficient kv1 regarding the weighting coefficient related to the identification information indicating the situation where the user U is in the non-private space. Further, the situation related information 37 indicates that the weighting coefficient kt2 is larger than the weighting coefficient kv2, the weighting coefficient kt3 is larger than the weighting coefficient kv3, and the weighting coefficient kt4 is larger than the weighting coefficient kv4.
As an example of user U in a non-private space, in a situation where user U is using a train, it is common for user U to speak without emotion, as uttering with emotion would attract the attention of others. I can say that. The situation in which user U is using a train is one of the situations in which it is difficult for user U to voice his or her emotions. As another example of user U being in a non-private space, in a situation where user U is in a meeting, it can be said that user U is unlikely to speak with emotion. A situation in which user U is in a meeting is one of the situations in which it is difficult for user U to voice his or her emotions.
Therefore, when user U is in the non-private space, weighting coefficient kt1 is larger than weighting coefficient kv1, weighting coefficient kt2 is larger than weighting coefficient kv2, weighting coefficient kt3 is larger than weighting coefficient kv3, and weighting coefficient kt4 is larger than weighting coefficient kt4. Since the weighting coefficient is larger than kv4, the user device 1 can accurately estimate joy, anger, sadness, and normality.

2. Second Embodiment The user device 1a according to the second embodiment calculates voice evaluation values CEV1 to CEV4 and character evaluation values CET1 to CET4 based on the degree to which emotion is expressed in the voice according to the recognized character string RT. This differs from the user device 1 according to the first embodiment in that the correction is performed. Note that in the second embodiment illustrated below, for elements whose actions or functions are equivalent to those in the first embodiment, reference numerals are used in the above description, and detailed descriptions of each are omitted as appropriate.

2.1. Functions of Second Embodiment FIG. 9 is a block diagram showing a user device 1a according to a second embodiment. The user device 1a is realized by a computer system including a processing device 2a, a storage device 3a, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS device 8. The storage device 3a is a recording medium readable by the processing device 2a, and stores a plurality of programs including the control program PRa executed by the processing device 2a, analysis dictionary information 31, emotion classification information 33, schedule information 35, and situation related information. 37, character string relationship information 38 and learning model LM are stored.

The character string related information 38 is used to determine the degree to which emotion is expressed in the voice according to the recognized character string RT. The processing device 2a functions as an acquisition section 21, a situation information generation section 23, an emotional information generation section 25a, and an output section 27 by reading and executing the control program PRa from the storage device 3a.

FIG. 10 is a diagram showing an overview of the functions of the user device 1a in the second embodiment. The emotional information generation section 25a includes a feature generation section 251, a first evaluation section 252, a recognition section 254, a second evaluation section 255, a correction section 257a, and an estimation section 258.

The correction unit 257a calculates the voice evaluation values EV1 to EV4 and the character evaluation value based on the degree to which emotion is expressed in the voice of the user U when the user U utters the recognized character string RT, and the situation information SiI. Correct ET1 to ET4. For example, the correction unit 257a corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 according to one of the two modes shown below.

The correction unit 257a in the first aspect classifies the character strings into character strings with a low degree of emotional expression in the voice of the user U and character strings with a high degree of emotion when uttered. Then, when the recognized character string RT includes a character string with a low degree of emotion expressed, the correction unit 257a determines that the degree of emotion expressed in the voice of the user U is low when the recognized character string RT is uttered. judge. On the other hand, if the recognized character string RT does not include a character string with a low degree of emotional expression, the correction unit 257a adjusts the degree of emotional expression in the voice of the user U when the recognized character string RT is uttered. It is judged to be high. Then, the correction unit 257a determines that if the degree of emotion expressed in the user U's voice is low when the recognized character string RT is uttered, the degree of emotion expressed in the user U's voice is low when the recognized character string RT is uttered. The weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4 are lowered compared to the case where the degree is high.
For example, "thank you" has a general meaning of gratitude. When the user U utters "Thank you," the emotion that the user U feels becomes more joyful. However, character strings used for general purposes are often uttered smoothly because they are used to being said, and it can be said that the degree to which emotions are expressed in the voice is low. Therefore, when the recognized character string RT includes "thank you", the correction unit 257a lowers the weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4. For example, the correction unit 257a subtracts a predetermined value from the weighting coefficients kv1 to kv4 corresponding to the voice evaluation values EV1 to EV4, and adds the predetermined value to the weighting coefficients kt1 to kt4 corresponding to the character evaluation values ET1 to ET4.

The correction unit 257a in the second aspect is configured to calculate the characters included in the recognized character string RT for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the character string related information 38 and the situation related information 37. Weighting coefficients kv1 to kv4 and kt1 to kt4 are set according to the situation indicated by the character string and situation information SiI. Then, the correction unit 257a corrects the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficient according to the situation indicated by the recognized character string RT and the situation information SiI.

FIG. 11 is a diagram showing an example of the stored contents of the character string related information 38. The character string related information 38 includes a character string uttered by a human, voice evaluation values EV1 to EV4 set based on the degree to which emotions are expressed in the human voice when this character string is uttered, and a character evaluation value ET1. The relationship between weighting coefficients kv1 to kv4 and increase/decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4 of weighting coefficients kt1 to kv4 for ~ET4 is shown.
The increase/decrease values Δkv1 to Δkv4 and the increase/decrease values Δkt1 to Δkt4 have the following conditions.

Δkv1+Δkt1=Δkv2+Δkt2=Δkv3+Δkt3=Δkv4+Δkt4=0

The character string related information 38 shown in FIG. 10 indicates that when a human utters the character string "Thank you", increase/decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4 are set as shown below.
Increase/decrease value Δkv1=-0.2
Increase/decrease value Δkt1=0.2
Increase/decrease value Δkv2=-0.2
Increase/decrease value Δkt2=0.2
Increase/decrease value Δkv3=-0.2
Increase/decrease value Δkt3=0.2
Increase/decrease value Δkv4=-0.1
Increase/decrease value Δkt4=0.1
Note that a value of 0 may be included among the increase/decrease values Δkv1 to Δkv4 and Δkt1 to Δkt4. For example, the increase/decrease values Δkv1 and Δkt1 for the character string "Thank you" are the values described above, and the increase/decrease values Δkv2 to Δkv4 and Δkt2 to Δkt4 may be zero.

When the character string included in the character string related information 38 is included in the recognized character string RT, the correction unit 257a performs the following calculation to obtain the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation. Outputs values CET1 to CET4.

CEV1=(kv1+Δkv1)×EV1
CEV2=(kv2+Δkv2)×EV2
CEV3=(kv3+Δkv3)×EV3
CEV4=(kv4+Δkv4)×EV4
CET1=(kt1+Δkt1)×ET1
CET2=(kt2+Δkt2)×ET2
CET3=(kt3+Δkt3)×ET3
CET4=(kt4+Δkt4)×ET4

The increase/decrease values Δkv1 to Δkv4 and the increase/decrease values Δkt1 to Δkt4 are values corresponding to character strings included in the recognized character string RT in the character string related information 38. Further, when the sum of the weighting coefficient kvx and the increase/decrease value Δkvx is less than 0, the correction unit 257a outputs the corrected audio evaluation value CEVx as 0, and the sum of the weighting coefficient kvx and the increase/decrease value Δkvx is less than 1. If it is larger, the corrected audio evaluation value CEVx is output as the same value as the audio evaluation value EVx. Similarly, when the sum of the weighting coefficient ktx and the increase/decrease value Δktx is less than 0, the correction unit 257a outputs the corrected character evaluation value CETx as 0, and the sum of the weighting coefficient ktx and the increase/decrease value Δktx is 1. If larger, the corrected character evaluation value CETx is output as the same value as the character evaluation value ETx. x is an integer from 1 to 4.
In the following description, the correction unit 257a will be described as being in the second aspect.

2.2. Operation of Second Embodiment Next, the operation of the user device 1a will be described using FIG. 12.

FIG. 12 is a flowchart showing the operation of the user device 1a. The processing from step S21 to step S27 shown in FIG. 12 is the same as the processing from step S1 to step S7 shown in FIG. 8, respectively, so the description thereof will be omitted.

After completing the process in step S27, the correction unit 257a determines whether the character string in the character string related information 38 is included in the recognized character string RT (step S28). If the determination result in step S28 is affirmative, the correction unit 257a sets the voice evaluation values EV1 to EV4 based on the character string included in the recognized character string RT and the weighting coefficients kv1 to kv4 according to the situation indicated by the situation information SiI. The character evaluation values ET1 to ET4 are corrected (step S29). On the other hand, if the determination result in step S28 is negative, the correction unit 257a calculates the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the weighting coefficients kv1 to kv4 according to the situation indicated by the situation information SiI. Correct it (step S30).

After the process in step S29 or the process in step S30 is completed, the estimating unit 258 calculates one or more values that the user U has based on the corrected voice evaluation values CEV1 to CEV4 and the corrected character evaluation values CET1 to CET4. The emotion is estimated and emotion information EI is output (step S31). The output unit 27 outputs information obtained by performing processing on the recognized character string RT according to the emotion indicated by the emotion information EI (step S32). After completing the process in step S10, the user device 1 ends the series of processes shown in FIG.

2.3. Effects of the Second Embodiment According to the above description, the user device 1a determines based on the degree to which emotion is expressed in the voice of the user U when the user U utters the recognized character string RT, and the situation information SiI. , the voice evaluation values EV1 to EV4, and the character evaluation values ET1 to ET4. In general, there are character strings that express a high degree of emotion when uttered, and character strings that express a low degree of emotion when uttered, so the degree to which emotions are expressed when uttered is By correcting the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 in accordance with the above, it is possible to accurately estimate the emotion felt by the user U.

Further, the user device 1a refers to the situation related information 37 and the character string related information 38, and calculates the characters included in the recognized character string RT for each of the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4. Weighting coefficients kv1 to kv4 and kt1 to kt4 are set according to the situation indicated by the column and situation information SiI. By setting the weighting coefficients kv1 to kv4 and kt1 to kt4 according to the character string included in the recognized character string RT and the situation indicated by the situation information SiI, the emotion felt by the user U can be estimated with high accuracy.

3. Modifications The present invention is not limited to the embodiments illustrated above. Specific modes of modification are illustrated below. Two or more aspects arbitrarily selected from the examples below may be combined.

(1) In each of the above embodiments, the processing device 2 of the user device 1 functions as the acquisition unit 21, the situation information generation unit 23, the emotional information generation unit 25, and the output unit 27, but the present invention is not limited thereto. In the first modification, the acquisition unit 21, the situation information generation unit 23, the emotional information generation unit 25, and the output unit 27 are distributed between the user device 1b and the server device 10.

FIG. 13 is a diagram showing the overall configuration of the emotion estimation system SYS. The emotion estimation system SYS includes a user device 1b owned by the user U, a network NW, and a server device 10. The user device 1b is an example of a "terminal device." The server device 10 is an example of the "emotion estimation device" in the first modification.

FIG. 14 is a block diagram showing the configuration of the user device 1b. The user device 1b is realized by a computer system including a processing device 2b, a storage device 3b, an input device 4, an output device 5, a communication device 6, an inertial sensor 7, and a GPS device 8. The storage device 3b is a recording medium readable by the processing device 2b, and stores a plurality of programs including the control program PRb executed by the processing device 2b, and schedule information 35.

The processing device 2b functions as an acquisition section 21, a situation information generation section 23, and an output section 27 by reading and executing the control program PRb from the storage device 3b.

The communication device 6 transmits the voice information VI and the situation information SiI to the server device 10, and receives the recognized character string RT and the emotional information EI from the server device 10.

FIG. 15 is a block diagram showing the configuration of the server device 10. As shown in FIG. The server device 10 is realized by a computer system including a processing device 2B, a storage device 3B, and a communication device 6B. Each element of the server device 10 is interconnected by one or more buses 9B for communicating information. The storage device 3B is a recording medium readable by the processing device 2B, and stores a plurality of programs including the control program PRB executed by the processing device 2B, analysis dictionary information 31, emotion classification information 33, situation related information 37, and Store the learning model LM.

The processing device 2B functions as the emotional information generation section 25 by reading and executing the control program PRB from the storage device 3B.

The communication device 6B receives the voice information VI and the situation information SiI from the user device 1b, and transmits the recognized character string RT and the emotional information EI to the user device 1b.

As described above, according to the first modification, the acquisition unit 21, the situation information generation unit 23, the emotional information generation unit 25, and the output unit 27 can be distributed between the user device 1 and the server device 10.

(2) In the second embodiment, the correction unit 257a calculates the voice evaluation value based on the degree to which emotion is expressed in the voice of the user U when the user U utters the recognized character string RT, and the situation information SiI. Although EV1 to EV4 and character evaluation values ET1 to ET4 are corrected, the correction is not limited thereto. For example, the correction unit 257a calculates the voice evaluation values EV1 to EV4 and the character evaluation values ET1 to ET4 based on the degree to which emotion is expressed in the voice of the user U when the user U utters the recognized character string RT. It may be corrected.

(3) Although it has been described that an example of a private space is inside a home, the example is not limited to this. For example, the private space may be in a hotel room where user U is staying. For example, it is assumed that the user device 1 has a function that can control locking and unlocking of a hotel room door. Based on this premise, the movement range of the user U is within a predetermined range between the time when the user device 1 instructs to unlock the door of the hotel room and a predetermined period of time elapses from the time when the user device 1 instructs the door to be unlocked. In some cases, the user device 1 determines that the user U is in a hotel room where the user U is staying.

(4) Non-private spaces are described as inside public transportation and inside workplaces, but are not limited to these. For example, non-private spaces include inside a school, inside a hospital, and in a library.

(5) As an example of the situation of using public transportation, the situation of using a train was given, but the situation of using public transportation is not limited to the situation of using a train. For example, the situation of using public transportation may include the situation of being inside a station. For example, if the user device 1 has a transportation IC (Integrated Circuit) card function, and the user U enters the station premises using the transportation IC card function, the situation is such that the user U uses public transportation. You may judge that. In addition, public transportation includes not only trains but also route buses, taxis, ferries, passenger planes, etc.

(6) Although the output unit 27 outputs information obtained by performing processing on the recognized character string RT in accordance with the emotion indicated by the emotion information EI, the present invention is not limited to this. For example, the output unit 27 may output a character string indicating the emotion indicated by the emotion information EI to the display device 51, or may output a pictogram indicated by the emotion information EI to the display device 51.

(7) The user device 1 does not need to have the sound collection device 41. If the user device 1 does not include the sound collection device 41, the user device 1 may acquire the audio information VI via the communication device 6, or may acquire the audio information VI stored in the storage device 3.

(8) The user device 1 does not need to have the sound emitting device 53.

(9) The user device 1 may be a smart speaker. When the user device 1 is a smart speaker, the user device 1 does not need to have the touch panel 43 and the display device 51.

(10) As shown in FIG. 4, the emotion classification information 33 categorizes each of multiple morphemes used in a certain word, such as “katsu” and “katsu”, into joy, anger, sadness, and normal. However, it is not limited to this. For example, the emotion classification information 33 may classify character strings registered in the original data of the analysis dictionary information 31 into any of joy, anger, sadness, and normality. For example, the emotion classification information 33 classifies the character strings "happy," "pass," and "win" registered in the original data of the analysis dictionary information 31 as joy. The calculation unit 2554 decomposes the corrected recognized character string CRT into morphemes and converts the decomposed morphemes into character strings registered in the original form data of the analysis dictionary information 31. Then, when the character string obtained by the conversion and the character string included in the emotion classification information 33 match, the calculation unit 2554 calculates the emotion corresponding to the character string included in the corrected recognized character string CRT. Increase character evaluation value ET.

(11) Although the calculation unit 2554 calculated the character evaluation value ET for each emotion for the corrected recognized character string CRT, it may also calculate the character evaluation value ET for each emotion for the recognized character string RT. . However, the recognized character string RT includes character strings that are unnecessary for estimating emotions. Therefore, by calculating the character evaluation value ET for each emotion for the corrected recognized character string CRT, it is possible to estimate the emotion by calculating the character evaluation value ET for each emotion for the recognized character string RT. Accuracy can be improved.

(12) Although the example in which the user U speaks Japanese is used, the above-mentioned aspects can be applied to any language the user speaks. For example, even if the user U speaks a language other than Japanese, such as English, French, or Chinese, the above-mentioned aspects can be applied. For example, when user U speaks English, the analysis dictionary information 31 is information regarding English morphemes, and the emotion classification information 33 is information that classifies English words as joy, anger, sadness, or normal. That's fine.

(13) The block diagrams used to explain each aspect described above show functional unit blocks. These functional blocks (components) are realized by any combination of hardware and/or software. Further, the means for realizing each functional block is not particularly limited. That is, each functional block may be realized by one device that is physically and/or logically coupled, or may be realized by directly and/or indirectly two or more physically and/or logically separated devices. It may also be realized by a plurality of these devices connected to each other (for example, by wire and/or wirelessly).

(14) The order of the processing procedures, sequences, flowcharts, etc. in each aspect described above may be changed as long as there is no contradiction. For example, the methods described herein present elements of the various steps in an exemplary order and are not limited to the particular order presented.

(15) In each of the aspects described above, input/output information may be stored in a specific location (for example, memory) or may be managed in a management table. Information etc. to be input/output may be overwritten, updated, or additionally written. The output information etc. may be deleted. The input information etc. may be transmitted to other devices.

(16) In each of the aspects described above, the determination may be made based on a value expressed by 1 bit (0 or 1), or may be made based on a truth value (Boolean: true or false). , may be performed by numerical comparison (for example, comparison with a predetermined value).

(17) In each of the above-mentioned aspects, a portable information processing device such as a smartphone is exemplified as the user device 1, but the specific form of the user device 1 is arbitrary and is not limited to the above-mentioned examples of each form. . For example, a portable or stationary personal computer may be used as the user device 1.

(18) In each of the above aspects, the storage device 3 is a recording medium readable by the processing device 2, and examples include ROM and RAM. discs, Blu-ray discs), smart cards, flash memory devices (e.g. cards, sticks, key drives), CD-ROMs (Compact Disc-ROMs), registers, removable disks, hard disks, floppies ) disk, magnetic strip, database, server, or other suitable storage medium. The program may also be sent from a network. The program may also be transmitted from a communication network via a telecommunications line.

(19) Each of the above aspects includes LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G, 5G, FRA (Future Radio Access), W-CDMA (registered trademark) , GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi), IEEE 802.16 (WiMAX), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark) ), other suitable systems, and/or next-generation systems expanded based on these.

(20) In each of the aspects described above, the information, signals, etc. described may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc., which may be referred to throughout the above description, may refer to voltages, currents, electromagnetic waves, magnetic fields or magnetic particles, light fields or photons, or any of these. It may also be represented by a combination of
Note that terms explained in this specification and/or terms necessary for understanding this specification may be replaced with terms having the same or similar meanings.

(21) Each function illustrated in FIG. 2, FIG. 6, FIG. 9, FIG. 10, FIG. 14, and FIG. 15 is realized by any combination of hardware and software. Further, each function may be realized by a single device, or may be realized by two or more devices configured separately from each other.

(22) The programs exemplified in each of the above embodiments are instructions, instruction sets, codes, code segments, regardless of whether they are called software, firmware, middleware, microcode, hardware description language, or by other names. , should be broadly construed to mean program code, subprogram, software module, application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, or function, etc.
Additionally, software, instructions, etc. may be sent and received via a transmission medium. For example, if the software uses wired technologies such as coaxial cable, fiber optic cable, twisted pair and digital subscriber line (DSL) and/or wireless technologies such as infrared, radio and microwave to When transmitted from a remote source, these wired and/or wireless technologies are included within the definition of transmission medium.

(23) In each of the embodiments described above, information, parameters, etc. may be expressed as absolute values, as relative values from a predetermined value, or as other corresponding information. Good too.

(24) The names used for the parameters described above are not limiting in any way. Furthermore, the mathematical formulas etc. using these parameters may differ from those explicitly disclosed herein.

(25) In each of the embodiments described above, the user equipment 1 may be a mobile station. A mobile station is defined by a person skilled in the art as a subscriber station, mobile unit, subscriber unit, wireless unit, remote unit, mobile device, wireless device, wireless communication device, remote device, mobile subscriber station, access terminal, mobile terminal, wireless It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable terminology.

(26) In each of the embodiments described above, the statement "based on" does not mean "based only on" unless specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

(27) As used herein, any reference to elements using the designations "first," "second," etc. does not generally limit the amount or order of those elements. These designations may be used herein as a convenient way of distinguishing between two or more elements. Thus, reference to a first and second element does not imply that only two elements may be employed therein or that the first element must precede the second element in any way.

(28) In each of the embodiments described above, as long as "including", "comprising", and variations thereof are used in the present specification or claims, these terms Like the term "comprising," it is intended to be inclusive. Furthermore, the term "or" as used in this specification or in the claims is not intended to be exclusive or.

(29) Throughout this application, where articles have been added by translation, such as a, an, and the in English, these articles shall be used unless the context clearly indicates otherwise. Including plural.

(30) It will be apparent to those skilled in the art that the present invention is not limited to the embodiments described herein. The present invention can be implemented as modifications and variations without departing from the spirit and scope of the present invention as determined based on the claims. Accordingly, the description herein is for illustrative purposes only and is not intended to be construed as limiting the invention. Further, a plurality of aspects selected from the aspects illustrated in this specification may be combined.

DESCRIPTION OF SYMBOLS 1, 1a, 1b... User device, 6... Communication device, 10... Server device, 21... Acquisition part, 23... Situation information generation part, 25, 25a... Emotional information generation part, 27... Output part, 37... Situation related information , 38...Character string related information, 41...Sound collection device, 251...Feature value generation section, 252...First evaluation section 254...Recognition section, 255...Second evaluation section, 257, 257a...Correction section, 258...Estimation section , CET1...Character evaluation value after correction, CET2...Character evaluation value after correction, CET3...Character evaluation value after correction, CET4...Character evaluation value after correction, CEV1...Speech evaluation value after correction, CEV2...After correction Voice evaluation value, CEV3...Voice evaluation value after correction, CEV4...Speech evaluation value after correction, EI...Emotional information, ET1...Text evaluation value, ET2...Text evaluation value, ET3...Text evaluation value, ET4...Text evaluation Value, EV1...Voice evaluation value, EV2...Voice evaluation value, EV3...Voice evaluation value, EV4...Voice evaluation value, kt1...Weighting coefficient, kt2...Weighting coefficient, kt3...Weighting coefficient, kt4...Weighting coefficient, kv1...Weighting coefficient , kv2...Weighting coefficient, kv3...Weighting coefficient, kv4...Weighting coefficient.

Claims (7)

a generation unit that generates a feature amount for the user's voice based on the user's voice information;
a first evaluation unit that generates a first voice evaluation value indicating the strength of the user's first emotion and a second voice evaluation value indicating the degree of the user's second emotion, based on the feature amount; ,
a recognition unit that generates a recognized character string indicating the content of the user's utterance based on the voice information;
A second character evaluation value that indicates the intensity with which the user feels the first emotion and a second character evaluation value that indicates the intensity with which the user feels the second emotion based on the recognized character string. Evaluation department and
a correction unit that corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value based on situation information indicating the user's situation;
an estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit;
Equipped with
The correction unit is
Identification information indicating a situation that a human can take, and each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value set according to the situation. With reference to the situational information that shows the relationship between the weighting coefficient and the
setting weighting coefficients for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value according to the situation indicated by the situation information;
The first voice evaluation value and the second voice evaluation are based on weighting coefficients for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value. correcting the value, the first character evaluation value, and the second character evaluation value,
The situation related information is
Regarding a weighting coefficient related to identification information indicating that the user is in a private space that is prohibited from entering without the user's permission, a weighting coefficient for the first voice evaluation value is larger than a weighting coefficient for the first character evaluation value. , and a weighting coefficient for the second voice evaluation value is larger than a weighting coefficient for the second character evaluation value.
to show that
Emotion estimation device. a generation unit that generates a feature amount for the user's voice based on the user's voice information;
a first evaluation unit that generates a first voice evaluation value indicating the strength of the user's first emotion and a second voice evaluation value indicating the degree of the user's second emotion, based on the feature amount; ,
a recognition unit that generates a recognized character string indicating the content of the user's utterance based on the voice information;
A second character evaluation value that indicates the intensity with which the user feels the first emotion and a second character evaluation value that indicates the intensity with which the user feels the second emotion based on the recognized character string. Evaluation department and
a correction unit that corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value based on situation information indicating the user's situation;
an estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit;
Equipped with
The correction unit is
Identification information indicating a situation that a human can take, and each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value set according to the situation. With reference to the situational information that shows the relationship between the weighting coefficient and the
setting weighting coefficients for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value according to the situation indicated by the situation information;
The first voice evaluation value and the second voice evaluation are based on weighting coefficients for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value. correcting the value, the first character evaluation value, and the second character evaluation value,
The situation related information is
Regarding a weighting coefficient related to identification information indicating a situation in which the user is in a non-private space that can be entered without the user's permission, a weighting coefficient for the first character evaluation value is greater than a weighting coefficient for the first voice evaluation value. , and a weighting coefficient for the second character evaluation value is larger than a weighting coefficient for the second voice evaluation value.
to show that
Emotion estimation device. a generation unit that generates a feature amount for the user's voice based on the user's voice information;
a first evaluation unit that generates a first voice evaluation value indicating the strength of the user's first emotion and a second voice evaluation value indicating the degree of the user's second emotion, based on the feature amount; ,
a recognition unit that generates a recognized character string indicating the content of the user's utterance based on the voice information;
A second character evaluation value that indicates the intensity with which the user feels the first emotion and a second character evaluation value that indicates the intensity with which the user feels the second emotion based on the recognized character string. Evaluation department and
a correction unit that corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value based on situation information indicating the user's situation;
an estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit;
Equipped with
The correction unit is
The first voice evaluation value, the second voice evaluation value, and the first correcting the character evaluation value and the second character evaluation value;
Emotion estimation device. a generation unit that generates a feature amount for the user's voice based on the user's voice information;
a first evaluation unit that generates a first voice evaluation value indicating the strength of the user's first emotion and a second voice evaluation value indicating the degree of the user's second emotion, based on the feature amount; ,
a recognition unit that generates a recognized character string indicating the content of the user's utterance based on the voice information;
A second character evaluation value that indicates the intensity with which the user feels the first emotion and a second character evaluation value that indicates the intensity with which the user feels the second emotion based on the recognized character string. Evaluation department and
a correction unit that corrects the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value based on situation information indicating the user's situation;
an estimation unit that estimates one or more emotions held by the user based on the correction result of the correction unit;
Equipped with
The correction unit is
identification information indicating a situation that a human can take; and the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value set according to the situation. Situation related information indicating the relationship with the weighting coefficient for each;
the first voice evaluation value, the second voice evaluation value, and the first voice evaluation value set based on the character string uttered by the human , and the degree to which emotion is expressed in the human voice when the character string is uttered; Character string relationship information indicating a relationship between a character evaluation value and an increase/decrease value of a weighting coefficient for each of the second character evaluation values;
Refer to
the character string included in the recognized character string and the situation indicated by the situation information for each of the first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the second character evaluation value; Set the weighting coefficient accordingly,
The first voice evaluation value, the second voice evaluation value, the first character evaluation value, and the Correct the two-character evaluation value,
Emotion estimation device. The first evaluation section includes:
The feature values generated by the generation unit are applied to a learning model that has already learned the relationship between the feature values corresponding to the human voice and the intensities for each of the first emotion and second emotion held by the person who uttered the voice. inputting the first voice evaluation value and the second voice evaluation value from the learning model;
The emotion estimation device according to any one of claims 1 to 4. The learning model has already learned, for a plurality of humans, the relationship between a plurality of feature amounts corresponding to the human voice and the intensity of each of the first emotion and the second emotion held by the person who uttered the voice. be,
The emotion estimation device according to claim 5. An emotion estimation system comprising the emotion estimation device according to any one of claims 1 to 6 and a terminal device capable of communicating with the emotion estimation device,
The terminal device is
a sound collection device that collects the user's voice;
a situation information generation unit that generates the situation information;
The voice information indicating the user's voice and the situation information are transmitted to the emotion estimation device, and the emotion estimation device transmits the recognized character string and one or more emotions held by the user estimated by the estimation unit. a communication device that receives emotional information indicating
an output unit that outputs information obtained by performing processing on the recognized character string according to the emotion indicated by the emotion information;
An emotion estimation system equipped with

Images