JP7805427B2 - Information processing system, information processing device, information processing method, and program - Google Patents

Description

Article 30, paragraph 2 of the Patent Act applies. (1) Date of website publication: March 22, 2023 Website address: https://www.mdis.co.jp/report-column/professional/202303_data-analysis.html <Reference> Mitsubishi Electric Information Systems Corporation website Interview article (2) Date of website publication: March 22, 2023 Website address: http://www.mitsubishielectric.co.jp/meltopia/productreport/case80. html <Materials> 2: Mitsubishi Electric Corporation website Product report (3) Date held (publication date) August 31, 2022 Name of meeting, location of publication Presentation on "Data analysis services" Service proposed to: Advanced Media Co., Ltd. (3-1-1 Higashiikebukuro, Toshima-ku, Tokyo) Held via online conference (4) Date held (publication date) September 2, 2022 Name of meeting, location of publication Presentation on "Data analysis services" Service proposed to: Mitsubishi UFJ Bank Ltd. (2-7-1 Marunouchi, Chiyoda-ku, Tokyo) Held via online conference (5) Date held (publication date) November 30, 2022 Name of meeting, location of publication Presentation on "Data analysis services" Service proposed to: Mitsubishi UFJ Trust and Banking Corporation (1-4-5 Marunouchi, Chiyoda-ku, Tokyo) Held via online conference (6) Date held (publication date) November 30, 2022 Name of meeting, location of publication Presentation on "Data Analysis Services" Service proposed to: Mitsubishi UFJ Trust Systems Co., Ltd. (2-9-8 Konan, Minato-ku, Tokyo) Held via online conference (7) Date held (public date) February 6, 2023 Meeting name, public location Presentation on Data Analysis Services Service proposed to: Mitsubishi UFJ Information Technology Co., Ltd. (4-10-2 Nakano, Nakano-ku, Tokyo) Held via online conference (8) Date held (public date) April 26, 2023 Meeting name, public location Presentation on "Data Analysis Services" Service proposed to: Yamanashi Chuo Bank, Ltd. (1-20-8 Marunouchi, Kofu, Yamanashi) Held via online conference <Materials> Presentation materials for "Data Analysis Services"

The present invention relates to an information processing system, an information processing device, an information processing method, and a program.

In call center operations, in addition to improving customer satisfaction, it is desirable to improve the skills of operators and reduce operator stress.
For example, Patent Document 1 discloses an information provision system that includes an analysis unit that uses voice data of a call between a customer and an operator that has been quantified based on emotions to classify the call into one of the evaluation indicators of the customer's emotions, and a provision unit that provides the results of the classification.

Japanese Patent Application Laid-Open No. 2022-154230

Typically, call center operations require operators to appropriately change intonation, words, and other aspects of their speech depending on the content of the conversation and the customer. For this reason, it is necessary to analyze emotions by understanding the content and context of the conversation from text data based on the audio data of the conversation, for example.
However, the technology described in Patent Document 1 is based on the premise that text data based on voice data is not used to accurately evaluate customer emotions. Therefore, an operator needs to appropriately change speaking style, such as intonation and words, depending on the content of the conversation and the customer. This poses a problem in that it is not possible to analyze emotions by understanding the content and context of the conversation from text data based on the voice data of the conversation.

One aspect of the present invention has been made in consideration of the above points, and aims to provide an information processing system, information processing device, information processing method, and program that can analyze emotions after understanding the content and context of a conversation from text data based on voice data.

The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is an information processing system comprising: a prediction unit that predicts an evaluation value by inputting a plurality of emotion values that correspond to text data for each speaker that represents the content of an utterance by the speaker and that represent the emotions of the speaker into a learning model that is generated based on the correspondence between the emotion values and an evaluation value for the conversation that corresponds to the text data.

Another aspect of the present invention is an information processing system comprising: a frequent word extraction unit that extracts one or more frequently occurring words included in text data for each speaker that represents the content of the speaker's utterance; and an estimation unit that estimates which of multiple patterns the speaker belongs to based on the frequently occurring words and multiple emotion values that represent the speaker's emotions, wherein the emotion values are either statistics of evaluation values that represent each emotion over a predetermined period from the start of the conversation, statistics of evaluation values that represent each emotion over a predetermined period going back a predetermined period from the end of the conversation, or statistics of evaluation values that represent each emotion over a predetermined period that has elapsed since a specific keyword was uttered in the conversation, and the estimation unit estimates which of multiple patterns the speaker belongs to using the emotion values, numerical values that represent the frequently occurring words, and weights for each of the multiple patterns.

Another aspect of the present invention is an information processing device that includes a prediction unit that predicts an evaluation value by inputting a plurality of emotion values that correspond to text data for each speaker that represents the content of an utterance by the speaker and that represent the emotions of the speaker into a learning model that is generated based on the correspondence between the emotion values and an evaluation value for the conversation that corresponds to the text data.

Another aspect of the present invention is an information processing device comprising: a frequent word extraction unit that extracts one or more frequently occurring words included in text data for each speaker that represents the content of an utterance by the speaker; and an estimation unit that estimates which of a plurality of types of pattern the speaker belongs to based on the frequently occurring words and a plurality of emotion values that represent the emotions of the speaker, wherein the emotion values are either statistics of evaluation values that represent each emotion over a predetermined period from the start of a conversation, statistics of evaluation values that represent each emotion over a predetermined period going back a certain period from the end of the conversation, or statistics of evaluation values that represent each emotion over a predetermined period that has elapsed since a specific keyword was uttered in the conversation, and the estimation unit estimates which of a plurality of types of pattern the speaker belongs to using the emotion values, numerical values that represent the frequently occurring words, and weights for each of the plurality of types of patterns.
is.

Another aspect of the present invention is an information processing method executed by a computer of an information processing device, which includes a prediction step of predicting an evaluation value by inputting multiple emotion values that correspond to text data for each speaker representing the content of the speaker's utterance and represent the emotions of the speaker into a learning model generated based on the correspondence between the emotion values and evaluation values for the conversation corresponding to the text data.

Another aspect of the present invention is an information processing method executed by a computer of an information processing device, comprising: a frequent word extraction step of extracting one or more frequently occurring words included in text data for each speaker that represents the content of the speaker's utterance; and an estimation step of estimating which of multiple patterns the speaker belongs to based on the frequently occurring words and multiple emotion values that represent the speaker's emotions, wherein the emotion values are either statistics of evaluation values that represent each emotion over a predetermined period from the start of the conversation, statistics of evaluation values that represent each emotion over a predetermined period going back a predetermined period from the end of the conversation, or statistics of evaluation values that represent each emotion over a predetermined period that has elapsed since a specific keyword was uttered in the conversation, and the estimation step estimates which of multiple patterns the speaker belongs to using the emotion values, numerical values that represent the frequently occurring words, and weights for each of the multiple patterns.

Another aspect of the present invention is a program for causing a computer of an information processing device to execute a prediction step of predicting the evaluation value by inputting the emotion values into a learning model generated based on the correspondence between multiple emotion values representing the emotions of the speaker and the evaluation value for the conversation corresponding to the text data, the emotion values corresponding to the text data for each speaker representing the content of the speaker's utterance.

Another aspect of the present invention is a program that causes a computer of an information processing device to execute: a frequent word extraction step of extracting one or more frequently occurring words included in text data for each speaker that represents the content of the speaker's utterance; and an estimation step of estimating which of multiple types of patterns the speaker belongs to based on the frequently occurring words and multiple emotion values that represent the speaker's emotions, wherein the emotion values are either statistics of evaluation values that represent each emotion over a predetermined period from the start of the conversation, statistics of evaluation values that represent each emotion over a predetermined period going back a predetermined period from the end of the conversation, or statistics of evaluation values that represent each emotion over a predetermined period that has elapsed since a specific keyword was uttered in the conversation; and the estimation step estimates which of multiple types of patterns the speaker belongs to using the emotion values, numerical values that represent the frequently occurring words, and weights for each of the multiple types of patterns.

According to the present invention, sentiment analysis can be performed by understanding the content and context of conversations from text data based on audio data.

1 is a system configuration diagram showing an example of the configuration of an information processing system according to a first embodiment of the present invention. FIG. 2 is a block diagram showing an example of a hardware configuration of the information processing device according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a functional configuration of the information processing device according to the present embodiment. 1A to 1C are explanatory diagrams showing examples of an emotion/word matrix, a weight matrix, and a pattern matrix according to the present embodiment. FIG. 2 is a diagram showing an example of text data stored in a storage unit of the information processing device according to the present embodiment. FIG. 4 is a diagram illustrating an example of integrated data stored in a storage unit of the information processing apparatus according to the present embodiment. FIG. 10 is a diagram showing an example of an output display in the information processing device according to the embodiment. 10 is a flowchart illustrating an example of information processing in the information processing device according to the present embodiment. FIG. 10 is a block diagram illustrating an example of a functional configuration of an information processing apparatus according to a second embodiment of the present invention. 5 is a diagram showing an example of learning data used by a pattern learning unit in the information processing device according to the present embodiment. FIG. FIG. 10 is a diagram showing an example of an output display in the information processing device according to the embodiment. 10 is a flowchart illustrating an example of information processing in the information processing device according to the present embodiment.

[First embodiment]
Hereinafter, each embodiment of the present invention will be described with reference to the drawings.
<Configuration of information processing system SYS>
First, the configuration of the information processing system SYS will be described.

FIG. 1 is a system configuration diagram showing an example of the configuration of an information processing system SYS according to the first embodiment of the present invention.
The information processing system SYS includes an information processing device 100 and a call terminal device 200. The information processing device 100 and the call terminal device 200 are connected to each other via a network NW so as to be able to communicate with each other.

The information processing system SYS will be explained in more detail.

The information processing system SYS is a system that extracts multiple emotional values representing multiple emotions of multiple people (e.g., customers and operators) from voice data of a call between a customer and an operator at a call center, analyzes the content of the conversation (content of the call) and context by extracting frequently occurring words and the like from text data obtained by speech recognition of the voice data, and infers which of multiple patterns the conversation and speaker (e.g., operator) falls into based on the emotional values and frequently occurring words, and outputs the inferred result.
Here, the frequently occurring words include single words and phrases consisting of multiple words, but may also be either single words or phrases consisting of multiple words.

More specifically, the information processing system SYS acquires voice data and uses voice recognition to extract, for each speaker, multiple emotion values that represent the emotions of the speaker in the voice data and text data that represents the content of the speaker's utterance. The information processing system SYS also extracts one or more frequently occurring words contained in the text data, estimates which of multiple patterns the speaker belongs to based on the frequently occurring words and emotion values, and outputs the estimated pattern.

With this configuration, the information processing system SYS can understand the context of a conversation from frequently occurring words extracted from text data based on speech data and estimate patterns based on frequently occurring words and emotional values, thereby performing emotional analysis according to the conversation.

The information processing device 100 acquires the voice data of the call and extracts from the voice data multiple emotion values that represent the emotions of each speaker (e.g., the operator and the customer) over a predetermined period of time, for example, one minute from the start of the conversation. The emotions include joy, anger, sadness, happiness, empathy, kindness, sincerity, and other feelings felt by the customer or operator during the call. The emotion values are numerical values that indicate the degree of the above-mentioned emotions.

The emotion is not limited to these, and any emotion that a customer or an operator may have may be used.
The predetermined period for extracting the emotion value of a conversation from the voice data may be a period going back a certain period from the end of the conversation, or a certain period from the timing when a specific keyword is uttered in the conversation.

Furthermore, the information processing device 100 extracts text data of the conversation from the voice data by voice recognition. The text data corresponds to the voice data from which the emotion value was extracted.
The information processing device 100 extracts frequently occurring words from text data to understand the content and context of the conversation. The information processing device 100 uses the frequently occurring words and emotion values to estimate which of multiple patterns the conversation falls into. The information processing device 100 outputs the estimation result.

The call terminal device 200 is a terminal device used for calls. The call terminal device 200 may be a telephone, smartphone, call application, chat tool, etc., or a recorder or camera capable of recording audio data. In this embodiment, the call terminal device 200 is a terminal device used by an operator for calls with customers. The call terminal device 200 has the function of making calls with customers and the function of recording those calls.

In this embodiment, we will use a call at a call center as an example, but the present invention can be applied to any conversation between multiple people, whether face-to-face or online.

Next, we will explain the hardware configuration of the information processing device 100.

<Hardware configuration>
FIG. 2 is a block diagram showing an example of the hardware configuration of the information processing apparatus according to this embodiment.
The information processing device 100 includes a CPU 101, a storage medium interface unit 102, a storage medium 103, an input device 104, an output device 105, a ROM 106 (Read Only Memory), a RAM 107 (Random Access Memory), an auxiliary storage unit 108, and a network interface unit 109. The CPU 101, the storage medium interface unit 102, the input device 104, the output device 105, the ROM 106, the RAM 107, the auxiliary storage unit 108, and the network interface unit 109 are connected to each other via a bus.

Note that the term "CPU 101" here refers to a processor in general, and includes not only devices known as CPUs in the narrow sense, but also, for example, GPUs and DSPs. Furthermore, the CPU 101 referred to here is not limited to being realized by a single processor, but may also be realized by combining multiple processors of the same or different types.

<CPU 101>
The CPU 101 controls the information processing device 100 by reading and executing programs stored in the auxiliary storage unit 108, the ROM 106, and the RAM 107, and by reading various data stored in the auxiliary storage unit 108, the ROM 106, and the RAM 107 and writing the various data to the auxiliary storage unit 108 and the RAM 107. The CPU 101 also reads various data stored in the storage medium 103 via the storage medium interface unit 102 and writes the various data to the storage medium 103.

<Storage medium 103>
The storage medium 103 is a portable storage medium such as a magneto-optical disk, a flexible disk, or a flash memory, and stores various data.

<Storage medium interface unit 102>
The storage medium interface unit 102 is an interface for reading and writing data from and to the storage medium 103 .

<Input device 104>
The input device 104 is an input device such as a mouse, a keyboard, a touch panel, a microphone, a volume control button, a power button, a setting button, and an infrared receiver.

<Output device 105>
The output device 105 is an output device such as a display unit and a speaker.

<ROM106, RAM107>
The ROM 106 and RAM 107 store programs and various data for operating the various functional units of the information processing apparatus 100 .

<Auxiliary storage unit 108>
The auxiliary storage unit 108 is a hard disk drive, a flash memory, or the like, and stores programs for operating each functional unit of the information processing device 100 and various data.

<Network Interface Unit 109>
The network interface unit 109 has a communication interface and is connected to the network NW via wireless communication.

For example, the CPU 101 of the information processing device 100 corresponds to the control unit 15 in the functional configuration shown in FIG. 3. Furthermore, the ROM 106, RAM 107, auxiliary storage unit 108 of the information processing device 100, or any combination thereof, corresponds to the storage unit 12 in the functional configuration shown in FIG. 3. Furthermore, the input device 104 and output device 105 of the information processing device 100 correspond to the input unit 13 and output unit 14 in the functional configuration shown in FIG. 3.

Note that the hardware configuration of the call terminal device 200 is not shown or described here, but has the same hardware configuration as the information processing device 100 shown in Figure 2.

Next, the functional configuration of the information processing device 100 will be described.

<Functional configuration of information processing device 100>
FIG. 3 is a block diagram showing an example of the functional configuration of the information processing apparatus 100 according to this embodiment.
The information processing device 100 includes a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15. The communication unit 11, the storage unit 12, the input unit 13, the output unit 14, and the control unit 15 are connected to each other via a bus.

<Communication unit 11>
The communication unit 11 has a function of communicating with the call terminal device 200. The communication unit 11 outputs various information received from the call terminal device 200 to the control unit 15. The communication unit 11 also transmits information input from the control unit 15 to the call terminal device 200.

<Storage unit 12>
The storage unit 12 is configured by a storage medium, for example, a hard disk drive (HDD), a flash memory, an electrically erasable programmable read-only memory (EEPROM), a random access read/write memory (RAM), a read-only memory (ROM), or any combination of these storage media.
The storage unit 12 may be, for example, a nonvolatile memory.

The storage unit 12 stores text data 121 , integrated data 122 , and pattern data 123 .
The text data 121 is generated by speech recognition of the voice data, and is associated with an emotion value extracted from the voice data.
The integrated data 122 is data in which frequently occurring words extracted from text data, emotional values of conversations in the voice data corresponding to the text data, and call types are associated with each other. The call type is attribute information that identifies the type of call, such as an inbound call (received to a call center), an outbound call (made from a call center), or the content of an inquiry.
The pattern data 123 is data of a plurality of model patterns in which frequently occurring words are associated with emotion values.

<Input unit 13>
The input unit 13 is an input device such as a mouse, keyboard, or microphone connected to the information processing device 100. The input unit 13 accepts an operation input input from outside. The input unit 13 outputs an operation signal corresponding to the operation input to the control unit 15.

<Output unit 14>
The output unit 14 is an output device such as a display device, etc. The output unit 14 outputs the presentation information output from the control unit 15 to the output device or another device such as the call terminal device 200.

<Control unit 15>
The control unit 15 has a function of controlling the information processing device 100. The control unit 15 reads out various data, applications, programs, etc. stored in the storage unit 12 and controls the information processing device 100.

The processing of the control unit 15 will be described in more detail.
The control unit 15 includes a voice data acquisition unit 151 , a text data extraction unit 152 , an emotion/text integration unit 153 , a pattern estimation unit 154 , and a pattern output unit 155 .

<Audio Data Acquisition Unit 151>
The voice data acquisition unit 151 acquires voice data from the communication terminal device 200 via the network NW and the communication unit 11. The voice data acquisition unit 151 outputs the voice data to the text data extraction unit 152.

<Text Data Extraction Unit 152>
When voice data is input from the voice data acquisition unit 151, the text data extraction unit 152 extracts text data by voice recognition of the voice data. The text data corresponds to voice data for a predetermined period of time, for example, one minute, from the start of the call. The text data is also text data in which speakers and the content of each speaker's speech are converted into text by speaker recognition. The text data extraction unit 152 stores the extracted text data in the storage unit 12 in association with identification information that identifies the call.
Furthermore, the text data extraction unit 152 extracts, for each utterance, a plurality of emotional values of the speaker, e.g., the customer and the operator, for a predetermined period from the start of the call from the voice data, using, for example, known technology. The text data extraction unit 152 associates the extracted emotional values with the text data and stores them in the storage unit 12.
Here, speaker recognition is performed by recognizing the voices of multiple speakers using existing voice separation technology or the like.

The emotion value may be a statistical quantity such as a mean value, median value, or variance value obtained by averaging each emotion value over a predetermined period.
The predetermined period may be a period until the end of the conversation, a period going back a certain period from the end of the conversation, or a period of a certain amount of time elapsed from the time a specific keyword was uttered during the conversation.

<Emotion and text integration unit 153>
The emotion/text integrator 153 reads the text data 121 from the storage unit 12 and integrates the emotion value with the text data for each utterance. Specifically, the emotion/text integrator 153 extracts words (frequently occurring words) and phrases for each utterance in the text data and quantifies the importance of each extracted word and phrase using, for example, TF-IDF (Term Frequency - Inverse Document Frequency). The emotion/text integrator 153 associates the emotion value, frequently occurring words, and scores such as the importance of each word or phrase for each utterance, and stores the associated data as integrated data 122 in the storage unit 12.

<Pattern Estimation Unit 154>
The pattern estimation unit 154 reads the integrated data 122 from the storage unit 12 and extracts patterns of emotions and words. Specifically, the pattern estimation unit 154 extracts emotion values and word scores from the integrated data 122 and defines them as an emotion-word matrix. The pattern estimation unit 154 decomposes the emotion-word matrix into a weight matrix and a pattern matrix. For example, singular value decomposition, non-negative matrix factorization, neural network, or the like is used to decompose the emotion-word matrix, weight matrix, and pattern matrix into correspondence with each other and stores them in the storage unit 12 as pattern estimation results.

In this embodiment, for example, non-negative matrix factorization is used to decompose into a weight matrix and a pattern matrix. By doing so, all of the values of the matrix decomposed into a weight matrix and a pattern matrix are non-negative, making it easier to intuitively understand the pattern matrix and weight matrix obtained as a result of the decomposition.

In addition, the pattern estimation unit 154 is not limited to linear algorithms such as matrix decomposition, and may extract emotion and word patterns using nonlinear algorithms such as k-nearest neighbor, UMAP (Uniform Manifold Approximation and Projection), and t-SNE (t-distributed stochastic neighbor embedding), or combinations of these.

Please refer to Figure 4 for explanation.

FIG. 4 is an explanatory diagram of the emotion/word matrix, weight matrix, and pattern matrix according to this embodiment.
The illustrated example is an example in which there is one type of operator emotion value (OP emotion value), one type of customer emotion value (CU emotion value), and one type of word, and the number of patterns is three.
If the number of calls is N, the (N×3) emotion/word matrix is decomposed into an (N×3) weight matrix and a (3×3) pattern matrix.
The weight matrix indicates the weight of each pattern in each call, and also indicates which emotions and words have a strong relationship in each conversation.
Each column of the pattern matrix indicates the relative strength of the relationship between emotion and word by its weight, and each row value of the pattern matrix represents a common pattern of words and emotions in N conversations.

Note that the size of the matrix is not limited to this, and can be changed depending on the number of emotion values that can be obtained, the type of words, and the number of words. Also, numerical values other than word scores and emotion values, such as call type, may be used.

<Pattern Output Unit 155>
The pattern output unit 155 reads out the pattern estimation result from the storage unit 12, and generates a display image including emotions corresponding to the emotion values using the emotion-word matrix, words corresponding to the word scores, weights in the weight matrix, patterns in the pattern matrix, and pattern data based on past cases. The pattern output unit 155 outputs the generated display image to an output device or a call terminal device 200.

The pattern output unit 155 may output both each pattern in the pattern matrix and the word corresponding to the word score, i.e., the score of the frequently occurring word, or it may output either one of them.

Next, we will explain the various data stored in the memory unit 12 of the information processing device 100.

FIG. 5 is a diagram showing an example of text data 121 stored in the storage unit 12 of the information processing device 100 according to this embodiment.
The illustrated text data is data in which a call identification number, a start time, an end time, a speaker, the content of the speech, an emotion value (A), an emotion value (B), and an emotion value (C) are associated with each other.
The call identification number is identification information (number) that identifies a call.
The start time and end time are time information indicating the start time and end time of an utterance, and are acquired from time information included in the voice data, for example.
The speaker represents a speaker whose voice has been separated by speaker recognition. Note that as long as the speaker can be distinguished, such as speaker A and speaker B, it is not necessary to distinguish between types of people, such as "operator" and "customer."
The speech content is text data representing the content of the speech by the speaker, and is generated by speech recognition.
Emotional value (A), emotional value (B), and emotional value (C) are numerical values that represent each of the emotions set in advance. For example, emotional value (A) is a numerical value that represents joy, emotional value (B) is a numerical value that represents anger, and so on. Emotional values corresponding to a plurality of emotions are output from the voice data for each utterance using known technology.
As shown in the figure, the text data associates the content of an utterance for each speaker with a plurality of emotion values for each utterance content.

FIG. 6 is a diagram showing an example of the integrated data 122 stored in the storage unit 12 of the information processing device 100 according to this embodiment.
The integrated data shown in the figure is data in which call identification numbers, operator word importance levels, customer word importance levels, operator emotion values, and customer emotion values are associated with each other.
The integrated data may be associated with a call type.
The call identification number is identification information that identifies a call.
The operator word importance indicates the score of a word extracted from the speech content of the operator. For example, the operator word importance indicates a predetermined number of words in descending order of word importance, and is represented as a numerical value associated with the importance of each word.
The customer word importance indicates the score of a word extracted from the customer's speech. For example, the customer word importance indicates a predetermined number of words (frequently occurring words) in descending order of word importance, and is represented as a numerical value associated with the importance of each word.
The operator emotion value and the customer emotion value are emotion values that represent emotions extracted from the speech content of the operator and the customer.

Note that while an example of generating integrated data by converting it into numerical values has been described, integrated data may also be generated by converting emotion values into text.

Next, we will explain an example of output from the output unit 14 of the information processing device 100.

FIG. 7 is a diagram showing an example of an output display in the information processing device 100 according to this embodiment.
The output display shown in the figure is an example of a case where there are three types of patterns and the display image generated by the pattern output unit 155 is output to the output device.
For example, assume that emotion values (A) for emotion A and (B) for emotion B for the operator and the customer are extracted from speech data, that the words UUU, VVV, and WWW are extracted as frequently occurring words based on the speech data, and that the word importance (word scores) of the frequently occurring words are extracted. The information processing device 100 generates an emotion-word matrix according to the emotion values and word scores, and decomposes the emotion-word matrix into a weight matrix and a pattern matrix.
The display image includes an area for displaying the weight of emotion/word patterns, an area for listing emotion/word patterns, an area for pattern examples, and an area for overall evaluation.

The emotion/word pattern weight display area reads each column of the weight matrix and displays the weight of the pattern for each call. The strength (large or small) of the weight of each pattern expresses the characteristics of the call content. By using the pattern weights in this way, it is possible to determine and compare the trends of each call. In the example shown, it is displayed as "Pattern 1 weight 1.5," "Pattern 2 weight 0.1," and "Pattern 3 weight 0.5."

The emotion/word pattern list area is an area that reads the values of each row of the pattern matrix and displays the patterns in various graphs. The emotion/word pattern list area is also an area that displays, for example, a predetermined number of highly important words (words with large values in the weight matrix) as "frequent words."
Instead of words, important phrases or phrases that are not allowed for the call center may be displayed.

The pattern example area is a surface area that displays pattern data stored as pattern data in the memory unit 12. For example, the pattern data may hold emotion and word patterns of highly skilled operators and emotion and word patterns of less experienced and less skilled operators. The pattern output unit 155 searches the pattern data stored in the memory unit 12 and displays good patterns to follow and bad patterns to serve as counterexamples.
This allows operators to, for example, refer to customer emotion values from past cases to check how emotionally they should speak, what words they should use, etc. It is also possible to identify the content of a call from words, for example, and compare it with the emotion of a highly skilled operator at that time.

The pattern data may also contain emotion and word patterns linked to other indicators related to call center and sales conversations. Indicators include, for example, operator turnover rate and customer satisfaction surveyed separately via social media, the web, or questionnaires. By linking these indicators to emotion and word patterns, it is possible to understand trends, such as which emotion and word patterns improve the indicators.

The comprehensive evaluation area is an area that displays advice that compares the emotion and word patterns in a certain call with patterns stored as pattern data.
In the example shown, the conversation content is estimated based on frequently occurring words and emotional values, and the display shows an evaluation of whether the conversation is being conducted with appropriate emotions and whether appropriate words are being used, advice for improving emotions, and advice such as words that should be used to improve customer impressions such as satisfaction, such as "Evaluation: In the current conversation content, pattern 1 is the closest. In the content of this call, it is close to an undesirable pattern.", "Emotional advice: Try to lower the tone of your voice," and "Word advice: Talking to XXX will improve customer satisfaction."

Next, we will explain the flow of information processing related to this embodiment.

FIG. 8 is a flowchart showing an example of information processing in the information processing device 100 according to this embodiment.
In step S101, the information processing device 100 acquires audio data. Then, the information processing device 100 executes the process of step S102.
In step S102, the information processing device 100 extracts text data from the voice data by voice recognition. Then, the information processing device 100 executes the process of step S103.

In step S103, the information processing device 100 extracts a plurality of emotion values for each conversation from the voice data. After that, the information processing device 100 executes the process of step S104.
In step S104, information processing device 100 generates integrated data by integrating the text data and the emotion values. Specifically, information processing device 100 calculates word importance for each conversation and generates integrated data by associating a predetermined number of words with high word importance with the word importance and the emotion values extracted from the audio data of the conversation. Information processing device 100 then performs the process of step S105.

In step S105, the information processing device 100 reads the integrated data and generates an emotion-word matrix. The information processing device 100 also decomposes the emotion-word matrix into a weight matrix and a pattern matrix to estimate conversation patterns. Then, the information processing device 100 executes the process of step S106.

Here, estimating a conversation pattern includes indicating weights for multiple patterns, such as "weight of pattern 1: 1.5," "weight of pattern 2: 0.1," and "weight of pattern 3: 0.5," as described above. Estimating a conversation pattern also includes selecting the weight with the largest weight value from the weights of multiple patterns and estimating, for example, that it is pattern 1 or that pattern 1 is strong based on the selected weight. Estimating a conversation pattern also includes setting a threshold for the weights of multiple patterns and estimating a tendency from patterns with weights above the threshold, such as when the threshold is 0.3 in the above example, pattern 1 is strong, followed by pattern 3.

In step S106, the information processing device 100 outputs the estimation result. Then, the information processing device 100 ends the information processing related to FIG. 8.

As such, the information processing system SYS according to this embodiment comprises a voice data acquisition unit 151 that acquires voice data; an extraction unit (text data extraction unit 152) that uses voice recognition to extract from the voice data, for each speaker, multiple emotion values that represent the emotions of the speaker in the voice data and text data that represents the content of the speaker's utterance; a frequent word extraction unit (emotion/text integration unit 153) that extracts one or more frequently occurring words included in the text data and further integrates the text data and emotion values into a numerical or text format to generate integrated data; an estimation unit (pattern estimation unit 154) that estimates which of multiple patterns the utterance falls into based on the frequently occurring words and emotion values; and an output unit (pattern output unit 155) that outputs the estimated pattern.

In this way, emotion analysis can be performed after understanding the content and context of the conversation from text data based on audio data. Furthermore, by extracting and displaying patterns along with words and phrases in addition to emotion values, it is possible to visualize the speaker's skill according to the content of the speech. Furthermore, by providing feedback on model emotions and speaking tendencies, it is possible to improve the speaker's skill according to the content of the speech.

Furthermore, in the above-mentioned information processing system SYS, the emotion values are statistical quantities such as the average values of each emotion value over a predetermined period of time from the start of the conversation, and the estimation unit (pattern estimation unit 154) estimates which of multiple patterns the speaker belongs to using the emotion values, numerical values representing frequently occurring words, and weights for each of multiple patterns.

By doing this, it is possible to use statistical values such as emotion values and average emotion values over a certain period from the start of a conversation, which are likely to reveal characteristics of the conversation content, thereby improving the accuracy of estimating patterns appropriate for the conversation content.

In the above-mentioned information processing system SYS, the output unit (pattern output unit 155) outputs patterns and frequently occurring words.

By doing this, you can check what words and emotions to use in the conversation.

In the information processing system SYS, the voice data is call voice data, and the output unit (pattern output unit 155) matches a speaker with a call partner of the speaker according to emotion and word patterns.
Specifically, for an operator, if the operator is similar (approximate) to, for example, "weight of pattern 1: 1.5,""weight of pattern 2: 0.1," or "weight of pattern 3: 0.5," the pattern is considered to be friendly, and if the operator is similar to, for example, "weight of pattern 1: 1.0,""weight of pattern 2: 1.0," or "weight of pattern 3: 1.0," the pattern is considered to be calm.
Furthermore, with regard to customers, if the pattern is similar to, for example, "Pattern 1 weight 1.0,""Pattern 2 weight 1.0," or "Pattern 3 weight 1.0," it is considered a sociable pattern, and if the pattern is similar to, for example, "Pattern 1 weight 1.5,""Pattern 2 weight 1.0," or "Pattern 3 weight 1.0," it is considered a critical pattern.

In this case, matching patterns are previously defined in the information processing system SYS such that a customer with a sociable pattern is matched with an operator with a friendly pattern, and a customer with a critical pattern is matched with an operator with a calm pattern.
As a result, the output unit (pattern output unit 155) determines patterns for the personalities of customers and agents, and matches customers with agents according to the matching patterns.

The information processing device 100 also includes an audio data acquisition unit 151 that acquires audio data, an extraction unit (text data extraction unit 152) that uses voice recognition to extract, for each speaker, from the audio data, multiple emotion values that represent the emotions of the speaker in the audio data and text data that represents the content of the speaker's utterance, a frequent word extraction unit (emotion/text integration unit 153) that extracts one or more frequently occurring words included in the text data, an estimation unit (pattern estimation unit 154) that estimates which of multiple patterns the speaker belongs to based on the frequently occurring words and emotion values, and an output unit (pattern output unit 155) that outputs the estimated pattern.

In this way, emotion analysis can be performed after understanding the content and context of the conversation from text data based on audio data. Furthermore, by extracting and displaying patterns along with words and phrases in addition to emotion values, it is possible to visualize the speaker's skill according to the content of the speech. Furthermore, by providing feedback on model emotions and speaking tendencies, it is possible to improve the speaker's skill according to the content of the speech.

Second Embodiment
In the second embodiment, an example of learning conversation-related indices through supervised learning will be described.
Here, in the second embodiment, the differences from the first embodiment will be mainly described, and the first embodiment will be used for the other parts, and the description thereof will be omitted.

FIG. 9 is a block diagram showing an example of the functional configuration of an information processing apparatus 100 according to the second embodiment of the present invention.
The information processing device 100 includes a communication unit 11, a storage unit 12, an input unit 13, an output unit 14, and a control unit 15. The communication unit 11, the storage unit 12, the input unit 13, the output unit 14, and the control unit 15 are connected to each other via a bus.

<Storage unit 12>
The storage unit 12 stores text data 121 , integrated data 122 , pattern data 123 , index data 124 , and a learning model 125 .
The index data 124 is, for example, data on indexes to be checked by a call center QA (Quality Administrator), management-related index data, index data to be improved by analyzing emotions and words, and related data thereof.

The data of the indicators that the QA should check is, for example, the following data of the indicators.
Response Time: An indicator of the time it takes for an operator to respond to a customer inquiry.
Average Handling Time: An indicator of the average length of time spent talking with a customer.
First Call Resolution Rate: An indicator of the percentage of customers who reach a resolution on their first inquiry.
- Customer Satisfaction: An indicator that measures how satisfied a customer is with the operator's response.
- Operator Compliance: An indicator that evaluates whether operators are complying with corporate and industry rules, legal compliance, and privacy protection.
- Operator Skills and Knowledge: An indicator that evaluates whether the operator has appropriate knowledge of products and services and customer service skills.
- Call Quality: An indicator that evaluates the quality of a call, such as how easy it is to hear the operator's voice, how clear it is, the expressions and phrases that should be spoken, the expressions and phrases that should not be spoken, and the timing of speaking.
- Escalation Rate: An indicator of the rate at which an operator escalates an inquiry to a senior or supervisor.

Furthermore, the management indicators include, for example, the following indicator data:
- Cost Performance: An indicator of the balance between operating costs and customer satisfaction.
・Operator Efficiency: An indicator that evaluates whether operators are performing their work efficiently, including call time, waiting time, and time to resolution.
- Operator Retention Rate: An indicator of the percentage of operators who have been working at the call center for a long period of time.

Furthermore, for example, it may be an index related to digital marketing collected from the web, SNS, etc.
Conversion Rate: An indicator that shows the percentage of visitors who actually make a purchase or make an inquiry as a result of marketing activities.
Bounce Rate: An indicator that shows the percentage of visitors who leave the site within a short period of time after accessing it.
Click-Through Rate: An indicator that shows the percentage of people who clicked on content such as an advertisement or email newsletter.
・Social Share Rate: An indicator showing the percentage of people who shared content on social media etc.
- Repeat Rate: An indicator showing the percentage of people who have visited a site in the past who visit again.
Net Promoter Score (NPS): A metric that measures how likely a customer is to recommend a brand or service to others.

The learning model 125 is a learning model generated by the pattern learning unit 156.

<Control unit 15>
The control unit 15 includes a voice data acquisition unit 151 , a text data extraction unit 152 , an emotion/text integration unit 153 , a pattern estimation unit 154 , a pattern output unit 155 , and a pattern learning unit 156 .

<Pattern Learning Unit 156>
The pattern learning unit 156 sets the index data stored in the storage unit 12 as a target variable, sets the text data including emotion values stored in the storage unit 12 as an explanatory variable, and generates a learning model by learning the correspondence between them using machine learning including deep learning, etc. As a result, by inputting the text data including emotion values into the learning model, various indices are output.

The pattern learning unit 156 may generate a learning model according to the format of the learning data. For example, in the case of quantified learning data, a machine learning model that is strong in table data, such as Lightgbm, may be used, and in the case of text learning data, a language model, such as BERT, may be used.

<Pattern Estimation Unit 154>
The pattern estimation unit 154 utilizes the learning model stored in the memory unit 12, inputs text data including emotional values as explanatory variables into the learning model, and obtains the index set as the objective variable as output, thereby estimating (predicting) the index.

<Pattern Output Unit 155>
The pattern output unit 155 generates a display image showing the relationship between the predicted index, the content of the conversation, and the emotion value.

Fig. 10 is a diagram showing an example of learning data used by the pattern learning unit 156 in the information processing device 100 according to this embodiment. Fig. 11 is a diagram showing an example of an output display in the information processing device 100 according to this embodiment.
The learning data in this embodiment is, for example, text data including emotion values stored in the storage unit 12. The learning data is data in which call identification numbers, text data, operator emotion values, and customer emotion values are associated with each other.
Furthermore, the learning data used is text data associated with emotion values shown in FIG. 10 and index data corresponding to the text data.
Note that instead of the text data including emotion values shown in Fig. 10, the text data including emotion values shown in Fig. 5 may be used as training data. For example, the emotion values are converted into text, such as "emotion A = 10, emotion B = 20, emotion C = 0", and combined with the text data of the speech recognition results to be used as training data.

In addition, in the learning data shown in FIG. 10, learning may be performed using learning data in which index data stored in the memory unit 12 is associated with the operator emotion value and the customer emotion value, instead of or in addition to the operator emotion value and the customer emotion value.

For example, when text data including the emotion values shown in FIG. 10 are input as explanatory variables to a learning model trained in this way, indicators A and B corresponding to the call identification number, text data, operator emotion value, and customer emotion value shown in FIG. 10 are output as objective variables, as shown in FIG. 11. The learning model may also output which words or emotion values were focused on when predicting an index. For example, a deep learning model with an attention mechanism can be used as the learning model, and the output values of the intermediate layer of the learning model can be used to output which words or emotion values were focused on when predicting an index.
Here, the output display shown in FIG. 11 in the above description is an example of a display image generated by the pattern output unit 155 based on the index obtained as an output from the learning model.
Note that which words or emotion values were focused on when estimating the index may be displayed in a way that is easy for the user to understand by changing the display mode, such as the text color or background color. In this way, it is possible to visualize in an easy-to-understand manner which words or emotion values contributed to the index prediction.

Next, we will explain the information processing related to this embodiment.

FIG. 12 is a flowchart showing an example of information processing in the information processing device 100 according to this embodiment.
In step S201, the information processing device 100 acquires audio data. Then, the information processing device 100 executes the process of step S202.
In step S202, the information processing device 100 extracts text data from the voice data by voice recognition. Then, the information processing device 100 executes the process of step S203.

In step S203, the information processing device 100 extracts a plurality of emotion values for each conversation from the voice data. After that, the information processing device 100 executes the process of step S204.
In step S204, the information processing device 100 generates integrated data by integrating the text data and the emotion values. Specifically, the information processing device 100 calculates word importance for each conversation and generates integrated data by associating a predetermined number of words with high word importance with the word importance and the emotion values extracted from the speech data of the conversation. Alternatively, the information processing device 100 may generate integrated data by converting the emotion values into text and combining the emotion values converted into text with the text data of the speech recognition result. Then, the information processing device 100 executes the process of step S205.

In step S205, when text data including emotion values is input, the information processing device 100 generates a learning model that outputs an index (index value). Thereafter, the information processing device 100 executes the process of step S206.
In step S206, the information processing device 100 reads out the learning model 125 stored in the storage unit 12 and inputs text data including emotion values into the learning model 125 to obtain an index as an output. The information processing device 100 estimates the output index as a pattern. Thereafter, the information processing device 100 executes the process of step S207.

In step S207, the information processing device 100 outputs the estimation result. Then, the information processing device 100 ends the information processing related to FIG. 12.

As such, the information processing system SYS according to this embodiment comprises a voice data acquisition unit 151 that acquires voice data, an extraction unit (text data extraction unit 152) that uses voice recognition to extract from the voice data, for each speaker, multiple emotion values that represent the emotions of the speaker in the voice data and text data that represents the content of the speaker's utterance, a frequent word extraction unit (emotion/text integration unit 153) that extracts one or more frequently occurring words included in the text data and further integrates the text data and emotion values into a numerical or text format to generate integrated data, a pattern learning unit 156 that machine-learns a learning model based on the training data, an estimation unit (pattern estimation unit 154) that inputs text and emotion values and predicts an index using the learning model, and an output unit (pattern output unit 155) that outputs the estimated index.

In this way, sentiment analysis can be performed after understanding the content and context of the conversation from text data based on audio data. Furthermore, by simultaneously performing machine learning on not only emotional values but also words and phrases, and predicting and displaying various indicators, it is possible to visualize the speaker's skill according to the content of the utterance. Furthermore, by displaying the emotional values and words that the learning model focused on when predicting indicators, it is possible to provide feedback on exemplary emotions and speaking tendencies, thereby improving the speaker's skill according to the content of the utterance.

In the above-mentioned information processing system SYS, the output unit (pattern output unit 155) displays the predicted index. Furthermore, by displaying the emotional values and words that were focused on when predicting the index, feedback can be provided on the emotions and speech tendencies that should be modeled, thereby improving the speaker's skills according to the content of the speech.

In the information processing system SYS, the voice data is call voice data, and the output unit (pattern output unit 155) matches the speaker with the call partner of the speaker according to the pattern of the predicted index.
Specifically, for example, when an operator's personality resembles "index 1 value 1.5" and "index 2 value 0.1," the operator's personality is deemed to be friendly, and when it resembles "index 1 value 1.0" and "index 2 value 1.0," the operator's personality is deemed to be calm. For example, when a customer's personality resembles "index 3 value 1.0" and "index 4 value 1.0," the operator's personality is deemed to be sociable, and when it resembles "index 3 value 1.5" and "index 4 value 1.0," the operator's personality is deemed to be critical. Matching patterns are predefined in the information processing system SYS, such that a customer with a sociable pattern is matched with an operator with a friendly pattern, and a customer with a critical pattern is matched with an operator with a calm pattern. The output unit (pattern output unit 155) determines the personality patterns of the customer and the operator, and matches the customer and the operator according to the matching pattern.

By doing this, it is possible to match the appropriate speaker to the person you are calling based on the pattern.

The information processing system SYS further includes a prediction unit (not shown) that predicts an evaluation value by inputting the emotional value and text data into a learning model generated by learning the correspondence between text data, the emotional value corresponding to the text data, and the evaluation value for the conversation corresponding to the text data.

Since evaluation values can be output along with the patterns, it is possible to visualize the speaker's skills according to the content of the utterance. In addition, it is possible to provide feedback on exemplary emotions and speaking tendencies, allowing the speaker's skills to be improved according to the content of the utterance.

The information processing device 100 also includes an audio data acquisition unit 151 that acquires audio data, an extraction unit (text data extraction unit 152) that uses speech recognition to extract from the audio data, for each speaker, multiple emotion values that represent the emotions of the speaker in the audio data and text data that represents the content of the speaker's utterance, an integration unit that integrates the text data and emotion values into a numerical value or text format, a frequent word extraction unit (emotion/text integration unit 153) that integrates the text data and emotion values into a numerical value or text format to generate integrated data in the process of extracting one or more frequently occurring words included in the text data, a pattern learning unit 156 that machine-learns a learning model based on the training data, an estimation unit (pattern estimation unit 154) that inputs text and emotion values and predicts an index using the learning model, and an output unit (pattern output unit 155) that outputs the estimated index.

In this way, sentiment analysis can be performed after understanding the content and context of the conversation from text data based on the voice data.
Furthermore, by simultaneously learning not only emotional values but also words and phrases through machine learning and predicting and displaying various indicators, it is possible to visualize the speaker's skill according to the content of the utterance. Furthermore, by displaying the emotional values and words that the learning model focused on when predicting the indicators, it is possible to provide feedback on the emotions and speaking tendencies that should be modeled, thereby improving the speaker's skill according to the content of the utterance.

<Additional Notes>
One aspect of the present invention is an information processing system comprising: a voice data acquisition unit that acquires voice data; an extraction unit that extracts, for each speaker, from the voice data using voice recognition, a plurality of emotional values that represent the emotions of the speaker in the voice data and text data that represents the content of the utterances made by the speaker; a frequent word extraction unit that extracts one or more frequent words included in the text data; an estimation unit that estimates which of a plurality of patterns the speaker belongs to based on the frequent words and the emotional values; and an output unit that outputs a score of the estimated pattern or the frequent words.

Another aspect of the present invention is an information processing device comprising: a voice data acquisition unit that acquires voice data; an extraction unit that extracts, for each speaker, from the voice data using voice recognition, a plurality of emotion values that represent the emotions of the speakers in the voice data and text data that represents the content of the utterances made by the speakers; a frequent word extraction unit that extracts one or more frequent words included in the text data; an estimation unit that estimates which of a plurality of patterns the speaker belongs to based on the frequent words and the emotion values; and an output unit that outputs a score for the estimated pattern or the frequent words.

Another aspect of the present invention is an information processing method executed by a computer of an information processing device, comprising: a voice data acquisition step for acquiring voice data; an extraction step for extracting, for each speaker, from the voice data using voice recognition, multiple emotion values representing the emotions of the speakers in the voice data and text data representing the content of the utterances made by the speakers; a frequent word extraction step for extracting one or more frequently occurring words included in the text data; an estimation step for estimating which of multiple patterns the speaker belongs to based on the frequent words and the emotion values; and an output step for outputting a score for the estimated pattern or the frequent words.

Another aspect of the present invention is a program that causes a computer of an information processing device to execute the following steps: a voice data acquisition step for acquiring voice data; an extraction step for extracting, for each speaker, from the voice data using voice recognition, multiple emotion values that represent the emotions of the speakers in the voice data and text data that represents the content of the utterances made by the speakers; a frequent word extraction step for extracting one or more frequently occurring words included in the text data; an estimation step for estimating which of multiple patterns the speaker belongs to based on the frequently occurring words and the emotion values; and an output step for outputting the score of the estimated pattern or the frequently occurring words.

<Appendix 1>
an extraction unit that extracts, for each speaker, from the voice data using voice recognition, a plurality of emotional values that represent the emotions of the speaker in the voice data and text data that represents the content of the speech by the speaker; a frequent word extraction unit that extracts one or more frequently occurring words included in the text data; an estimation unit that estimates which of a plurality of patterns the speaker belongs to based on the frequent words and the emotional values; and an output unit that outputs a score of the estimated pattern or the frequent words.

<Appendix 2>
The information processing system according to claim 1, wherein the emotion values are either statistics of emotion values for a predetermined period from the start of a conversation, statistics of emotion values for a predetermined period going back a certain period from the end of the conversation, or statistics of emotion values for a predetermined period a certain period after a specific keyword was uttered in the conversation, and the estimation unit estimates which of a plurality of patterns the speaker belongs to using the emotion values, numerical values representing the frequently occurring words, and weights for each of a plurality of types of patterns.

<Appendix 3>
and a prediction unit that predicts the evaluation value by inputting the emotional value and the text data into a learning model that is generated by learning a correspondence relationship between the text data, the emotional value corresponding to the text data, and an evaluation value for a conversation that corresponds to the text data.

<Appendix 4>
2. The information processing system according to claim 1, wherein the frequent word extraction unit integrates the text data and the emotion values corresponding to the text data as all numerical values or all text.

<Appendix 5>
The information processing system according to claim 3, wherein the voice data is call voice data, and the output unit matches the speaker with a call partner of the speaker according to the pattern or the evaluation value.

<Appendix 6>
an extraction unit that extracts, for each speaker, from the voice data using voice recognition, a plurality of emotional values that represent the emotions of the speaker in the voice data and text data that represents the content of the utterance by the speaker; a frequent word extraction unit that extracts one or more frequently occurring words included in the text data; an estimation unit that estimates which of a plurality of patterns the speaker belongs to based on the frequent words and the emotional values; and an output unit that outputs a score of the estimated pattern or the frequent words.

<Appendix 7>
An information processing method to be executed by a computer of an information processing device, the information processing method comprising: a voice data acquisition step of acquiring voice data; an extraction step of extracting, for each speaker, from the voice data by voice recognition, a plurality of emotional values that represent the emotions of the speakers in the voice data and text data that represent the content of the utterances made by the speakers; a frequent word extraction step of extracting one or more frequently occurring words included in the text data; an estimation step of estimating which of a plurality of patterns the speaker belongs to based on the frequent words and the emotional values; and an output step of outputting a score of the estimated pattern or the frequent words.

<Appendix 8>
A program for causing a computer of an information processing device to execute the following steps: an audio data acquisition step for acquiring audio data; an extraction step for extracting, for each speaker, from the audio data using voice recognition, a plurality of emotional values that represent the emotions of the speaker in the audio data and text data that represents the content of the speech by the speaker; a frequent word extraction step for extracting one or more frequently occurring words included in the text data; an estimation step for estimating which of a plurality of patterns the speaker belongs to based on the frequent words and the emotional values; and an output step for outputting a score of the estimated pattern or the frequent words.

Each embodiment of the present invention has been described in detail above with reference to the drawings, but the specific configuration is not limited to that described above, and various design modifications can be made without departing from the spirit of the present invention.

For example, in each of the above-described embodiments, an example has been described in which the information processing device 100 and the call terminal device 200 are configured as separate devices, but one aspect of the present invention may also be realized by a device that combines some or all of these devices, or a device in which some of these devices are rearranged.

In one aspect of the present invention, the program running on the information processing device 100 and the call terminal device 200 may be a program (a program that causes a computer to function) that controls one or more processors, such as a CPU (Central Processing Unit), so as to realize the functions described in the above-described embodiments and variations related to one aspect of the present invention. The term "computer" here also includes quantum computers. Information handled by each of these devices is temporarily stored in RAM (Random Access Memory) during processing, and then stored in various storage devices such as flash memory and HDD (Hard Disk Drive), and may be read, modified, or written by the CPU or the like as needed.

In addition, part or all of the information processing device 100 and the call terminal device 200 in each of the above-mentioned embodiments and variations may be implemented by a computer having one or more processors. In this case, the program for implementing this control function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed to implement the function.

Note that the term "computer system" used here refers to the computer system built into the information processing device 100 and the communication terminal device 200, and includes hardware such as the OS and peripheral devices. Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into the computer system.

Furthermore, "computer-readable recording medium" may include something that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or something that stores a program for a fixed period of time, such as volatile memory within a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes some of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

Furthermore, part or all of the information processing device 100 and the call terminal device 200 in each of the above-described embodiments and variations may be realized as an LSI, which is typically an integrated circuit, or as a chipset. Furthermore, each functional block of the information processing device 100 and the call terminal device 200 in each of the above-described embodiments and variations may be individually formed into a chip, or part or all may be integrated into a chip. Furthermore, the integrated circuit method is not limited to LSI, and may be realized using a dedicated circuit and/or a general-purpose processor. Furthermore, if an integrated circuit technology that can replace LSI emerges due to advances in semiconductor technology, it may also be possible to use an integrated circuit based on that technology.

The above has described in detail each embodiment and variant of this invention with reference to the drawings, but the specific configuration is not limited to each embodiment or variant, and includes design modifications within the scope of the invention. Furthermore, various modifications of this aspect of the invention are possible within the scope of the claims, and embodiments obtained by appropriately combining the technical means disclosed in different embodiments are also included in the technical scope of the present invention. Furthermore, configurations in which elements described in the above embodiments and variants are substituted for elements that achieve the same effect are also included.

SYS Information processing system 100 Information processing device 101 CPU
102 Storage medium interface unit 103 Storage medium 104 Input device 105 Output device 106 ROM
107 RAM
108 Auxiliary storage unit 109 Network interface unit 11 Communication unit 12 Storage unit 121 Text data 122 Integrated data 123 Pattern data 124 Index data 125 Learning model 13 Input unit 14 Output unit 15 Control unit 151 Voice data acquisition unit 152 Text data extraction unit 153 Emotion/text integration unit 154 Pattern estimation unit 155 Pattern output unit 156 Pattern learning unit 200 Call terminal device

Claims (10)

a prediction unit that predicts the evaluation value by inputting text data for each speaker that represents the content of an utterance by the speaker, a plurality of emotion values that correspond to the text data and represent the emotions of the speaker, and an evaluation value for a conversation that corresponds to the text data ; and
An information processing system comprising: the emotion values include an operator emotion value representing an emotion extracted from the content of the operator's utterance, and a customer emotion value representing an emotion extracted from the content of the customer's utterance,
the prediction unit predicts the evaluation value by inputting the text data, the operator emotion value, and the customer emotion value into the learning model generated based on correspondence relationships among the text data, the operator emotion value, the customer emotion value, and the evaluation value;
The evaluation value includes at least one of the following: the customer satisfaction level of the customer with respect to the response of the operator; an index value evaluating the response skills of the operator; an index value evaluating the knowledge of the operator regarding the product or service; an index value evaluating the call quality of the operator; and a net promoter score indicating the degree to which the customer recommends the product or service.
The information processing system according to claim 1 . further comprising an output unit that outputs the evaluation values corresponding to the text data, the operator emotion value, and the customer emotion value.
The information processing system according to claim 2 . The learning model is a deep learning model having an attention mechanism,
the prediction unit predicts, based on an output value of an intermediate layer of the learning model, a word in the text data that contributed to the output of the evaluation value or the emotion value;
the output unit outputs the words in the text data that contributed to the output of the evaluation value or the emotion value in a display format different from other displays.
The information processing system according to claim 3 . a prediction unit that predicts the evaluation value by inputting text data for each speaker that represents the content of an utterance by the speaker, a plurality of emotion values that correspond to the text data and represent the emotions of the speaker, and an evaluation value for a conversation that corresponds to the text data ; and
An information processing device comprising: the emotion values include an operator emotion value representing an emotion extracted from the content of the operator's utterance, and a customer emotion value representing an emotion extracted from the content of the customer's utterance,
the prediction unit predicts the evaluation value by inputting the text data, the operator emotion value, and the customer emotion value into the learning model generated based on correspondence relationships among the text data, the operator emotion value, the customer emotion value, and the evaluation value;
The evaluation value includes at least one of the following: the customer satisfaction level of the customer with respect to the response of the operator; an index value evaluating the response skills of the operator; an index value evaluating the knowledge of the operator regarding the product or service; an index value evaluating the call quality of the operator; and a net promoter score indicating the degree to which the customer recommends the product or service.
The information processing device according to claim 5 . further comprising an output unit that outputs the evaluation values corresponding to the text data, the operator emotion value, and the customer emotion value.
The information processing device according to claim 6 . The learning model is a deep learning model having an attention mechanism,
the prediction unit predicts, based on an output value of an intermediate layer of the learning model, a word in the text data that contributed to the output of the evaluation value or the emotion value;
the output unit outputs the words in the text data that contributed to the output of the evaluation value or the emotion value in a display format different from other displays.
The information processing device according to claim 7 . An information processing method to be executed by a computer of an information processing device,
a prediction step of predicting the evaluation value by inputting text data for each speaker that represents the content of an utterance by the speaker, a plurality of emotion values that correspond to the text data and represent the emotions of the speaker, and an evaluation value for the conversation that corresponds to the text data;
An information processing method comprising: The computer of the information processing device
a prediction step of predicting the evaluation value by inputting text data for each speaker that represents the content of an utterance by the speaker, a plurality of emotion values that correspond to the text data and represent the emotions of the speaker, and an evaluation value for the conversation that corresponds to the text data;
A program to execute.