JP7209663B2 - Response evaluation device, response evaluation method, and computer program - Google Patents

Description

The present invention relates to a response evaluation device, a response evaluation method, and a computer program.

Conventionally, Patent Literature 1 describes a topic model learning technique for clarifying a shared topic structure between speakers in a dialogue. The topic model learning technology described in Patent Literature 1 clarifies the topic structure such as "In the dialogue between Mr. E and Mr. F, the topic B is dominant." A topic model capable of capturing information such as "The words in the B-th topic are C and D" is realized.

JP 2015-45915 A

However, with the above-described conventional technology, it is not possible to determine whether or not the extracted topic is useful for achieving the purpose. Data could not be generated.

SUMMARY OF THE INVENTION The present invention has been made in consideration of such circumstances, and its object is to generate useful and accurate dialogue evaluation data for achieving the purpose of past reception dialogues. .

(1) One aspect of the present invention is a goal success/failure prediction unit that calculates a goal success/failure prediction score of each sentence included in the dialogue text based on the result of success/failure of the purpose of the response with respect to the past dialogue text. a decision-making judgment unit for judging a decision-making timing for the other party of the reception to make a decision related to the success or failure of the purpose of the reception, based on the goal success/failure prediction score; and an evaluation unit that generates evaluation data of the response using only sentences up to the decision-making timing.
(2) One aspect of the present invention is a model in which the goal success/failure prediction unit outputs a goal success/failure prediction score representing the probability of success or failure of the goal of the response from the input text, wherein the dialogue of the response for a specific purpose The response evaluation device according to (1) above, comprising a machine learning model machine-learned using a set of text learning data and a target success/failure result.
(3) In one aspect of the present invention, the decision-making judgment unit uses the target success/failure prediction score calculated by the target success/failure prediction unit as the target success/failure prediction score of each sentence along the time series of responses in the dialogue text. (1) or ( 2) is one of the response evaluation devices.

(4) In one aspect of the present invention, the response evaluation device calculates a goal success/failure prediction score of each sentence included in the dialogue text based on the success/failure result of the purpose of the response for the dialogue text of past responses. a goal success/failure prediction step, and a decision-making judgment in which the response evaluation device determines a decision-making timing at which the reception party makes a decision related to the success or failure of the purpose of the response, based on the purpose success/failure prediction score. and an evaluation step in which the response evaluation device generates response evaluation data using only sentences up to the decision-making timing among sentences included in the dialogue text. be.

(5) According to one aspect of the present invention, a computer calculates a purpose success/failure prediction score of each sentence included in the dialogue text based on the success/failure result of the purpose of the correspondence with respect to the past dialogue text. a prediction step, a decision-making determination step of determining a decision-making timing for the other party of the reception to make a decision related to success or failure of the purpose of the reception, based on the goal success/failure prediction score; a computer program for executing an evaluation step of generating evaluation data of a response using only sentences up to the decision-making timing out of the sentences that are written.

ADVANTAGE OF THE INVENTION According to this invention, the effect of being able to produce|generate useful and exact evaluation data of dialogue for achievement of the objective of the response in the past dialogue of reception is acquired.

It is a block diagram which shows the structural example of the response evaluation apparatus which concerns on one Embodiment. It is a figure which shows the structural example of the dialogue text storage part which concerns on one Embodiment. It is a figure which shows the structural example of the objective success/failure result storage part which concerns on one Embodiment. It is a figure which shows the structural example of the dialogue text data with purpose success or failure which concerns on one Embodiment. FIG. 10 is a diagram showing a configuration example of dialog text data with goal success/failure prediction score according to one embodiment; 6 is a flowchart illustrating an example of decision-making judgment processing according to one embodiment; It is a figure which shows the example of the dialogue text data with the target success-failure prediction score, and the dialogue text data before decision-making which concerns on one Embodiment. It is a figure which shows the structural example of the evaluation data which concerns on one Embodiment. It is a flowchart which shows an example of the procedure of the response evaluation method which concerns on one Embodiment.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 is a block diagram showing a configuration example of a response evaluation device according to one embodiment. In FIG. 1 , the response evaluation device 1 includes a purpose success/failure result combining unit 11 , a purpose success/failure prediction unit 12 , a decision making determination unit 13 , and a topic extraction unit 14 . In this embodiment, the topic extraction unit 14 is an example of an evaluation unit.

Each function of the response evaluation device 1 is realized by the response evaluation device 1 having computer hardware such as a CPU (Central Processing Unit) and memory, and the CPU executing a computer program stored in the memory. be done.
The response evaluation device 1 may be configured using a general-purpose computer device, or may be configured as a dedicated hardware device. For example, the response evaluation device 1 may be configured using a server computer connected to a communication network such as the Internet. Moreover, each function of the response evaluation apparatus 1 may be implemented by cloud computing.

The response evaluation device 1 accesses the database 30 . The database 30 may be directly connected to the response evaluation apparatus 1, or may be connected via a communication line. For example, database 30 may be configured using a server computer connected to a communication network such as the Internet. Also, the database 30 may be realized by cloud computing.

The database 30 includes a dialogue text storage section 31 , a goal success/failure result storage section 32 , a pre-decision text storage section 33 , and a topic storage section 34 .

The dialogue text storage unit 31 accumulates dialogue texts of past responses. For example, the dialog text storage unit 31 stores a text (dialogue text) in which a dialog between a customer and a customer at a company's service center is converted into text. For example, the dialog text storage unit 31 stores a dialog text in which a dialog between a customer and a customer at a merchandise store is converted into text.

FIG. 2 is a diagram showing a configuration example of a dialogue text storage unit according to this embodiment. As shown in FIG. 2, the dialogue text storage unit 31 stores dialogue identifiers (dialogue IDs) and dialogues (dialogue texts) in association with each other.

Here, multiple dialogue texts that are evaluated together have the same answering purpose. For this reason, as an example of the present embodiment, one dialogue text storage unit 31 stores only dialogue texts for the same purpose, and all dialogue texts stored in one dialogue text storage unit 31 are stored. be subject to evaluation together. Therefore, dialogue texts for different purposes are stored in separate dialogue text storage units 31, respectively. For example, in "Case 1: Dialogue text between a customer and a customer at a service center of a company," the purpose of the response is to satisfy the customer, and "Case 2: A customer and a customer at a store selling goods It is assumed that the purpose of the response of "dialogue text of response" is for the customer to purchase the product. In this case, since cases 1 and 2 have different purposes of correspondence, separate dialog text storage units 31 are provided for cases 1 and 2 so that case 1 and case 2 are evaluated separately.

As a modification of this embodiment, case 1 and case 2 are stored in one dialogue text storage unit 31 so as to be identifiable, and only case 1 is extracted when case 1 is to be evaluated. If case 2 is to be evaluated, a target dialogue extraction unit for extracting only case 2 may be provided.

The goal success/failure result storage unit 32 accumulates the result of success or failure of the purpose of the response for each dialog text stored in the dialog text storage unit 31 . For example, the customer's satisfaction with the service can be cited as the purpose of the service between the customer and the customer at the company's service center. For example, one of the purposes of correspondence between a customer and a customer at a merchandise store is for the customer to purchase merchandise. FIG. 3 is a diagram showing a configuration example of a goal success/failure result storage unit according to the present embodiment. As shown in FIG. 3, the purpose success/failure result storage unit 32 associates and stores the dialogue ID and the success or failure of achieving the purpose of the dialogue text corresponding to the dialogue ID "1: success" or "0: failure".

The purpose success/failure result linking unit 11 associates the dialogue text in the dialogue text storage unit 31 with the purpose achievement success/failure “1: success” or “0: failure” in the purpose success/failure result storage unit 32 using the dialogue ID as a key. . As a result, as illustrated in FIG. 4, dialogue text data with purpose success/failure is generated in which the dialogue text with the same dialogue ID is associated with the success/failure of purpose achievement “1: success” or “0: failure”.

The purpose success/failure prediction unit 12 calculates the purpose success/failure prediction score of each sentence included in the dialogue text based on the success/failure result of the purpose of the past dialogue text. The purpose success/failure prediction unit 12 calculates a purpose success/failure prediction score of each sentence included in the dialogue text of the dialogue text data with purpose success/failure generated by the purpose success/failure result combining unit 11 .

[Example of goal success/failure prediction score calculation method]
The target success/failure prediction unit 12 includes a model (target success/failure prediction score calculation model) that outputs a target success/failure prediction score representing the probability of success or failure of the target of the response from the input text. The target success/failure prediction score calculation model is a machine that has been machine-learned using a set of dialogue text learning data for a specific target response and the target success/failure result (target achievement success or failure “1: success” or “0: failure”) Learning model.

The interactive text used as input to the model (input interactive text) is digitized. In this embodiment, the input dialogue text is vectorized. For vectorization of the input dialogue text, for example, a document vectorization technology called "Doc2Vec" can be applied.

In the training phase, the machine learning model is trained to predict the target success/failure result from the sentence vectors of the input dialogue text using pairs of input dialogue text and objective success/failure result vectorized on the dialogue text training data. done. A trained machine learning model is used as a target success/failure prediction score calculation model.

In the goal success/failure prediction score calculation stage, when a sentence vector (input sentence) of the input dialogue text vectorized for the dialogue text data with the goal success/failure is input to the goal success/failure prediction score calculation model, the goal success/failure prediction score for the input sentence is calculated. It is output from the target success/failure prediction score calculation model. In this embodiment, the target success/failure prediction score is a real number from "0.0" to "1.0". The larger the purpose success/failure prediction score, the higher the probability that the purpose of the reception will succeed.

The purpose success/failure prediction unit 12 uses a purpose success/failure prediction score calculation model to obtain a purpose success/failure prediction score for each sentence (including sentences preceding the sentence) included in the dialogue text of the dialogue text data with purpose success/failure.

The above is the description of an example of the target success/failure prediction score calculation method according to the present embodiment.
The purpose success/failure prediction unit 12 gives each purpose success/failure prediction score to each sentence of the dialogue text data with purpose success/failure. As a result, dialogue text data with goal success/failure prediction score is generated, in which each goal success/failure prediction score is assigned to each sentence of the dialogue text data with goal success/failure, as illustrated in FIG.

The decision-making judgment unit 13 determines the intention of the other party to make a decision related to the success or failure of the purpose of the response based on the purpose success/failure prediction score given to each sentence of the dialogue text data with purpose success/failure prediction score. Determine decision timing. The intention related to the success or failure of the purpose of service is, for example, the purpose of service between a service person and a customer (reception partner) at a company's service center, which is ``the customer is satisfied with the service.'' are satisfied with the customer service and end the service (willingness related to the success of the purpose)” and “the mindset of the customer ending the service without being satisfied with the service (intention related to the failure of the purpose)”. In addition, the intention related to the success or failure of the purpose of the service is, for example, for the purpose of the service between the service person and the customer (the other party of the service) at the product sales store, "the customer purchases the product" The customer's willingness to purchase the product (the intention related to the success of the purpose)," and the "customer's intention not to purchase the product (the intention related to the failure of the purpose)."

Here, decision-making judgment processing according to the present embodiment will be described with reference to FIG. FIG. 6 is a flowchart showing an example of decision-making judgment processing according to this embodiment.

(Step S<b>101 ) The decision making determination unit 13 acquires dialogue text data with goal success/failure prediction score from the goal success/failure prediction unit 12 .

(Step S102) The decision making determination unit 13 acquires a set of a sentence and a target success/failure prediction score for each sentence of the dialogue text with the same dialogue ID from the dialogue text data with the purpose success/failure prediction score.

(Step S103) The decision-making judgment unit 13 assigns a sentence identifier (sentence ID (here, 1, 2, 3, . . . )).

(Step S104) The decision-making judgment unit 13 acquires the target success/failure prediction score in chronological order of the response (ascending order of the sentence ID).

(Step S<b>105 ) The decision making determination unit 13 determines whether the obtained one purpose success/failure prediction score is equal to or greater than the success score threshold or equal to or less than the failure score threshold. A success score threshold and a failure score threshold are preset. "success score threshold≧failure score threshold". As a result of the determination in step S105, if the score is equal to or greater than the success score threshold or equal to or less than the failure score threshold (step S105, YES), the process proceeds to step S106; otherwise (step S105, NO), the process returns to step S104. .

(Step S106) The decision making determination unit 13 determines whether or not the result of determination in step S105, ie, "the target success/failure prediction score is greater than or equal to the success score threshold" continues for the number of consecutive times threshold or more. Further, the decision-making judgment unit 13 judges whether or not the judgment result of step S105, "the target success/failure prediction score is equal to or less than the failure score threshold", continues for the consecutive number of times equal to or more than the threshold. As a result of the determination in step S106, if the number of consecutive times exceeds the threshold (step S106, YES), the process proceeds to step S107, otherwise (step S106, NO), the process returns to step S104.

(Step S107) The decision-making judging unit 13 first determines that the threshold is exceeded (success score threshold or more or failure score threshold or less) among the same judgment results in step S105 that are consecutive for the number of consecutive times or more. The data of the sentence ID after the ID (dialogue text, goal achievement success/failure, goal success/failure prediction score) are deleted from the dialogue text data with goal success/failure prediction score to generate pre-decision-making dialogue text data.

(Step S108) The decision making determination unit 13 determines whether or not all dialogue texts in the dialogue text data with target success/failure prediction score have been processed. As a result of this determination, if all the dialogue texts have been processed, the process of FIG. 6 is terminated, and if unprocessed dialogue texts remain, the process returns to step S104.

FIG. 7 is a diagram showing an example of dialogue text data with goal success/failure prediction score and pre-decision dialogue text data according to the present embodiment. In the example of FIG. 7, "success score threshold=0.7". In the dialog text data with target success/failure prediction score shown in FIG. and the number of consecutive times of "the target success/failure prediction score is equal to or greater than the success score threshold" is equal to or greater than the consecutive count threshold "3". For this reason, the decision-making determination unit 13 selects sentence IDs "6" to "10" after sentence ID "5", which is first determined to be equal to or greater than the success score threshold, among sentence IDs "5" to "10". ” (dialogue text, goal achievement success/failure, goal success/failure prediction score) are deleted from the dialogue text data with goal success/failure prediction score of FIG. to generate

As shown in FIG. 7(2), the pre-decision-making dialogue text data includes data from sentence ID "1" to sentence ID "5" that is first determined to be equal to or greater than the success score threshold (dialogue text, goal achievement success/failure, goal success/failure prediction score). The pre-decision-making dialogue text data in FIG. 7(2) is from the first sentence (sentence ID "1") of the response to the decision-making timing at which the other party makes a decision related to the success of the purpose of the response. It has up to the corresponding sentence (sentence ID "5") and does not have sentences (sentence IDs "6" to "10") after it (after the decision-making timing).

Therefore, although the pre-decision-making dialogue text data in FIG. 7(2) includes the dialogue up to the decision-making timing at which the other party makes a decision related to the success of the purpose of the reception, the dialogue after the decision-making is included. Not included. Here, it is considered that the post-decision dialogue often includes dialogue unrelated to the purpose of the response.

For example, when dealing with sales of a main product, after the customer decides to purchase the main product, it is often considered that the customer is further recommended to purchase products related to the main product. In this case, it is considered that the dialogue about the related product, which takes place after the dialogue about the main product is finished, is an irrelevant dialogue that is not very useful for the sales of the main product. For this reason, from the series of customer dialogues, the remaining dialogue after deleting the dialogue on related products that takes place after the dialogue on the main product ends, that is, only the dialogue up to the decision-making timing when the customer decides to purchase the main product. is to extract useful and accurate dialogues for achieving the purpose of the reception of selling main products. The response evaluation data generated based on the extracted dialogue is useful and accurate dialogue evaluation data for achieving the purpose of the response.

The example of FIG. 7 is an example in which the goal success/failure prediction score is equal to or higher than the success score threshold, but pre-decision dialogue text data is similarly generated when the goal success/failure prediction score is equal to or lower than the failure score threshold. be done. The consecutive count threshold (failure determination consecutive count threshold) when the target success/failure prediction score is less than or equal to the failure score threshold is the same as the consecutive count threshold (success determination consecutive count threshold) when the target success/failure prediction score is greater than or equal to the success score threshold. or may be different.

Returning to FIG.
The decision-making judgment unit 13 stores the generated pre-decision-making dialogue text data in the pre-decision-making text storage unit 33 .

The topic extraction unit 14 uses the pre-decision-making dialogue text data stored in the pre-decision-making text storage unit 33 to generate response evaluation data. In other words, the topic extracting unit 14 uses only the sentences up to the decision-making timing among the sentences included in the dialogue texts of the past responses to generate the response evaluation data.

As an example of this embodiment, the response evaluation data is data indicating a topic effective in achieving the purpose of the response. The topic extracting unit 14 divides the dialogue text of the pre-decision-making dialogue text data into morphemes, and generates a morpheme group by grouping morphemes that are close in meaning to each of the divided morphemes. Further, the topic extracting unit 14 calculates the degree of usefulness for achieving the purpose of the response based on the success or failure result of the purpose of the response for each of the divided morphemes.

The usefulness is calculated by the following formula.
Usefulness = (A/(A+B))/(C/(C+D))
However, A is the total number of dialogue texts in which the goal achievement success/failure is “1: success” among the dialogue texts in which the usefulness calculation target morpheme appears. B is the total number of dialogue texts in which the morpheme for which the usefulness is to be calculated appears, and whose goal achievement success/failure is “0: failure”. C is the total number of dialogue texts in which the goal achievement success/failure is “1: success” among the dialogue texts in which the usefulness calculation target morpheme does not appear. D is the total number of dialogue texts in which the morpheme for which the usefulness is to be calculated does not appear and which is "0: failure" in achieving the purpose.

The topic extraction unit 14 calculates a usefulness representative value of a morpheme group based on the usefulness of each morpheme contained in the morpheme group, and extracts a morpheme group having a usefulness representative value equal to or higher than a predetermined threshold as a topic. do. As usefulness representative values, for example, average values, median values, and best values can be used. The topic extraction unit 14 generates evaluation data indicating the extracted morpheme group (topic). For example, the evaluation data may be a list of all morphemes and their usefulness from the extracted morpheme group, or may be a list of morphemes whose usefulness is above a certain level. The topic extraction unit 14 stores the generated evaluation data in the topic storage unit 34 . This evaluation data is data indicating a topic of conversation that is useful and accurate for achieving the purpose of the response.

FIG. 8 is a diagram showing a configuration example of evaluation data according to this embodiment. In FIG. 8, the topic number is an identifier attached to the morpheme group. In the configuration example of the evaluation data shown in FIG. 8, a list includes a set of morphemes, topic numbers, and degrees of usefulness.

According to this embodiment, by extracting a topic effective for achieving the purpose of the response using only sentences up to the decision making of the customer who is the other party of the response in one past conversation, By excluding topics that appear after the goal is achieved, it is possible to extract topics that are influencing the customer's decision making. Based on this extracted topic, create a model talk script that is effective in achieving goals in customer service, and people with low customer service skills or little experience in customer service respond based on the model talk script. Therefore, it becomes possible to contribute to the improvement of the target success rate of the response.

In the above-described embodiment, the topic extraction unit 14 is used as an example of the evaluation unit, but the evaluation unit is not limited to this. The evaluation unit may use the pre-decision dialogue text to generate the evaluation data of the response.

Next, a response evaluation method according to this embodiment will be described with reference to FIG. FIG. 9 is a flow chart showing an example of the procedure of the response evaluation method according to this embodiment.

(Step S1) The response evaluation device 1 compares the dialogue text in the dialogue text storage unit 31 and the objective success/failure "1: success" or "0: failure" in the objective success/failure result storage unit 32 using the dialogue ID as a key. associated with. As a result, as illustrated in FIG. 4, dialogue text data with purpose success/failure is generated in which the dialogue text with the same dialogue ID is associated with the success/failure of purpose achievement “1: success” or “0: failure”.

(Step S2) The response evaluation device 1 calculates a goal success/failure prediction score of each sentence included in the dialogue text of the dialogue text data with purpose success/failure. The response evaluation device 1 assigns each purpose success/failure prediction score to each sentence of the dialogue text data with purpose success/failure. As a result, dialogue text data with goal success/failure prediction score is generated, in which each goal success/failure prediction score is assigned to each sentence of the dialogue text data with goal success/failure, as illustrated in FIG.

(Step S3) The response evaluation device 1 evaluates the intention of the other party to be related to the success or failure of the purpose of the response based on the purpose success/failure prediction score assigned to each sentence of the dialog text data with purpose success/failure prediction score. Determine decision-making timing to decide. The decision-making judgment unit 13 generates pre-decision-making dialogue text data by deleting sentences after the decision-making timing as a result of the judgment from the dialogue text data with purpose prediction score. This pre-decision-making dialogue text data includes only sentences up to the decision-making timing as a result of the judgment, among the sentences included in the dialogue text data with goal success/failure prediction score.

(Step S4) The response evaluation device 1 uses the pre-decision dialogue text data to generate response evaluation data.

According to the above-described embodiment, from the past conversations in response, the dialogue up to the decision-making timing at which the other party of the response makes a decision related to the success or failure of the purpose of the response is extracted, and the extracted dialogue only can be used to generate response evaluation data. As a result, it is possible to generate useful and accurate dialogue evaluation data for achieving the purpose of the past dialogue.

Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like are included within the scope of the present invention.

Alternatively, a computer program for realizing the functions of the devices described above may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read and executed by the computer system. Note that the “computer system” referred to here may include hardware such as an OS and peripheral devices.
In addition, "computer-readable recording medium" includes writable nonvolatile memories such as flexible discs, magneto-optical discs, ROMs and flash memories, portable media such as DVDs (Digital Versatile Discs), and computer system built-in media. A storage device such as a hard disk that

Furthermore, "computer-readable recording medium" means a volatile memory (e.g., DRAM (Dynamic Random Access Memory)), which holds the program for a certain period of time, is also included.
Further, the above program may be transmitted from a computer system storing this program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in a transmission medium. Here, the "transmission medium" for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.
Further, the program may be for realizing part of the functions described above. Further, it may be a so-called difference file (difference program) that can realize the above-described functions in combination with a program already recorded in the computer system.

1 Response evaluation device 11 Goal success/failure result combination unit 12 Goal success/failure prediction unit 13 Decision decision determination unit 14 Topic extraction unit 30 Database 31 Dialogue text storage unit 32 Goal success/failure result Storage unit, 33: Pre-decision text storage unit, 34: Topic storage unit

Claims (5)

a goal success/failure prediction unit that calculates a goal success/failure prediction score of each sentence included in the dialogue text based on the result of success/failure of the purpose of the response with respect to the past dialogue text;
a decision-making determination unit that determines a decision-making timing for the other party of the reception to make a decision related to success or failure of the purpose of the reception, based on the goal success/failure prediction score;
an evaluation unit that uses only sentences up to the decision-making timing among sentences included in the dialogue text to generate evaluation data of the response;
Response evaluation device. The goal success/failure prediction unit is a model that outputs a goal success/failure prediction score representing the probability of success or failure of the goal of the response from the input text, and is a combination of dialogue text learning data of a specific goal response and the goal success/failure result. with a machine learning model machine-learned using
The response evaluation device according to claim 1. The decision-making determination unit determines whether the target success/failure prediction score calculated by the target success/failure prediction unit is a predetermined threshold value for the target success/failure prediction score of each sentence along the time series of the response in the dialogue text, which is consecutive for a predetermined number of times or more. When the condition is satisfied, the first sentence among consecutive sentences whose target success/failure prediction score satisfies the threshold condition is determined at the decision-making timing;
The response evaluation device according to claim 1 or 2. a target success/failure prediction step in which the response evaluation device calculates a target success/failure prediction score of each sentence included in the dialogue text based on the result of success/failure of the purpose of the response with respect to the dialogue text of past responses;
a decision-making determination step in which the response evaluation device determines a decision-making timing for the response partner to make a decision related to the success or failure of the purpose of the response, based on the goal success/failure prediction score;
an evaluation step in which the response evaluation device generates response evaluation data using only sentences up to the decision-making timing among sentences included in the dialogue text;
Response evaluation methods, including to the computer,
a purpose success/failure prediction step of calculating a purpose success/failure prediction score of each sentence included in the dialogue text based on the result of success/failure of the purpose of the response with respect to the past dialogue text;
a decision-making judgment step of judging a decision-making timing for the other party of the reception to make a decision related to the success or failure of the purpose of the reception, based on the purpose success/failure prediction score;
an evaluation step of generating response evaluation data using only sentences up to the decision-making timing among sentences included in the dialogue text;
A computer program for executing

Images