JP6760138B2 - Dialogue corpus creation program, dialogue corpus creation method, and information processing device - Google Patents

Description

The present invention relates to a dialogue corpus creation program, a dialogue corpus creation method, and an information processing apparatus.

In recent years, a dialogue system in which a machine responds to a dialogue with a person with a natural response sentence has been developed and commercialized. The sentence pattern of the corresponding response sentence is obtained from the sentence pattern of the response sentence accumulated in advance for the human input sentence, and the obtained response sentence pattern is obtained by using the characteristic words of the words extracted from the input sentence and the words having a high degree of relevance. A technique for generating a response sentence to an input sentence from is known.

In addition, in order for the machine to output more natural sentences in dialogue with humans, it accepts dialogue sentences of two or more users, processes at least one conversational sentence in natural language, and acquires and accumulates conversation connection data. A technique for generating a dialogue corpus by doing so has been proposed.

Japanese Unexamined Patent Publication No. 2015-153261 Japanese Unexamined Patent Publication No. 2008-299754 JP-A-2007-102104 Japanese Unexamined Patent Publication No. 2001-166785

The above-mentioned technique has a problem that when two people work, it is inefficient because a waiting time occurs while one of them is working. If two parties try to have a dialogue efficiently, it is necessary to manage the work of finding a pair of two parties and the time adjustment of the two parties, which is a burden on the administrator.

Therefore, one aspect aims to efficiently create a dialogue corpus.

According to one aspect, the worker IDs of workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties, and the worker IDs of a plurality of workers in the order in which they can participate in the dialogue. The work of the worker ID taken out from the stored storage unit and managed by the storage unit by referring to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues. According to the worker allocation procedure based on the past participation status of the person, the computer is made to execute the allocation process for selecting the dialogue for assigning the worker who creates the speech data one by one, and the dialogue is completed by the plurality of workers. Provided is a dialogue corpus creation program that causes the computer to execute a process of obtaining the dialogue data.

Further, as a means for solving the above problems, a dialogue corpus creation method and an information processing device can be used.

You can efficiently create a dialogue corpus.

It is a figure for demonstrating the creation of the dialogue corpus by the dialogue of a pair. It is a figure which shows the example of corpus creation which breaks down a task and uses crowdsourcing. It is a figure for demonstrating the common consciousness for dialogue and the method of maintaining the quality of dialogue. It is a figure which shows the dialogue screen example. It is a figure for demonstrating the example of the utterance tree structure based on the presence or absence of an inappropriate check. It is a figure for demonstrating the utterance scoring in the case of the multiplicity "1". It is a figure for demonstrating the utterance scoring in the case of the multiplicity "3". It is a figure for demonstrating the worker scoring. It is a figure for demonstrating dialogue scoring. It is a figure for demonstrating the outline of the worker allocation procedure. It is a figure which shows the example of the worker allocation processing in the case of only one worker ID with the highest selection probability. It is a figure which shows the 1st example of the worker allocation processing when there are a plurality of the highest selection probabilities. It is a figure which shows the 2nd example of the worker allocation processing when there are a plurality of the highest selection probabilities. It is a figure which shows the 3rd example of the worker allocation processing when there are a plurality of the highest selection probabilities. It is a figure which shows the system configuration example in this Example. It is a figure for demonstrating the outline of the whole processing in the system of this Example. It is a figure which shows the functional configuration example of a server device. It is a figure which shows the data structure example of a worker list. It is a figure which shows the data structure example of an interactive task table. It is a figure which shows the data structure example of the utterance table. It is a figure which shows the data structure example of the utterance data DB. It is a figure which shows the data structure example of the worker score table. It is a figure which shows the data structure example of the dialogue data with a score. It is a flowchart for demonstrating the initialization process. It is a flowchart for demonstrating the worker registration process and the worker allocation process. It is a flowchart for demonstrating the interactive task selection process in step S222 of FIG. It is a flowchart for demonstrating the dialogue task generation process in step S237 of FIG. It is a flowchart for demonstrating the utterance making process. It is a flowchart for demonstrating the utterance information update process in step S252 of FIG. 28. It is a flowchart for demonstrating the interactive data fetch process. It is a flowchart for demonstrating the worker score calculation process in step S271 of FIG. It is a flowchart for demonstrating the dialogue score calculation process in step S272 of FIG.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. First, a case where a dialogue corpus is created by a pair of two parties will be described. FIG. 1 is a diagram for explaining the creation of a dialogue corpus by a pair dialogue.

In FIG. 1, subjects α and β to be a dialogue pair are prepared in advance, the pair is asked to have a dialogue, and the content of the dialogue is recorded to create a dialogue corpus 1a. The roles of subjects α and β will be explained in advance. For example, a clerk of a travel agency and a customer who reserves a means of transportation, a hotel, or the like.

FIG. 1 shows an example of a dialogue in which the subject α has a conversation as a clerk of a travel agency and the subject β has a conversation as a customer. Subject α asks in the first utterance, "Where are you going?", And subject β replies in the second utterance, "I'm going to Chicago."

Then, subject α asks a further question in the third utterance "How many nights are you planning?", Subject β replies in the fourth utterance "I'm thinking about 3 nights", and so on. The conversation with β is continued. When the subject α, who acts as a clerk of the travel agency, obtains sufficient information from the subject β to plan the trip, the dialogue ends.

In this way, by performing machine learning using the dialogue corpus 1a collected from many pairs of subjects, it is possible to appropriately proceed with the dialogue when the machine deals with a person. Machine learning using the dialogue corpus 1a eliminates the need to manually describe the rules of dialogue.

In order to proceed with the dialogue accurately, it is necessary to create a large-scale dialogue corpus 1a. The large-scale dialogue corpus 1a enables the development of a dialogue system using machine learning with high accuracy.

However, in order to create the dialogue corpus 1a in this way, in general, subjects α and β to be a pair of dialogues are prepared, and dialogues are provided at a predetermined time by providing a place or via a communication means. And record the content of the ongoing dialogue. In this method, the subject works in collaboration with the other, and while one subject is working, the other person in the pair has time to wait for the work.

As a simple solution to such a problem, instead of having the dialogue recorded in the dialogue corpus 1a performed from the start to the end by two defined parties, the dialogue is divided into a plurality of individual tasks and crowdsourced. It is conceivable to carry out independently with the participation of a large number of people by using such as.

Dividing the dialogue into a plurality of individual tasks means that one person does not perform one role, but a person who plays a role is selected for each conversation and the conversation is continued. As an example, in the dialogue between the subject α and the subject β in FIG. 1, the person who plays the role is changed each time either the subject α or β speaks. In this case, making one utterance is regarded as one individual task.

Crowdsourcing is to decompose the target main task into independent small unit tasks (hereinafter referred to as small tasks) and execute the small tasks by an unspecified number of workers via a network. ..

Since the distribution and execution of small tasks are mainly performed using a web browser, a smartphone, or the like, as a merit of the worker, the worker can carry out the work at any place at any time. Individual subtasks are small and easy tasks and often do not require expertise.

In addition, by having an unspecified number of workers perform small tasks, it is possible to complete a large main task quickly at low cost. From this, as a simple example, it is conceivable that the main task is a dialogue consisting of a plurality of utterances, and the utterance unit is a small task.

FIG. 2 is a diagram showing an example of creating a corpus by decomposing tasks and using crowdsourcing. FIG. 2 shows an example of progress of dialogue by a plurality of workers 9 including workers A, B, C, and D.

The conversation is started when Worker A asks the first utterance, "Where are you going?". Worker B responds to this question with the second utterance, "I'm going to Chicago."

In response to the response of the worker B, in the example of FIG. 1, the worker A of the first utterance should ask the next question, but in the example of FIG. 2, a worker C different from the worker A is asked. Ask more questions in the third utterance, "How many nights are you planning?"

The worker 9 who answers the question of the worker C is not the worker B but the worker D. Worker D replies with the fourth utterance, "I'm thinking about three nights." In this way, the two persons having a dialogue are not fixed, a worker 9 is selected for each utterance input, and a series of conversations are individually and independently carried out by a large number of workers 9.

Before explaining this embodiment, the terms are defined below.

An utterance is a statement described by a single worker.

A dialogue is a series of utterances consisting of multiple utterances.

A worker is an individual person who creates an utterance and is sometimes referred to as the speaker.

In the corpus creation method using crowdsourcing as described in FIG. 2, if the dialogue flow is decomposed into individual subtasks as described above, it may not be possible to generate a consistent utterance example. Therefore, in this embodiment,
(I) Consistency check (II) Scoring of utterances, workers, and dialogues (III) Efficient allocation of workers.

(I) Consistency check First, in order to maintain the common consciousness of workers 9 and the quality of dialogue by presenting the context.
-Present the context from the beginning to the worker 9-The worker 9 can share the same context as the previous speaker (worker 9) by reading and understanding the context. Maintains the same quality as utterances created by the same pair.

Then, to ensure the integrity of the context
-Every time one utterance progresses, the worker 9 checks whether or not the content of the dialogue so far is appropriate. This can be done at the same time as understanding the context. In other words, after reading everything to understand the context, if you cannot understand the content, it is judged to be inappropriate.
-The problem with creating a dialogue corpus by an independent worker 9 is that there is a risk that a consistent utterance sequence cannot be created, but this check ensures the consistency of the dialogue.
-If the worker 9 determines that the previous dialogue is inappropriate, the worker 9 creates an utterance that will be made there.

(II) Scoring of utterances, workers, and dialogues First, utterances and scoring of the reliability of workers 9 will be described.
-The level of worker 9 generally varies, and as a result, inappropriate utterances may be created.
-For each utterance, the ratio at which the utterance is judged to be correct at the multiplicity given as a parameter is defined as the utterance score.
・ After branching, continue utterance with a high utterance score. It may be described as extending the branch of the utterance after branching.
-The average value of the utterance scores of all the utterances created by the worker 9 is set as the worker score of the worker 9.

Next, dialogue scoring using worker scores will be described.
-Calculate the score for each dialogue.
・ If there is a branch, select the most specified path and extract one path.
-The geometric mean of the worker scores of the workers 9 appearing in the path is calculated, and this is used as the score of the dialogue, which is hereinafter referred to as the dialogue score.

(III) Efficient allocation of workers When the utterance line in a dialogue becomes long, it takes time to understand the context and check its contents. In order to omit this check as much as possible and make it efficient, the same task is preferentially assigned to the same worker 9.
-For this purpose, assign tasks using probabilities based on the task history of the dialogue.
The task here corresponds to the dialogue to be created, and may be referred to as a dialogue task in the following description. Dialogues to be created include those that are currently in progress and those that have not yet started.

The above-mentioned (I) to (III) will be described with reference to examples. First, (I) consistency check will be described with reference to FIGS. 3 to 5.

FIG. 3 is a diagram for explaining a method of standardizing awareness of dialogue and maintaining dialogue quality. In FIG. 3, a plurality of different workers 9, including workers A, B, C, and D, are asked to imagine a situation of booking transportation, accommodation, etc. at a travel agency, and have a series of conversations. An example is shown when you receive it.

In this embodiment
-The pair of workers 9 is not managed, and each worker 9 is accepted independently.
-Each worker 9 is requested to present the context of the dialogue up to that point (the sequence of utterances generated) and to create the next utterance.

The plurality of workers 9 connect to the service according to the present embodiment from a terminal such as a mobile phone via a network. The utterance at least immediately before is presented to the worker 9. Further explanations of the scenes in which the dialogue proceeds may be presented. On the other hand, the scene of the dialogue may be determined by the worker 9 from the utterances up to the last minute.

In this example, a plurality of workers 9 are asked to have a dialogue between a clerk and a customer regarding a reservation at a travel agency. After grasping the situation, each worker 9 inputs the most probable content for the immediately preceding utterance as the next utterance. Alternatively, if it is determined that the straight line utterance is inappropriate, enter the utterance to be corrected.

Of the plurality of workers 9, the worker A who inputs the conversation first determines that he / she is a clerk and inputs the first utterance. For example, "Where are you going?" Is input as the first utterance.

Worker B, who is presented with the first utterance for the first utterance, determines that he / she is a customer and inputs the second utterance. For example, "I'm going to Chicago" is input as the second utterance.

Worker C, who is presented with the first and second utterances, determines that he is a clerk and inputs the third utterance. For example, "How many nights are you planning?" Is input as the third utterance.

Worker D, who is presented with the first to third utterances, determines that he is a customer and inputs the fourth utterance. For example, "I'm thinking about 3 nights" is entered as the fourth utterance.

By providing the utterances up to the last minute to the worker 9 in this way, even if the worker 9 is changed for each utterance, the worker 9 can grasp the situation and his / her role. Further, in this embodiment, the screen as shown in FIG. 4 is displayed on the terminal of the worker 9, so that the worker 9 checks the consistency of the dialogue up to the last minute.

FIG. 4 is a diagram showing an example of a dialogue screen. The dialogue screen G70 illustrated in FIG. 4 has an utterance history display area 71, an utterance input area 72, an inappropriate check area 73, and an end check area 74. On the dialogue screen G70, the worker C reads the two utterances from the beginning to the immediately preceding of the dialogue displayed in the utterance history display area 71, and grasps the situation of the dialogue.

Worker C was first displayed in the utterance history display area 71. Where are you going?
2. 2. From the utterances I'm going to Chicago, determine if the last (second) utterance is appropriate.

When the worker C determines that the last utterance is appropriate, the worker C creates and inputs an appropriate utterance following the last utterance in the utterance input area 72. On the other hand, when it is determined that it is inappropriate, the worker C checks the inappropriate check area 73. When the inappropriate check area 73 is checked, the worker C is corrected by inputting an utterance considered to be appropriate into the utterance input area 72 instead of the last utterance.

When the inappropriate check area 73 is checked, it is desirable that this correction is indispensable. By having the utterance to be corrected created, the worker C can be made to make a proper judgment of the last utterance in good faith. In this embodiment, since the next worker 9 is made to judge the appropriateness of the last utterance for each utterance in this way, it can be said that the utterances before the last utterance are the utterances judged to be appropriate.

In this example, the worker C creates an utterance in the utterance input area 72 as "How many nights are you planning?" Without checking the inappropriate check area 73. That is, it is the case that the utterance of "I will go to Chicago" is judged to be appropriate and the next utterance following this utterance is created.

When it is determined that the end check area 74 is the end of the dialogue, the worker C checks the end check area 74. The end of the dialogue is when it is determined that the purpose of the dialogue has been achieved. When the purpose of the dialogue is to reserve transportation, hotels, etc. related to accommodation, the dialogue ends when it is judged that information such as the departure place, destination, place of stay, and length of stay can be obtained. In addition to the case where such necessary information is obtained, the dialogue ends when the question can be answered and the other party's satisfaction is obtained.

In this embodiment, when the inappropriate check area 73 described above is checked, as shown in FIG. 5, a plurality of utterances are composed of a tree structure (hereinafter referred to as an utterance tree structure) that branches the progress of the utterance. Express the progress of the dialogue.

FIG. 5 is a diagram for explaining an example of an utterance tree structure based on the presence or absence of an inappropriate check. In FIG. 5 (A), in response to the first (first) utterance "Where are you going?" By worker A, the next worker B is the second utterance "I will go to Chicago." Is created, and the dialogue screen G70 when this dialogue is assigned to the worker C is shown. This is the same as the dialogue screen G70 of FIG.

In this example, the worker C judges that the utterance of the worker B is appropriate for the utterance of the worker A, and the utterance by the worker C "How many nights are you planning?" Without the inappropriate check. Is created.

The progress of this dialogue is represented by the utterance tree structure 8a. In the utterance tree structure 8a, the utterance is briefly indicated by a node of the alphabet representing the worker 9 who created the utterance. The same applies to FIGS. 6 to 9.

The utterance tree structure 8a starts from the first (first) utterance A, extends downward to the second utterance B, and further downwards to the third utterance C. There are no branches in the utterance tree structure 8a. That is, at this point, it indicates that the inappropriate check has not been performed yet.

In FIG. 5 (B), a dialogue in which the next worker D creates the second utterance "anywhere" in response to the first (first) utterance "Where are you going?" By the worker A. The screen G70 is shown.

The worker E determines that the utterance "anywhere" of the immediately preceding worker D is inappropriate, and creates an utterance to be corrected. The utterance tree structure 8b when the utterance E is created with the improper check is shown. In the utterance tree structure 8b, the utterance tree structure 8b starts from the first (first) utterance A and extends downward to the second utterance D, but the utterance E is positioned directly under the utterance A due to an inappropriate check for the utterance D.

In the following description, unless otherwise specified such as the utterance tree structure 8a, 8b, etc., the utterance tree structure 8 may be generically referred to.

Next, scoring of (II) utterance, worker, and dialogue will be described with reference to FIGS. 6 to 9. First, utterance scoring will be described. In this embodiment
-When the utterance branches, the multiplicity for checking is given as a parameter.
-The multiplicity gives an odd number.
-When a branch occurs, let the operator evaluate whether or not the branch occurs up to the number of multiplicities. When the multiplicity is 1, only the utterance after the branch is used when the branch is made. The multiplicity is preferably 3 or more.
-The utterance score is the ratio instructed by the worker 9.

First, utterance scoring will be described with reference to FIGS. 6 and 7. FIG. 6 is a diagram for explaining utterance scoring when the multiplicity is “1”. FIG. 6 (A) shows the utterance score of each utterance of the utterance tree structure 8a of FIG. 5 (A). In the utterance tree structure 8a, since there is no branch, all the utterance scores US of the utterances A, B, and C show "1".

FIG. 6B describes the utterance score US before and after the branch in the utterance tree structure 8c. At the time when the utterance D before the branch is created, the utterance score US of both the utterance A and the utterance D is "1". When branching from utterance A when creating utterance C, that is, when utterance D is determined to be inappropriate, if the multiplicity is 1, the utterance score US of utterance D determined to be inappropriate is "0". Then, the utterance score US "1" is given to the utterance C for correction.

FIG. 7 is a diagram for explaining utterance scoring when the multiplicity is “3”. FIG. 7A shows a state in which utterance D is created in response to utterance A. In this state, the utterance score US of utterance A is "1", and the utterance score US of utterance D is also "1".

In FIG. 7B, the utterance D is determined to be inappropriate, and the utterance C is created for correction. At this point, it cannot be determined whether utterance D is really inappropriate. In the utterance tree structure 8d in this state, the utterance score US of the utterance D is changed from "1" to "1/2", and the utterance C is given the utterance score US "1/2". It is conceivable that the utterance tree structure 8d of FIG. 7 (B) transitions to the state of FIG. 7 (C) or FIG. 7 (D).

FIG. 7C shows a state in which the utterance E is generated by another worker E who determines that the utterance D is inappropriate. It can be understood that the utterance D is not inappropriate by adding the utterance E, but it cannot be determined yet which is more appropriate between the utterance C and the utterance E. In this case, the utterance D is deleted, and the utterance score US of the utterance C and the utterance E becomes "1/2" each.

FIG. 7D shows a state in which the utterance F is added without being judged to be inappropriate for the utterance D. Therefore, the utterance score US of utterance D is changed from "1/2" to "2/3", and the utterance score US of utterance C is changed from "1/2" to "1/3".

Next, worker scoring is performed using the utterance tree structure 8 that is generated and changed as described above. In the worker scoring, the average value of the utterance scores of all the utterances created by the worker 9 is calculated and used as the worker score.

FIG. 8 is a diagram for explaining worker scoring. FIG. 8 describes a case where the worker scores WS of the workers A to G are calculated by using the utterance scores US of the utterances A to G in the three utterance tree structures 8e, 8f, and 8g.

In this example
About worker A
Obtain the worker score WS _A = (1 + 1 + 1) / 3 = 1 and
About worker B
Obtain the worker score WS _B = (1/3 + 1 + 1) / 3 = 0.78,
About worker C
Worker score WS _C = (1 + 2/3) / 2 = 0.83 is obtained,
About worker D
Obtain the worker score WS _D = (0 + 1/3) / 2 = 0.17 and
About worker E
Obtain the worker score WS _E = (1 + 1 + 1) / 3 = 1 and
About worker F
Obtain the worker score WS _F = (1 + 1) / 2 = 1 and
About worker G
Worker score WS _G = (2/3) / 1 = 0.67
To get.

The lower the worker score WS, the more the worker 9 creates an inappropriate utterance. By filtering the worker score WS using a threshold value, the worker 9 who creates an inappropriate utterance may be excluded. In this case, it is possible to suppress the creation of inappropriate utterances.

Next, dialogue scoring will be described. Dialogue scoring is performed by the geometric mean of the worker scores WS of worker 9 who created the utterances on the most favored path. The most favored path is determined by selecting the branch with the highest number of utterances.

FIG. 9 is a diagram for explaining dialogue scoring. In FIG. 9, the case of calculating the dialogue score DS of each dialogue represented by the three utterance tree structures 8e, 8f, and 8g similar to those in FIG. 8 will be described.

The most favored path Pe in the utterance tree structure 8e is the context of utterances A, G, C, E, and F in that order. Therefore, the geometric mean of the worker score WS of the worker 9 who created the utterances constituting the path Pe is
(1 x 0.67 x 0.83 x 1 x 1) ^ (1/5) = 0.89
And a dialogue score of DSe = 0.89 is obtained.

The most supported path Pf in the utterance tree structure 8f is the context of utterances E, F, A, and B in that order. Therefore, the geometric mean of the worker score WS of the worker 9 who created the utterances constituting the path Pf is
(1 x 1 x 1 x 0.78) ^ (1/4) = 0.94
The dialogue score DSf = 0.94 is obtained.

The most supported path Pg in the utterance tree structure 8g is the context of utterances B, E, C, and A in that order. Therefore, the geometric mean of the worker score WS of the worker 9 who created the utterances constituting the path Pg is
(0.78 × 1 × 0.83 × 1) ^ (1/4) = 0.90
The dialogue score DSg = 0.90 is obtained.

The higher the dialogue score DS, that is, the closer it is to 1, the more appropriate the context is for the dialogue. In this example, since the dialogue score DSf shows the highest value "0.94", the context by the path Pf is considered to indicate a proper sequence of utterances.

Next, (III) efficiency improvement of worker allocation will be described with reference to FIGS. 10 to 14. FIG. 10 is a diagram for explaining an outline of the worker allocation procedure.

In FIG. 10, the worker list 41 is a list having a queue structure in which the worker IDs of the workers 9 are taken out in the order in which the worker IDs are added. As an example, the queue structure may be in a FIFO (First-In, First-Out) system. Just do it. In this example, the ID of the worker 9 is added from the right end, and the ID of the worker 9 is taken out from the left end.

The dialogue task table 42 is a table in which the workers 9 who created the utterances for each dialogue are shown in order. Each dialogue is registered in the dialogue task table 42 as a dialogue task, and is subsequently managed in the dialogue task table 42. The progress of the dialogue is indicated by a worker ID that identifies the worker 9 recorded in the dialogue task table 42.

In this example, as a plurality of workers 9 who can participate in the dialogue, referring to the worker list 41, the workers w4, w9, w21, w10, and w5 are already in the waiting state. Further, the first worker w4 is assigned to the dialogue, and the worker w7 is added to the worker list 41 following the last worker w5.

Dialogue tasks d1, ..., D5, etc. are registered in the dialogue task table 42. In the dialogue task d1, after the worker w4 creates the first utterance, the worker w11 is second with respect to the first utterance. Shows the state in which the utterance of is created. Similarly, for the dialogue tasks d2, d3, d4, etc., the worker IDs are shown in the order in which the utterances are created (utterance creation order). The dialogue task d5 indicates a state in which the utterance has not been created yet, although it has been registered.

The worker 9 is assigned to the dialogue according to the following worker assignment procedure.
-Procedure (1) The worker ID is taken out from the head (left) of the worker list 41 (FIG. 10).
-Procedure (2) Obtain the selection probability for each dialogue task registered in the dialogue task table 42.
Selection probability = Number of occurrences of worker ID / number of utterances In the above formula, the number of utterances does not count the dialogue task whose last worker ID is taken out.
-Procedure (3) Select the dialogue task with the highest selection probability.
If there are multiple largest dialogue tasks, select the one with the least number of utterances. If there are a plurality of the least interactive tasks, one interactive task is randomly selected.

Using the data example shown in FIG. 10, the worker allocation process according to the worker allocation procedure described above will be described with reference to FIGS. 11 to 14. FIG. 11 is a diagram showing an example of a worker allocation process when there is only one worker ID having the highest selection probability.

In the worker allocation process in FIG. 11, the worker ID “w4” is extracted from the worker list 41 by the procedure (1), and the interactive task table 42 is referred to by the procedure (2), and the extracted worker ID is extracted. The selection probability of "w4" is calculated.

In the case of this example, in the dialogue task d1, since one of the two created utterances is due to the worker ID “w4”, the selection probability “1/2” is obtained. In the dialogue task d2, since there is no utterance by the worker ID “w4” among the four created utterances, the selection probability “0” is obtained.

In the dialogue task d3, since the worker ID "w4" creates the utterance at the end, the selection probability is not calculated. The calculation of the selection probability is suppressed.

In the dialogue task d4, since there is no utterance by the worker ID “w4” among the two created utterances, the selection probability “0” is obtained. In the dialogue task d5, since the created utterance does not exist yet, the selection probability “0” is obtained.

Next, according to the procedure (3), the dialogue task d1 having the highest selection probability is selected, and "w4" is added to the end of the utterance creation order. In the dialogue task d1, "w4" is added after the worker ID "w11". From the worker list 41, the worker ID "w4" is followed by the worker ID "w9". The worker allocation process for the worker ID “w9” will be described with reference to FIG.

FIG. 12 is a diagram showing a first example of the worker allocation process when a plurality of the highest selection probabilities exist. In FIG. 12, the worker ID “w9” is extracted from the worker list 41 by the procedure (1), the interactive task table 42 is referred to by the procedure (2), and the extracted worker ID “w9” is selected. Calculate the probability.

In the case of this example, in the dialogue tasks d1 to d5, since there is no utterance by the worker ID "w9", the selection probability "0" is obtained for all the dialogue tasks d1 to d5. In this case, it corresponds to the case where there are a plurality of maximum dialogue tasks, and the dialogue task with the smallest number of utterances is selected according to the procedure (3).

The number of utterances of the dialogue task d1 is "3", the number of utterances of the dialogue task d2 is "4", the number of utterances of the dialogue task d3 is "1", and the number of utterances of the dialogue task d4 is "2". Yes, the number of utterances in the dialogue task d5 is "0".

Therefore, the dialogue task d5 is selected for the worker ID “w9”. The worker ID "w9" is assigned to the dialogue task d5 with the smallest number of utterances. In this example, "w9" is recorded as the first utterance for the dialogue task d5. Then, from the worker list 41, the worker ID "w9" is followed by the worker ID "w21". The worker allocation process for the worker ID “w21” will be described with reference to FIG.

FIG. 13 is a diagram showing a second example of the worker allocation process when a plurality of the highest selection probabilities exist. In FIG. 13, the worker ID “w21” is extracted from the worker list 41 by the procedure (1), the interactive task table 42 is referred to by the procedure (2), and the extracted worker ID “w21” is selected. Calculate the probability.

In the case of this example, since there is no utterance by the worker ID "w21" in the dialogue tasks d1 to d5, the selection probability "0" is obtained for all the dialogue tasks d1 to d5. In this case, it corresponds to the case where there are a plurality of maximum dialogue tasks, and the dialogue task with the smallest number of utterances is selected according to the procedure (3).

The number of utterances of the dialogue task d1 is "3", the number of utterances of the dialogue task d2 is "4", the number of utterances of the dialogue task d3 is "1", and the number of utterances of the dialogue task d4 is "2". Yes, the number of utterances in the dialogue task d5 is "1". Therefore, there are two dialogue tasks with the smallest number of utterances.

In this case, according to the procedure (3), one of the plurality of dialogue tasks with the smallest number of utterances is randomly selected. The random selection method is not particularly limited. An existing function or the like may be used.

It is assumed that the dialogue task "d3" is selected from the dialogue tasks "d3" and "d5" for the worker ID "w21". In the dialogue task d3, the worker ID "w21" is added next to the already assigned worker ID "w4".

From the worker list 41, the worker ID "w21" is followed by the worker ID "w10". The worker ID "w10" does not exist in the interactive task table 42. The situation similar to that of the worker ID "w9" is shown. Therefore, it is added to the dialogue task d5 with the smallest number of utterances. Then, from the worker list 41, the worker ID "w10" is followed by the worker ID "w5". The worker allocation process for the worker ID “w5” will be described with reference to FIG.

FIG. 14 is a diagram showing a third example of the worker allocation process when a plurality of the highest selection probabilities exist. In FIG. 14, the worker ID “w5” is extracted from the worker list 41 by the procedure (1), the interactive task table 42 is referred to by the procedure (2), and the extracted worker ID “w5” is selected. Calculate the probability.

In the case of this example, in the dialogue task d1, since there is no utterance by the worker ID “w5” among the three created utterances, the selection probability “0” is obtained. In the dialogue task d2, since two of the four created utterances are due to the worker ID "w5", the selection probability "1/2" (= 2/4) is obtained.

In the dialogue task d3, as in the dialogue task d1, since there is no utterance by the worker ID “w5”, the selection probability “0” is obtained. In the dialogue task d4, since one of the two created utterances is due to the worker ID "w5", the selection probability "1/2" is obtained. Similar to the dialogue task d1, the dialogue task d5 also has a selection probability of “0” because there is no utterance by the worker ID “w5”.

In the case of this example, in the dialogue tasks d1 to d5, the dialogue tasks having the highest selection probability with respect to the worker ID "w5" are "d2" and "d4", and there are a plurality of dialogue tasks. Therefore, according to the procedure (3), the dialogue task with the smallest number of utterances is selected.

The number of utterances may be counted for the dialogue tasks d2 and d4. The number of utterances of the dialogue task d2 is "4", and the number of utterances of the dialogue task d4 is "2". Therefore, the dialogue task d4 with the smallest number of utterances is selected.

The worker allocation process as described above is repeated every time the worker ID is sequentially taken out from the worker list 41. On the other hand, the worker ID is added to the worker list 41 at any time independently of the worker allocation process.

Various processes according to this embodiment are performed in the system configuration as shown in FIG. FIG. 15 is a diagram showing a system configuration example in this embodiment. The system 1000 shown in FIG. 15 includes a server device 100, a plurality of worker terminals 3 including a mobile terminal, a PC (Personal Computer), and one or more user terminals 6, and the server device 100 and a plurality of user terminals 6. The worker terminal 3 and one or more user terminals are connected via the network 2.

The server device 100 is a device that executes the dialogue corpus creation process according to the present embodiment, and is more useful by having an unspecified number of workers 9 create utterances according to the purpose of each of the plurality of dialogues. Create a dialogue corpus by accumulating high dialogue cases.

The plurality of worker terminals 3 are terminals used by the worker 9, and each worker 9 connects the worker terminal 3 to the server device 100 via the network 2 to perform the operation via the network 2. You can participate in the dialogue.

The one or more user terminals 6 are terminals used by the user 7 who uses a series of conversation data (dialogue corpus) related to the dialogue accumulated by the server device 100, and the user 7 is the user terminal 6. Is connected to the server device 100 via the network 2, and conversation data is acquired from the server device 100.

The server device 100 is an information processing device controlled by a computer, and is a CPU (Central Processing Unit) 11, a main storage device 12, an auxiliary storage device 13, an input device 14, a display device 15, and a communication I. It has a / F (interface) 17 and a drive device 18, and is connected to the bus B1.

The CPU 11 corresponds to a processor that controls the server device 100 according to a program stored in the main storage device 12. A RAM (Random Access Memory), a ROM (Read Only Memory), or the like is used in the main storage device 12, and is obtained by a program executed by the CPU 11, data required for processing by the CPU 11, and processing by the CPU 11. Store or temporarily store the collected data.

An HDD (Hard Disk Drive) or the like is used in the auxiliary storage device 13, and data such as a program for executing various processes is stored in the auxiliary storage device 13. Various processes are realized by loading a part of the program stored in the auxiliary storage device 13 into the main storage device 12 and executing the program in the CPU 11.

The input device 14 has a mouse, a keyboard, and the like, and is used by the administrator to input various information necessary for processing by the server device 100. The display device 15 displays various necessary information under the control of the CPU 11. The input device 14 and the display device 15 may be a user interface using an integrated touch panel or the like. The communication I / F 17 communicates through a network such as wired or wireless. Communication by communication I / F17 is not limited to wireless or wired. A program that realizes processing performed by the server device 100 is a server using a storage medium 19 such as a CD-ROM (Compact Disc Read-Only Memory). Provided to device 100.

The drive device 18 interfaces the storage medium 19 (for example, a CD-ROM or the like) set in the drive device 18 with the server device 100.

Further, a program for realizing various processes according to the present embodiment described later is stored in the storage medium 19, and the program stored in the storage medium 19 is installed in the server device 100 via the drive device 18. To. The installed program can be executed by the server device 100.

The storage medium 19 for storing the program is not limited to the CD-ROM, and one or more non-transitory, tangible (tangible) having a structure as data that can be read by a computer. ) Any medium. As the computer-readable storage medium, in addition to the CD-ROM, a portable recording medium such as a DVD disk or a USB memory, or a semiconductor memory such as a flash memory may be used.

The plurality of worker terminals 3 are information processing terminals such as tablet-type and mobile phones controlled by a computer, and include a CPU 11a, a main storage device 12a, a user I / F (interface) 16a, and a communication I / F 17a. And a drive device 18a, which are connected to the bus B1.

The CPU 11a corresponds to a processor that controls the worker terminal 3 according to a program stored in the main storage device 12a. A RAM (Random Access Memory), a ROM (Read Only Memory), or the like is used in the main storage device 12a, and a program executed by the CPU 11a, data required for processing by the CPU 11a, and a process obtained by the CPU 11a are obtained. Store or temporarily store the collected data. Various processes are realized by executing the program stored in the main storage device 12a in the CPU 11a.

The user I / F 16a is a touch panel or the like that displays various information required under the control of the CPU 11a and enables the user to input operations. Communication by communication I / F17a is not limited to wireless or wired.

The program that realizes the processing performed by the worker terminal 3 is downloaded from the external device via the network 2. Alternatively, it may be stored in advance in the main storage device 12a or the storage medium 19a of the worker terminal 3. The main storage device 12a and / or the storage medium 19a corresponds to the storage unit 130a.

The drive device 18a interfaces the storage medium 19a (for example, SD (Secure Digital) memory card or the like) set in the drive device 18a with the worker terminal 3. The storage medium 19a may be one or more non-transitory, tangible media having a structure as data that can be read by a computer.

Since the hardware configuration of the user terminal 6 is the same as that of the worker terminal 3, detailed description thereof will be omitted. Further, the worker terminal 3 and the user terminal 6 may be information processing terminals such as a desktop type, a notebook type, and a laptop type, and the hardware configuration thereof is the same as the hardware configuration of the server device 100. Since there is, the description thereof will be omitted.

FIG. 16 is a diagram for explaining an outline of the overall processing in the system of this embodiment. In FIG. 16, an outline of the overall processing in the system of this embodiment will be described using a processing sequence.

The worker 9 transmits the dialogue participation request 3a from the worker terminal 3 to the server device 100 (step S1). The worker ID is included in the dialogue participation request 3a. The worker ID may be acquired from the server device 100 before the dialogue participation request 3a is performed, or the server device 100 assigns the worker ID in response to the first dialogue participation request 3a and the worker terminal 3 May be notified to. After that, the worker ID given by the server device 100 is specified, and the dialogue participation request 3a is transmitted to the server device 100.

The server device 100 uses the worker ID specified in the dialogue participation request 3a to perform a worker allocation process for specifying the dialogue for which the worker ID is assigned (step S2), and is the request source of the dialogue participation request 3a. The dialogue task is presented to the worker terminal 3 and the dialogue task is displayed on the worker terminal 3 (step S4). The dialogue task presentation 3b includes dialogue premise information, utterance history information, and the like. The dialogue premise information includes an explanation of the scene (premise) in which the dialogue is performed, and is information for making the worker 9 understand his / her role. The utterance history information corresponds to the content of the already performed dialogue displayed on the dialogue screen G70.

On the worker terminal 3, the dialogue screen G70 is displayed on the user I / F16a, and when the utterance is created by the worker 9 (step S5), the utterance creation information 3c is transmitted to the server device 100 (step S6). The utterance creation information 3c includes the presence / absence of an inappropriate check, the presence / absence of termination, utterance data representing the utterance input by the worker 9 in the utterance input area 72 of the dialogue screen G70, and the like.

The server device 100 performs the utterance creation process in response to the reception of the utterance creation information 3c (step S7). In the utterance creation process, the server device 100 stores the utterance data in the storage unit 130, adds an utterance ID that identifies the utterance data in association with the dialogue task assigned to the worker 9, and updates the utterance score in the dialogue task. (Step S7). Such steps S1 to S7 are repeated as appropriate.

On the other hand, when the user 7 makes the dialogue data acquisition request 6a from the user terminal 6 (step S8), the server device 100 performs the dialogue data retrieval process (step S9). In the dialogue data retrieval process, the server device 100 calculates the worker score for the worker 9 based on the utterance score for all the dialogues. Further, the server device 100 calculates the dialogue score for each dialogue based on the utterance score for each utterance associated with the dialogue.

Then, the dialogue data 6b with a score based on a predetermined condition is provided (step S10). The predetermined condition is, for example, a threshold value of the dialogue score desired by the user 7, may be specified in the dialogue data acquisition request 6a, or may be a default threshold value preset in the dialogue data retrieval process. Good.

FIG. 17 is a diagram showing a functional configuration example of the server device. In FIG. 17, the server device 100 mainly includes an initialization unit 50, a worker registration unit 51, a worker allocation unit 52, an utterance creation unit 55, a dialogue data extraction unit 57, and a user I / F 60. Has. The initialization unit 50, the worker registration unit 51, the worker allocation unit 52, the utterance creation unit 55, the dialogue data extraction unit 57, and the user I / F60 are such that the CPU 11a executes a corresponding program. Corresponds to the processing performed.

The storage unit 130 stores the worker list 41, the dialogue task table 42, the utterance table 43, the utterance data DB 44, the worker score table 45, the dialogue data 6b with a score, and the like.

The initialization unit 50 initializes the worker list 41, the dialogue task table 42, and the like when resetting the creation of the dialogue corpus at the start of operation of the system according to the present embodiment.

The worker registration unit 51 is a processing unit that registers the worker ID specified in the dialogue participation request 3a in the worker list 41 in response to the reception of the dialogue participation request 3a via the user I / F60.

The worker allocation unit 52 identifies the worker 9 for each utterance in response to the reception of the dialogue participation request 3a via the user I / F60, and asks the specified worker 9 to create an utterance. This is a processing unit that selects and performs the dialogue task presentation 3b on the worker terminal 9. The process of identifying the worker 9 for each utterance is a part of the process performed by the worker allocation unit 52, and corresponds to the worker ID extraction unit. The worker allocation unit 52 further has a dialogue task selection unit 53 including a dialogue task generation unit 54.

The dialogue task selection unit 53 is a processing unit called from the worker allocation unit 52, and corresponds to the selection unit. The dialogue task selection unit 53 selects the dialogue task to be assigned to the worker 9 according to the worker allocation procedure described above. The dialogue task selection unit 53 calls the dialogue task generation unit 54 to generate a dialogue task, and transmits the generated dialogue task to the worker terminal 3 via the user I / F60 to display it.

When the utterance creation unit 55 receives the utterance creation information 3c from the worker terminal 3 via the user I / F60, the utterance creation unit 55 updates the dialogue task table 42. The utterance data acquired from the utterance creation information 3c is accumulated in the utterance data DB 44 together with the newly assigned utterance ID. The utterance data DB 44 corresponds to the dialogue corpus 1a.

The utterance creation unit 55 further has an utterance information update unit 56, and by calling the utterance information update unit 56, the utterance data is added, and the utterance score of the dialogue task is recalculated and updated.

The utterance information update unit 56 updates the utterance tree structure managed in the utterance table 43 based on the utterance creation information 3c, and updates the utterance score.

When the dialogue data acquisition unit 57 receives the dialogue data acquisition request 6a from the user terminal 6 via the user I / F60, the dialogue data acquisition unit 57 refers to the utterance table 43, calculates the worker score and the dialogue score, and has a score. The dialogue data 6b is output. A part or all of the dialogue data 6b may be provided to the user terminal 6. When the user 7 is the administrator of the server device 100, the dialogue data acquisition request 6a may be made from the input device 114.

The dialogue data extraction unit 57 further includes a worker score calculation unit 58 and an utterance score calculation unit 59. The dialogue data extraction unit 57 calls the worker score calculation unit 58 to calculate the worker score of each worker 9 using the utterance table 43. The worker score calculation unit 58 outputs and stores the worker score table 45 in the storage unit 130.

The dialogue data extraction unit 57 calls the utterance score calculation unit 59 and calculates the dialogue score for each dialogue task using the worker score table 45. The dialogue data extraction unit 57 outputs the scored dialogue data 6b to the storage unit 130 and stores it.

The user I / F 60 performs an interface between the server device 100 and the worker 9 of the worker terminal 3 and an interface between the server device 100 and the user 7 of the user terminal 6 via the communication I / F 117.

Next, FIGS. 18 to 23 will be described with respect to various data configurations stored in the storage unit 130.

FIG. 18 is a diagram showing an example of data structure of the worker list. As described with reference to FIG. 10, the worker list 41 shown in FIG. 18 is a list having a queue structure in which the worker IDs of the workers 9 are taken out in the order of addition.

In this example, w4, w9, w21, w10, w5, w7, w8, and w40 are shown in order of addition from the left of the worker list 41. The length of the worker list 41 is expanded and contracted by adding and taking out the worker ID, and the worker list 41 is an expandable storage area.

FIG. 19 is a diagram showing an example of data configuration of the interactive task table. In FIG. 19, the dialogue task table 42 is a table in which worker IDs are recorded in the order in which utterances are created for each dialogue task, and has items such as a dialogue ID and a list of worker IDs.

The dialogue ID is identification information that identifies the dialogue task. All interactive tasks are registered and managed in the task table 42. The list of worker IDs is a list in which worker IDs are recorded in the order of creation for each utterance creation.

In this example, the worker IDs "w4" and "W11" are recorded for the dialogue ID "d1". Worker IDs "w5", "w7", "w5", and "W8" are recorded for the dialogue ID "d2".

FIG. 20 is a diagram showing an example of data structure of the utterance table. In FIG. 20, the utterance table 43 is a table for recording and managing the utterances created for the dialogue task in the utterance tree structure 8. The utterance table 43 has items such as a dialogue ID, an utterance ID, a parent utterance, an operator ID, an utterance score (US), and a number of supporters.

The dialogue ID indicates identification information that identifies the dialogue, and is identification information registered in the dialogue task table 42. The utterance ID is identification information that is associated with the utterance data representing the created utterance and is managed in the utterance data DB 44.

The parent utterance indicates an utterance ID having a parental relationship in the utterance tree structure 8. The worker ID indicates identification information that identifies the worker 9 who created the utterance. The utterance score (US) indicates a value calculated based on the number of supporters, and is updated every time an utterance is added. The number of support indicates a value added or subtracted for each additional utterance, and is referred to when calculating the utterance score.

In this data example, the utterance IDs "u21" to "u27" are recorded with respect to the dialogue ID "d2". In the record of the utterance ID "u21", since the utterance ID corresponds to the parent utterance, the utterance ID does not exist, the utterance creator is the worker 9 of the worker ID "w1", and the utterance score is "1". Yes, it shows the number of support "1".

In the record of the utterance ID "u22", the parent utterance ID is "u21", the utterance creator is the worker 9 of the worker ID "w4", the utterance score is "0", and the number of support is "0". Is shown. In the record of the utterance ID "u23", the parent utterance ID is "u21", the utterance creator is the worker 9 of the worker ID "w7", the utterance score is "2/3", and the number of support is "2/3". 2 ”is shown. In the record of the utterance ID "u24", the parent utterance ID is "u21", the utterance creator is the worker 9 of the worker ID "w2", the utterance score is "1/3", and the number of support " 1/3 "is shown.

In the record of the utterance ID "u25", the parent utterance ID is "u23", the utterance creator is the worker 9 of the worker ID "w3", the utterance score is "1", and the number of support is "1". Is shown. In the record of the utterance ID "u26", the parent utterance ID is "u25", the utterance creator is the worker 9 of the worker ID "w5", the utterance score is "1", and the number of support is "1". Is shown. In the record of the utterance ID "u27", the parent utterance ID is "u26", the utterance creator is the worker 9 of the worker ID "w6", the utterance score is "1", and the number of support is "1". Is shown.

With such data, the utterance tree structure 8k can be represented. The utterance tree structure 8k represents a genealogy of utterance creation that constitutes dialogue d2. The node indicates an utterance, and the utterance ID is indicated in the node. In addition, the worker ID is shown on the branch, and the utterance score is shown near the node.

FIG. 21 is a diagram showing a data configuration example of the utterance data DB. In FIG. 21, the utterance data DB 44 is a database that stores and manages utterance data received from the worker terminal 3 in association with the utterance ID, and has items such as the utterance ID and the utterance data.

The utterance ID is identification information uniquely assigned to each utterance data received from the worker terminal 3, and all the utterance data is managed by the utterance ID in the utterance data DB 44. The utterance ID associates the utterance data DB 44 with the utterance table 43.

The utterance data is data representing the utterance created by the worker 9 acquired from the utterance creation information 3c received from the worker terminal 3. As an example, the utterance data is text, voice data, an image or a moving image expressing a dialogue situation, or the like.

In this data example, the utterance data of the utterance ID "u11" is "Where are you going?", The utterance data of the utterance ID "u12" is "Go to Chicago", and the utterance data of the utterance ID "u13". "Please enjoy" etc. are recorded. Similarly, for the other utterance IDs, the content itself created by the worker 9 is shown.

FIG. 22 is a diagram showing an example of data structure of the worker score table. In FIG. 22, the worker score table 45 is a table for recording and managing the worker score for each worker 9, and has items such as a worker ID and a worker score.

The worker ID indicates a worker ID that has been added to the worker list 41 at least once. The utterance table 43 and the worker score table 45 are associated with each other by the worker ID.
The worker score indicates a score calculated using the utterance score associated with the worker ID of the utterance table 43, and the degree of sincere contribution to the utterance creation of the worker 9 (reliability of the content of the created utterance). ) Is shown.

FIG. 23 is a diagram showing a data structure example of dialogue data with a score. In FIG. 23, the scored dialogue data 6b is a table showing the result of evaluating the dialogue task, and has items such as dialogue ID, dialogue score, utterance 1, utterance 2, and utterance 3.

The dialogue ID is identification information that identifies the dialogue task, and indicates any of the dialogue IDs existing in the dialogue task table 42. The dialogue score identifies the most supported path based on the utterance score of the utterance table 43, indicates a value calculated using the utterance score shown on the identified path, and indicates the degree of completion of the dialogue task. When the dialogue score is "1", the degree of completion is the highest, and the closer it is to "0", the lower the degree of completion. Utterance 1, utterance 2, utterance 3, etc. indicate utterance data made on the most supported path.

In this data example, for the dialogue ID "d1", the dialogue score is "1", and the utterances made on the most supported path are utterance 1 "Where are you going?" And utterance 2 "Go to Chicago." It is shown that it was "Masu", utterance 3 "Please enjoy", etc.

For the dialogue ID "d3", the dialogue score is "0.75", and the utterances made on the most supported path are utterance 1 "where are you going", utterance 2 "see off", utterance 3 "[End]" is shown. When the worker 9 determines that this dialogue is completed in the immediately preceding utterance and checks the end check area 74 of the dialogue screen G70, [End] is indicated as in utterance 3.

Next, the processing in each processing unit 50 to 58 will be described. In the figure below, the dotted line shows the data flow and the solid line shows the processing flow. First, the initialization process by the initialization unit 50 will be described.

FIG. 24 is a flowchart for explaining the initialization process. In FIG. 24, the initialization unit 50 acquires the number of dialogues (step S201). The number of dialogues may be acquired by reading the number of dialogues preset in the storage unit 130, or by displaying an input screen on the display device 115 and having the administrator of the server device 100 input the number of dialogues as an initial setting. .. The initialization unit 50 empties the worker list 41 (step S202) and initializes the dialogue task table 42 based on the number of dialogues (step S203).

As an example of initialization of the dialogue task table 42, when the number of dialogues "5" is specified, five dialogue IDs that identify the dialogue tasks are generated and registered in the dialogue task table 42. When the dialogue IDs "d1", "d2", "d3", "d4", and "d5" are generated, a record is generated in the dialogue task table 42 for each dialogue ID.

Next, the worker registration process by the worker registration unit 51 and the worker allocation process by the worker allocation unit 52 will be described. FIG. 25 is a flowchart for explaining the worker registration process and the worker allocation process.

In FIG. 25, when the worker registration unit 51 receives the dialogue participation request 3a via the user I / F60, the worker registration unit 51 acquires the worker ID from the dialogue participation request 3a and adds it to the worker list 41 (step S211). .. The worker registration unit 51 performs the same process each time the dialogue participation request 3a is received.

On the other hand, the worker allocation unit 52 takes out one worker ID in order from the worker list 41 at predetermined intervals (step S221), calls the dialogue task selection unit 53, and notifies the worker ID and the dialogue task table 42. Then, the dialogue task selection unit 53 performs the dialogue task selection process (step S222). The dialogue task to be assigned to the worker ID is specified by the dialogue ID. Step S221 by the worker allocation unit 52 corresponds to the worker ID extraction unit, and step S222 by the dialogue task selection unit 53 corresponds to the selection unit.

When the interactive task is specified by the task selection process, the worker allocation unit 52 determines whether or not the worker list 41 is empty (step S224). If the worker list 41 is not empty (NO in step S224), the worker allocation unit 52 returns to step S221 and repeats the same process as described above. On the other hand, when the worker list 41 is empty (YES in step S224), the worker allocation unit 52 ends the worker allocation process.

FIG. 26 is a flowchart for explaining the dialogue task selection process in step S222 of FIG. 25. In FIG. 26, when the dialogue task selection unit 53 receives the worker ID and the dialogue task table 42 from the worker allocation unit 52, the dialogue task selection unit 53 calculates the selection probabilities for all the dialogue IDs in the dialogue task table 42 (step S231). ..

The selection probability is obtained for each dialogue ID according to the worker allocation procedure described above. If the interactive task ends with the notified worker ID,
Selection probability = 0
And. Otherwise
Selection probability = Calculated based on the number of occurrences / number of utterances of the worker ID.

The dialogue task selection unit 53 selects the dialogue ID having the maximum selection probability from the dialogue task table 42 (step S232), and determines whether or not the number of selected dialogue IDs is one (step S233). When the number of selected dialogue IDs is one (YES in step S233), the dialogue task selection unit 53 proceeds to step S237.

On the other hand, when the number of selected dialogue IDs is plural (NO in step S233), the dialogue task selection unit 53 selects the dialogue ID having the smallest number of utterances (step S234), and the number of selected dialogue IDs is one. It is determined whether or not (step S235). When the number of selected dialogue IDs is one (YES in step S235), the dialogue task selection unit 53 proceeds to step S237.

On the other hand, when the number of selected dialogue IDs is plural (NO in step S235), the dialogue task selection unit 53 randomly selects one from the selected plurality of dialogue IDs (step S236), and the dialogue task generation unit 54 Is called to notify the dialogue ID and the utterance table 43, and the dialogue task generation unit 54 performs the dialogue task generation process (step S237). A dialogue task corresponding to the dialogue ID is generated.

When the dialogue task selection unit 53 acquires the dialogue task, it sets it as a return value to the worker allocation unit 52 and ends the dialogue task selection process.

FIG. 27 is a flowchart for explaining the dialogue task generation process in step S237 of FIG. 26. In FIG. 27, a case where the dialogue task specified by the dialogue ID transitions in the order of <case 1> → <case 2> → <case 3> → <case 4> for each utterance creation will be described. The utterance tree structure 8 of the dialogue ID designated by the dialogue task generation unit 54 corresponds to any one of <Case 1>, <Case 2>, <Case 3>, or <Case 4>.

The dialogue task generation unit 54 traces the utterance ID associated with the designated dialogue ID in the utterance table 43 according to the utterance tree structure 8 by using the utterance table 43 and the dialogue ID received from the dialogue task generation unit 54, and ends. Of the utterance IDs of the above, the utterance ID whose utterance score is not 0 is acquired (step S241).

The dialogue task generation unit 54 determines whether the number of acquired utterance IDs is 1 or 2 (step S242).

When there is only one utterance ID acquired, the dialogue task generation unit 54 refers to the utterance data DB 44 using the utterance ID acquired from the utterance table 43, acquires utterance history information, and creates a dialogue task ( Step S243).

In this case, it corresponds to <Case 1>. The utterance ID of the child is only "D" with respect to the utterance ID "A" of the parent. Therefore, in step S243, the utterance data specified by the utterance ID “A” from the utterance data DB 44 and the utterance data specified by the utterance ID “D” are arranged in chronological order according to the utterance tree structure 8 of <Case 1>. Get utterance history information by arranging them. A dialogue task containing this utterance history information is created.

After that, the dialogue task generation unit 54 sets the created dialogue task as a return value, and ends the dialogue task generation process.

When it is determined that the number of utterance IDs acquired in step S242 is two, the dialogue task generation unit 54 acquires the number of support for each utterance ID from the utterance table 43, and the total value of the acquired number of support indicates 2. It is determined whether or 3 is shown (step S244).

When it is determined in step S244 that the total value of the number of instructions indicates 2, the dialogue task generation unit 54 refers to the utterance data DB 44 using the utterance ID excluding the last added utterance ID, and acquires the utterance history information. To create an interactive task (step S245).

In this case, it corresponds to <Case 2> or <Case 3>. In the case of <Case 2>, since the last added utterance ID is "C", this utterance ID "C" is excluded. Therefore, the utterance data specified by the utterance ID "A" from the utterance data DB 44 and the utterance data specified by the utterance ID "D" are arranged in chronological order according to the utterance tree structure 8 of <Case 2>. Get history information. A dialogue task containing this utterance history information is created.

In the case of <Case 3>, since the last added utterance ID is "E", this utterance ID "E" is excluded. Further, since it is determined that the utterance of the utterance ID "D" is inappropriate, it is not considered, and the utterance data specified by the utterance ID "A" and the utterance ID "C" are specified from the utterance data DB44. Based on the utterance data, the utterance history information is acquired by arranging them in chronological order according to the utterance tree structure 8 of <Case 3>. A dialogue task containing this utterance history information is created.

After that, the dialogue task generation unit 54 sets the created dialogue task as a return value, and ends the dialogue task generation process.

When it is determined that the total value of the number of instructions acquired in step S244 indicates 3, the dialogue task generation unit 54 refers to the utterance table 44, selects the utterance with the highest utterance score, and creates a dialogue task. (Step S246).

In this case, it corresponds to <Case 4>. In the case of <Case 4>, the utterance ID having the highest utterance score is "F". According to the path having the utterance ID "F", the utterance data specified by the utterance ID "A" from the utterance data DB44, the utterance data specified by the utterance ID "D", and the utterance specified by the utterance ID "F". Based on the data, the utterance history information is acquired by arranging them in chronological order according to the utterance tree structure 8 of <Case 4>. A dialogue task containing this utterance history information is created.

After that, the dialogue task generation unit 54 sets the created dialogue task as a return value, and ends the dialogue task generation process.

Next, the utterance creation process by the utterance creation unit 55 will be described. FIG. 28 is a flowchart for explaining the utterance creation process. In FIG. 28, when the utterance creation unit 55 receives the utterance creation information 3c via the user I / F60, it assigns the utterance ID to the utterance data in the utterance creation information 3c, records it in the utterance data DB 44, and records the utterance ID. Is added to the interactive task table 42 (step S251).

Then, the utterance creation unit 55 acquires the presence / absence of the inappropriate check from the utterance creation information 3c, notifies the utterance information update unit 56 of the acquired improper check presence / absence, the utterance ID, and the utterance table 43, and utters. Information update processing is performed (step S252). The utterance table 43 is updated by the utterance information update process, and the utterance creation unit 55 ends the utterance creation process.

Next, the utterance information update process by the utterance information update unit 56 will be described. FIG. 29 is a flowchart for explaining the utterance information update process in step S252 of FIG. 28.

In FIG. 29, the utterance information updating unit 56 determines whether or not to newly add the utterance ID (step S261). If there is no other utterance ID having the last utterance ID "D" of the utterance tree structure 8p as the parent utterance ID, it is determined that the utterance is newly added.

As an example, in FIG. 28, when the assigned dialogue is represented by the utterance tree structure 8p, it is determined that a new utterance ID is added. On the other hand, when the assigned dialogue is represented by the utterance tree structure 8q, it is determined that the new utterance ID is not a new addition.

When it is determined in step S261 that a new utterance is to be added, the utterance information update unit 56 further determines whether or not the last utterance is inappropriate (step S262). If there is no inappropriate check, the utterance information update unit 56 determines that the last utterance is not inappropriate, additionally registers a new utterance ID in the utterance table 43, and sets the number of support to 1. (Step S263).

A record of a new utterance ID having a support number of "1" with the last utterance ID "D" as the parent utterance ID is added to the utterance table 43. The utterance tree structure 8p transitions to the utterance tree structure 8p-1. Then, the utterance information update unit 56 proceeds to step S268.

On the other hand, in step S262, when there is an inappropriate check, the utterance information update unit 56 determines that the last utterance is inappropriate, additionally registers a new utterance ID in the utterance table 43, and sets it as the last utterance ID. The same parent utterance ID is set, and the number of supports is set to 1 (step S264).

The same parent utterance ID as the last utterance ID is set, and a record of a new utterance ID with a support number of "1" is added to the utterance table 43. The utterance tree structure 8p transitions to the utterance tree structure 8p-2. Then, the utterance information update unit 56 proceeds to step S268.

If it is determined in step S261 that no new utterance is to be added, the utterance information updating unit 56 further determines whether or not the last utterance is inappropriate (step S265). If there is no inappropriate check, the utterance information update unit 56 determines that the last utterance is not inappropriate, increases the number of support by 1 with respect to the parent utterance ID in the utterance table 43, and additionally registers a new utterance ID. , The number of supports is set to 1 (step S266).

The number of support of the record of the parent utterance ID is added by 1, and the record of the new utterance ID with the number of support "1" is added to the utterance table 43. Speaking tree structure 8q Transition to the utterance tree structure 8q-1. Then, the utterance information update unit 56 proceeds to step S268.

On the other hand, in step S265, when there is an inappropriate check, the utterance information updating unit 56 determines that the last utterance is inappropriate. Then, the utterance information update unit 56 sets the number of support to 0 for the last utterance ID in the utterance table, additionally registers a new utterance ID, sets the same parent utterance ID as the last utterance ID, and sets the same parent utterance ID. The number of supports is set to 1 (step S267).

The number of support of the record of the last utterance ID is set to 0. Further, the record of the new utterance ID having the same parent utterance ID as the last utterance ID and the number of support "1" is added to the utterance table 43. The utterance tree structure 8q transitions to the utterance tree structure 8q-2. Then, the utterance information update unit 56 proceeds to step S268.

In step S268, the utterance information update unit 56 calculates the utterance score for each of the node whose support number has changed and its sibling node, and sets it in each record. The utterance score is
Support number ÷ Obtained by the total support number of sibling nodes.

In the utterance tree structure 8p-1, since there is no sibling node of the new utterance ID “N”, the utterance score is “1”.

Further, in the utterance tree structure 8p-2, the sibling node of the new utterance ID “N” is the utterance ID “D”. Therefore, "1/2" is set in the utterance score of the utterance ID "N", and "1/2" is set in the utterance score of the utterance ID "D".

On the other hand, in the utterance tree structure 8q-1, since the utterance ID “D” whose support number has changed has a sibling node of the utterance ID “C”, the utterance score of the utterance ID “D” is “2/3”. It is set, and "1/3" is set in the utterance score of the utterance ID "C". Since there are no siblings in the utterance ID "N", the utterance score is "1".

Further, in the utterance tree structure 8q-2, the utterance ID "C" and the newly added utterance ID "N" exist as sibling nodes in the utterance ID "D" whose support number has changed. The utterance score of the utterance ID "D" is "0", indicating a deleted state. The utterance score of the utterance ID "C" of the sibling node is set to "1/2", and the utterance score of the utterance ID "N" is set to "1/2".

Next, the dialogue data retrieval process by the dialogue data retrieval unit 57 will be described. FIG. 30 is a flowchart for explaining the dialogue data retrieval process. In FIG. 30, when the dialogue data acquisition unit 57 receives the notification of the dialogue data acquisition request 6a from the user I / F60, it calls the worker score calculation unit 58, notifies the utterance table 43, and performs the worker score calculation process. (Step S271). The dialogue data extraction unit 57 acquires, as a return value, a worker score table 45 indicating the reliability of the content of the utterance creation for each worker 9 from the worker score calculation unit 58.

Then, the dialogue data extraction unit 57 calls the dialogue score calculation unit 59, notifies the worker score table 45 obtained from the worker score calculation unit 58, and performs the dialogue score calculation process (step S272). The dialogue data extraction unit 57 acquires scored dialogue data 6b indicating the appropriateness of the context for each dialogue as a return value from the dialogue score calculation unit 59.

Next, the worker score calculation process by the worker score calculation unit 58 will be described. FIG. 31 is a flowchart for explaining the worker score calculation process in step S271 of FIG. 30.

In FIG. 31, the worker score calculation unit 58 refers to the utterance table 43 and obtains the total number of utterances and the utterance score for each worker ID (step S281).

Then, the worker score calculation unit 58 calculates an average value from the obtained number of utterances and the total of the utterance scores for each worker ID, and the worker score table in which the worker ID and the calculated average value are associated with each other. 45 is created (step S282). The worker score calculation unit 58 sets the created worker score table 45 as a return value, and ends the worker score calculation process.

Next, the dialogue score calculation process by the dialogue score calculation unit 59 will be described. FIG. 32 is a flowchart for explaining the dialogue score calculation process in step S272 of FIG. 30.

In FIG. 32, the dialogue score calculation unit 59 is most supported by selecting the utterance ID based on the utterance score according to the parent-child relationship of the utterance tree structure 8 for each dialogue ID with reference to the utterance table 43. The path is specified (step S291). When the same parent utterance ID is associated with two or more utterance IDs, the dialogue score calculation unit 59 selects the utterance ID having the maximum utterance score.

The dialogue score calculation unit 59 refers to the utterance table 43 and acquires the worker ID associated with the utterance ID on the specified path for each dialogue ID (step S292). Further, the dialogue score calculation unit 59 refers to the worker score table 45 and acquires the worker ID associated with the utterance ID on the specified path for each dialogue ID (step S293).

In addition, the dialogue score calculation unit 59 calculates the geometric mean for each pass using the total value of the worker scores and the total number of dialogue IDs (step S294).

Then, the dialogue score calculation unit 59 acquires utterance data from the utterance data DB 44 in the order of the utterance IDs of the most supported paths for each dialogue ID, and obtains the dialogue ID, the dialogue score, and the utterance data. The associated dialogue data 6b with a score is created (step S295). The dialogue score calculation unit 59 sets the created dialogue data 6b with a score as a return value, and ends the dialogue score calculation process.

From the above, in this embodiment, since it is not necessary to make a pair of workers 9, it is possible to reduce the work cost such as making a pair and adjusting the time. When creating a pair, the size of the dialogue corpus that can be created at a fixed cost becomes small due to the burden of managing the pair and the high unit cost of work. Therefore, only a small dialogue corpus could be created.

On the other hand, in the present embodiment, since the individual worker 9 can perform the utterance work of the dialogue, the worker 9 can perform the work at an arbitrary place and time by using the worker terminal 3. Therefore, an unspecified number of workers 9 can perform the work, and a large-scale dialogue corpus can be created. In addition, crowdsourcing, which is a method of ordering work from an unspecified number of individual workers 9, can be used.

Further, it is possible to identify the worker 9 who creates an inappropriate utterance based on the worker score, and by filtering the worker 9 by the worker score, the creation of an inappropriate utterance can be suppressed. In addition, since it is possible to calculate a score indicating the appropriateness of utterance, it is possible to identify a high-quality dialogue and provide the content of the dialogue (a set of utterance data in the flow of the dialogue).

Further, by assigning the worker 9 to the dialogue using the selection probability, the load on the worker 9 can be reduced, and thus the work unit price can be kept low.

In crowdsourcing, the number of workers 9 who can work is not constant in time series, but appropriate work can be assigned regardless of the number.

The present invention is not limited to the specifically disclosed examples, and major modifications and modifications can be made without departing from the scope of claims.

The following additional notes will be further disclosed with respect to the above embodiments including the present embodiment.
(Appendix 1)
From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. Take out for each utterance
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. According to the worker allocation procedure based on, select the dialogue to which the worker who creates the utterance data is assigned, and
A dialogue corpus creation program that causes a computer to execute a process of obtaining dialogue data that completes the dialogue by the plurality of workers.
(Appendix 2)
Displaying the dialogue information of the dialogue on the worker terminal of the worker to which the dialogue is assigned, checking when the content of the utterance immediately before being indicated by the dialogue information is inappropriate, and the input area of the utterance data. Display the creation screen with
When at least the utterance data and the presence / absence of the inappropriate check are received from the worker terminal, the received utterance data is managed by a tree structure having a parent-child relationship according to the progress of the dialogue, and the inappropriate check is performed. If there is, the received utterance data is branched from the parent's utterance data and recorded in the storage unit as a sibling relationship with the immediately preceding utterance data.
The dialogue corpus creation program according to Appendix 1, which causes the computer to execute a process of determining the inappropriateness of the immediately preceding utterance data according to the presence or absence of the inappropriateness check received together with the immediately preceding utterance data.
(Appendix 3)
In the tree structure, a support number is given to the received utterance data, and the support number is given.
When another utterance data having a sibling relationship exists with respect to the immediately preceding utterance data, the number of supports of the immediately preceding utterance data is increased or decreased depending on the presence or absence of the inappropriate check received together with the received utterance data. The interactive corpus creation program according to Appendix 2, which causes the computer to execute the processing to be performed.
(Appendix 4)
The utterance score is calculated from the utterance data in which the number of support increases or decreases and the utterance data of the sibling relationship of the utterance data, and the ratio of the number of support of each of the utterance data to the total number of support is calculated and stored in the storage unit. The interactive corpus creation program according to Appendix 3, which causes the computer to execute a process of storing data.
(Appendix 5)
With reference to the storage unit, the number of the created utterance data and the total value of the utterance score are acquired for each worker ID.
The dialogue according to Appendix 4, wherein an average value of the utterance scores is calculated for each worker ID, and a process of associating the calculated average value with the worker ID and storing the calculated average value in the storage unit is executed by the computer. Corpus creation program.
(Appendix 6)
With reference to the storage unit, the path of the most supported series of utterance data in the dialogue is identified based on the parent-child relationship of the tree structure of the utterance data for each dialogue and the utterance score.
For each pass, the average value associated with the worker ID of the worker who created the speech data is acquired from the storage unit and totaled, and the geometric mean is calculated using the total value and the total number of passes. The dialogue corpus creation program according to Appendix 5, which causes the computer to execute a process of associating a calculated dialogue ID for specifying a dialogue with a calculated geometric mean and storing it in the storage unit.
(Appendix 7)
The dialogue corpus creation program according to Appendix 6, wherein the computer is made to execute a process of arranging utterance data in order according to a parent-child relationship on the path and creating dialogue data with a score associated with the dialogue ID together with the geometric mean.
(Appendix 8)
On the computer
In each of the plurality of dialogues, the ratio of the extracted worker IDs present in the record of the worker IDs of the workers who participated in the dialogue is calculated.
Addendum 1 to 7, characterized in that the worker ID taken out at the end of the highest percentage of dialogue is added, and the worker specified by the added worker ID is assigned to the highest percentage of dialogue. The dialogue corpus creation program described in any one of the above.
(Appendix 9)
On the computer
The dialogue corpus creation program according to Appendix 8, wherein when a plurality of the dialogues having the highest ratio exist, the dialogue having the largest number of occurrences of the worker ID extracted is selected from the plurality of dialogues.
(Appendix 10)
On the computer
When a plurality of the highest ratio dialogues exist and the number of occurrences of the extracted worker IDs is the same, one dialogue is randomly selected from the plurality of dialogues.
The dialogue corpus creation program described in Appendix 8 characterized by the above.
(Appendix 11)
From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. Take out for each utterance
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. According to the worker allocation procedure based on, select the dialogue to which the worker who creates the utterance data is assigned, and
A method of creating a dialogue corpus in which a computer performs a process of obtaining dialogue data in which the dialogue is completed by the plurality of workers.
(Appendix 12)
From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. The worker ID extraction section that retrieves each utterance,
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. A selection unit that selects the dialogue to which a worker who creates the utterance data is assigned according to a worker allocation procedure based on the above, and a utterance creation unit that causes the plurality of workers to create the utterance data and collects the created utterance data. Information processing device with.

3 Worker terminal 3a Dialogue participation request 3b Dialogue task presentation 6 User terminal 6a Dialogue data acquisition request 6b Dialogue data with score 7 User 9 Worker 41 Worker list 42 Dialogue task table 43 Speech table 44 Speech data DB
45 Worker score table 50 Initialization unit 51 Worker registration unit 52 Worker allocation unit 53 Dialogue task selection unit 54 Dialogue task generation unit 55 Speaking creation unit 56 Speaking information update unit 57 Dialogue data extraction unit 58 Worker score calculation unit 59 Dialogue score calculation unit 60 User I / F
100 server device

Claims (6)

From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. Take out for each utterance
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. According to the worker allocation procedure based on, select the dialogue to which the worker who creates the utterance data is assigned, and
A dialogue corpus creation program that causes a computer to execute a process of obtaining dialogue data that completes the dialogue by the plurality of workers. Displaying the dialogue information of the dialogue on the worker terminal of the worker to which the dialogue is assigned, checking when the content of the utterance immediately before being indicated by the dialogue information is inappropriate, and the input area of the utterance data. Display the creation screen with
When at least the utterance data and the presence / absence of the inappropriate check are received from the worker terminal, the received utterance data is managed by a tree structure having a parent-child relationship according to the progress of the dialogue, and the inappropriate check is performed. If there is, the received utterance data is branched from the parent's utterance data and recorded in the storage unit as a sibling relationship with the immediately preceding utterance data.
The interactive corpus creation program according to claim 1, wherein the computer executes a process of determining the inappropriateness of the immediately preceding utterance data according to the presence or absence of the inappropriateness check received together with the immediately preceding utterance data. In the tree structure, a support number is given to the received utterance data, and the support number is given.
When another utterance data having a sibling relationship exists with respect to the immediately preceding utterance data, the number of supports of the immediately preceding utterance data is increased or decreased depending on the presence or absence of the inappropriate check received together with the received utterance data. The interactive corpus creation program according to claim 2, wherein the computer executes the processing to be performed. The utterance score is calculated from the utterance data in which the number of support increases or decreases and the utterance data of the sibling relationship of the utterance data, and the ratio of the number of support of each of the utterance data to the total number of support is calculated and stored in the storage unit. The interactive corpus creation program according to claim 3, wherein the computer executes a memorizing process. From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. Take out for each utterance
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. According to the worker allocation procedure based on, select the dialogue to which the worker who creates the utterance data is assigned, and
A method of creating a dialogue corpus in which a computer performs a process of obtaining dialogue data in which the dialogue is completed by the plurality of workers. From the storage unit that stores the worker IDs of the workers who create utterance data representing one utterance based on the dialogue information related to the dialogue between the two parties in the order in which the worker IDs of a plurality of workers can participate in the dialogue. The worker ID extraction section that retrieves each utterance,
With reference to the record of the worker ID of the worker who participated in the dialogue in chronological order for each of the plurality of dialogues managed by the storage unit, the past participation status of the worker with the extracted worker ID can be obtained. According to the worker allocation procedure based on the above, the selection unit that selects the dialogue to which the worker who creates the utterance data is assigned, and the utterance creation unit that causes the plurality of workers to create the utterance data and collects the created utterance data. Information processing device with.

Images