JP7400040B2 - Reinforcement learning agent for multidimensional dialogue act selection - Google Patents

Description

TECHNICAL FIELD Embodiments described herein generally relate to methods and devices for generating responses in interaction systems.

Dialogue systems, such as task-oriented speech dialogue systems (SDS), include systems that can continuously interact with humans to help them accomplish tasks through the use of speech. These tasks can include information retrieval, customer support, electronic commerce, physical environment control, and human-robot interaction.

Dialogue systems track what happens in an interaction by incorporating information about current and previous user inputs (i.e., user turns) and system outputs (i.e., system turns) into an "interaction state." Can contain components.

The action selection component of the dialogue system selects actions from a fixed set of actions based on the dialogue policy and the dialogue state.

Utterances within a dialogue often have multiple communication functions. For example, moving tasks forward, giving and eliciting feedback, and managing who has the right to speak (i.e., turn management). Speech using multiple communication functions is called multifunctional speech. The ISO 24617-2 annotation standard is designed to support multifunctional utterances and define an annotation scheme with multiple dimensions, each dimension dedicated to a different aspect of the interaction process.

Multidimensional interaction systems have been developed in which the decision-making process is driven by several interaction policies rather than one. This approach reflects the multifunctionality of human interaction behavior. In this case, each dialogue act (DA) agent has its own dialogue policy with a set of actions, and each dialogue act (DA) agent generates at most one dialogue act candidate.

Multifunctional speech requires a combination of dialogic acts from several different dimensions. However, some dimensions cannot be considered sufficiently independent. For example, several interaction acts from different dimensions may conflict with each other such that only one of them should be output. Therefore, an evaluation agent (EA) is required to select the final candidate combination of interaction acts.

The arrangement will be better understood and appreciated from the following detailed description, which is given by way of example only and taken in conjunction with the drawings.

FIG. 1 is a diagram illustrating an example dialogue system. FIG. 2 is a diagram illustrating a multidimensional action selection component according to an example. FIG. 1 illustrates an example implementation of a reinforcement learning (RL) machine learning model. FIG. 2 illustrates a second example implementation of a reinforcement learning (RL) machine learning model. FIG. 2 is a diagram illustrating an implementation of an evaluation agent 205 according to an example. The figure which shows the representation of the vector of action probability by an example. FIG. 3 is a diagram illustrating an implementation of an evaluation agent 205 according to a second example. FIG. 2 is a diagram illustrating a method for generating system utterances according to an example. FIG. 2 is a diagram illustrating a method for training a multidimensional action selection component according to an example. FIG. 3 is a diagram illustrating a comparison of interaction success rates for multidimensional systems using hard-coded evaluation agents and evaluation agents, according to examples described herein. FIG. 2 is a diagram illustrating a method of transfer learning using multidimensional action selection components, according to an example. FIG. 7 is a diagram illustrating a method of transfer learning using a multidimensional action selection component according to a second example. 1 is a schematic diagram of hardware that can be used to implement methods according to embodiments; FIG.

According to a first aspect, a computer-implemented method is provided for generating a response in an interaction system, the interaction system being for interacting with a user. The method includes the steps of: receiving an utterance from a user; updating a dialog state based on the utterance; generating information identifying the dialog state; selecting a first action candidate from a first set of action candidates using the learning model and information identifying an interaction state, the first set of action candidates having a first interaction dimension; associated selecting and selecting, by a second interaction act agent, a second action candidate from the second set of action candidates using the second machine learning model and information identifying the interaction state. a second set of action candidates is associated with a second interaction dimension; selecting an output behavior based on the output behavior, the output behavior comprising a combination of the first behavior candidate and/or the second behavior candidate; and generating a system response.

In one embodiment, the interaction system includes a microphone for receiving vocalizations.

In one embodiment, the interaction system includes a speaker, and the method further includes outputting the system response via the speaker.

In one embodiment, the output behavior comprises a first candidate behavior and a second candidate behavior.

In one embodiment, the output behavior comprises a first candidate behavior or a second candidate behavior.

In one embodiment, the output behavior comprises neither the first candidate behavior nor the second candidate behavior, and preferably the output behavior indicates that no response can be generated.

In one embodiment, the first machine learning model, the second machine learning model, and the third machine learning model are trained independently using a single, same, reward signal.

In one embodiment, the first machine learning model enforces the first interaction policy, the second machine learning model enforces the second interaction policy, and the third machine learning model enforces the third interaction policy. implement.

In one embodiment, the natural language generator is a template-based natural language generator.

In one embodiment, the interaction state comprises a data structure comprising items that are mentioned between the system and the user during the interaction.

In one embodiment, each interaction dimension is associated with a different aspect of the interaction process and associated with multiple actions in response to said aspects of the interaction process.

In one embodiment, the interaction dimension comprises a dimension defined according to the ISO 25617-2 annotation standard.

In one embodiment, the information identifying the interaction state is the interaction state.

In one embodiment, the information identifying the interaction state is a set of features extracted from the interaction state.

In one embodiment, generating the information identifying the interaction state includes: generating a first instance of the information identifying the interaction state; and generating a second instance of the information identifying the interaction state. , the first action candidate is selected based on the first instance of the information identifying the interaction state, and the second action candidate is selected based on the second instance of the information identifying the interaction state. Ru.

In one embodiment, the first instance of the information identifying the interaction state and the second instance of the information identifying the interaction state comprise different sets of interaction state characteristics.

In one embodiment, the system response is generated by a natural language generator.

In one embodiment, the system response is a vocalization.

In one embodiment, selecting the output action is associated with, for each combination of the first action candidate and/or the second action candidate, selecting the respective combination as a system response if in an interactive state. and selecting an output behavior based on the vector of values.

In one embodiment, the vector of values is generated using a function parameterized by a first set of trainable weights, the input to the function is information identifying the interaction state, and the output of the function is the estimated cumulative reward for each combination.

In one embodiment, the function is a linear function.

In one embodiment, the third machine learning model comprises a linear layer configured to implement a linear function.

In one embodiment, the linear layer is implemented using an (artificial) neural network.

In one embodiment, selecting the output behavior based on the vector of values includes converting the vector of values into a probability distribution to generate a vector of behavior probabilities, and selecting the output behavior based on the vector of behavior probabilities. and selecting.

In one embodiment, the vector of values is transformed into a probability distribution by applying a softmax function to the vector of values.

In one embodiment, the probability distribution is a normalized probability distribution.

In one embodiment, transforming a vector of values into a probability distribution to generate a vector of behavioral probabilities comprises transforming the vector of values into a vector of transformed behavioral values, The vector of behavior values is masked by transforming it to have a value greater than or equal to zero and setting the first behavior value in the transformed vector of behavior values equal to zero based on a predetermined criterion. forming a vector of masked transformed behavior values, and normalizing the vector of masked transformed behavior values to form a vector of behavior probabilities.

In one embodiment, the vector of transformed behavioral values only includes zero or more values.

In one embodiment, the vector of masked probabilities is formed by a masking layer configured to take the vector of transformed behavioral values as input and provide the vector of masked probabilities as output.

In one embodiment, the predetermined criteria comprises at least one of a list of meaningless combinations and/or a list of combinations that are not supported by a downstream component, such as a natural language generator.

In one embodiment, setting the first action value equal to zero disables combinations of the first action candidate and/or the second action candidate.

In one embodiment, the first interaction dimension is different than the second interaction dimension.

In one embodiment, the user is a human user.

In one embodiment, the interaction system comprises a microphone, and receiving speech from the user comprises generating a representation of the human speech received via the microphone.

In one embodiment, the dialogue state is updated based on utterances and previous instances of the dialogue state.

In one embodiment, the interaction system comprises a speaker, and the method further comprises outputting the system response via the speaker so that the system response can be heard by a user.

According to a second aspect, a computer-implemented method for training a dialogue system, comprising: generating a system response by the method described above; providing the system response to a user; and reinforcement learning. A method is provided comprising training a first machine learning model, a second machine learning model, and a third machine learning model using the method.

In one embodiment, the user is a simulated user.

In one embodiment, using reinforcement learning to train the first machine learning model, the second machine learning model, and the third machine learning model includes determining a cumulative reward; training a first machine learning model, a second machine learning model, and a third machine learning model based on the cumulative reward.

In one embodiment, the first machine learning model, the second machine learning model, and the third machine learning model are trained using Monte Carlo control with linear value function approximation.

In one embodiment, the first machine learning model, the second machine learning model, and the third machine learning model are trained separately.

In one embodiment, the cumulative reward is the sum of the individual rewards obtained between providing the system response and terminating the interaction.

In one embodiment, the first machine learning model calculates a first estimated cumulative reward for the first output behavior candidate based on a first function parameterized by a first set of trainable weights. and training the first machine learning model based on the cumulative reward comprises: determining a difference between the cumulative reward and the first estimated cumulative reward; updating a first set of possible weights.

In one embodiment, the difference is the mean squared error.

In one embodiment, updating the first set of trainable weights includes determining a weight change for each weight in the first set of weights based on the difference using gradient descent. Equipped with

In one embodiment, the first set of trainable weights is updated to minimize the difference between the cumulative reward (ie, the "true" cumulative reward) and the first estimated cumulative reward.

In one embodiment, the third machine learning model is configured to estimate a second estimated cumulative reward for the output behavior based on a second function parameterized by a second set of trainable weights. and training a third machine learning model based on the cumulative reward comprises determining the difference between the cumulative reward and the second estimated cumulative reward and determining the trainable weights based on this difference. and updating the second set.

In one embodiment, the cumulative reward is based on a total reward that comprises a first reward generated to provide a system response.

In one embodiment, the first reward comprises the sum of a first component and a second component, the first component providing a penalty to discourage long interactions, and the second component provides rewards for completing predefined goals.

In one embodiment, a method includes receiving a second utterance in response to providing a system response and generating a second system response by a method for generating a response in an interaction system. , the utterance in the method for producing a response in a dialog system is a second utterance, and the system response produced by the method for producing a response in a dialog system is a second system response. , further comprising: generating a second reward based on the second system response; and calculating a cumulative reward based on a sum of the first reward and the second reward.

In one embodiment, the sum of the first reward and the second reward is a discounted sum such that the value of the second reward in the sum is reduced by a discount factor.

In one embodiment, the interaction system includes a first interaction policy implemented by a first machine learning model, a second interaction policy implemented by a second machine learning model, and a third interaction policy implemented by a third machine learning model. a first hyperparameter configured to control the amount of exploration of a third interaction policy performed, the first hyperparameter having a first value during the first interaction; The interaction comprises an utterance and a system response, the method determining whether the first interaction is complete and, in response to determining that the first interaction is complete, a state-action space. modifying the first hyperparameter such that the amount of exploration of the first interaction policy, the second interaction policy, and the third interaction policy within the interaction policy is smaller than during the first interaction; and retraining the dialog system using the dialog.

In one embodiment, each of the first machine learning model, the second machine learning model, and the third machine learning model is configured to transform a vector of behavioral values into a transformed vector of behavioral values. a conversion layer controlled according to a temperature hyperparameter, the first hyperparameter modification comprising decreasing a value of the temperature hyperparameter closer to unity.

In one embodiment, the output of the first machine learning model is represented using one-hot encoding.

In one embodiment, the output of the first machine learning model comprises a first vector comprising a plurality of entries, each entry being associated with a different behavior from the first set of actions. , the entry associated with the first action candidate includes information indicating that the action candidate has been selected by the first machine learning model, and the other entries in the plurality of entries are associated with the other entries. The action candidate includes information indicating that it has not been selected by the first machine learning model.

According to a third aspect, there is provided a device for generating a response in a dialog system, the dialog system being for interacting with a user, and the device receiving utterances from the user. , updating a dialogue state based on the utterance, generating information identifying the dialogue state, and using the first machine learning model and the information identifying the dialogue state by the first dialogue act agent. selecting a first action candidate from a first set of action candidates, the first set of action candidates being associated with a first interaction dimension; selecting, by the action agent, a second action candidate from the second set of action candidates using the second machine learning model and information identifying the interaction state, the action agent selecting a second action candidate from the second set of action candidates; the set is associated with the second interaction dimension; and selecting, by the evaluation agent, an output action using the third machine learning model, the first action candidate, and the second action candidate. the output behavior comprises a combination of the first candidate behavior and/or the second candidate behavior, the output behavior being configured to select and generating a system response based on the output behavior; device is provided.

In one embodiment, when selecting the output action, the device is configured to, for each combination of the first action candidate and/or the second action candidate, select the respective combination as a system response if in the interactive state. The method is further configured to generate a vector of values with an associated estimated cumulative reward indicator and select an output behavior based on the vector of values.

In one embodiment, the device is further configured to generate a vector of values using a function parameterized by the first set of trainable weights, the input to the function identifying an interaction state. information, and the output of the function is the estimated cumulative reward for each combination.

In one embodiment, the function is a linear function.

In one embodiment, when selecting an output action based on the vector of values, the device converts the vector of values into a probability distribution to generate a vector of action probabilities, and selects the output action based on the vector of action probabilities. Further configured to select.

In one embodiment, when converting a vector of values, the device converts the vector of values into a vector of transformed behavioral values, the vector of transformed behavioral values having a value of zero or more. forming a masked vector of transformed behavior values by converting the vector of transformed behavior values to equal to zero, and setting the first behavior value in the vector of transformed behavior values equal to zero based on a predetermined criterion. and normalizing the vector of masked transformed behavioral values to form a vector of behavioral probabilities.

In one embodiment, the first interaction dimension is different than the second interaction dimension.

According to a fourth aspect, a training system for training a dialogue system, comprising a device for generating a response within the dialogue system discussed above, the training system comprising: a device for generating a response within the dialogue system discussed above; generate a system response using a device for generating a system response, provide the system response to a user, and use reinforcement learning to generate a first machine learning model, a second machine learning model, and a third machine learning model. A training system configured to train a machine learning model is provided.

In one embodiment, the training system uses reinforcement learning to determine a cumulative reward when training the first machine learning model, the second machine learning model, and the third machine learning model; The device is further configured to train a first machine learning model, a second machine learning model, and a third machine learning model based on the cumulative reward.

In one embodiment, the first machine learning model calculates a first estimated cumulative reward for the first output behavior candidate based on a first function parameterized by a first set of trainable weights. and the training system is configured to, when training the first machine learning model based on the cumulative reward, determine a difference between the cumulative reward and the first estimated cumulative reward, and based on this difference, Further configured to update the first set of trainable weights.

In one embodiment, the third machine learning model is configured to estimate a second estimated cumulative reward for the output behavior based on a second function parameterized by a second set of trainable weights. and the training system is configured to, when training a third machine learning model based on the cumulative reward, determine a difference between the cumulative reward and the second estimated cumulative reward, and based on this difference, trainable weights. further configured to update the second set of .

In one embodiment, the cumulative reward is based on a total reward that comprises a first reward generated to provide a system response.

In one embodiment, the training system includes receiving a second utterance in response to providing a system response and generating a second utterance using a device for generating a response within the interaction system discussed above. generating a system response, the utterance received by the device being a second utterance, and the system response generated by the device being a second system response; and calculating a cumulative reward based on the sum of the first reward and the second reward.

In one embodiment, the interaction system includes a first interaction policy implemented by a first machine learning model, a second interaction policy implemented by a second machine learning model, and a third interaction policy implemented by a third machine learning model. a first hyperparameter configured to control the amount of exploration of a third interaction policy performed, the first hyperparameter having a first value during the first interaction; The interaction comprises an utterance and a system response, the training system determining whether the first interaction is complete, and in response to determining that the first interaction is complete, the training system Modify the first hyperparameter and use the second interaction such that the amount of exploration of the first interaction policy, second interaction policy, and third interaction policy is smaller than during the first interaction. and is further configured to retrain the interaction system.

According to a fifth aspect, a non-transitory computer-readable medium comprising computer program instructions suitable for execution by a processor, the instructions, when executed by the processor, causing the processor to perform the method discussed above. A non-transitory computer-readable medium is provided that is configured to perform a non-transitory computer-readable medium.

FIG. 1 shows an example dialogue system. The dialogue system includes a dialogue act tagger component 100, a dialogue state tracking component 101, an action selection component 102, and a natural language generator 103.

The interaction act tagger component 100 is configured to obtain information identifying user input, optionally audio user input (eg, received via a microphone). Alternatively, the user input may be in the form of text.

In examples where voice user input is received via a microphone, automatic speech recognition (ASR) is performed by an automatic speech recognition component (not shown) to recognize the user's utterances and convert them to text. be exposed. In one example, the automatic speech recognition component is implemented using the Google® web speech API.

The input to the interaction act tagger component 100 is user input text. The interaction act tagger component 100 is configured to identify one or more interaction act tags based on user input text and output the identified interaction act tags to the interaction state tracking component 101. Dialogue act tagger component 100 is also configured to output user input text to dialogue state tracking component 101 . The interaction act tagger component is used to recognize domain-independent interaction acts, such as greeting and thanking, required by the action selection component 102. Optionally, interaction act tagger component 100 is a general purpose interaction act tagger.

Dialogue state tracking component 101 is configured to generate and update dialogue state after each received utterance. In one example, dialogue state tracking component 101 takes as input user text and recognized dialogue act tags (both output by dialogue act tagger 100) and generates an updated dialogue state. , which contains recognized interaction act tags. In one example, the dialogue state tracker component 101 recognizes updates to the user's goals if the user utterances (as conveyed by the dialogue act tagger component 100) and the current dialogue state (as remembered from previous turns). configured to do so.

In one example, the representation of the interaction state takes the form of a data structure comprising items mentioned during the interaction between the system and the user. In a further example, the representation of interaction state 101 comprises structured information about the interaction so far, such as user goals.

In another example, a dialog state stores information by providing slots. In the specific example where the dialogue system is used within the restaurant search domain, the dialogue state is represented by three slots: food, region, and price range (e.g., cheap, medium, or expensive). target” constraint. At the beginning of the interaction, each slot is empty, and the slots are populated by the interaction state tracking component 101 as more information is collected from the user.

In another example, the interaction state tracking component 101 behavioral state update model for interaction state tracking is described in "Svetlana Stoyanchev, Simon Keizer, and RamaDoddipatla, "Action state update approach to dialog management”, ICASSP2021 -2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021, pages 7398-7402.

More generally, the dialog state maintained by the dialog state tracking component 101 includes information identifying the system's beliefs about user goals and, optionally, an interaction history including previously discussed items. Be prepared.

Dialogue state tracking component 101 is configured to output a representation of the dialogue state to action selection component 102 .

The behavior selection component 102 is configured to receive a representation of the interaction state and generate information indicative of a system response based on the interaction policy of the behavior selection component and the received representation of the interaction state. In one example, behavior selection component 102 is implemented according to the example multidimensional behavior selection component described below. The action selection component 102 is configured to output information identifying interaction acts that can form a system response. As discussed in more detail below, system responses may take the form of multifunctional utterances.

Natural language generator 103 is configured to receive the output information of action selection component 102 (i.e., information identifying interaction acts that can form a system response) and configured to generate a system response in the form of an utterance. It is composed of

Optionally, natural language generator 103 is a template-based natural language generator (NLG). In one example, the template-based natural language generator 103 is described in Reiter, E., & Dale, R. (2000), Building Natural Language Generation Systems (Studies in Natural Language Processing). g), Cambridge: Cambridge University Press. It will be implemented as follows.

In one example, the natural language system response (i.e., the generated utterance) is output through a speaker in the form of an audio waveform so that the system response can be received by a human user. Optionally, when the output of the natural language generator 103 is spoken natural language in textual form, the output of the natural language generator 103 is optionally output from Google before being output via the speaker. Text is converted to speech using the web speech API.

In the example shown in FIG. 1, the interaction system is used in the restaurant search domain and receives the input utterance, "I'm looking for a cheap restaurant." Using the functionality of the interaction state tracking component 101, the action selection component 102, and the natural language generator 103, the interaction system provides a response to the user that recommends a restaurant (e.g., the interaction system provides a response that recommends a restaurant such as "Prezzo is (Provides a system response of ``Popular Italian Restaurant.'')

Although the dialogue system is discussed above in relation to the restaurant search domain, it is emphasized that the dialogue system is adaptable to many uses. One possible use is information retrieval. However, other uses are also possible. For example, information gathering, troubleshooting, customer support, electronic commerce, physical environment control, and human-robot interaction.

FIG. 2 illustrates a multidimensional action selection component according to an example. The multidimensional action selection component shown in FIG. 2 comprises information (eg, received from interaction state tracking component 101) that identifies interaction state 201. As discussed above, information identifying the interaction state 201 comprises the system's thoughts about user goals and interaction history, including previously discussed items (eg, previous user and system utterances).

In a further example, the information identifying interaction state 201 is a set of features extracted from the full interaction state. In this context, a feature is a property or characteristic of a full interaction state.

In a further example, the information identifying the interaction state 201 comprises a plurality of feature sets (e.g., a first feature set, a second feature set, and a third feature set), each set of features being (as discussed in more detail in ) are associated with different dialogue act (DA) agents. Optionally, feature sets in the plurality of feature sets may overlap. That is, two feature sets in the plurality of feature sets may include a common feature.

The multidimensional action selection component further comprises a first dialogue act (DA) agent 202, a second dialogue act (DA) agent 203, and a third dialogue act (DA) agent 204.

Each of the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, and the third dialogue act (DA) agent 204 has a respective dialogue act (DA) agent as discussed above. ) configured to access information identifying an interaction state 201 that includes a set of features extracted from a full interaction state that is uniquely associated with an agent.

Each of the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, and the third dialogue act (DA) agent 204 uses the dialogue act classification method described in the ISO 24617-2 annotation standard. It can also be associated with different dimensions. A short introduction to the standard is provided below. However, a more comprehensive explanation is found in Bunt, Harry & Petukhova, Volha & Traum, David & Alexandersson, January (2017), Dialogue Act Annotation with the ISO 24617-2, which is incorporated herein by reference. Standard. 10.1007/978- 3-319-42816-1_6" is also available.

The ISO 24617-2 annotation standard is designed for the annotation of audio, written, and multimodal interactions with information about the interaction acts that make up the interaction. A dialogue act is defined as a communicative activity of dialogue participants that is interpreted as having a certain communicative function and semantic content. As discussed above, vocalizations during interaction often have multiple communication functions. To capture different aspects of the interaction process, the ISO 24617-2 annotation standard defines the concept of "dimensions".

Dimensions specified in the ISO 24617-2 annotation standard include, but are not limited to, a task dimension, an automatic feedback dimension, and a social responsibility management (SOM) dimension. Other dimensions include turn-taking and task management.

The task dimension concerns the interaction act that moves forward the task or activity that motivates the interaction. For example, when a multidimensional behavioral selection component is used within a restaurant search domain, the task dimension is responsible for recommending restaurants. More generally, the task dimension pertains to behaviors focused on the underlying task or activity (e.g., information navigation, tutoring, negotiation, etc.).

The automatic feedback dimension concerns interactional acts that provide or elicit information about the current speaker's processing and the addressee's previous utterances (e.g., asking the user to repeat/reword previous input after a processing problem). do. The automatic feedback dimension may be used to indicate misunderstood user input when the speech recognition component (upstream from the multidimensional action selection component) returns no results. An example output generated from a behavior in the automatic feedback dimension may be "I didn't quite understand. Could you repeat that?" Actions in the automatic feedback dimension may also be used to provide clear feedback in the form of explicit or implicit confirmation when the system is uncertain about the user's input. An example output in this case may be "Did you say it was expensive?"

The Social Responsibility Management (SOM) dimension concerns dialogic acts for dealing with social customs such as greetings, self-introductions, apologies, and gratitude.

In the system of Figure 2, each dimension is associated with an action set of possible summary actions (referred to throughout as actions). Summary actions are underspecified interaction acts that may require additional information from the interaction state to be mapped to a fully specified interaction act. For example, a summary action of "REQUEST" is mapped to an interaction act of type "request" by determining which slot is requested, eg, request(area), based on the interaction state. Similarly, the summary action "OFFER" is mapped to a full interaction action by determining which venue should be offered. The mapping of summary actions to dialogue actions is performed by a dialogue action (DA) agent. In one example, summary actions are mapped to interaction actions using fixed heuristics (eg, using a predetermined mapping). Dialogue acts (DA) Dialogue acts generated by agents are used by natural language generator 103 to generate system utterances.

As discussed above, each dialogue act (DA) agent generates a dialogue act based on the selected action, optionally using additional information from the dialogue state. In some cases, the policy of a dialogue act (DA) agent selects an action that corresponds directly to the dialogue act in some cases (without additional information required, e.g. ACCEPT_THANKING). In other cases, a dialogue act (DA) agent selects a summary action that requires additional information to form a complete dialogue act (eg, REQUEST).

Advantageously, using summary actions instead of fully specified dialogue acts reduces the size of the action set used by a dialogue act (DA) agent's policy, and therefore Improving the ease and speed of training agents.

In one example, the interaction acts performed by the system are based at least in part on some standard, such as ISO 24617-2. However, these standards tend to focus only on interaction act types/communication functions and aim to be domain independent. As a result, in a further example, the system implements interaction acts that are specific to the domain in which the system is used. In one example, the interaction acts performed by the system require semantic content, optionally expressed as a list of slot-value pairs (food=Chinese food, region=city center, etc.). In one example, the range of slot-value pairs is described in a domain ontology. The specific interaction acts performed by the system are determined by the fact that several interaction system components are part of this set of interaction acts, for example components for natural language understanding and generation, and state tracking and action selection (policy) components. This is a problem for system design as it must be supported.

In a particular example, the task dimension is associated with an action set comprising an "OFFER" action, an "ANSWER" action, a "REQUEST" action, and a "NONE" action, and the automatic feedback dimension is "IMPL-CONFIRM". associated with an action set comprising an action, an "EXPL-CONFIRM" action, an "AUTO_NEGATIVE" action, and a "NONE" action, and the social responsibility management (SOM) dimension is "ACCEPT_THANKING" and "RETURN_GOODBYE" It is associated with an action set that includes "NONE".

In this example, the "OFFER" behavior may generate an output utterance that comprises an offer to satisfy the goal (e.g., if a multidimensional behavior selection component is used within a restaurant search domain, the "OFFER" behavior can result in the output utterance "How about Rice Boat?" where "Rice Boat" is the name of the restaurant). The "ANSWER" action may generate an output utterance that comprises an answer to a question (eg, "Rice Boat's address is..."). A "REQUEST" action may generate an output utterance requesting more information (eg, "What price range are you considering?"). The "IMPL-CONFIRM" action is an output utterance that implicitly confirms that the system understands the input (e.g., "I understand. Indian food...", that the system understands that the input request pertains to Indian food). (which implicitly confirms that The "EXPL-CONFIRM" action is used to generate an output utterance that explicitly confirms that the system understands the user input (for example, "I would like Indian food, is that correct?"). obtain. The "AUTO_NEGATIVE" behavior may be used to generate an output utterance (e.g., "I didn't hear you clearly. Could you repeat that?") that conveys that the system is unsure about the user's input. The "ACCEPT_THANKING" and "RETURN_GOODBYE" actions are self-explanatory. An example utterance based on the "ACCEPT_THANKING" behavior may include "Welcome" and an example utterance based on "RETURN_GOODBYE" may include "Have a nice day." In one example, the "NONE" action causes the selection of a predetermined interaction act (eg, in one example, the "NONE" action causes the interaction system to select the "autoNegative" interaction act).

In the particular example above, the action set comprises a particular action. However, for the avoidance of any doubt, it is emphasized that each action set may comprise other actions not explicitly discussed above.

In the multidimensional action selection component of FIG. 2, a first dialogue act (DA) agent 202 is associated with the "task" dimension, a second dialogue act (DA) agent 203 is associated with the "automatic feedback" dimension, and A third dialogue act (DA) agent 204 is associated with the "Social Responsibility Management (SOM)" dimension. Advantageously, these three agents (corresponding to three dimensions) are considered the minimum required for a task-oriented multidimensional interaction system. Although three dialogue act (DA) agents are shown in FIG. 2, it is emphasized that any number greater than one dialogue act (DA) agent may be provided.

Consider the first dialogue act (DA) agent 202 in more detail. The first dialogue act (DA) agent 202 takes actions based on the information associated with the "task" dimension and identifying the dialogue state 201 and the dialogue policy being implemented by the first dialogue act (DA) agent 202. configured to generate candidates; A dialogue policy associated with a first dialogue act (DA) agent 202 is used to select candidate actions from a set of possible actions based on input information identifying a dialogue state 201.

As discussed above, the set of possible actions comprises multiple summary actions. Summary actions are poorly specified interaction acts that may require additional information from the interaction state to be mapped to a fully specified interaction act. In one example, actions are mapped to full interaction acts using additional information from interaction states. In a further example, actions within the set of possible actions correspond to communication functions from the ISO24617-2 annotation standard.

The first interaction act (DA) agent 202 comprises a reinforcement learning (RL) machine learning model, or in other words, a machine learning model trained using reinforcement learning (RL). As is known in the art, Reinforcement Learning (RL) is a machine learning model's policy (i.e., when a given state is This is one type of machine learning in which the behavior (which actions are selected) is learned.

Training each of the machine learning models within a dialogue act (DA) agent is discussed in more detail below. However, briefly, during training, a first dialogue act (DA) agent 202 starts in state S (corresponding to information identifying dialogue state 201) and is trained on a particular dimension (i.e., the "task" dimension). ) is configured to learn an action value function that estimates the cumulative reward obtained when selecting an output action from a set of possible actions for As discussed in more detail below, the reinforcement learning (RL) machine learning model of the first dialogue act (DA) agent 202 better estimates the cumulative reward associated with taking an action from a given state. In order to do so, use a reward signal (e.g., generated based on a (simulated) user interaction) to train the parameters of the behavior value function.

During inference, a first interaction act (DA) agent 202 is configured to generate output action candidates from a set of possible actions associated with a "task dimension" based on the input interaction state 201 and the learned interaction policy. It is composed of In particular, the machine learning model of the first dialogue act (DA) agent 202 is associated with selecting each of the actions in the action set if the current state S (corresponding to information identifying the dialogue state 201) is and configured to determine a cumulative reward. The machine learning model is further configured to select a (single) action candidate for the task dimension based on the estimated cumulative reward.

An example implementation of a first reinforcement learning (RL) machine learning model (also referred to as a first machine learning model) is discussed in more detail in connection with FIG. 3A.

FIG. 3A shows an example implementation of a reinforcement learning (RL) machine learning model.

In one example, the reinforcement learning (RL) machine learning model for the first interaction act (DA) agent 202 may include several stages. In one example, the machine learning model for the first interaction act (DA) agent 202 comprises a linear layer 302 of trainable parameters.

Input to the linear layer 302 includes information identifying the dialog state 301 (e.g., a set of features representing a full dialog state, optionally a full dialog state that is uniquely associated with the first dialog act (DA) agent 202). a set of features).

Linear layer 302 is configured to generate an expected long-term cumulative reward that is associated with selecting an action from the action set given the current state (ie, given the current interaction state). In this case, linear layer 302 implements an action value function. In one example, the linear layer is implemented using an (artificial) neural network. In one example, the linear layer is a neural network with a single hidden layer. In a further example, the linear layer is implemented using multiple hidden layers.

The output of the linear layer 302 comprises a vector of values 303 (also referred to as a vector of behavior values), where each value represents the extent to which the behavior (from the set of behaviors associated with the relevant dimension) maximizes the long-term cumulative reward. shows. Optionally, the value indicates a long-term cumulative reward associated with taking each action.

As discussed in more detail below, during training of a reinforcement learning (RL) machine learning model, the reward signal is used to train the linear layer so that the expected return (i.e., cumulative reward) is closer to the actual return received. Used to update possible parameters.

In one example, the machine learning model of the first interaction act (DA) agent 202 also includes a transformation layer 304. The output of linear layer 302 (ie, the vector of behavior values 303) is the input to transformation layer 304.

The transformation layer 304 is configured to transform a vector of behavioral values, where each value takes a value between negative infinity and positive infinity, and a transformed behavioral value that takes any value greater than or equal to 0. 305 vectors. In one example, the transformation layer 304 calculates e ^yi for each input behavior value y _i in the vector of behavior values 303. In fact, all action values are converted to values greater than or equal to zero.

In one embodiment, the transformation layer 304 _includes a

where τ is the temperature hyperparameter. As known in the art, hyperparameters are parameters chosen to control the learning process. In particular, temperature hyperparameters are used to control the level of exploration during training of interaction policies implemented by reinforcement learning (RL) machine learning models. High temperatures are used during the early stages of training to force a more random selection of actions (from a set of possible actions associated with a dimension). However, during later stages of training, lower temperatures are used, forcing the estimates to follow more closely (ie, choosing high-value actions with higher probabilities). Therefore, the temperature hyperparameter controls exploration versus exploitation of interaction policies.

In the above equation, e is a standard exponential function and τ is a temperature hyperparameter that takes values between 0 and infinity. As the temperature hyperparameter approaches infinity, the probability distribution becomes a uniform distribution. As discussed in more detail below, when specifically discussing the training of multidimensional action selection components, the temperature hyperparameters may vary throughout training. In one example, the temperature hyperparameter is optionally fixed to 1 during inference.

The output 304 of the transformation layer is a vector with transformed behavior values 305. The transformed vector of behavior values 305 is input to the masking layer 306.

Masking layer 306 is coupled to the output of transform layer 304. Masking layer 306 disables behaviors in vector of transformed behavior values 305 (e.g., by setting associated values to zero) to generate a vector of masked transformed behavior values 307. configured to enable. In one example, masking within a first dialogue act (DA) agent because the required information is missing in a full interaction state or the output behavior is not supported by a downstream component (e.g., natural language generator 103) Layer 306 disables an action if, for example, the selected summary action cannot be mapped to a full interaction action. The output of masking layer 306 is coupled to the input of normalization layer 308.

Normalization layer 308 is configured to receive a vector of masked transformed behavior values 307. The normalization layer 308 is configured to normalize the values in the vector of masked transformed behavioral values 307 to generate a vector of behavioral probabilities whose sum of probabilities is equal to one. In other words, the masked transformed behavior values 307 (which can take any value greater than or equal to zero) are normalized to a probability distribution where the masked behavior has probability zero.

In one example, normalizing the values in the vector of masked transformed behavior values 307 includes converting each behavior value in the vector of masked transformed behavior values 307 to the masked transformed behavior value 307 vectors. The output of normalization layer 308 is a vector of action probabilities.

The first interaction act (DA) agent 202 is further configured to select output action candidates from the set of actions based on the vector of action probabilities.

In one example, the output of the first dialogue act (DA) agent 202 comprises a one-hot expression indicating a selected action from a set of possible actions (e.g., a binary "1" indicates a selected action). , where a binary '0' indicates an unselected action). Actions are selected based on the values in the vector of action probabilities (e.g., by selecting the action associated with the highest probability, or by sampling actions based on a probability distribution within the vector of action probabilities). It's okay to be.

The example discussed in connection with FIG. 3A shows a machine learning model that includes a masking layer 306. However, it is emphasized that masking layer 306 is optional. In this case, the vector of transformed behavior values 305 is input to the normalization layer 308. Optionally, when no masking layer is present, the functions of transform layer 304 and normalization layer 308 can be combined in a softmax layer.

As is known in the art, a softmax function is a function that reduces a vector of K real values to a vector of K real values that sum to 1. Input values can be positive, negative, zero, or greater than 1, but softmax turns them into values between 0 and 1 so that they can be interpreted as probabilities. In this case, the softmax layer is configured to convert behavioral values indicative of long-term cumulative rewards (i.e., a vector of behavioral values 303) into a normalized probability distribution. The output of the softmax layer comprises a vector of action probabilities, each probability associated with a different action within the set of possible actions, and each probability expressing the probability that a given action maximizes the long-term cumulative reward. show.

In one example, the softmax layer is a softmax type

, where σ is a softmax function, y is the input vector (i.e., a vector of action values 303), n is the magnitude of the input vector, and e is a standard exponential function. . In one example, temperature hyperparameters are used as discussed above, and in this case the softmax layer is

where τ is a temperature hyperparameter that takes values between 0 and infinity.

In a further example, if masking layer 306 is present, the functions of transformation layer 304 (ie, transforming behavioral values to a number greater than or equal to zero) and masking layer 306 are combined within one layer.

FIG. 3B shows a second example implementation of a reinforcement learning (RL) machine learning model. In FIG. 3B, the same reference numbers as in FIG. 3A are used to indicate the same components. FIG. 3B further includes a modified masking layer 350. Modified masking layer 350 is configured to receive a vector of action values 303 comprising an estimated cumulative reward Q(s,a) for performing action a in state s. First, exp(Q(s,a)/τ) values are calculated for all behaviors (where τ is a temperature hyperparameter and “exp” is an exponential function). Then a mask (mask(s,a)) is applied (i.e. mask(s,a)*exp(Q(s,a)/τ)) and some of these values are set to zero. , thereby generating a second vector of action values 352. Finally, the resulting values (i.e., the values in the second vector of behavior values 352) are distributed by the normalization layer 308 to a probability distribution (where the masked behavior has a probability of zero), as discussed above. ).

In the example of FIG. dimension, with an indication that the system response is selected from a set of possible actions. In other words, the output of the first dialogue act (DA) agent 202 comprises an indicator that the action for the "task" dimension is an "OFFER" summary action.

In addition to outputting an indicator of the selected action candidate, the first interaction act (DA) agent 202 is also configured to output an interaction action candidate associated with the selected action candidate. In one example, a first dialogue act (DA) agent 202 generates a dialogue act candidate based on information identifying the selected action candidate and the dialogue state 201.

As discussed above, the multidimensional dialogue system of FIG. 1 also includes a second dialogue act (DA) agent 103 and a third dialogue act (DA) agent 104.

The second dialogue act (DA) agent 103 and the third dialogue act (DA) agent 104 are similar reinforcement learning (RL) machines discussed above in relation to the first dialogue act (DA) agent 102. Equipped with a learning model. However, since each dialogue act (DA) agent is associated with a different dimension of the ISO 24617-2 annotation standard and therefore selects action candidates from a different set of possible actions, actions may be selected from different sets of actions. will be understood. Moreover, if each dialogue act (DA) agent uses a different feature set extracted from the dialogue state, the information identifying the dialogue state 101 for each dialogue act (DA) agent is a different feature set (optional). It will be appreciated that the selection may have overlapping characteristics.

Consider the second dialogue act (DA) agent 203 in more detail.

Training the second dialogue act (DA) agent 203 starts with state S (corresponding to information identifying dialogue state 201) and starts with a possible The method comprises learning an action value function that estimates the cumulative reward obtained when selecting an output action from a set of actions. As discussed in more detail below, the reinforcement learning (RL) machine learning model of the second dialogic act (DA) agent 203 uses a reward signal (i.e., the same reward signal as the first dialogic act (DA) agent 202). ) to train the parameters of the linear layer that implements the behavior value function. Although training the second interaction act (DA) agent 203 is based on a common reward signal, it is emphasized that the training is independent.

During inference, a second dialogue act (DA) agent 203 selects an output action from a set of possible actions associated with the "automatic feedback" dimension based on input information identifying the dialogue state 201 and the learned dialogue policy. configured to generate candidates; In particular, the machine learning model of the second dialogue act (DA) agent 203 is associated with selecting each of the actions in the action if the current state S (corresponding to information identifying the dialogue state 201) configured to determine cumulative compensation; The machine learning model is further configured to select a (single) action candidate for the "automatic feedback" dimension based on the estimated cumulative reward.

In the example of FIG. In the "Feedback" dimension, it is provided with an indicator that it has been selected as a system response. The summary behavior selected by the second dialogue act (DA) 203 agent is output to the evaluation agent 205, which determines whether this candidate behavior becomes final. As discussed above, the second dialogue act (DA) agent 202 performs a dialogue action based on the action candidate, which in the example of FIG. It is also configured to generate and output candidates.

The third dialogue act (DA) agent 204 is configured in a manner similar to the first dialogue act (DA) agent 202 and the second dialogue act (DA) agent 203, but instead, it is management (SOM)” dimension.

In the example of FIG. 2, the output of the third dialogue act (DA) agent 204 is that the "ACCEPT_THANKING" action is in response to information identifying the dialogue state 201 in the "Social Responsibility Management (SOM)" dimension. Provides an indicator that it has been selected as a system response. The summary behavior selected by the third dialogue act (DA) 204 agent is output to the evaluation agent 205, which determines whether this candidate behavior becomes final. The third dialogue act (DA) agent 204 is also configured to output a candidate dialogue act, which in the example of FIG. 2 is the "ACCEPT_THANKING" action, and information identifying the dialogue state 201. .

The multidimensional behavior selection component of FIG. 2 also includes an evaluation agent 205. The evaluation agent 205 is configured to evaluate the suggestions made by the dialogue act (DA) agents (i.e., the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, and the third dialogue act (DA) agent 204). configured to select a combination of actions to be performed. By selecting action candidates and providing a means to combine actions, the system of Figure 2 produces multifunctional system utterances (i.e., utterances that convey multiple interaction acts), thereby improving the naturalness and It is possible to improve both effectiveness.

In FIG. 2, information identifying the action candidates selected by each of the dialogue act (DA) agents is provided as input to evaluation agent 205. In FIG. In one example, each dialogue act (DA) agent is configured to generate at most one action candidate in every turn. Evaluation agent 205 is configured to generate a combination of these action candidates as system output.

FIG. 4A illustrates one implementation of evaluation agent 205 according to an example. Evaluation agent 205 comprises a fourth reinforcement learning (RL) machine learning model comprising a linear layer 402 , a transformation layer 404 , a masking layer 406 and a normalization layer 408 . The evaluation agent 205 evaluates each dialogue act (DA) agent (i.e., the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, and the third dialogue act (DA) agent 204). communicatively coupled to the output.

The output of each dialogue act (DA) agent uses one-hot encoding to determine which action (associated with the dimension of the dialogue act (DA) agent) from the set of possible actions is selected as a candidate action. has a vector shown. In this case, the magnitude of the output vector from a particular dialogue act (DA) agent corresponds to the number of actions in the set of possible actions for the associated dimension. Evaluation agent 205 is configured to concatenate the output vectors from each of the dialogue action (DA) agents to form input vector 401 .

In the example of FIG. 1 with three dialogue act (DA) agents, input vector 401 comprises three concatenated one-hot vectors. In this case, there are 2 ³ =8 possible interaction combinations that may be selected as the output of evaluation agent 205 .

The fourth reinforcement learning (RL) machine learning model for evaluation agent 205 has a similar structure as discussed above in connection with the machine learning model for dialogue act (DA) agents. In particular, a fourth reinforcement learning (RL) machine learning model comprises a linear layer of trainable parameters that transforms an input vector 401 into a vector of action values 403 that estimates a long-term cumulative reward for each possible combination of input interaction act candidates. 402. The vector of action values 403 is an input to the transformation layer 404.

The output of linear layer 402 (ie, comprising a vector of behavioral values 403) is input to transform layer 404. The conversion layer 404 converts the behavior values in the vector of behavior values 403, which can take any value between negative infinity and positive infinity, into a vector of transformed behavior values 405 having values greater than or equal to zero. It is configured as follows. The output of the transformation layer 404 (i.e., the vector of transformed behavioral values 405) is coupled to the input of the masking layer 406 (i.e., the masking layer 406 is configured to receive the vector of transformed behavioral values 405). ). Conversion layer 404 may perform the same conversions discussed in connection with conversion layer 304 of FIG. 3A.

Masking layer 406 is configured to disable some output behavior candidate combinations by setting their values to zero to generate a vector of masked transformed behavior values 407. In one example, the masking layer is configured to disable combinations of action candidates that result in a conflict. For example, it would be unreasonable to ask a question and then say goodbye. Therefore, some combinations of actions are determined to be nonsensical and therefore represent contradictions based on normal speech conventions. In one example, the combinations of actions that are disabled by the masking layer are predetermined and/or hard-coded.

In another example, the masking layer only detects a single interaction act or a combination of an ``IMPL-CONFIRM'' (of the ``Automatic Feedback'' dimension) with an ``OFFER'' behavior or a ``REQUEST'' behavior (of the ``Task'' dimension). The evaluation agent 205 is configured to ensure that permission is granted. The table below shows specific combinations allowed by masking layer 406 according to this specific example. An example of utterances that may be generated based on the combination of actions is also provided.

Although the table above shows specific combinations of actions from different dimensions, it is emphasized that other combinations of other actions may also be allowed. In a further example, the set of actions associated with the "Social Responsibility Management (SOM)" dimension includes the "APOLOGY" action, and the allowed combinations include "APOLOGY" + "AUTO_NEGATIVE", which is "Sorry". "No, but I didn't really understand."

The masking layer 406 of the evaluation agent 205 represents a sophisticated manner of limiting the range of interaction combinations that the agent can select to meet the input requirements of downstream components, such as the natural language generator 103, for example. Moreover, the masking layer can also speed up training of the component, since the multidimensional action selection component does not need to learn contradictory combinations (eg, meaningless combinations).

The output 406 of the masking layer is a vector of masked transformed behavior values 407. The output of masking layer 406 is coupled to normalization layer 408. A normalization layer 408 normalizes the behavior values present in the vector of masked transformed behavior values 407 such that the sum is equal to 1, thus presenting the probability of the behavior that maximizes the long-term cumulative reward. It is configured as follows. The output of normalization layer 408 is an output vector of action probabilities.

FIG. 4B shows an output vector representation of action probabilities according to an example. As discussed in connection with FIG. 2, the three dimensions for which interaction act candidates are generated include "task," "automatic feedback," and "social responsibility management (SOM)." FIG. 4B shows the probabilities associated with the combinations of action candidates for each of the respective dimensions. For example, Figure 4B shows that the combination of action candidates for the "task" dimension (i.e. "OFFER") and action candidates associated with the "automatic feedback" dimension (i.e. "IMPL-CONFIRM") maximizes long-term cumulative reward. indicates that it is associated with the highest probability of

FIG. 4B also shows an entry (ie, "---") associated with a blank state. This is the probability associated with not selecting a candidate action. This is similar to not generating a response. Finally, FIG. 4B also shows multiple behavior combinations (eg, "automatic feedback" + "SOM") that are associated with no probability (ie, "0") by masking layer 406.

As discussed in connection with FIG. 3A, masking layer 406 is optional. In this case, the functions of transform layer 404 and normalization layer 408 can be combined in a softmax layer that optionally implements temperature hyperparameters. Alternatively, if a masking layer is present, the functions of conversion layer 404 and masking layer 406 can be combined in a modified masking layer.

FIG. 4C shows one implementation of evaluation agent 205 according to a second example. In FIG. 4C, the same reference numbers as in FIG. 4A are used to indicate the same components. FIG. 4C further includes a modified masking layer 450. Modified masking layer 450 is configured to receive a vector of action values 403 comprising an estimated cumulative reward Q(s,a) for performing action a in state s. First, the exp(Q(s,a)/τ) value is calculated for every behavior (where τ(tau) is the temperature hyperparameter and “exp” is the exponential function) . Then a mask (mask(s,a)) is applied (i.e. mask(s,a)*exp(Q(s,a)/τ)) and some of these values are set to zero. , thereby generating a second vector of action values 452. Finally, the resulting values (i.e., the second vector of behavior values 452) are normalized to a probability distribution (the masked behavior has probability zero) by a normalization layer 408, as discussed above. be done.

After generating the output vector of action probabilities, the evaluation agent 205 is configured to select a combination of one or more action candidates by sampling from the resulting probability distribution (i.e., the probability distribution of the output vectors). be done.

Considering the example of FIG. 2, this means that the evaluation agent 205 selects the combination of the action "OFFER" from the "Task" dimension and "IMPL-CONFIRM" from the "Auto Feedback" dimension as the output action, but " From the "Social Responsibility Management (SOM)" dimension, the "ACCEPT_THANKING" behavior results in not being selected.

After selecting a combination of candidate actions, the evaluation agent 205 performs a process by retrieving the dialogue acts generated by the dialogue act (DA) agent that are associated with the actions that form the final combination of actions generated by the evaluation agent 205. , configured to generate a combination of interaction acts. In the example of FIG. 2, the combination of action candidates is "OFFER" from the "task" dimension and "IMPL-CONFIRM" from the "automatic feedback" dimension. In this case, the evaluation agent 205 uses the first interaction act (DA) agent 202 (“OFFER”) together with the interaction act candidates output by the second interaction act (DA) agent 203 (associated with the “IMPL-CONFIRM” action). It is configured to generate a combination of dialogue acts by extracting and combining the dialogue act candidates output by ``(associated with the action). The combination of dialogue acts is output by the evaluation agent 205.

The output of evaluation agent 205 is a natural language generator (e.g., a natural language 103).

For example, a multifunctional utterance generated by a natural language generation (NLG) module may be "Prezzo is a popular Italian restaurant" in response to user input "I'm looking for a cheap Italian restaurant." In this case, the utterances generated by the system include an "OFFER" dialog act (i.e. "Prezzo") and an "IMPL-CONFIRM" dialog act (which indicates that the dialog system understands that the user wants Italian food). implicitly confirming to the user, "...is a popular Italian restaurant").

As discussed above, like each of the other reinforcement learning (RL) machine learning models, the fourth reinforcement learning (RL) machine learning model comprises a linear layer 402 of trainable model parameters. As discussed in more detail below, the model parameters of the reinforcement learning (RL) machine learning model for each of the interaction agents and the evaluation agent 205 are optimized based on online interactions with (simulated) users. be done. In particular, the model parameters of each reinforcement learning (RL) machine learning model are optimized based on a common reward signal obtained from (simulated) users. During an interaction with a (simulated) user, four agents (i.e., a first dialogue act (DA) agent 202, a second dialogue act (DA) agent 203, a third dialogue act (DA) agent 204, and evaluation agent 205) receive the same shared reward and update parameters (i.e., model parameters for the linear layer) based on the reward with the goal of obtaining a policy that achieves the maximum long-term cumulative reward. Moreover, like each of the other reinforcement learning (RL) machine learning models, the fourth reinforcement learning (RL) machine learning model includes a transformation layer 404 that is associated with a temperature hyperparameter. In one example, each of the agents (i.e., first dialogue act (DA) agent 202, second dialogue act (DA) agent 203, third dialogue act (DA) agent 204, and evaluation agent 205 has a respective Use the same value of temperature hyperparameter in the conversion layer.

FIG. 5 illustrates a method for generating system utterances according to an example. The method begins at step 501 by receiving a user utterance. In one example, the speech is from a human user and is captured via a microphone. In this case, automatic speech recognition is performed on the user input to recognize the user's utterances and convert them into text. In one example, the Google(R) Web Speech API is used for automatic speech recognition.

In step 502, the dialogue state is updated based on the user utterances received in step 501. In one example, interaction state is updated by interaction state tracking component 101.

In step 503, a first action candidate in a first dimension is generated. In one example, the first action candidate is generated by a first dialogue act (DA) agent 202, as discussed above.

In step 504, a second action candidate in the second dimension is generated. In one example, the second action candidate is generated by a second dialogue act (DA) agent 203, as discussed above.

In step 505, a third action candidate in the third dimension is generated. In one example, the third action candidate is generated by a third dialogue act (DA) agent 204, as discussed above.

The method of FIG. 5 shows that a first action candidate, a second action candidate, and a third action candidate are generated in parallel. This indicates that the generation of each action candidate is independent. However, for the avoidance of doubt, it is emphasized that steps 503, 504, 505 may be performed in series or in parallel.

At step 506, one or more final interaction acts are generated by selecting a combination of the first candidate action, the second candidate action, and the third candidate action. For the avoidance of any doubt, one combination of these actions does not include any of the action candidates. In one example, the final interaction act is selected by evaluation agent 205, as discussed above. In particular, a combination of candidate actions is selected and one or more final interaction act combinations are generated based on interaction acts associated with the selected action.

As discussed above, each of the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, the third dialogue act (DA) agent 204, and the evaluation agent 205 are trained Equipped with a machine learning model with possible parameters. The method shown in FIG. 5 concerns the inference phase. Therefore, the parameters of the machine learning model are fixed (i.e., do not change during use). In one example, the inference phase shown in FIG. 5 occurs after training various machine learning models (as discussed in more detail below). In another example, the trainable parameters were retrieved from storage, where the trainable parameters were stored after the training phase (at a different time).

In step 507, a system response is generated based on the final interaction act selected in step 506. In one example, the final interaction act is converted into a natural language system response using natural language generator 103.

At step 508, a natural language response is output. In one example, the output of natural language generator 103 is in text format and is converted from text to speech, optionally using the Google(R) Web Speech API. The system response is then output via the speaker.

In the above example, the machine learning models for the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, the third dialogue act (DA) agent 204, and the evaluation agent 205 are trained in advance. be done. That is, the trainable parameters of the machine learning model are obtained during the training phase (optionally at different times) and used by the machine learning model for inference. Training the trainable parameters of the multidimensional action selection component is discussed in detail next. DA) agent 204, and evaluation agent (EA) 205 are each trained using reinforcement learning.

Briefly, reinforcement learning (RL) is a framework that allows agents to reason about sequential decision-making as an optimization process. The problem is formulated as a Markov Decision Process (MDP) with a state S and a set A of agent actions. A basic reinforcement learning (RL) agent interacts with its environment in discrete time steps. At each time t, the agent receives the current state s _t and the reward _rt . The policy then selects an action a _t from the set of available actions, and the action a _t is sent to the environment. The environment is moved to a new state s _t+1 and the reward r _t+1 associated with transition (s _t , a _t , s _t+1 ) is determined. The goal of a reinforcement learning (RL) agent is to maximize the expected cumulative reward, policy π:A×S→[0,1], π(a,s)=Pr(a _t =a|s _t =s) It is about learning. A more detailed discussion of reinforcement learning is provided in Richard S. Sutton and Andrew G. Barto, Reinforcement learning: An introduction, MIT press, 2018, which is incorporated herein by reference.

As discussed above, each of the dialogue act (DA) agents and the evaluation agent comprises a linear layer. In one example, a linear layer is a single layer of (artificial) neurons configured to generate an output based on a linear combination of inputs. The linear layer generates an indication of the expected return (i.e., cumulative reward) for each action when starting from state S (i.e., the input to the machine learning model) and taking action a following the current policy π. It is configured as follows.

For a dialogue act (DA) agent, the state S corresponds to the dialogue state of the system (i.e., the dialogue state maintained by the dialogue state tracking component 101), and the behavior is based on a particular dimension (i.e., the machine learning model is the action candidate in the action set for the dimension configured to provide action candidates). In a further example, state S (i.e., the input to the machine learning model) is a set of features extracted from the full interaction state maintained by interaction state tracking component 101.

For the evaluation agent, the state S corresponds to action candidates selected by the dialogue act (DA) agent, and an action is a combination of action candidates.

The linear layer is configured to generate values for the behavioral value function. In other words, the linear layer is configured to generate an expected return (ie, cumulative reward) associated with each action in the action set based on the input state.

Thus, the output of the linear layer comprises, for each action in the action set, a value that indicates the expected return (i.e., cumulative reward) for taking that action. For example, for a linear layer in the first dialogue act (DA) agent 202 that is associated with the "task" dimension, the outputs of the linear layer are: "OFFER" action, "ANSWER" action, "REQUEST" action, " NONE” for each of the actions (i.e., the expected return/cumulative reward associated with taking the action).

As discussed above, each of the interaction act (DA) agents and the evaluation agent (EA) also comprises a transformation layer (or softmax layer) controlled by a temperature hyperparameter. In one example, the behavioral candidates are generated by sampling at least a probability distribution generated by normalizing the transformed behavioral values (optionally after masking the behavioral values), and the probability distribution is Shows the likelihood that each action in the set will maximize the long-term cumulative reward.

In reinforcement learning, there is a trade-off between exploitation and exploration (i.e., whether the machine learning model continues to do what it did, or whether the machine learning model tries new actions in the state and thereby (whether to explore the action space or not). In one example, machine learning models for dialogue act (DA) agents and evaluation agents use Boltzmann search.

In a Boltzmann search, behavior is sampled from a softmax distribution applied to an estimate of the behavior value function. As discussed above, the conversion layer is controlled by temperature hyperparameters. High temperatures are used during the early stages of training to force a more random selection of actions (from a set of possible actions associated with a dimension). However, during later stages of training, lower temperatures are used, forcing the estimates to follow more closely (ie, choosing high-value actions with higher probabilities). In one example, the temperature hyperparameter is decreased linearly until the policy (of the reinforcement learning machine learning model) selects only the actions with the highest expected return (i.e., associated with the highest estimated cumulative reward). Gradually reduce the level of.

FIG. 6 illustrates a method of training a multidimensional action selection component according to one example.

In one example, the machine learning model's trainable weights (specifically, the linear layer weights for each interaction act (DA) agent and evaluation agent) are optionally initialized with random values. .

In step 601, a dialogue is initiated. In one example, an interaction is initiated between the system and the simulated user. The simulated user is configured to generate utterances and/or interaction acts based on the goal.

In one example, initiating the interaction in step 601 comprises randomly initializing user goals from a domain ontology and providing user goals to a simulated user. For example, in a restaurant search domain, a user goal may be to find a cheap Italian restaurant and get its phone number. Generally, a user goal comprises constraints (eg, slot-value pairs) and requests (eg, slots to obtain values).

In one example, the simulated user is the author of "Jost Schatzmann, Blaise Thomson, Karl Weilhammer, Hui Ye, Steve Young, Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics: "Agenda-Based User Simulation for Bootstrapping a POMDP Dialogue System" in Companion Volume, Short Papers, Rochester, New York , April 2007, pages 149-152, Association for Computational Linguistics. It is carried out using a simulator.

In step 602, utterances are generated by a simulated user and the generated utterances are provided to a dialog system being trained (i.e., the dialog system discussed in connection with FIG. , an action selection component 102 (implemented using a trained multidimensional action selection component), and a natural language generator 103).

At step 603, the interaction state is updated (eg, by interaction state tracking component 101) in response to receiving utterances from the simulated user.

In one example, a dialogue act is generated by a simulated user, and the simulated dialogue act is passed to a retrieval-based natural language generator to generate user utterances, which then states the behavior state to update the dialogue state. Passed to Update Model (ASU).

At step 604, a system response is generated. As discussed above, each dialogue act (DA) agent obtains information that identifies the dialogue state (e.g., a feature set extracted from the complete dialogue state) and uses the information within the action set when starting from the current state. Estimate the expected return (i.e., cumulative reward) associated with taking each of the actions in . Expected returns are estimated using behavioral value functions. The behavior value function is implemented using a linear layer (eg, an artificial neural network) in which the output of the behavior value function (i.e., cumulative reward) is generated according to a linear combination of input feature sets. The linear combination is based on trainable weights, which for the first training iteration are the initialized weights. Optionally, estimated cumulative rewards associated with the candidate actions are stored for future use.

Each dialogue act agent generates a behavior candidate, and the evaluation agent generates a combination of behavior candidates. After generating the combination of action candidates, the evaluation agent generates the combination of interaction acts by combining the interaction acts associated with each action in the combination of action candidates as discussed above.

In a first example, generating the system response in step 604 comprises generating an utterance. In this case, the combination of interactional acts associated with the combination of action candidates is then provided to a natural language generator (eg, natural language generator 103) that generates natural language utterances. In one example, generating the system response comprises performing steps 503-508 discussed in connection with FIG. 5.

In step 605, the system response generated in step 604 is provided to the simulated user.

At step 606, a reward signal is calculated and, starting from state s, the reward generated when performing action a (e.g., selecting a candidate action from an action set) is stored, optionally in a storage. . This reward value is then used to determine the cumulative reward for the interaction.

The reward signal is calculated after each turn of the system (ie, after each time the system generates an utterance in response to the simulated user). As is common in reinforcement learning, the generated rewards are used to train the machine learning model (i.e., to encourage/prevent certain behaviors of the machine learning model).

In one example, the reward signal comprises two components: a reward generated by the simulated user and a reward generated internally.

In one example, the simulated user gives a +100 reward upon task completion. In the restaurant search domain, this means that the interaction system recommends a restaurant that matches the user's preferences (as specified in the user goals provided to the simulated user in step 601) and includes all requested information about this restaurant. This is when we provide

Additionally, the user issues a punishment when the system fails to respond to an appreciative action, or inserts a social action when not asked for (-5 for each occurrence). This is to force the system to learn basic reactive social behaviors. In a further example, more sophisticated social patterns may be promoted through the design of reward signals.

The second component of the reward signal is an internally generated reward. Internally, a penalty of -1 is applied each turn (to encourage shorter interactions) and a penalty of -25 indicates that a "processing problem" has been encountered and the system signals this to the user with a feedback act. Applies when there is no A processing problem is recorded in the dialog state when speech recognition or natural language understanding fails, ie, does not return a result. During training, such processing problems are randomly simulated in 5% of user turns, and the original simulated user actions are discarded.

The reward signal at each turn (i.e., when choosing action a in state s) is calculated by summing the user-generated reward and the internal reward.

In step 607, it is determined whether the interaction is finished. In reinforcement learning terminology, a dialogue represents an episode (i.e., a sequence of states, actions, and rewards that ends in an endpoint state). In one example, the interaction is ended when the simulated user says "goodbye" and the system responds with a "returnGoodbye" interaction act. In one example, the simulated user is configured to say "goodbye" when the goal is completed. Optionally, after thanking the system first.

In another example, the interaction is terminated when the user "hangs up." When a multidimensional action selection component is trained using a telephone-based application, the user "hangs up" by ending the call (i.e., disconnecting the communication link from the user to the system). Alternatively, if the multidimensional behavior selection component is trained within the software system, the user clicks a button on the user interface or enters a typed command (e.g., "<hangup>") into the interface. By doing so, it "hangs up".

If the interaction is finished, the method proceeds to step 608. At step 608, the learnable weights of the multidimensional action selection component are trained.

The weights of the linear layer (implementing the behavior value function) are updated after every interaction/episode based on the reward signals received throughout the interaction. Policies (of each machine learning model) coordinate behavior only indirectly through shared rewards, i.e. each policy operates independently without direct communication with other policies. In other words, each of the dialogue action (DA) agents and the evaluation agent (EA) are trained individually based on a common reward signal.

In one example, a dialogue act (DA) agent and an evaluation agent (EA) are trained simultaneously using Monte Carlo controlled reinforcement learning with linear value function approximation. Monte Carlo controlled reinforcement learning is described in Richard S. Sutton and Andrew G. Barto, Reinforcement Learning: An introduction, the MIT Press, 2nd edition, 2018, which is incorporated herein by reference.

Briefly, Monte Carlo controlled reinforcement learning uses Monte Carlo methods (i.e., uses repeated random sampling to obtain numerical estimates). In Monte Carlo controlled reinforcement learning, a return (ie, cumulative reward) is calculated for each state-action pair (s, a) that appears in an episode. The cumulative reward for this state-action is added to the list of cumulative rewards for the same state-action pair (i.e., if an episode produces the same state-action pair twice, a list with two values of cumulative reward is added to the list of cumulative rewards for the same state-action pair). be). The cumulative rewards are then averaged to obtain the "true" cumulative reward associated with taking action a in state s.

As discussed above, in step 606, only the instantaneous value of the reward signal is calculated and stored. As a result, during step 608, a cumulative reward is generated for each state-action pair at the end of the interaction (in natural language processing terminology), or equivalently in this example, the episode (in reinforcement learning terminology). .

To do this, the sum of rewards received between each state-action pair and the endpoint state in the sequence is calculated. Optionally, a discounted reward formula is used where a discount factor is applied to reduce the impact of future rewards on the value of cumulative rewards (eg, to prevent infinite cumulative rewards).

The accumulated rewards for the state-action pairs based on the reward signals are then used to update the trainable parameters of the dialogue act (DA) agent and the evaluation agent (EA).

For clarity, we define two terms. A "true" cumulative reward is a cumulative reward obtained during an interaction/episode with a simulated user. The "estimated" cumulative reward is the cumulative reward estimated by the action-value function implemented by the linear layer.

As discussed above, the machine learning model (specifically, the linear layer) is configured to generate an estimate of the cumulative reward for performing a given action from a given state. In one example, a linear function (specified by trainable weights) is used to generate an action value for each action given the input state.

After calculating the "true" cumulative return, it is possible to obtain a loss value (i.e., the difference between the "true" cumulative reward and the "estimated" cumulative reward). The loss is then used to update the trainable parameters of the machine learning model (specifically, the linear layer weights).

In one example, training a machine learning model for each interaction act (DA) agent and evaluation agent (EA) to minimize the mean squared error between the "true" and "estimated" cumulative rewards. Equipped with In this example, the mean squared error is calculated in step 608 and the trainable weights of the machine learning model (specifically, the linear layer) are updated using stochastic gradient descent.

Consider the example shown in FIG. 2 and assume that FIG. 2 represents the interaction during the training phase. In this example, the action generated by the first dialogue act (DA) agent 202 (associated with the "task" dimension) is "OFFER." The action generated by the second interaction act (DA) agent 203 (associated with the "Automatic Feedback" dimension) is "IMPL-CONFIRM". The action generated by the third dialogue act (DA) agent 204 (associated with the "Social Responsibility Management (SOM)" dimension) is "ACCEPT_THANKING." The combination of actions generated by the evaluation agent 205 is "OFFER" + "IMPL-CONFIRM".

During training, the "true" cumulative reward consists of 1) the instantaneous reward (generated by the simulated user in response to "OFFER" + "IMPL-CONFIRM") and 2) other utterances within the interaction until the end of the interaction. generated based on subsequent instantaneous rewards received for.

Each of the dialogue act (DA) agents and the evaluation agent (EA) are trained separately based on a common reward signal (i.e., each of the agents is trained using the same "true" cumulative reward) .

Consider a first dialogue act (DA) agent 202. The machine learning model (specifically, the linear layer) of the first interaction act (DA) agent 202 determines whether the input state (e.g., features extracted from the full interaction state) according to the trainable weights of the linear layer; The cumulative reward for the "OFFER" behavior was estimated. The weights of the machine learning model (which generates an "estimated" cumulative reward and thus represents the "estimated" action value function) are then calculated for the "OFFER" action if the input state is the previously generated "estimated" ” is trained according to the difference between the cumulative reward and the “true” cumulative reward calculated above.

Consider a third dialogue act (DA) agent 204. The machine learning model (specifically, the linear layer) of the third dialogue act (DA) agent 204 is based on the input state (e.g., features extracted from the full dialogue state) according to linearly trainable weights. , estimated the cumulative reward for the "ACCEPT_THANKING" behavior.

From the perspective of the third dialogue act (DA) agent 204, the dialogue act (DA) agent "thinks" that the "true" cumulative reward calculated above was generated for the "ACCEPT_THANKING" action (i.e. (not aware that the evaluation agent 205 did not select the "ACCEPT_THANKING" action). As a result, the weights of the machine learning model within the third dialogue act (DA) agent (representing the "approximate" action value function for the "Social Responsibility Management (SOM)" dimension) are then It is trained based on the difference between the cumulative reward and the "estimated" cumulative reward estimated for the "ACCEPT_THANKING" action.

Consider evaluation agent (EA) 205. The evaluation agent (EA) 205 has a different state than any of the dialogue act (DA) agents (each has a different set of features as input, but all share a common full dialogue state representation as discussed above). ). In particular, the state for the evaluation agent (EA) 205 is determined by the state of the various dialogue act (DA) agents (e.g., the first dialogue act (DA) agent 202, the second dialogue act (DA) agent 203, and the third dialogue act (DA) agent 203). Based on the action candidates output by the dialogue act (DA) agent 204).

Therefore, the machine learning model of the evaluation agent (EA) estimates the “estimated” cumulative reward for the states {“OFFER”・“IMPL-CONFIRM”・“ACCEPT_THANKING”} and actions {“OFFER” + “IMPL-CONFIRM”}. configured to generate. The combination of actions (i.e., "OFFER" + "IMPL-CONFIRM") can then be used to generate the combination of interaction acts discussed above, which can then be used to generate a reward. It is noted. The difference between this "estimated" cumulative reward and the "true" cumulative reward, generated above, is used to train a linear function learnable weight that approximates the action value function within the evaluation agent (EA) 205. be done.

To summarize the above, in this example, the "true" cumulative reward generated above is the following (state, action) pair: (information identifying the interaction state_1, "OFFER") - the first In the case of the dialog act (DA) agent 202, (information identifying the dialog state_2, "IMPL-CONFIRM") - In the case of the second dialog act (DA) agent 203, (information identifying the dialog state_3, "ACCEPT_THANKING") ”)-for the third dialogue act (DA) agent 204, and (“OFER”/“IMPL-CONFIRM”/“ACCEPT_THANKING”, “OFFER+IMPL-CONFIRM”)-for the evaluation agent 205.

As discussed above, the information identifying the dialogue states may include, in one example, different sets of features extracted from a common (i.e., the same) full dialogue state representation, as indicated by labels 1, 2, and 3. It may be.

The weight updates are based on the difference between the (common) "true" and "estimated" cumulative rewards generated by each of the interaction act (DA) agents and the evaluation agent (EA) within the machine learning model. , is generated for each weight of each dialogue act (DA) agent and evaluation agent (EA). As discussed above, the weights of each linear layer are updated to minimize the mean squared error between the "true" and "estimated" cumulative rewards. Optionally, weight updates for each linear layer for each weight are computed using stochastic gradient descent.

At step 608, the weights of each linear layer in each of the interaction act (DA) agents and evaluation agents (EA) are updated according to the calculated weight updates. The method then proceeds to step 609.

In step 609, temperature hyperparameters are controlled. As discussed above, in one example, each transformation layer (e.g., the transformation layer of the first dialogue act (DA) agent 202, the transformation layer of the second dialogue act (DA) agent 203, the transformation layer of the third dialogue act The temperature hyperparameters of the transformation layer of the (DA) agent 204 and the transformation layer of the evaluation agent (EA) 205 decay linearly as the number of interactions (or equivalently, episodes, in reinforcement learning terminology) increases. Gradually reduce the number of levels.

In one example, the weights are updated (at step 608) based on a learning rate hyperparameter that controls the amount of change to the weights resulting from a given determined error. In this case, step 609 also comprises gradually decreasing the value of the learning rate hyperparameter as the number of interactions increases.

In one example, both hyperparameters (i.e., temperature hyperparameter and learning rate hyperparameter) are attenuated after each training interaction to reach a minimum level.

After adjusting the hyperparameters, the method proceeds to step 610.

At step 610, it is determined whether the number of times to train the interaction is equal to the maximum number of times to train the interaction. Essentially, step 610 comprises determining whether the training phase is complete. In one example, the maximum number of training interactions is predetermined (ie, set before starting the method of FIG. 6).

If in step 610 it is determined that the number of interactions is equal to the maximum number of training interactions, training is terminated/completed as indicated by step 611. Optionally, once training is complete, the trainable weights for each interaction act (DA) agent and evaluation agent (EA) are stored for future use. As explained further below, the trainable weights of each dialogue act (DA) agent are stored with an indication of which dimension they were generated for, thereby allowing the pre-generated weights for the various dimensions to be Modular systems may be able to be created if weights are reused.

If it is determined in step 610 that the number of interactions is not equal to the maximum number of interactions, the method proceeds to step 601 where a new interaction is initiated with the simulated user. In one example, step 601 further comprises incrementing the value of the number of times to train the interaction.

In the training method discussed above, steps 601 (initiate dialogue), step 602 (generate user utterances), and step 603 (update dialogue state) pass user utterances/text to the dialogue system. and then using a behavioral state update model. Similarly, in the example discussed above, step 604 comprises generating a system response in the form of an utterance.

In another example, a multidimensional action selection component is trained by passing (simulating) user interaction actions to an interaction system using a rule-based state update model.

In this case, in steps 601 and 602 the (mock) user generates user interactions, and in step 603 the interaction state is updated based on the user interactions according to a rule-based state update model. In one example, a simulated user generates an interaction act that is made into an interaction hypothesis with a confidence score using a semantic error model. The dialogue state is then updated using the top hypothesis based on hand-crafted rules.

In one example, the simulated user considers interaction acts as input. Therefore, generating the system response in step 604 involves providing the simulated user with the combination of interaction acts associated with the combination of action candidates (i.e., without generating an utterance based on the combination of interaction acts). Be prepared. In this case, natural language generator functionality is not required as part of the training method.

Using dialogic acts directly (i.e., without generating and/or processing utterances) speeds up training of multidimensional action selection components, behavioral state update models, or natural language generator components. By not requiring less resources can be used for training.

Although the training methods discussed above used simulated users, for the avoidance of any doubt, the multidimensional behavior selection component may instead interact with a physical user (i.e., a real human) or a human It is emphasized that the training may be based on information representing interactions between users (eg, a corpus of data). However, advantageously, using simulated users speeds up the training process.

Moreover, although the training method discussed above uses a specific approach to reinforcement learning (i.e. Monte Carlo control with linear value function approximation), for the avoidance of any doubt, training the reinforcement learning model It is emphasized that other reinforcement learning techniques may be used. Some of these are described in Richard S Sutton and Andrew G Barto, Reinforcement learning: An introduction, MIT press, 2018, which is incorporated herein by reference. For example, a machine learning model may be trained using a state-action-reward-state-action (SARSA) algorithm. In particular, the multidimensional action selection component is trained using other reinforcement learning (RL) techniques such as Gaussian process state-action-reward-state-action (GP-SARSA) and/or deep Q-learning (DQN). It's fine.

Finally, the training methods discussed above used a specific reward function (i.e., generating a reward signal). Other reward functions may also be used.

FIG. 7 illustrates a comparison of interaction success rates for a multidimensional system using hand-coded evaluation agents and evaluation agents, according to examples described herein.

Testing was conducted to evaluate the impact that a trainable evaluation agent (such as those described herein) can have on the overall interaction policy optimization process (of the overall system). Testing involved training 10 sets of interaction policies (ie, completing 10 training runs). Each training run includes hand-coded policies for the evaluation agents (i.e., first dialogue act (DA) agent 202, second dialogue act (DA) agent 203, and third dialogue act (DA) agent 204). and four policies in a system using an evaluation agent according to the examples described herein (i.e., a first interaction act (DA) agent 202, a second interaction act (DA) agent 202, (DA) agent 203, third dialogue act (DA) agent 204, and evaluation agent 205 policies). Figure 7 shows the learning curve for 40000 simulated training interaction policy optimizations for both systems. Each curve represents the sliding window success rate over the past 100 interactions during training, averaged over 10 training runs.

As noted from Figure 7, both systems converge to comparable performance levels with an average success rate of approximately 95%. However, for between 5000 and 15000 interactions, the performance level of the version using a trainable evaluation agent (ie, the embodiment described herein) is quite good.

The results of evaluating a fully trained policy over 3000 test interactions (e.g. after 40000 training interactions) are shown in the table below, which is based on the trained evaluation agent (e.g. (similar to the embodiment described in this paper) shows that a system with a hand-coded evaluation agent obtains a better success rate than a system with a hand-coded evaluation agent.

A system with a trained rating agent (i.e., similar to that illustrated and described herein) was tested as part of a voice interaction system in crowdsourced human user ratings. Results for a fully trained system (including a trained assessment agent) are shown below, with subjective partial and full completion rates (%), as well as for perceived understanding. The average scores (1-6), speech recognition, adequacy and naturalness of the system response are obtained from a questionnaire that the subjects were asked to fill out after each interaction.

As can be appreciated from the above, the examples described herein provide better performance than systems with hand-coded evaluation agents. Moreover, the behavior selection component implemented by the examples described herein can be implemented by, for example, including more dialogue act (DA) agents to support additional dimensions, or individual interaction within a multidimensional system. Expanding the behavior set of a dialogue act (DA) agent has the advantage that it is easier to train as dialogue systems become more complex. In contrast to the examples described herein, it requires an increasingly deeper knowledge of interaction act taxonomy to scale increasingly tedious and hand-coded/rule-based evaluation agents. I need. Using evaluation agents (EAs) that can be trained automatically, in collaboration with dialogue act (DA) agents in the system, from online interactions is a more scalable and reliable way to build multidimensional systems. provide a high-quality method. Therefore, a general method for automatically learning combinatorial processes (as described herein) is preferred for long runs.

Moreover, hand-coded evaluation agents select only a single interaction act candidate from a one-dimensional/dialogue act (DA) agent (i.e., some hand-coded evaluation agents select a combination of actions from different dimensions). (not configured to select) is common. Therefore, these hand-coded evaluation agents do not produce multifunctional utterances (i.e., utterances that convey multiple dialogic acts) in a manner based on the principles provided by the ISO 24617-2 annotation standard, and do not produce multifunctional utterances (i.e., utterances that convey multiple dialogic acts), but rather Limit system response with respect to both. In contrast, embodiments described herein select combinations of one or more interaction actions based on policies learned using reinforcement learning (RL). Moreover, the masking mechanisms described herein are used to control the range of possible combinations, particularly to accommodate downstream components.

Advantageously, the evaluation agent (EA) described herein opens up the opportunity to learn the strategic and stylistic aspects of selecting dialogue act combinations directly from dialogue interactions. More generally, the embodiments described herein help push the agenda for developing interoperable dialogue system components that comply with the ISO 24617-2 standard, which may lead to a wide range of applications. For example, the production of dialogue systems can be streamlined.

In summary, multidimensional policy models facilitate training multiple policies/agents to make behavioral choices in an interaction, as opposed to using a single policy model. Such models facilitate the output of combinations of interaction acts, and multiple agents are trained to handle different aspects of the interaction. In the example described above, a trainable evaluation agent (EA) is provided that is trained with multiple dialogue action (DA) agents to determine the next action based on a common reward signal. This selected action may be an action determined by a single dialogue act (DA) agent, or a combination of actions from multiple dialogue act (DA) agents within a multidimensional system. In the examples described herein, the evaluation agent (EA) is trained, as opposed to being manually created/hand-coded. As can be seen from the experimental results provided above, trainable evaluation agents (EAs) help reduce complexity and improve interaction success when compared to manually created evaluation agents.

Another issue with previous dialog systems is scalability to new domains and applications. Previous interaction systems typically scale as the domain expands with new slots (i.e., new information is tracked by the interaction state), and as new applications emerge that require new task-specific system behavior. is poor. This is partly due to the lack of availability of annotated datasets within the domain.

One approach to solving this problem is to use transfer learning techniques to efficiently develop interaction systems for new domains with limited or no data in the target domain.

Advantageously, the multidimensional action selection component presented herein is particularly well suited for transfer learning. In particular, the multidimensional behavior selection component according to the examples presented herein opens opportunities for efficient adaptation of interaction managers to new tasks and domains.

For example, automatic feedback dimensions and social responsibility management (SOM) dimensions are considered domain and task independent, and therefore interaction policies for these dimensions are transferred (forming previously trained models). ), leaving only the policy for the "task" dimension to be trained from scratch in the target domain.

FIG. 8A illustrates a method of transfer learning using multidimensional action selection components, according to one example.

The method of FIG. 8A is discussed in connection with adapting a multidimensional behavioral selection component to a new use case within a domain (eg, within a restaurant search domain). In particular, the method of FIG. 8A is discussed in connection with an example in which a "task" agent's behavior set is expanded to include slot-specific behaviors. However, it is emphasized that transfer learning techniques may be used to adapt the multidimensional behavioral selection component to new domains (eg, from a hotel search domain to a restaurant search domain). Additionally, the transfer learning techniques discussed below may be used when adding new behaviors that extend the behavior of the system (eg, when extending a restaurant search system to a restaurant reservation system).

In one example, the action set of a "task" agent is expanded by replacing the summary action "REQUEST" for requesting slots in an interactive state with a separate request action for each slot. In one example, where the interaction state includes slots for food type (TYPE), price range (PRICERANGE), and search area (AREA), the action sets include "REQUEST-AREA", "REQUEST-FOOD", and It is expanded to include "REQUEST-PRICERANGE".

For ease of understanding, the following terms are defined. A source scenario is a scenario (eg, a use case within a domain, or the domain itself) in which the trainable weights of the multidimensional behavior selection component were originally generated. The target scenario is a new scenario (ie, a new domain or a new use case) in which the multidimensional action selection component is being used.

In the source scenario, the policy may select the REQUEST summary action, and then heuristics determine which slot is requested, whereas in the target scenario, by way of example, the interaction policy may select the REQUEST summary action. For example, the "REQUEST-AREA" action may be selected to request more information about the regional slot. Therefore, in a target scenario with an expanded action set, the system automatically learns which slots to request, rather than relying on heuristics as in the source scenario.

As discussed above, within the restaurant search domain, this means that the "REQUEST" summary behavior corresponds to the three slot-specific request behaviors (i.e., "REQUEST-AREA") corresponding to the slots Food, Region, and Price Range. ”, “REQUEST-FOOD”, “REQUEST-PRICERANGE”).

Returning to the original set of actions outlined in the table discussed above, the set of actions for the "task" agent is thus changed from 4 actions (OFFER, REQUEST, ANSWER, NONE) to 6 actions (OFFER, REQUEST- FOOD, REQUEST-AREA, REQUEST-PRICERANGE, ANSWER, NONE), and the set input features for the evaluation agent (EA) are also expanded to include a new set of behaviors for the "task" agent.

Therefore, both the "task" agent and the evaluation agent (EA) must be retrained for the target scenario, especially to account for the new behaviors available in the "task" dimension.

At step 801, a multidimensional action selection component is generated. This step may include initiating a multidimensional behavior selection component within the software.

At step 802, trainable weights for a dialogue act (DA) agent are obtained. In one example, a dialogue act (DA) agent is associated with a dimension that is not specific to the source scenario (eg, the dimension is domain independent). As discussed above, domain-independent dimensions may include an "automatic feedback" dimension and a "social responsibility management (SOM)" dimension. Optionally, these agent trainable weights are retrieved from storage when the machine learning model is trained for the source scenario.

In another example, obtaining pre-trained weights for a dialogue action (DA) agent comprises training a multidimensional system (eg, according to FIG. 6) for a source scenario. In this case, a "task" interaction act (DA) agent uses a summary action (ie, "REQUEST") to request a slot. This results in four trained policies, corresponding to the three conversation dimensions plus the evaluation agent. Among these, the domain-independent interaction policies for "Auto Feedback" and "SOM" are then reused for the target scenario.

At step 803, a multidimensional action selection component is trained for the target scenario. In this case, the task agent uses an extended action set (comprising "REQUEST-AREA", "REQUEST-FOOD", "REQUEST-PRICERANGE"). The interaction policy for the domain/use case independent dimension is reused and the multidimensional action selection component is retrained (e.g., according to Figure 6) so that the system uses the expanded action set in the "task" dimension. Ru.

It has been observed that by reusing policies in this way, multidimensional action selection components can increase training speed (i.e., obtain higher conversion success rates in fewer training interactions). ing.

FIG. 8B illustrates a method of transfer learning using multidimensional action selection components according to a second example. Step 850 comprises training a multidimensional action selection component for the source scenario. Step 851 comprises storing a trained policy that can be transferred to a target scenario. Step 852 comprises training and reusing the multidimensional action selection component for the target scenario and adapting the pre-trained transferable policy.

FIG. 9 shows a schematic diagram of hardware that can be used to implement methods according to embodiments. This hardware comprises a computing system 900. In this particular example, the components of the system will be described together. However, it will be appreciated that the components are not necessarily co-located.

The components of computing system 900 include, but are not limited to, a processing unit 913 (such as a central processing unit or CPU, otherwise referred to as a "processor"), a system memory 901, and a system memory 901. and a system bus 911 that couples various system components to a processing unit 913. System bus 911 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a local bus using any of a variety of bus architectures, and the like. Computing system 900 may also include memory 915 coupled to system bus 911 . Memory 915 may be external memory.

System memory 901 comprises computer storage media in the form of volatile/non-volatile memory, such as read only memory (ROM). System memory 901, when executed by processing unit 913 (e.g., a processor), causes processing unit 913 (also referred to as a "processor") to perform the functions described herein, particularly with respect to FIG. The systems discussed, the multidimensional action selection component described in connection with Figure 2, the method of Figure 5 (inference), the method of Figure 6 (training), and/or the method of Figure 8A or 8B (transfer learning). ), storing computer-readable instructions for performing at least one of the following:

System memory 901 may also include an operating system 905 , application programs 907 , and program data 909 in use by processing unit 913 . In one embodiment, system memory 901 includes a basic input/output system (BIOS) that contains routines that help transfer information between elements within a computer, such as typically stored in system memory 901 during startup. 903 is also provided.

The computing system further includes an interface 925 communicatively connected to system bus 911 . In one embodiment, interface 925 includes the ability to communicate via wired and/or wireless connections, specifically to receive information from and send information to other devices. Additionally or alternatively, interface 925 includes an audio input (e.g., a microphone) for receiving user input (e.g., vocalizations) and an audio output (e.g., a speaker) for providing system responses to the user. Equipped with Additionally and alternatively, interface 925 includes means for receiving text input data from a device (eg, a keyboard) configured to generate text data. In another embodiment, interface 925 may be any form of user interface that allows a user to interact with the system, for example, by generating commands and/or questions and receiving responses. .

In the example of FIG. 9, a video interface 917 is provided. Video interface 917 includes a graphics processing unit (GPU) 919 communicatively connected to graphics processing memory 921 .

Graphics processing unit (GPU) 919 is particularly well suited for training machine learning models, such as neural network training, due to its suitability for data parallel operations. Accordingly, in one embodiment, processing for training a machine learning model may be split between CPU 913 and GPU 919.

It should be noted that in some embodiments, different hardware may be used to train the machine learning model and to perform state updates. For example, training of a machine learning model may occur on one or more local desktop or workstation computers, or on a device in a cloud computing system, which may include one or more discrete desktop or workstation GPUs, one or more discrete desktop or workstation CPUs, e.g., processors with a PC-oriented architecture and a significant amount of volatile system memory, e.g., 16 GB or more. For example, the performance of the interaction may use mobile or embedded hardware that may or may not include a mobile GPU as part of a system-on-chip (SoC), or one or more A mobile or embedded CPU, eg, a processor with a mobile-oriented architecture, or a microcontroller-oriented architecture, and a smaller amount of volatile memory, eg, less than 1 GB, may be used. For example, the interacting hardware may be a voice assistance system 120, such as a smart speaker or a mobile phone that includes a virtual assistant.

The hardware used to train machine learning models for interaction act (DA) agents and evaluation agents can have significantly greater computing power, e.g. It can perform more operations per second than hardware and may have more memory.

Computing system 900 may be implemented as a mobile phone with a virtual assistant, a smart speaker, an embedded device, a desktop computer, a laptop computer, or as part of a server.

Although FIG. 9 shows a specific example of hardware that can be used to implement the methods described herein, this is only one example and other arrangements may also be used. will be understood.

Although several arrangements have been described, the arrangements are presented by way of example only and are not intended to limit the scope of protection. The inventive concepts described herein may be implemented in various other forms. Moreover, various omissions, substitutions, and modifications to the specific implementations described herein may be made without departing from the scope of protection as defined in the following claims.

Claims (25)

A computer-implemented method for generating a response within a dialog system, the dialog system being for interacting with a user, the method comprising:
receiving utterances from the user;
updating a dialogue state based on the utterance;
generating information identifying the interaction state;
selecting, by a first interaction act agent, a first action candidate from a first set of action candidates using a first machine learning model and the information identifying the interaction state; , the first set of action candidates is associated with a first interaction dimension;
selecting, by a second interaction act agent, a second action candidate from a second set of action candidates using a second machine learning model and the information identifying the interaction state; , the second set of action candidates is associated with a second interaction dimension;
selecting an output action by an evaluation agent using a third machine learning model, the first action candidate, and the second action candidate; a combination of one action candidate and/or the second action candidate;
generating a system response based on the output behavior. Selecting the output action comprises:
For each combination of the first action candidate and/or the second action candidate, an estimated cumulative reward associated with selecting the respective combination as the system response in the dialog state. generating a vector of values comprising indicators;
and selecting the output behavior based on the vector of values. said vector of values is generated using a function parameterized by a first set of trainable weights, wherein an input to said function is said information identifying said interaction state; 3. The computer-implemented method of claim 2, wherein an output of a function is the estimated cumulative reward for each combination. 4. The computer-implemented method of claim 3, wherein the function is a linear function. selecting the output behavior based on the vector of values;
converting the vector of values into a probability distribution to generate a vector of behavioral probabilities;
5. The computer-implemented method according to any one of claims 2 to 4, comprising: selecting the output behavior based on the vector of behavior probabilities. transforming the vector of values into the probability distribution to generate the vector of action probabilities;
converting the vector of values into a vector of transformed behavioral values, wherein the transformed vector of behavioral values comprises a value greater than or equal to zero;
forming a vector of masked transformed behavior values by setting a first behavior value in the vector of transformed behavior values equal to zero based on a predetermined criterion;
and normalizing the vector of masked transformed behavioral values to form the vector of behavioral probabilities. 2. The computer-implemented method of claim 1, wherein the first interaction dimension is different than the second interaction dimension. A computer-implemented method for training a dialogue system, the method comprising:
generating a system response by the method of claim 1;
providing the system response to the user;
A computer-implemented method comprising training the first machine learning model, the second machine learning model, and the third machine learning model using reinforcement learning. training the first machine learning model, the second machine learning model, and the third machine learning model using reinforcement learning;
determining cumulative compensation;
9. The computer-implemented method of claim 8, comprising training the first machine learning model, the second machine learning model, and the third machine learning model based on the cumulative reward. . the first machine learning model estimates a first estimated cumulative reward for the first candidate action based on a first function parameterized by a first set of trainable weights; configured such that, and where:
training the first machine learning model based on the cumulative reward;
determining a difference between the cumulative reward and the first estimated cumulative reward;
and updating the first set of trainable weights based on the difference. the third machine learning model is configured to estimate a second estimated cumulative reward for the output behavior based on a second function parameterized by a second set of trainable weights; , and here,
training the third machine learning model based on the cumulative reward;
determining a difference between the cumulative reward and the second estimated cumulative reward;
and updating the second set of trainable weights based on the difference. 11. The computer-implemented method of claim 9 or 10, wherein the cumulative reward is based on a total reward, the total reward comprising a first reward generated for providing the system response. receiving a second utterance in response to providing the system response;
generating a second system response by the method of claim 1, wherein the utterance in the method of claim 1 is the second utterance; the system response generated is the second system response;
generating a second reward based on the second system response;
13. The computer-implemented method of claim 12, further comprising calculating the cumulative reward based on a sum of the first reward and the second reward. The dialogue system has a first dialogue policy implemented by the first machine learning model, a second dialogue policy implemented by the second machine learning model, and a second dialogue policy implemented by the third machine learning model. a first hyperparameter configured to control the level of exploration of a third interaction policy;
the first hyperparameter has a first value during a first interaction, the first interaction comprises the utterance and the system response, and the method includes:
determining whether the first interaction is complete; and in response to determining that the first interaction is complete;
the level of exploration of the first interaction policy, the second interaction policy, and the third interaction policy in a state-action space is less than the level of exploration during the first interaction; modifying the first value that is a first hyperparameter;
9. The computer-implemented method of claim 8, further comprising retraining the interaction system with a second interaction. A device for generating a response within a dialog system, wherein the dialog system is for interacting with a user, and the device is configured to:
Receive utterances from the user,
updating a dialogue state based on the utterance;
generating information identifying the dialogue state;
a first interaction act agent selects a first action candidate from a first set of action candidates using a first machine learning model and said information identifying said interaction state; a first set of action candidates is associated with a first interaction dimension;
a second interaction act agent selects a second action candidate from a second set of action candidates using a second machine learning model and said information identifying said interaction state, wherein said a second set of action candidates is associated with a second interaction dimension;
an evaluation agent selects an output behavior using a third machine learning model, the first action candidate, and the second action candidate, wherein the output action is different from the first action candidate; comprising a combination of action candidates and/or the second action candidates;
A device configured to generate a system response based on the output behavior. When the device selects the output action,
For each combination of the first action candidate and/or the second action candidate, an estimated cumulative reward associated with selecting the respective combination as the system response in the dialog state. generate a vector of values with indicators,
16. The device of claim 15, further configured to select the output behavior based on the vector of values. The device is further configured to generate the vector of values using a function parameterized by a first set of trainable weights, wherein an input to the function determines the interaction state. 17. The device of claim 16, wherein the information identifying and the output of the function is the estimated cumulative reward for each combination. 18. The device of claim 17, wherein the function is a linear function. when the device selects the output behavior based on the vector of values;
converting the vector of values into a probability distribution to generate a vector of behavioral probabilities;
19. The device of any of claims 16 to 18, further configured to select the output behavior based on the vector of behavior probabilities. When the device transforms the vector of values,
converting the vector of values into a vector of transformed behavioral values, where the vector of transformed behavioral values comprises a value greater than or equal to zero;
forming a vector of masked transformed behavior values by setting a first behavior value in the vector of transformed behavior values equal to zero based on a predetermined criterion;
20. The device of claim 19, further configured to normalize the vector of masked transformed behavioral values to form the vector of behavioral probabilities. 16. The device of claim 15, wherein the first interaction dimension is different than the second interaction dimension. A training system for training a dialogue system, comprising a device according to claim 15,
generating a system response using the device of claim 15;
providing the system response to the user;
A training system configured to train the first machine learning model, the second machine learning model, and the third machine learning model using reinforcement learning. when the training system uses reinforcement learning to train the first machine learning model, the second machine learning model, and the third machine learning model;
Determine the cumulative reward,
23. The training of claim 22, further configured to train the first machine learning model, the second machine learning model, and the third machine learning model based on the cumulative reward. system. the first machine learning model estimates a first estimated cumulative reward for the first candidate action based on a first function parameterized by a first set of trainable weights; and wherein the training system trains the first machine learning model based on the cumulative reward;
determining a difference between the cumulative reward and the first estimated cumulative reward;
24. The training system of claim 23, further configured to update the first set of trainable weights based on the difference. A non-transitory computer readable medium comprising computer program instructions suitable for execution by a processor, the instructions configured, when executed by the processor, to cause the processor to perform the method of claim 1. non-transitory computer-readable medium.