JP7598076B2 - Information processing device and program - Google Patents

Description

The present invention relates to an information processing device and a program.

In recent years, various tools have been developed to manage users' schedules. Among these tools, there is known a technology that registers and manages schedules based on specific keywords contained in the user's conversations (see, for example, Non-Patent Document 1, etc.).

"iPhone User Guide", [online], Apple Computer, [searched December 16, 2019], Internet <URL: https://support.apple.com/ ja-jp/guide/iphone/iph3d110f84/ios>

However, while the above conventional tools communicate with users using conversational sentences, they are only capable of managing future plans and are not capable of providing conversational sentences related to those plans, and do not provide sufficient information tailored to the circumstances of each user.

The present invention has been made in consideration of the above-mentioned situation, and one of its objectives is to provide an information processing device and program that can provide conversational text tailored to the circumstances of each user.

One aspect of the present invention that solves the problems of the above-mentioned conventional examples is an information processing device that includes an acquisition means for acquiring event-specific information that identifies a user's event associated with date and time information, and a conversation sentence generation means for generating a related conversation sentence related to the event-specific information in a process of speaking at a date and time after the date and time represented by the date and time information.

This invention makes it possible to provide conversational text tailored to the circumstances of each user.

FIG. 1 is a block diagram illustrating an example of the configuration of an information processing system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating a configuration example of a terminal device according to an embodiment of the present invention. FIG. 2 is a functional block diagram illustrating an example of a server according to an embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating an example of a conversation sentence queue used in the information processing system according to the embodiment of the present invention. 1 is an explanatory diagram illustrating an example of the contents of an action database used in an information processing system according to an embodiment of the present invention. FIG. FIG. 2 is a functional block diagram illustrating an example of a terminal device according to an embodiment of the present invention. FIG. 2 is an explanatory diagram illustrating an example of setting information used in the information processing system according to the embodiment of the present invention. 4 is a flowchart illustrating an example of the operation of the information processing system according to the embodiment of the present invention. 6 is another flowchart illustrating an example of the operation of the information processing system according to the embodiment of the present invention. FIG. 11 is a flowchart illustrating an example of a conversation sentence selection process of the information processing system according to the embodiment of the present invention.

An embodiment of the present invention will be described with reference to the drawings. As shown in FIG. 1, an information processing system 1 according to an embodiment of the present invention includes a server 10 as an information processing device, and a terminal device 20 communicatively connected to the server 10 via a communication means such as a network.

As shown in FIG. 1, the server 10 includes a control unit 11, a memory unit 12, and a communication unit 13. The terminal device 20 is a robot, and includes at least a leg unit 21 and a main body unit 22, as shown in FIG. 2, in which the control unit 31, the memory unit 32, the sensor unit 33, the display unit 34, the audio output unit 35, the communication unit 36, and the drive unit 37 are housed. The leg unit 21 and the main body unit 22 are connected via an actuator that can rotate around at least one axis, and the orientation of the main body unit 22 can be rotated relative to the leg unit 21.

The control unit 11 of the server 10 is a program control device such as a CPU, and operates according to a program stored in the memory unit 12. In this embodiment, the control unit 11 accepts request information from the terminal device 20. The control unit 11 also executes processing based on the accepted request information. For example, as one of the processes, the control unit 11 transmits action information including an action instruction instructing an action to be executed by the terminal device 20 and character string information representing the content of the voice to be uttered by the terminal device 20 to the terminal device 20 that is the sender of the request information.

In one example of this embodiment, the control unit 11 acquires information representing a user's event, such as a "date," associated with date and time information representing a date and time (hereinafter referred to as event-specific information), and generates a related conversation sentence related to the acquired event-specific information in the process of executing a process of sending to the terminal device 20 a voice uttered at the terminal device 20 at a date and time after the date and time represented by the date and time information. The detailed contents of the process of the control unit 11 will be described later.

The storage unit 12 is a disk device or memory device, and holds programs executed by the control unit 11. The storage unit 12 also operates as a work memory for the control unit 11. In one example of this embodiment, the storage unit 12 may store information for generating instructions for the terminal device 20. For example, the storage unit 12 holds a conversation sentence queue that accumulates conversation sentence candidates to be spoken by the terminal device 20. The contents of this conversation sentence queue will be described later.

The communication unit 13 is a network interface or the like, and sends various information to the terminal device 20 via the network according to instructions input from the control unit 11. The communication unit 13 also outputs information received via the network to the control unit 11.

The control unit 31 of the terminal device 20 is a program-controlled device such as a CPU, and operates according to a program stored in the storage unit 32. In this embodiment, the control unit 31 sends request information to the server 10 at a predetermined timing together with device identification information unique to the terminal device 20, which will be described later.

In one example of this embodiment, when a sensor unit 33, which will be described later, receives voice input from a user, the control unit 31 of the terminal device 20 converts the input voice into character string information. This process can use widely known voice recognition processing, and the control unit 31 may execute this process, for example, by sending information on the input voice to a voice recognition server that executes the voice recognition process and receiving the recognized character string information.

The control unit 31 also sends request information to the server 10 when the user inputs voice as a trigger. This request information includes information that identifies the trigger (e.g., information that the user has input voice) and information necessary for processing by the server 10, for example, character string information that is the recognition result of the voice input by the user.

In other words, when the control unit 31 determines that a predetermined trigger has occurred, it collects information necessary for processing in the server 10 and sends request information including the collected information to the server 10 together with information identifying the trigger. This trigger can be any trigger, such as the user inputting voice as in the previous example, or the arrival of a specified time. The detailed operation of the control unit 31 will be described later.

The storage unit 32 is a memory device or the like, and stores the programs executed by the control unit 31. The storage unit 32 also operates as a work memory for the control unit 31.

In this embodiment, the storage unit 32 may store setting information that associates the trigger with information that specifies the information to be sent to the server 10. This setting information will be described later with a specific example. The storage unit 32 also stores device identification information that is uniquely set for the terminal device 20.

Furthermore, in one example of this embodiment, the memory unit 32 stores image data of an animation to be displayed on the display unit 34. Specifically, the memory unit 32 may store multiple image data of parts that represent the facial expressions of a character, such as image data of eye animation, such as image data of smiling eyes, image data of animated eyes with tears flowing, etc., and image data of animated mouths, such as image data of a closed mouth in a smiling state, image data of a closed mouth in a crying state, image data of an animated mouth during speech, etc.

The sensor unit 33 includes at least a microphone, which is an audio sensor. The sensor unit 33 may also include a touch sensor, an acceleration sensor, etc. The sensor unit 33 outputs audio signals detected by each sensor, information indicating the position touched by the user, acceleration information, etc. to the control unit 31.

The display unit 34 is a liquid crystal display or the like, and displays image data according to instructions input from the control unit 31. In one example of this embodiment, the display unit 34 displays the character's facial expression using image data of the eyes and mouth. The audio output unit 35 is a speaker or the like, and produces sound according to an audio signal input from the control unit 31.

The communication unit 36 includes a network interface. This communication unit 36 transmits and receives information to and from the server 10 via the network, either wirelessly or via a wired connection. Specifically, the communication unit 36 sends request information and the like to the server 10 in accordance with instructions input from the control unit 31. In addition, this communication unit 36 outputs information received from the server 10 to the control unit 31.

The drive unit 37 drives the actuator to rotate the main body unit 22 relative to the legs 21 according to instructions input from the control unit 31.

Next, the operation of the control unit 11 of the server 10 in this embodiment will be described. In this embodiment, the control unit 11 of the server 10 includes a receiving unit 41, an event management unit 42, an event information acquisition unit 43, a conversation sentence generation unit 44, a conversation history management unit 45, an action information generation unit 46, and an instruction transmission unit 47, as illustrated in FIG. 3.

The receiving unit 41 receives request information and device identification information from the terminal device 20. This request information includes information identifying the cause (trigger) of the request for an action to be executed by the terminal device 20. Types of triggers will be described later, and an example would be voice input by the user. The request information based on the trigger that there has been voice input by the user may include information identifying the trigger as well as information representing the content of the voice input by the user. Here, the information representing the content of the voice input by the user may be character string information that is the result of voice recognition.

The receiving unit 41 outputs the information included in the received request information, such as information identifying the trigger and character string information representing the content of the voice input by the user, to the event information acquiring unit 43 and the action information generating unit 46.

The event management unit 42 stores and manages event information for each user in the storage unit 12. Here, event information is a user's schedule, and associates date and time information with event specification information that specifies the event.

The event information acquisition unit 43 receives input of character string information representing the content of the voice input by the user from the receiving unit 41. The event information acquisition unit 43 then determines whether the received character string information is information related to an event. Specifically, in this embodiment, the event information acquisition unit 43 compares the character string information with a predetermined pattern of information related to an event.

Here, the pattern includes a portion that matches words expressing a predetermined date and time (date and time related words) such as "tomorrow,""the day after tomorrow," and "next week," and a portion that matches words related to an event (event related words) such as "date,""work," and "beauty salon." This pattern can be expressed, for example, as a regular expression string, and for example, "[tomorrow | the day after tomorrow | weekend | next week] [D+day] [W+day]
It is set as "[日][は｜に]W+[が][あ][ある｜なん]*".

In the above example, "*" is a regular expression that matches any character including spaces, "+" is a regular expression that matches one or more characters of a type specified by the preceding character, and [X|Y|...] is a regular expression that matches either X, Y.... [Z] means a pattern that may or may not exist, "D" matches a number, and "W" matches a letter. Therefore, the above string matches string information such as "I have a date tomorrow" or "I have an exam next Monday." In the above pattern, the part "[tomorrow|the day after tomorrow|weekend|next week][D+day][W+day][day]" corresponds to a date-time related word, and the part of "W+[ga][aru|nan]" at the beginning of "W+" (matching any character) corresponds to an event-related word.

The number of patterns is not limited to one, and may be multiple. For example, in addition to the above patterns, a pattern such as "W+[ga][aru|nan]*[tomorrow|the day after tomorrow|weekend|next week][D+day][W+day][day]" may be set. This example matches an example that does not match the patterns exemplified above, such as "There's a live show tomorrow," and the event information acquisition unit 43 determines that the accepted character string information is event-related information when the character string information representing the content of the voice input by the user matches a pattern of event-related information such as the above example (or any of the patterns if there are multiple patterns).

In other words, if the character string information representing the content of the voice input by the user matches the pattern of event-related information such as the above example and determines that the information is event-related, the event information acquisition unit 43 extracts date and time information and event specific information that identifies the event from the character string information. Specifically, if the character string information is "I have a date tomorrow," the event information acquisition unit 43 obtains date and time information for the day following the date and time when the process is being executed.

The date and time information here is information indicating the date and time at which an event will occur, estimated based on the date and time related words found by the above pattern. Widely known technology can be used for the process of estimating date and time information from date and time related words, etc., so a detailed explanation will be omitted here.

For example, if the processing is performed on December 15th, the event information acquisition unit 43 acquires date and time information of December 16th, which is "tomorrow" from that date. In addition, as event specific information that identifies the event, the event information acquisition unit 43 extracts "date" from the character string information. A widely known method can be used to extract this event specific information.

The event information acquisition unit 43 outputs the date and time information and event specification information that specifies the event that are extracted here to the conversation sentence generation unit 44. Furthermore, the event information acquisition unit 43 interrupts the process if the character string information that represents the content of the voice input by the user does not match the pattern of information related to the event (if there are multiple patterns, none of them).

The conversation sentence generation unit 44 generates a conversation sentence upon receiving the date and time information and the event specific information output by the event information acquisition unit 43. Specifically, the conversation sentence generation unit 44 determines a date and time (utterance period) that is later than the date and time output by the event information acquisition unit 43.

The speech period is determined, for example, as follows. In one example of this embodiment, the server 10 stores in the storage unit 12, as the candidate day information, a number of days data table that associates predetermined candidate days such as "1 day," "2 days," and "1 week," the number of days from the first day to the last day of the speech period (speech period length) determined for each candidate day, and a word that specifies the date on which the event occurred (called a date and time indicator word).

The conversation sentence generation unit 44 randomly selects one of the number of days candidates included in the number of days data table and adds it to the date and time output by the event information acquisition unit 43 to determine the first day of the speech period. The conversation sentence generation unit 44 also adds the speech period length associated with the selected number of days candidate to the first day of the speech period to determine the last day of the speech period.

As an example, when the date and time output by the event information acquisition unit 43 is "December 15th," if "2 days" is selected from the candidate days (with the associated speech period length set to "0 days"), the conversation sentence generation unit 44 will set the first day of the speech period to "December 17th" and the last day to "December 17th."

The conversation sentence generation unit 44 also generates a conversation sentence using a date and time indicator associated with the selected number of days candidate and the event specific information output by the event information acquisition unit 43. For example, the conversation sentence generation unit 44 randomly selects one of predetermined conversation sentence pattern candidates such as "Now that I think of it, how was the <event specific information> of the <date and time indicator>?", "Ah, the <date and time indicator> was the <event specific information>, right?", etc., and inserts the date and time indicator and the event specific information into the selected conversation sentence pattern.

This allows the conversation sentence generation unit 44 to generate a conversation sentence such as, "By the way, how was your date the day before yesterday?"

The conversation sentence generation unit 44 associates the conversation sentence thus generated with the determined speech period as a speech condition, and accumulates the associated conversation sentence in the conversation sentence queue stored in the storage unit 12. That is, in this example, the conversation sentence queue stored in the storage unit 12 of the server 10 stores the speech condition (C) and the conversation sentence (D) in association with each other, as illustrated in FIG.

Note that in addition to the above-mentioned speech period, the speech conditions may also include conditions related to the time of speech and conditions such as the temperature at the time of speech. Furthermore, speech conditions are not necessarily required and do not have to be defined. If no speech conditions are defined (if there are no speech conditions), the conversation sentence without the defined speech condition becomes a conversation sentence that can be spoken at any time.

In this embodiment, in addition to the conversation sentences generated by the conversation sentence generation unit 44, conversation sentences may be preregistered in the conversation sentence queue. In this way, examples of preregistered conversation sentences include, for example, "It was hot today," associated with the speech conditions that the maximum temperature was 35 degrees or higher and the time was after 6 p.m., and "Aren't you going to bed now?" associated with the speech condition that the time was between 2 a.m. and 4 a.m.

The conversation history management unit 45 sequentially records, for each user, the contents of the voice input by the user that is accepted by the receiving unit 41, and the contents of the conversation sentences that are spoken by the terminal device 20 of the user that are instructed by the action information generating unit 46, which will be described later. In other words, the conversation history management unit 45 records the history of the conversation between the user and the terminal device 20.

The action information generating unit 46 determines the action to be executed by the terminal device 20 that sent the request based on the information input from the receiving unit 41, generates action information including information instructing the action (action instruction) and information required to execute the action (hereinafter referred to as parameter information), and outputs the action information to the instruction transmitting unit 47.

In one example of this embodiment, the memory unit 12 of the server 10 stores an action database including at least one record (R) that associates, as information for generating instructions to the terminal device 20, information (T) that identifies a trigger, information (V) to be compared with information representing the content of the voice input by the user (hereinafter referred to as comparison string information; however, this comparison string information may not be necessary depending on the type of trigger), and information (S) representing the process executed by the server 10 to generate action information, as illustrated in FIG. 5.

The action information generation unit 46 obtains comparison string information (V, if any) associated with the information (T) specifying the trigger input from the receiving unit 41, and information indicating the process that the server 10 should execute to generate the action information.

If the comparison string information is acquired (if the comparison string information is associated with the information that identifies the trigger), the action information generation unit 46 compares the comparison string information with the string information output by the receiving unit 41. If the action information generation unit 46 determines that the string information output by the receiving unit 41 matches the comparison string information, it executes the process represented by the acquired information to generate action information.

In addition, if comparison string information has not been acquired, the action information generation unit 46 executes the process represented by the acquired information to generate action information.

As a specific example, the action database contains records that associate information specifying a trigger that "there has been voice input by the user" with comparison string information such as "*News [to | is] * [not | tell me | read it out] *" as comparison string information to be compared with information expressing the content of the voice input by the user, and information such as "obtain the news string information from a specified web server on the Internet and instruct to read out the string information."

This comparison string information is also expressed as a regular expression. Therefore, the above string will match string information such as "Tell me about today's news" and "Is there any news?".

When the action information generating unit 46 receives input from the receiving unit 41, for example, information identifying a trigger such as "there has been voice input by the user" and character string information such as "Is there any news?" as information representing the content of the voice input by the user, the action information generating unit 46 searches the action database for a record including information identifying the trigger.

Here, the action information generating unit 46 finds the above record from the action database, and compares the comparison string information contained in the record with the accepted string information. In the above example, the accepted string information "Is there any news?" is determined to match the comparison string information "*News [wo|ha] * [no|tell me|read it out]*", so the action information generating unit 46 compares the information representing the process to be executed by the server 10, which is contained in the above record found by the search, such as "(Step 1) Obtain the news string information from a specified web server on the Internet,
(Step 2) Generate an instruction to read out the character string information, and (Step 3) Generate an instruction to display animation information to be played back when reading out. Then, the process according to this information is executed.

That is, the action information generating unit 46 obtains news character string information from a specified web server on the Internet according to the read information. The action information generating unit 46 also generates an instruction to display animation information including information (which may be the file name of each image data) specifying the image data of the animation to be displayed at the start of execution of the action processes performed in parallel, the image data of the animation to be displayed during the execution, and the image data of the animation to be displayed at the end of the execution.

In this example, the action information generating unit 46 generates action information including an action instruction and parameter information, and outputs the action information to the instruction transmitting unit 47. Here, the action instruction includes an instruction to read out the character string information, and an instruction to display the animation information. In addition, the parameter information includes the acquired character string information and information that identifies the image data of the animation.

In addition, the information representing the process to be executed by the server 10 may include an instruction to "select a conversation sentence." If such an instruction is included, the action information generating unit 46 selects a conversation sentence in accordance with the instruction, for example, in the following manner.

The action information generation unit 46 extracts conversation sentences that satisfy the speech conditions associated with the conversation sentences from among the conversation sentences stored in the conversation sentence queue. Various pieces of information required to determine whether or not the speech conditions are satisfied, such as the current date and time (date and time when the process is being executed) and weather information, can be obtained from an NTP (Network Time Protocol) server or a specified web server via the network.

The action information generating unit 46 selects, for example, randomly one of the conversation sentences extracted as satisfying the speech conditions. The action information generating unit 46 also reads out the conversation history related to the user of the terminal device 20 that sent the request, which is recorded by the conversation history managing unit 45. The action information generating unit 46 then judges whether the selected conversation sentence will result in a natural conversation when spoken following the currently recorded conversation history.

This determination can be realized, for example, by using a neural network that has learned the text of a conversation between humans through machine learning to determine whether or not the sentence is valid as a continuation of the currently recorded conversation history, that is, whether or not there is continuity in the conversation. This type of processing is known as Next Sentence Prediction (NSP). As a neural network for performing next sentence prediction, for example, a model known as BERT (https://arxiv.org/pdf/1706.03762.pdf) can be used. As training data for machine learning for such next sentence prediction, widely known training data and machine learning processing methods using the same can be used, such as a method using a pair of conversation sentences (a first conversation sentence and a second conversation sentence) and information indicating the continuity of the first and second conversation sentences associated with each other.

In this way, when a neural network is used to determine whether a selected conversational sentence is appropriate as a sentence following the currently recorded conversation history, i.e., whether there is continuity in the conversation, the output of the neural network represents that appropriateness as a numerical value. The action information generating unit 46 determines that the conversation is natural (determines that there is continuity) when the numerical value exceeds a predetermined threshold value.

When the action information generating unit 46 determines that there is continuity as described above, it generates an action instruction including an instruction to read out the character string information and an instruction to display the animation information. The action information generating unit 46 also generates parameter information including the character string information of the selected conversational text and information specifying the image data of the animation, and instructs the instruction transmitting unit 47 to send the generated action instruction and parameter information as action information to the terminal device 20.

The instruction sending unit 47 sends the action information generated by the action information generating unit 46 to the terminal device 20 that is the sender of the request information received by the receiving unit 41.

Next, the operation of the control unit 31 of the terminal device 20 will be described. In this embodiment, the control unit 31 is configured to functionally include a request sending unit 51, an action information receiving unit 52, a voice synthesis unit 53, and an action processing execution unit 54, as illustrated in FIG. 6.

When the request sending unit 51 determines that a predetermined trigger has occurred, it collects information necessary for processing in the server 10 and sends request information including the collected information together with information identifying the trigger to the server 10. Specifically, triggers such as voice input by the user or the arrival of a specified time are listed in advance, included in the setting information, and stored in the storage unit 32.

As an example, as shown in FIG. 7, the setting information lists and records, for each trigger, its name (trigger name: N), in association with information related to the processing of the trigger, such as occurrence conditions (C), information (P) that identifies information related to the trigger that is necessary for processing on the server 10, and interval time (T).

When the request sending unit 51 determines that the occurrence condition is satisfied by referring to this setting information, it regards the occurrence of a trigger for which the occurrence condition is satisfied as occurring, and refers to information (P) related to the trigger that is necessary for processing on the server 10.

Then, the request sending unit 51 collects information required for processing on the server 10, which is specified by the referenced information, and sends request information including the collected information and information specifying the trigger that has occurred (which may be the trigger name) to the server 10.

The action information receiving unit 52 receives action information from the server 10 and outputs the received action information to the action processing execution unit 54.

The voice synthesis unit 53 synthesizes voice data based on character string information input from the action processing execution unit 54, which will be described later. The voice synthesis unit 53 outputs the voice data obtained by synthesis to the action processing execution unit 54.

The action processing execution unit 54 extracts the action instruction and parameter information from the action information sent by the server 10, and executes processing according to the action instruction. Specifically, as in the above example, a case will be described where the action information receiving unit 52 receives action information including an instruction to read out the acquired character string information and an instruction to display the animation information, and parameter information including the acquired character string information and image data of the animation.

In this example, the action processing execution unit 54 outputs the acquired character string information to the voice synthesis unit 53 to acquire voice data. The action processing execution unit 54 also reads out image data of an animation specified by information included in the action information from the storage unit 22. The action processing execution unit 54 then outputs the voice data output by the voice synthesis unit 53 to the voice output unit 35 to play a sound, and outputs the image data of the read animation to the display unit 34 to play the image data of the animation.

[Action]
The information processing system 1 of this embodiment has the above configuration and operates as in the following example. In the following example, it is assumed that the storage unit 12 of the server 10 stores, as an action database, information representing the process executed by the server 10 to generate action information for each trigger that is the cause of an action request, in association with the trigger.

In the following example, the action database contains the following information: Information specifying a trigger (T): A user is having a conversation Process to be executed:
(Step 1) Select character string information of a conversation sentence; (Step 2) Generate an instruction to read out the character string information; (Step 3) Generate an instruction to display animation information to be played when reading out. 0 is assumed to be included in the information.

In addition, the storage unit 32 of the terminal device 20 stores, as setting information, for each trigger, information (C) that specifies the information required for processing on the server 10 in relation to the trigger, and the like, in association with the trigger, as shown in FIG. 7.

In the following example, this setting information includes the following information: - Information identifying the trigger (trigger name N): there was voice input by the user - Occurrence condition (C): the user uttered a specific wake word - Information identifying the information required for processing in server 10 (P): character string information of what the user spoke... Here, a wake word is a word such as "Hey, listen to me" or "Wake up" that should be recognized as the start of voice input when spoken by the user, and is determined in advance. The terminal device 20 may remove the character string portion corresponding to this wake word from the character string information of what the user spoke that is required for processing in server 10.

The operation of the server 10 that holds such setting information and the terminal device 20 will be described below with reference to Figures 8 and 9.

When a user speaks to the terminal device 20, such as "Hey, listen. I have a date tomorrow..." (S11 in FIG. 8), the terminal device 20 executes a process of recognizing the user's voice (S12) and acquires character string information corresponding to the voice spoken by the user. As already mentioned, the voice recognition process does not have to be performed by the terminal device 20 itself, but may be performed by accessing a voice recognition processing service via a network.

The terminal device 20 refers to the setting information to check whether any of the trigger occurrence conditions have been satisfied (S13). In this case, since the user has uttered the wake word "Hey, listen to me," it is assumed that a trigger has been generated indicating that "there has been voice input by the user" (S13: Yes), and according to the setting information, character string information of the content uttered by the user is collected. Note that, if it is determined in step S13 that none of the trigger occurrence conditions have been satisfied (S13: No), the terminal device 20 ends the process.

In this case, since the content uttered by the user has already been acquired as character string information in step S12, the terminal device 20 sends request information including the character string information and information identifying the trigger that has occurred (trigger name "Voice input by the user has been received") to the server 10 (S14).

The server 10 receives the request information from the terminal device 20. The server 10 then determines whether the character string information included in the request information is event-related information by comparing the character string information included in the request information with a predetermined pattern of information related to the event (S15).

If the pattern is "* [tomorrow | the day after tomorrow | weekend | next week] [D+day] [W+day] [day] [ha|ni] W+[ga] [aru|nan] *," the accepted character string information is "Hey, listen. Tomorrow is a date...," so the server 10 determines that it matches the pattern. In other words, this character string information is determined to be information related to an event (S15: Yes).

1
On the other hand, if it is determined that the information is not event-related (S15: No), the server 10 executes another process, that is, a process obtained by referring to the action database.

The server 10 then extracts date and time information and event specific information from this character string information (S16). Since "tomorrow" is used here, the server 10 obtains date and time information of "December 16th", which is the day following the date and time when this process is being executed (e.g. December 15th). The server 10 also extracts "date" from the character string information as event specific information.

Next, the server 10 generates a conversation sentence. The server 10 first randomly acquires one number of days from the number of days candidates predefined in the number of days data table as the utterance period of the conversation sentence (S17: Decision of utterance period). Here, it is assumed that "7 days" is acquired as the number of days candidate. In addition, in the process of this step S16, the server 10 acquires the utterance period length (here, "7 days") and the date and time indicator (here, "during this time") recorded in the number of days data table in association with the acquired number of days candidate, and determines the utterance period to be "from December 23rd to December 30th" using the date and time information (December 16th) extracted in step S16, the acquired number of days candidate, and the utterance period length.

In the process of determining the speech period, an example has been given in which one number of days is randomly determined from the number of days candidates predefined in the number of days data table, but this embodiment is not limited to this, and the server 10 may obtain a number of days predefined in association with the event-specific information obtained in step S16 as the speech period.

In this example, for example, if the event-specific information is a "date" (an event that can occur relatively frequently and whose topic life-time is considered to be relatively short), a relatively short number of days is set to "4 days," and if the event-specific information is a "trip" (an event whose topic life-time is considered to be relatively long), a relatively long number of days is set to "14 days," and a predetermined data table is stored in the storage unit 12. The server 10 sets the speech period by referring to this data table.

In another example, the speech period may be determined by referring to a data table that stores multiple candidate days associated with each of the event-specific information acquired in step S16.

In this example, when the event-specific information is a "date" (an event that can occur relatively frequently and whose topic life time is considered to be relatively short), the relatively short number of days are "1 day, 2 days, 3 days, 4 days" as candidate days, and when the event-specific information is a "trip" (an event whose topic life time is considered to be relatively long), the relatively long number of days are included, and a predetermined data table such as "1 day, 2 days, ..., 13 days, 14 days" is stored in the storage unit 12.

Then, the server 10 randomly selects one number of days from the number of days candidates predefined in the data table in association with the event-specific information acquired in step S16, and sets the selected number of days as the speech period.

In this example, if the event specified by the event specification information obtained in step S16 is a "date," a speech period of 1 to 4 days is set, and if the event specified by the event specification information obtained in step S16 is a "trip," a speech period of 1 to 14 days is set.

The server 10 generates a conversation sentence using the date and time noun associated with the selected number of days candidate, the event specific information output by the event information acquisition unit 43, and information on the predetermined conversation sentence pattern candidate (S18). Here, the server 10 selects one from a plurality of predetermined conversation sentence pattern candidates, and generates a conversation sentence, for example, a conversation sentence such as "By the way, how was your date the other day?" by inserting the date and time noun ("the other day") and the event specific information ("date").

The server 10 associates the conversation sentence generated in the process of step S18 with the speech period determined in the process of step S17 as a speech condition, and accumulates the conversation sentence in a conversation sentence queue stored in the memory unit 12 (S19).

By this process, the conversation sentence queue contains the conversation sentences,
(1) Utterance condition: The maximum temperature was 35 degrees or higher, and the time was after 6 p.m. Conversation: "It was hot today."
(2) Utterance condition: The time is between 2:00 and 4:00 a.m. Conversation sentence: "Aren't you going to bed soon?" ... etc.
(n) Utterance condition: within the utterance period from December 23 to December 30. Conversation sentence: "By the way, how was your date the other day?" is stored.

During this operation, the server 10 sequentially records the contents of the voice input by the user and the contents of the conversation spoken by the user's terminal device 20, and generates conversation history information. For example, if the following conversation takes place between the user and the server 10, the conversation history information is stored.

That is, the server 10 records conversation history information such as: User: "Hey, listen. We have a date tomorrow." Terminal device 20's speech: "Oh, really?" User: "I wonder what the weather's like." Terminal device 20's speech: "It looks like it's going to be sunny tomorrow."

Note that the server 10 may execute other processes, i.e. processes obtained by referring to the action database, in parallel with the processes in steps S16 to S19.

After that, for example, on December 24th, if the user speaks to the terminal device 20, such as "Wake me up. Is there any news?" (S21 in FIG. 9), the terminal device 20 executes a process of recognizing the user's voice (S22) and obtains character string information corresponding to the voice spoken by the user.

The terminal device 20 also refers to the setting information to check whether any of the trigger occurrence conditions have been satisfied (S23). In this case, since the user has spoken the wake word, it is assumed that a trigger has occurred indicating that "there has been voice input by the user" (S23: Yes), and according to the setting information, it collects the character string information of what the user said, "Wake up. Is there any news?". The terminal device 20 then sends request information including the character string information and information identifying the trigger that has occurred (trigger name "there has been voice input by the user") to the server 10 (S24).

The server 10 compares the comparison string information "*News [to | is] * [not | tell | read] *" associated with the trigger indicating that "there has been voice input by the user" with the string information "Wake up. Is there any news?" sent by the terminal device 20, and determines whether they match (S25).

In this case, the character string information "Wake up. Any news?" that the server 10 received from the terminal device 20 matches the comparison character string information, so the server 10 determines in step S25 that the comparison character string information matches the received character string information, and starts a process specified by the trigger indicating that "voice input has been made by the user" and the information associated with the comparison character string information.

For example, the server 10 acquires news string information from a specific web server on the Internet, and sends it to the terminal device 20 as action information together with an instruction to read out the string information (S26).

In response to this instruction, the terminal device 20 executes a process of reading out the news information (S27). For example, the news that is read out may be something like, "The warm weather continues from last week, but a cold wave is expected to hit at the end of the year."

The server 10 records the character string information sent and received between the server 10 and the terminal device 20 up to this point as a conversation history. In the above example, the conversation history is recorded as follows: User: "Wake up. Is there any news?" Terminal device 20 utterance: "They say, 'It's been warm since last week, but a cold wave is expected to hit at the end of the year.'"

In the process up to this point, the server 10 determines whether the character string information included in the request information is event-related information by comparing the character string information included in the request information with a predetermined pattern of event-related information. However, since the character string information "Wake up. Is there any news?" does not match the pattern "* [tomorrow | the day after tomorrow | weekend | next week] [D+day] [W+day] [day] [is | is] W+[is] [is | is] *", it is not considered to be event-related information, and the corresponding process is not performed.

After that, when the user further speaks to the terminal device 20, such as "Yes, it certainly was warm last week" (S28), the terminal device 20 acquires character string information corresponding to the voice spoken by the user. Then, the terminal device 20 assumes that a trigger has occurred, for example, that "the user is having a conversation", and sends request information to the server 10 according to the setting information, including the character string information of what the user spoke, "Yes, it certainly was warm last week", and information identifying the trigger that has occurred (S29).

The server 10 determines whether the character string information included in this request information is event-related information by comparing the character string information included in the request information with a predetermined pattern of information related to an event. However, since this character string information does not match the above pattern, it is not determined to be event-related information, and the corresponding process is not performed.

Meanwhile, the server 10 executes a process specified by information associated with information specifying the trigger "users are having a conversation," selects character string information of the conversational text, and sends an instruction to the terminal device 20 to vocalize the character string information (S30).

That is, as shown in FIG. 10, in the process of step S30, the server 10 extracts a conversation sentence that satisfies the four utterance conditions associated with the conversation sentence from among the conversation sentences stored in the conversation sentence queue (S41). The server 10 also selects one of the unselected conversation sentences from among the conversation sentences, for example at random (S42).

On the other hand, the server 10 refers to the conversation history related to the user of the terminal device 20 that sent the request, and determines whether the conversation sentence selected in step S42 will result in a natural conversation when spoken following the currently recorded conversation history (S43).

Specifically, the server 10 obtains a numerical value (the more appropriate the value, the larger the value) that indicates the appropriateness of the selected conversational sentence as a sentence that follows the currently recorded conversation history, through a next sentence prediction process using a BERT model that is a machine-learned version of the text of conversations between humans. The server 10 then determines that the conversation is natural (there is continuity) when the obtained value exceeds a predetermined threshold value, and otherwise determines that the conversation is not natural (there is no continuity).

In step S43, when the server 10 determines that the selected conversational sentence will result in a natural conversation when spoken (S43: Yes), the server 10 generates action information including an instruction to read out the character string information of the selected conversational sentence. The server 10 then transmits the action information to the terminal device 20 (S44).

If it is determined in step S43 that the selected conversation sentence will not result in a natural conversation when spoken (S43: No), the server 10 returns to process S42 and continues the process. If there are no unselected conversation sentences in process S42, the server 10 executes a predetermined process, such as generating action information including an instruction to read out character string information of a predetermined conversation sentence and sending it to the terminal device 20 (S45: Execute default process).

Returning to FIG. 9, the terminal device 20 executes a process of reading out the character string information in accordance with the instructions of the action information (S31).

As a specific example, in the process of step S43 described above, following a conversation history such as: User: "Wake up. Is there any news?" Speech from terminal device 20: "It's been warm since last week, but they say a cold wave is expected to hit at the end of the year," User: "Yeah. It certainly was warm last week," it is determined whether the conversation sentence "By the way, how was your date the other day?" naturally follows as one of the conversation sentences for which the utterance condition is satisfied.

If it is determined that there is continuity in the conversation, the server 10 instructs the terminal device 20 to speak this conversational sentence, so that the overall conversation will be something like this: User: "Wake up. Is there any news?" Speech of the terminal device 20: "They say, 'It's been warm since last week, but a cold wave is expected to hit at the end of the year.'" User: "Yeah. It was certainly warm last week." Speech of the terminal device 20: "Now that I think about it, how was your date the other day?"

In this way, in this embodiment, conversations about past plans appear, allowing for conversations that are tailored to the user, evoking a sense of closeness.

[In the case of a single terminal device]
In the explanation so far, the terminal device 20 recognizes the voice uttered by the user and transmits the obtained character string information to the server 10. Then, the server 10 generates and provides character string information (conversational text) that is the basis of the voice data to be uttered by the terminal device 20.

However, in one embodiment of the present invention, the terminal device 20 may function as an information processing device. In this case, the server 10 is not necessarily required.

In this example, the control unit 31 of the terminal device 20 realizes the functions of a request sending unit 51, an action information receiving unit 52, a voice synthesis unit 53, and an action processing execution unit 54, and also operates as a receiving unit 41, an event management unit 42, an event information acquisition unit 43, a conversation sentence generation unit 44, a conversation history management unit 45, and an action information generation unit 46.

In this case, all of the data stored in the memory unit 12 in the above description, such as the action database and conversation sentence queue, are stored in the memory unit 32 of the terminal device 20.

In this example, the request sending unit 51 outputs the request information to a receiving unit 41 realized by the control unit 31 itself. The receiving unit 41 then accepts the output request information and processes it. Furthermore, the action information generating unit 46 outputs the generated action information to an action information receiving unit 52 realized by the control unit 31 itself.

The output and reception of this information may be performed via a network (communication unit 36) through a local loopback, or may be performed by storing and reading out the information in memory unit 32 without going through a network.

[Speech stop action]
In addition, the terminal device 20 may be configured to interrupt playback of the voice data (including voice data synthesized based on string information received from the server 10) when the user performs a specified operation on the terminal device 20 during playback of the voice data.

For example, if the sensor unit 33 is equipped with a touch sensor, this predetermined operation may be performed by touching the touch sensor. In this example, when the terminal device 20 detects that the user has touched the touch sensor during playback of audio data, the terminal device 20 interrupts playback of the audio data.

In another example, the predetermined operation may be the user uttering a predetermined word. For example, when the terminal device 20 determines that the user uttered the word "shush" requesting quietness while playing back audio data, the terminal device 20 interrupts the playback of the audio data.

In yet another example, the information processing system 1 may be controlled so that a specified conversational sentence is not spoken even if the user does not perform an operation.

Specifically, the sensor unit 33 of the terminal device 20 may be equipped with a human presence sensor (a widely known device, and therefore a detailed description thereof will be omitted) capable of detecting the number of people. Then, information on the number of people detected by this human presence sensor around the terminal device 20 may be used to select a conversation sentence.

In this example, when the server 10 or the terminal device 20, which executes the function of the conversation sentence generation unit 44, generates a conversation sentence based on the information output by the event information acquisition unit 43 and registers it in the conversation sentence queue, the speech condition includes a condition related to the information on the number of people detected by the terminal device 20 in addition to the information on the speech period. As an example, the condition related to the number of people is set to a condition that the number of people is "1" (singular).

When the terminal device 20 selects a conversation sentence on the server 10, the terminal device 20 sends request information to the server 10 including number information indicating the number of people detected in the vicinity.

Then, the server 10 or the terminal device 20, which executes the function of the action information generation unit 46, selects a conversation sentence that also satisfies the condition using the number of people information when extracting a conversation sentence that satisfies the speech condition associated with the conversation sentence from among the conversation sentences stored in the conversation sentence queue.

As in the above example, if the condition for the number of people to be "1" is included as a speech condition for a conversation sentence related to past event information, when there are multiple people around the terminal device 20, the conversation sentence related to past event information will not be selected, thereby reducing the chance of the user's private information being leaked.

[Expression of emotions]
In one example of the present embodiment, the action information may further include information related to the emotion of the character displayed on the display unit 34 of the terminal device 20. This emotion information may be information specifying, for example, joy or sadness. This emotion information may also include information indicating no emotion (flat).

The terminal device 20 refers to the emotion information included in the received action information and selects the image data of the eyes to be displayed on the display unit 34. As an example, when the emotion information is "Sad," the terminal device 20 selects and displays the image data of an animation of crying eyes from the eye image data.

Furthermore, in one example of this embodiment, when determining the emotion information to be included in the action information, the server 10 or the terminal device 20 that generates the character string information to be spoken may determine the emotion based on the words included in the character string information. Such processing can be performed, for example, as a process of determining the polarity of the sentence expressed by the character string information based on the words included in the character string information. Specifically, when speaking news character string information, if the news character string information contains words that represent a tragic incident, the news character string information is included in the action information along with information that specifies the emotion of sadness.

[Delete information]
In addition, in this embodiment, the server 10 or the terminal device 20 storing the conversation sentence queue may delete conversation sentences associated with speech periods that have already passed (speech periods that have ended) from the conversation sentence queue at predetermined timing.

Furthermore, among the information stored in the conversation sentence queue, the conversation sentence stored by the conversation sentence generation unit 44 may be deleted from the conversation sentence queue when the conversation sentence is spoken. This prevents the same conversation sentence from being played multiple times.

[Get from schedule information]
Furthermore, in the above description of the present embodiment, the server 10 or the terminal device 20 functioning as the event information acquisition unit 43 extracts date and time information and event specification information from character string information when character string information representing the contents of the voice input by the user matches a pattern of information related to the event. However, the present embodiment is not limited to this.

In one example of this embodiment, instead of or in addition to the above process, the event information acquisition unit 43 may acquire predetermined schedule information and extract date and time information and event 7 specific information. Specifically, the server 10 or terminal device 20 functioning as the event information acquisition unit 43 receives in advance from the user settings of access rights to a web service in which the user registers schedule information. The terminal device 20 then acquires schedule information from the web service in accordance with the settings.

When the schedule information acquired here includes date and time information and event specific information that identifies an event in which the user is involved on that date and time (e.g., "date" or "work"), the event information acquisition unit 43 extracts the date and time information and event specific information and outputs them to the conversation sentence generation unit 44.

In this example, conversations can be provided based on the user's past schedule, even if it does not appear in the conversation with the user.

[When the speech period is wide]
In the above explanation, the conversation sentence generation unit 44 generates the entire conversation sentence in advance and stores it in the conversation sentence queue. Therefore, the conversation sentence generation unit 44 includes in advance a relative demonstrative pronoun (such as "yesterday" or "the day before yesterday") that indicates the day when the past event, which is the subject of the conversation, occurred, according to the utterance period.

However, this embodiment is not limited to this, and the conversation sentence generation unit 44 may generate a conversation sentence template that indicates only the position of the word indicating the date on which a past event occurred in the conversation sentence and store it in the conversation sentence queue. In this case, after a conversation sentence is selected, in the processing of the action information generation unit 46, when an instruction to speak the selected conversation sentence is generated, a word indicating the date on which the target past event occurred may be generated from that point on.

In this example, the event information acquisition unit 43 causes the event management unit 42 to record the acquired event information and issue an event identifier unique to the event that identifies the recorded event.

When generating a conversation related to the event information (including date and time information and event-specific information) acquired by the event information acquisition unit 43, the conversation generation unit 44 generates a conversation template that indicates only the position of a word indicating the date on which a past event occurred, and stores the template in a conversation queue in association with the corresponding event identifier. Here, the conversation template includes a symbol to be replaced with a word specifying the date and time, such as "How was the date on <date and time>?".

After that, when the action information generating unit 46 selects the conversational sentence associated with this event identifier as the target for speech, the action information generating unit 46 refers to the event identifier associated with the selected conversational sentence and obtains event information (including at least the date and time information) corresponding to the event identifier from the event management unit 42.

Then, a word that specifies the current date and time is generated, and replaced with the code in the selected conversational text that should be replaced with the word that specifies the date and time. This word can be generated based on the difference (the interval in days) between the current date and time and the date and time when the target event occurred, if the interval is predetermined based on the number of days, such as "yesterday" for the 1st, "the day before yesterday" for the 2nd, "the other day" for the 3rd to 6th, "last week" for the 7th to 13th, etc.

Then, the action information generating unit 46 generates action information that includes an instruction to speak a conversational sentence that includes a word that specifies the date and time.

[Features of the embodiment]
The present embodiment is also characterized as follows: That is, one aspect of the present embodiment is an information processing device, comprising: an acquisition means for acquiring event specification information such as "date" that specifies a user's event associated with date and time information; and a conversation sentence generation means for generating a related conversation sentence that is a conversation sentence related to the event specification information, such as "How was your date last week?", in a process of uttering at a date and time after the date and time represented by the date and time information.

This information processing device may be implemented as a terminal device provided for each user, or may be implemented as a server communicatively connected to the terminal device to realize each of the above means.

In this example, the information processing device provides conversational text about past plans, enabling conversation about past events, which allows for conversations of more private nature for the user and elicits a sense of intimacy.

The system further includes a storage means for storing, when a conversational sentence input by a user includes an event-related word such as "date" related to a predetermined event and a date/time-related word specifying a date and time such as "next weekend," the event-specific information related to the event-related word and the date/time information represented by the date/time-related word in association with each other.

In this example, event-related information can be accumulated simply by inputting it as conversation text, without the user having to register it as schedule information or the like.

The conversation sentence generation means may further include an execution means for recording the related conversation sentence, such as "How was your date last week?", in association with information indicating a period during which the related conversation sentence is to be spoken, and for determining whether or not there is continuity between the conversation sentence input by the user and the related conversation sentence when the period includes the current date and time, and for executing speech processing of the related conversation sentence when it is determined that there is continuity.

In this example, conversations about past events appear at times that are deemed to be natural continuations, making the conversation sound more natural.

Here, the execution means may determine whether or not the accepted conversational sentence and the related conversational sentence have continuity by using a neural network trained by machine learning using learning data that associates a first conversational sentence, a second conversational sentence, and information indicating the continuity between the first and second conversational sentences.

Furthermore, while the related conversation sentence is being spoken, if the user performs a predetermined input, such as an action to forcibly stop the speech, the speech may be stopped. This allows the speech to be stopped when it is inappropriate.

The system may further include a playback device that speaks the generated related conversation sentences and a means for detecting a person in the vicinity of the playback device, and may cause the playback device to speak the related conversation sentences only when there is a single person in the vicinity of the playback device. This makes it possible to prevent speech in inappropriate situations.

1 Information processing system, 10 Server, 11 Control unit, 12 Storage unit, 13 Communication unit, 20 Terminal device, 21 Leg unit, 22 Main body unit, 31 Control unit, 32 Storage unit, 33 Sensor unit, 34 Display unit, 35 Audio output unit, 36 Communication unit, 37 Drive unit, 41 Receiving unit, 42 Event management unit, 43 Event information acquisition unit, 44 Conversation sentence generation unit, 45 Conversation history management unit, 46 Action information generation unit, 47 Instruction transmission unit, 51 Request transmission unit, 52 Action information reception unit, 53 Audio synthesis unit, 54 Action processing execution unit.

Claims (6)

A processor is provided.
The processor,
generating a related conversation sentence related to the event identification information, which is to be uttered at a date and time after the event, from event identification information and date and time information that are obtained by recognizing the voice of the user;
After the event, speaking to the user a conversation sentence that was prepared before the generation of the related conversation sentence and the related conversation sentence;
a period for uttering the related conversation sentence is varied according to the event identification information;
Information processing device. The processor,
The information processing apparatus according to claim 1 , wherein the conversation sentence and the related conversation sentence are uttered when an utterance condition is satisfied. The processor,
3 . The information processing device according to claim 1 , further comprising: a processor configured to: (a) generate a conversation sentence and a related conversation sentence; and (b) continue speaking the conversation sentence when a numerical value representing a validity of the conversation sentence as a sentence following a conversation history exceeds a predetermined threshold value. a processor that generates, from event identification information and date and time information that identify an event of the user obtained by recognizing the voice of the user, a related conversation sentence that is to be uttered at a date and time after the event and that is related to the event identification information;
having a processor speak, to the user after the event, a prepared conversation sentence and the related conversation sentence that have been prepared before the generation of the related conversation sentence ;
a period for uttering the related conversation sentence is varied according to the event identification information;
program. a processor generates a related conversation sentence related to the event identification information, which is to be uttered at a date and time after the event, from event identification information and date and time information that identify the event of the user , obtained by recognizing the voice of the user;
a processor, after the event, speaking to the user a prepared conversation sentence and the related conversation sentence that were prepared before the related conversation sentence was generated ;
a period for uttering the related conversation sentence is varied according to the event identification information;
method. A server and a terminal are provided,
The server,
generating a related conversation sentence related to the event identification information, which is to be uttered at a date and time after the event, from event identification information and date and time information that are obtained by recognizing the voice of the user;
After the event, the terminal is caused to speak the conversation sentence that was prepared before the related conversation sentence was generated and the related conversation sentence ;
a period for uttering the related conversation sentence is varied according to the event identification information;
system.

Images