JP7613796B2 - Line-of-sight control device and method, non-transitory storage medium, and computer program - Google Patents

Description

This invention relates to technology for controlling the gaze of an agent, such as a robot, when interacting with a person. This application claims priority to Japanese Application No. 2022-086674, filed on May 27, 2022, and incorporates all of the contents of said Japanese application by reference.

A variety of conversational agents, including robots and virtual agents, are entering society. Opportunities for people to come into contact with conversational agents are increasing. It is becoming more common to see robots and other agents in places where people gather, such as retail stores, hotel lobbies, and train stations.

When agents such as robots converse with people, not only the linguistic information contained in speech but also non-verbal information such as gestures, facial expressions, prosody, and gaze are important for smooth dialogue. Furthermore, linguistic and non-verbal information can be thought of as a medium through which people consciously or unconsciously express their individuality.

Therefore, when agents such as robots converse with people, not only verbal but also non-verbal information is of great significance. In particular, when an agent converses with a person, controlling the agent's gaze is important not only for smooth dialogue but also for allowing the agent to express its individuality.

Non-Patent Document 1 reports that by observing the behavior of two people while they are performing some kind of task, an extrovert and an introvert differ in the amount of time they direct their gaze at the other person and the amount of time they direct their gaze at the object of their work. Non-Patent Document 1 reports the results of an experiment in which the movements of a robot were controlled based on these results to see what impression a person interacting with the robot would have.

However, the report in Non-Patent Document 1 is based on human behavior when performing a specific task in a specific environment. The disclosure in Non-Patent Document 1 does not provide any guidance on how to control a robot's gaze in general interactions.

In response to this, Patent Document 1, listed below, discloses technology relating to controlling a robot's gaze when conversing with one or more people. In the technology disclosed in Patent Document 1, the robot's gaze is directed in the direction of the person who is speaking. However, even if one person speaks while another person is speaking, the robot's gaze is generally not directed in the direction of the new speaker. The robot's level of interest in the speech of the first speaker is calculated, and if this value is lower than a threshold value, the robot's gaze is directed in the direction of the new utterance.

According to non-patent document 1, controlling the robot's gaze in this way has the effect of preventing the robot from moving unnaturally around in a short period of time.

Patent Publication No. 2022-057507

Sean Andrist et al., “Look Like Me: Matching Robot Personality via Gaze to Increase Motivation,” Proceedings of the 33rd annual ACM conference on human factors in computing systems, 2015.

When people are conversing, characteristic movements in the direction of the gaze are observed around the time when the right to speak (turn) passes from one person to another (when changing turns). Not only that, but even when a speaker is speaking, other speakers do not always look in the direction of the speaker. People may look in a direction other than the speaker, or when there are three or more participants, they may look in the direction of another participant who is not speaking. The same is true for the speaker's gaze. Furthermore, the movement of the gaze direction also differs depending on the person's personality (individuality). The technology disclosed in Patent Document 1 does not control the gaze of the robot based on such information. As a result, the gaze movement of the robot obtained is not necessarily natural.

Therefore, there is a demand for a gaze control device that can enable agents such as robots to gaze more naturally when interacting with humans.

A gaze control device according to a first aspect of the present invention is a gaze control device for controlling the gaze of a robot in a multi-person dialogue, and includes a gaze direction setting means for determining the gaze direction of the robot based on a combination of the role of the robot in the multi-person dialogue and the state of the dialogue flow in response to the timing for determining the gaze direction, and a control parameter generation means for generating control parameters for controlling the direction of the robot's face and eyeballs in response to the gaze direction of the robot being determined by the gaze direction setting means.

Preferably, the gaze direction setting means includes a direction determination model storage means for storing a direction determination model that determines the probability that each of the multiple participants will face in a predetermined number of directions for each role, depending on the combination of each role of the multiple participants in the multi-person dialogue and the state of the dialogue flow, a probability distribution extraction means for extracting a probability distribution from the direction determination model depending on the combination of the robot's role and the state of the dialogue flow in response to the timing for determining the gaze direction, and a first sampling means for sampling the robot's gaze direction from the probability distribution extracted by the probability distribution extraction means.

More preferably, the multiple directions of the orientation determination model include multiple participant orientations and an averting direction that is different from any of the multiple participant orientations.

More preferably, the gaze direction setting means further includes an averted gaze direction model storage means for storing an averted gaze direction model consisting of a probabilistic model for probabilistically determining the averted gaze direction in accordance with a combination of the robot's role and the state of the dialogue flow, and a second sampling means for sampling the direction in which the robot averts its gaze from the averted gaze direction model in response to the gaze direction sampled by the first sampling means being the averted gaze direction.

Preferably, the gaze control device further includes a duration calculation unit for calculating the duration of the robot's gaze based on a combination of the robot's role, the state of the dialogue flow, and the gaze direction determined by the gaze direction setting means.

More preferably, the timing for determining the gaze direction is different when the dialogue flow is in a turn alternation state than when it is not.

More preferably, the timing for determining the gaze direction is a predetermined timing during the turn alternation state when the dialogue flow state is a turn alternation state, and when the dialogue flow state is not a turn alternation state, it is the timing at which the duration calculated immediately before by the duration calculation unit expires.

Preferably, the gaze direction setting means includes a direction determination model storage means for storing a direction determination model that determines the probability that each of the multiple participants will face in a predetermined number of directions for each role, depending on a combination of the roles of each of the multiple participants in the multi-person dialogue, the state of the dialogue flow, and the personality assumed for the robot; a probability distribution extraction means for extracting a probability distribution from the direction determination model depending on the combination of the robot's role, the state of the dialogue flow, and the personality, in response to the timing for determining the gaze direction, and a first sampling means for sampling the robot's gaze direction from the probability distribution extracted by the probability distribution extraction means.

More preferably, the multiple directions of the orientation determination model include multiple participant orientations and an averting direction that is different from any of the multiple participant orientations.

More preferably, the gaze direction setting means further includes an averting direction model storage means for storing an averting direction model consisting of a probabilistic model for probabilistically determining the averting direction according to a combination of the robot's role, the state of the dialogue flow, and the personality, and a second sampling means for sampling the direction in which the robot averts its gaze from the averting direction model in response to the gaze direction sampled by the first sampling means being the averting direction.

Preferably, the gaze control device further includes a duration calculation unit for calculating the duration of the robot's gaze based on a combination of the robot's role, the state of the dialogue flow, the personality, and the gaze direction determined by the gaze direction setting means.

More preferably, the timing for determining the gaze direction is different when the dialogue flow is in a turn-changing state than when it is not.

More preferably, the timing for determining the gaze direction is a predetermined timing during the turn alternation state when the dialogue flow state is a turn alternation state, and when the dialogue flow state is not a turn alternation state, it is the timing at which the duration calculated immediately before by the duration calculation unit expires.

A gaze control method according to a second aspect of the present invention is a gaze control method implemented by a computer for controlling the gaze of a robot in a multi-person dialogue, and includes the steps of: determining the gaze direction of the robot based on a combination of the robot's role in the multi-person dialogue and the state of the dialogue flow, in response to the timing for determining the gaze direction by the computer; and generating control parameters for controlling the orientation of the robot's face and the direction of its eyes, in response to the robot's gaze direction being determined in the gaze direction determining step.

A computer program according to a third aspect of the present invention is a computer program for controlling the gaze of a robot in a multi-person dialogue, which causes the computer to function as a gaze direction setting means for determining the gaze direction of the robot based on a combination of the robot's role in the multi-person dialogue and the state of the dialogue flow in response to the timing for determining the gaze direction, and a control parameter generation means for generating control parameters for controlling the orientation of the robot's face and the direction of the eyes in response to the gaze direction of the robot being determined by the gaze direction setting means.

As described above, this invention provides an eye gaze control device, method, and computer program that enable an agent, such as a robot, to gaze more naturally when interacting with a person.

FIG. 1 is a schematic diagram showing the setup of the preliminary experiment. FIG. 2 is a schematic diagram for explaining a method of tagging the utterances of each participant in the preliminary experiment. FIG. 3 is a schematic diagram for explaining the timing of turn-taking in speech. FIG. 4 is a graph showing the frequency of gaze directions of each speaker during turn-taking in a three-party dialogue in a preliminary experiment. FIG. 5 is a graph showing the proportion of time in the gaze direction of each speaker other than at turn-taking times in a three-party dialogue in a preliminary experiment. FIG. 6 is a diagram showing, in the form of a table, the proportion of gaze directed toward each of the other participants during speech and the duration of gaze averting for each participant in a conversation. FIG. 7 is a graph showing the proportion of time spent in the gaze direction of a specific extroverted participant in each of the participation roles in the dialogue in a three-way dialogue in a preliminary experiment. FIG. 8 is a graph showing the proportion of time spent in the gaze direction of a specific introverted participant in each of the participation roles in the dialogue in a three-way dialogue in a preliminary experiment. FIG. 9 is a graph showing the distribution of gaze time directed at each participant when an extroverted dialogue participant was the speaker. FIG. 10 is a graph showing the distribution of gaze time for each participant when an introverted dialogue participant is the speaker. FIG. 11 is a graph showing the frequency of participants of each role averting their gaze while speaking. FIG. 12 is a graph showing a histogram indicating the distribution of the time when dialogue participants look away and an approximation curve thereof. FIG. 13 is a diagram showing the proportion of the pupil positions when the dialogue participants look away. FIG. 14 is a diagram showing the proportion of pupil positions when an extroverted dialogue participant looks away. FIG. 15 is a diagram showing the proportion of pupil positions when an introverted dialogue participant looks away. FIG. 16 is a block diagram showing a hardware configuration of a conversational robot system 150 according to an embodiment of the present invention. FIG. 17 is a diagram showing the external appearance of the robot shown in FIG. FIG. 18 is a block diagram showing the hardware configuration of the computer shown in FIG. FIG. 19 is a block diagram showing the functional configuration of a line of sight control device realized by the robot control device shown in FIG. FIG. 20 is a schematic diagram showing the configuration of the gaze direction model shown in FIG. FIG. 21 is a diagram showing an example of the gaze direction model at the time of speaking by personality and role shown in FIG. FIG. 22 is a schematic diagram showing an example of the configuration of the gaze aversion model shown in FIG. FIG. 23 is a schematic diagram showing an example of the gaze direction model for each individual, when changing turns, and when looking away, shown in FIG. FIG. 24 is a flowchart showing a control structure of a computer program causing a computer to function as a line of sight control device in the embodiment of the present invention. FIG. 25 is a flowchart showing a control structure of a computer program causing a computer to execute the state sensing steps shown in FIG. FIG. 26 is a flowchart showing a control structure of a computer program causing a computer to execute the steps of determining the line of sight direction and duration shown in FIG. FIG. 27 is a flowchart showing a control structure of a computer program causing a computer to execute the line of sight direction determining step shown in FIG. FIG. 28 is a flowchart showing a control structure of a computer program causing a computer to execute the gaze duration determining step shown in FIG. 26. FIG. 29 is a flowchart showing a control structure of a computer program causing a computer to execute the steps of determining the gaze direction when the gaze is averted, shown in FIG. FIG. 30 shows the layout of video screens for the evaluation experiment. FIG. 31 is a graph showing the results of the evaluation experiment. FIG. 32 is a graph showing the results of the evaluation experiment.

In the following description and drawings, identical parts are given the same reference numbers. Therefore, detailed description thereof will not be repeated. In the following embodiment, a human-shaped robot is used as the agent. However, the present invention is not limited to such an embodiment. The robot does not necessarily have to be human-shaped, and can be of any shape as long as it has eyeballs and is assumed to be able to speak. Furthermore, the present invention can be applied to virtual agents that do not have a three-dimensional entity like a robot, for example, virtual agents that are represented as two-dimensional images, or virtual agents that are represented as three-dimensional images in a virtual space.

Part 1 Preliminary Experiment 1. Purpose of Data Collection In order to make the gaze movement of a robot during multiple conversations natural, it is necessary to investigate the gaze of real humans and collect the necessary information. For this purpose, we conducted the following preliminary experiment. Figure 1 shows the setup of the preliminary experiment.

With reference to FIG. 1, the preliminary experiment was conducted as a three-way dialogue 50. The participants in the three-way dialogue 50 are participants 60, 62, and 64. In FIG. 1, participants 60, 62, and 64 are all standing, but in the preliminary experiment, the direction of gaze is important, and it is necessary to fix each other's positions. For this reason, in the preliminary experiment, chairs were prepared as described below, and participants were asked to sit on those chairs and talk. In the preliminary experiment, these participants were asked to talk freely in a form without any particular purpose and without a clear way of proceeding with the discussion, like a chit-chat, and information on the direction of gaze of each participant at that time was collected. In such a dialogue, the conversation often flows lively, even though there are no clear rules for how to proceed with the discussion. In order for an agent to enter society, it is desirable to enable a wide range of interactions, from purposeful dialogue such as a meeting to casual dialogue.

When a three-way dialogue like this is taking place, who speaks, or in other words who has the right to speak, is of great significance. Here, this right is called the right to speak, and the participant who has the right to speak is called the speaker. The other participants take on the role of listeners, which we will call listeners. Speakers take turns as appropriate. This shift in speaking rights is called turn-taking. Each time a turn takes place, each participant in the dialogue becomes either the speaker or the listener. The position of each participant taking part in the dialogue in this way is called a participant role, or simply a role.

Past research has shown that participants in a dialogue consciously or unconsciously recognize their own role depending on the situation of the dialogue. That is, each participant is always aware of whether he or she is the speaker, the listener who is primarily addressed by the speaker, or a listener who is only slightly involved in the conversation. Here, the listener who is primarily addressed by the speaker is called the main listener (ML), and the listener who is only slightly involved in the conversation is called the sub-listener (SL).

Previous research has also shown that there is a correlation between the role of each participant in a conversation and their eye movements. Following this idea, in the preliminary experiment, we collected information on how participants in a three-way dialogue 50 moved their eyes according to their roles.

2. Data analysis (labeling)
A. Data Set A total of nine people, both male and female, participated in the preliminary experiment. These participants formed six groups, each with three participants. In each group, the three participants freely conversed. The content of the conversation was free talk. The duration of the conversation ranged from 20 to 30 minutes.

Participants in each group sat in three chairs, positioned at the three vertices of a triangle, while they spoke. A camera was placed in front of each participant, capturing their facial and body movements from directly in front of them. Each participant was equipped with a headset microphone, which collected audio data from each participant as well as a video of the entire discussion. After the dialogue ended, a transcript of the dialogue was created from these recordings and audio data.

Next, annotators labeled the transcripts, paying attention to gaze direction, eye direction, speaking rights, turn taking, and each participant's role in the dialogue. Figure 2 shows an example of the resulting labeled dialogue data. The following analysis was performed on the resulting labeled dialogue data.

B. Data Analysis FIG. 2 shows examples of gaze label strings 80, 82, and 84 for three participants A, B, and C. With reference to FIG. 2, for example, the label "B" in the gaze label string 80 for participant A indicates that participant A was looking at participant B during this time period. Similarly, the label "C" in the gaze label string 80 indicates that participant A was looking at participant C. "Looking away" in the gaze label string 80 indicates that participant A was not looking at either participant B or participant C. In other words, this label indicates that participant A was looking away from other participants during this period. In this specification, a participant not looking at other participants is referred to as "looking away," and the duration of this is referred to as "looking away duration."

Gaze label sequences 82 and 84 were created in the same manner as gaze label sequence 80.

C. Eye direction When a person looks at another person, there is not much deviation between the label and the face and pupil position. On the other hand, when looking away, the gaze, especially the pupil position, is very important. However, considering that there is a limit to the accuracy of determining the gaze direction, in this preliminary experiment, the eye direction corresponding to the pupil position was labeled in nine directions consisting of up, down, left, and right, including the center and diagonal to the center.

Below the part of the gaze label column 80 in Figure 2 that says "averted gaze," additional information is provided, which is a label indicating in which direction participant A's gaze was directed at that time.

For example, the period marked "up" indicates that Participant A's gaze was directed upwards, and the period marked "upper right" indicates that Participant A's gaze was directed to the upper right. In this case, the gaze is determined not only by the angle of the face but also by the direction of the eyes, and the direction determined comprehensively from these two elements is the gaze direction. Participant B and C were also labeled in the same way.

D. Presence or Absence of Speaking Rights and Turn Taking Figure 3 shows an actual turn taking situation. In this example, participant A first has the speaking right and makes an utterance 100. This is indicated by a turn taking label 102. Next, speaker C takes the speaking right and makes an utterance 104. This is indicated by a turn taking label 106. The timing of the turn taking is determined by the annotator. For example, it is clear from the turn taking label 106 that the speaking right has been transferred from participant A to participant C.

In this experiment, the turn alternation 108 lasted a total of two seconds, including one second before and after the moment when the two participants took turns speaking, and in particular, the movement of the gaze direction of each participant during this period was analyzed using gaze labels.

E. Roles In gaze analysis of a three-way dialogue, the role of each participant in the dialogue is important. In this preliminary experiment, the speaker, main listener, and sub-listener were defined as described above, and gaze analysis was performed for each of these separately during turn-taking and other times. The right to speak moves during turn-taking. In order to clarify the role of each participant during turn-taking, here, the speaker during turn-taking refers to the person who takes the right to speak due to the turn-taking. In the case of FIG. 3, participant C is the speaker. The main listener during turn-taking refers to the person who had the right to speak one turn ago and handed it over to the speaker during the turn-taking. In the case of FIG. 3, participant A is the main listener. The person who was not involved in this change of the right to speak is the sub-listener. In the case of FIG. 3, participant B is the sub-listener.

F. Analysis Results a. When changing turns Figure 4 shows the transition of the rate of eye movement for each role when changing turns. In Figure 4, the horizontal axis is time (unit: seconds). At time t = 0.0, the speaker takes the turn and starts speaking. The vertical axis shows the rate (0.0-1.0) based on statistics of where the speaker is looking at time t. For example, looking at the values on the vertical axis at t = 0.0 seconds on the horizontal axis in Figure 3 (A), the ML, SL, and averted gaze are 0.37, 0.19, and 0.44, respectively. This indicates that of the total distance the speaker was looking at when he started speaking, 0.37 was looking at the main listener, 0.19 was looking at the sub-listener, and the remaining 0.44 was looking away. The gaze direction was calculated every 0.1 seconds, and Figure 3 shows how the proportion of the speaker's gaze direction changes over time during a period of 2 seconds, 1 second before and 1 second after the speaker takes the floor and begins to speak.

In the embodiment described below, the 2-second period during turn alternation is divided into three periods, from -1.0 seconds to -0.3 seconds, -0.3 seconds to 0.3 seconds, and 0.3 seconds to 1 second, with the timing of the turn alternation as the origin (0 seconds), and the gaze direction is determined at the beginning of each period. Therefore, in the preliminary experiment, statistics are taken separately for each of these periods to create a model.

・Speaker (SP)
FIG. 4A shows the transition of the rate of gaze of the speaker (SP) during turn taking. One second before taking a turn, the speaker has a high rate of looking at the main listener (ML), i.e., the previous speaker. However, as the speaker gets the right to speak and starts speaking (t=0.0 seconds), the rate of looking away from the main listener increases. Then, the rate of the speaker looking away reaches a peak about 0.1 seconds after the speaker starts speaking. After that, the rate of the speaker looking toward the main listener and the rate of looking away become almost equal, and the transition of gaze during turn taking ends. From this result, people tend to look at the previous speaker at the beginning of a turn taking, look away when they start speaking, and then keep looking away or look at the previous speaker, the main listener.

・Main Listener (ML)
4B shows the transition of the gaze ratio of the main listener who had the right to speak until just before the turn change. When the turn change occurs, various situations are considered, such as the main listener handing over the right to speak to the speaker, the speaker taking it, or the speaker naturally changing over. However, overall, the main listener knows or has already decided who the next speaker will be before the next speaker starts speaking, and the main listener tends to keep looking at the speaker during the speaker change period from one second before the turn change to one second after handing over the right to speak.

・Sub-listener (SL)
Finally, Figure 3(C) shows the transition in the proportion of gazes of sub-listeners who were not involved in the turn-taking. In Figure 4(C), at t = -1.0 seconds on the horizontal axis, that is, until one second before the speaker speaks, the proportion of sub-listeners looking at the main listener and the speaker is equally high. However, from there to t = 1.0 seconds, that is, one second after the speaker starts speaking, the proportion of sub-listeners looking at the speaker increases. Therefore, it is thought that sub-listeners tend to guess who will be the next speaker from one second before the turn-taking until the turn-taking, and until then, they look toward the main listener or the next speaker, and then move their gaze to the next speaker.

b. Outside turn-taking (utterance period)
The time other than the turn-taking time is referred to as the speech period. The beginning and end of the speech period are considered to be affected by the turn-taking. Therefore, in the following gaze analysis of the speech period, the first and last two seconds of the speech period were excluded from the analysis. Figure 5 shows the proportion of gaze directions of each participant during the speech period. Figure 6 shows the specific figures of the gaze proportions in the speech section shown in Figure 5 in a table format.

In Figure 5, the horizontal axis represents the direction of gaze, and the vertical axis represents the percentage of time spent looking in each direction relative to the total time.

5A, the speaker has a slightly higher rate of looking at the main listener while speaking. However, the speaker generally distributes his/her gaze in a balanced manner.

Main listener: Referring to Fig. 5B, the main listener faces the speaker for nearly 70% of the speaking period, and looks at the sub-listeners very rarely. The main listener looks away much more often than the sub-listeners, but much less often than the speaker.

Sub-listeners: Referring to Figure 5C, the tendency of sub-listeners is almost the same as that of main listeners. That is, sub-listeners also look in the direction of the speaker for nearly 70% of the speaking period. The percentage of looking away is higher than the percentage of looking at the main listener, but lower than the percentage of looking away by the main listener.

c. Eye Movement Due to Personality Based on the concern that eye movement may differ depending on personality, another preliminary experiment (second preliminary experiment) was conducted in addition to the above preliminary experiment (first preliminary experiment) to analyze eye movement related to personality. In the second preliminary experiment, a three-way dialogue was conducted with 17 participants, both male and female. A total of 14 sessions were conducted, and each session involved free conversation for about 10 to 20 minutes. The three speakers sat on three chairs arranged at the three vertices of an equilateral triangle with sides of about 2 meters each, and wore headset microphones and acceleration sensors. In addition to these, the movements of each participant were recorded using an image depth camera.

There is an index called the BIG5 that is used to express individuality. The BIG5 explains human personality characteristics in five dimensions, one of which is "extraversion." In this preliminary experiment, we focused on "extraversion" and aimed to investigate how participants' eye movements differed depending on their extroversion.

In this experiment, two participants who gave different impressions of extroversion were selected from the 17 participants, and the gaze movements of each were analyzed from the data obtained from these two participants. In the following explanation, these two participants are referred to as Speaker A and Speaker B. Speaker A gave the impression of being more extroverted, while Speaker B gave the impression of being more introverted. By constructing different models using the analysis results of the gaze of these two people, it may be possible to express different personalities using agents such as robots.

As a speaker Figure 7(A) shows the percentage of time speaker A looked at the main listener (ML), the sub-listener (SL), and the gaze averted (GA) when he was the speaker (SP). Figure 8(A) similarly shows the results obtained for speaker B.

From Figure 7 (A), we can see that when Speaker A is the speaker, he or she spends a relatively high proportion of time looking not only at the main listener, but also at the sub-listener. In other words, the proportions of time they spend looking at the main listener and the sub-listener are roughly equal, and the proportions of time they spend looking away are also roughly the same. Furthermore, compared to Figure 5 (A), which shows the results of a previous preliminary experiment that did not distinguish between personalities, the probability that Speaker A looks at the main listener is relatively low.

On the other hand, Figure 8 (A) shows that when Speaker B is the speaker, he or she tends to look away more often and less at the sub-listener than Speaker A. In particular, when compared to Figure 5 (A), it can be seen that Speaker B spends a lot of time looking away.

As the main listener, Figure 7B shows the percentage of time spent looking at the speaker, the percentage of time spent looking at the sub-listener, and the percentage of time spent looking away when speaker A was the main listener. Figure 7B also shows the results obtained for speaker B.

Referring to Figure 7 (B), it can be seen that when speaker A is the main listener, they spend a very large proportion of their time looking at the speaker, and they tend to look away from time to time. On the other hand, referring to Figure 8 (B), it can be seen that speaker B, like speaker A, spends a large proportion of their time looking at the speaker, but they also look away much more than speaker A. It can also be seen that the proportion of time that speakers A and B spend looking at the sub-listener is low.

Comparing Figures 7(B) and 8(B) with Figure 5(B), we can see that they show the same tendency. However, it is noteworthy that Speaker B, as shown in Figure 8(B), spends a lot of time looking away.

As a sub-listener Figure 7C shows the percentage of time that speaker A looked at the speaker, the main listener, and the time that he looked away when he was a sub-listener. Figure 8C shows the results for speaker B in a similar way.

As can be seen from Figure 7 (C), when speaker A is the sub-listener, as with the main listener, the proportion of time spent looking at the speaker is very high, and the proportion of time spent looking at the main listener or looking away is quite low. On the other hand, as can be seen from Figure 8 (C), when speaker B is the sub-listener, the proportion of time spent looking at the speaker is high, but it is lower than that of speaker A. Instead, in the case of speaker B, the proportion of time spent looking away is high, and is equal to the proportion of time spent looking at the speaker.

Comparing these with Figure 5 (C), the difference in the tendencies between Speaker A and Speaker B becomes even clearer. That is, Speaker A spends a greater proportion of time looking at the speaker and averts his gaze less than when personality is not taken into account. Conversely, Speaker B spends a lesser proportion of time looking at the speaker and a greater proportion of time averting his gaze than when personality is not taken into account. Furthermore, in the case of Speaker B, the proportion of time spent looking at the main listener is also significantly lower.

d. Gaze duration by personality For each of speaker A and speaker B, the distribution of gaze duration during the speaking period (period other than the turn-taking period) when they were in the role of speaker, main listener, and sub-listener was analyzed. The results are shown in Figures 9 (A) to (C) and Figures 10 (A) to (C), respectively. These figures show histograms of gaze duration under each condition and distribution curves approximated by the ^χ2 distribution expressed by the following formula:

上式においてｎはχ^２分布の自由度であり、ｓｃａｌｅは分布曲線の縦軸方向の振幅の大きさを、ｌｏｃは分布曲線の横軸方向のバイアスを、それぞれ決めるパラメータである。

In the above equation, n is the degree of freedom of the ^χ2 distribution, scale is a parameter that determines the magnitude of the amplitude in the vertical axis direction of the distribution curve, and loc is a parameter that determines the bias in the horizontal axis direction of the distribution curve.

For each of Speaker A and Speaker B, we calculated the distribution of three types of duration: the duration of looking at the main listener when speaking, the duration of looking at the sub-listener, and the duration of looking away. Figure 9 shows the results for Speaker A, and Figure 10 shows the results for Speaker B.

With reference to FIG. 9(A), for speaker A, the approximation curve of the distribution of the time spent looking at the main listener is given by n=3.4, loc=0.15, scale=0.2. The approximation curve of the distribution of the duration of looking at the sub-listener shown in FIG. 9(B) is given by n=4.34, loc=0.0, scale=0.18. Therefore, when speaker A is the speaker, the duration of looking at the interlocutor (main listener and sub-listener) tends to be longer than when speaker B is the speaker. On the other hand, the approximation curve of the distribution of the duration of looking away by speaker A shown in FIG. 9(C) is n=4.67, loc=0.0, scale=0.12. From this result, it can be seen that the duration of looking away tends to be shorter in the case of speaker A.

From the above, we can see that when Speaker A is the speaker, he or she tends to look at the interlocutor for longer periods of time and occasionally look away for shorter periods of time.

In contrast, in the case of speaker B, referring to FIG. 10(A), the approximation curve of the distribution of the time spent looking at the main listener is given by n=7.9, loc=0.0, scale=0.1. Also, the approximation curve of the distribution of the duration of looking at the sub-listener shown in FIG. 10(B) is given by n=4.7, loc=0.12, scale=0.12. From these results, it can be seen that in the case of speaker B, both the duration of looking at the main listener and the duration of looking at the sub-listener are shorter. Furthermore, the approximation curve of the distribution of the duration of looking away shown in FIG. 10(C) is given by n=3.28, loc=0.1, scale=0.2. From these results, it can be seen that in the case of speaker B, the duration of looking away tends to be longer.

From the above, we can see that when speaker B is the speaker, the duration of gaze averting is longer and there is a tendency for the listener to occasionally look briefly at the main listener and sub-listener.

Details about the distribution of gaze direction duration when Speaker A and Speaker B were the main listener and sub-listener, respectively, are not provided here. However, a similar distribution trend of duration was observed when these speakers were the speakers.

e. Averting Eyes - Interval of Averting Eyes Figure 11 shows a scatter plot of the number of times participants looked away while speaking in the first preliminary experiment. The horizontal axis indicates the time during speaking. The vertical axis indicates the number of times they looked away from a person. As mentioned above, the time on the horizontal axis is the time excluding the beginning and end of the speech. Figure 11 also shows an approximation line of the averting eyes. The regression coefficient r of the approximation line is 0.83, 0.82, and 0.67 for Figures 11 (A), (B), and (C), respectively. It can be seen from Figures 11 and 6 that the interval of averting eyes does not change regardless of the role.

- Eye direction The movement of the eyes when a person looks at another person is very simple. Therefore, when a robot looks at a person, it is sufficient to turn the robot's face towards the other person in a way that does not seem unnatural. However, it was found that when a robot averts its gaze from a conversation partner, a simple movement is not sufficient. Looking at the data of an actual recorded three-way conversation with a human, it was found that in the case of a human, the gaze was averted by moving only the eyeballs. For this reason, in order to analyze the human eye movement required when a robot averts its gaze, we focused in particular on where the human eyeballs are directed when averting their gaze. We also analyzed the distribution of time for which a person averts their gaze when averting their gaze.

Duration of gaze aversion Figure 12 shows a histogram of the duration of gaze aversion. This distribution is approximated by the following exponential distribution:

ただしμ＝０．２、λ＝０．６３である。なお、０．２秒未満のサンプルは実際のエージェントへの実装を考えたとき、非現実的だと考え、０．２秒以上のサンプルに限って分析した。解析対象となったサンプルの継続時間の平均値は０．８３秒、中央値は０．５５秒だった。

where μ = 0.2 and λ = 0.63. Samples shorter than 0.2 seconds were considered unrealistic when considering implementation in an actual agent, so only samples longer than 0.2 seconds were analyzed. The average duration of the samples analyzed was 0.83 seconds, and the median was 0.55 seconds.

- Eye distribution when looking away When people look away from a conversation partner, they mainly move their pupils (eyeballs) and not just their face. We also analyzed how many times people move their eyes when looking away and what pattern they follow.

From the results of the first preliminary experiment, the eye direction when averting gaze for each speaker was classified into nine directions: {forward, up, down, left, right, upper left, upper right, lower left, lower right}, and the proportions were calculated. The results are shown in Figure 13. Referring to Figure 13, it can be seen that when people avert their gaze, they tend to avert their pupils to two locations: the center and the lower center. It can also be seen that, although there is some bias, the pupils are dispersed to the other seven locations in roughly equal proportions when people avert their eyes.

Pupil distribution by personality The results of the second preliminary experiment were analyzed for pupil distribution when looking away in the same manner as the first preliminary experiment. Figure 14 shows the results for speaker A, and Figure 15 shows the results for speaker B.

With reference to Figures 14 and 15, in the second preliminary experiment, as in the results of the first preliminary experiment, it can be seen that both Speaker A and Speaker B look downward more frequently than upward when looking away. However, Speaker B tends to look downward more frequently. Both speakers' eyes face forward (i.e., their pupils are located in the center) relatively frequently. In the three-way dialogue data obtained in the second preliminary experiment, the interlocutors are located diagonally to the left and right. Therefore, when a speaker's eyes face forward, it means that the speaker is looking away from both of his or her dialogue partners.

In addition, in Figures 14 and 15, the proportion of pupils that are deflected to the lower right is higher than in Figure 13, and is also higher than the other six locations that are neither the center nor the lower center. The cause of this bias is unknown. However, in light of the results in Figure 13, in the embodiment described below, we have implemented the case of Figures 14 and 15 so that when the gaze is deflected, the pupils are almost evenly directed to the seven locations other than the center and the lower center.

- Number of times of looking away In both the first and second experiments, when the dialogue participants looked away, they tended to move their eyes multiple times, rather than staring at one point for a long time. The longer the time of looking away, the more the number of times of eye movement tended to increase. Based on the results of analyzing the distribution of the duration of each number of times of looking away, in the embodiment described below, the number of times the robot looked away was increased by one for every 0.7 seconds, i.e., once when the duration of looking away was 0.7 seconds or less, twice when it was 1.4 seconds or less, and three times when it was 2.1 seconds or less.

Second embodiment 1. System configuration With reference to FIG. 16, the configuration of a conversational robot system 150 employing a gaze control system according to an embodiment of the present invention for controlling a robot 160 will be described below. In order for an agent such as a robot to converse with a person, it is necessary to implement a voice recognition function, a voice synthesis function, and a dialogue function using these functions. In this embodiment, it is assumed that these functions are existing. In addition, in order to conduct a three-way dialogue, it is necessary to dynamically recognize turn taking and the roles of each dialogue participant. Although it is difficult to recognize turn taking, it is possible to use a technology for recognizing the timing of turn taking, for example, as described in the following reference document.

Chaoran Liu et al, “Turn-taking Estimation Model Based on Joint Embedding of Lexical and Prosodic Contents”, INTERSPEECH 2017, [Online], August 20-24, 2017, Stockholm, Sweden, [Retrieved May 3, 2022], Internet <URL:https://isca-speech.org/archive_v0/Interspeech_2017/pdfs/0965.PDF>
In the above-mentioned reference, a model is used to detect the timing of turn alternation. In contrast, in the present embodiment, it is necessary to detect one second before the timing of turn alternation. However, even in this case, by setting the timing to be detected to one second before the turn alternation and training the model using the method described in the above-mentioned reference, it is possible to detect one second before the timing of turn alternation.

In the following embodiments, the recognition of the roles of the participants, including the robot, and the recognition of whether the robot should take action to acquire the right to speak are assumed to be given from the outside. However, for example, who is speaking can be identified by grasping the position of the speaker and then localizing the sound source as described below. Also, who is the main listener can be estimated based on the facial images of the dialogue participants and the proportion of time that the speaker is facing each participant. If the robot is the main listener, it can simply start speaking when a turn change is detected. When the robot is the speaker, a method can be considered in which the participant who is facing the robot for a longer period of time from the facial images of the other participants is designated as the main listener.

Figure 16 shows the hardware configuration of a conversational robot system 150 according to this embodiment. Referring to Figure 16, the conversational robot system 150 includes a probability model storage device 172 that stores a probability model for controlling the gaze of the robot 160, an operation control PC (Personal Computer) 162 for controlling the gaze-related operation of the robot 160 using the probability model stored in the probability model storage device 172, and a network 176 to which the operation control PC 162 is connected. The robot 160 according to this embodiment is for carrying out a three-way dialogue with two other dialogue participants.

The conversation robot system 150 further includes a human position sensor 178 for detecting the position of the conversation partner, and a human position recognition PC 180 connected to the human position sensor 178 and the network 176. The human position sensor 178 is for detecting the position of the conversation partner and providing a detection signal to the human position recognition PC 180. The human position recognition PC 180 is for calculating the position of the conversation partner as seen by the robot 160 based on the signal from the human position sensor 178, and transmitting the calculated position to the operation control PC 162 via the network 176.

The conversation robot system 150 further includes a microphone 164 and a voice processing PC 166 for recognizing the contents of the dialogue participants' utterances, turn changes, and the roles of each participant based on the voice signal received from the microphone 164, and transmitting the results to the operation control PC 162 via the network 176. The microphone 164 is a microphone array, and the voice processing PC 166 can identify the position of the sound source based on the output of the microphone 164. The voice processing PC 166 has a function of transmitting to the operation control PC 162 information indicating the position of the person who made the utterance, along with the spoken text, turn change detection information, and the role of each participant. The operation control PC 162 has a function of determining whether or not it should acquire the right to speak based on the contents of the spoken text and the detection result of the timing of turn changes and the role recognition result by the voice processing PC 166.

The conversational robot system 150 further includes a voice synthesis PC 170 for receiving spoken text and generating a voice signal corresponding to the text, and a speaker 168 for receiving the voice signal from the voice synthesis PC 170 and converting it into voice. The voice quality of the voice synthesis by the voice synthesis PC 170 is selected to be suitable for the appearance of the robot 160.

The conversation robot system 150 further includes an interaction PC 174 that is connected to the network 176 and generates and outputs a response to an utterance in response to a speech text received from another PC via the network 176. The operation control PC 162 has a function of sending the text of the speaker's utterance to the interaction PC 174 when the voice processing PC 166 determines that the robot 160 should acquire the right to speak. The interaction PC 174 generates a speech text in response to this input and sends it to the operation control PC 162 and the probability model storage device 172, so that a voice corresponding to the speech text is generated by the voice synthesis PC 170 and the speaker 168, and the operation control PC 162 can control the movement of the voice processing PC 166 in response to the utterance.

The voice processing PC 166 includes a voice recognition unit 190 that performs voice recognition on the voice signal output by the microphone 164, and further performs sound source localization based on the output of the microphone 164, and transmits text obtained as a result of the voice recognition and information indicating the position of the speaker to the operation control PC 162. The voice processing PC 166 further includes a turn recognition unit 192 that performs a process of detecting turn alternation according to the method disclosed in the above-mentioned reference document using the text output by the voice recognition unit 190 and prosodic information including the voice power and fundamental frequency F0 obtained from the voice signal of the microphone 164, and a process of recognizing the role of each participant in the turn alternation, and transmits the results to the operation control PC 162.

Figure 17 shows the external appearance of robot 160. As shown in Figure 17, robot 160 is a relatively small robot, and its upper body and head can rotate left and right. Furthermore, the eyes of robot 160 have large pupils, and by controlling the rotation angle of the eyeballs, the line of sight of robot 160 can be moved up, down, left and right.

2. Hardware Configuration Fig. 18 is a hardware block diagram of a computer system that operates as, for example, the operation control PC 162 shown in Fig. 16. The voice processing PC 166, voice synthesis PC 170, and human position recognition PC 180 shown in Fig. 16 can also be realized by a computer system having a configuration substantially similar to that of the operation control PC 162. Here, only the configuration of computer system 250 will be described as a representative of these PCs 162, and the configuration of each PC will not be described in detail.

18, this computer system 250 includes a computer 270 having a DVD (Digital Versatile Disc) drive 302, and a keyboard 274, a mouse 276, and a monitor 272 for interacting with a user, all of which are connected to the computer 270. Of course, these are just examples of configurations for when user interaction is required, and any general hardware and software available for user interaction (e.g., a touch panel, voice input, pointing devices in general) can be used.

Referring to FIG. 18, computer 270 includes, in addition to DVD drive 302, a CPU (Central Processing Unit) 290, a GPU (Graphics Processing Unit) 292, a bus 310 connected to CPU 290, GPU 292, and DVD drive 302, a ROM (Read-Only Memory) 296 connected to bus 310 and storing a boot-up program for computer 270, etc., a RAM (Random Access Memory) 298 connected to bus 310 and storing instructions constituting a program, system programs, working data, etc., and an SSD (Solid State Drive) 300 which is a non-volatile memory connected to bus 310. The SSD 300 is for storing programs executed by the CPU 290 and the GPU 292, and data used by the programs executed by the CPU 290 and the GPU 292. The computer 270 further includes a network I/F (Interface) 308 that provides a connection to a network 176 that enables communication with other terminals, and a Universal Serial Bus (USB) port 306 to which a USB memory 284 is detachable and that provides communication between the USB memory 284 and each unit in the computer 270.

The computer 270 further includes an audio I/F 304 that is connected to the microphone 164 and speaker 168 and the bus 310, and has the function of reading out audio signals, video signals and text data generated by the CPU 290 and stored in the RAM 298 or SSD 300 in accordance with instructions from the CPU 290, performing analog conversion and amplification processing to drive the speaker 168, and digitizing the analog audio signal from the microphone 164 and storing it in any address of the RAM 298 or SSD 300 specified by the CPU 290.

In the above embodiment, programs for realizing the functions of the operation control PC 162, the voice processing PC 166, the voice synthesis PC 170, and the human position recognition PC 180 are all stored, for example, in the SSD 300, RAM 298, DVD 278, or USB memory 284 shown in FIG. 18, or in a storage medium of an external device (not shown) connected via the network I/F 308 and the network 176. Typically, these data and parameters are written, for example, from outside into the SSD 300 and loaded into the RAM 298 when the computer 270 is executed.

7, the operation control PC 162, the voice processing PC 166, the voice synthesis PC 170, and the human position recognition PC 180, and computer programs for operating the computer system to realize the functions of each of these components are stored in a DVD 278 inserted in a DVD drive 302, and transferred from the DVD drive 302 to the SSD 300. Alternatively, these programs are stored in a USB memory 284, which is inserted in a USB port 306, and the programs are transferred to the SSD 300. Alternatively, the programs may be transmitted to the computer 270 via the network 176 and stored in the SSD 300.

The program is loaded into RAM 298 when executed. Of course, the source program may be input using keyboard 274, monitor 272 and mouse 276, and the compiled object program may be stored in SSD 300. In the case of a scripting language as in the above embodiment, a script input using keyboard 274 or the like may be stored in SSD 300. In the case of a program that runs on a virtual machine, a program that functions as a virtual machine must be installed in computer 270 in advance. A neural network is used for voice recognition and voice synthesis. A trained neural network may be used, or training may be performed in conversational robot system 150.

The CPU 290 reads a program from the RAM 298 according to an address indicated by a register (not shown) called a program counter inside the CPU 290, interprets the command, reads data required for executing the command from the RAM 298, the SSD 300, or other devices according to the address specified by the command, and executes the process specified by the command. The CPU 290 stores the execution result data in the RAM 298, the SSD 300, a register in the CPU 290, or an address specified by the program. Depending on the address, the computer outputs the command to the robot actuator, a voice signal, or the like. At this time, the value of the program counter is also updated by the program. The computer program may be loaded directly into the RAM 298 from the DVD 278, the USB memory 284, or via the network 176. Note that some tasks (mainly numerical calculations) among the programs executed by the CPU 290 are dispatched to the GPU 292 according to the command included in the program or according to the analysis result when the CPU 290 executes the command.

The program for implementing the functions of each part according to the above-mentioned embodiment by the computer 270 includes a plurality of instructions written and arranged to operate the computer 270 to implement those functions. Some of the basic functions required to execute the instructions may be provided by an operating system (OS) or a third-party program running on the computer 270, or by modules of various toolkits or the execution environment of the program installed on the computer 270. Thus, the program does not necessarily include all of the functions required to implement the system and method of this embodiment. The program only needs to include instructions to perform the operations of each of the above-mentioned devices and their components by statically linking or dynamically calling appropriate functions or modules in a controlled manner to obtain the desired results. The method of operation of the computer 270 for this purpose is well known, so it will not be repeated here.

The GPU 292 is capable of parallel processing, and can execute a large amount of calculations associated with machine learning simultaneously in parallel or in a pipeline manner. For example, parallel calculation elements discovered in a program when the program is compiled, or parallel calculation elements discovered when the program is executed, are dispatched from the CPU 290 to the GPU 292 as needed, and executed. The results are returned to the CPU 290 directly or via a specified address in the RAM 298, and assigned to a specified variable in the program.

3. Functional Configuration FIG. 19 shows the functional configuration of the gaze control system 350, which is a part related to gaze control in the conversation robot system 150 according to this embodiment. Referring to FIG. 19, the gaze control system 350 includes a role determination unit 360 for determining and outputting the current roles of the robot 160 and other dialogue participants, and a turn change detection unit 362 for detecting turn changes. In this embodiment, the turn changes and the roles of the dialogue participants are detected each time. However, for example, when a three-way dialogue including a robot is performed according to a scenario prepared in advance, the roles and turn changes of each participant are almost specified by the scenario. Therefore, it is not necessary to provide the role determination unit 360 and the turn change detection unit 362. In the evaluation experiment described later, gaze control is performed under such conditions.

The gaze control system 350 further includes a gaze direction model 364 for probabilistically determining the gaze direction of the robot 160 when changing turns and while speaking, a gaze aversion model 366 for efficiently determining the gaze direction of the robot 160 when averting its gaze, and a personality information storage unit 368 for storing information about the personality of the robot 160 (extrovert, introvert, neutral). Details of the gaze direction model 364 will be described later with reference to Figures 20 and 21. Details of the model 366 will be described later with reference to Figures 22 and 23. Both the gaze direction model 364 and the gaze aversion model 366 are stored in the probability model storage unit 172 shown in Figure 16. The personality information storage unit 368 is also actually realized by the probability model storage unit 172. The probability model storage unit 172 is realized by the SSD 300 in Figure 18.

The gaze control system 350 further includes a gaze action generation unit 370 for receiving information regarding the roles of the robot 160 and other dialogue participants from the role determination unit 360 in response to receiving a signal indicating that a turn change has been detected from the turn change detection unit 362, and for creating parameters for controlling the gaze action of the robot 160 using the gaze direction model 364, the personality information storage units 368 and 368, and a gaze action control unit 372 for controlling various actuators (not shown) for controlling the gaze of the robot 160 in accordance with the parameters created by the gaze action generation unit 370.

4. Model configuration A. Gaze direction model 364
Referring to FIG. 20 , the gaze direction model 364 includes a gaze direction model 400 during speech that is prepared in advance according to a combination of personality and role, a gaze duration model 402 that is prepared in advance according to a combination of personality and role, and gaze direction models 404, 406, and 408 during turn alternation that are prepared in advance according to a combination of personality and role for determining the gaze direction in the first, second, and third sections during a turn alternation, respectively.

The configuration of the gaze direction model 400 during speech is substantially the same as that shown in FIG. 6. The model shown in FIG. 6 is, for example, a model for an extroverted participant. Therefore, in addition to that shown in FIG. 6, the gaze direction model 400 during speech includes a model for an introverted participant and a model for a neutral participant.

Fig. 21 shows an example of the configuration of the gaze duration model 402. Referring to Fig. 21, the gaze duration model 402 stores the values of the degree of freedom (n), the bias (loc) in the horizontal axis direction, and the amplitude (scale) in the vertical axis direction of the chi ^-square distribution curve for each gaze destination (role of other participants excluding one's own role) according to a combination of the role of the robot 160 (speaker, main listener, and sub-listener) and the personality (extrovert, introvert, neutral) assumed for the robot 160. The distribution is specified by substituting these values into the chi- ^square distribution formula, and the duration can be determined when the gaze direction is determined by randomly sampling values from the distribution.

The gaze direction models 404, 406, and 408 during turn alternation have the same configuration as the gaze direction model during speech 400. The difference is that the stored values are limited to each interval during the turn alternation.

B. Averted Eyes Model 366
Fig. 22 shows the configuration of the gaze aversion model 366 shown in Fig. 19. Referring to Fig. 22, the gaze aversion model 366 includes a turn-taking gaze aversion direction model 450 and a speech gaze aversion direction model 452. Both of these models are provided for each personality.

Figure 23 shows an example of the configuration of the turn-changing gaze aversion direction model 450. With reference to Figure 23, the turn-changing gaze aversion direction model 450 is a model of the gaze aversion direction of an extroverted participant shown in Figure 14, for example. In this embodiment, the turn-changing gaze aversion direction model 450 is realized by an array. Each element is arranged in the order when the center column and the right column in Figure 14 are moved below the leftmost column. Each element indicates the gaze aversion direction (nine directions from the top left to the bottom right) and the proportion (probability) of averting the gaze in that direction. In Figure 14, "L" indicates left, "C" indicates center, "R" indicates right, "U" indicates up, and "D" indicates down. For example, "LD" indicates bottom left, and "CD" indicates bottom center.

As described below, the program can refer to the turn-changing gaze averting direction model 450 to probabilistically determine the direction in which the robot 160 will avert its gaze when changing turns, depending on the role of the robot 160.

5. Program Configuration A. Overall Configuration FIG. 24 shows, in a flow chart format, the overall configuration of the program executed by the gaze control system 350 to control the robot 160 according to this embodiment. Referring to FIG. 24, this program is repeatedly started at a certain time interval, and includes step 480 for sensing the state of the dialogue (whether it is a turn change, what is the role of each dialogue participant, etc.) at the time of starting, and step 482 for branching the flow of control according to whether the current time is the timing to determine the gaze direction. For example, if it is a turn change, the timing for determining the gaze direction is when it is −1 second, −0.3 second, and 0.3 second, with the timing of the turn change being 0 second. Other than the turn change, that is, when speaking, it is the time when the duration of the gaze determined when the previous gaze direction was determined expires. If the direction is the gaze averting direction and the duration is 0.7 seconds or more, a process of re-determining the gaze direction is performed every 0.7 seconds. These timings can be detected by setting the time to a timer when each event occurs and depending on whether the timer expires or not.

This program further includes step 484, which is executed when the determination in step 482 is positive, and determines the gaze direction and its duration according to a model selected from gaze direction model 364 or further from gaze aversion model 366, according to a combination of the role assigned to the current robot 160, its personality, and information on whether it is speaking or turning, and step 486, which is executed after step 484 or directly when the determination in step 482 is negative, and controls each part of the robot 160 according to the parameters set in step 484 and ends the execution of the program. Note that in this program, processing for voice recognition, voice synthesis, posture control, and the like for the robot is also executed in other steps not shown. Control based on such processing is also performed in step 486.

In step 486, the head direction and eye position of robot 160 are controlled in accordance with the parameters determined in step 484 in order to control the line of sight of robot 160 in accordance with the progression of the timer. Since time progresses each time the program is executed, the head position and eye position of robot 160 change each time the program is executed.

B. Step 480 (Status Sensing)
Fig. 25 shows, in the form of a flowchart, the configuration of a program for realizing step 480 in Fig. 24. Referring to Fig. 25, step 480 includes step 510 for reading information indicating the roles of robot 160 and each participant from role determination unit 360 shown in Fig. 19, step 512 for reading information indicating a turn state from turn alternation detection unit 362 shown in Fig. 19, and step 514 for reading information indicating the personality given to robot 160 from personality information storage unit 368 shown in Fig. 19 and terminating execution of this program.

The personality given to the robot 160 can be considered to be fixed. Therefore, information about the personality does not necessarily need to be read out for each repetition of the program execution. However, if the robot 160 is to be given not only a personality but also states that are expected to change over time, such as emotions, it is desirable to read out that information for each repetition of the program execution, as shown in Figure 25.

C. Step 484 (Determine Gaze Direction and Duration)
26, step 484 includes a step 550 of selecting and reading from the gaze direction model 364 a gaze direction model and a gaze duration model corresponding to the role, turn state, and personality assigned to the robot 160, and a step 552 of selecting and reading from the gaze aversion model 366 a turn alternation gaze averting direction model 450 or a speech averting direction model 452 corresponding to the role, turn state, and personality assigned to the robot 160. In step 550, if the turn state is speech, the speech gaze direction model 400 shown in FIG. 20 is selected. If the turn state is turn alternation, any one of the turn alternation gaze direction models 404, 406, and 408 is selected according to the timing.

Following step 552, the program further includes step 554 of sampling a value p in the range [0, 1] from a uniform distribution, and step 556 of determining the gaze direction from the gaze direction model selected in step 550 using the value p sampled in step 554. In this specification, determining the gaze direction using a value randomly sampled in the range [0, 1] of the gaze direction model in this manner is referred to as "sampling the gaze direction." Details of step 556 will be described later with reference to FIG. 27.

This program further includes step 558 of sampling a value p in the range of [0, 1] from a uniform distribution, similar to step 554, and step 560 of reading parameters n, loc, and scale of a ^chi -square distribution according to the role of the robot 160 and the personality given to the robot 160 set by the process shown in Fig. 25 from the gaze duration model 402 read in step 550, and determining the gaze duration using these values and the value p sampled in step 558. Details of step 560 will be described later with reference to Fig. 28.

The program further includes step 562, which branches the flow of control depending on whether the gaze direction determined in step 556 is an averted gaze or not. If the determination in step 562 is negative, execution of the program ends.

The program further includes step 564, in response to the determination in step 562 being affirmative, sampling a value p in the range [0, 1] from a uniform distribution, and step 566, using the value p sampled in step 564 and the gaze direction model selected in step 556 to determine the gaze direction, and terminating execution of the program. Details of step 566 will be described later with reference to FIG. 29.

FIG. 27 shows the details of step 556 in FIG. 26 in the form of a flow chart. In this figure, a variable S _M is used as a variable representing the probability of directing the gaze to the main listener, S _S is used as a probability of directing the gaze to the speaker, and S _B is used as a probability of directing the gaze to the sub-listener. Referring to FIG _. 27, step 556 includes step 600 of substituting the sum of the value of the variable S _S and the value of the variable S _M into the variable S _M , and substituting the sum of the value of the variable S _M thus calculated and the value of the variable S B into the variable S _B. By performing this calculation, the probability of turning in the direction of the speaker is stored in the variable S _S. The variable S _M stores a value obtained by adding up the probability of turning in the direction of the speaker and the probability of turning in the direction of the main listener. The variable S _B stores a value obtained by adding up the probabilities of turning in the direction of the speaker, the main listener, and the sub-listener.

This program further includes, following step 600, step 602 for branching the flow of control depending on whether or not the value p is smaller than the value of the variable S-- _S , and, when the determination in step 602 is affirmative, step 604 for determining that the gaze direction is the direction of the speaker and terminating execution of this program.

The program further includes step 606, in response to the determination in step 602 being negative, of branching the flow of control according to whether or not the value p is smaller than the value of the variable S-- _M , and step 608, in response to the determination result in step 606 being positive, of determining that the line of sight direction is the direction of the main listener and terminating execution of the program.

The program further includes step 610, in response to the determination in step 606 being negative, branching the flow of control according to whether or not the value p is less than the value of the variable _{S_B} , step 612, when the determination in step 610 is positive, determining the gaze direction to be in the direction of the sub-listener and terminating execution of the program, and step 614, when the determination in step 610 is negative, determining the gaze direction to be an averting direction and terminating execution of the program.

By performing the processing in step 600, such an algorithm can be used to determine the gaze direction according to the value p and the probability represented by the model.

Fig. 28 shows, in the form of a flowchart, a control structure of the program executed in step 560 in Fig. 26. Referring to Fig. 28, this program includes step 650 of reading values n, loc, and scale from the gaze duration model 402 selected in step 556 in Fig. ^{26, step 652 of sampling a value p from a uniform distribution of [0, 1] and substituting this value for a variable x, and step 654 of calculating the gaze duration by substituting the values of n, loc, scale, and x into the chi-square} distribution equation for calculating the gaze duration already described, and then terminating execution of this program.

Figure 29 shows, in the form of a flowchart, the control structure of a program for implementing step 566 in Figure 26. Referring to Figure 29, this program includes step 680 for substituting 0 for repetition variable i for loop processing in the program, and 0 for variable S for accumulating each of the probabilities shown in Figure 23 in order starting from array number 0, and step 682 following step 680, for branching the flow of control depending on whether the value of variable i is less than 8. Here, "8" is the maximum value of the subscript in the array shown in Figure 23.

23 to variable S, step 686 branches the flow of control depending on whether the value of variable p is smaller than the value of variable S, and step 688 adds 1 to the value of variable i and returns control to step 682 when the determination in step 686 is negative, i.e., when the value of variable p is equal to or greater than the value of variable S. If the determination in step 686 is positive, step 690 determines the direction of averted gaze to be the direction of averted gaze D(i) in the array in FIG. 23 and terminates execution of the program.

The program further includes step 692, in response to a negative determination in step 682, determining the gaze aversion direction to be gaze aversion direction D(8) and terminating execution of the program.

By executing this program, the direction of gaze aversion can be determined, for example, according to the probability distribution shown in Figure 14.

6. Operation The conversational robot system 150 according to this embodiment operates as follows. Note that the following description will be given only about the gaze control system 350, mainly with reference to Figs. 24 to 29.

When the gaze control system 350 is started, step 480 in Figure 24 is executed. In this example, it is assumed that the roles of the dialogue participants, including the robot 160, and the turn state are set as predetermined initial values, with role = speaker and the turn state being the speaking period (other than the turn alternation period). It is also assumed that the personality information storage unit 368 shown in Figure 19 stores information indicating the personality to be given to the robot 160. Here, it is assumed that the robot 160 has been given the personality of "extroverted". In step 480, this information is read into the RAM 298 in Figure 18.

Next, in step 482, it is determined whether it is time to determine the gaze direction. If the speech period is set as the initial value of the turn state and the gaze duration timer is cleared, the determination in step 482 in FIG. 24 is positive and control proceeds to step 484.

With reference to FIG. 26, in step 550, the gaze direction model 400 during speech corresponding to role=speaker, turn state=speaking period, and personality=extroversion is read out, and in step 552, the gaze duration model 402 corresponding to role=speaker and personality=extroversion is selected. Furthermore, in step 554, the value p is sampled, and step 556 is executed. In step 556, the process shown in FIG. 27 is executed, and the gaze direction is determined. Here, since the role of the robot 160 is a speaker, one of main listener, sub-listener, and averted gaze is selected as the gaze direction.

Returning to FIG. 26, the value p is again sampled in step 558. In the following step 560, the gaze duration is determined as follows:

28, in step 650, parameters n, loc, and scale of the chi- ^square distribution according to the role of the robot 160 and the personality given to the robot 160 are read from the gaze duration model 402 read in step 552 in Fig. 26. In step 652, the value p sampled in step 558 is substituted for the variable x. In the following step 654, the gaze duration is determined by substituting the parameters n, loc, and scale and the value of the variable x into an equation for duration calculation to calculate a value.

Returning to FIG. 26 again, in the following step 560, it is determined whether the gaze direction determined in step 556 is an averted gaze or not. Here, it is assumed that the gaze direction is not an averted gaze. In that case, the process shown in FIG. 26 (step 484 in FIG. 24) immediately ends. Returning to FIG. 24, in step 486, control of robot 160 is started according to the gaze direction and gaze duration determined in step 484. Note that in step 486, control of robot 160 is also executed in addition to these gaze controls. If the gaze direction determined in step 556 is an averted gaze, steps 564 and 566 in FIG. 26 are executed to determine the direction in which robot 160 will avert its gaze. In step 486, the gaze of robot 160 is controlled so that it is in a direction that is neither the main listener nor the sub-listener.

Next, assume that the processing shown in FIG. 24 is started. Step 480 is the same as the first processing. In step 482, since the duration of the gaze direction has not yet expired, the processing of step 484 is not executed, and only the processing of step 486 is executed. Therefore, if the change in the gaze direction of the robot 160 has not yet been completed, the change in the gaze direction continues, and if it has been completed, the gaze direction is maintained.

After the process of FIG. 24 is thus executed multiple times, it is assumed that a turn change has been detected. Here, one second before the actual timing of the turn change is detected and notified to robot 160 in step 480 of FIG. 24. One second before the timing of the turn change is the timing for determining the gaze direction. Therefore, the process of step 484 described above is executed again, and a new gaze direction and a new gaze duration are determined using a gaze direction model and a gaze duration model selected according to robot 160's new role, the turn state = turn change period, and the personality of robot 160. Thereafter, control of robot 160's gaze according to the new parameters is started in step 486 of FIG. 24.

Thereafter, the gaze control system 350 repeats the above-mentioned process. As already mentioned, the gaze direction of the robot 160 is determined when the gaze duration determined immediately before expires, or when the timing of the turn change is -1 second, -0.3 second, and 0.3 second, assuming that the timing is 0 second.

In addition, in actual three-way dialogues, it was observed that when people looked away, they did not look at one point for a long time, but rather changed their gaze direction midway. The number of changes tended to increase the longer the gaze was averted. Therefore, based on the analysis of the distribution of duration for each number of gaze averts, it was decided to increase the number of gaze direction changes by once every 0.7 seconds, i.e. once when the duration was 0.7 seconds or less, twice when it was 1.4 seconds or less, and three times when it was 2.1 seconds or less.

In this case, too, the gaze direction was basically changed according to the distribution shown in Figures 13 to 15. However, if the gaze direction was changed further while looking away, the previous gaze direction was set as the center, and the distribution shown in Figures 13 to 15 was applied on the premise that the gaze direction was changed to a location other than the center. Also, when the gaze direction was changed two or more times while looking away, the actual observations showed a tendency for the gaze to return to the previous direction, so the implementation was also set to return the gaze direction to the previous direction.

3. Evaluation experiment 1. Evaluation experiment 1
An evaluation experiment was conducted on the robot 160 according to the above embodiment to see how a change in the gaze direction when no personality is specified is perceived by an observer.

A. Model In this embodiment, the gaze pattern for each role and the movement of the pupils when looking away are important elements in the dialogue. Therefore, in the evaluation experiment, the evaluation of the movement of the robot 160 was compared under the following four conditions.

a. Baseline (same proportions - head model)
As a baseline, a model was used in which the robot 160, when its role was a speaker, directed its gaze in the direction of the two interlocutors at the same rate. The ratio of the robot's gaze to the two interlocutors and other places was set to be approximately 3:3:4, with the first interlocutor, the second interlocutor, and other places being the first:3:4 ratio. In this case, the robot was made to look at a place obtained by applying a two-dimensional Gaussian distribution with a variance of 4 degrees between the human face or the two interlocutors. When the robot was a listener, it was made to look in the direction of the speaker to avoid creating a sense of incongruity.

b. Comparison model (equal proportions - head-eye model)
Eye movement is important for the gaze control of a robot. Therefore, in the equal proportion model mentioned above, we added the eyeballs to the head as well as the head to move together.

c. Evaluation model 1 (head-eyeball model according to the embodiment)
This is a model in which gaze control and gaze aversion are implemented based on the above embodiment.

d. Evaluation model 2 (head model according to the embodiment)
Among the aforementioned evaluation models, this is a model that eliminates eye movement and only involves head movement.

B. Impression Evaluation The most desirable effect of this embodiment is whether the robot's behavior seems natural. Therefore, the question "Did you feel that the robot's behavior was natural overall?" was included as a question for the evaluators.

C. Experimental Setup In this evaluation experiment, a video of one of the conversations between three people was actually recorded and loaded onto the robot, and was created for each of the above models. As a result, four videos were obtained. The evaluators watched these videos and evaluated the eye movements at the time. In practice, each evaluator watched a video like that shown in Figure 30.

Thirty men and women (average age 32.1 years, variance 10.3) were asked to watch four videos and to evaluate their impressions of each. Figure 30 shows an example of the video image 720 used in this experiment. The experiment is solely concerned with changes in the gaze direction of the robot 160. Therefore, in this evaluation experiment, a three-way dialogue between real people was used as the speech, and one of the people was spoken by the robot 160. In this case, the gaze direction of the robot 160 was controlled to move in the same manner as in the above embodiment according to the role, personality, and turn state of the robot 160. Since the actual three-way dialogue is labeled, the role determination unit 360 and turn change detection unit 362 shown in Figure 16 are not necessary, and information corresponding to the labels is provided to the gaze action generation unit 370 at an appropriate timing.

As shown in Figure 30, video image 720 has a moving image of robot 160 placed in the center, with simple character images 730 and 732 showing the other two speakers on the left and right. Character images 730 and 732 are intended to indicate that this is a three-way dialogue, and when a speaker on the right side of robot 160 (left side of Figure 30) is speaking, for example, a sign indicating that the speaker is speaking is displayed around character image 730. The same applies to the speaker on the left side of robot 160 (right side as you face it). Figure 30 shows a situation where the speaker on the right side is speaking.

In the evaluation experiment, 30 men and women were asked to watch these four videos and their impressions were collected. When watching the videos, the speaker on the right side was heard in the evaluator's right ear, the speaker on the left side in the evaluator's left ear, and the robot's speech in both ears. In this way, the evaluators were asked to individually evaluate the impressions of each of the four models mentioned above.

The experimental procedure was as follows. First, the presentation order of the four videos was randomized to reduce order effects. Next, the evaluators were asked to rate the impressions of each of the four methods individually on a 7-point scale (1: very unnatural, 4: neither strong nor weak, 7: very natural). The above constituted one set, and impressions of a total of 12 videos in this set were evaluated for three dialogue sections (each dialogue section was approximately 1 minute long), and the results were compiled.

D. Evaluation Results Among the three dialogue sections, no significant difference was observed between the conditions in two of them. In the remaining dialogue section, a significant difference was observed between the conditions as shown in FIG. 31. FIG. 31 shows the evaluation results for the question, "Overall, did you feel that the robot's behavior was natural?" For each model, the average value of the results, standard error, and the result of multiple comparison based on Ryan's method were calculated.

The results of the Ryan method showed significant differences between the baseline model and evaluation model 1, with a p-value of 0.020 (≦.05), between the comparison model and evaluation model 1, and between evaluation model 2 and evaluation model 1, with a p-value of 0.001 (≦.05), and between evaluation model 2 and evaluation model 1, respectively.

The above results show that, among the four models, the behavior of the robot based on evaluation model 3 based on the embodiment is the most natural.

2. Evaluation experiment 2
In addition to the settings of the above evaluation experiment 1, an experiment was conducted to consider personality in the gaze control of the robot and to evaluate the impression of personality (extroversion) based on gaze movements.

A. Experimental Method In order to conduct an experiment to evaluate gaze behavior, first, a dialogue section in which speaker A or speaker B participated was extracted from the three-way dialogue data. Furthermore, the conversational voice of each speaker was used to generate the gaze behavior of the robot, and a video was created. In the experiment, the participants were asked to evaluate the gaze behavior generated under multiple conditions. In order to investigate the difference in the impression of personality expression due to gaze behavior, the gaze behavior of the robot was generated using a model (PA-M) created from the data of speaker A and a model (PB-M) created from the data of speaker B. For comparison, we also prepared a model that reproduced the gaze behavior of the speaker (PA-R or PB-R) and a model that presented only the voice without any movement (PA-S or PB-S).

Conversational segments of about 40 seconds each were extracted for Speaker A and Speaker B, and videos were recorded under the four conditions mentioned above and used in the subject experiment. The content of the videos was the same as that shown in Figure 30. The way the speech was heard by the evaluators was also the same as in Evaluation Experiment 1. In order to make it clear that there were interlocutors on the left and right (diagonally in front), character images representing the interlocutors were drawn on both sides of the screen as shown in Figure 7 (b), and when a voice was spoken, a radiating line was drawn. The interlocutors' voices were also heard in the left and right ears, respectively, and the voice of Speaker A or Speaker B was heard in both ears. In the video with audio only, the target speaker was also drawn as a character image without any movement instead of a robot.

In each dialogue section, participants first watched an audio-only video (S-M), and then watched three videos that included the robot's gaze movements. To reduce order effects, the presentation order of the three videos was randomized. Immediately after watching each video, participants were asked to rate their impressions on a 7-point scale (1: very introverted, 4: neutral, 7: very extroverted). Participants were not informed that gaze movements were controlled in this experiment.

B. Experimental Results Forty-one men and women (average age 37.5 years, standard deviation 14.1 years) participated in the experiment. Figure 32 shows the results of the impression evaluation.

Speaker A's voice Figure 32 (A) shows that the impression of extroversion when listening to only the voice (PA-S) is on the extroverted side (4 or higher). When gaze movements are controlled through a robot, the gaze movement generation model (PA-M) constructed from speaker A's data gives the same impression of extroversion as the model (PA-R) in which speaker A's gaze movements are reproduced, while the model (PB-M) constructed from speaker B's data, which is on the introverted side, gives a significantly lower impression of extroversion.

b. Speaker B's voice Figure 32 (B) shows that when only the voice was heard (PB-S), the speaker was more introverted (4 or less). When gaze movements were controlled through a robot, the gaze movement generation model (PB-M) constructed from speaker B's data gave the same impression of extroversion as the model (PB-R) in which speaker B's gaze movements were reproduced, while the model (PA-M) constructed from speaker A's data, who is more extroverted, gave a significantly higher impression of extroversion.

C. Discussion The above results show that by using different gaze generation models for speakers with different personalities (extroversion), it is possible to change the impression of personality expressed even with the same voice. This result also suggests that the impression of extroversion can be continuously controlled by interpolating or extrapolating the model parameters.

Fourth Modification In the above embodiment, the model stores the probability of directing the gaze in each direction, and when actually determining the gaze direction, these probabilities are added and then the gaze direction is sampled using a random number. However, the present invention is not limited to such an embodiment. For example, the model may store a value (cumulative probability) in which each probability is added in advance. In the above embodiment, there are nine directions when the gaze is averted. However, the present invention is not limited to such an embodiment. The directions may be classified in more detail.

Furthermore, in the above embodiment, a gaze direction model and gaze continuation model are prepared and used based on a combination of the dialogue role, turn state, and personality of an agent such as a robot. However, the present invention is not limited to such an embodiment. A model may be prepared that omits any of these. Also, criteria other than these may be used to select a gaze direction model. For example, age, or social status such as teacher and student, parent and child, etc. may be used as criteria.

The above embodiment relates to a three-party dialogue. However, as is clear from the description of the above embodiment, the present invention can also be applied to a dialogue involving four or more people, provided that each speaker can be distinguished and their role can be classified.

The embodiments disclosed herein are merely illustrative, and the present invention is not limited to the above-described embodiments. The scope of the present invention is indicated by the claims in the scope of the claims, taking into consideration the detailed description of the invention, and includes all modifications within the meaning and scope equivalent to the wording described therein.

50 Three-way dialogue 60, 62, 64 Participants 80, 82, 84 Gaze label sequence 100, 104 Utterance 102, 106 Turn-taking label 108 Turn-taking 150 Conversation robot system 160 Robot 162 Operation control PC
164 Microphone 166 Audio processing PC
168 Speaker 170 Voice synthesis PC
172 Probability model storage device 174 Interactive PC
176 Network 178 Human position sensor 180 Human position recognition PC
190 Voice recognition unit 192 Turn recognition unit 250 Computer system 270 Computer 272 Monitor 274 Keyboard 276 Mouse 278 DVD
284 USB memory 290 CPU
292 GPUs
296 ROM
298 RAM
300 SSD
302 DVD drive 304 Audio I/F
306 USB port 308 Network I/F
310 Bus 350 Gaze control system 360 Role determination unit 362 Turn change detection unit 364 Gaze direction model 366 Model 368 Individual information storage unit 370 Gaze movement generation unit 372 Gaze movement control unit 400 Gaze direction model during speech 402 Gaze duration model 404, 406, 408 Gaze direction model during turn change 450, 452 Direction model

Claims (18)

A gaze control device for controlling a gaze of an agent in a multi-person dialogue including three or more agents , comprising:
a gaze direction setting means for determining the gaze direction of the agent based on a combination of roles of participants in the multi-person dialogue and a state of a dialogue flow in response to a timing for determining the gaze direction of the agent ;
a control parameter generating means for generating control parameters for controlling a face orientation and an eye direction of the agent in response to the gaze direction of the agent being determined by the gaze direction setting means,
The state of the dialogue flow includes a turn-taking state that starts a predetermined time before the turn-taking and continues until a predetermined condition is satisfied after the turn-taking, and an utterance state that is a state other than the turn-taking state,
The gaze direction setting means detects the start and end of the turn alternation state, determines the timing using different methods depending on the state of the dialogue flow, and at each of the timings, determines the gaze direction of the agent using different probability models depending on the state of the dialogue flow and the role of the agent in that state . The line of sight direction setting means
a direction determination model storage means for storing a direction determination model that determines, for each role, a probability that the gaze direction of the agent will turn in each of a plurality of predetermined directions in accordance with a combination of each role of the participants and a state of the dialogue flow;
a probability distribution extracting means for detecting a start and an end of the turn-changing state, determining the timing in accordance with the role of the agent in the multi-person dialogue and whether the dialogue flow is in the turn-changing state, and extracting a probability distribution corresponding to a combination of the role of the agent and the state of the dialogue flow from the direction determination model in response to the timing being reached;
2. The gaze control device according to claim 1, further comprising: a first sampling means for sampling the gaze direction of said agent from the probability distribution extracted by said probability distribution extracting means. The gaze control device according to claim 2 , wherein the plurality of directions of the direction determination model include a direction of the participants other than the agent and an averting direction different from any of the directions of the participants . The line of sight direction setting means further comprises:
an eye gaze aversion direction model storage means for storing an eye gaze aversion direction model comprising a probabilistic model for probabilistically determining the eye gaze aversion direction according to a combination of the role of the agent and a state of the dialogue flow;
and second sampling means for sampling a direction in which the agent averts its gaze from the averting direction model in response to the gaze direction sampled by the first sampling means being the averting direction. 5. The gaze control device according to claim 1, further comprising a duration calculation unit for calculating a duration of the gaze of the agent in response to the timing being reached, in accordance with a combination of the role of the agent , a state of the dialogue flow, and the gaze direction determined by the gaze direction setting means. 6. The gaze control device according to claim 5, wherein when the state of the dialogue flow is a turn-alternating state , the timing is a predetermined timing in the turn-alternating state, and when the state of the dialogue flow is not a turn-alternating state, the timing is the expiration of the duration calculated by the duration calculation unit immediately before that . The line-of-sight control device according to claim 6 , wherein the timing for determining the line-of-sight direction in the turn-alternating state is a plurality of predetermined timings after the start of the turn-alternating state. The line of sight direction setting means
a direction determination model storage means for storing a direction determination model that determines, for each role , a probability that the gaze direction of the agent will turn in each of a plurality of predetermined directions according to a combination of each role of the participants, a state of the dialogue flow, and a personality assumed for the agent;
a probability distribution extracting means for detecting a start and an end of the turn alternation state, determining the timing in accordance with the role of the agent in the dialogue and the state of the dialogue flow, and, in response to the timing being reached, extracting a probability distribution corresponding to a combination of the role of the agent , the state of the dialogue flow, and the personality from the direction determination model;
2. The gaze control device according to claim 1, further comprising: a first sampling means for sampling the gaze direction of said agent from the probability distribution extracted by said probability distribution extracting means. The gaze control device according to claim 8 , wherein the plurality of directions of the direction determination model include a direction of the participants other than the agent and an averting direction different from any of the directions of the participants . The line of sight direction setting means further comprises:
an eye gaze aversion direction model storage means for storing an eye gaze aversion direction model comprising a probabilistic model for probabilistically determining the eye gaze aversion direction of the agent according to a combination of the role of the agent , the state of the dialogue flow , and the personality;
and second sampling means for sampling a direction in which the agent averts its gaze from the averting direction model in response to the gaze direction sampled by the first sampling means being the averting direction. 11. The gaze control device according to claim 8, further comprising a duration calculation unit for calculating a duration of the gaze of the agent in response to the timing being reached, according to a combination of the role of the agent , a state of the dialogue flow, the personality, and the gaze direction determined by the gaze direction setting means. The gaze control device according to claim 11, wherein the timing for determining the gaze direction is a predetermined timing in a turn-alternating state when the state of the dialogue flow is a turn-alternating state, and is the timing at which the duration calculated by the duration calculation unit immediately before the turn-alternating state expires when the state of the dialogue flow is not a turn-alternating state. The gaze control device according to claim 12 , wherein the timing for determining the gaze direction in the turn alternating state is a plurality of predetermined timings after the start of the turn alternating state. The agent is an agent that is equipped with an eyeball capable of controlling the direction of gaze and a speaking function, and is selected from the group consisting of a robot, a three-dimensional virtual agent represented as a three-dimensional image in a virtual space, and a two-dimensional virtual agent represented as a two-dimensional image.A gaze control device as described in any one of claims 1 to 4 and claims 8 to 10. 1. A gaze control method implemented by a computer for controlling a gaze of an agent in a multi-person dialogue of three or more people including the agent, comprising:
A step of determining a gaze direction of the agent by the computer in response to a timing for determining a gaze direction based on a combination of a role of a participant in the multi-person dialogue and a state of a dialogue flow;
and generating control parameters for controlling a face orientation and an eye direction of the agent in response to the gaze direction of the agent being determined in the step of determining the gaze direction by the computer ,
The state of the dialogue flow includes a turn-taking state that starts a predetermined time before the turn-taking and continues until a predetermined condition is satisfied after the turn-taking, and an utterance state that is a state other than the turn-taking state,
The step of determining the gaze direction includes the steps of detecting the start and end of the turn alternation state, determining the timing by different methods depending on the state of the dialogue flow, and determining the gaze direction of the agent at each of the timings using different probability models depending on the state of the dialogue flow and the role of the agent in that state . A computer program for controlling the gaze of an agent in a multi-person dialogue involving three or more agents , the computer comprising:
a gaze direction setting means for determining the gaze direction of the agent based on a combination of roles of participants in the multi-person dialogue and a state of a dialogue flow in response to a timing for determining the gaze direction of the agent ;
functioning as a control parameter generating means for generating control parameters for controlling a facial orientation and an eye direction of the agent in response to the gaze direction of the agent being determined by the gaze direction setting means;
The state of the dialogue flow includes a turn-taking state that starts a predetermined time before the turn-taking and continues until a predetermined condition is satisfied after the turn-taking, and an utterance state that is a state other than the turn-taking state,
the gaze direction setting means detects the start and end of the turn alternation state, determines the timing by different methods depending on the state of the dialogue flow, and determines the gaze direction of the agent at each of the timings using different probability models depending on the state of the dialogue flow and the role of the agent in that state . A gaze control device for controlling a gaze of an agent in a multi-person dialogue of three or more people including the agent , the gaze control device being realized by a computer having a processor, comprising:
The processor,
determining a gaze direction of the agent based on a combination of roles of participants in the multi-person dialogue and a state of a dialogue flow in the multi-person dialogue in response to a timing for determining a gaze direction of the agent ;
a control parameter is generated for controlling a face orientation and an eye direction of the agent in response to the gaze direction of the agent being determined ;
The state of the dialogue flow includes a turn-taking state that starts a predetermined time before the turn-taking and continues until a predetermined condition is satisfied after the turn-taking, and an utterance state that is a state other than the turn-taking state,
the processor is further configured to detect the start and end of the turn alternation state, determine the timing in different manners depending on the state of the dialogue flow, and at each of the timings, determine the gaze direction of the agent using a different probability model depending on the state of the dialogue flow and the role of the agent in that state . A non-transitory, computer-readable, non-transitory storage medium storing a computer program for operating a computer to function as the gaze control device according to claim 17 .

Images