JP7624660B2 - Prediction device, subjective impression prediction method, and program - Google Patents

Description

This disclosure relates to a prediction device, a subjective impression prediction method, and a program.

Head movements are known to play a variety of roles among non-verbal behaviors that occur during human dialogue. For example, speakers use head movements to emphasize what they are saying or to confirm a response, while listeners use head movements to provide interjections, responses, or signs of agreement. As such, head movements have multiple functions, and a single head movement may simultaneously have multiple meanings. Focusing on the multiplicity and ambiguity of the functions of head movements, Non-Patent Document 1 defines functional categories of head movements and proposes a convolutional neural network (CNN) that uses data that has been non-exclusively annotated to allow for overlapping functions and detects the presence or absence of individual functions from the time series of head posture and the presence or absence of speech.

K. Otsuka and M. Tsumori, "Analyzing multifunctionality of head movements in face-to-face conversations using deep convolutional neural networks," IEEE Access, vol.8, pp.217169-217195, 2020.

However, as mentioned above, although head movement function is thought to be deeply related to behavior and emotions during a conversation, there is still no established method to predict the subjective impressions that interlocutors themselves have during a conversation.

The present invention has been made in consideration of the above points, and aims to predict the subjective impressions held by the interlocutors themselves during a conversation.

In order to solve the above problem, the invention of claim 1 is a prediction device that predicts the subjective impression held by an interlocutor himself during a conversation , comprising: a data acquisition means for acquiring data regarding the head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement function detection means for detecting head movement function in relation to the data regarding head movement using a head movement function detection model; a feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and a subjective impression prediction means for predicting the subjective impression of the interlocutor regarding the feature of the head movement function using a regression model.

The invention of claim 2 is a prediction device for predicting a subjective impression held by an interlocutor himself during a conversation , comprising: a data acquisition means for acquiring data regarding head movement of the interlocutor from video data obtained by photographing the interlocutor; a head movement detection means for detecting the presence or absence of head movement in relation to the data regarding head movement using a head movement detection model; a head movement function detection means for detecting head movement function in relation to head movement presence or absence information indicating the presence or absence of head movement using a head movement function detection model; a feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and a subjective impression prediction means for predicting the subjective impression of the interlocutor regarding the feature of the head movement function using a regression model.

The invention of claim 5 is a prediction device for predicting a subjective impression held by an interlocutor himself during a conversation , comprising: data acquisition means for receiving interlocutor data related to an interlocutor and acquiring information on a speech section indicating a speech section of the interlocutor from the interlocutor data; head movement detection means for detecting the presence or absence of speech in response to the speech section information using a head movement detection model; head movement function detection means for detecting head movement function in response to speech presence or absence information indicating the presence or absence of speech using a head movement function detection model; feature calculation means for calculating a feature of the head movement function based on the detection result by the head movement function detection means; and subjective impression prediction means for predicting the subjective impression of the interlocutor for the feature of the head movement function using a regression model.

As described above, the present invention has the effect of being able to predict the subjective impressions held by the interlocutors during a conversation.

FIG. 1 is a schematic diagram of a communication system. FIG. 2 is a diagram illustrating an electrical hardware configuration of the prediction device. FIG. 2 is a functional configuration diagram of a prediction device. FIG. 2 is a diagram showing the relationship between input information (data) and output information (data) in each function. 11 is a flowchart showing the behavior of data learning. 1 is a flowchart illustrating the behavior of data prediction. FIG. 13 is a diagram showing categories and functional contents of head movement functions. 1 is a diagram showing categories and functional contents of head movement functions used in this embodiment. FIG. FIG. 13 is a diagram showing a list of kinematic features. FIG. 1 illustrates characteristics of head movement function. FIG. 13 shows predicted values of subjective impressions as introspection report scores for each item.

The following describes an embodiment of the present invention with reference to the drawings.

[System configuration of the embodiment]
First, an outline of the configuration of a communication system according to the present embodiment will be described with reference to Fig. 1. Fig. 1 is a schematic diagram of a communication system according to an embodiment of the present invention.

As shown in FIG. 1, the communication system 1 of this embodiment is constructed by an input device 2, a prediction device 3, and a monitor 4. The input device 2 is a camera 2a, a smartphone 2b, a headphone with a microphone 2c, a microphone 2d, etc. The camera 2a is a luminance camera, a LiDAR (Light Detection and Ranging), an infrared camera, a thermal camera, etc.

The input device 2 and the prediction device 3 can communicate with each other via a communication network 100 such as the Internet. The communication network 100 may be connected in either a wireless or wired manner.

The prediction device 3 is composed of one or more computers. When the prediction device 3 is composed of multiple computers, it may be referred to as a "prediction device" or a "prediction system."

The prediction device 3 predicts the subjective impression (feelings) of the interlocutor based on the interlocutor data about the interlocutor acquired by the input device 2, and outputs result data indicating the prediction result. As an output method, the result data may be sent to a monitor 4 as an output device, and a graph or the like related to the result data may be displayed on the monitor 4.

[Hardware configuration]
<Hardware configuration of prediction device>
Next, the electrical hardware configuration of the prediction device 3 will be described with reference to Fig. 2. Fig. 2 is a diagram showing the electrical hardware configuration of the prediction device.

As shown in FIG. 2, the prediction device 3 is a computer and includes a CPU (Central Processing Unit) 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a HD (Hard Disk) 304, a HDD (Hard Disk Drive) controller 305, an external device connection I/F (Interface) 308, a network I/F 309, a bus line 310, and a media I/F 314.

Of these, CPU 301 controls the operation of the entire prediction device 3. ROM 302 stores programs used to drive CPU 301, such as IPL (Initial Program Loader). RAM 303 is used as a work area for CPU 301.

HD304 stores various data such as programs. HDD controller 305 controls the reading and writing of various data from HD304 under the control of CPU301. Note that instead of HD304 and HDD controller 305, an SSD (Solid State Drive) and SSD controller may be installed.

The external device connection I/F 308 is an interface for connecting various external devices. In this case, the external devices include a display, a speaker, a keyboard, a mouse, a USB (Universal Serial Bus) memory, a printer, etc.

The network I/F 309 is an interface for data communication via the communication network 100. The bus line 310 is an address bus, a data bus, etc., for electrically connecting each component such as the CPU 301 shown in FIG. 2.

The media I/F 314 also controls the reading and writing (storing) of data from and to a recording medium 313 such as a flash memory. The recording medium 313 also includes a DVD (Digital Versatile Disc) and a Blu-ray Disc (registered trademark).

[Functional configuration of the prediction device]
Next, the functional configuration of the prediction device will be described with reference to Fig. 3. Fig. 3 is a diagram showing the functional configuration of the prediction device.

In FIG. 3, the prediction device 3 has a data acquisition unit 31, a detection unit 32, a feature calculation unit 33, a subjective impression prediction unit 34, and a model storage unit 39. The data acquisition unit 31, the detection unit 32, the feature calculation unit 33, and the subjective impression prediction unit 34 are functions realized by instructions from the CPU 301 in FIG. 2 based on a program. The data acquisition unit 31 and the detection unit 32 perform data learning and data prediction behavior. The data acquisition unit 31, the detection unit 32, the feature calculation unit 33, and the subjective impression prediction unit 34 perform data prediction behavior. In addition, the detection unit 32 includes a head movement detection unit 32a, and a head movement function detection unit 32b.

Furthermore, a model storage unit 39 is constructed in the RAM 303 or HD 304 in FIG. 2. The model storage unit 39 stores a head movement detection model 40a, a head movement function detection model 40b, and a regression model 50.

<Each model>
Next, each model will be described.

(Head movement detection model)
The head movement detection model 40a is based on the Late-fusion CNN model (see Non-Patent Document 1) proposed by Otsuka & Tsumori as a CNN (Convolutional Neural Networks) model structure, with a Dropout layer inserted in fc4 of the fully connected layer. The head movement detection model 40a uses the angular velocity of the head pose angles (Azimuth, Elevation, Roll) of 32 video frames in the video data centered on the target frame as input information (data), and performs CNN learning to output the presence or absence of head movement for this input information. The number of video frames can be set arbitrarily. Hereinafter, a section of video frames in which head movement is detected continuously is referred to as one "head movement section." The machine-learned head movement detection model 40a is stored in the model storage unit 39.

(Head movement function detection model)
Head movement function detection model 40b uses a late-fusion CNN model similar to head movement detection model 40a. Here, each video frame in the head movement section detected by head movement detection model 40a is used as a video frame to be predicted, and at least one of the angular velocity of the head posture angle and speech section data for 32 frames centered on each video frame is used as input information. The number of video frames can be set arbitrarily.

The speech data used is a time series that has been smoothed using a moving average filter with a width of 50 video frames before and after the binary time series indicating the presence or absence of speech. The width of this video frame can be set arbitrarily. In the preliminary machine learning, the head movement function detection model 40b is stored in the model storage unit 39.

Figure 7 is a diagram showing the categories and function contents of head movement functions. In pre-learning, a corpus (data in which information about dialogue is written as time-series data) is used to learn a head movement function detection model for one or more of the functions of any category listed in Figure 7. The corpus includes data in which one or more of the functions of the categories in the head movement function listed in Figure 7 are non-exclusively labeled. For example, data in which the presence or absence of each of the 32 types of head movement functions listed in Figure 7 is non-exclusively labeled for each video frame for a total of four consensus-building dialogues (total of 27 minutes) between two groups of eight women in their 20s and 30s is the target. However, in this embodiment, an example of machine learning for head movement function detection for each of the 10 functions listed in Figure 8 will be described. Note that Figure 8 is a diagram showing the categories and function contents of head movement functions used in this embodiment. The head movement function detection model 40b for which machine learning has been performed is stored in the model storage unit 39.

(Regression model)
Regression model 50 is a random forest regression model (Reference 1) for predicting introspection report scores as predictors of subjective impressions.

(Reference 1) L. Breiman, "Random forests," Machine Learning, vol.45, pp.5-32, 2001.
The regression model 50 is not limited to a random forest, but may be any model such as Linear Regression or Support Vector Regression.

A model (MK+F) is constructed that adds at least one of the 22 head movement function features to six kinematic features as input information (feature amounts in this case). Then, the regression model 50 performs machine learning on the four introspection report score items "atmosphere," "fun," "motivation," and "concentration" as output information (correct answer data).

The number of items in the introspection report score can be set arbitrarily. This introspection report score is data that is divided into arbitrary time intervals and input, and each divided interval is called a "slot." This slot is the unit of prediction. The length of this interval may be a uniquely determined value, or the length of the interval may vary. If the length of the interval varies, the length of each interval is stored separately in the model storage unit 39.

<Functional configuration>
Next, each functional configuration of the prediction device will be described with reference to FIG.

The data acquisition unit 31 acquires coordinate information regarding the head movement of the interlocutor from video data obtained by capturing an image of the interlocutor using the camera 2a or the like.

Furthermore, the data acquisition unit 31 receives interlocutor data on the interlocutor obtained by the camera 2a or the like, and acquires information on speech sections indicating speech sections of the interlocutor from the interlocutor data. When the data acquisition unit 31 acquires information on speech sections, there are the following patterns.
(1) The data acquisition unit 31 acquires information about speech sections from audio data included in video data serving as interlocutor data obtained by photographing an interlocutor.
(2) The data acquisition unit 31 acquires information about speech sections from voice data serving as interlocutor data obtained by collecting the voices of interlocutors.
(3) The data acquisition unit 31 acquires information about a speech section by determining that an utterance has occurred when the motion vector of a feature point existing around the mouth of a conversation partner in the video data is equal to or greater than a predetermined threshold value.
(4) The data acquisition unit 31 acquires information on speech sections from a corpus in which speech content is transcribed with time-series information for video data.

After acquiring the video data (partner data) from the input device 2, the data acquisition unit 31 acquires coordinate information (information on the speech section) related to head movement in real time, but this is not limited to this. For example, after acquiring the video data (partner data) from the input device 2, the data acquisition unit 31 may temporarily store the video data (partner data) in the RAM 303 or the like, and when the prediction device 3 outputs a predicted value of the subjective impression, the data acquisition unit 31 may read out the video data (partner data) from the RAM 303 or the like.

The head movement detection unit 32a uses a head movement detection model to detect the presence or absence of head movement in relation to coordinate information relating to head movement. The head movement detection unit 32a also uses the head movement detection model to detect the presence or absence of speech in relation to information about a speech section.

If a corpus exists in which dialogue information is transcribed, the head movement detection unit 32a may output the presence or absence of speech from the corpus. In this case, the processing of the head movement detection unit 32a is omitted.

Furthermore, when video data is input as data acquired by the data acquisition unit 31, the head movement detection unit 32a may extract time series information of the coordinates located at the head from the video data by using a library such as OpenFace. If the Euclidean distance of movement between certain times in this extracted time series information exceeds a threshold, it can be determined that there is head movement, in which case the processing by the head movement detection unit 32a is omitted and the data is input directly from the data acquisition unit 31 to the head movement function detection unit 32b.

The head movement function detection unit 32b uses a head movement function detection model to detect head movement function in response to head movement presence/absence information indicating the presence or absence of head movement. The head movement function detection unit 32b also uses a head movement function detection model to detect head movement function in response to speech presence/absence information indicating the presence or absence of speech.

The feature amount calculation unit 33 calculates the feature amount of the head movement function based on the detection result by the head movement function detection unit 32b. The feature amount calculation unit 33 also calculates the head kinematic feature amount based on the presence or absence of head movement detected by the head movement detection unit 32a. Furthermore, the feature amount calculation unit 33 calculates the head kinematic feature amount based on the presence or absence of speech detected by the head movement detection unit 32a.

The subjective impression prediction unit 34 uses a regression model to predict the interlocutor's subjective impression of at least the head movement function feature quantity, out of the head movement function feature quantity and the head kinematic feature quantity. An example of the output information (data) of the subjective impression prediction unit 34 is shown in FIG. 11, which will be described later.

<Relationship between input and output information>
Here, the relationship between input information (data) and output information (data) in each function will be described with reference to Fig. 4. Fig. 4 is a diagram showing the relationship between input information (data) and output information (data) in each function.

As shown in FIG. 4, when the data acquisition unit 31 acquires video data, if the input information in the head movement detection unit 32a is coordinate information related to head movement, the output information in the head movement detection unit 32a is information on the presence or absence of head movement. Also, when the data acquisition unit 31 acquires voice data, etc., the output information in the head movement detection unit 32a is information on the presence or absence of speech.

Next, in the head movement function detection unit 32b, the input information is at least one of information on the presence or absence of head movement and information on the presence or absence of speech, and the output information is the detection result of the head movement function.

Next, in the feature calculation unit 33, when the input information is the detection result of head movement function, the output information is the feature of head movement function (see FIG. 10 described below). In addition, in the feature calculation unit 33, when the input information is at least one of information on the presence or absence of head movement and information on the presence or absence of speech, the output information is head kinematic feature (see FIG. 9 described below). Note that while the detection result of head movement function is essential as input information, information on the presence or absence of head movement and information on the presence or absence of speech are not essential. In other words, the feature calculation unit 33 must calculate the feature of head movement function, but does not must calculate the feature of head kinematic feature.

Next, in the subjective impression prediction unit 34, the input information is at least the head movement function feature among the head movement function feature and the head kinematic feature, and the output information is a predicted value of the interlocutor's subjective impression.

[Processing or Operation of the Embodiment]
Next, the processing or operation of this embodiment will be described with reference to FIGS.

<Data learning behavior>
First, the behavior of data learning will be described with reference to Fig. 5. Fig. 5 is a flowchart showing the behavior of data learning.

The data acquisition unit 31 receives interlocutor data output in time series from the input device 2 (S11). The data acquisition unit 31 then acquires coordinate information related to head movement (information indicating speech intervals) from the interlocutor data (S12).

Here, we will explain in detail the processing of the data acquisition unit 31.

(Use of video data)
First, a case where the data acquisition unit 31 utilizes video data will be described. The data acquisition unit 31 acquires video data from the camera 2a, the smartphone 2b, etc. When the data acquisition unit 31 receives the video data, it performs face tracking using OpenFace face recognition technology and measures the head attitude angle. The data acquisition unit 31 detects, for example, three components of the azimuth, elevation, and roll as the head attitude angle.

In addition, instead of Azimuth, Elevation, and Roll, angles based on other coordinate systems may be used. Furthermore, the data acquisition unit 31 may detect angular acceleration.

The data acquisition unit 31 may acquire coordinate information related to head movement from any program generated by a computer based on the movements of a person in the dialogue video. In this case, the data acquisition unit 31 extracts coordinates output from the program corresponding to time frame t. Furthermore, the data acquisition unit 31 may acquire coordinate information related to head movement using a wearable sensor, an acceleration sensor, motion capture, a magnetic sensor, or a device with these built in. In this case, the data acquisition unit 31 extracts information on angular velocity and angular acceleration corresponding to time frame t.

As a result of the above, the data acquisition unit 31 sets each component in time frame t to the following values as coordinate information related to head movement, and outputs these to the head movement detection unit 32a.

（対話者データの活用）
次に、データ取得部３１が、対話者データを活用する場合について説明する。データ取得部３１は、カメラ２ａ、スマートフォン２ｂ、マイク付きヘッドフォン２ｃ、及びマイク２ｄ等から対話者データを取得する。データ取得部３１は、対話者ごとの発話区間を取得する。

(Utilizing interlocutor data)
Next, a case where the data acquisition unit 31 utilizes interlocutor data will be described. The data acquisition unit 31 acquires interlocutor data from the camera 2a, the smartphone 2b, the headphones with microphone 2c, the microphone 2d, etc. The data acquisition unit 31 acquires speech sections for each interlocutor.

When the data acquisition unit 31 acquires voice data as interlocutor data, it acquires speech section data from sound pressure information. For example, the data acquisition unit 31 detects a sound pressure having a value equal to or greater than an arbitrary threshold as a speaking section.

When the data acquisition unit 31 acquires video data as interlocutor data, it extracts voice data from the video data. The data acquisition unit 31 may also use the video data to extract certain feature points (for example, feature points around the mouth) that can be acquired using OpenFace facial recognition technology, and may determine that speech is occurring when the motion vector of this feature point is equal to or greater than a certain threshold. Furthermore, if there is already a corpus in which speech content has been transcribed with time-series information for the video data, the data acquisition unit 31 may use the information from this corpus to treat the section in which the speech content exists as speech section data.

Furthermore, the data acquisition unit 31 may acquire the information from any program generated by a computer based on information about speech sections in the spoken video. In this case, the head movement detection unit 32a described below extracts information about the presence or absence of speech output from the program corresponding to time frame t.

As a result, the data acquisition unit 31 outputs information about the speech section (including information about time frame t) to the head movement detection unit 32a. Note that, as described above, the data acquisition unit 31 may acquire data about head movement instead of coordinate information about head movement.

Next, the head movement detection unit 32a performs machine learning using the head movement detection model to detect the presence or absence of head movement (presence or absence of speech) for coordinate information related to head movement (information indicating a speech section) (S13). Note that, if data related to head movement is acquired in step S12, the head movement detection unit 32a performs machine learning using the head movement detection model to detect the presence or absence of head movement for the data related to head movement. Also, as described above, step S13 by the head movement detection unit 32a may be omitted. In this case, the data acquired in step S12 is directly input to the head movement function detection unit 32b.

Next, the head movement detection unit 32a stores the head movement detection model after machine learning in the model storage unit 39 (S14).

Next, head movement function detection unit 32b performs machine learning using the head movement function detection model to detect the head movement function for head movement presence/absence information (information on the presence/absence of speech) (S15). Note that if step S13 by head movement detection unit 32a is omitted, head movement function detection unit 32b performs machine learning using the head movement function detection model to detect the head movement function for the data acquired in step S12.

Next, the head movement function detection unit 32b stores the head movement function detection model after machine learning in the model storage unit 39 (S16).

This concludes the explanation of data learning behavior.

<Data prediction behavior>
Next, the behavior of data prediction will be described with reference to Fig. 6 to Fig. 11. Fig. 6 is a flowchart showing the behavior of data prediction. Note that steps S21, S22, S23, and S24 below are the same as steps S11, S12, S13, and S15 described above, respectively, and therefore descriptions of the overlapping processes will be omitted.

The data acquisition unit 31 receives interlocutor data output in time series from the input device 2 (S12). Then, the data acquisition unit 31 acquires coordinate information related to head movement (information indicating a speech section) from the interlocutor data (S22). Note that the method of acquiring the coordinate information related to head movement and the information indicating a speech section may be a method other than machine learning. Also, as described above, the data acquisition unit 31 may acquire data related to head movement instead of coordinate information related to head movement.

Next, head movement detection unit 32a uses the trained head movement detection model to detect the presence or absence of head movement (presence or absence of speech) for coordinate information related to head movement (information indicating a speech section) (S23). Note that, if data related to head movement is acquired in step S22, head movement detection unit 32a uses the trained head movement detection model to detect the presence or absence of head movement for the data related to head movement. Also, as described above, step S23 by head movement detection unit 32a may be omitted. In this case, the data acquired in step S22 is directly input to head movement function detection unit 32b.

Next, the head movement function detection unit 32b uses the trained head movement function detection model to detect the head movement function in response to head movement presence/absence information (information on the presence/absence of speech) (S24).

In this case, head movement function detection unit 32b performs detection including at least one of the 10 types of head movement functions that have been machine-learned. When detecting all 10 types of head movement functions, head movement function detection unit 32b expresses the detection results of the 10 types of head movement functions at frame t during a conversation as follows, using the numbers shown in Figure 8.

検出結果には、上述の参考文献１が示唆するように、誤差が含まれていると考えられるが、本研究では誤差も含めて、この検出結果を頭部運動機能とみなし、頭部運動機能検出部３２ｂが検出結果を特徴量算出部３３へ出力する。

As suggested by Reference 1 mentioned above, the detection results are likely to contain errors. However, in this study, the detection results, including the errors, are regarded as head movement function, and the head movement function detection unit 32b outputs the detection results to the feature calculation unit 33.

As described above, if step S23 is omitted, the head movement function detection unit 32b uses the trained head movement function detection model to detect the head movement function for the data acquired in step S22.

Then, the feature calculation unit 33 calculates the feature of the head movement function based on the detection result of the head movement function (S25). The feature calculation unit 33 also calculates the head kinematic feature based on the information on the presence or absence of head movement (information on the presence or absence of speech) (S26). Note that this step S26 is not a required process.

Now, we will explain in more detail the processing performed by the feature calculation unit 33.

The feature calculation unit 33 acquires at least the head movement function detection results from the information on the presence or absence of head movement and the head movement function detection results, and calculates the feature amounts to be input to the next stage regression model 50.

Below, we define six types of head kinematic features and 22 types of head movement function features, and explain the case where at least the 22 types of head movement function features are used. Note that the features used are not limited to these, and other features may also be used.

Figure 9 shows a list of kinematic features. Here, if the number of frames included in a slot is T, the total number of frames in which head movement is detected within the slot is NF, and the number of head movement intervals within the slot is NI, then each feature amount is defined as shown in Figure 9. This will be explained in more detail.

Hrate indicates the ratio of frames in which head movement segments are detected to the total number of frames in one slot. Hnum indicates the number of head movement segments in each slot. Hdur indicates the average number of frames per head movement segment detected in each slot. These are expressed by (Equation 1).

は、スロット内の各頭部姿勢角である

is the head attitude angle in each slot

の標準偏差を示し、スロット内の頭部姿勢角の平均を

The standard deviation of the head attitude angle in the slot is shown.

とすると，次の様に表せられる。

Then, it can be expressed as follows:

続いて、図１０は、頭部運動機能の特徴を示す図である。図１０において、各スロットにおける総フレーム数をT ，発話関連機能si,t の検出フレーム数を| si |、反応機能ri,t の検出フレーム数を| ri |、その他機能c1,t の検出フレーム数を| c1 |とし、各機能区分の検出フレームの総和を|s|, |r|, |c| とすると，各特徴量は以下のように定義される。

Next, Fig. 10 is a diagram showing the features of head movement functions. In Fig. 10, if the total number of frames in each slot is T, the number of detection frames for speech-related functions si,t is |si|, the number of detection frames for reaction functions ri,t is |ri|, the number of detection frames for other functions c1,t is |c1|, and the sums of detection frames for each function division are |s|, |r|, |c|, then each feature is defined as follows:

(a) Function occurrence rate (first feature amount)
Feature occurrence rate

は各スロットのフレーム数T に対して各頭部運動機能

is the head movement function for each slot frame number T

の出現したフレームの割合を示し、（式２）ように表される。

This indicates the proportion of frames in which the

この（式２）は、１スロット内の各機能の現出量を表し，対話活動の量的側面を示唆する。

This (Equation 2) represents the amount of each function occurring within one slot, and suggests the quantitative aspect of dialogue activity.

(b) Functional content (second characteristic amount)
Functional content

は、スロット内の発話関連機能の総フレーム数| s |に対する各発話関連機能

is the ratio of each utterance-related feature to the total number of frames of utterance-related features in the slot |s|

の占める割合を示す。これは発話活動において、どの発話機能が相対的に多く現出していたかという発話活動の性質を表す特徴を有する。
また、

This shows the proportion of speech functions that were relatively frequently used in speech activities.
Also,

はスロット内の反応機能の総フレーム数| r |に対する各反応機能

is the total number of frames of the reaction function in the slot |r| for each reaction function

の占める割合を示す。これは、他者に対する反応の中で、どの反応機能が相対的に多く現出されていたかという反応の性質を表す特徴を有する。これらは（式３）表せられる。

This shows the proportion of the reaction function that is relatively more frequently expressed in the reaction to others. These are expressed by (Equation 3).

上述の機能出現率が、スロット内において機能の現出した量を表すものであるのに対して、機能含有率は、機能現出の絶対的な時間の長さには依存せず、これらの表出に含まれる機能の違いを機能間の相対的な量として捉える特徴を有する。例えば、発話時の頭部運動の時間長が同じであったとしても、機能含有率によって、強調の多い発話や問いかけの多い発話など話し手の振る舞いの違いを定量化することができる。同様に聞き手においても相槌に終始する場合や長く思考する場合など、機能含有率によって、傾聴時の振る舞いの性質を捉えることができる。

While the above-mentioned function occurrence rate represents the amount of a function that appears within a slot, the function content rate has the characteristic of not depending on the absolute length of time that the function appears, but rather capturing the difference in the functions contained in these expressions as the relative amount between functions. For example, even if the length of time of head movement during speech is the same, the function content rate can quantify the difference in speaker behavior, such as speech with a lot of emphasis or speech with a lot of questions. Similarly, the function content rate can capture the nature of the listener's behavior during attentive listening, such as whether the listener only responds with a nod or whether the listener thinks for a long time.

(c) Functional division composition ratio (third characteristic amount)
Functional division composition ratio

はスロット内の１０種の全機能の出現フレーム数の総和

is the total number of frames in which all 10 features appear in the slot

に対する発話関連機能、反応機能、その他機能の各々の占める割合であり、（式４）によって表される。

This is the ratio of speech-related functions, reaction functions, and other functions to the total, and is expressed by (Equation 4).

この（式４）は、スロット内における各参加者の対話活動の概略を示す。例えば、話し手としての活動が多い場合には

This (Equation 4) shows an outline of the dialogue activity of each participant in a slot. For example, if there is a lot of activity as a speaker,

が大きくなる。

becomes larger.

Also, if you are mostly active as a listener

が相対的に大きくなる。

becomes relatively large.

The feature calculation unit 33 outputs the feature calculated as described above to the subjective impression prediction unit 34.

Next, the subjective impression prediction unit 34 uses a regression model to output predicted values (predicted subjective impressions) of each item of the interlocutor's introspection report score for at least the head movement function feature quantities among the head movement function feature quantities and the head kinematic feature quantities (S27).

Here, the processing of the subjective impression prediction unit 34 will be described in more detail. FIG. 11 is a diagram showing predicted values of subjective impressions as introspection report scores for each item. In FIG. 11, the four introspection report score items, "atmosphere," "fun," "motivation," and "concentration," are divided into two-minute intervals and values are expressed as integers from 1 to 9. The integer value may be an arbitrarily input value (A) or a value output from some device (B). The integer value may also be a value obtained as a result of calculating each value (A) and (B) using an arbitrary method.

The subjective impression prediction unit 34 also inputs the feature quantities obtained from each slot during the conversation into the machine-learned regression model 50, and outputs a predicted value for each item of the introspection report score. Note that the predicted value may be either a nine-level discrete value or a real value.

This concludes the explanation of data prediction behavior.

[Effects of the embodiment]
As described above, the present embodiment has an effect of being able to predict the subjective impressions (feelings) of the interlocutors during a conversation. This leads to the establishment of technology that improves the conversation experience of the interlocutors, such as building relationships between the interlocutors and ensuring psychological safety.

In addition, by using both head movement during dialogue and features of head movement function, it becomes possible to make predictions that are robust to differences in situations during dialogue.

Furthermore, by considering the rate of occurrence of functions, it is possible to improve the accuracy of predictions by using the quantitative aspect of dialogue activity, by considering the rate of content of functions, it is possible to determine the nature of speech activity, and by considering the composition ratio of functional categories, it is possible to use the outline of each participant's dialogue activity as a feature.

〔supplement〕
The present invention is not limited to the above-described embodiment, and may have the following configurations or processes (operations).
(1) The prediction device 3 of the present invention can be realized by a computer and a program. This program can be recorded on a recording medium or provided via a communication network.
(2) In the above embodiment, a camera 2a or the like is shown as an example of the input device 2, but this is not limited to this and may be, for example, a desktop personal computer, a notebook computer, a tablet terminal, a smart watch, a car navigation device, etc.
(3) Each of the CPUs 301 and 501 may be multiple, rather than just single.

1 Communication system 3 Prediction device 4 Monitor 31 Data acquisition unit (an example of a data acquisition means)
32 Detector 32a Head movement detector (an example of a head movement detector)
32b Head movement function detection unit (an example of a head movement function detection means)
33 Feature amount calculation unit (an example of a feature amount calculation means)
34 Subjective impression prediction unit (an example of a subjective impression prediction means)
39 Model storage unit (an example of a model storage means)
40a Head movement detection model 40b Head movement function detection model 50 Regression model

Claims (11)

A prediction device for predicting a subjective impression held by a person in a conversation , comprising:
A data acquisition means for acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor;
a head movement function detection means for detecting a head movement function for the data relating to the head movement using a head movement function detection model;
a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means;
a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model;
A prediction device comprising: A prediction device for predicting a subjective impression held by a person in a conversation , comprising:
A data acquisition means for acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor;
head movement detection means for detecting the presence or absence of head movement with respect to the head movement related data using a head movement detection model;
a head movement function detection means for detecting a head movement function corresponding to the head movement presence/absence information indicating the presence or absence of head movement using a head movement function detection model;
a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means;
a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model;
A prediction device comprising: the data regarding head movement is coordinate information regarding the head movement,
3. The prediction device according to claim 2, wherein the head movement detection means detects the presence or absence of the head movement with respect to the coordinate information relating to the head movement by using the head movement detection model. the feature amount calculation means calculates a head kinematic feature amount based on the presence or absence of the head movement detected by the head movement detection means;
The prediction device according to claim 2 or 3, wherein the subjective impression prediction means predicts the interlocutor's subjective impression of the head motor function feature and the head kinematic feature by using the regression model. A prediction device for predicting a subjective impression held by a person in a conversation , comprising:
a data acquisition means for receiving interlocutor data relating to an interlocutor and acquiring, from the interlocutor data, information on a speech section indicating an utterance section of the interlocutor;
a head movement detection means for detecting the presence or absence of speech in the speech section information by using a head movement detection model;
a head movement function detection means for detecting head movement function in response to the speech presence/absence information indicating the presence or absence of speech using a head movement function detection model;
a feature amount calculation means for calculating a feature amount of the head movement function based on a detection result by the head movement function detection means;
a subjective impression prediction means for predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity by using a regression model;
A prediction device comprising: The feature amount calculation means calculates a head kinematic feature amount based on the presence or absence of the speech detected by the head movement detection means,
The prediction device according to claim 5 , wherein the subjective impression prediction means predicts the interlocutor's subjective impression of the head motor function feature amount and the head kinematic feature amount by using the regression model. The prediction device according to claim 5 or 6, characterized in that the data acquisition means acquires the information on the speech section from audio data included in the video data as the interlocutor data obtained by filming the interlocutor, acquires the information on the speech section from audio data as the interlocutor data obtained by collecting the interlocutor's voice, acquires the information on the speech section by determining that the interlocutor has spoken when the motion vector of a feature point existing around the mouth of the interlocutor in the video data is equal to or greater than a predetermined threshold, or acquires the information on the speech section from a corpus in which the speech content of the video data is transcribed while having time series information. A prediction method executed by a prediction device for predicting a subjective impression held by a person in a conversation , comprising:
a data acquisition step of acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor;
a head movement function detection step of detecting a head movement function for the data relating to the head movement using a head movement function detection model;
a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result in the head movement function detection step;
a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model;
A prediction method comprising the steps of: A prediction method executed by a prediction device for predicting a subjective impression held by a person in a conversation , comprising:
a data acquisition step of acquiring data on head movement of the interlocutor from video data obtained by photographing the interlocutor;
a head movement detection step of detecting the presence or absence of head movement with respect to the head movement related data using a head movement detection model;
a head movement function detection step of detecting a head movement function corresponding to head movement presence/absence information indicating the presence/absence of head movement using a head movement function detection model;
a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result by the head movement function detection step;
a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model;
A prediction method comprising the steps of: A prediction method executed by a prediction device for predicting a subjective impression held by a person in a conversation , comprising:
a data acquiring step of receiving interlocutor data related to an interlocutor and acquiring, from the interlocutor data, information on a speech section indicating an utterance section of the interlocutor;
a head movement detection step of detecting the presence or absence of speech in the speech section information by using a head movement detection model;
a head movement function detection step of detecting head movement function in response to the speech presence/absence information indicating the presence or absence of speech using a head movement function detection model;
a feature amount calculation step of calculating a feature amount of the head movement function based on a detection result by the head movement function detection step;
a subjective impression prediction step of predicting a subjective impression of the interlocutor with respect to the head movement function feature quantity using a regression model;
A prediction method comprising the steps of: A program for causing a computer to execute the method according to any one of claims 8 to 10.

Images