JP7697400B2 - Communication prediction device and communication prediction method - Google Patents

Description

The present invention relates to a communication prediction device and a communication prediction method.

Patent Document 1 discloses a conversation support system that encourages participants who have missed the appropriate timing to speak during a conversation to speak. This system estimates the next speaker probability, which is the probability that each participant will be the next to speak at any given time, based on the measurement results of the non-verbal behavior of each participant during the conversation, and estimates the predicted next speaker, who is the participant who should speak next, based on the next speaker probability of each participant.

JP 2017-123027 A

The technology in Patent Document 1 requires a great deal of effort to collect data, since data acquired from past conversations is used as training data to train a model for estimating the next speaker probability.

The objective of the present invention is to provide a technology that can accurately predict the next speaker while reducing the effort required to collect training data.

In order to solve the above problem, a communication prediction device according to an embodiment of the present invention includes an acquisition unit that acquires information input by multiple users in a social networking service, a learning unit that performs machine learning to learn a model for predicting a next speaker from multiple conversation participants based on the acquired information, and a derivation unit that derives linguistic features of multiple users based on posted statements input by the multiple users included in the acquired information, and derives relationship features representing relationships between a target user and other users and information representing the number of speaker turns from the target user to other users based on information on reactions of a target user to posted statements of other users input by the target user included in the acquired information . The learning unit receives the linguistic features of the target user, the linguistic features of the other users, and the relationship features between the target user and other users as input, and learns a model using information representing the number of speaker turns from the target user to other users as a correct answer label.

Another aspect of the present invention is also a communication prediction device. The device includes an acquisition unit that acquires information about communication between each conversation participant and other conversation participants during a conversation between multiple conversation participants, a prediction unit that predicts a next speaker from among the multiple conversation participants based on the acquired information about communication using a trained model that has been machine-learned based on information input by multiple users in a social networking service , and a derivation unit that derives a second feature vector for each of the multiple conversation participants based on the acquired information about communication . The prediction unit acquires a first feature vector for each of the multiple conversation participants from a storage unit that stores a first feature vector for each user derived from information input by multiple users in the social networking service, and predicts a next speaker based on the acquired first feature vectors and the derived second feature vectors using the trained model.

Yet another aspect of the present invention is a communication prediction method. This method is a communication prediction method executed by a computer, and includes the steps of: acquiring information input by a plurality of users in a social networking service; learning a model for predicting a next speaker from among a plurality of conversation participants based on the acquired information; deriving linguistic features of a plurality of users based on posted statements input by the plurality of users included in the acquired information; and deriving relationship features representing relationships between a target user and other users and information representing the number of speaker turns from the target user to other users based on information on reactions to posted statements input by the target user included in the acquired information . In the machine learning step, the linguistic features of the target user, the linguistic features of the other users, and the relationship features between the target user and other users are input, and a model is learned using information representing the number of speaker turns from the target user to other users as a correct answer label.

Yet another aspect of the present invention is also a communication prediction method. This method is a communication prediction method executed by a computer, and includes the steps of: acquiring information on communication between each conversation participant and other conversation participants during a conversation between multiple conversation participants; predicting a next speaker from among the multiple conversation participants based on the acquired information on communication using a trained model that has been machine- learned based on information input by multiple users in a social networking service; and deriving a second feature vector for each of the multiple conversation participants based on the acquired information on communication . In the predicting step, a first feature vector for each of the multiple conversation participants is acquired from a storage unit that stores a first feature vector for each user derived from information input by multiple users in a social networking service, and predicting a next speaker based on the acquired first feature vectors and the derived second feature vectors using the trained model.

The present invention provides technology that can accurately predict the next speaker while reducing the effort required to collect training data.

FIG. 2 is a diagram for explaining an overview of the processing of the communication prediction system according to the first embodiment; FIG. 1 illustrates a configuration of a communication prediction system according to a first embodiment. FIG. 11 is a diagram showing an example of SNS information. FIG. 13 is a diagram illustrating an example of linguistic features of a plurality of users. FIG. 13 is a diagram illustrating an example of the relationship between the number of speaker turns and frequency. FIG. 1 is a diagram illustrating a distribution of characteristic feature vectors of multiple users. 3 is a flowchart showing a learning process of the communication prediction system of FIG. 2 . 3 is a flowchart showing a prediction process of the communication prediction system of FIG. 2 .

(First embodiment)
1 is a diagram for explaining an overview of the process of the communication prediction system according to the first embodiment. The communication prediction system predicts information about a conversation between multiple conversation participants in a physical space by using communication information in a cyberspace. An example of three conversation participants u1, u2, and u3 will be described below, but the number of participants is not particularly limited.

The communication prediction system acquires voice data, video data, and the like of multiple conversation participants u1, u2, and u3 during a conversation, and periodically acquires information about the communication between each conversation participant and the other conversation participants based on the acquired data. The information about communication includes information about the content of the conversation, gaze, nodding, and the like.

The trained model 50 is a model for predicting the next speaker from among multiple conversation participants, and has been machine-trained in advance based on information such as posts and reactions entered by multiple users on a social networking service (hereinafter also referred to as SNS). Each of the conversation participants u1, u2, and u3 uses an SNS, and it is assumed that the multiple users to be trained include the conversation participants u1, u2, and u3, but as described below, the conversation participants do not need to use an SNS.

The communication prediction system uses the trained model 50 to periodically predict, for each conversation participant, the number of speaker turns per unit time from that conversation participant to each of the other conversation participants, based on information regarding the communication between the conversation participants u1, u2, and u3 and information entered into the SNS by the conversation participants u1, u2, and u3. The prediction results can be expressed as a matrix as shown in the figure. The unit time and prediction frequency can be appropriately determined through experiments or simulations; for example, the unit time may be one minute, and the prediction frequency may be once per minute.

In the illustrated example, the number of speaker turns per unit time from the speaker, conversation participant u1, to the next speaker, conversation participant u2, is predicted to be "2," and the number of speaker turns per unit time to conversation participant u3 is predicted to be "20."

The number of turns per unit time from the speaker, conversation participant u2, to the next speaker, conversation participant u1, is predicted to be "5," and the number of turns per unit time to conversation participant u3 is predicted to be "10."

The number of turns per unit time from the speaker, conversation participant u3, to the next speaker, conversation participant u1, is predicted to be "10," and the number of turns per unit time to conversation participant u2 is predicted to be "3."

In this example, transitions in which the first speaker and the next speaker are the same are excluded, and the number of speaker changes is fixed to "0." However, depending on the application, transitions in which the first speaker and the next speaker are the same may also be defined as speaker changes, and the number of speaker changes may also be predicted.

The number of speaker changes per unit time represents the frequency with which speakers change, and also represents the speaker transition probability. For example, if the speaker is conversation participant u1, the probability that the next speaker will be conversation participant u2 is 100 x 2/(2 + 20) [%], and the probability that the next speaker will be conversation participant u3 is 100 x 20/(2 + 20) [%], so it can be predicted that the next speaker will be conversation participant u3. In other words, the communication prediction system predicts the next speaker from among multiple conversation participants.

Compared to collecting data from actual conversations as training data, collecting SNS data requires less effort and allows more data about more users to be collected in a shorter time. By using a prediction model trained using such a large amount of training data, it is possible to accurately predict the next speaker.

By predicting the next speaker, for example, it is possible to inform multiple conversation participants of the predicted next speaker. Depending on whether the predicted next speaker matches the actual next speaker, it is also possible to check whether the speech was delivered at the appropriate time.

The number of speaker changes per unit time can also be said to represent the amount of conversation per unit time. In the example of FIG. 1, the amount of conversation of the conversation participant u2 following the speech of the conversation participant u1 can also be expressed as "2", and the amount of conversation of the conversation participant u3 following the speech of the conversation participant u1 can also be expressed as "20". In other words, the communication prediction system also predicts the amount of conversation per unit time of each of the other conversation participants following the speech of the conversation participant for each conversation participant. Also, for example, the amount of conversation of the conversation participant u2 following the speech of the conversation participant u1 and the amount of conversation of the conversation participant u1 following the speech of the conversation participant u2 are each regarded as the amount of conversation between the conversation participants u1 and the conversation participants u2. Therefore, the amount of conversation between the conversation participants u1 and the conversation participants u2 is predicted to be "2" + "5" = "7". In other words, the communication prediction system can also be said to predict the amount of conversation per unit time for each combination of two conversation participants among multiple conversation participants. In this way, the amount of conversation can be predicted in advance. In other words, it is possible to predict in advance who is likely to talk and how much.

The predicted conversation volume can be used, for example, as the conversation volume per unit time between two members in the communication support system described in Patent Application No. 2021-163089 previously filed by the applicant. For example, the future stress state value of conversation participant u1 can be predicted from the current stress state values of each of conversation participants u1, u2, and u3, the predicted conversation volume between conversation participant u1 and conversation participant u2, and the predicted conversation volume between conversation participant u1 and conversation participant u3. In other words, it is possible to predict an individual's future stress state value taking into account the influence of conversations with others.

Figure 2 shows the configuration of the communication prediction system 1 of the first embodiment. The communication prediction system 1 includes a microphone 2, a camera 4, a sensor 6, and a communication prediction device 10. Although not shown, the communication prediction system 1 has multiple microphones 2, multiple cameras 4, and multiple sensors 6.

The microphone 2 captures the voices of multiple conversation participants and supplies the captured voice data to the communication prediction device 10. The camera 4 captures images of the multiple conversation participants and supplies the captured image data to the communication prediction device 10. The sensor 6 is attached to the body of each conversation participant, detects the posture of each conversation participant, and supplies the detected sensor data to the communication prediction device 10. The sensor 6 may also detect the heart rate of each conversation participant.

The communication prediction device 10 includes a processing unit 12, an SNS data storage unit 14, a feature dictionary storage unit 16, and a trained model storage unit 18. The processing unit 12 includes a first acquisition unit 20, a first derivation unit 22, a learning unit 24, a second acquisition unit 26, a voice recognition unit 28, a second derivation unit 30, a prediction unit 32, and an output unit 34.

The configuration of the processing unit 12 can be realized in hardware terms by the CPU, memory, and other LSIs of any computer, and in software terms by programs loaded into memory, but here we depict functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by hardware alone, software alone, or a combination of both.

First, the learning process will be described. The first acquisition unit 20 acquires information entered by multiple users on the SNS (hereinafter also referred to as SNS data) and stores the acquired SNS data in the SNS data storage unit 14. The SNS data includes information such as the content of posted messages, reactions to the posted messages using emoticons and the like, replies to the posted messages, and images or profile images of the user's accounts.

The first acquisition unit 20 performs preprocessing on the acquired SNS data as necessary, creates learning data from the SNS data, and supplies the learning data to the first derivation unit 22. The preprocessing is, for example, a formatting process for the text of the posted message, and includes processes for standardizing full-width characters such as alphabets and katakana to half-width characters, deleting emoticons, and performing word segmentation.

The first derivation unit 22 derives a first feature vector for each user from the learning data. The first feature vector includes linguistic features, image features, and relationship features.

The first derivation unit 22 derives linguistic features of multiple users based on the posts, reactions to the posts, and replies to the posts entered by multiple users, which are included in the information acquired by the first acquisition unit 20. The linguistic features express topics that the user is good at, topics that the user is interested in, the user's expertise, etc.

The first derivation unit 22 derives the linguistic feature of the target user based on the frequency of occurrence of each word in the posted text entered by the target user and the frequency of occurrence of each word in the posted text of other users that was the subject of a reaction or reply entered by the target user. The frequency of occurrence of each word in the reply entered by the target user may also be used. In other words, the first derivation unit 22 derives the linguistic feature of the target user based on the posted text entered by the target user and the reaction and reply, which are information entered by the target user regarding reactions to the posted text of other users.

Figure 3 shows an example of SNS information. Figure 3 shows an example of a screen displaying a message 60 posted by user A, a reply 62 to the message 60 posted by user B, and reactions 64 to the message posted by user A by multiple users. An account image 66 of user A and an account image 68 of user B are also displayed.

Figure 4 shows an example of linguistic features of multiple users u1 to um (m is an integer of 2 or more). w1 to wn (n is an integer of 2 or more) represent words. Each element of the matrix represents the frequency of occurrence of a word in a posted text. One row is a vector representing the linguistic features of a certain user. In the example shown, in the vector of the linguistic features of user u1, the frequency of occurrence of word w1 is "0", and the frequency of occurrence of word w2 is "10". Since the vocabulary of words is very large, the linguistic features can have, for example, tens of thousands of dimensions, and the matrix also becomes sparse, so the dimensions may be reduced using a low-rank approximation method (NMF) or the like. The dimension-reduced linguistic features of each user may have, for example, 200 dimensions.

The first derivation unit 22 also derives image features of each user based on an image for identifying each user in the SNS. The image for identifying a user is an image of the user's account or a profile image. The first derivation unit 22 derives image features based on RGB values of multiple pixels such as an account image. If there is a model that estimates personality using an account image or the like as input, the first derivation unit 22 may use the output value of the model as the image feature. The image feature is, for example, one-dimensional, and represents the individual characteristics and inner nature of each user, such as personality. For example, a larger numerical value of the image feature may represent a more extroverted personality. By using the image feature, the user's personality can also be reflected in the model, and prediction accuracy can be improved.

The first derivation unit 22 derives relationship features representing the relationship between the target user and other users and information representing the number of speaker changes from the target user to other users, based on information on reactions and replies to posts by other users entered by the target user contained in the acquired information.

Relationship features can also be called network features, and represent the relationships between users. Each user is a node, and when there is a reaction or reply, an edge is created between the original poster and the user who made the reaction or reply. If there are multiple reactions, the weight of the edge is added up.

The value of the relationship feature may be the difference between the average of the entire network, such as the shortest path length, similar selectivity, or degree, and each value between specific people. The relationship feature is three-dimensional, including, for example, a value based on the shortest path length, a value based on similar selectivity, and a value based on degree.

The number of times the target user takes turns speaking to other users is derived based on a predetermined relational expression, and the greater the number of reactions, etc., input by the target user to posts by other users, the greater the number of times the target user takes turns speaking to other users. The predetermined relational expression can be determined appropriately through experiments, for example, so as to approximate the relationship between the number of reactions, etc., input by the target user to posts by other users, and the number of times the target user takes turns speaking to other users in actual conversations. A distinction is made between the number of times the target user A takes turns speaking to other users B and the number of times the other users B takes turns speaking to the target user A.

The first derivation unit 22 concatenates the linguistic features and image features for each user to create a feature vector unique to the user, creates a feature dictionary including this feature vector, and stores the created dictionary in the feature dictionary storage unit 16. The feature vector unique to one user has, for example, 200 + 1 = 201 dimensions.

The first derivation unit 22 also stores, in the feature dictionary storage unit 16, the relationship features of each combination of two users among the multiple users, and information representing the number of speaker changes. In other words, it can be said that the feature dictionary storage unit 16 stores the first feature vector of each user.

The learning unit 24 performs machine learning on a model for predicting the next speaker from among multiple conversation participants based on the information acquired by the first acquisition unit 20, and stores the learned model in the learned model storage unit 18. Specifically, the learning unit 24 acquires the first feature vector of the target user, the first feature vectors of the other users, and information representing the number of speaker changes from the target user to the other users from the feature dictionary storage unit 16, and learns a model based on these. The first feature vector of the target user includes the linguistic feature and image feature of the target user, and the relationship feature between the target user and the other users, and is, for example, 204 dimensions. The first feature vector of the other users includes the linguistic feature and image feature of the other users, and the relationship feature between the other users and the target user, and is, for example, 204 dimensions. The relationship feature between the target user and the other users may be the same as the relationship feature between the other users and the target user.

More specifically, the learning unit 24 inputs the linguistic features and image features of the target user, the relationship features between the target user and other users, the linguistic features and image features of the other users, and the relationship features between the other users and the target user, and learns a model using information representing the number of speaker turns from the target user to the other users as the correct answer label. The learning unit 24 repeats learning a model for every two users out of the multiple users. For two users, learning can be performed twice, once when information representing the number of speaker turns from the target user to the other users is used as the correct answer label, and once when information representing the number of speaker turns from the other users to the target user is used as the correct answer label. By learning for every two users, the number of learning samples can be inflated, and a model that is robust to the number of conversation participants can be created. For example, in the case of 100 users, 100 x 100 - 100 = 9900 samples can be secured.

The model is, for example, a multi-layer neural network model. The model may be a regression model or a classification model. In the case of a regression model, the learning unit 24 learns the numerical value of the number of speaker turns as the correct label. For example, in a conversation from user A to user B, the 204-dimensional feature of user A and the 204-dimensional feature of user B are input, and "54", which is the number of speaker turns from user A to user B derived from SNS data, becomes the correct label.

On the other hand, for example, if the target is a conversation from user B to user A, the 204-dimensional feature of user A and the 204-dimensional feature of user B are input, and the number of speaker turns from user B to user A, derived from the SNS data, "30" is the correct label.

In order to distinguish and learn the number of speaker changes from user A to user B and the number of speaker changes from user B to user A, a vector for identifying the order of the first feature vectors of the two users may also be supplied to the learning unit 24.

In the case of a classification model, the learning unit 24 learns the quantized value of the number of speaker changes as the correct label. The quantized value of the number of speaker changes is information that represents the number of speaker changes.

Figure 5 shows an example of the relationship between the number of speaker turns and frequency. If the number of speaker turns is smaller than threshold th1, the number of speaker turns is defined as "small." If the number of speaker turns is equal to or greater than threshold th1 and less than threshold th2, the number of speaker turns is defined as "medium." If the number of speaker turns is equal to or greater than threshold th2, the number of speaker turns is defined as "large." Thresholds th1 and th2 are set by a human in advance.

For example, when targeting a conversation from user A to user B, the 204-dimensional feature of user A and the 204-dimensional feature of user B are input, and the number of speaker turns from user A to user B derived from SNS data, "large", is the correct label.

When targeting a conversation from user B to user A, the 204-dimensional feature values of user A and the 204-dimensional feature values of user B are input, and the number of speaker turns from user B to user A derived from SNS data, "medium," is the correct label.

However, it is possible that the actual conversation participants do not use SNS, and their data is not included in the SNS data. In this case, the following process is performed.

Figure 6 shows a schematic distribution of unique feature vectors for multiple users. As mentioned above, the dimension of the unique feature vector is, for example, 201 dimensions, but in Figure 6 it is simplified to 2 dimensions.

The first derivation unit 22 classifies the unique feature vectors of multiple users into multiple clusters by unsupervised learning using k-means or the like, and derives the unique feature vector of a virtual representative user representing each cluster. The first derivation unit 22 derives the unique feature vector of the representative user representing the center of gravity of the multiple feature vectors of the cluster for each cluster. In the example of FIG. 6, the users are classified into five clusters C1 to C5, and five unique feature vectors V1 to V5 are derived for five fictitious representative users. In other words, if there are 100 users, features for 105 users are created.

When a user who has no information entered into the SNS is also a target for learning, the first derivation unit 22 sets the unique feature vector of the representative user of one of the clusters as the unique feature vector of the user, and stores the unique feature vector in the feature dictionary storage unit 16. For example, a system administrator or the like may check whether a user for whom no SNS data exists has an extroverted or introverted personality, and select the unique feature vector of a representative user of a cluster to which a user whose personality is relatively close to that of the user belongs.

The first derivation unit 22 derives relationship features and correct answer labels for users whose information has not been entered into the SNS based on the users classified into each cluster, and stores them in the feature dictionary storage unit 16.

For example, if it is known that the number of speaker changes from user A to user B is "54," the number of speaker changes from cluster C1, in which user A is classified, to cluster C3, in which user B is classified, is considered to be "54." Furthermore, the relationship feature between cluster C1, in which user A is classified, and cluster C3, in which user B is classified, is considered to be the relationship feature between user A and user B.

In other words, if a user (say, User X) who has no information entered into the SNS is classified into cluster C1, and the target conversation from User X to User B is the target, User X's unique feature vector V1 and the relationship feature between User A and User B, and User B's unique feature vector 70 and the relationship feature between User B and User A are input, and "54", which is the number of speaker changes from User A to User B, is the correct label.

In this way, even if the actual conversation participants do not use SNS, learning can be done using feature vectors that may represent the characteristics of those conversation participants.

In addition, if there are missing values in the SNS data for some reason, the first derivation unit 22 may complement the missing values using a trained missing value estimation model, and derive the feature vector of each user based on the SNS data after completion of the missing values. For example, the technology described in Patent Application No. 2021-165372 previously filed by the present applicant may be used to complement the missing values.

Next, we will explain the process of prediction that is performed by conversation participants during a conversation after learning.

The second acquisition unit 26 acquires voice data supplied from the microphone 2, image data supplied from the camera 4, and sensor data supplied from the sensor 6 during a conversation between multiple conversation participants, and supplies the voice data to the voice recognition unit 28 and the image data and sensor data to the second derivation unit 30. This process corresponds to the second acquisition unit 26 acquiring information regarding the communication between each conversation participant and the other conversation participants.

The speech recognition unit 28 performs speech recognition on the speech data acquired by the second acquisition unit 26, performs preprocessing on the speech recognition result as necessary, creates text data, and supplies the text data to the second derivation unit 30. The preprocessing includes, for example, standardizing full-width characters to half-width characters, anonymizing people's proper nouns, and performing word segmentation.

The second derivation unit 30 derives linguistic features for each conversation participant based on the text data. The second derivation unit 30 derives linguistic features for the target conversation participant from the frequency of occurrence of each word in the text data representing the conversation content of the target conversation participant. This linguistic feature is also a vector with the same structure as the linguistic feature derived by the first derivation unit 22 shown in Figure 4, and has the same dimensions.

The second derivation unit 30 derives image features for each conversation participant based on the sensor data. The posture of a conversation participant during a conversation identified by the sensor data is considered to represent the personality of the conversation participant. Therefore, the second derivation unit 30 derives image features of the target conversation participant from the posture derived from the sensor data of the target conversation participant. This image feature is also a one-dimensional vector, like the image feature derived by the first derivation unit 22.

The second derivation unit 30 derives relationship features for each pair of two conversation participants based on the image data. The second derivation unit 30 treats each conversation participant as a node, and if the image data identifies that the two conversation participants exchanged glances or nodded with each other, increases the weight of the edge between the two conversation participants. Similar to the processing by the first derivation unit 22, the second derivation unit 30 derives three-dimensional relationship features based on the nodes and edges of multiple conversation participants. In other words, the second derivation unit 30 derives relationship features between a target conversation participant and the other conversation participants based on information regarding the behavior of the target conversation participant toward the other conversation participants.

These processes correspond to the second derivation unit 30 deriving a second feature vector for each of the multiple conversation participants based on the acquired information about the communication. The second feature vector includes linguistic features, image features, and relationship features. The number of dimensions of the second feature vector is the same as the number of dimensions of the first feature vector.

The prediction unit 32 predicts the next speaker from among the multiple conversation participants based on the acquired communication information, using a trained model that is machine-learned based on information entered by multiple users on the SNS and stored in the trained model storage unit 18. For each conversation participant, the prediction unit 32 predicts the amount of conversation per unit time of each of the other conversation participants following the speech of the conversation participant.

The prediction unit 32 acquires the first feature vector of each of the multiple conversation participants from the feature dictionary storage unit 16, which stores the first feature vector of each user derived from information entered by the multiple users on the SNS. Even if a conversation participant who does not use the SNS is included, as described above, the first feature vector of that conversation participant is also stored in the feature dictionary storage unit 16.

The prediction unit 32 uses the trained model to predict the next speaker based on the acquired multiple first feature vectors and the derived multiple second feature vectors. The prediction unit 32 mixes the acquired first feature vector and the derived second feature vector for each conversation participant. Specifically, the prediction unit 32 derives a third feature vector by performing a weighted addition of corresponding features in the acquired first feature vector and the derived second feature vector for each conversation participant, and inputs the derived third feature vector into the trained model to predict the next speaker.

The weights may be different for the linguistic features, image features, and relationship features. The weight for a specific feature of the first feature vector may be zero, in which case the specific feature of the first feature vector is replaced with the feature of the second feature vector. The weights may be determined separately by machine learning.

More specifically, the prediction unit 32 inputs the third feature vector of the target conversation participant and the third feature vector of the other conversation participants into the learned model, and outputs the number of speaker turns per unit time from the target conversation participant to the other conversation participants. The third feature vector of the target conversation participant includes the linguistic feature and image feature of the target conversation participant, and the relationship feature between the target conversation participant and the other conversation participants. The third feature vector of the other conversation participant includes the linguistic feature and image feature of the other conversation participant, and the relationship feature between the other conversation participant and the target conversation participant. The prediction unit 32 repeats the above prediction process for each two conversation participants out of the multiple conversation participants. Two predictions can be made for two conversation participants. A vector for specifying the order of the third feature vectors of the two conversation participants may also be supplied to the prediction unit 32.

The output unit 34 outputs the result predicted by the prediction unit 32. The output unit 34 may output the matrix of FIG. 1 output from the model trained by the prediction unit 32.

By inputting both the information entered into SNS and the information acquired during conversation into the trained model, it is possible to make more accurate predictions by also reflecting information about the physical space during the conversation.

Figure 7 is a flowchart showing the learning process of the communication prediction system 1 of Figure 2. The first acquisition unit 20 acquires SNS data of multiple users (S10) and creates learning data from the SNS data (S12). The first derivation unit 22 creates a feature dictionary from the learning data and stores it in the feature dictionary storage unit 16 (S14). The learning unit 24 trains a prediction model by machine learning (S16), stores the trained model (S18), and ends the process.

Figure 8 is a flowchart showing the prediction process of the communication prediction system 1 of Figure 2. This process is executed after learning is completed. It is determined whether a conversation has started (S30), and if not (N in S30), the process returns to S30.

If a conversation has started (Y in S30), the second acquisition unit 26 acquires data from the microphone 2, camera 4, and sensor 6 (S32), and the voice recognition unit 28 recognizes the voice data and creates text data (S34).

The second derivation unit 30 derives features for each conversation participant based on the text data, image data, and sensor data (S36). For each conversation participant, the second derivation unit 30 creates a feature vector by mixing the features acquired from the feature dictionary with the derived features (S38). The prediction unit 32 inputs the created feature vector into the trained model and makes a prediction (S40). If a conversation is in progress (Y in S42), the process returns to S32. If a conversation is not in progress (N in S42), the process ends.

Next, a specific example of the process performed by the communication prediction system 1 will be described.
[study]
Assume that the frequency of occurrence of the word "camping" in user A's SNS posts is 100, and the frequencies of occurrence of "fishing" and "politics" are 0.

Let's assume that the frequency of occurrence of "fishing" in User B's SNS posts is 200, and the frequency of occurrence of "camping" and "politics" is 0.

Let us assume that the frequency of occurrence of "camping" and "fishing" in User C's SNS posts is 0, and the frequency of occurrence of "politics" is 50. Hereinafter, User A, User B, and User C will also be simply referred to as A, B, and C.

Assume that the relationship feature indicating the relationship between A and B is 10, and the relationship feature indicating the relationship between A and C and between B and C is 0.

Assume that the number of speaker turns from A to B = 50, the number of speaker turns from B to A = 40, and the number of speaker turns from A to C, C to A, B to C, and C to B = 0.

The model is trained using SNS data from 100 users, including A, B, and C.

[prediction]
Three users A, B, and C are having a conversation, and a matrix is output from the trained model at the current time t1.

Assume that by time t1, the frequency of occurrence of "camping" in A's speech is 10, and the frequency of occurrence of "fishing" and "politics" is 1.

In B's speech, assume that the frequency of occurrence of "camping" is 4, and the frequency of occurrence of "fishing" and "politics" is 1.

In C's speech, assume that the frequency of occurrence of "camping" is 1, and the frequency of occurrence of "fishing" and "politics" is 0.

Let us assume that the relationship feature between A and B captured by camera 4 (e.g., the frequency of eye contact, and so on) = 8, the relationship feature between A and C = 1, and the relationship feature between B and C = 0.

In the output matrix, if the current speaker is A, assume that the number of speaker turns to the next speaker, B, is 40, and the number of speaker turns to the next speaker, C, is 1.

If the current speaker is B, assume that the number of speaker turns to the next speaker, A, is 30, and the number of speaker turns to the next speaker, C, is 0.

If the current speaker is C, assume that the number of turns to the next speaker, A, is 30, and the number of turns to the next speaker, B, is 20.

In reality, if the speaker at time t1 is A, the next speaker will be B, and the amount of conversation that B will speak after A is predicted to be "40".

Next, after time t1, a matrix is output at time t2, and a matrix is output at time t3. The processing at time t2 is omitted.

Assume that from time t2 to t3, the frequency of occurrence of "camping," "fishing," and "politics" in A's speech is 0.

Assume that in B's speech, the frequency of occurrence of "camping", "fishing", and "politics" = 0, and in C's speech, the frequency of occurrence of "camping" and "fishing" = 0, and the frequency of occurrence of "politics" = 10.

Assume that the relationship feature between A and B obtained by the sensor is 0, the relationship feature between A and C is 0, and the relationship feature between B and C is 5.

In the output matrix, if the current speaker is A, assume that the number of speaker turns to the next speaker, B, is 10, and the number of speaker turns to the next speaker, C, is 20.

If the current speaker is B, assume that the number of turns to the next speaker, A, is 10, and the number of turns to the next speaker, C, is 20.

If the current speaker is C, assume that the number of speaker turns to the next speaker, A, is 0, and the number of speaker turns to the next speaker, B, is 20.

If the speaker at time t3 is actually C, the next speaker will be B, and the amount of conversation that B will have after C is predicted to be "20".

Second Embodiment
The second embodiment differs from the first embodiment in that the feature amount obtained from the SNS and the feature amount obtained during the conversation for each user are treated in parallel as separate feature amounts. The following mainly describes the differences from the first embodiment.

In this embodiment, additional learning is performed on the model that has already been trained from SNS data according to the processing of the first embodiment. A conversation experiment is performed in advance with multiple conversation participants to be predicted, and the model is additionally trained using the data obtained from the conversation experiment.

First, the learning will be described. The second derivation unit 30 derives a second feature vector for each of the multiple conversation participants based on information about communication acquired in the conversation experiment. The second feature vector can include features different from the first feature vector, and may have a different number of dimensions from the first feature vector. This improves the degree of freedom in designing the features.

For example, the second feature vector may include linguistic features, prosodic features, and features obtained from image data and sensor data. The linguistic features may have the same structure as in the first embodiment, for example, 200 dimensions.

The second derivation unit 30 derives prosodic features for each conversation participant based on the speech data. The prosodic features may be, for example, two-dimensional and may include features related to the pitch of the speech and features related to the power of the speech.

The features obtained from the image data and sensor data may be, for example, two-dimensional and may include features related to gaze and features related to posture.

The second derivation unit 30 derives a feature quantity related to gaze for each pair of two conversation participants based on the image data. For example, the second derivation unit 30 may derive a larger feature quantity related to gaze between the two conversation participants identified by the image data, the higher the frequency of gaze exchange between the two conversation participants.

The second derivation unit 30 derives posture-related features for each conversation participant based on the sensor data.

The learning unit 24 acquires the first feature vectors of each of the multiple conversation participants from the feature dictionary storage unit 16, and for each conversation participant, concatenates the acquired first feature vector with the derived second feature vector to derive a third feature vector. The third feature vector has, for example, 204 + 204 = 408 dimensions.

The learning unit 24 receives as input the third feature vector of the target conversation participant and the third feature vector of the other conversation participants, and learns a model using information representing the number of speaker turns from the target conversation participant to the other conversation participants as a correct answer label. The information representing the number of speaker turns may be the same as in the first embodiment, or may be derived from the results of a conversation experiment.

Next, prediction will be described. During a conversation between multiple conversation participants that is carried out after learning, the second derivation unit 30 derives a second feature vector for each of the multiple conversation participants based on the acquired information about the communication.

The prediction unit 32 acquires the first feature vector of each of the multiple conversation participants from the feature dictionary storage unit 16, and for each conversation participant, derives a third feature vector by concatenating the acquired first feature vector with the derived second feature vector. The prediction unit 32 inputs the derived third feature vector into the trained model and predicts the next speaker. Specifically, the prediction unit 32 inputs the third feature vector of the target conversation participant and the third feature vectors of the other conversation participants into the trained model, and outputs the number of speaker changes per unit time from the target conversation participant to the other conversation participants.

The same effects as in the first embodiment can be obtained in this embodiment as well.

Note that in actual application scenarios, the amount of data that can be secured may be limited, so input features may be created as follows:

For example, if there is a user for whom SNS data can be obtained but conversation experiment data cannot be obtained, the second feature vector of that user may be sampled from a uniform distribution or the like to create fictitious data. Alternatively, similar users for whom conversation data exists may be extracted based on the features of the SNS data, and the features of the similar users may be substituted. Similar users may be extracted by calculating the inner product between users using the first feature vector from the SNS data.

In addition, if there are users for whom SNS data cannot be obtained but conversation experiment data can be obtained, the process of obtaining the feature vector of the representative user of the cluster in the first embodiment can be applied. If there are users for whom neither SNS data nor conversation experiment data can be obtained, the above two processes can be used in combination.

The present invention has been described above based on the embodiments. The embodiments are merely examples, and it will be understood by those skilled in the art that various modifications are possible in the combination of each component and each processing process, and that such modifications are also within the scope of the present invention.

For example, the communication prediction device 10 may be configured not to perform learning. In this case, the communication prediction device 10 may not include the first acquisition unit 20, the first derivation unit 22, the learning unit 24, and the SNS data storage unit 14. The feature dictionary and the trained model are acquired from the outside. In this case, the machine learning device that performs only machine learning may be configured as a device separate from the communication prediction device 10. The machine learning device that performs learning of the first embodiment includes the first acquisition unit 20, the first derivation unit 22, the learning unit 24, the SNS data storage unit 14, the feature dictionary storage unit 16, and the trained model storage unit 18. The machine learning device that performs learning of the second embodiment includes the first acquisition unit 20, the first derivation unit 22, the learning unit 24, the second acquisition unit 26, the voice recognition unit 28, the second derivation unit 30, the SNS data storage unit 14, the feature dictionary storage unit 16, and the trained model storage unit 18.

1... communication prediction system, 10... communication prediction device, 12... processing unit, 14... SNS data storage unit, 16... feature dictionary storage unit, 18... trained model storage unit, 20... first acquisition unit, 22... first derivation unit, 24... learning unit, 26... second acquisition unit, 28... speech recognition unit, 30... second derivation unit, 32... prediction unit, 34... output unit.

Claims (8)

an acquisition unit that acquires information input by a plurality of users in a social networking service;
A learning unit that performs machine learning to learn a model for predicting the next speaker from among multiple conversation participants based on the acquired information;
a derivation unit that derives linguistic features of a plurality of users based on posted statements input by the plurality of users included in the acquired information, and derives relationship features that represent relationships between the target user and the other users and information that represents the number of times the target user takes turns to speak to the other users based on information regarding reactions to posted statements input by the target user included in the acquired information;
Equipped with
the learning unit receives as input linguistic features of a target user, linguistic features of other users, and relationship features between the target user and the other users, and learns a model using information indicating the number of speaker turns from the target user to the other users as a correct answer label;
A communication prediction device comprising: The derivation unit derives image feature amounts of the target user and the other users based on an image for identifying the target user and an image for identifying the other users, which are included in the acquired information;
The learning unit also receives image features of the target user and other users as input and learns a model.
The communication prediction device according to claim 1 . The lead-out portion is
Classify the linguistic features of multiple users into clusters,
Derive linguistic features of a virtual representative user representing each cluster,
When a user who has not entered any information into the social networking service is also to be the target of learning, the linguistic features of the representative user of one of the clusters are set as the linguistic features of the user.
The communication prediction device according to claim 1 . an acquisition unit that acquires information regarding communication between each conversation participant and other conversation participants during a conversation between the plurality of conversation participants;
a prediction unit that predicts a next speaker from among the plurality of conversation participants based on acquired communication information, using a trained model that has been machine-learned based on information input by a plurality of users in a social networking service;
a derivation unit that derives a second feature vector for each of the plurality of conversation participants based on the acquired communication information;
Equipped with
The prediction unit is
acquiring a first feature vector of each of the plurality of conversation participants from a storage unit that stores a first feature vector of each user derived from information input by the plurality of users in the social networking service;
predicting a next speaker based on the acquired plurality of first feature vectors and the derived plurality of second feature vectors using the trained model;
A communication prediction device comprising: The prediction unit is
deriving a third feature vector by performing a weighted addition of corresponding features in the acquired first feature vector and the derived second feature vector for each conversation participant;
The derived third feature vector is input into the trained model to predict the next speaker.
5. The communication prediction device according to claim 4 . The prediction unit is
deriving a third feature vector by concatenating the acquired first feature vector and the derived second feature vector for each conversation participant;
The derived third feature vector is input into the trained model to predict the next speaker.
5. The communication prediction device according to claim 4 . 1. A computer implemented method for communication prediction, comprising:
acquiring information entered by a plurality of users in a social networking service;
A step of machine learning a model for predicting a next speaker from among a plurality of conversation participants based on the acquired information;
deriving linguistic features of a plurality of users based on the posted statements input by the plurality of users included in the acquired information, and deriving relationship features representing relationships between the target user and the other users and information representing the number of times the target user takes turns to the other users based on information regarding reactions to the posted statements of the other users input by the target user included in the acquired information;
Equipped with
In the machine learning step, the linguistic features of the target user, the linguistic features of the other users, and the relationship features between the target user and the other users are input, and a model is trained using information indicating the number of speaker changes from the target user to the other users as a correct answer label.
A communication prediction method comprising: 1. A computer implemented method for communication prediction, comprising:
acquiring information regarding communication between each of the conversation participants and the other conversation participants during a conversation between the plurality of conversation participants;
predicting a next speaker from among the plurality of conversation participants based on acquired communication information using a trained model that has been machine-learned based on information input by a plurality of users in a social networking service;
deriving a second feature vector for each of the plurality of conversation participants based on the acquired communication information;
Equipped with
In the step of predicting,
acquiring a first feature vector of each of the plurality of conversation participants from a storage unit that stores a first feature vector of each user derived from information input by the plurality of users in the social networking service;
predicting a next speaker based on the acquired plurality of first feature vectors and the derived plurality of second feature vectors using the trained model;
A communication prediction method comprising:

Images