JP7716627B2 - Estimation method, estimation device, and estimation program - Google Patents

Description

The present invention relates to an estimation method, an estimation device, and an estimation program.

Research and development has been conducted on technology to automatically estimate mental states expressed in non-verbal and paralinguistic information such as human voice, facial expression, and gestures. For example, it is expected that in dialogue with agents or robots, the mental state of the person being spoken to can be reflected when generating responses, the results of estimation can be used as part of mental health care, and the state of participants in web conferences can be quantified to make it easier to understand.

The estimation of mental states expressed in such non-verbal and paralinguistic information is generally defined as supervised learning, which outputs the posterior probability of each label representing a defined mental state in response to input such as features extracted from speech or video images or the data itself (see non-patent document 1).

Here, emotions and facial expressions are classified into several classes, such as neutral, joy, sadness, surprise, fear, hatred, anger, and contempt. Specific levels, such as understanding, are also classified into arbitrary stages. In supervised learning, labels corresponding to these defined classes are annotated by one or more workers.

However, this annotation, or labeling, task becomes more difficult the finer the label granularity. For example, if the level of understanding is increased from three levels (no understanding, normal, and understood) to five levels (no understanding, slightly no understanding, normal, slightly understanding, and understood), the difficulty of labeling increases.

For such fine-grained classification problems, annotation results by different workers may agree globally but are unlikely to agree locally. For example, when multiple workers annotate a five-level comprehension scale, all workers may agree that the level of comprehension is low, but their assessments of "not understood" and "somewhat not understood" may differ slightly. In this case, if biases such as fatigue, experience, and judgment criteria affect the workers change, the results may also change. This means that supervised learning will result in the mixing of noisy and inaccurate correct labels, which has a negative impact on learning and evaluation.

Therefore, label correction techniques known as relabeling have been known in the past (see non-patent literature 2 and 3).

D. Rangulov and M. Fahim, “Emotion Recognition on large video dataset based on Convolutional Feature Extractor and Recurrent Neural Network”, 2020 IEEE 4th International Conference on Image Processing, Applications and Systems (IPAS), 2020 K. Wang, X. Peng, J. Yang, S. Lu, and Y. Qiao, “Suppressing Uncertainties for Large-Scale Facial Expression Recognition”, 2020 B. Zhang, L. Li, S. Wang, Z. Zha, and Q. Huang, “State-Relabeling Adversarial Active Learning”, 2020

However, with conventional technology, it has been difficult to accurately correct labels that represent mental states expressed in non-verbal and paralinguistic information. For example, conventional technology only retains the label assignment results of a single worker or the maximum value of multiple votes as a label, which makes it difficult to fully utilize human knowledge.

The present invention has been made in consideration of the above, and aims to accurately correct labels that represent mental states expressed in non-verbal and paralinguistic information.

In order to solve the above-mentioned problems and achieve the objectives, the estimation method of the present invention is an estimation method executed by an estimation device, and is characterized by including: an acquisition step of acquiring training data including non-verbal information or paralinguistic information and correct labels indicating mental states expressed in the non-verbal information or paralinguistic information assigned by multiple workers; a calculation step of calculating the posterior probability of the mental states for the acquired non-verbal information or paralinguistic information; and a learning step of learning model parameters of a model that estimates mental states expressed in input non-verbal information or paralinguistic information using the training data and the calculated posterior probability of the mental states.

According to the present invention, it is possible to accurately correct labels that represent mental states expressed in non-verbal and paralinguistic information.

FIG. 1 is a schematic diagram illustrating a schematic configuration of an estimation device. FIG. 2 is a diagram for explaining the processing of the estimation device. FIG. 3 is a diagram illustrating an example of the data structure of the learning data. FIG. 4 is a flowchart showing the estimation process procedure. FIG. 5 is a flowchart showing the procedure of the estimation process. FIG. 6 is a diagram illustrating a computer that executes the estimation program.

One embodiment of the present invention will be described in detail below with reference to the drawings. Note that the present invention is not limited to this embodiment. In addition, in the drawings, identical parts are designated by the same reference numerals.

[Configuration of the estimation device]
FIG. 1 is a schematic diagram illustrating the overall configuration of an estimation device. FIG. 2 is a diagram illustrating the processing of the estimation device. The estimation device 10 of this embodiment uses a neural network to estimate a level of understanding on a five-point scale as a mental state expressed in non-verbal and para-linguistic information, based on a video showing the upper body of a subject, which is non-verbal and para-linguistic information. The levels of understanding are defined as, for example, 1. not understanding, 2. somewhat not understanding, 3. normal state, 4. somewhat understanding, and 5. understanding, with higher numbers indicating greater understanding.

First, as illustrated in Figure 1, the estimation device 10 of this embodiment is realized by a general-purpose computer such as a personal computer, and includes an input unit 11, an output unit 12, a communication control unit 13, a memory unit 14, and a control unit 15.

The input unit 11 is implemented using input devices such as a keyboard and mouse, and inputs various instruction information, such as starting processing, to the control unit 15 in response to input operations by the implementer. The output unit 12 is implemented by a display device such as an LCD display, a printing device such as a printer, an information communication device, etc. The communication control unit 13 is implemented by a NIC (Network Interface Card), etc., and controls communication via a network between the control unit 15 and external devices such as servers and devices that manage learning data.

The memory unit 14 is realized by a semiconductor memory element such as RAM (Random Access Memory) or flash memory, or a storage device such as a hard disk or optical disk. The memory unit 14 may be configured to communicate with the control unit 15 via the communication control unit 13. In this embodiment, the memory unit 14 stores, for example, learning data 14a used in the estimation process described below, and model parameters 14b generated and updated in the estimation process.

Here, Figure 3 is a diagram illustrating the data structure of the training data. As shown in Figure 3, the training data 14a includes at least video data showing the upper body of the subject as non-verbal/paralinguistic information, a data ID that identifies each video data, and a correct answer label that represents the mental state, such as the level of understanding, that appears in each video data. In the example shown in Figure 3, a correct answer label is included as the level of understanding label.

The training data 14a may include labels representing personal attributes such as a personal ID that can identify an individual, age, and gender. Furthermore, the training data 14a may be divided into training, development, or evaluation sets, or the data may be expanded as needed.

In addition, pre-processing such as contrast normalization and face detection may be performed so that only certain areas of the video data are used. Furthermore, there are no particular restrictions on the codec of the input data (video data). Furthermore, the learning data 14a holds correct labels of understanding level updated by the processing of the update unit 15d, which will be described later.

Specifically, when estimating comprehension levels from video data using the estimation process described below, video data in H264 format recorded at 30 frames per second using a webcam may be resized to have 224 pixels on each side. Each of the X video data is assigned the personal IDs of S subjects, a correct comprehension level label assigned by name A, and a correct comprehension level label updated by the processing of the update unit 15d. In the example shown in Figure 3, the updated comprehension level label is included as the updated correct label.

Returning to the explanation of Figure 1, the control unit 15 is implemented using a CPU (Central Processing Unit), NP (Network Processor), FPGA (Field Programmable Gate Array), etc., and executes processing programs stored in memory. As a result, the control unit 15 functions as an acquisition unit 15a, a calculation unit 15b, a learning unit 15c, and an update unit 15d, as illustrated in Figure 1. Note that each of these functional units may be implemented in different hardware. For example, the acquisition unit 15a may be implemented in hardware different from the other functional units. The control unit 15 may also include other functional units.

The acquisition unit 15a acquires training data 14a including non-verbal or paralinguistic information and correct labels representing mental states, such as the level of understanding, assigned by multiple workers. Specifically, the acquisition unit 15a acquires training data 14a via the input unit 11 or from a device that generates training data, etc., via the communication control unit 13, including video data showing the upper body of the subject as non-verbal or paralinguistic information, a data ID that identifies each video data, and correct labels representing mental states, such as the level of understanding, that appear in each video data.

The acquisition unit 15a stores the learning data 14a acquired in advance prior to the following processing in the memory unit 14. Note that the acquisition unit 15a may transfer the acquired learning data 14a to the calculation unit 15b described below without storing it in the memory unit 14.

The calculation unit 15b calculates the posterior probability of a mental state for the acquired non-verbal information or paralinguistic information. For example, the calculation unit 15b uses a neural network to calculate the posterior probability of a mental state, such as the level of understanding, that appears in the video data of the training data 14a.

Note that the processing using neural networks described below is not limited to this embodiment, and elements of well-known technologies such as batch normalization, dropout, and L1/L2 regularization may be added at any point.

Specifically, the calculation unit 15b extracts frame-by-frame features from video data x1 _:T having a frame length T using a 2D convolutional neural network (CNN). Next, the calculation unit 15b calculates a time-direction embedded representation tensor _Hx using a recurrent neural network (RNN) having an output dimension of D, as shown in the following equation (1). Here, θ is a parameter set of the CNN, and φ is a parameter set of the RNN.

Next, the calculation unit 15b uses a multi-head self-attention mechanism to calculate the time to be focused on in the time direction, and calculates the weighted sum vector v in the time direction, as shown in the following equation (2).

In the above formula (2), the calculation unit 15b calculates the attention weight from the query Q _i and the key K _i , applies it to the value V _i , and finally calculates the sum in the time direction.

Here, d represents the number of attention heads, i represents each attention head, and W _i ^Q , W _i ^K , and W _i ^V represent the weights for the query, key, and value in each attention head, respectively.

Finally, the calculation unit 15b calculates the posterior probability p(C|x _1:T ) for each of the five levels of understanding using two fully connected layers, as shown in the following equation (3).

Here, _W1FC ^and _W2FC represent the weights of the two fully connected layers, ^DFC represents the number ^of output dimensions of the first fully connected layer, and C represents the number of predicted labels (C = 5 in this embodiment). The activation function of the first fully connected layer is the ReLU function.

The learning unit 15c uses the learning data 14a and the calculated posterior probability of the mental state to learn model parameters 14b of a model that estimates the mental state expressed in the input non-verbal information or paralinguistic information.

Specifically, the learning unit 15c updates the model parameter set Ω and obtains a trained model parameter set Ω'. The learning unit 15c can apply well-known loss functions and update methods. For example, the model parameter set Ω may include parameters previously trained in any other task, initial values may be generated using any random number, or some model parameters may not be updated.

For example, the learning unit 15c uses a stochastic gradient descent (SGD) to update the model parameter set Ω using the cross entropy L shown in the following equation (4) as a loss function. At this time, arbitrary values are used for hyperparameters such as the learning rate.

Here, m ^x is the correct distribution of the input video data x _1:T . The method of expressing the correct distribution is not particularly limited, and for example, it may be expressed as a one-hot vector using the understanding level label L _x illustrated in Figure 3. Alternatively, the correct distribution may be expressed by approximating a normal distribution centered on the correct class, or it may be expressed by using the annotation result as a soft-label as it is.

In addition, the learning unit 15c stores the acquired learned model parameter set Ω' in the memory unit 14 as model parameters 14b.

Returning to the explanation of Figure 1, the update unit 15d updates the correct label of the training data 14a using the learned model parameters 14b. Specifically, the update unit 15d updates the correct label of the training data 14a when the similarity between the posterior probability of the mental state calculated using the learned model parameters 14b and the correct label of the training data 14a is equal to or greater than a predetermined threshold.

For example, the update unit 15d updates the correct label L, which is obtained by normalizing the distribution of labels in the training data 14a assigned by multiple workers. First, the update unit 15d predicts the posterior probability of the understanding level for the training data 14a using the trained model parameter set Ω'. Then, the update unit 15d calculates the similarity between the correct label and the posterior probability, and updates the correct label if the calculated similarity is equal to or greater than a predetermined threshold.

When updating the correct label _Lx of the understanding level of input video data x using the trained model parameter Ω', the update unit 15d first calculates the posterior probability p(C|x1 _:T , Ω') for each understanding level. Next, the update unit 15d calculates the similarity between _Lx and p(C|x1 _:T , Ω'). The similarity calculated by the update unit 15d is not particularly limited, and may be calculated using an algorithm capable of calculating the distance or similarity between vectors, such as cross entropy, Kullback-Leibler divergence, cosine similarity, or Euclidean distance.

The update unit 15d calculates the cosine similarity c _x (-1≦c _x ≦1) as shown in the following equation (5), for example.

The update unit 15d may also determine whether to update Lx based on whether a specific condition is satisfied. For example, the update unit 15d may determine that _Lx should be updated as TRUE if max(p(C|x1 _:T , Ω′)) is equal to or greater than a predetermined threshold.

Alternatively, the update unit 15d may determine that the update judgment is TRUE if the two highest values of Lx are adjacent and the ratio is within the range of 4:6 to 6:4, and the correct answer changes within the two classes. For example, in the comprehension label of the data with data ID = 0000002 shown in Figure 3, the top two values (0.4, 0.6) are adjacent and the ratio is within the range of 4:6 to 6:4, so the update judgment is TRUE. On the other hand, in the comprehension label of the data with data ID = 0001459, the top two values (0.8, 0.2) are adjacent, but the ratio is not within the range of 4:6 to 6:4, so the update judgment is FALSE.

Next, the update unit 15d determines whether the calculated similarity is equal to or greater than a predetermined threshold, and if true, updates _Lx and assigns p(C|x1 _:T , Ω') to the updated understanding level label _Lx '. At this time, the update unit 15d may make the update determination based on a single condition, or may make the update determination based on a combination of multiple AND conditions or OR conditions.

Furthermore, the update unit 15d may substitute p(C|x1 _:T , Ω') for L _x ' after performing pre-processing such as re-normalizing a small value, such as less than 0.1, as 0. L _x ' is used in the calculation of the loss function instead of L _x as a correct label when subsequently learning the model parameters 14b.

The processing by the update unit 15d can be initiated at any timing during the learning of the model parameters 14b by the learning unit 15c. For example, the update unit 15d may be configured to initiate the processing when the number of updates due to learning of the model parameters 14b in the learning unit 15c reaches a predetermined threshold or greater. Alternatively, the update unit 15d may be configured to be more complex, for example, to initiate the first processing after the model parameters 14b in the learning unit 15c have been updated 1000 times, and to initiate the second and subsequent processing after the model parameters 14b in the learning unit 15c have been updated 100 times.

[Estimation process]
Next, the estimation process performed by the estimation device 10 will be described. Figures 4 and 5 are flowcharts showing the estimation process procedure. The estimation process of this embodiment includes a learning process and an update process. First, Figure 4 shows the learning process procedure. The flowchart in Figure 4 starts, for example, when an input is made to instruct the start of the learning process.

First, the acquisition unit 15a acquires training data 14a including non-verbal or paralinguistic information and correct labels indicating mental states expressed in the non-verbal or paralinguistic information assigned by multiple workers (step S1). The acquisition unit 15a stores the acquired training data 14a in the memory unit 14. Alternatively, the acquisition unit 15a may transfer the acquired training data 14a to the calculation unit 15b without storing it in the memory unit 14.

In addition, the calculation unit 15b calculates the posterior probability of the mental state for the acquired non-verbal information or paralinguistic information (step S2).

Next, the learning unit 15c uses the learning data 14a and the calculated posterior probability of the mental state to learn model parameters 14b of a model that estimates the mental state expressed in the input non-verbal information or paralinguistic information (step S3). This completes the learning process.

Next, Figure 5 shows the update processing procedure. The flowchart in Figure 5 starts, for example, when an input is received instructing the start of the update processing.

First, the update unit 15d uses the learned model parameters 14b to calculate the posterior probability of understanding for the learning data 14a (step S11).

Next, if the similarity between the calculated posterior probability of the mental state and the correct label of the training data 14a is equal to or greater than a predetermined threshold, the update unit 15d updates the correct label of the training data 14a (step S12). This completes the update process.

[effect]
As described above, in the estimation device 10 of this embodiment, the acquisition unit 15a acquires training data 14a including non-verbal information or paralinguistic information and correct labels assigned by multiple workers that represent mental states expressed in the non-verbal information or paralinguistic information. The calculation unit 15b calculates posterior probabilities of mental states for the acquired non-verbal information or paralinguistic information. The learning unit 15c uses the training data 14a and the calculated posterior probabilities of mental states to learn model parameters 14b of a model that estimates mental states expressed in input non-verbal information or paralinguistic information.

This enables the estimation device 10 to accurately estimate mental states expressed in non-verbal or paralinguistic information by learning global label assignment using correct labels assigned by multiple people. Therefore, the estimation device 10 can accurately assign labels representing mental states using the estimation results. In this way, the estimation device 10 makes it possible to accurately correct labels representing mental states expressed in non-verbal and paralinguistic information.

The update unit 15d also updates the correct label of the training data 14a using the learned model parameters 14b. Specifically, the update unit 15d updates the correct label of the training data 14a when the similarity between the posterior probability of the mental state calculated using the learned model parameters 14b and the correct label of the training data 14a is equal to or greater than a predetermined threshold.

This allows the estimation device 10 to correct the correct label only if it is somewhat similar to the distribution of correct label assignment results from multiple people. This eliminates the possibility of mistakenly assigning a label to a class with a distant meaning, making it possible to locally correct labels without changing the overall label assignment. Furthermore, by referencing the tendency of human label assignment, certain constraints are imposed, making it possible to stably correct labels with a small amount of data. In this way, the estimation device 10 makes it possible to accurately correct labels even for data that is difficult to label.

When the number of updates due to learning of model parameters 14b in learning unit 15c reaches a predetermined threshold or more, processing by update unit 15d is initiated. This enables label correction with even higher accuracy.

[program]
A program describing the processing performed by the estimation device 10 according to the above embodiment in a computer-executable language can also be created. In one embodiment, the estimation device 10 can be implemented by installing an estimation program that executes the above estimation processing as package software or online software on a desired computer. For example, by executing the estimation program on an information processing device, the information processing device can function as the estimation device 10. Other examples of information processing devices include mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone Systems), as well as slate terminals such as PDAs (Personal Digital Assistants). The functions of the estimation device 10 may also be implemented on a cloud server.

Figure 6 shows an example of a computer that executes an estimation program. The computer 1000 has, for example, a memory 1010, a CPU 1020, a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1031. The disk drive interface 1040 is connected to a disk drive 1041. A removable storage medium such as a magnetic disk or optical disk is inserted into the disk drive 1041. The serial port interface 1050 is connected to, for example, a mouse 1051 and a keyboard 1052. The video adapter 1060 is connected to, for example, a display 1061.

Here, the hard disk drive 1031 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. Each piece of information described in the above embodiment is stored, for example, in the hard disk drive 1031 or memory 1010.

The estimation program is stored on the hard disk drive 1031, for example, as a program module 1093 that describes instructions to be executed by the computer 1000. Specifically, the program module 1093 that describes each process executed by the estimation device 10 described in the above embodiment is stored on the hard disk drive 1031.

In addition, data used for information processing by the estimation program is stored, for example, as program data 1094 on the hard disk drive 1031. The CPU 1020 then reads the program module 1093 and program data 1094 stored on the hard disk drive 1031 into the RAM 1012 as needed, and executes each of the procedures described above.

The program module 1093 and program data 1094 related to the estimation program are not limited to being stored on the hard disk drive 1031, but may be stored, for example, on a removable storage medium and read by the CPU 1020 via the disk drive 1041, etc. Alternatively, the program module 1093 and program data 1094 related to the estimation program may be stored in another computer connected via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network), and read by the CPU 1020 via the network interface 1070.

The above describes an embodiment of the invention made by the inventor, but the present invention is not limited to the descriptions and drawings that form part of the disclosure of the present invention according to this embodiment. In other words, all other embodiments, examples, operational techniques, etc. made by those skilled in the art based on this embodiment are included in the scope of the present invention.

REFERENCE SIGNS LIST 10 Estimation device 11 Input unit 12 Output unit 13 Communication control unit 14 Storage unit 14a Learning data 14b Model parameters 15 Control unit 15a Acquisition unit 15b Calculation unit 15c Learning unit 15d Update unit

Claims (5)

An estimation method executed by an estimation device,
an acquisition step of acquiring training data including non-verbal information or para-verbal information and correct labels indicating mental states expressed in the non-verbal information or para-verbal information assigned by a plurality of workers;
a calculation step of calculating a posterior probability of the state of mind for the acquired non-linguistic information or paralinguistic information;
a learning step of learning model parameters of a model that estimates the mental state expressed in input non-linguistic information or paralinguistic information using the learning data and the calculated posterior probability of the mental state;
an updating step of updating the correct labels of the training data using the trained model parameters;
An estimation method comprising: 2. The estimation method according to claim 1, wherein the updating step updates the correct label of the training data when a similarity between the posterior probability of the mental state calculated using the trained model parameters and the correct label of the training data is equal to or greater than a predetermined threshold. 2. The estimation method according to claim 1 , wherein the updating step is initiated when the number of updates due to learning of the model parameters in the learning step reaches a predetermined threshold value or more. an acquisition unit that acquires learning data including non-verbal information or para-verbal information and correct labels that represent mental states expressed in the non-verbal information or para-verbal information assigned by a plurality of workers;
a calculation unit that calculates a posterior probability of the state of mind for the acquired non-linguistic information or paralinguistic information;
a learning unit that uses the learning data and the calculated posterior probability of the mental state to learn model parameters of a model that estimates the mental state expressed in input non-linguistic information or paralinguistic information;
an update unit that updates the correct labels of the training data using the trained model parameters;
An estimation device comprising: an acquisition step of acquiring learning data including non-verbal information or para-verbal information and correct labels indicating mental states expressed in the non-verbal information or para-verbal information assigned by a plurality of workers;
a calculation step of calculating a posterior probability of the state of mind for the acquired non-linguistic information or paralinguistic information;
a learning step of learning model parameters of a model that estimates the mental state expressed in input non-linguistic information or paralinguistic information using the learning data and the calculated posterior probability of the mental state;
an updating step of updating the correct labels of the training data using the trained model parameters;
An estimation program for causing a computer to execute the above.