JP7766297B2 - Behavior recognition device, behavior recognition method, and behavior recognition program - Google Patents

Description

Embodiments of the present invention relate to an action recognition device, an action recognition method, and an action recognition program.

Currently, a variety of industries are being called upon to digitize their operations due to a labor shortage. In order to digitize operations, it is necessary to understand how the operations being digitized are being carried out. However, accurately understanding the operations can be difficult. For example, even if an operations manual has been prepared to understand the operations, there may be a discrepancy between the manual and the actual operations, making it difficult to accurately understand the operations.

There are methods for visualizing human behavior using cameras, etc. For example, Non-Patent Document 1 discloses a technology that uses behavior recognition technology to visualize a person's actual behavior in order to automatically acquire human behavior from a camera.

Furthermore, Non-Patent Document 2 discloses technology that can learn a class of behavior from multiple video images.

Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He, “SlowFast Networks for Video Recognition”, ICCV2019. Jiahao Wang, Yunhong Wang, Sheng Liu, Annan Li, “Few-shot Fine-Grained Action Recognition via Bidirectional Attention and Contrastive Meta-Learning”, ACMMM2021. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding” NAACL, 2019. Kenichiro Ishii, Shuko Ueda, Eisaku Maeda, Hiroshi Murase, "Easy-to-understand Pattern Recognition", Ohmsha.

Non-Patent Document 1 has the problem that it is costly to define the actions to be recognized (define action labels).

Furthermore, the technology disclosed in Non-Patent Document 2 has the problem that it is difficult to classify classes with similar behavioral features.

The object of this invention was made in light of the above circumstances, and its purpose is to provide technology that can improve the stability and accuracy of learning by generating features that take into account behavioral and linguistic features and that can classify classes with similar behavioral features, and by applying criteria that use the generated features to a method for selecting support data for episodes.

The behavior recognition device according to the embodiment includes a data acquisition unit that acquires input data including a first video segment; a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model that includes a pre-trained behavior recognition model and an encoder; a behavior label generation unit that generates a behavior label by inputting the latent feature vector into a pre-trained behavior classification model; and an output control unit that outputs the behavior label.

According to one aspect of the present invention, it is possible to provide a behavior recognition device, a behavior recognition method, and a behavior recognition program that can improve the stability of learning and increase accuracy by generating features that take into account behavioral and linguistic features and that can classify classes with similar behavioral features, and applying criteria using the generated features to a method for selecting support data in an episode.

FIG. 1 is a block diagram illustrating an example of a hardware configuration of a behavior recognition apparatus according to an embodiment. FIG. 2 is a block diagram showing the software configuration of the behavior recognition apparatus according to the embodiment in relation to the hardware configuration shown in FIG. FIG. 3 is a flowchart showing an example of a learning procedure of the behavior recognition device according to this embodiment. FIG. 4 is a diagram showing an example of a conceptual diagram of a neural network model of the latent feature generation model building unit. FIG. 5 is a flowchart showing an example of detailed processing for generating an episode set in step ST103. FIG. 6 is a flowchart showing an example of detailed processing for updating the behavior classification model in step ST104. FIG. 7 is a flowchart showing an example of an activity label estimation procedure of the activity recognition device according to this embodiment. FIG. 8 is a diagram showing an example of a conceptual diagram of a neural network model of the behavior classification unit.

The behavior recognition device, behavior recognition method, and behavior recognition program will be described in detail below with reference to the drawings. Note that in the following embodiments, parts with the same numbers perform similar operations, and redundant explanations will be omitted. For example, when there are multiple identical or similar elements, a common symbol may be used to describe each element without distinguishing between them, or a sub-number may be used in addition to the common symbol to describe each element with distinction between them.

[Embodiment]
First, in the following description, an "action" is an action that represents a human movement such as walking, running, or carrying, and an "action label" is a verb that represents an action such as walking or running.

Non-Patent Document 2 uses a learning method known as "few-shot learning." Few-shot learning constructs training data by generating a large amount of task data called episodes. Specifically, suppose there is training data with a large number of classes, from which five classes are extracted. If each class holds three samples as training data, the "support data" becomes a 5-way 3-shot (5 classes, 3 samples) classification task. In this case, if the "query data" holds one sample of test data for each of two of the five classes, the task of few-shot learning is to learn from the support data and accurately classify each sample from those two classes. Here, this combination of support data and query data is called an "episode."

However, the few-shot learning used in Non-Patent Document 2 generates a large number of episodes and further trains a model that has already been trained from a large amount of training data. However, the episode construction method is random sampling, and if the class selection method is incorrect initially, the neural network will fall into a local solution, affecting generalization performance and reducing accuracy. In particular, there is a problem in that accuracy is affected when classes with low separability, which is the ratio of inter-class variance to intra-class variance, are mixed. Specifically, the technology disclosed in Non-Patent Document 2 has difficulty classifying classes with similar behavioral features.

In the following explanation, as in Non-Patent Document 2, "episode" refers to the combination of the support data and query data described above.

Furthermore, in the following description, a "video segment" is a unit consisting of a predetermined number of video frames. For example, 30 frames may be considered as one segment. For example, when considering a long-term video of hundreds or thousands of frames, a collection of frames sampled at regular intervals may be considered as a video segment.

(composition)
FIG. 1 is a block diagram showing an example of a hardware configuration of a behavior recognition device 1 according to an embodiment.
The behavior recognition device 1 is a computer that analyzes input data, generates output data, and outputs the output data. The behavior recognition device 1 may be installed in any location.

As shown in FIG. 1, the behavior recognition device 1 includes a control unit 10, a program storage unit 20, a data storage unit 30, a communication interface 40, and an input/output interface 50. The control unit 10, the program storage unit 20, the data storage unit 30, the communication interface 40, and the input/output interface 50 are communicatively connected to one another via a bus. Furthermore, the communication interface 40 may be communicatively connected to an external device via a network. Furthermore, the input/output interface 50 is communicatively connected to an input device 51 and an output device 52.

The control unit 10 controls the behavior recognition device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 10 may be an integrated circuit capable of executing various programs.

The program storage unit 20 can use a combination of non-volatile memory that can be written to and read from at any time, such as an EPROM (Erasable Programmable Read Only Memory), HDD (Hard Disk Drive), or SSD (Solid State Drive), and non-volatile memory such as a ROM (Read Only Memory) as a storage medium. The program storage unit 20 stores the programs necessary to execute various processes. In other words, the control unit 10 can achieve various controls and operations by reading and executing the programs stored in the program storage unit 20.

The data storage unit 30 is a storage medium that combines non-volatile memory, such as a HDD or memory card, which can be written to and read from at any time, with volatile memory, such as RAM (Random Access Memory). The data storage unit 30 is used to store data acquired and generated in the process of the control unit 10 executing programs and performing various processes.

The communication interface 40 includes one or more wired or wireless communication modules. For example, the communication interface 40 includes a communication module that establishes a wired or wireless connection with an external device via a network. The communication interface 40 may also include a wireless communication module that establishes a wireless connection with an external device such as a Wi-Fi access point or base station. Furthermore, the communication interface 40 may also include a wireless communication module that establishes a wireless connection with an external device using short-range wireless technology. In other words, the communication interface 40 may be any general communication interface that can communicate with an external device under the control of the control unit 10 and send and receive various information, including past performance data.

The input/output interface 50 is connected to the input device 51, output device 52, etc. The input/output interface 50 is an interface that enables the transmission and reception of information between the input device 51 and the output device 52. The input/output interface 50 may be integrated with the communication interface 40. For example, the behavior recognition device 1 and at least one of the input device 51 or the output device 52 may be wirelessly connected using short-range wireless technology, etc., and information may be transmitted and received using this short-range wireless technology.

The input device 51 includes, for example, a keyboard or pointing device that allows the user to input various data to the behavior recognition device 1. The input device 51 may also include a reader for reading data to be stored in the program storage unit 20 or data storage unit 30 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

The output device 52 includes a display that displays images from the behavior recognition device 1 to be displayed to the administrator of the behavior recognition device 1.

FIG. 2 is a block diagram showing the software configuration of the behavior recognition device 1 according to the embodiment in association with the hardware configuration shown in FIG.
The control unit 10 includes a data acquisition unit 101, a latent feature generation model construction unit 102, a training data generation unit 103, a model update unit 104, a behavior classification unit 105, and an output control unit 106.

The data acquisition unit 101 acquires training data during learning. For example, the training data may be a collection of video segments, action labels, and object labels. Here, action labels are verbs that express actions such as walking and running, as described above. Object labels may be general labels used in object detection, etc. Furthermore, the data acquisition unit 101 acquires input data during behavior inference. The input data may include video segments.

The latent feature generation model construction unit 102 constructs a latent feature generation model. The latent feature generation model construction unit 102 constructs a latent feature generation model that takes video segments included in the received training data as input and outputs latent feature vectors. The latent feature generation model construction unit 102 may output parameters of the constructed latent model to the training data generation unit 103. Alternatively, the latent feature generation model construction unit 102 may output information for using the latent feature generation model in the training data generation unit 103 to the training data generation unit 103.

The training data generation unit 103 includes a set extraction unit 1031, a latent feature vector generation unit 1032, a degree of separation calculation unit 1033, an episode set addition unit 1034, a determination unit 1035, and a memory control unit 1036.

The training data generation unit 103 generates a set of episodes. The training data generation unit 103 receives as input video segments, parameters of a latent feature generation model, and a set of action labels, and outputs a set of episodes to be used for training.

The set extraction unit 1031 randomly extracts a set of N-way K-shot support data and query data from a collection of video segments and action labels.

The latent feature vector generation unit 1032 generates a latent feature vector. The latent feature vector generation unit 1032 constructs a latent feature generation model from the parameters received from the latent feature generation model construction unit 102, and generates a latent feature vector for each video segment of the support data using the constructed latent feature generation model.

The separability calculation unit 1033 calculates the separability using the latent feature vector. The specific method for calculating the separability will be described later.

The episode set addition unit 1034 adds the degree of separation and the episode (combination of support data and query data) as an episode set.

The determination unit 1035 determines whether the number of episode sets is greater than a predetermined number.

The storage control unit 1036 stores the episode sets in the learning data storage unit 301. The storage control unit 1036 stores episode sets that exceed a predetermined number in the learning data storage unit 301.

The model update unit 104 includes a batch extraction unit 1041, a loss calculation unit 1042, a model parameter update unit 1043, a separability threshold update unit 1044, a convergence determination unit 1045, and a memory control unit 1046.

The model update unit 104 updates the behavior classification model. During learning, the model update unit 104 acquires parameters of the behavior classification model used by the behavior classification unit 105, and acquires data necessary for learning the behavior classification model. Here, the behavior classification model may be a general model as long as it can classify behavior labels. Then, the model update unit 104 updates the model parameters based on the loss calculated from the acquired data, and stores the updated parameters in the model storage unit 302.

The batch extraction unit 1041 acquires a set of episodes from the training data storage unit 301. From the acquired set of episodes, the batch extraction unit 1041 extracts a batch of episodes whose degree of separation is equal to or greater than a threshold. Here, a batch is a training unit for the neural network and consists of multiple episodes.

The loss calculation unit 1042 estimates the behavior label of the batch and calculates the loss. Details of the method for estimating the behavior label of the batch and the method for calculating the loss will be described later.

The model parameter update unit 1043 updates the model parameters by backpropagating the loss.

The separability threshold update unit 1044 updates the separability threshold. Details of how to update the separability threshold will be described later.

The convergence determination unit 1045 determines whether the loss value is less than a threshold value.

The memory control unit 1046 stores the updated parameters in the model memory unit 302. The memory control unit 1046 may store the updated parameters in the model memory unit 302 when the loss value becomes less than the threshold value.

The behavior classification unit 105 includes a latent feature vector generation unit 1051 and a behavior label generation unit 1052.

The latent feature vector generation unit 1051 generates a latent feature vector using a latent feature generation model. The latent feature generation model may be a model constructed by the latent feature generation model construction unit 102.

The behavior label generation unit 1052 generates behavior labels. The behavior label generation unit 1052 obtains the parameters of the behavior classification model stored in the model storage unit 302 and generates behavior labels by inputting a latent feature vector into the behavior classification model. The behavior label generation unit 1052 outputs the generated behavior labels to the output control unit 106.

The output control unit 106 outputs the behavior label. The output control unit 106 outputs the behavior label to the output device 52 via the input/output interface 50. The output control unit 106 may then control the display of the output device 52 to display the behavior label.

(Learning behavior)
First, the learning operation will be described. In this learning operation, it is assumed that the parameters of the behavior categorization model used by the behavior categorization unit 105 are stored in the model storage unit 302.

FIG. 3 is a flowchart showing an example of a learning procedure of the behavior recognition device 1 in this embodiment.
The operation of this flowchart is realized by the control unit 10 of the behavior recognition device 1 reading and executing a program stored in the program storage unit 20. This operation starts when a user of the behavior recognition device 1 inputs training data into the input device 51. Here, the training data may be a collection of video segments, behavior labels, and object labels.

In step ST101, the data acquisition unit 101 acquires training data. The input device 51 receives the training data and outputs the training data to the data acquisition unit 101. The data acquisition unit 101 receives the training data and outputs the training data to the latent feature generation model construction unit 102.

In step ST102, the latent feature generation model construction unit 102 constructs a latent feature generation model. The latent feature generation model construction unit 102 constructs a latent feature generation model that takes as input video segments included in the received training data and outputs latent feature vectors. The latent feature generation model construction unit 102 may output parameters of the constructed latent model to the training data generation unit 103. Alternatively, the latent feature generation model construction unit 102 may output information for using the latent feature generation model in the training data generation unit 103 to the training data generation unit 103.

FIG. 4 is a diagram showing an example of a conceptual diagram of a neural network model of the latent feature generation model construction unit 102.
The neural network model may be any neural network model that can define input and output and is effective for the output target, such as a convolutional neural network or a neural network model using a Transformer. For example, in the example of Figure 2, the latent feature generation model includes a pre-trained action recognition model 1021, an encoder 1022, a decoder 1023, and a language model 1024.

The pre-trained behavior recognition model 1021 may be any behavior recognition model, such as the SlowFast model disclosed in Non-Patent Document 1.

The encoder 1022 and decoder 1023 may be general encoders 1022 and decoders 1023, and detailed explanations will be omitted here.

The language model 1024 may be a pre-trained language model 1024 such as BERT as disclosed in Non-Patent Document 3.

Furthermore, the action labels and object labels shown in FIG. 4 may be included in the training data and are assumed to be annotated to video segments in advance. If object labels are not annotated, the latent feature generation model construction unit 102 may extract object labels using a technique such as object detection.

Referring to Figure 4, when a video segment is input, the pre-trained action recognition model 1021 outputs an action feature vector from the fully connected layer of the model. The output action feature vector is output to the encoder 1022.

The encoder 1022 generates a latent feature vector by encoding the behavioral feature vector. The generated latent feature vector may be, for example, a 256-dimensional vector. The generated latent feature vector is output to the decoder 1023.

The decoder 1023 reconstructs an action feature vector by decoding the latent feature vector, and also reconstructs a language feature vector obtained from the language model 1024. Here, the language feature vector is a language feature vector obtained from a combination of an action label and an interacting object label, etc.

When training the encoder 1022 and decoder 1023 to reconstruct an action feature vector and a language feature vector from a latent feature vector, the latent feature generation model construction unit 102 calculates the L2 loss (first loss), which is the mean squared error between the action feature vector obtained from the pre-trained action recognition model 1021 and the action feature vector generated by the decoder 1023 from the latent feature vector, and calculates the L2 loss (second loss) between the language feature vector output by the language model 1024 in response to input action labels and object labels, and the language feature vector generated by the decoder 1023 from the latent feature vector. The latent feature generation model construction unit 102 then trains the encoder 1022 and decoder 1023 to minimize these L2 losses, thereby enabling the desired latent feature vector to be encoded. The loss may be, for example, the L1 loss, which is the mean absolute error.

By reconstructing the action feature vector in this way and generating a language feature vector at the same time, we can obtain a latent feature vector that takes into account the differences in the language feature vectors. As a result, we can obtain latent feature vectors with high discriminatory ability for actions that are similar but have different meanings. For example, typical action recognition models are not good at distinguishing between the actions of "push" and "pull." However, the language feature vectors for "push" and "pull" are different. Therefore, even if the action feature vectors are similar, the language feature vectors are different, so different latent feature vectors can be obtained for "push" and "pull."

Here, a trained latent feature generation model is sufficient to obtain a latent feature vector. Therefore, once the encoder 1022 has been trained once to obtain a latent feature vector, the decoder 1023 is not used to generate the latent feature vector unless new training is performed. That is, in the example of Figure 2, the latent feature generation model refers to the pre-trained action recognition model 1021 and encoder 1022.

In other words, the latent feature vector is a vector obtained by using the pre-trained behavior recognition model 1021 as input to an action feature vector that can be extracted from a video segment, and training the latent feature generation model (encoder 1022) to reconstruct a language feature vector extracted from the extracted action feature vector, the action label corresponding to the video segment, and the label of the interacting object.

Furthermore, when reconstructing the action feature vector and language feature vector, commonly used losses such as the L1 norm and L2 norm of the original and reconstructed action feature vectors and the L1 norm and L2 norm of the original and reconstructed language feature vectors can be used as the reconstruction loss, making it possible to reconstruct the action feature vector and language feature vector. When actually performing reconstruction, the sum of the reconstruction loss of the action feature vector and the reconstruction loss of the language feature vector can be used as the overall loss. For example, reconstruction can be performed using a weighted sum of the reconstruction loss of the action feature vector and the reconstruction loss of the language feature vector.

Returning to FIG. 3, in step ST103, the training data generation unit 103 generates a set of episodes. The training data generation unit 103 receives as input video segments, parameters of a latent feature generation model, and a set of action labels, and outputs a set of episodes to be used for training.

FIG. 5 is a flowchart showing an example of detailed processing for generating an episode set in step ST103.
In step ST201, the set extraction unit 1031 randomly extracts sets of N-way K-shot support data and query data from a set of video segments and action labels. The set extraction unit 1031 of the training data generation unit 103 extracts sets of N-way K-shot support data and query data from the set of video segments and action labels among the input data. That is, the set extraction unit 1031 extracts sets of support data for N classes and K samples and query data for evaluation of the same classes as the extracted N classes. The number of combinations of support data and query data and the number of query data samples to prepare for each of the N classes may be determined experimentally.

In step ST202, the latent feature vector generation unit 1032 generates a latent feature vector. The latent feature vector generation unit 1032 constructs a latent feature generation model from the parameters received from the latent feature generation model construction unit 102, and generates a latent feature vector for each video segment of the support data using the constructed latent feature generation model. Alternatively, the latent feature vector generation unit 1032 generates a latent feature vector for each video segment of the support data using the latent feature generation model of the latent feature generation model construction unit 102.

In step ST203, the separability calculation unit 1033 calculates the separability using the latent feature vector. The separability calculation unit 1033 calculates the intra-class variance and inter-class variance of the support data based on the latent feature vector. The separability calculation unit 1033 then calculates the separability, which is the ratio of the inter-class variance to the intra-class variance.

Here, let σ _w ² be the intra-class variance, σ _b ² be the inter-class variance, and let X _i be the set of samples belonging to class c _i, where i is a positive integer equal to or greater than 1. Let n _i be the number of samples belonging to class c _i , and m _i be the mean vector of the samples. If N _i is the total number of samples and m is the mean vector of all samples, the intra-class variance and inter-class variance can be expressed as follows:

From the above formula, the degree of separation is expressed as σ _b ² /σ _w ² .

In step ST204, the episode set addition unit 1034 adds the degree of separation and the episode (combination of support data and query data) as an episode set.

In step ST205, the determination unit 1035 determines whether the number of episode sets is greater than a predetermined number. If the number of episode sets is equal to or less than the predetermined number, the process returns to step ST201. On the other hand, if the number of episode sets is greater than the predetermined number, the process proceeds to step ST206.

In step ST206, the storage control unit 1036 stores the episode set in the learning data storage unit 301.

Returning to FIG. 3 , in step ST104, the model update unit 104 updates the behavior classification model. During learning, the model update unit 104 acquires parameters of the behavior classification model used by the behavior classification unit 105, and acquires data necessary for learning the behavior classification model. The model update unit 104 then updates the model parameters based on the loss calculated from the acquired data, and stores the updated parameters in the model storage unit 302.

FIG. 6 is a flowchart showing an example of detailed processing for updating the behavior classification model in step ST104.
In step ST301, the batch extraction unit 1041 acquires an episode set from the training data storage unit 301.

In step ST302, the batch extraction unit 1041 extracts a batch of episodes from the acquired set of episodes whose degree of separation is equal to or greater than a threshold. Here, a batch is a learning unit for the neural network and consists of multiple episodes.

In step ST303, the loss calculation unit 1042 estimates the behavior label of the batch and calculates the loss. The loss calculation unit 1042 estimates the behavior label of the batch. Specifically, the loss calculation unit 1042 inputs support data in multiple episodes that make up the batch into a behavior classification model and obtains the behavior label from the results estimated by the behavior classification model. The loss calculation unit 1042 then calculates the loss based on the estimated behavior label and pre-defined behavior labels included in the training data. Here, the loss may be, for example, cross-entropy loss, which calculates the classification loss between the estimated behavior label and the pre-defined behavior label.

In step ST304, the model parameter update unit 1043 updates the model parameters by backpropagating the loss. The model parameter update unit 1043 obtains the parameters used in the behavior classification model stored in the model storage unit 302. Then, the model parameter update unit 1043 updates the parameters by backpropagating the loss using a common method such as gradient descent.

In step ST305, the separability threshold update unit 1044 updates the separability threshold. For example, the model update unit 104 extracts a batch and uses it for learning if the separability (the ratio of inter-class variance to intra-class variance) exceeds the threshold up to a certain number of episodes. On the other hand, if the separability does not exceed the threshold, the batch is not extracted and therefore not used for learning. This prevents the initial learning data from being forgotten. Here, forgetting the initial learning data means that the model parameters are updated as the data learned in the initial learning stage learns different data, and the model adapts to the different data, resulting in a loss of separation performance for the initial learning data. Furthermore, the separability threshold update unit 1044 updates the separability threshold to a smaller threshold for episodes that exceed a certain number. In this way, by using episodes with low separability as targets for random sampling in learning, it is possible to prevent initial forgetting and prevent a decrease in sampling patterns.

In step ST306, the convergence determination unit 1045 determines whether the loss value is less than the threshold value. If the loss value is less than the threshold value, i.e., if it is determined that sufficient learning has been achieved, the convergence determination unit 1045 considers that the loss value has converged. The process then proceeds to step ST307. On the other hand, if the loss value is greater than or equal to the threshold value, the convergence determination unit 1045 determines that the loss value has not converged. The process then returns to step ST302.

In step ST307, the memory control unit 1046 stores the updated parameters in the model memory unit 302.

In this way, the behavior recognition device 1 can update the parameters of the behavior classification model through learning.

(Inference operation)
Next, the processing performed by the control unit 10 during behavior inference will be described.
FIG. 7 is a flowchart showing an example of an activity label estimation procedure of the activity recognition device 1 according to this embodiment.

The operation of this flowchart is realized by the control unit 10 of the behavior recognition device 1 reading and executing a program stored in the program storage unit 20. This operation begins when the user of the behavior recognition device 1 inputs input data into the input device 51. Here, the input data may include video segments.

In step ST401, the data acquisition unit 101 acquires input data. The input device 51 receives the input data and outputs the input data to the data acquisition unit 101. The data acquisition unit 101 receives the input data and outputs the input data to the latent feature generation model construction unit 102, the behavior recognition unit. The data acquisition unit 101 may also output a signal indicating that the input data has been received to the latent feature generation model construction unit 102. In response to receiving the signal, the latent feature generation model construction unit 102 outputs parameters to be used in the latent feature generation model or information for using the latent feature generation model to the behavior classification unit 105.

In step ST402, the latent feature vector generation unit 1051 generates a latent feature vector using a latent feature generation model. Upon receiving input data from the data acquisition unit 101 and parameters to be used in the latent feature generation model from the latent feature generation model construction unit 102, the latent feature vector generation unit 1051 constructs a latent feature generation model and inputs video segments included in the input data into the latent feature generation model to generate a latent feature vector. Alternatively, the latent feature vector generation unit 1051 generates a latent feature vector using the latent feature generation model of the latent feature generation model construction unit 102.

In step ST403, the behavior label generation unit 1052 generates a behavior label. The behavior label generation unit 1052 acquires the parameters of the behavior classification model stored in the model storage unit 302 and generates a behavior label by inputting a latent feature vector into the behavior classification model. The behavior label generation unit 1052 outputs the generated behavior label to the output control unit 106.

FIG. 8 is a diagram showing an example of a conceptual diagram of the neural network model of the behavior classification unit 105.
8 includes a pre-trained behavior recognition model 1021, an encoder 1022, and a behavior classification model 1053. The pre-trained behavior recognition model 1021 and the encoder 1022 configure a latent feature generation model. That is, the pre-trained behavior recognition model 1021 and the encoder 1022 are used to generate the latent feature vector described in step ST401. Then, the behavior classification model 1053 is used to generate the behavior label described in step ST402.

In step ST404, the output control unit 106 outputs the behavior label. The output control unit 106 outputs the behavior label to the output device 52 via the input/output interface 50. The output control unit 106 may then control the display of the output device 52 to display the behavior label.

In this way, by inputting a video segment into the behavior recognition device 1, a behavior label corresponding to that video segment is generated, and the generated behavior label is displayed on the display of the output device 52.

(Effects of the embodiment)
According to the embodiment described above, the behavior recognition device 1 can generate behavior labels by performing learning in consideration of the behavioral features and linguistic features of the behavior labels. Furthermore, the behavior recognition device 1 can also classify behavior labels with similar behavioral features.

Other Embodiments
It should be noted that the present invention is not limited to the above embodiment. For example, in the learning operation, learning of the encoder 1022 in step ST102 does not necessarily have to be performed. For example, once learning of the encoder 1022 has been completed, it may be performed as needed.

While several embodiments of the present invention have been described, these embodiments are presented as examples and are not intended to limit the scope of the invention. These novel embodiments may be embodied in a variety of other forms, and various omissions, substitutions, and modifications may be made without departing from the spirit of the invention. These embodiments and their variations are within the scope and spirit of the invention, and are also included in the scope of the invention and its equivalents as set forth in the claims.

DESCRIPTION OF SYMBOLS 1... Behavior recognition device 10... Control unit 101... Data acquisition unit 102... Latent feature generation model construction unit 1021... Pre-trained behavior recognition model 1022... Encoder 1023... Decoder 1024... Language model 103... Training data generation unit 1031... Set extraction unit 1032... Latent feature vector generation unit 1033... Separability calculation unit 1034... Episode set addition unit 1035... Determination unit 1036... Memory control unit 104... Model update unit 1041... Batch extraction unit 1042... Loss calculation unit 1043... Model parameter update unit 1044... Separability threshold update unit 1045... Convergence determination unit 1046... Memory control unit 105... Behavior classification unit 1051... Latent feature vector generation unit 1052... Behavior label generation unit 1053... Behavior classification model 106... Output control unit 20... Program memory unit 30: Data storage unit 301: Learning data storage unit 302: Model storage unit 40: Communication interface 50: Input/output interface 51: Input device 52: Output device

Claims (7)

a data acquisition unit that acquires input data including a first video segment , and acquires training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model ;
a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model , an encoder , a decoder, and a language model ;
a latent feature generation model construction unit that calculates a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculates a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and trains the encoder and the decoder so as to minimize the first loss and the second loss;
an action label generation unit that generates an action label by inputting the latent feature vector into the action classification model;
an output control unit that outputs the behavior label;
An activity recognition device comprising: a data acquisition unit that acquires input data including a first video segment , and acquires training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model ;
a latent feature vector generation unit that generates a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder;
a model storage unit that stores parameters used by the behavior classification model;
a training data generation unit that generates a set of episodes based on the video segments and the training data;
a model update unit that acquires the parameters stored in the model storage unit, extracts batches from the set of episodes that satisfy a predetermined condition, estimates an action label of the extracted batch, calculates a loss between the estimated action label and an action label included in the training data, and updates the parameters by backpropagating the calculated loss; and
an action label generation unit that generates an action label by inputting the latent feature vector into the action classification model;
an output control unit that outputs the behavior label;
The learning data generation unit comprises:
a set extraction unit that extracts episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data;
a latent feature vector generation unit that generates a second latent feature vector by inputting the second video segment into the latent feature generation model;
a separability calculation unit that calculates a separability using the second latent feature vector;
an episode set addition unit that adds the degree of separation and the episode to the episode set;
Equipped with
The model update unit extracts the batch and uses it for learning if the degree of separation exceeds a threshold up to a specific number of episodes, does not extract the batch and does not use it for learning if the degree of separation does not exceed the threshold, and updates the threshold to be smaller for episodes that exceed the specific number of episodes . 3. The behavior recognition device according to claim 2, wherein the separability calculation unit calculates an inter-class variance and an intra-class variance of the support data based on the second latent feature vector, and calculates a separability that is a ratio of the inter-class variance to the intra-class variance. acquiring input data including a first video segment , and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model ;
generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model , an encoder , a decoder, and a language model ;
calculating a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculating a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and training the encoder and the decoder to minimize the first loss and the second loss;
generating an action label by inputting the latent feature vector into the action classification model;
outputting the activity label;
The activity recognition method includes: acquiring input data including a first video segment , and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an action classification model; generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder;
storing parameters used by the behavior classification model;
generating a set of episodes based on the video segments and the training data;
acquiring the parameters, extracting batches from the set of episodes that satisfy a predetermined condition, estimating action labels of the extracted batches, calculating a loss between the estimated action labels and action labels included in the training data, and updating the parameters by backpropagating the calculated loss;
generating an action label by inputting the latent feature vector into the action classification model;
outputting the activity label;
and generating the set of episodes comprises:
extracting episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data;
generating a second latent feature vector by inputting the second video segment into the latent feature generation model;
calculating a degree of separation using the second latent feature vector;
the degree of separation and the episodes as the episode set;
Equipped with
The behavior recognition method includes: updating the parameters by extracting the batch and using it for learning if the degree of separation exceeds a threshold up to a specific number of episodes; not extracting the batch and not using it for learning if the degree of separation does not exceed the threshold; and updating the threshold to be smaller for episodes that exceed the specific number of episodes . acquiring input data including a first video segment , and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model ;
generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model , an encoder , a decoder, and a language model ;
calculating a first loss between an action feature vector obtained from the pre-trained action recognition model and an action feature vector generated by inputting the latent feature vector to the decoder, calculating a second loss between a first language feature vector generated by inputting the action label and the object label to the language model and a second language feature vector generated by inputting the latent feature vector to the decoder, and training the encoder and the decoder to minimize the first loss and the second loss;
generating an action label by inputting the latent feature vector into the action classification model;
outputting the activity label;
An action recognition program comprising instructions for causing a processor of an action recognition device to execute the above. acquiring input data including a first video segment , and acquiring training data including a second video segment and an action label and an object label annotated to the second video segment when training an activity classification model ;
generating a latent feature vector by inputting the video segment into a latent feature generation model including a pre-trained action recognition model and an encoder;
storing parameters used by the behavior classification model;
generating a set of episodes based on the video segments and the training data;
acquiring the parameters, extracting batches from the set of episodes that satisfy a predetermined condition, estimating action labels of the extracted batches, calculating a loss between the estimated action labels and action labels included in the training data, and updating the parameters by backpropagating the calculated loss;
generating an action label by inputting the latent feature vector into a pre-trained action classification model;
outputting the activity label;
and generating the episode set comprises:
extracting episodes, which are sets of N-way K-shot support data and query data, from the set of the second video segments and the action labels included in the training data;
generating a second latent feature vector by inputting the second video segment into the latent feature generation model;
calculating a degree of separation using the second latent feature vector;
the degree of separation and the episodes as the episode set;
Equipped with
The behavior recognition program includes updating the parameters by extracting the batch and using it for learning if the degree of separation exceeds a threshold up to a specific number of episodes, not extracting the batch and not using it for learning if the threshold is not exceeded, and updating the threshold to be smaller for episodes that exceed the specific number .