JP7521382B2 - BEHAVIOR RECOGNITION LEARNING DEVICE, BEHAVIOR RECOGNITION ... LEARNING METHOD, BEHAVIOR RECOGNITION METHOD, AND BEHAVIOR RECOGNITION LEARNING PROGRAM - Google Patents

Description

The disclosed technology relates to a behavior recognition learning device, a behavior recognition device, a behavior recognition learning method, a behavior recognition method, and a behavior recognition learning program.

There is behavior recognition technology that estimates and recognizes the behavior of a moving object contained in an input video (or skeletal sequence). Behavior recognition technology has a wide range of industrial applications, such as analysis of surveillance camera video and human behavior understanding by robots. There is a wide range of behaviors that people can take in the real world. For example, the NTU RGB+D120 dataset (see Non-Patent Document 1), which recognizes daily human movements, defines 120 classes of behavior types. Behavior recognition technology is required to have the ability to accurately classify such many types of behavior.

Publicly known techniques with high accuracy include a technique that uses a Convolutional Neural Network (CNN) to extract and classify features from video (see Non-Patent Document 2). In addition, there is a technique that uses Graph Convolutional Networks (GCN) to extract and classify features from skeletal information acquired by skeletal detection (see Non-Patent Document 3).

J. Liu, A. Shahroudy, M. Perez, G. Wang, L-Y. Duan and A. C. Kot, “NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 2019. J. Lin, C. Gan, S. Han, “TSM: Temporal Shift Module for Efficient Video Understanding,” in IEEE Int. Con. on Computer Vision, 2019. S. Yan, Y. Xiong and D. Lin, “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition,” in the Thirty-Second AAAI Conference on Artificial Intelligence, 2018.

As the number of behavior types to be classified increases, similar behaviors are often included between classes, making behavior classification more difficult. In known technologies, training of feature extractors that are effective for classification is performed in an end-to-end manner, and features to be obtained from multiple convolutional layers are learned solely based on the correctness of the classification results in the learning process. In this way, conventional methods that do not explicitly consider the similarity of the behaviors to be classified tend to misrecognize similar behaviors, leaving room for improvement in accuracy.

The disclosed technology has been made in consideration of the above points, and aims to provide a behavior recognition learning device, a behavior recognition device, a behavior recognition learning method, a behavior recognition method, and a behavior recognition learning program that enable accurate estimation of the behavior type of a moving object.

A first aspect of the present disclosure is a behavior recognition learning device that includes: a behavior similarity calculation unit that receives, as learning data, behavior data including each of the behaviors of a motion target object and behavior labels corresponding to the behavior types of each of the behaviors, and calculates a similarity regarding the behaviors for each combination of the behaviors; a behavior clustering unit that clusters the behavior labels based on the similarity regarding the behaviors and generates a hierarchical structure of the behavior labels; and a hierarchical learning unit that learns parameters of a behavior recognition model for estimating the behavior type based on a loss calculated using the behavior data and the hierarchical structure.

A second aspect of the present disclosure is an action recognition learning device that includes: action data including each action of an action target object and an action label corresponding to each action type of the action as learning data; a language similarity calculation unit that calculates a language similarity for each of the action labels using a function that extracts language features; an action clustering unit that clusters the action labels based on the language similarity and generates a hierarchical structure of the action labels; and a hierarchical learning unit that learns parameters of an action recognition model for estimating an action type based on a loss calculated using the action data and the hierarchical structure.

A third aspect of the present disclosure is a behavior recognition learning method, characterized in that it causes a computer to execute a process of accepting, as learning data, behavior data including each of the behaviors of a moving object and behavior labels corresponding to the behavior types of each of the behaviors, calculating a similarity related to the behaviors for each combination of the behaviors, clustering the behavior labels based on the similarity related to the behaviors, generating a hierarchical structure of the behavior labels, and learning parameters of a behavior recognition model for estimating the behavior type based on a loss calculated using the behavior data and the hierarchical structure.

A fourth aspect of the present disclosure is a behavior recognition learning method, which is characterized by having a computer execute a process of accepting, as learning data, behavior data including each of the behaviors of a moving object and behavior labels corresponding to the behavior types of each of the behaviors, calculating a language similarity for each of the behavior labels using a function that extracts language features, clustering the behavior labels based on the language similarity, generating a hierarchical structure of the behavior labels, and learning parameters of a behavior recognition model for estimating the behavior type based on a loss calculated using the behavior data and the hierarchical structure.

The disclosed technology makes it possible to accurately estimate the behavior type of a moving object.

FIG. 1 is a diagram illustrating an example of a conventional training method for a CNN or GCN-based action recognizer. FIG. 13 is a diagram showing an example of a case in which the correct answer for an activity label obtained by a conventional method differs from the estimated result. FIG. 13 is a diagram showing an example of a case in which the correct answer for an activity label obtained by a conventional method differs from the estimated result. 2 is a block diagram showing the hardware configuration of the behavior recognition learning device and the behavior recognition device. FIG. 1 is a block diagram showing a configuration of an action recognition learning device according to a first embodiment. FIG. 13 is a diagram showing an example of identifying abstract labels in each layer and identifying a final action label. 1 is a block diagram showing a configuration of a behavior recognition device according to a first embodiment. FIG. 13 is a diagram showing an example of a case where classification into abstract labels and final action labels is performed in stages. FIG. 13 is a diagram showing an example of a case where classification into abstract labels and final action labels is performed in stages. 5 is a flowchart showing a flow of an action recognition learning process by the action recognition learning device of the first embodiment. 5 is a flowchart showing a flow of a behavior classification process performed by the behavior recognition device of the first embodiment. FIG. 11 is a block diagram showing a configuration of an action recognition learning device according to a second embodiment. 13 is a flowchart showing a flow of an action recognition learning process by the action recognition learning device of the second embodiment.

Below, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

First, an overview of the present disclosure will be described. FIG. 1 is a diagram showing an example of a conventional learning method of a CNN or GCN-based action recognizer. The example of FIG. 1 is an image diagram of a mode of learning an action recognizer that estimates an action label by inputting a video of an action of drinking water and a skeleton sequence of the action, and performing convolution multiple times. When such an action recognizer is used, as described in the above problem, it becomes more difficult to distinguish between similar actions as the number of action types to be classified increases. FIGS. 2A and 2B are diagrams showing an example of a case in which the correct answer and the estimated result of the action label by the conventional method differ. As shown in FIGS. 2A and 2B, the correct answer and the estimated result are different, indicating that it is difficult. In the example of FIG. 2A, the correct answer and the recognition result are different for the V sign action (make victory sign) and the OK sign action (make OK sign). In the example of FIG. 2B, the correct answer and the recognition result are different for the action of putting an object into a bag and the action of taking an object out of a bag. Since it is difficult to distinguish between such similar actions, the accuracy of the estimation decreases. Table 1 shows the accuracy of the estimation of the behavior type and an example of an erroneous estimation.

In the method disclosed herein, as one means of simplifying complex classification problems, a hierarchical classification is assumed in which the granularity of the classes to be identified is set in stages, with coarse classification being used for rough classification and fine classification being used for detailed classification of actions. By applying this hierarchical classification to action recognition, the accuracy of action recognition is improved. If a hierarchical structure can be set based on the similarity of actions, the learning of a classifier that focuses on rough differences in actions in coarse classification and on detailed differences in actions in fine classification can be promoted, and it is hoped that actions can be recognized with high accuracy even when similar actions are included between classes.

In this disclosure, we have considered introducing the above-mentioned hierarchical classification into a CNN or GCN-based action recognizer (hereinafter referred to as an action recognition model), which is one of the most highly accurate known technologies. Generally, action labels for action recognition are not given a hierarchical structure. Therefore, in order to abstract the granularity of the classes to be identified, we first provide an "action clustering unit" that hierarchically clusters classes using the features of the actions to be recognized and generates a hierarchical structure. Furthermore, we provide a "hierarchical learning unit" that trains the CNN to perform hierarchical classification based on the hierarchical structure obtained by clustering.

[Premise of the embodiment]
For simplicity, in the embodiment of the present disclosure, a GCN is used as a base action recognition model. Generally, in action recognition using a GCN (Non-Patent Document 1, etc.), during learning, human body skeletal sequence information estimated from a video by a skeleton detector (Reference Document 1, etc.) and label information of actions occurring in the video are input, and an action recognition model is learned. During estimation, skeleton information estimated from the video is input, and action labels occurring in the input video are output.
[Reference 1] K. Sun, B. Xiao, D. Liu and J. Wang, “Deep High-Resolution Representation Learning for Human Pose Estimation,” in IEEE Conf. on Computer Vision and Pattern Recognition, 2019.

The present disclosure is applicable not only to the above behavior recognition method, but also to other suitable behavior recognition methods such as a method of performing behavior recognition directly from video (for example, Non-Patent Document 2).
[Reference 2] J. Carreira and A. Zisserman, “Quo vadis, action recognition? a new model and the kinetics dataset,” in Int. Conf. on Computer Vision and Pattern Recognition, 2018.

The following description assumes that the object to be moved is a human body, but is not limited to this. For example, any object that can be classified can be classified as long as it is possible to take into account changes in joint movements. In addition to the human body, machine tools, animals, and multiple objects whose connection points are treated as joints may also be considered as objects to be moved.

Hereinafter, as a first embodiment of the present disclosure, a method for hierarchizing behavior labels based on the similarity of skeletal movements of training data ("similarity related to movements" in the technology of the present disclosure) and training an behavior recognition model will be described. Also, as a second embodiment, a method for hierarchizing behavior labels based on the linguistic similarity of behavior labels ("similarity related to language" in the technology of the present disclosure) and training an behavior recognition model will be described. In the present disclosure, in addition to the above two types of measures of behavior similarity that can be used to create a hierarchical structure, video features (for example, a significant measure such as the similarity in Reference 2) can also be used.

[First embodiment]
(Configuration of the First Embodiment)
The configuration of the first embodiment will be described below. The behavior recognition learning device and the behavior recognition device will be described separately. Note that they may be configured by the same device.

Figure 3 is a block diagram showing the hardware configuration of the behavior recognition learning device 100.

As shown in FIG. 3, the behavior recognition learning device 100 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads a program from the ROM 12 or the storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or the storage 14. In this embodiment, the behavior recognition learning program is stored in the ROM 12 or the storage 14.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive), and stores various programs including an operating system, and various data.

The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

The display unit 16 is, for example, a liquid crystal display, and displays various information. The display unit 16 may also function as the input unit 15 by adopting a touch panel system.

The communication interface 17 is an interface for communicating with other devices such as terminals. For example, a wired communication standard such as Ethernet (registered trademark) or FDDI, or a wireless communication standard such as 4G, 5G, or Wi-Fi (registered trademark) is used for the communication. Similarly, the behavior recognition device 150 has a CPU 21, a ROM 22, a RAM 23, a storage 24, an input unit 25, a display unit 26, and a communication I/F 27. Each component is connected to each other via a bus 29 so that they can communicate with each other. Explanations of each component of the hardware configuration are omitted because they are the same as those of the behavior recognition learning device 100. Note that the behavior recognition learning device of the second embodiment can also have a similar hardware configuration.

Next, each functional configuration of the behavior recognition learning device 100 will be described. FIG. 4 is a block diagram showing the configuration of the behavior recognition learning device 100 of the first embodiment. Each functional configuration is realized by the CPU 11 reading out the behavior recognition learning program stored in the ROM 12 or storage 14, expanding it in the RAM 13, and executing it.

As shown in FIG. 4, the behavior recognition learning device 100 includes a hierarchical structure generation unit 110, a hierarchical structure storage unit 116, a parameter storage unit 118, and a hierarchical learning unit 120. The hierarchical structure generation unit 110 includes an action similarity calculation unit 112 and a behavior clustering unit 114. The hierarchical learning unit 120 includes a behavior recognition unit 122, a loss calculation unit 124, and a parameter optimization unit 126.

The behavior recognition learning device 100 accepts motion data and behavior labels as input learning data. The motion data is each video and a skeletal sequence corresponding to each video. Each video includes a motion target performing a predetermined motion. The skeletal sequence is sequence data corresponding to the motion of the motion target in the video. The behavior label indicates a class corresponding to each behavior type of the motion included in the motion data. For example, the 120 classes in Non-Patent Document 1 can be used as the behavior labels. Note that it is also possible to use only the video as the motion data without accepting the skeletal sequence, and to use the skeletal sequence estimated by analyzing the video. It is also possible to use only the skeletal sequence without accepting the video as the motion data.

In the behavior recognition learning device 100, first, the hierarchical structure generation unit 110 generates a hierarchical structure of behavior labels for the learning data. In the hierarchical structure generation unit 110, the behavior similarity calculation unit 112 calculates the similarity of behavior between each behavior type using the skeleton sequence in the learning data. Then, the behavior clustering unit 114 clusters the behavior types based on the calculated similarity of behavior, and similar classes are hierarchically organized. The clusters of each hierarchy thus obtained are used as abstract labels for the behavior classes, thereby generating a hierarchical structure. Note that the behavior recognition learning device 100 may be configured not to include the hierarchical structure generation unit 110, but to accept a hierarchical structure generated by an external device.

In the behavior recognition learning device 100, the hierarchical learning unit 120 learns the parameters of the behavior recognition model using the hierarchical structure of behavior types previously generated as described above. First, the behavior recognition unit 122 estimates for each layer which abstract label defined in the hierarchical structure the skeletal sequence input as learning data corresponds to. Next, the loss calculation unit 124 calculates the loss based on the accuracy of the estimation result at each layer. Then, the parameter optimization unit 126 updates the parameters in the parameter storage unit 118 based on the loss, thereby progressing the learning of the behavior recognition model.

The action similarity calculation unit 112 calculates the action similarity for each combination of actions in the action data. The action similarity calculation unit 112 uses the skeletal sequence included in the action data of the learning data to calculate the movement distance between each action type (i.e., between actions), and uses the movement distance to calculate the action similarity. Since calculating the distance between all actions would be computationally expensive, first, distance calculation is performed between actions of the same type, and one piece of data showing an average action within the same type is selected from each action type as a representative action. After that, the distance between the representative actions is calculated, and this is used as the action similarity.

The distance between actions is calculated as the following distance F by DP matching using the Euclidean distance of the skeleton. By using DP matching, it is possible to calculate the similarity taking into account partial expansion and contraction while maintaining the time series order relationship between data, so that the similarity (distance F) can be calculated without being affected by the time difference between the actions of the data. Specifically, the distance F between two skeleton series A = {A ₁ , ..., A _t , ..., A _T } and B = {B ₁ , ..., B _u , _... , B _U } is defined by the following formula (1 ₎ as a problem of determining the correspondences t ₁ , t ₂ , ..., t _U that minimize the cumulative evaluation formula, which is the distance d(t, u) between A t and B u.

... (1)

Here, the distance d(t, u) between certain frames of the skeleton sequence is defined as the sum of Euclidean distances between each joint included in the skeleton information according to the following equation (2).

... (2)

Here, A _t,j and B _u,j represent the j-th joint coordinates in each skeletal series. The representative action is selected by selecting an action that has the smallest sum of distances from other actions of the same type included in the learning data, using the distance F between skeletal series defined above. Specifically, if the n-th skeletal series of a certain action type c in the learning data is A ^c,k (k∈N), the skeletal sequence A ^c' of the representative action is calculated by the following formula (3).

...(3)

Then, a similarity S(c _l, c _m ) between two behavior types c _l and c _m is defined by the following formula (4) using the distance between the representative behaviors of each behavior. The similarity S is an example of a similarity related to the operation of the technology of the present disclosure.

...(4)

The behavior clustering unit 114 clusters the behavior labels based on the similarity S of the actions calculated for each behavior type, generates a hierarchical structure of the behavior labels, and stores the generated hierarchical structure in the hierarchical structure storage unit 116. In this embodiment, the number of clusters is changed while performing the classification multiple times, and the behavior labels included in each cluster are classified into abstract labels and final behavior labels to generate a hierarchical structure. In other words, for each cluster level, the behavior labels are classified into abstract labels corresponding to the abstracted levels and final behavior labels corresponding to the final levels. Any significant method can be used for clustering, but for example, using a hierarchical clustering method (such as a complete link method) ensures that each behavior type has a hierarchical structure between clusters when the number of clusters is changed. This generates a hierarchical structure in which the upper level has a high level of abstraction and the lower level has a low level of abstraction. A high level of abstraction means that the clusters are related to behavior types of rough actions. A low level of abstraction means that the clusters are related to behavior types of fine actions. The behavioral clustering unit 114 stores the hierarchical structure learned in this manner in the hierarchical structure storage unit 116.

The hierarchical structure storage unit 116 stores the hierarchical structure. The parameter storage unit 118 stores the parameters of the behavior recognition model. The initial values of the parameters can be set before learning.

The hierarchical learning unit 120 performs each process to learn the parameters of the behavior recognition model for estimating the behavior type, using the loss calculated using the motion data and the behavior label for each layer in the hierarchical structure. Thereafter, the behavior recognition unit 122 and the loss calculation unit 124 perform processing that takes the hierarchical structure into account.

The behavior recognition unit 122 uses a predetermined method for performing classification for each layer according to the hierarchical structure, and outputs the classification results to the loss calculation unit 124 using motion data as input. Classification is performed by concatenating features from each layer to identify abstract labels for intermediate layers (hereinafter, corresponding to intermediate blocks), and final behavior labels for the final layer (hereinafter, corresponding to final blocks).

The specified method used by the behavior recognition unit 122 will be described. In general behavior recognition using GCN, convolution blocks each consisting of a certain number of convolution layers are connected together while increasing the dimensions of the extracted features, and classification is performed based on the features extracted from the final block. Note that a convolution block is a block consisting of multiple convolution layers each having the same number of dimensions of the output features, and corresponds to a hierarchical structure. In the behavior recognition unit 122, classification is performed not only in the final block but also in the intermediate blocks. Each convolution block of the GCN corresponds to each layer of the hierarchical structure. By identifying abstract labels with a high level of abstraction in shallow blocks and abstract labels with a low level of abstraction in deep blocks, the classification granularity of behavior types is refined hierarchically, and all behavior types required for classification are identified in the final block.

Figure 5 shows an example of the identification of abstract labels in each layer and the identification of final action labels. For example, when this mechanism is added to ST-GCN (see Non-Patent Document 1), global average pooling (GAP) is applied to the output of the intermediate block to extract features, and then identification is performed by the final block. Note that the "convolution" shown in Figure 5 corresponds to the convolution block, and the output of GAP is the feature to be extracted. By adding this process to multiple intermediate blocks, action recognition is performed hierarchically. Furthermore, since the features extracted in the intermediate block are considered to be useful features for estimation in the subsequent intermediate blocks, a certain layer may be concatenated with the features of the previous intermediate block (concatenation: the + symbol in Figure 5 indicates concatenation) before identification by that layer. The feature extraction may be performed from either the image or the skeletal sequence. In the example shown in FIG. 5, in the second layer classification, the feature quantities extracted by GAP in the first layer classification are linked with the feature quantities of the next second layer to perform classification. Then, in the final block, the feature quantities of the first layer and the feature quantities of the second layer are linked with the feature quantities of the final block to perform classification, thereby classifying the final behavior label. Through the above processing by the behavior recognition unit 122, classification results are obtained for the abstract labels for each layer and the final behavior label.

The loss calculation unit 124 calculates the loss by finding the error between the first value and the second value, which will be described below, using the classification result of the behavior recognition unit 122. The loss calculation unit 124 calculates the loss by comparing each of the classified abstract labels and the final behavior label with their true values (teaching data). For example, when using cross-entropy error following ST-GCN (see Non-Patent Document 1), the loss L is calculated by the following formula (5).

...(5)

Here, s is a hierarchical level, k ^s is an abstract label in a certain hierarchical level, q ^s (·) is a teacher of the abstract label in a certain hierarchical level, and p ^s (·) is an estimation result in a certain hierarchical level (likelihood for each abstract label). Also, q(k) is a teacher label, and p(k) is an estimation result of the final block. In the technology disclosed herein, the hierarchical level s corresponds to "each of the hierarchical levels in the hierarchical structure", the abstract label k ^s is an "abstract label in the hierarchical level", and the estimation result p ^s (·) in a certain hierarchical level corresponds to "likelihood for the abstract label of the hierarchical level", and the first term of the above formula (5) corresponds to the first value. Also, the teacher label q(k) corresponds to "teacher label for each action", and the final block p(k) corresponds to "estimated result p(k) of the final action label in the final block", and the second term corresponds to the second value.

The parameter optimization unit 126 optimizes the parameters of the behavior recognition model based on the loss L calculated by the loss calculation unit 124 using the hierarchical structure, and stores the result in the parameter storage unit 118. At this time, a useful algorithm such as the method described in Reference 3 can be used as the parameter optimization algorithm.
[Reference 3] D. Kingma, J. Ba, “Adam: A Method for Stochastic Optimization,” in Int. Conf. on Learning Representations, 2018.

Next, each functional configuration of the behavior recognition device 150 will be described. The behavior recognition device 150 estimates a behavior type using parameters of the behavior recognition model learned by the behavior recognition learning device 100. FIG. 6 is a block diagram showing the configuration of the behavior recognition device 150 of the first embodiment. Each functional configuration is realized by the CPU 21 reading out a behavior recognition program stored in the ROM 22 or storage 24, expanding it in the RAM 23, and executing it.

As shown in FIG. 6, the behavior recognition device 150 includes a hierarchical structure storage unit 116, a parameter storage unit 118, and a behavior estimation unit 152. The same reference numerals are used for each storage unit that stores the learning results of the behavior recognition learning device 100, and the description thereof will be omitted.

The behavior estimation unit 152 inputs the motion data (video or skeletal sequence) of the estimation target to the behavior recognition model, and estimates the behavior type of the motion included in the motion data using the hierarchical structure and parameters of the behavior recognition model. The processing of the behavior recognition model may be the same as that of the behavior recognition unit 122 of the behavior recognition learning device 100, and the behavior type may be estimated using the hierarchical structure stored in the hierarchical structure storage unit 116 and the parameters of the behavior recognition model stored in the parameter storage unit 118. In addition, since the behavior estimation unit 152 outputs the abstract label identification result as the output of the intermediate layer of the behavior recognition model during the estimation process, each of the abstract labels of the intermediate layers and the final final behavior label may be output. By classifying the motions in this way in stages, it is possible to perform identification focusing on differences in how the motions are perceived.

Figures 7A and 7B are diagrams showing an example of a case where abstract labels and final action labels are classified in stages. Figures 7A and 7B show a two-stage case where the first stage is classified by abstract labels and the final stage is classified by final action labels. In Figure 7A, the top two actions in the first stage show that both actions are classified by the abstract label "hand sign". In the top two actions in the final stage show that the left action is classified by the final action label "make victory sign" and the right action is classified by the final action label "make OK sign". In Figure 7B, the bottom two actions in the first stage show that both actions are classified by the abstract label "bag operation". In the two actions below in the final stage, the left action is classified with the final action label "put object into bag," and the right action is classified with the final action label "take object out of bag."

(Operation of the First Embodiment)
Next, the operation of the behavior recognition learning device 100 will be described.

Figure 8 is a flowchart showing the flow of the behavior recognition learning process by the behavior recognition learning device 100 of the first embodiment. The behavior recognition learning process is performed by the CPU 11 reading out the behavior recognition learning program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. The behavior recognition learning device 100 accepts learning data (motion data and behavior labels) as input and performs the following processes.

In step S100, the CPU 11, as the action similarity calculation unit 112, calculates the action similarity for each combination of actions in the action data according to the above formulas (1) to (4).

In step S102, the CPU 11, as the behavior clustering unit 114, clusters the behavior labels based on the similarity S of the actions calculated for each behavior type, generates a hierarchical structure of the behavior labels (abstract labels and final behavior labels), and stores the hierarchical structure in the hierarchical structure storage unit 116.

In step S104, the CPU 11, as the behavior recognition unit 122, uses a predetermined method for performing classification for each layer according to the hierarchical structure, inputs the action data, and outputs the classification result to the loss calculation unit 124. The classification uses the features of each layer in combination to identify abstract labels as intermediate block classification, and final behavior labels as final block classification.

In step S106, the CPU 11, as the loss calculation unit 124, calculates the loss L by using the identification result of the behavior recognition unit 122 and determining the error between the first value and the second value according to equation (5).

In step S108, the CPU 11, as the parameter optimization unit 126, optimizes the parameters of the behavior recognition model based on the loss L calculated by the loss calculation unit 124 using the hierarchical structure, stores the result in the parameter storage unit 118, and ends the process.

According to the behavior recognition learning device 100 of the first embodiment described above, it is possible to learn a behavior recognition model that enables the behavior type of an action target object to be estimated with high accuracy.

Figure 9 is a flowchart showing the flow of the behavior classification process by the behavior recognition device 150 of the first embodiment. The behavior classification process is performed by the CPU 21 reading out a behavior classification program from the ROM 22 or the storage 24, expanding it into the RAM 23, and executing it.

In step S150, the CPU 21, as the behavior estimation unit 152, receives the motion data of the object to be estimated.

In step S152, the CPU 21, as the behavior estimation unit 152, acquires the hierarchical structure stored in the hierarchical structure memory unit 116 and the behavior classification model parameters stored in the parameter memory unit 118.

In step S154, the CPU 21, as the behavior estimation unit 152, inputs the motion data (video or skeletal sequence) of the estimation target to the behavior recognition model, and estimates the behavior type of the motion included in the motion data using the hierarchical structure and parameters of the behavior recognition model.

In step S156, the CPU 21, as the behavior estimation unit 152, outputs the estimation result of the behavior type and ends the process.

As described above, the behavior recognition device 150 of this embodiment can accurately estimate the behavior type of the moving object.

[Second embodiment]
Next, a second embodiment will be described. FIG. 10 is a block diagram showing the configuration of an action recognition learning device 200 of the second embodiment. As shown in FIG. 10, the action recognition learning device 200 includes a hierarchical structure generating unit 210, a hierarchical structure storage unit 116, a parameter storage unit 118, and a hierarchical learning unit 120. The hierarchical structure generating unit 210 includes a language similarity calculating unit 212 and an action clustering unit 114. The hierarchical learning unit 120 includes an action recognizing unit 122, a loss calculating unit 124, and a parameter optimizing unit 126. The same reference numerals are used for the same parts as those in the first embodiment, and the description thereof will be omitted.

The second embodiment differs from the first embodiment in that the language similarity calculation unit 212 is used to calculate the language similarity when generating the hierarchical structure. In general, action names (such as "jump" and "pick up") are given to action labels in learning data for action recognition. In this embodiment, the hierarchical structure is generated by using the semantic closeness of these action names as the similarity of the actions.

The second embodiment will be described. The behavior recognition learning device 200 receives, as learning data, behavior data including each behavior of a behavior target object and behavior labels corresponding to the behavior types of each of the behaviors.

The language similarity calculation unit 212 calculates the language similarity (language similarity) for each combination of actions in the action data using a function that extracts language features from the action label. The language similarity calculation unit 212 calculates language features from the action label, and the distance in the feature space is taken as the language similarity. Specifically, if the function that extracts language features from a certain action label is W(·), then S(c _l , c _m ) in this embodiment is defined as in the following formula (6).

...(6)

In this case, a meaningful method can be used for extracting language features from the action label. For example, GloVe (see Reference 4) or BART (see Reference 5) can be used, which has a feature extractor trained from a huge amount of text. Here, since GloVe is a method for extracting features from words, in the case of a label consisting of multiple words such as "pick up," the average of the features extracted from each word is used as the feature for that label.
[Reference 4] J. Pennington, R. Socher, CD Manning, “Glove: Global vectors for word representation,” in proc. of the 2014 conference on empirical methods in natural language processing, 2014.
[Reference 5] M. Lewis, Y. Liu, N. Goyal, M. Ghazvininejad, A. Mohamed, O. Levy, V. Stoyanov, L. Zettlemoyer, “BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension,” in Proc of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.

The behavior clustering unit 214 clusters behavior labels based on linguistic similarity and generates a hierarchical structure of behavior labels. As in the first embodiment, the hierarchical learning unit 120 learns parameters of a behavior recognition model for estimating behavior types based on the loss calculated using the motion data and the hierarchical structure. The hierarchical structure used for learning differs from that in the first embodiment.

The hardware configuration of the behavior recognition learning device 200 in the second embodiment may be the same as that in the first embodiment.

Figure 11 is a flowchart showing the flow of the behavior recognition learning process by the behavior recognition learning device 200 of the second embodiment. As shown in Figure 11, in step S200, the CPU 11, as the language similarity calculation unit 212, calculates the language similarity (language similarity) for each combination of actions in the action data using a function that extracts language features from the action label. The subsequent processing is the same as in the first embodiment.

<<Experimental Example>>
An experimental example using the method described in the second embodiment will be described. In this experimental example, the accuracy of action recognition is investigated using NTU RGB+D120 (see Non-Patent Document 1), which is a data set for action recognition using a skeletal sequence. In order to investigate the effectiveness of the technology disclosed herein for hierarchically recognizing actions, the action recognition accuracy was compared between the baseline method ST-GCN (see Non-Patent Document 3) and two methods in which the method described in the second embodiment is added to the baseline method. The hierarchical structure in the method disclosed herein is two-staged, and the previous 120 classes of action types are clustered once to create 80 classes of abstract labels.

As a result of the experiment, it was confirmed that the average accuracy (correct answer rate) improved by about 2 points by adding a hierarchical structure, as shown in Table 2. In addition, Table 3 shows a comparison of the individual accuracy for the bottom five behavior types for which ST-GCN had low recognition accuracy. Tables 2 and 3 confirm that hierarchical recognition improves the accuracy of many behaviors that were difficult to recognize with conventional baseline methods.

In addition, the behavior recognition learning process or behavior classification process executed by the CPU by reading the software (program) in each of the above embodiments may be executed by various processors other than the CPU. Examples of the processor in this case include a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture such as an FPGA (Field-Programmable Gate Array), and a dedicated electric circuit that is a processor having a circuit configuration designed exclusively for executing a specific process such as an ASIC (Application Specific Integrated Circuit). In addition, the behavior recognition learning process or behavior classification process may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, etc.). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor devices.

In addition, in each of the above embodiments, the behavior recognition learning program is described as being pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network. The same applies to the behavior classification program.

The following additional notes are provided regarding the above embodiment.

(Additional Note 1)
Memory,
at least one processor coupled to the memory;
Including,
The processor,
accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
Calculating a similarity score for each combination of the actions;
Clustering the action labels based on the similarity regarding the actions, and generating a hierarchical structure of the action labels;
learning parameters of an action recognition model for estimating an action type based on the motion data and the loss calculated using the hierarchical structure;
The behavior recognition learning device is configured as follows.

(Additional Note 2)
A non-transitory storage medium storing a program executable by a computer to execute an action recognition learning process,
accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
Calculating a similarity score for each combination of the actions;
Clustering the action labels based on the similarity regarding the actions, and generating a hierarchical structure of the action labels;
learning parameters of an action recognition model for estimating an action type based on the motion data and the loss calculated using the hierarchical structure;
Non-transitory storage media.

100, 200 Behavior recognition learning device 110, 210 Hierarchical structure generation unit 112 Action similarity calculation unit 114, 214 Behavior clustering unit 116 Hierarchical structure storage unit 118 Parameter storage unit 120 Hierarchical learning unit 122 Behavior recognition unit 124 Loss calculation unit 126 Parameter optimization unit 150 Behavior recognition device 152 Behavior estimation unit 212 Language similarity calculation unit

Claims (8)

accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
an action similarity calculation unit that calculates a distance between actions of the same type for each combination of actions, selects one piece of data showing an average action within the same type as a representative action from each action type, and calculates a distance between the representative actions to calculate a similarity between the actions;
an action clustering unit that clusters the action labels based on the similarity regarding the actions and generates a hierarchical structure of the action labels;
a hierarchical learning unit that learns parameters of an action recognition model for estimating an action type based on the action data and a loss calculated using the hierarchical structure;
An action recognition learning device comprising: accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
a linguistic similarity calculation unit that calculates a linguistic similarity for each of the action labels using a function that extracts a linguistic feature;
an action clustering unit that clusters the action labels based on the linguistic similarity and generates a hierarchical structure of the action labels;
a hierarchical learning unit that learns parameters of an action recognition model for estimating an action type based on the action data and a loss calculated using the hierarchical structure;
An action recognition learning device comprising: the behavior clustering unit generates the hierarchical structure by classifying the clusters into a hierarchy for each level of abstraction according to the similarity, and classifying each hierarchy into an abstract label corresponding to the abstract hierarchy and a final behavior label corresponding to a final hierarchy;
the hierarchical learning unit includes an action recognition unit, a loss calculation unit, and a parameter optimization unit;
the action recognition unit uses a predetermined method for performing classification for each layer according to the hierarchical structure, receives the action data as an input, and uses a concatenated feature amount extracted from a predetermined convolution block of each layer to output, as a classification result, a classification of the abstract label as a classification of an intermediate block, and a classification of the final action label as a classification of a final block;
the loss calculation unit calculates the loss by using the classification result to obtain an error between a first value using the abstract label in each layer of the hierarchical structure and a likelihood for the abstract label in the layer, and a second value using a teacher label for each of the actions and an estimation result of the final action label of a final block;
The behavior recognition learning device according to claim 1 , wherein the parameter optimization unit optimizes parameters of the behavior recognition model based on the loss. an action estimation unit that estimates an action type of an action included in an input video or a skeletal sequence by using the hierarchical structure and parameters of the action recognition model learned by the action recognition learning device according to any one of claims 1 to 3;
An action recognition device comprising: accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
For each combination of the actions, a distance calculation is performed between actions of the same type, one piece of data showing an average action within the same type is selected as a representative action from each action type, and a distance between the representative actions is calculated to calculate a similarity between the actions;
Clustering the action labels based on the similarity regarding the actions, and generating a hierarchical structure of the action labels;
learning parameters of an action recognition model for estimating an action type based on the motion data and the loss calculated using the hierarchical structure;
An action recognition learning method that causes a computer to execute processing. accepting, as learning data, motion data including each motion of the motion object and a motion label corresponding to a motion type of each of the motions;
Calculating a linguistic similarity for each of the behavior labels using a function that extracts linguistic features;
Clustering the activity labels based on the linguistic similarity and generating a hierarchical structure of the activity labels;
learning parameters of an action recognition model for estimating an action type based on the motion data and the loss calculated using the hierarchical structure;
An action recognition learning method that causes a computer to execute processing. estimating an action type of an action included in an input video or a skeletal sequence by using the hierarchical structure and parameters of the action recognition model trained by the action recognition learning method according to claim 5 or 6;
An action recognition method that causes a computer to execute processing. A behavior recognition learning program that causes a computer to execute the processing of each part of the behavior recognition learning device described in any one of claims 1 to 3.