JP7624941B2 - Motion prediction device, method and program - Google Patents

Description

The present invention relates to an apparatus, method, and program for predicting the continuation of a person's action based on video of that person's action, and in particular to an action prediction apparatus, method, and program for predicting a person's action by taking into account the surrounding situation.

Predicting detailed human movements, such as hand and foot movements, several seconds in advance is necessary in areas such as quickly detecting dangerous behavior and controlling robots and actuators that work in cooperation with people.

Non-Patent Document 2 discloses a technology that detects the skeletal position of a person from 2D video and predicts the future skeletal position from past skeletal positions using LSTM (Long Short Term Memory). While Non-Patent Document 2 makes it easy to predict simple movements, it is necessary to add additional restrictions on the postures that a person can take, making it difficult to impose strict constraints.

Non-Patent Document 1 discloses a technology that uses a CVAE (Conditional Variational Autoencoder) model as its base, which employs a GRU (Gated Recurrent Unit) instead of an LSTM as the encoder-decoder module. CVAE can learn to output plausible subsequent actions using past actions as conditions, based on latent representations randomly sampled from a standard normal distribution.

Non-Patent Document 1 further imposes diversity on the latent representation, taking into account the fact that there are many possible future human actions, so that multiple, diverse prediction results are output as subsequent actions. Non-Patent Document 1 applies various linear transformations to randomly sampled latent variables based on given past actions, and provides this to the CVAE as a new latent representation. This makes it possible to predict plausible actions that cover a wide range of future possibilities.

Patent document 1 discloses a technology that predicts a person's behavior by reflecting the characteristics of each individual. Patent document 1 analyzes the behavioral state and person identifier of a photographed person, and generates behavior prediction information based on rules that are matched against the obtained person information.

JP 2013-186556 A

Yuan, Ye and Kitani, Kris (2020). Dlow: Diversifying latent flows for diverse human motion prediction. ECCV Erwin Wu and Hideki Koike: FuturePose - Mixed Reality Martial Arts Training Using Real-Time 3D Human Pose Forecasting With a RGB Camera, WACV(2019) J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You Only Look Once: Unified, Real-Time Object Detection," 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779- 788, doi: 10.1109/CVPR.2016.91. Wojke, Nicolai and Bewley, Alex and Paulus, Dietrich, "Simple Online and Realtime Tracking with a Deep Association Metric", ICIP2017 Zhe Cao and Tomas Simon and Shih-En Wei and Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", CVPR2017 Yang, Ceyuan and Xu, Yinghao and Shi, Jianping and Dai, Bo and Zhou, Bolei, "Temporal Pyramid Network for Action Recognition", CVPR2020

In Non-Patent Document 1, it is possible to predict a variety of reasonable actions as a continuation of a given action. However, there are cases where it predicts actions that are unlikely due to a lack of consideration of the surrounding situation. For example, when making an autonomous robot refer to the predicted action results and create an action plan, it is desirable to have it concentrate on predicting actions that are more likely to be likely depending on the surrounding situation. However, Non-Patent Document 1 presents more prediction results than necessary, including actions that are unlikely, which may hinder the autonomous robot's action plan.

Patent document 1 predicts behavior that reflects the characteristics of each individual, but because it does not take into account the surrounding situation, there is a possibility that the predicted behavior will be inconsistent with the situation.

The object of the present invention is to provide a motion prediction device, method, and program that can solve the above technical problems and accurately predict the subsequent motions of a photographed person while taking into account the surrounding circumstances.

To achieve the above object, the present invention is a motion prediction device that predicts subsequent motions based on video of a person performing a motion, and is characterized by having the following configuration.

(1) The system includes a means for analyzing a person's current movements, a means for acquiring a destination point of the movements, a means for extracting a latent expression of a subsequent movement based on the person's current movements and the destination point, and a means for predicting the person's subsequent movement based on the person's current movements, the destination point, and the latent expression.

(2) The means for acquiring the destination point of the movement estimates the destination point based on the person's movement up to the present.

(3) The means for acquiring the destination point includes a means for detecting an object from an image and a means for estimating the direction of movement of a person based on the image, and is configured to estimate the position of the object detected in the direction of movement as the destination point.

(1) Accurate predictions are possible because the system takes into account the surrounding circumstances to predict the next steps of a person's actions.

(2) The destination point is related to movement, and since the destination point is estimated based on the movements up to the present, the destination point can be estimated with high accuracy.

(3) The destination point depends on the direction of movement, but since the position of an object detected in the direction of movement is estimated as the destination point, it becomes possible to estimate the destination point with even greater accuracy.

1 is a functional block diagram showing a configuration of a main part of an action prediction device according to an embodiment of the present invention. FIG. 2 is a functional block diagram showing a configuration of an expression extraction unit. 13 is a flowchart showing an operation of an expression extraction unit. FIG. 2 is a functional block diagram showing a configuration of an operation prediction unit.

The following describes in detail an embodiment of the present invention with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a motion prediction device 1 according to one embodiment of the present invention, and the main components include a video acquisition unit 10, a motion analysis unit 20, a destination point estimation unit 30, an expression extraction unit 40, and a motion prediction unit 50.

Such a three-dimensional shape reconstruction device 1 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

In this embodiment, for a single action made by a person, the subsequent action following that action is predicted based on the analysis results of past actions up to the present, and in particular, prediction of a scene in which a person is about to pick up an object (stationary object) is used as an example. Note that if there are multiple people for whom action prediction is to be made, individual predictions can be made by repeating the same process for each person.

The video acquisition unit 10 uses an RGB camera or a depth camera to capture a subject person whose movements are to be predicted for a predetermined period of time (e.g., 3 seconds) and acquires frame images showing the subject. Alternatively, a video file in which the subject has been captured in advance for a predetermined period of time may be acquired.

The motion analysis unit 20 analyzes the acquired video to detect and track people, and estimates the person's current motion based on the results of the detection and tracking. A deep learning model such as YOLO (You Only Look Once) disclosed in Non-Patent Document 3 can be used for person detection, and DeepSORT disclosed in Non-Patent Document 4 can be used for person tracking. This makes it possible to accurately track the same person across multiple frames.

The motion analysis unit 20 further performs posture estimation processing on each frame image in which a person appears. For posture estimation, a method can be used that outputs the positions of skeletal key points in the video as postures, such as OpenPose disclosed in Non-Patent Document 5. The positions of key points can be expressed in two-dimensional or three-dimensional coordinates.

Since the output of OpenPose and other tools is usually two-dimensional coordinates, additional processing is required to express it in three-dimensional coordinates. Examples include triangulation (multiple viewpoints in the input), adding depth information (depth images in the input), or fitting a 3D model of a human.

The motion analysis unit 20 outputs the estimated result of the person's motion by combining the results of all frames as the motion analysis result . Note that if the number of frames is T, the number of skeletal key points is J, and the posture is expressed in three-dimensional coordinates, the data format of the motion analysis result is a T×3J matrix.

In addition to the above analysis results, the actions of a person may also be analyzed and labeled, and an action label may be attached to the action analysis results. The action label can be estimated, for example, by using the deep learning model disclosed in Non-Patent Document 6.

The destination point estimation unit 30 includes an object detection unit 301 and a direction estimation unit 302, and detects an object that is the object of the action based on the results of the action analysis , and estimates the person's destination point based on the person's movement direction.

To predict the action of a person picking up an object as in this embodiment, the object detection unit 301 is given several categories of objects that a person can pick up in advance, and object detection is performed on objects that belong to those categories using the method described in Non-Patent Document 3.

In addition to the action of picking up an object, similar processing is possible if the objects associated with the target action are defined in advance. For example, to predict the action of opening a door, the detection target would be a handle, and to predict the action of sitting down, a chair, etc. If the action analysis unit 20 has recognized an action label, the detection target may be limited to objects associated with that action label.

Once an object is detected, the direction of the person's movement is used as a clue to identify the object that is the target of the movement. The direction estimation unit 302 calculates the person's direction of movement as the frame average of the movement direction of the center coordinates of the body, and identifies the target object as the object that is closest to the person on an extension of the direction of movement.

The estimated object position is expressed using the same coordinate system as the coordinate system that expresses the person's movements. In the case of two-dimensional coordinates, the coordinates are the center of the area output by the detector, and in the case of three-dimensional coordinates, the three-dimensional coordinates of the center of the object are obtained by triangulation, depth imaging, fitting of a predefined object model, etc. The estimated object position is sent to the downstream expression extraction unit 40 as the destination point of the movement together with the person's movement.

In addition, if the action is to pick up an object, it is considered that the way of picking up the object will change depending on the type of object, such as a cup or a bag, so the type of object may also be sent to the expression extraction unit 40 at the subsequent stage. The type of object can be represented by a numerical value, such as an ID, and by using the detector disclosed in Non-Patent Document 3, the ID can be output together with the detection result.

When the type of object is taken into consideration, for example, the object ID may be converted into a one-hot vector representation and combined with the destination point data before subsequent processing. Note that instead of the destination point estimation unit 30, a configuration for inputting the destination point may be provided so that the destination point can be input manually.

Based on the estimated results of the person's action x and destination point y, the expression extraction unit 40 extracts multiple valid latent expressions z by inferring latent expressions of the person's future actions using an approximate probability distribution.

Figure 2 is a functional block diagram showing the configuration of the expression extraction unit 40, in which the person's current movement x is input to a gated regression unit (GRU) 401, and the destination point y is input to a multilayer perceptron (MLP) 402. The GRU 401 weights the person's movement x according to each time, and then outputs it to the downstream multilayer perceptron 403.

Figure 3 is a flowchart showing the procedure by which the expression extraction unit 40 extracts a latent expression z based on the estimation results of a person's movement x and a destination point y. Here, the person's movement x and the destination point y are assumed to be expressed in three-dimensional coordinates.

In step S1, K vector values ε={ε ₁ , ..., ε _K } are randomly sampled from a multivariate Gaussian distribution N(0,I) and provided to the conversion unit 405. In step S2, each MLP 403 calculates K matrix-vector pairs A _k , b _k (k=1, ..., K) by encoding a person's motion x∈R ^T×3J using K complex nonlinear functions φ (such as a pre-trained neural network).

In step S3, the subsequent conversion unit 405 converts the K vector values ε into K vector values δ (δ ₁ to δ _k ) by applying the K vector values ε to the following equation (1).

_δk = _{Ak εk} + _bk (1)

In step S4, the MLP 402 encodes the destination point y using L complex nonlinear functions ψ (such as a pre-trained neural network) to calculate L matrix-vector pairs C _l , d _l (l=1, . . . , L).

In step S5, the subsequent conversion unit 406 applies the K vector values δ to the following equation (2) to convert them into K·L latent expressions Z={Z _{1,1 , .} . . , Z _K,L }.

Z _k,1 =C ₁・δ ₁ +d ₁
…
Z _{k, L} = C _L・δ _k +d _L (2)

The nonlinear function φ in step S2 and the nonlinear function ψ in step S4 are learned together on a dataset including pairs of a person's movement x and a destination point y. The person's movement x in the dataset is separated into intermediate frames and treated as the "observed movement" and the correct "continued movement." The learning method may be very similar to that in Non-Patent Document 1. Specifically, the objective function in the following equation (3) is minimized using an optimization method such as stochastic gradient descent.

L _Recon +L _Div +L _KL (3)

Here, L _Recon is the reconstruction error, and L _Div is an energy function that promotes diversity in the prediction results. If the prediction results of K·L actions obtained by inputting the latent representation z to the action prediction unit 50 described below are expressed by the following formula (4) and the correct answer is expressed by the following formula (5), then L _Recon and L _Div can be obtained by the following formulas (6) and (7). Here, λ is set to an appropriate constant.

L _KL denotes the average Kullback-Leibler distance between the probability distribution of the K·L latent representations z and N(0,I). The probability distribution of the latent representations z _{k, l} can be expressed as the following equation (8) as a probability distribution obtained by affine transforming N(0,I) in order with A _k , b _k and C _l , d _l .

If the Kullback-Leibler distance between the above equation (8) and N(0,I) is Dk _,l , then _LKL can be found by the following equation (9). Note that since the output of the motion prediction unit 50 is used in this learning, it is necessary to train the motion prediction unit 50 in advance.

The order of steps S2 and S3 and steps S4 and S5 may be swapped. In that case, the symbols in the above equation would be swapped as appropriate, such as sampling L vector values in step S1.

Steps S2 and S3 and steps S4 and S5 can be performed simultaneously to encode the person's movement x and destination point y using K x L complex nonlinear functions (such as pre-trained neural networks), calculate K x L matrix and vector pairs A and b, and obtain the latent representation z by converting the vector value ε just once.

The obtained latent representation z can be regarded as a value sampled from a probability distribution conditioned by the person's movement x and the object's position y, and a total of K·L latent representations z are sent to the subsequent movement prediction unit 50 as latent representations of subsequent movements.

The action prediction unit 50 predicts multiple appropriate actions of a person based on the person's actions up to the present time x, destination point y, and latent expression z. For example, the person's action x, destination point y, and K·L latent expressions z are input to a complex nonlinear function ρ (such as a pre-trained neural network), and K·L predictions of subsequent actions x' are output. An example of a model in which the complex nonlinear function is a neural network is shown in Figure 4.

The GRU 501 weights the past actions x of the person up to the present according to each time. The MLP 502 encodes the destination point y by L complex nonlinear functions ψ and calculates L matrix-vector pairs C _l , d _l (l=1, ..., L). The combination module 503 combines the outputs of the GRU 501 and MLP 502 with the latent representation z and outputs it to the downstream GRU 504.

GRU504 infers a predicted motion x' from x, y, and z using a nonlinear function ρ. The nonlinear function ρ is learned on a dataset that includes pairs of human motion x and destination y. It is also possible to estimate the ideal state by using a method similar to CVAE, which is a deep generative model on which Non-Patent Document 1 is based, and taking into account information on the objective variable in generating the latent variables.

GRU504 considers an objective function with two terms: the reconstruction error of the prediction x', the approximate probability distribution of the latent representation z, and the Kullback-Leibler distance with N(0,I), and minimizes it using optimization methods such as stochastic gradient descent.

In the above embodiment, both the expression extraction unit 40 and the action prediction unit 50 take the destination point y into consideration, but it is also possible to have only one of them take it into consideration. Specifically, it is possible to omit either the processing of steps S4 and S5 performed by the expression extraction unit 40 or the input of the destination point y to the action prediction unit 50.

The above embodiment allows accurate prediction of a person's next movements based on video of the person moving, making it possible to provide an inexpensive and convenient movement prediction system to many people regardless of geographic or economic disparity. As a result, it will be possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the United Nations-led Sustainable Development Goals (SDGs).

1...Movement prediction device, 10...Video acquisition unit, 20...Movement analysis unit, 30...Destination point estimation unit, 40...Expression extraction unit, 50...Movement prediction unit, 401, 501, 504...GRU, 402, 403, 404, 502...MLP, 405, 406...Conversion unit, 503...Combination module

Claims (8)

A motion prediction device for predicting a subsequent motion of a person based on a video of the person performing the motion, comprising:
a means for analyzing a video of a person in motion , estimating each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result ;
A means for estimating a destination point of the person's movement based on the result of the movement analysis ;
A means for extracting a latent expression of a subsequent motion of the person based on the motion analysis result and a destination point;
and means for predicting a subsequent action of the person based on the result of the action analysis , the destination point and the latent expression. The motion prediction device according to claim 1, characterized in that the means for outputting the motion analysis result analyzes a video of a person moving, detects, tracks and estimates a posture of the person, and outputs a series of postures of the person up to the present. 3. The motion prediction device according to claim 1, further comprising means for acquiring a manually input destination point instead of the means for estimating the destination point. The means for estimating a destination point includes:
A means for detecting an object from the video;
means for estimating a moving direction of a person based on the image;
3. The motion prediction device according to claim 1, wherein the position of the object detected in the moving direction is estimated as a destination point. The means for estimating the destination point is notified of a category of the object in advance;
The motion prediction device according to claim 4 , wherein the means for detecting an object detects an object belonging to the category. The means for outputting the motion analysis result identifies a person's motion and outputs a motion label of the identification result;
The action prediction device according to claim 4 , wherein the means for estimating the destination point detects an object corresponding to the action label. A motion prediction method in which a computer predicts a subsequent motion of a person based on a video of the person performing the motion, comprising:
Analyzing a video of a person in motion to estimate each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result ;
Estimating a destination point of the person's movement based on the movement analysis result ;
Extracting a latent expression of a subsequent motion of the person based on the motion analysis result and the destination point;
A motion prediction method, comprising: predicting a subsequent motion of the person based on the motion analysis result , a destination point, and a latent expression. A motion prediction program for predicting a subsequent motion based on a video of a person performing a motion,
a step of analyzing a video of a person in motion , estimating each of the person's current postures, and outputting a sequence of the estimated postures as a motion analysis result ;
estimating a destination point of the person's movement based on the movement analysis result ;
extracting a latent expression of a subsequent motion of the person based on the motion analysis result and the destination point;
and predicting the person's subsequent actions based on the results of the action analysis , the destination point, and the latent expression.

Images