JP7648501B2 - Posture estimation device, method and program - Google Patents

Description

The present invention relates to a posture estimation device, method, and program for estimating the posture of an object captured in an image.

Patent Document 1 discloses a method for estimating the orientation of an object captured in an image. In Patent Document 1, a first orientation estimation is performed from an image of the object captured by a first image capture unit, and a second orientation estimation is performed using this result and an image captured by a second image capture unit, and the result is output. As a result, it may be possible to use the second orientation estimation to interpolate points where the orientation could not be estimated using the first orientation estimation due to the effects of occlusion, etc.

Non-Patent Document 1 discloses a technology in which a first pose estimation is performed from an image, and by combining the image features obtained there, a feature amount is generated that also takes into account the positional relationship of the entire pose, and a second pose estimation is performed using this feature amount and the result is output. As a result, it is possible to interpolate and correct to a natural shape by the second pose estimation for parts where the pose could not be estimated by the first pose estimation due to the influence of occlusion, etc.

Non-Patent Document 2 discloses a technique in which a first pose estimation is performed from an image by some means, and the result is corrected to a more natural form in a second pose estimation unit. For the second pose estimation, a Convolutional Neural Network (CNN) that has learned the distribution of errors in general pose estimation is used.

Although not related to pose estimation, Non-Patent Document 3 discloses a technique for improving the correction performance of a task of correcting an original image with defects into a natural image without defects by replacing each convolutional layer (hereinafter referred to as Conv layer) in a CNN with a Gated Conv layer that has the function of filtering information derived from defective pixels.

JP 2017-097577 A

Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun, "Cascaded pyramid network for multi-person pose estimation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 7103-7112. Moon, Gyeongsik, Ju Yong Chang, and Kyoung Mu Lee. "Posefix: Model-agnostic general human pose refinement network." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019. Yu, Jiahui, et al. "Free-form image inpainting with gated convolution." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.

In Patent Document 1 and Non-Patent Documents 1 and 2, the second attitude estimation is performed based on the results of the first attitude estimation, so if the results of the first attitude estimation contain many errors, it is difficult to improve the results by the second attitude estimation. In some cases, the accuracy may be reduced by the second attitude estimation due to the strong influence of the erroneous results.

On the other hand, in order to avoid outputting erroneous results in the first pose estimation, restrictions are placed on the shooting conditions, such as the subject's position and background. Non-Patent Document 3 does not take into account its application to pose estimation technology.

The object of the present invention is to solve the above technical problems and provide a posture estimation device, method, and program that determines the certainty of each part of the posture for the output of the first posture estimation, explicitly determines which parts of the information are uncertain or missing, and selects information to refer to when making corrections to perform a second posture estimation.

To achieve the above object, the present invention is characterized in that a posture estimation device that estimates the posture of an object captured in a camera image has the following configuration.

(1) The system includes a means for estimating a first posture of a posture estimation target captured in a camera image, a means for acquiring a degree of certainty for each part of the first posture, and a means for estimating a second posture based on the first posture and its degree of certainty, and the means for estimating the second posture selectively corrects a part of the first posture for which the degree of certainty is relatively low, based on a part for which the degree of certainty is relatively high.

(2) The means for estimating the second posture includes a means for selecting a part of the first posture to be corrected based on the degree of certainty, and a means for correcting the part to be corrected.

(3) The means for estimating the second posture is a convolutional neural network, and the convolutional layer does not perform convolutions on the parts to be corrected.

(4) The convolutional layer is a gated convolutional layer, and the gated convolutional layer includes a means for applying an activation function to each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level to output a correction target, and a means for normalizing each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level by the confidence level to output a confidence level, and further convolution of the feature quantity selected from the correction target based on the confidence level is repeated.

(5) The means for estimating the second orientation estimates the second orientation at time t based on the first orientations at time t and at least one of the times before and after t and their certainties.

(1) Even in difficult situations that can occur when the shooting conditions are not restricted, such as when part of the target for pose estimation is not captured in the image or when the background contains an image similar to the target for pose estimation, the adverse effect of an erroneous estimation result of the first pose on the estimation of the second pose can be reduced, making it possible to realize pose estimation that is more robust and capable of stable operation without requiring restrictions on the shooting conditions.

(2) The incorrectly estimated parts of the first posture are determined based on the confidence level obtained during posture estimation, making it possible to quantitatively and accurately grasp the incorrectly estimated parts.

(3) By replacing the convolutional layer of the CNN model used to estimate the second pose with a gated convolutional layer, the adverse effect of an erroneous estimation result of the first pose on the estimation of the second pose can be reduced, thereby improving the accuracy of estimating the second pose using the CNN estimation model.

(4) The second orientation is estimated based on not only the first orientation at time t but also the first orientation at multiple times before and after time t, which enables more accurate orientation estimation.

1 is a functional block diagram of a posture estimation device according to a first embodiment of the present invention. FIG. 13 illustrates an example of pose estimation. FIG. 11 is a block diagram showing an example in which a confidence level is calculated separately and used for pose estimation. FIG. 13 is a diagram showing an example of estimating a second pose by applying a first pose and its confidence factor to a CNN. FIG. 1 is a block diagram showing an example of a Gated Conv layer that can replace the Conv layer of a CNN. FIG. 11 is a block diagram showing a second embodiment of the present invention. FIG. 11 is a block diagram showing a modified example of the second embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. FIG. 1 is a functional block diagram showing the configuration of the main parts of a posture estimation device 1 according to a first embodiment of the present invention, and the main components include an image acquisition unit 10, a target region extraction unit 20, a first posture estimation unit 30, a confidence factor acquisition unit 40, a second posture estimation unit 50, and a posture output unit 60.

Such a posture estimation device 1 can be configured by implementing applications (programs) that realize each function on at least one general-purpose computer or server equipped with a CPU, ROM, RAM, bus, interface, etc. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

The image acquisition unit 10 acquires a camera image (e.g., an RGB image) capturing a posture estimation target. In this embodiment, an example will be described in which the posture estimation target is a person. The target area extraction unit 20 extracts a person area from the camera image as the image area of the posture estimation target. The person area may be extracted as a rectangular area circumscribing the person, such as a bounding box, as in Non-Patent Document 1, or as a silhouette following the contours of the person.

The first posture estimation unit 30 estimates the first posture of the person from the image of the extracted person area. In this embodiment, as shown in an example in FIG. 2, feature points related to the structure of the person are extracted from the person area, key points such as the face, hands, torso, and feet and their positions are estimated, and a collection of key points is output as an estimation result of the first posture. In this embodiment, it is assumed that the 23 key points shown in FIG. 2 and their position coordinates are estimated.

The keypoints whose positions are to be estimated may be only those that are visible in the camera image, or the positions of all types of keypoints may be estimated, including those that are not visible. The positions of keypoints may be represented by two-dimensional coordinates on the image, or may be represented using other appropriate coordinate systems such as a three-dimensional coordinate system. When representing the positions in three-dimensional coordinates, images from multiple viewpoints are acquired and the three-dimensional position of each keypoint is calculated by triangulation.

The certainty acquisition unit 40 acquires the likelihood of the estimation result for each keypoint of the first posture as a certainty. As the certainty, the certainty of the estimation result of each keypoint calculated from image features at the same time as posture estimation in Non-Patent Document 1 can be used. The certainty of each keypoint may be represented as a score that takes a value between 0 and 1, or may be represented as a truth value of 0 or 1 by setting a threshold value.

If a confidence level cannot be obtained in the first posture estimation process in the first posture estimation unit 30, or if a confidence level from a different viewpoint is desired even if a confidence level can be obtained, a memory unit 70 may be provided as shown in FIG. 3 to store estimation results at a different point in time, and the confidence level may be calculated based on whether or not the position transition is appropriate taking into account physical constraints, etc. Alternatively, reference information such as a person's body model may be stored, and the confidence level may be calculated based on whether or not the positional relationship of each key point satisfies constraint conditions specific to the person, etc.

The second posture estimation unit 50 includes a correction part selection unit 501 and a selective correction unit 502, and estimates a second posture of the person based on the first posture estimated by the first posture estimation unit 30 and its confidence level. The second posture is output from the posture output unit 60 as a result of posture estimation.

The correction part selection unit 501 selects parts to be corrected from each part of the first posture based on the confidence level. For example, if the first posture is estimated by a set of position coordinates of each key point as a main part, a method can be considered in which the confidence level is used as a mask and each key point of the first posture is multiplied by the mask value. The selective correction unit 502 estimates the second posture by correcting parts of the first posture with a relatively low confidence level as correction targets and parts with a relatively high confidence level using reference information.

With this configuration, uncertain parts of the first posture can be treated explicitly, so that the influence of uncertain parts on the estimation result in the second posture estimation can be suppressed. For example, it becomes possible to prevent erroneous correction of certain parts and to prevent reference to information on uncertain parts when correcting uncertain parts.

Figure 4 shows an example of the second posture estimation unit 50 configured based on an encoder-decoder (autoencoder) type CNN model, where the Conv layer is replaced with a gated convolution layer (Gated Conv layer) in this embodiment.

The encoder receives a heat map showing the image positions of all keypoints and a mask generated for each keypoint using the confidence level as prior knowledge. The features compressed by the encoder are expanded by upsampling in the decoder. In this embodiment, by replacing the Conv layer in the CNN model with a Gated Conv layer, keypoints in the first pose with relatively low confidence levels can be corrected by features obtained from keypoints with relatively high confidence levels.

Figure 5 is a block diagram showing an example of the configuration of a Gated Conv layer used in this embodiment. In the first stage, for each Conv layer in the CNN model, the masked heat map of each keypoint is used as input and the convolution layer extracts intermediate features. In the correction target system R1, an activation function (ReLU: Rectified Linear Unit)/ramp function is applied to the intermediate features extracted by the convolution layer. With "0" as the base point, if the input value is 0 or less, "0" is output, and if it is greater than 0, the same value as the input value is output, both as the features to be corrected.

In the reference information system R2, a softmax function is applied to the intermediate features extracted by the Conv layer to calculate the reliability (0-1) for each element of the feature, and this is output as reference information. Each Conv layer functions as the selective correction unit 502. The multiplication unit 503 functions as the correction part selection unit 501, and multiplies the feature to be corrected by the reference information (reliability) to select a feature with a relatively high reliability, which is then input as new input information to the next Gated Conv layer.

This configuration makes it possible to suppress the propagation of information from areas where the estimation results are uncertain in each layer of the CNN model, making it possible to make inferences using only strictly certain information.

Figure 6 is a functional block diagram showing the configuration of a second embodiment of the present invention. In the first embodiment, the second orientation at time t is estimated based on the first orientation at time t and its certainty factor. In contrast, in the second embodiment, the second orientation at time t is estimated by processing the first orientation at multiple times before and after time t and its certainty factor multiple times in a one-dimensional (1D) Conv layer in the time dimension.

In this embodiment, all Conv layers are replaced with Gated Conv layers, and for each time t for which the second posture is to be estimated and the times before and after it, such as t-2, t-1, t+1, t+2, etc., a vector (23 x 4 = 92 dimensions) is calculated whose elements are the position information (x, y, z) of each part of the first posture (23 keypoints in this embodiment) and its confidence level (k). Then, the 92-dimensional vectors calculated for each time are provided as input matrices to the Gated Conv layers to estimate the second posture at time t.

If you want to provide the first poses estimated at more time instants and their confidence vectors as input matrices, it is desirable to use multiple Gated Convolution layers, as shown in Figure 7.

In addition, in this embodiment, the first postures estimated at multiple times before and after the time t at which the posture is to be estimated and their certainty factors are used to estimate the second posture, but the present invention is not limited to this, and the second posture may be estimated based on multiple first postures estimated at time t and at least one other time before and after t and their certainty factors.

According to this embodiment, the second posture at time t is estimated based not only on the first posture at time t and its certainty factor, but also on the first posture at multiple times before and after time t and its certainty factor, enabling more accurate posture estimation.

Furthermore, according to each of the above embodiments, it becomes possible to provide highly accurate pose estimation at low cost, so that a variety of services can be provided to many people regardless of geographic or economic disparities. As a result, it becomes possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the Sustainable Development Goals (SDGs) led by the United Nations.

1... Posture estimation device, 10... Image acquisition unit, 20... Target area extraction unit, 30... First posture estimation unit, 40... Confidence factor acquisition unit, 50... Second posture estimation unit, 60... Posture output unit, 70... Memory unit, 501... Correction area selection unit, 502... Selective correction unit, 503... Multiplication unit

Claims (9)

In a posture estimation device that estimates the posture of an object captured in a camera image,
A means for estimating a first orientation of a posture estimation target captured in a camera image;
A means for acquiring a confidence level for each part of the first posture;
means for estimating a second attitude based on the first attitude and a confidence factor thereof;
The means for estimating the second attitude
a means for selecting a portion to be corrected in the first posture based on the degree of certainty;
and a means for correcting the correction target portion,
The means for correcting is an autoencoder type convolutional neural network including an encoder and a decoder ,
A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty.
In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount;
The decoder is configured to upsample the feature amount to estimate each part including the part to be corrected . The convolution layer is a gated convolution layer,
The gated convolutional layer comprises:
a means for applying an activation function to each element of an intermediate feature after convolution of the first orientation masked by the confidence level, and outputting a correction target;
and a means for normalizing each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level by the confidence level, and outputting a confidence level;
The posture estimation apparatus according to claim 1 , further comprising: repeating convolution of the feature quantity selected from the correction target based on the reliability. The posture estimation device according to claim 1 or 2, characterized in that the means for estimating the second posture estimates the second posture at time t based on each of the first postures at time t and at least one of the times before and after t and their confidence levels. A posture estimation method in which a computer estimates the posture of an object captured in a camera image, comprising:
A first posture of the posture estimation target captured in the camera image is estimated,
Obtain confidence for each part of the first posture,
Estimating a second pose based on the first pose and the confidence level thereof;
In estimating the second attitude,
selecting a portion to be corrected in the first posture based on the degree of certainty;
The correction target portion is corrected by an autoencoder type convolutional neural network including an encoder and a decoder ;
A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty.
In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount;
The posture estimation method , wherein the decoder estimates each part including the part to be corrected by upsampling the feature amount . The convolution layer is a gated convolution layer,
The gated convolutional layer comprises:
applying an activation function to each element of the intermediate feature after convolution of the first posture masked by the confidence level to output a correction target;
Normalizing each element of the intermediate feature after convolution of the first posture masked by the confidence level by the confidence level to output a confidence level;
The posture estimation method according to claim 4 , further comprising repeating convolution of the feature quantity selected from the correction target based on the reliability. In estimating the second attitude,
6. The method according to claim 4, further comprising estimating the second orientation at time t based on the first orientations at time t and at least one of the times before and after t and their certainties. In a posture estimation program that estimates the posture of an object captured in a camera image,
A step of estimating a first posture of a posture estimation target captured in a camera image;
A procedure for obtaining confidence levels for each part of the first posture;
and estimating a second pose based on the first pose and a certainty factor thereof;
In the step of estimating the second attitude,
selecting a portion to be corrected in the first posture based on the degree of certainty;
and correcting the correction target portion using an autoencoder type convolutional neural network including an encoder and a decoder ;
A portion of the first posture having a relatively low certainty is selectively corrected based on a portion of the first posture having a relatively high certainty.
In the encoder, a convolution layer convolves a portion excluding the portion to be corrected to extract a feature amount;
The decoder upsamples the feature amount to estimate each part including the part to be corrected . The convolution layer is a gated convolution layer,
The gated convolutional layer comprises:
applying an activation function to each element of the intermediate feature after convolution of the first posture masked by the confidence level to output a correction target;
and normalizing each element of the intermediate feature quantity after convolution of the first posture masked by the confidence level by the confidence level to output a confidence level;
8. The posture estimation program according to claim 7, further comprising repeating convolution of the feature quantity selected from the correction target based on the reliability. The posture estimation program according to claim 7 or 8, characterized in that in the step of estimating the second posture, the second posture at time t is estimated based on each of the first postures at time t and at least one of the times before and after t, and their certainties.

Images