JP7789798B2 - Multi-view Neural Human Prediction with an Implicit Differentiable Renderer for Facial Expression, Body Pose Shape, and Clothing Performance Capture - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
This application is related to the publication entitled "Multiview Neural Human Prediction Using Implicit Differentiable Renderer for Facial Expression, Body Pose Shape, and Clothing Performance Capture," filed on November 16, 2021. No. 63/279,916, filed March 31, 2021, entitled "Multiview Neural Human Prediction Using Implicit Differentiable Renderer for Facial Expression, Body Pose Shape, and Clothing Displacement," and U.S. Provisional Patent Application Serial No. 63/279,916, filed March 31, 2021, entitled "Multiview Neural Human Prediction Using Implicit Differentiable Renderer for Facial Expression, Body Pose Shape, and Clothing Displacement." This application claims the benefit of priority under 35 U.S.C. §119 of U.S. Provisional Patent Application Serial No. 63/168,467, entitled "DISPLACEMENT," both of which are incorporated herein by reference in their entireties for all purposes.

The present invention relates to 3D computer vision and graphics for the entertainment industry. Specifically, the present invention relates to acquiring and processing 3D computer vision and graphics for film, TV, music, and gaming content production.

Conventional systems, such as Facebook FrankMocap, only predict the shape and pose of a nude body from a single image. They cannot predict clothing surfaces. These systems are 2D image transformation methods and cannot handle multi-view input.

Implicit part networks predict both the body and clothes from scanned or reconstructed point clouds, but they require 3D scans and cannot handle RGB images as input, nor facial expression and appearance. Implicit part networks also explicitly fit a human prior model after predicting only labels that identify voxels as body or clothes, making them slow. Neural Body and Animatable NeRF use neural radiance fields (NeRFs) to predict clothes human bodies without facial expression. However, these require the creation of dense latent code volumes, which are limited to low resolution, resulting in coarse body shapes. Furthermore, these methods can only reconstruct volumetric human models that do not include mesh vertex correspondences.

Multiview neural human prediction involves predicting a 3D human model, including bone structure, body shape, and clothing displacement and appearance, from a set of multiview images given camera calibration.

In one aspect, a neural network receives an input image set, which can be a single image or multiple images from different views, and predicts a layered 3D human model. The image set contains a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel. The camera information for the image set is known. The output model contains three layers, from inside to outside: a skeleton with a predicted pose; a naked 3D body with a predicted shape including facial expression (e.g., an SMPL-X model parameterized by blendshapes and joint rotations); and a 3D field of clothing displacement and appearance RGB color inferred from the input images. A clothed body mesh is obtained by deforming a naked 3D body mesh according to the clothes displacement field.

In another aspect, the neural network is composed of three sub-networks: a multi-view stereo 3D convolutional neural network (MVS-3DCNN) that encodes the input image set into features; a human mesh recovery multilayer perceptron (HMR MLP) that regresses the features onto human parameters; and a neural radiance field multilayer perceptron (NeRF MLP) that fine-tunes the MVS-3DCNN to decode query 3D rays (3D position and orientation) into RGB color and clothing-body displacement.

In another aspect, in test/inference mode, prediction of a layered 3D human model is device-independent, fully automatic, and real-time for a small input set within the view range of the cameras in the training data, without explicit numerical optimization. When predicting using a trained neural network, the MVS-3DCNN receives a multi-view image set as input, selects a frontal view as the reference view, and extracts features. The HMR MLP regresses all features onto human pose, shape, and facial expression parameters. The SMPL-X model generates a human nude body mesh according to the parameters. The nude body mesh is then converted into an occupancy field within a bounding box. The trained NeRF MLP generates RGB colors and 3D displacement vectors representing the nude body surface for any 3D point near the body mesh associated with the ray direction from each center of view. By querying all rays cast from all pixels in a camera view (either the same view as the input view or any new view), the appearance of a clothed human body can be rendered as an RGB image. By deforming the nude body using 3D displacement vectors from the sampled points, a clothed human body mesh such as SMPL-X+D can be obtained with the same vertex correspondence as the SMPL-X model.

In another aspect, neural network training includes two cases: supervised and self-supervised. In the supervised case, a labeled dataset with known human parameters, such as the H36M dataset, is given. The ground truth (GT) parameters and shape are compared with the CNN-regressed parameters and shape. The difference is calculated as the shape loss. On the other hand, rays are cast from sampled pixels in the input image set, and NeRF MLP renders the rays to regress the parameters to color and density, which are functions of nude body density and 3D clothing displacement. The color loss is calculated by summing the difference between the sampled pixel color and the rendered color. On the other hand, for existing datasets where GT human parameters are unknown, such as motion capture datasets, self-supervised/self-improving training is used. In each training iteration, parameters are regressed from the MVS 3DCNN, then fed into an optimization-based human predictive algorithm such as SMPLifyX and optimized using explicit numerical optimization approaches. The optimized parameters are compared to the CNN regressed parameters to obtain the shape loss. The remaining steps are the same as supervised training, but self-improvement training requires more epochs and longer time than supervised training. Overall neural network training is performed using a parallel optimization algorithm such as Adam, which minimizes both the shape loss and color loss, and the optimized network weights are output.

FIG. 1 illustrates a flowchart for neural human prediction according to some embodiments. FIG. 1 illustrates the workflow of forward prediction represented by tensor notation, where the weights of all networks MVS 3DCNN, HMR MLP and NeRF MLP are known, according to some embodiments. FIG. 1 illustrates a workflow for training a network using supervision, according to some embodiments. FIG. 1 illustrates a workflow for training a network in a self-improvement strategy, according to some embodiments. FIG. 10 illustrates the alignment of MVS 3DCNN for each view to NeRF MLP according to some embodiments.

Neural human prediction involves predicting a 3D human model, including skeletal pose, body shape, and clothing displacement and appearance, from a set of images (single or multi-view images). An embodiment of neural human prediction describes the use of neural networks. Multi-view neural human prediction surpasses single-image-based motion capture (mocap) and human lifting in quality and robustness, simplifies the architecture of body-clothes prediction networks such as implicit part networks, which take sparse point clouds as input and run slowly, and avoids the resolution limitations of latent code-based networks such as Neural Body, which encode the entire 3D volume.

Figure 1 is a flowchart of neural human prediction according to some embodiments. In step 100, an input image set I, a single image, or multi-view images, such as a set of photographs taken around a subject, is obtained as input. The input I is represented as a 4D tensor of size Nxwxhxc, where N is the number of views, and w, h, and c are the image width, image height, and image channel, respectively. The camera has already been calibrated, so all camera information (e.g., camera parameters) is known. For image preprocessing, a bounding box of the subject and a foreground mask are extracted using existing techniques such as Detectron2 and Image Grab-Cut. The image is cropped by the bounding box and zoomed to a size of wxh with the same aspect ratio. The image border is filled with black.

The neural network (MVS-PERF) 102 consists of three components: a multi-view stereo 3D convolutional neural network (MVS-3DCNN) 104 that encodes the input image set into features; a human mesh reconstruction multi-layer perceptron (HMR MLP) 106 that regresses the features onto human parameters; and a neural luminance field multi-layer perceptron (NeRF MLP) 108 that fine-tunes the MVS-3DCNN to decode query 3D rays (3D position and orientation) into RGB color and clothing-body displacement.

In step 104, a deep 2D CNN extracts image features from each view. Each convolutional layer, except the last, is followed by a batch normalization (BN) layer and a rectified linear unit (ReLU). Two downsampling layers are also deployed. The output of the 2D CNN is a feature map of size w/4 x h/4 x 32.

Then, a view is selected as the reference view, and its view frustum is set to cover the entire workspace of the subject according to the perspective projection and near-far planes. This frustum is sampled from near to far with d depth planes parallel to both the near and far planes. All feature maps are transformed to each depth plane and blended. For any view i, i = 1, 2, ..., N, the 3x3 homography image warping matrix for the reference view (indexed from 1) is given by the following formula:

Here, K, [R, t] represent the intrinsic and extrinsic parameters of the camera, z is the distance from the depth plane to the camera center of the reference view, and n is the normal direction to the depth plane.

After all images are warped to the depth plane, the variance of all features
The cost at coordinates (u, v, z) is determined by:
is the average feature value of all views.
The size of the cost volume is d×w/4×h/4.

In step 106, a human mesh reconstruction multi-layer perceptron (HMR MLP) contains three layers of linear regression separated by flatten and dropout layers. The HMR MLP regresses the features from the MVS 3DCNN onto the body parameters θ _reg 114.

The human body parameters θ _reg can manipulate a human body parametric model such as SMPL-X into a 3D nude body mesh 202. Typically, the SMPL-X representation θ _reg includes a skeleton pose (3D rotation angle of each joint), body blendshape parameters that control body shape such as height and weight, and facial blendshape parameters that control facial expression. _{θ reg} uses the blendshape parameters to construct a T-pose mesh, which is then deformed into a pose mesh by the skeleton pose of a linear skinning model.

Meanwhile, in step 108, the cost volume is sent to a differentiable rendering MLP, such as a neural luminance field (NeRF). The NeRF MLP is formulated as c(x, φ) = M(x, φ, f; Γ), where M is a function that maps a query ray, represented by a 3D position x and direction φ, to a four-channel color RGB σ. f is the feature map from the cost volume to the NeRF volume in the frustum MVS 3DCNN 104, Γ is the weight of the NeRF MLP network, and σ represents the occupancy density of the probability that a 3D point exists within the mesh. The occupancy density field σb for a nude body can be directly obtained by transforming the mesh 202 (Figure 2) of the frustum 104. The density field σ for a clothed body can be expressed as σ(D, f), a function of the 3D displacement vector field D and the feature map f. The 3D displacement vector field D116 represents how points on the clothed body surface 204 relate to points on the nude body surface. Training the NeRF MLP also optimizes the displacement vector field D.

2 illustrates a workflow for forward prediction represented by tensor notation, where the weights of all networks (MVS 3DCNN, HMR MLP, and NeRF MLP) are trained and fixed, according to some embodiments. An appearance image 112 is rendered by querying all rays 200 of pixels from a perspective projection image. In some embodiments, 3D human prediction 110 is implemented. A displacement field D116 is obtained by querying sampled points near the human body. For human performance capture tasks, where the clothed output mesh has the same topology as the template, a nude body mesh _Vb 202 can be deformed into a clothed body mesh _Vc 204 by adding an interpolated displacement vector to each vertex.

3 illustrates a workflow for training a network with supervision, according to some embodiments. A supervised training dataset, such as Human3.6M, includes not only image inputs I100 but also ground truth human parameters θ _gt 300 and a nude body mesh V _b,gt 302, typically acquired by sensors or existing techniques. In this case, the shape loss 304 is obtained directly by summing the differences between the predicted nude body and the ground truth.

where J is the joint of the bare body and Π represents the perspective projection of the 3D points in each camera view. To train the network effectively, in each training step, all views are selected in turn as reference views for the MVS 3DCNN.

Meanwhile, rays 306 are sampled from the input image set 100 using a non-uniform sampling strategy that is typically proportional to image saliency. More rays are sampled in high-saliency regions and fewer rays are sampled from flat or background regions. These rays, along with the feature map from the MVS 3DCNN 104, are sent to the NeRF MLP 106, which renders the sample's appearance RGBσ color 308. The color loss 310 is calculated by summing all the differences between the sampled colors in the input image and the rendered color 308.

We apply a parallelized stochastic optimization algorithm such as Adam to train the weights of all networks (MVS 3DCNN, HMR MLP, NeRF MLP) by minimizing both shape and color losses.

4 illustrates a workflow for training a network in a self-improvement strategy, according to some embodiments. In this case, the training dataset provides only human images without annotations or human ground truth parameters. For each image in the input set 100, we apply an optimization-based prediction 400, such as the SMPLifyX algorithm, by selecting the regressed parameters θ _reg 114 as an initial guess. The optimization-based prediction first detects 2D human keypoints on each image and applies nonlinear optimization to fit a 3D human.
A mesh V _b,opt 404 (parameterized by θ _opt 402 ) is applied to these 2D keypoints.

Here, K denotes the detected 2D position of the keypoint, and the sum is over all corresponding keypoints and all views.

Nonlinear least-squares optimization is numerically slow, and the fitting accuracy depends on the initial guess _θreg , but it is highly reliable. After enough fitting iterations, _θopt approaches the ground truth. Therefore, a self-improvement training workflow can efficiently improve _θopt toward the ground truth, as summarized below.
Self-improvement training workflow:
Do the following: Calculate θ _reg from MVS-3DCNN and calculate HMR MLP from input I Calculate θ _opt from SMPLifyX using θ _reg as the initial guess and I as input Sample rays from I and calculate sampled color c from NeRF MLP Calculate ShapeLoss and ColorLoss Update the network weights of MVS 3DCNN, HMR MLP, and NeRF MLP by minimizing ShapeLoss and ColorLoss Iterate on all training data until the weights converge

Figure 5 shows the alignment of MVS 3DCNN to NeRF MLP for each view, according to some embodiments.

In operation, neural human prediction can be directly applied in commercial and/or personal markerless performance capture applications, such as markerless motion capture in game studios or human 3D surface reconstruction RGB camera setups. Other applications of multi-view neural human prediction embodiments can combine, for example, depth sensing input, 3D modeling, or using the output to create novel animations as a real-time backbone technology that can be combined with any extensions. Multi-view neural human prediction can also be applied in gaming applications, VR/AR applications, and any real-time human interaction applications. Depending on the hardware used (e.g., GPU processor speed and GPU memory size), multi-view neural human prediction can be real-time when processing a small number of views for prediction, or can implement near-real-time processing and prediction for a larger number of views (e.g., 20).

The methods described herein may be implemented on any computing device. Examples of suitable computing devices include a personal computer, a laptop computer, a computer workstation, a server, a mainframe computer, a handheld computer, a personal digital assistant, a cellular/mobile phone, a smart appliance, a gaming console, a digital camera, a digital camcorder, a camera phone, a smartphone, a portable music player, a tablet computer, a mobile device, a video player, a video disc writer/player (such as a DVD writer/player, a high-definition disc writer/player, or an ultra-high-definition disc writer/player), a television, a home entertainment system, an augmented reality device, a virtual reality device, smart jewelry (e.g., a smart watch), a vehicle (e.g., an autonomous vehicle), or any other suitable computing device.

Some embodiments of multi-view neural human prediction using an implicit differentiable renderer for facial expression, body pose shape, and clothing performance capture: 1. A method programmed into a non-transient device, comprising:
Taking as input a set of images;
Processing the image set using a neural network;
and the processing comprises:
encoding the set of images into one or more features;
Regressing features onto human parameters;
Fine-tuning the neural network and
Decoding the query 3D ray into RGB color and clothes-body displacement based on the image set;
A method comprising:

2. The method of clause 1, wherein the image set comprises a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel.

3. The method of clause 1, in which the neural network selects a frontal view from the image set as a reference view and extracts features.

4. The neural network regresses all features onto human pose, shape, and facial expression parameters, a method described in clause 3.

5. The method of clause 4, wherein the neural network generates a nude human body mesh according to the parameters.

6. The nude mesh is converted to an occupancy field within a bounding box, as in clause 5.

7. The method of clause 6, wherein the neural network generates an RGB color and a 3D displacement vector representing the surface of the nude body for any 3D point near the body mesh associated with a ray direction from each center of view.

8. The method of clause 7, wherein the appearance of the clothed human body is rendered as an RGB image by querying all rays cast from all pixels in the camera view, and the clothed body mesh is obtained by deforming the nude body using 3D displacement vectors from the sampled points.

9. The method of clause 1, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

10. A non-transitory memory configured to store an application;
a processor configured to process the application;
an application comprising:
It takes a set of images as input,
and processing the set of images using a neural network, the processing comprising:
encoding the set of images into one or more features;
Regressing features onto human parameters;
Fine-tuning the neural network and
Decoding the query 3D ray into RGB color and clothes-body displacement based on the image set;
1. An apparatus comprising:

11. The apparatus of clause 10, wherein the image set comprises a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel.

12. The apparatus of clause 10, wherein the neural network selects a frontal view from the image set as a reference view and extracts features.

13. The neural network is the device of clause 12 that regresses all features onto human pose, shape, and facial expression parameters.

14. The apparatus of clause 13, wherein the neural network generates a nude human body mesh according to the parameters.

15. The apparatus of clause 14, wherein the nude mesh is converted into an occupancy field within a bounding box.

16. The apparatus of clause 15, wherein the neural network generates an RGB color and a 3D displacement vector representing the surface of the nude body for any 3D point near the body mesh associated with a ray direction from each center of view.

17. The apparatus of clause 16, wherein the appearance of the clothed human body is rendered as an RGB image by querying all rays cast from all pixels in the camera view, and a clothed body mesh is obtained by deforming the nude body using 3D displacement vectors from the sampled points.

18. The apparatus of clause 10, wherein the neural network is implemented in a supervised mode or a self-supervised mode.

19. A non-transitory memory configured to store an application;
a processor configured to process the application;
an application comprising:
a multi-view stereo 3D convolutional neural network (MVS-3DCNN) configured to encode an input image into features;
a human mesh reconstruction multi-layer perceptron (HMR MLP) configured to regress features onto human parameters;
A Neural Luminance Field Multilayer Perceptron (NeRF MLP) configured to fine-tune the MVS-3DCNN and decode the query 3D ray (3D position and orientation) into RGB color and clothes-body displacement;
1. An apparatus comprising:

20. The apparatus of clause 19, wherein the image set comprises a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel.

21. The MVS-3DCNN is the apparatus of clause 20, which selects a frontal view from the image set as a reference view and extracts features.

22. HMR MLP is the device of Clause 21 that regresses all features onto human pose, shape, and facial expression parameters.

23. The apparatus of clause 22, further comprising a model configured to generate a nude human body mesh according to the parameters.

24. The apparatus of clause 23, wherein the nude mesh is converted into an occupancy field within a bounding box.

25. The apparatus of clause 24, wherein the NeRF MLP generates an RGB color and a 3D displacement vector representing the surface of the nude body for any 3D point near the body mesh associated with a ray direction from each center of view.

26. The apparatus of clause 25, wherein the appearance of a clothed human body is rendered as an RGB image by querying all rays cast from all pixels in the camera view, and a clothed body mesh is obtained by deforming the nude body using 3D displacement vectors from the sampled points.

The present invention has been described in terms of specific embodiments containing details to facilitate an understanding of the principles of construction and operation of the invention. Reference herein to such specific embodiments and details of these embodiments is not intended to limit the scope of the claims appended hereto. It will be readily apparent to those skilled in the art that various other modifications can be made in the embodiments chosen for illustration without departing from the spirit and scope of the invention as defined by the claims.

100 Image Input I
102 Neural Network (MVS-PERF)
104 Multi-view stereo 3D convolutional neural network (MVS-3DCNN)
106 Human Mesh Reconstruction Multilayer Perceptron (HMR MLP)
108 Neural Luminance Field Multilayer Perceptron (NeRF MLP)
110 3D human prediction 112 Appearance image 114 Human body parameters θ _reg
116 Three-dimensional displacement vector field D

Claims (17)

1. A method programmed into a non-transient device, comprising:
Taking as input a set of images;
processing the set of images using a neural network;
wherein the process comprises:
encoding the set of images into one or more features by inputting the set into a multi-view stereo 3D convolutional neural network (MVS-3DCNN) ;
inputting the features into a human mesh reconstruction multi-layer perceptron (HMR MLP) to regress the human body parameters, including parameters controlling the shape of the human body;
fine-tuning the neural network;
inputting a query 3D ray represented by a 3D position and orientation into a neural luminance field multi-layer perceptron (NeRF MLP) to decode the 3D ray into an RGB color based on the image set and a clothing-body displacement indicating the displacement of a point on the body surface after clothing relative to the point on the body surface before clothing is worn;
Including,
A method characterized by: The image set comprises a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel.
The method of claim 1. The neural network selects a front view from the image set as a reference view and extracts features.
The method of claim 1. The neural network regresses all features onto human pose, shape, and facial expression parameters.
The method of claim 3. the neural network generates a nude human body mesh according to the human body parameters;
The method of claim 4. The nude mesh is converted into an occupancy field within a bounding box.
The method of claim 5. the neural network generates, for any 3D point near the body mesh associated with a ray direction from each center of view, the RGB color and a 3D displacement vector representing the surface of the nude body;
The method of claim 6. The appearance of the clothed human body is rendered as an RGB image by querying all rays cast from all pixels in the camera view, and the clothed body mesh is obtained by deforming the nude body using the 3D displacement vectors from the sampled points.
The method of claim 7. The neural network is implemented in a supervised or self-supervised mode.
The method of claim 1. a non-transitory memory configured to store an application;
a processor configured to process the application;
and wherein the application comprises:
a multi-view stereo 3D convolutional neural network (MVS-3DCNN) configured to encode an input image into features;
a human mesh reconstruction multi-layer perceptron (HMR MLP) configured to regress the features to human body parameters, including parameters controlling the shape of the human body;
a Neural Luminance Field Multilayer Perceptron (NeRF MLP) configured to fine-tune the MVS-3DCNN and decode a query 3D ray (3D position and orientation) represented by a 3D position and orientation into an RGB color and a clothes-body displacement indicating the displacement of a point on the body surface after wearing the clothes relative to the point on the body surface before wearing the clothes;
Including,
An apparatus characterized in that The input image comprises a 4D tensor of size Nxwxhxc, where N is the number of views, w is the image width, h is the image height, and c is the image channel.
11. The apparatus of claim 10 . The MVS-3DCNN selects a front view from the input image as a reference view and extracts features.
12. The apparatus of claim 11 . The HMR MLP regresses all features onto human pose, shape, and facial expression parameters.
13. The apparatus of claim 12 . further comprising a model configured to generate a nude human body mesh according to the parameters;
14. The apparatus of claim 13 . The nude mesh is converted into an occupancy field within a bounding box.
15. The apparatus of claim 14 . The NeRF MLP generates, for any 3D point near the body mesh associated with a ray direction from each center of view, the RGB color and a 3D displacement vector representing the surface of the nude body.
16. The apparatus of claim 15 . The appearance of the clothed human body is rendered as an RGB image by querying all rays cast from all pixels in the camera view, and the clothed body mesh is obtained by deforming the nude body using the 3D displacement vectors from the sampled points.
17. The apparatus of claim 16 .