JP7643546B2 - Learning device, estimation device, their methods, and programs - Google Patents

Description

The present invention relates to an estimation technology for estimating facial key points and facial pose from a facial image, and a learning technology for parameters used when making the estimation.

Facial keypoints are plots of several to a dozen feature points on the human face, including eyebrows, eyes, nose, mouth, and contours. These features quantitatively capture facial expressions and facial movements, and estimating facial keypoints from facial images is important for understanding human states and inner thoughts. On the other hand, facial orientation is the rotation angle of the three axes (roll, pitch, yaw) of the face, and estimating facial orientation from facial images allows for analysis of facial movements and attention direction. Facial keypoints and facial orientation are generally estimated using neural networks. In conventional technology, facial keypoints and facial orientation are estimated independently using independent neural network models trained only with their respective training data (Non-Patent Documents 1 and 2). Correct labels for both facial keypoints and facial orientation are available as vector data. Facial keypoints are expressed as continuous data of the two-dimensional or three-dimensional coordinate values of each keypoint in the image. For example, any number of keypoints, such as 21 or 68 points, can be used. Facial orientation is continuous data of the rotation angle values of the roll, pitch, and yaw components of the rotation angle. Facial keypoints and facial direction can be estimated using a neural network by extracting image features using a convolutional layer and pooling layer, which are widely used in image recognition, and then regressing the facial keypoints and facial direction into vector data using a subsequent fully connected layer.

In non-patent documents 1 and 2, facial keypoints and facial pose are estimated using neural networks.

Zongping Deng, Ke Li, Qijun Zhao, Yi Zhang, Hu Chen, "Effective face landmark localization via single deep network", CoRR, 2017. Nataniel Ruiz, Eunji Chong, James M. Rehg, "Fine-Grained Head Pose Estimation Without Keypoints", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2018

In conventional technology, the model parameters used to estimate facial keypoints and the model parameters used to estimate facial direction are trained in independent neural networks using independent training data. However, since both facial keypoints and facial direction are features that capture facial movement, if model parameters can be trained taking into account the features of both, more universal facial features can be acquired, and highly accurate estimation of facial keypoints and facial direction can be expected. However, conventional technology has the problem that only local facial features can be captured because the model parameters used to estimate facial keypoints and the model parameters used to estimate facial direction are trained independently.

The present invention aims to provide an estimation device that can estimate facial keypoints and facial orientation that capture more universal facial features by learning model parameters used to estimate facial keypoints and facial orientation using a single neural network system, a learning device for the parameters used in the estimation, and methods and programs for carrying out the above mentioned estimation.

In order to solve the above problem, according to one aspect of the present invention, a learning device includes a shared network unit that converts a training face image ^Sb into an intermediate feature ^vb by a neural network function using a model parameter ^ _θk or ^ _θp , a face keypoint network unit that converts the intermediate feature ^vb into an estimated face keypoint vector ^Z'k , _b by a neural network function using the model parameter ^ _θk when the training face image ^Sb is composed of face keypoint training face images, a face keypoint model parameter optimization unit that updates the model parameter ^ _θk by using _the estimated face keypoint vector Z'k _,b ^and a face keypoint correct label ^Tk1 , ..., ^Tk _|M| for the training face image Sb, a face orientation network unit that converts the intermediate feature ^vb into an estimated face orientation vector Z'p _{,b by} a neural network function using the model parameter ^ _θp when _the training face ^image Sb is composed of face orientation training ^face images, _and The face direction model parameter optimization unit updates the face direction model parameter θ _k corresponding to the model parameter ^θ _k and obtains the learned model parameter θ _p corresponding to the model parameter ^θ _{p .}

The present invention has the effect of enabling estimation of facial keypoints and facial pose that capture more general facial features.

FIG. 1 is a diagram showing an example of the configuration of an estimation system according to a first embodiment. FIG. 2 is a functional block diagram of the learning device according to the first embodiment. FIG. 4 is a diagram showing an example of a processing flow of the learning device according to the first embodiment. FIG. 1 is a functional block diagram of an estimation device according to a first embodiment. FIG. 4 is a diagram showing an example of a processing flow of the estimation device according to the first embodiment. FIG. 13 is a diagram showing an example of the configuration of a computer to which the present technique is applied.

Below, an embodiment of the present invention will be described. In the drawings used in the following description, components having the same functions and steps performing the same processing are given the same reference numerals, and duplicated explanations will be omitted. In the following description, symbols such as "^" used in the text should be written directly above the character immediately preceding it, but due to limitations in text notation, they are written immediately after the character in question. In formulas, these symbols are written in their original positions. Furthermore, unless otherwise specified, processing performed on an element-by-element basis of a vector or matrix is assumed to apply to all elements of that vector or matrix.

<Key Points of the First Embodiment>
In this embodiment, both the learning data of the model parameters used to estimate the face keypoints and the learning data of the model parameters used to estimate the face direction are learned by a learning device that is a single neural network system. The face images for learning the model parameters used to estimate the face keypoints and the face images for learning the model parameters used to estimate the face direction are alternately input to a shared network section in the learning device, where more universal face features acquired from both the learning face images are learned. The face keypoint network section and the face direction network section are branched out at the rear of the shared network section, and the error of the estimated value is minimized using the respective correct answer labels, and the model parameters are updated. During estimation, the face keypoint estimation section and the face direction estimation section are used, respectively. The face keypoint estimation section estimates the face keypoints using the network architecture of the shared network section and the subsequent face keypoint network section, and the model parameters learned by the learning device. Similarly, the face direction estimation section estimates the face direction using the network architecture of the shared network section and the subsequent face direction network section, and the model parameters learned by the learning device.

First Embodiment
FIG. 1 shows an example of the configuration of an estimation system according to the first embodiment.

The estimation system includes a learning device 100 and an estimation device 200.

The learning device 100 and the estimation device 200 are special devices configured by loading a special program into a publicly known or dedicated computer having, for example, a central processing unit (CPU), a main storage device (RAM), etc. The learning device 100 and the estimation device 200 execute each process under the control of the central processing unit, for example. Data input to the learning device 100 and the estimation device 200 and data obtained in each process are stored in, for example, the main storage device, and the data stored in the main storage device is read out to the central processing unit as necessary and used for other processes. At least a part of each processing unit of the learning device 100 and the estimation device 200 may be configured by hardware such as an integrated circuit. Each storage unit provided in the learning device 100 and the estimation device 200 can be configured by, for example, a main storage device such as a RAM (Random Access Memory), or middleware such as a relational database or a key-value store. However, each storage unit does not necessarily need to be provided inside the learning device 100 and the estimation device 200, but may be configured as an auxiliary storage device made up of a hard disk, an optical disk, or a semiconductor memory element such as a flash memory, and may be configured to be provided outside the learning device 100 and the estimation device 200.

First, we will explain the learning device 100.

<Learning device 100>
The learning device 100 receives as input learning data K=(S ^k ₁ ,T ^k ₁ ),...,(S k |M| ,T _k |M| ) for learning model parameters θ _k used to estimate face keypoints (hereinafter also referred to as "face keypoint parameters θ _k ") and learning data P=(S ^p ₁ ,T ^p ₁ ),...,(S ^p _{|N| ,} T ^p _|N| ) for learning model parameters θ _p used to estimate face direction ⁽ hereinafter also referred to as "face direction parameters θ ^p _" ), learns the model parameters θ _k , θ _p using the learning data K, P, acquires and outputs the learned model parameters θ _k , θ _p . Here, M is the data size of the learning data K, and N is the data size of the learning data P. ^{S k} _m (m=1,...,M) are learning face images in the learning data for learning the face keypoint parameters θ _k . The training face image is, for example, an image in which only the face is cut out, has a resolution of 224x224 pixels, and has three RGB channels. ^{T k} _m (m=1,...,M) is a correct label in the training data for learning the face keypoint parameter θ _k (hereinafter, also referred to as a "face keypoint correct label"), and is, for example, vector data storing the coordinates of each point of the face keypoint, and has a data format such as [[X1,Y1],...,[X68,Y68]]. ^{S p} _n (n=1,...,N) is a training face image in the training data for learning the face direction parameter θ _p , and is, for example, an image in which only the face is cut out, similar to the training face image S ^k _m of the training data K, and has a resolution of, for example, 224x224 pixels, and has three RGB channels. T ^p _n (n = 1, ..., N) is a correct label in the training data for learning the face direction parameter θ _p (hereinafter also referred to as a "face direction correct label"), a vector that stores the rotation angles in three directions, roll, pitch, and yaw, of the face direction, and has a data format such as [roll, pitch, yaw].

Figure 2 shows a functional block diagram of the learning device 100, and Figure 3 shows its processing flow.

The learning device 100 includes a data selection unit 110, a shared network unit 120, a facial keypoint network unit 130, a facial keypoint model parameter optimization unit 140, a facial orientation network unit 150, and a facial orientation model parameter optimization unit 160.

Below, an overview of each part is provided.

The data selection unit 110 receives as input face images S ^k ₁ , ..., S ^k _|M| for training of face keypoint parameter θ _k (hereinafter also referred to as "face images for face keypoint training") and face images S _{p 1} , ..., S ^p | _N| for training of face direction parameter θ ^p (hereinafter also referred to as "face images for face direction training"), and selects training data to be used in one batch.

The shared network unit 120 takes a batch of training face images as input and obtains intermediate features using any neural network that outputs intermediate features.

The facial keypoint network unit 130 obtains estimates of facial keypoints using any neural network that takes intermediate features as input and outputs estimates of facial keypoints.

The face keypoint model parameter optimization unit 140 receives the estimated values of the face keypoints and the correct labels of the face keypoints as input, calculates the error in the estimated coordinates of the face keypoints, and updates the model parameters θ _k based on the error.

The face orientation network unit 150 obtains an estimate of face orientation using any neural network that takes intermediate features as input and outputs an estimate of face orientation.

The face direction model parameter optimization unit 160 receives the estimated face direction value and the correct label of the face direction as input, calculates the error in the estimated rotation angle of the face direction, and updates the model parameter θ _p based on the error.

The above process shows the learning procedure for one batch (a part of the training data that has been selected), and this can be repeated any number of times to learn all the data.

Details of each part are explained below.

<Data Selection Unit 110>
Input: Face images S ^k ₁ , …, S ^k _|M| for face keypoint learning, face images S ^p ₁ , …, S ^p _|N| for face pose learning
Output: One batch of training face images S ^b , data identifier i ^b
The data selection unit 110 receives face images S ^k ₁ , ..., S ^k _|M| for face keypoint learning and face images S ^p ₁ , ..., S ^p _|N| for face direction learning, and selects data to be used in one learning, that is, one batch of face images S ^b (S110). Note that b is an index indicating a batch number. For example, the face images S ^k ₁ , ..., S ^k _|M| for face keypoint learning and the face images S ^p ₁ , ..., S ^p _|N| for face direction learning are each divided into B batches, and b=1, 2, ..., 2B. In this case, one batch of face images S ^b is composed of, for example, 16 images. This one batch of face images S ^b is created for each face image for face keypoint learning and each face image for face direction learning, and the face images for learning in one batch are all composed of face images for face keypoint learning or all of the face images for face direction learning. As a data selection procedure, for example, the first batch of learning is a batch composed entirely of face images for face keypoint learning, the second batch of learning is a batch composed entirely of face images for face pose learning, and so on. The data identifier i ^b is an identifier for identifying whether one batch of learning face images S ^b is composed of data of face images for face keypoint learning or data of face images for face pose learning.

<Shared Network Unit 120>
Input: one batch of face images for learning S ^b , data identifier i ^b , updated model parameters ^θ _k or ^θ _p corresponding to the neural network constituting the shared network unit 120 Output: one batch of intermediate features v ^b , data identifier i ^b
The shared network unit 120 is configured with any neural network, for example, with four convolutional layers.

Prior to the conversion process, the shared network unit 120 receives updated model parameters ^θ _k (output value of the facial keypoint model parameter optimization unit 140) or ^θ _p (output value of the facial direction model parameter optimization unit 160) corresponding to any neural network that constitutes the shared network unit 120.

The shared network unit 120 converts one batch of training face images S b into intermediate features v b by a function of an arbitrary neural network using the updated model parameters ^θ _k or ^ ^θ _p corresponding to an arbitrary neural network constituting the shared network unit 120 (S120). For example, when the training face images S ^b of the ^b -th batch are composed of Q images, the intermediate feature corresponding to the q-th image is defined as v ^b _q , and v ^b = [v ^b ₁ , v ^b ₂ , ..., v ^b _Q ], where q = 1, 2, ..., Q.

Moreover, the shared network unit 120 outputs the data identifier ^i_b as is.

<Face Keypoint Network Unit 130>
Input: one batch of intermediate features v ^b , data identifier i ^b , and updated model parameters ^θ _k corresponding to the neural network constituting the face keypoint network unit 130. Output: estimated face keypoint vector Z' _k,b
The face keypoint network unit 130 is configured with any neural network, for example, with two fully connected layers.

Prior to the conversion process, the facial keypoint network unit 130 receives parameters corresponding to any neural network constituting the facial keypoint network unit 130 from among the updated model parameters ^θ _k (output values of the facial keypoint model parameter optimization unit 140).

When the data identifier i ^b indicates that the learning face image S ^b is composed of a face image for learning face keypoints (YES in S129), the face keypoint network unit 130 converts one batch of intermediate features v ^b into an estimated face keypoint vector Z' _k,b by using the received parameters and an arbitrary neural network function (S130). The estimated face keypoint vector is vector data that stores the coordinates of the estimated values of each point of the face keypoint in an array. In other words, the estimated face keypoint vector is an estimated value of the face keypoint vector. The face keypoint vector is vector data that stores the coordinates of each point of the face keypoint. For example, in the case of a model that estimates 68 two-dimensional keypoints, if X1 is the X coordinate value of the first keypoint and Y1 is the Y coordinate value of the first keypoint, the face keypoint vector will be a vector such as [[X1,Y1],...,[X68,Y68]]. In addition, in the case of a model that estimates three-dimensional keypoints, if Z1 is the Z coordinate value of the first keypoint, the vector will be something like [[X1,Y1,Z1],…,[X68,Y68,Z68]]. For example, let Z' _k,b,q be the estimated face keypoint vector corresponding to the q-th intermediate feature v ^b _q of the b-th batch, and let Z' _k,b =[Z' _k,b,1 ,Z' _k,b,2 ,…,Z' _k,b,Q ]. The estimated face keypoint vector Z' _k,b,q will be a vector like [[X1,Y1],…,[X68,Y68]] or [[X1,Y1,Z1],…,[X68,Y68,Z68]].

When the data identifier i ^b indicates that the training face image S ^b is composed of face images for face pose training, the facial keypoint network unit 130 and the facial keypoint model parameter optimization unit 140 do not perform any processing.

<Facial Keypoint Model Parameter Optimization Unit 140>
Input: Estimated face keypoint vector Z' _k,b , ground truth labels of face keypoints T ^k ₁ ,…,T ^k _|M|
Output: Updated model parameters ^θ _k or trained model parameters θ _k
The model parameters ^θ _k and θ _k are a concatenation of parameters corresponding to the neural network constituting the shared network unit 120 and parameters corresponding to the neural network constituting the face keypoint network unit 130 .

The face keypoint model parameter optimization unit 140 updates the model parameters ^ _θk using the estimated face keypoint vector _{Z'k ,b} and the face keypoint correct labels ^Tk1 , ..., ^Tk _|M| (S140) to perform optimization. For example, the face keypoint model parameter optimization unit 140 calculates the _error between the estimated face keypoint vector Z'k _,b and the face keypoint correct labels ^Tk1 , ..., ^Tk _|M| , updates the model parameters ^ _θk so as to minimize the error, and performs optimization. For example, the error may be an MSE error or an MAE error, and the parameter update method may be a gradient descent method or the like.

The above-mentioned processes S120 to S140 are repeated until a predetermined condition is met (S170). The predetermined condition is a condition for determining whether the parameter update has converged or not, and may be, for example, (i) the number of updates has exceeded a predetermined number, or (ii) the difference between the parameters before and after the update is smaller than a predetermined value.

<Face Direction Network Unit 150>
Input: one batch of intermediate features v ^b , data identifier i ^b , and updated model parameters ^θ _p corresponding to the neural network constituting the face direction network unit 150. Output: estimated face direction vector Z′ _p,b
The face direction network unit 150 is configured with any neural network, for example, with two fully connected layers.

Prior to the conversion process, the face direction network unit 150 receives parameters corresponding to any of the neural networks constituting the face direction network unit 150 from among the updated model parameters ^θ _p (output values of the face direction model parameter optimization unit 160).

When the data identifier i ^b indicates that the learning face image S ^b is composed of a face orientation learning face image (NO in S129), the face orientation network unit 150 converts one batch of intermediate features v ^b into an estimated face orientation vector Z' _p,b by using the received parameters and an arbitrary neural network function (S150). The estimated face orientation vector is vector data that stores estimated values of the rotation angles in three directions, roll, pitch, and yaw, of the face orientation. In other words, the estimated face orientation vector is an estimated value of the face orientation vector. The face orientation vector is a vector that stores the rotation angles in three directions, roll, pitch, and yaw, of the face orientation. For example, the rotation angle value has a range of -180 degrees to 180 degrees, and the face orientation vector is a vector such as [20, 43, -151]. For example, let Z' _p,b,q be the estimated face direction vector corresponding to the q-th intermediate feature v ^b _q of the b-th batch, and let Z' _p,b = [Z' _p,b,1 , Z' _p,b,2 , … , Z' _p,b,Q ]. The estimated face direction vector Z' _p,b,q is a vector such as [20, 43, -151].

When the data identifier i ^b indicates that the training face image S ^b is composed of face keypoint training face images, the face pose network unit 150 and the face pose model parameter optimization unit 160 do not perform any processing.

<Facial pose model parameter optimization unit 160>
Input: Estimated face direction vector Z' _p,b , face direction ground truth labels T ^p ₁ ,…,T ^p _|N|
Output: Updated model parameters ^θ _p or trained model parameters θ _p
The model parameters ^θ _p and θ _p are a combination of parameters corresponding to the neural network constituting the shared network unit 120 and parameters corresponding to the neural network constituting the face direction network unit 150 .

The face direction model parameter optimization unit 160 updates the model parameters ^ _θp using the estimated face direction vector _{Z'p ,b} and the face direction correct labels ^Tp1 , ..., ^Tp _|N| (S160) and performs optimization. For example, the face direction model parameter optimization unit 160 calculates the error between the estimated face direction vector Z'p _,b and the face direction correct labels ^Tp1 _, ..., ^Tp _|N| , updates the model parameters ^ _θp so as to minimize the error, and performs optimization. For example, the error can be an MSE error or an MAE error, and the parameter update method can be a gradient descent method or the like.

The above-mentioned processes S120 to S160 are repeated until the specified conditions are met (S170).

Furthermore, the above-mentioned processes S110 to S170 are performed on all batch data (learning data). For example, it is determined whether there is unprocessed batch data (S180), and if there is unprocessed batch data, the above-mentioned processes S110 to S170 are performed (NO in S180), and if there is no unprocessed batch data (YES in S180), the process ends.

After performing the above-mentioned processing on all the batch data, the finally obtained updated model parameters ^θ _k , ^θ _p are output as trained model parameters θ _k , θ _p .

Next, we will explain the estimation device 200.

<Estimation device 200>
Prior to the estimation process, the estimation device 200 receives trained model parameters _θk and _θp . The estimation device 200 receives a face image S to be estimated as an input, estimates face keypoints using the trained model parameter _θk , estimates a face direction using the trained model parameter _θp , and outputs an estimated face keypoint vector _Zk and an estimated face direction vector _Zp .

Figure 4 shows a functional block diagram of the estimation device 200, and Figure 5 shows its processing flow.

The estimation device 200 includes a face key point estimation unit 210 and a face direction estimation unit 220.

Details of each part are explained below.

<Facial Keypoint Estimation Unit 210>
Input: Face image S, model parameters θ _k
Output: Estimated face keypoint vector Z _k for face image S
Prior to the estimation process, the facial keypoint estimation unit 210 receives the model parameters θ _k .

The facial keypoint estimation unit 210 estimates facial keypoints from the facial image S (S210) using the network architecture of the shared network unit 120 and the subsequent facial keypoint network unit 130, and the model parameter θ _k , to obtain an estimate (estimated facial keypoint vector Z _k ). For example, if the shared network unit 120 is composed of an arbitrary neural network consisting of four convolutional layers, and the facial keypoint network unit 130 is composed of an arbitrary neural network consisting of two fully connected layers, the facial keypoint estimation unit 210 is composed of a neural network consisting of four convolutional layers and two fully connected layers corresponding to the shared network unit 120 and the facial keypoint network unit 130, and uses the model parameter θ _k in this neural network.

<Facial Direction Estimation Unit 220>
Input: Face image S, model parameters θ _p
Output: Estimated face direction vector Z _p for face image S
Prior to the estimation process, the face direction estimation unit 220 receives the model parameter θ _p .

The face direction estimation unit 220 estimates a face direction from the face image S using the network architecture of the shared network unit 120 and the subsequent face direction network unit 150 and the model parameter θ _p (S220), and obtains an estimated value (estimated face direction vector Z _p ). For example, if the shared network unit 120 is configured with an arbitrary neural network consisting of four convolution layers, and the face direction network unit 150 is configured with an arbitrary neural network consisting of two fully connected layers, the face direction estimation unit 220 is configured with a neural network consisting of four convolution layers and two fully connected layers corresponding to the shared network unit 120 and the face direction network unit 150, and uses the model parameter θ _p in this neural network.

＜Effects＞
With the above configuration, it is possible to estimate face keypoints and face poses that capture more general facial features.

<Modification>
In this embodiment, the estimation device 200 estimates face key points and face direction, but may estimate only one of them. Even in this case, since model parameters used for estimating face key points and face direction are learned in a single neural network system during learning, it is possible to estimate face key points or face direction that capture more general facial features.

In addition, in this embodiment, learning is performed by alternating between batches composed entirely of face images for face keypoint learning and batches composed entirely of face images for face orientation learning as learning data, but it is not necessary to alternate between batches one at a time; for example, ten batches may be used alternately. However, it is preferable to alternate between batches one at a time to avoid bias towards any of the learning data.

<Other Modifications>
The present invention is not limited to the above-mentioned embodiment and modified examples. For example, the above-mentioned various processes may be executed not only in chronological order as described, but also in parallel or individually depending on the processing capacity of the device executing the processes or as necessary. In addition, appropriate modifications are possible within the scope of the present invention.

<Program and recording medium>
The various processes described above can be implemented by loading a program that executes each step of the above method into the memory unit 2020 of the computer shown in Figure 6, and operating the control unit 2010, input unit 2030, output unit 2040, etc.

The program describing the processing can be recorded on a computer-readable recording medium. Examples of computer-readable recording media include magnetic recording devices, optical disks, magneto-optical recording media, and semiconductor memories.

The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or the program transferred from a server computer in its own storage device. Then, when executing a process, the computer reads the program stored on its own recording medium and executes the process according to the read program. As another execution form of this program, the computer may read the program directly from the portable recording medium and execute the process according to the program, or may execute the process according to the received program each time a program is transferred from the server computer to this computer. In addition, the server computer may not transfer the program to this computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes the processing function only by issuing an execution instruction and obtaining the results. Note that the program in this embodiment includes information used for processing by an electronic computer that is equivalent to a program (such as data that is not a direct command to the computer but has a nature that specifies the processing of the computer).

In addition, in this embodiment, the device is configured by executing a specific program on a computer, but at least a portion of the processing content may be realized by hardware.

Claims (8)

A shared network unit that converts the learning face image S ^b into an intermediate feature v ^b by a neural network function using the model parameter ^θ _k or ^θ _p ;
a face keypoint network unit that converts the intermediate feature v ^b into an estimated face keypoint vector Z' _k, b using the model parameter ^θ _k and a neural network function when the training face image S ^b is composed of face keypoint training face images;
a face keypoint model parameter optimization unit that updates the model parameter ^θ _k using the estimated face keypoint vector Z' _k,b and face keypoint correct labels T ^k ₁ , ..., T ^k _|M| for the learning face image S ^b ;
a face direction network unit that converts an intermediate feature v ^b into an estimated face direction vector Z' _p, b by a neural network function using the model parameter ^θ _p when the learning face image S ^b is composed of face direction learning face images;
a face direction model parameter optimization unit that updates the model parameter ^θ _p by using the estimated face direction vector Z′ _p,b and face direction correct labels T ^p ₁ , . . . , T ^p _|N|;
Acquire a trained model parameter θ _k corresponding to the model parameter ^θ _k and a trained model parameter θ _{p corresponding to the model parameter ^θ p .}
Learning device. An estimation device using the model parameter θ _k learned by the learning device of claim 1,
a face keypoint estimation unit that estimates face keypoints from an estimation target face image S using a network architecture of the shared network unit and the subsequent face keypoint network unit and the model parameter θ _k ;
Estimation device. An estimation device using a model parameter θ _p learned by the learning device of claim 1,
a face direction estimation unit that estimates a face direction from the face image S by using a network architecture of the shared network unit and the subsequent face direction network unit and the model parameter θ _p ;
Estimation device. An estimation device that uses the model parameters θ _k and θ _p learned by the learning device of claim 1,
a face keypoint estimation unit that estimates face keypoints from an estimation target face image S using the network architecture of the shared network unit and the subsequent face keypoint network unit and the model parameter θ _k ;
a face direction estimation unit that estimates a face direction from the face image S by using a network architecture of the shared network unit and the subsequent face direction network unit, and the model parameter θ _p ;
Estimation device. A shared network step of converting the training face image S ^b into intermediate features v ^b by a neural network function using the model parameter ^θ _k or ^θ _p ;
a face keypoint network step of converting the intermediate feature v ^b into an estimated face keypoint vector Z' _k,b by a neural network function using the model parameter ^θ _k when the training face image S ^b is composed of face keypoint training face images;
a face keypoint model parameter optimization step of updating the model parameter ^θ _k using the estimated face keypoint vector Z' _k,b and face keypoint ground truth labels T ^k ₁ , ..., T ^k _|M| for the learning face image S ^b ;
a face direction network step of converting the intermediate feature v ^b into an estimated face direction vector Z' _p, b by a neural network function using the model parameter ^θ _p when the learning face image S ^b is composed of face direction learning face images;
a face direction model parameter optimization step of updating the model parameter ^θ _p using the estimated face direction vector Z′ _p,b and face direction ground truth labels T ^p ₁ , . . . , T ^p _|N|;
Acquire a trained model parameter θ _k corresponding to the model parameter ^θ _k and a trained model parameter θ _p corresponding to the model parameter ^θ _p .
How to learn. An estimation method using a model parameter θ _k learned by the learning method of claim 5,
a face keypoint estimation step of estimating face keypoints from an estimation target face image S using a network architecture of the shared network step and the subsequent face keypoint network step and the model parameter θ _k ;
Estimation method. An estimation method using a model parameter θ _p learned by the learning method of claim 5,
a face direction estimation step of estimating a face direction from the face image S by using a network architecture of the shared network step and the subsequent face direction network step and the model parameter θ _p ;
Estimation method. A program for causing a computer to function as the learning device of claim 1 or as any one of the estimation devices of claims 2 to 4.

Images