JP5231183B2 - 3D shape restoration device - Google Patents

Description

  The present invention relates to a three-dimensional shape restoration apparatus that restores a three-dimensional shape of an object from a series of images (two-dimensional shape) obtained by photographing the object.

  Conventionally, an image sequence obtained by photographing an object is input, the positions of the feature points of a plurality of feature points set in advance for the object are tracked on the image plane, and the object is obtained from the feature point position sequence obtained by the tracking. An apparatus for restoring the three-dimensional shape is known.

  As one of the main three-dimensional reconstruction devices, an observation data matrix is generated from a feature point position sequence, and the observation data matrix is represented by a shape matrix representing the shape of the object (three-dimensional position of the feature point) and the motion of the object. There is a method using a method of factoring into a motion matrix representing (see, for example, Patent Documents 1 to 3).

  However, the factorization method cannot obtain effective results unless all the data that make up the observation data matrix is available, so the feature point position series (and thus the observation) There has been a problem that it is very difficult to apply to actual problems in which data (data constituting a data matrix) is frequently lost.

  On the other hand, the applicant has proposed a method for dealing with data loss by statistically treating the data and performing a calculation considering the variance of the probability distribution of the data as the reliability of the data (Patent Literature). 4).

  However, even in this method, when the object whose shape is to be estimated is a human face, detection accuracy deteriorates if the state in which the feature points are hidden by decorative items such as sunglasses and a scarf continues.

For this reason, it has not only feature points fixed in advance (hereinafter referred to as “fixed feature points”) but also pre-defined features (such as edges and corners of the imaged object) that are detected each time. It has been studied to detect a point (hereinafter referred to as “arbitrary feature point”) and to use the arbitrary feature point.
JP 2000-194863 A JP 2003-141552 A JP 2004-347332 A JP 2008-261661 A

  However, the arbitrary feature point is extracted not only on the object but also on the background of the object or on an object other than the object if it has the feature.

  However, in the conventional devices described in Patent Documents 1 to 4, since the feature points constituting the observation data matrix are premised on being derived from a single object, the feature points are In the case of being derived from a plurality of objects, there is a problem that the shape matrix (and consequently the shape of the object) cannot be obtained correctly.

  Note that it is conceivable to identify which object each feature point is derived from, and factorize each observation data matrix created for each object using the identified feature point. However, the feature points need to be identified by another image processing or the like, which not only increases the amount of processing, but also makes it difficult to identify the feature points, resulting in a problem that sufficient accuracy cannot be obtained.

  In addition, in the method of obtaining the shape matrix by factoring the observation data matrix, it is necessary to increase the image sequence used for the calculation (that is, to increase the dimension of the observation data matrix) in order to improve the estimation accuracy of the three-dimensional shape. is there.

  However, it is known that the amount of calculation required for factorization is usually proportional to the cube of the number of rows of the observation data matrix × the number of columns, and explosive by increasing the dimension of the observation data matrix. To increase.

  And when the amount of calculation increases, there is a problem that it becomes difficult to apply when there is a limit to the calculation capability of an arithmetic unit that can be used, such as when a three-dimensional shape restoration device is mounted on a vehicle. It was.

  In order to solve the above problems, the present invention enables the factorization to be performed even if data derived from a plurality of objects is mixed in the observation data matrix (feature point position series) in the three-dimensional shape restoration apparatus. Furthermore, it aims at reducing calculation time.

  In the three-dimensional shape restoration apparatus of the present invention made to achieve the above object, the feature point tracking means inputs an image sequence and extracts a plurality of feature points having preset features from each input image. At the same time, the position of each feature point is tracked on the image plane.

  The observation data matrix generation unit generates an observation data matrix representing the position of the feature point according to the extraction and tracking result of the feature point extraction and tracking unit. Then, the decomposition means factorizes the observation data matrix generated by the observation data matrix generation means to generate a shape matrix that represents the three-dimensional shape of the object imaged in the image and a motion matrix that represents the motion of the object. To do.

  At this time, the decomposition means x is the observation data matrix, z is the shape matrix, h is the designation matrix that specifies the object to which each feature point belongs, h is the motion matrix, the accuracy of measurement of the feature point, and the feature point is the respective object. Θ, z, h are obtained by calculating a posteriori probability P (θ, z, h | x) from the observation data matrix x, where θ is a model parameter that summarizes the probabilities belonging to.

  According to the three-dimensional shape restoration apparatus of the present invention configured as described above, factorization can be performed even if the observed data matrix x is a mixed data matrix composed of feature points derived from a plurality of objects. The shape matrix and the motion matrix can be obtained for each feature point classified by the designation matrix h, that is, for each object from which the feature point is derived.

  In the present invention, the decomposition means has a test distribution q (θ, h, z), which is an approximate function of the posterior probability P (θ, z, h | x), as described in claim 2. The posterior probabilities P (θ, z, h | x) and the test distribution q (θ, h, z) are factorized into the test distribution q (θ) and the second test distribution q (h, z). The first test distribution q (θ) and the second test distribution q (h, z) that minimize the KL distance may be calculated and the factorization calculation may be executed probabilistically.

  According to the three-dimensional shape restoration apparatus of the present invention configured as described above, a special process other than factorization is performed because the specified matrix h is also obtained probabilistically by Bayesian estimation simultaneously with obtaining the shape matrix z. Therefore, the feature points can be easily identified for each object.

  When factorization using Bayesian estimation is performed as described above, the observation data matrix generation unit generates the observation data matrix x based on images for a preset number of frames as described in claim 3. The decomposition means fixes the first test distribution q (θ) and estimates the second test distribution q (z, h), and the second test distribution q (z, h). When the second step of estimating the first test distribution q (θ) in a fixed manner is repeatedly executed alternately using the estimation results, and when a preset end condition is satisfied And ending means for ending the estimating means, and may be configured to obtain the shape matrix z and the designated matrix h from the finally obtained second test distribution q (z, h).

  In this case, the decomposition means implements a variational Bayesian estimation method, and the first step corresponds to the E step and the second step corresponds to the M step.

  However, even when the variational Bayes estimation method is used, the amount of calculation (calculation time) is prevented from increasing exponentially in accordance with the increase in the image sequence used for the calculation (that is, the increase in the dimension of the observation data matrix). It is not possible.

  Therefore, as described in claim 4, the observation data matrix generation unit generates the observation data matrix x for each image of one frame, and the decomposition unit determines that the observation data matrix x is generated by the observation data matrix generation unit. A first step of fixing the first test distribution q (θ) and estimating the second test distribution q (z, h) each time it is generated, and fixing the second test distribution q (z, h) Then, when the second step of estimating the first test distribution q (θ) is executed using the mutual estimation results, and when the preset end condition is satisfied, the estimation unit and the update It is desirable that it is configured so as to obtain the shape matrix z and the designated matrix h from the second test distribution q (z, h) sequentially obtained by the estimation means.

  In other words, the estimation means limits the image series used for the calculation, executes the first step (E step) and the second step (M step) once, repeatedly executes this, and from the estimation means Each time a calculation result is obtained, the distribution of the shape matrix is updated based on the calculation result. This realizes the variational Bayesian estimation method by sequential calculation (so-called online).

  Thus, according to the three-dimensional shape restoration apparatus of the present invention, the image sequence to be used is divided into a plurality of parts, and factorization is performed for each of the divided image sequences. Since it only increases incrementally in units of one process in the estimation means and does not increase exponentially, the processing load required for factorization (and thus the time required for processing) can be greatly reduced. it can.

  And the said three-dimensional shape decompression | restoration apparatus may be comprised so that it may mount and use for a motor vehicle, as described in Claim 5.

  An arithmetic unit mounted on an automobile generally has a low calculation capability, but can be applied even in such a case.

  Embodiments of the present invention will be described below with reference to the drawings.

[First Embodiment]
FIG. 1 is a block diagram showing an overall configuration of an image processing apparatus 1 as a three-dimensional shape restoration apparatus to which the present invention is applied. The image processing apparatus 1 is mounted on a vehicle and executes processing for obtaining a head posture in time series, which is information for detecting a driver's sidewalk, doze, or the like from an image obtained by photographing the driver's head. Device.

<Overall configuration>
As shown in FIG. 1, an image processing apparatus 1 is installed on the front upper part or lower part (for example, in a meter) of a driver seat, acquires an image including a driver's head, and an image acquired by the camera 2. The position of a plurality of feature points having a preset feature (image plane) from the image capturing device 3 that captures the image and the input image s _f (f is an identifier for identifying a frame) captured via the image capturing device 3 Features for extracting the two-dimensional coordinates) x ^(p) (p = 1, 2,... P) and tracking the extracted feature points to define the distribution of the positions of the feature points And a feature point tracking unit 4 for obtaining point estimation distribution parameters (average value x _f , variance V _f ).

In addition, the image processing apparatus 1 includes a face model supply unit 5 that supplies _a face model (shape matrix) C _a that defines a three-dimensional position of a feature point (and thus a three-dimensional shape of the head), and a feature point tracking unit 4. An estimation unit 6 for estimating an affine parameter A _f representing the head posture of the driver using a Kalman filter based on the feature point estimation distribution parameter supplied from the face model and the face model C _a supplied from the face model supply unit 5; According to the affine parameter A _f estimated by the estimation unit 6, the head posture calculation unit 7 is configured to calculate the head postures θx, θy, and θz.

<Feature point tracking unit>
The feature point tracking unit 4 includes a plurality of feature points representing the features of the driver's face, such as the corners of the eyes, the eyes, the mouth, the nose (the nostrils, the center of the nose, the left and right ends of the nose), etc. (see FIG. 2A). A fixed feature point tracking unit 41 that extracts and tracks the position of the “fixed feature point” (hereinafter also referred to as “fixed feature point”), and a part (hereinafter, an edge or a corner of the imaged object) that is easy to track in the image And an arbitrary feature point tracking unit 43 for extracting and tracking the position of “optional feature point”.

The fixed feature point tracking unit 41 extracts and tracks fixed feature points by template matching with the input image s _f using a template prepared in advance. The template is specifically as shown by a dotted white frame in FIG. However, in the figure, only the left mouth, the center of the nose, and the left eye corner are shown.

  On the other hand, the arbitrary feature point tracking unit 43 extracts and tracks arbitrary feature points using a known feature point detector such as SIFT (Scale Invariant Feature Transform) or KLT (Kanade-Lucas-Tomasi).

  Here, FIG. 3 is an explanatory diagram illustrating an example of arbitrary feature points extracted by the arbitrary feature point tracking unit 43. However, (a) is an input image, (b) is a fixed feature point extracted by the fixed feature point tracking unit 41, and (c) is an arbitrary feature point extracted by the arbitrary feature point tracking unit 43.

  As shown in FIG. 3, the arbitrary feature points are extracted not only from the driver's head but also from the background (vehicle compartment). However, even if the driver wears sunglasses or the like and the fixed feature points (eye corners, eye heads, etc.) are hidden, many arbitrary feature points are extracted from the contours of the sunglasses.

  That is, in the feature point tracking unit 4, arbitrary feature points are appropriately added as tracking targets in addition to the fixed feature points that are always tracked targets.

  However, the arbitrary feature point tracking unit 4 excludes any of the arbitrary feature points that coincide with the fixed feature points from the tracking target, and also features points (hereinafter referred to as “hereinafter“ After the information of “head feature points” is obtained, arbitrary feature points other than the head feature points are excluded from the tracking target based on the information.

Further, both the fixed feature point and the arbitrary feature point are tracked using a particle filter, and the posterior probability p (x _f | A _f ) necessary for the processing is supplied from the estimation unit 6. It is configured.

<Face model>
The face model C _a supplied from the face model supply unit 5 three-dimensionally defines the position of the feature point tracked by the feature point tracking unit 4 on the driver's face, As shown in FIG. 4, the horizontal coordinate axis on the image plane is the u axis, the vertical coordinate axis is the v axis, the horizontal coordinate axis in the three-dimensional space is the X axis, the vertical coordinate axis is the Y axis, and the optical axis direction ( The coordinate axis in the depth direction is expressed by the equations (1) and (2) with the Z axis as the Z axis. However, z ^(p) is the coordinate in the three-dimensional space of the p-th feature point (feature point position x ^(p) = (u ^(p) , v ^(p) ) extracted by the feature point tracking unit 4. It is.

<Estimation unit>
The affine parameter A _f estimated by the estimation unit 6 is used to express coordinate transformation in the real world (three-dimensional XYZ space) as coordinate transformation on an image plane (two-dimensional uv plane) that is a projection plane. It is a transformation matrix and has the relationship shown in equations (3) and (4).

However, the three-dimensional coordinates of a certain feature point are (X, Y, Z) ^T , the three-dimensional coordinates after coordinate transformation are (X ′, Y ′, Z ′) ^T , and these three-dimensional coordinates are projected onto the image plane. Two-dimensional coordinates, that is, the coordinates of feature points on the image plane are (u, v) ^T and (u ′, v ′) ^T , and X = u, Y = v (that is, X ′ = u ′, Y ′ = v ′).

<Head posture calculation unit>
The head posture calculation unit 7 calculates the three-axis angles θx, θy, θz of the head using equations (5) to (7) as head posture information.

Then, as long obtained affine parameters A _f representing the head pose in the frame f, the affine parameters A _f Is applied to the face model C _a , it is possible to predict the positions of all feature points in the frame f including the positions of hidden feature points that are not reflected in the image in the frame f.

In addition, such affine parameters _Af can reflect all changes in camera parameters (camera focal length, installation location, orientation, etc.) and driver's head posture.

<Outline of processing in feature point tracking unit and estimation unit>
Here, FIG. 5 is a graph showing a state space model handled by the feature point tracking unit 4 and the estimation unit 6.

As shown in FIG. 5, the upper layer (estimator 6) that obtains the affine parameter A _f representing the posture of the face model in the frame f as a hidden state, and the position x _f = (x _f ^{(1 )} , X _f ⁽²⁾ ,..., X _f ^(P) ) It has a lower layer (feature point tracking unit 4) consisting of P number of trackers that find ^T as a hidden state. The hidden states A _f and x _f are estimated from the input image sequence s _{1: f} by time series Bayesian estimation.

  As shown in FIG. 6, time series Bayesian estimation treats all state variables as random variables, obtains a predicted distribution in frame f from an estimation result (estimated distribution) in frame f-1 regarding the hidden state, By repeating the procedure of obtaining the likelihood (likelihood) that is the hidden state to be detected from the observation data at, and obtaining the estimation result (estimation distribution) in the frame f in consideration of these prediction distribution and likelihood, The hidden state is estimated.

That is, based on the input image sequence (observation data) s _{1: f} , the posterior probability (estimated distribution) p (A _f | s _{1: f} ) obtained by estimating the affine parameter (head posture hidden state) A _f is obtained. When this is described by mathematical formulas, it is represented by formulas (8) and (9).

Here, p (A _f | s _{1: f-1} ) is the prior probability distribution (predicted distribution) of the affine parameter A _f , and p (x _f | A _f , s _{1: f-1} ) is the position of the feature point group. prior probability distribution of x _f (prediction _{distribution), p (x f | a} f, s 1: f-1), p (s f | x f) represents the likelihood.

  The lower layer (feature point tracking unit 4) estimates the part of equation (8) using a particle filter, and the upper layer (estimator 6) estimates the portion of equation (9) using a Kalman filter.

When the probability distribution of a certain parameter follows a Gaussian distribution (normal distribution), the calculation of the probability distribution of the parameter is actually sufficient if the average x _f and the variance V _f are obtained.

  The specific configuration of the estimation unit 6 is described in, for example, Japanese Patent Application No. 2005-368124, and the specific configuration of the feature point tracking unit 4 is, for example, in Japanese Patent Application No. 2007-102914. Since it is described in detail, a detailed description thereof is omitted here.

<Face model supply department>
Next, the face model supply unit 5 which is a main part of the present invention will be described in detail.

As shown in FIG. 1, the face model supply unit 5 extracts fixed feature points from the images of a plurality of people whose heads are photographed, and average fixed feature points set based on the extracted results. Based on the average face model storage unit 51 for storing the average face model representing the arrangement of the image and the feature point estimation distribution parameter x _f that is activated by an activation command from the outside and is sequentially supplied from the feature point tracking unit 4 An individual face model learning unit 52 that learns and generates a personal face model representing the arrangement of feature points (fixed feature points and arbitrary feature points) derived from the driver's head, which is a subject, and an individual face model learning unit 52 In accordance with the switching signal, either the average face model stored in the average face model storage unit 51 or the individual face model generated by the individual face model learning unit 52 is selected as the face model C _a and supplied to the estimation unit 6. Face to do And a model switching unit 53.

Note that the individual face model learning unit 52 corresponds to the three-dimensional shape restoration device of the present invention, and the feature point x _f derived from a plurality of objects (for example, a driver, a vehicle, a scenery outside the vehicle reflected in a vehicle window, etc.) Information indicating which object each is derived from and the shape of each object (especially the head of the driver) from which the feature points are derived are simultaneously learned.

<Summary of learning>
The personal face model learning unit 52 performs learning by the following method.

<Model>
First, a model applied to learning will be described.

  Here, considering the generation process of observation data (feature point group) when there are multiple objects, there are multiple generation processes, and each process is selected probabilistically. Modeled by dynamic principal component analysis).

First, it is assumed that image capturing by a camera is a linear mapping (affine camera) from a three-dimensional space to a two-dimensional plane. Assuming that the shape of the object to be imaged is a time-invariant rigid body, a matrix z representing the three-dimensional positions of the P feature points is expressed by equation (10), and a matrix x _f representing the position of each feature point on the image of the frame _f. Is expressed by equation (11). However, the three-dimensional position of the p-th feature point is z ^(p) , and the two-dimensional position of the p-th feature point in the frame f is x _f ^(p) .

The generation process of the two-dimensional position x _f ^(p) of the feature point under the affine camera can be expressed by the equation (12) with the addition of the observation noise ε.

However, W _f is the projection matrix, the mu _f represents the translation component.

  Then, if P feature points are observed over the number of frames F (obtained from the feature point tracking unit 4), an observation data matrix x represented by equation (13) is obtained.

However, W _e is defined by the equation (14), hereinafter referred to as motion matrix. Z _e is defined by equation (15), and is hereinafter referred to as a shape matrix.

When a plurality of objects are photographed in the image, since there is a motion matrix W _e for each object, an object corresponding to the object u (u = 1, 2,..., U) is represented by We _{/ u.} It shall be expressed as However, since it is not known on the image which object each feature point is derived from, the observation data matrix x in which the feature points of each object are mixed is measured.

Further, assuming that the number of feature points measured for the object u is represented by P _u , the total number P of feature points is represented by equation (16).

Here, an instruction variable h ^(p) for designating which object the feature point p is derived from is expressed by equation (17). This indicating variable has U elements h _u ^{(p) which} are the same as the total number of objects assumed.

When the instruction variable h ^(p) for the feature point p expresses that the feature point p is derived from the object u, only the u-th element is 1 ( _hu ^(p) = 1). And all other elements are zero. In addition, since there are as many instruction variables h ^{(p) as} the number of feature points P, a collection of these is referred to as a designation matrix h and is represented by equation (18).

Also, when one point is selected at random from all the feature points, the probability that the selected feature point belongs to the object u is referred to as _gu , and the sum of all the objects is called the mixture ratio g. , (19).

When the number of feature points (P ₁ ,..., P _U ) measured for each object is known, g _u is randomly selected from P feature points and specified P _u feature points. Therefore, g _u = P _u / P. However, since _Pu is unknown, it is also necessary to estimate g.

Further, it is assumed that the noise ε mixed in the measurement relating to the object u follows a normal distribution with an expected value of 0 and a variance τ _{| u} ⁻¹ .

Assuming that the feature point p belongs to the object u, the noise distribution p (ε _p ) of the feature point p is expressed by equation (20). However, N (x | m, Σ) indicates that the random variable x expressed as a d-dimensional vector follows the normal distribution of the expected value m and covariance Σ, and the content is defined by the equation (21). To do.

From the above settings, the shape matrix z, the instruction matrix h, the mixing ratio g, the affine parameters W _{e | u} (u = 1, 2,..., U) for each object, and the accuracy of measurement of the feature point position of each object ( The reciprocal of noise variance) τ _{| u} is the target of estimation (learning).

  A graphical model illustrating these relationships is shown in FIG.

  In the following, model parameters θ and J shown in equations (22) and (23) are used to simplify the description of the equations.

Under the graphical model shown in FIG. 7, if the feature point (observation data) x is obtained, the shape matrix z and the model parameter θ are estimated by calculating the posterior probability p (θ, z, h | x). Can do.

  Further, the posterior distribution (posterior probability) p (θ, z, h | x) is described as shown in Equation (24) by Bayesian estimation.

In order to calculate this equation, the prior distribution p ₀ of the shape matrix z and the model parameter θ (ie, W _{e | u} , τ _{| u} , g) is set according to equations (25) to (28). However, α _{e | u} is a four-dimensional vector, and α _{e | u} = [1,1,1,1] ^T is used here, but some distribution may be provided.

However, we _{, i | u} is a column vector corresponding to the i-th row of We _{| u} . Further, the gamma distribution Gam (x | a, b) is defined by equation (29), and the Dirichlet distribution Dir (x | γ _x ) is defined by equation (30).

Since it is impossible to calculate equation (24) analytically, the posterior distribution is approximately obtained using the variation estimation method.

<Variation estimation method>
In the variation estimation method, a test distribution q (θ, h, z) that is an approximate function of the posterior distribution p (θ, h, z | x) is calculated. However, assuming that q (θ, h, z) = q (θ) q (h, z) is factorized, the test distribution q (θ, h, z) and the true posterior distribution p (θ, The test distribution q (θ) q (h, z) is adjusted so as to minimize a KL (Kullback-Leibler) pseudorange between h, z | x).

  By the way, it is known that obtaining a test distribution that minimizes the KL pseudorange is equivalent to obtaining a test distribution that maximizes free energy. That is, in the variation estimation method, the test distribution that maximizes the free energy is the most appropriate estimated solution.

In this case, free energy _FE is expressed by equation (31). In this case, free energy F _E It is known that the test distribution q (θ, h, z) that maximizes the condition satisfies the conditions shown in the equations (32) and (33) based on the prior simultaneous distribution p ₀ (J).

However, <> q means an operation for obtaining an expected value based on the distribution q.

Expressions obtained by developing the conditional expressions (32) and (33) are in a form in which θ, h, and z that are estimation targets are mutually dependent, and thus an explicit solution cannot be obtained. However, applying the conditional expressions (32) and (33) repeatedly is free energy F _E Since this is equivalent to performing the gradient method for, an approximate solution can be obtained.

  Specifically, in the E step of the variational Bayes method, the test distribution q (θ) is fixed and the test distribution q (h, z) is obtained. In the M step of the variational Bayes method, the test distribution q ( The test distribution q (θ) is obtained after fixing h, z).

  Here, each test distribution obtained in the E step is parameterized as shown in equations (34) to (36).

Also, each test distribution obtained in M steps is parameterized as shown in equations (37) to (40).

Expressions (32) and (33) are developed based on Expressions (25) to (28) and Expression (41) shown below.

Then, the conditions shown in the following equations (42) to (46) are imposed on the parameters of each test distribution obtained in the E step. Where Σ _{z | u} is the covariance matrix (reliability) of the estimated structure of the object u, z _{| u} ^(p) is the estimated coordinates when the feature point p is assumed to be derived from the object u, and c _u ^(p) Represents the reliability of the assumption that the feature point p is derived from the object u, and c _u represents a normalization term of the test distribution q (h ^(p) ).

Similarly, the conditions shown in the following equations (47) to (52) are imposed on the parameters of each test distribution obtained in M steps. However, the expected value of γ ratio of _gu mixing ratio (estimate), M _{We | u} is W _{e |} expected value of _u (estimated value),? W e _{| u} is W _{e |} covariance of each line _u putative matrix (reliability), mτ _{| u} is tau _| expected value of _u (estimated value), γτ _{| u} is tau _| represents the components of _{u |} variance estimate of _u (reliability) mτ _{| u} / γτ.

The distribution parameter Θ _S that summarizes all the parameters calculated in the E step is expressed by the equation (53), and the distribution parameter Θ _M that summarizes all the parameters calculated in the M step is expressed by the equation (54). Is done.

Further, a sufficient statistic r depending on h and z is expressed by equation (55). Specifically, in each of the equations (47), (48), (49), (51), and (52), the term enclosed by <<>> _fc is a sufficient statistic.

<Processing in the personal face model learning unit>
Here, the process executed by the individual face model learning unit 52 will be described with reference to the flowchart shown in FIG. Note that this process is activated by an input of an activation command from the outside.

When this processing is started, first, in S110, the face model C _a supply face model switching section 53 to the estimating portion 6, a switching signal to select the average face model stored in the average face model storage unit 51 Output, and proceed to S120.

Thereby, every time an image (frame) is input, the estimation unit 6 calculates the affine parameters A _f and the feature point estimation distribution parameters x _f and V _f using the average face model. Further, the head posture calculation unit 7 calculates the head postures θx, θy, and θz based on the affine parameters A _f obtained by the estimation unit 6.

  In S120, the first test distribution q (θ) that is the test distribution of the model parameter θ is initialized, and the process proceeds to S130.

Specifically, the distribution parameter Θ _M shown in the equation (54) and the sufficient statistic r shown in the equation (55) are initialized. For example, all the elements have the same constant value (for example, 0 or 1 or 0.5).

In S130, it is determined whether or not the acquisition of feature point parameters x _f (see formula (11)) for a predetermined number F of preset frames from the feature point tracking unit 4 is completed. Wait for completion.

When the acquisition of the feature point parameters x _{f for} the specified number of frames F is completed, an observation data matrix x (see equation (13)) that summarizes these is generated, and the process proceeds to S140, and the set distribution parameter Θ _M Based on the above, the calculation of the second test distribution q (z, h), that is, the process as the E step of variational Bayes is executed, and the process proceeds to S150.

Specifically, the distribution parameter Θ _S shown in the equation (53) is calculated according to the equations (42) to (46).

  In S150, the statistical amount r is sufficiently updated according to the observation data matrix x and the calculation result of the second test distribution q (z, h) in S140, and in S160, the calculation of the first test distribution q (θ), that is, Then, the process as the M step of variational Bayes is executed, and the process proceeds to S170.

Specifically, the distribution parameter Θ _M shown in the equation (54) is calculated according to the equations (47) to (52) using the sufficient statistic r calculated in S150.

In S170, S140, based on the calculation result in S160, the free energy F _E using the above-described (31) equation Is calculated and the process proceeds to S190.

However, the free energy F _E is estimated based on the motion matrix W and the shape matrix z obtained from the first test distribution q (θ) and the second test distribution q (h, z) obtained by the above-described processing. The closer the position is to the position (observation data matrix x) obtained by observation, the larger the value.

In S180, the free energy F _E calculated in S170. Is set to be larger than a preset energy threshold, and it is determined whether or not the end condition is satisfied. If not satisfied, the process returns to S140, and the processes of S140 to S170 are repeated. If the end condition is satisfied, the process proceeds to S190.

  Here, as the energy threshold, for example, the free energy calculated using the shape matrix representing the average face model and the motion matrix W obtained by the above processing, or a fixed value set to a value larger than the free energy Is used.

  In S190, for each feature point p (p = 1, 2,..., P), the object u from which the feature point is derived is discriminated using the discriminant shown in equation (56), and the driver's Only feature points derived from the head (hereinafter referred to as “head feature points”) are extracted and notified to the feature point tracking unit 4.

As a result, the feature point tracking unit 4 can subsequently select and track only those derived from the driver's head from the arbitrary feature points based on the notified head feature points.

In S200, a partial shape matrix in which only a portion based on the head feature point obtained in S190 is extracted from the shape matrix obtained from the second test distribution q (h, z) is generated, and the partial shape matrix is generated. the outputs as an individual face model, as face model C _a supply face model switching section 53 to the estimating unit 6, and outputs a switching signal to select the individual face model, and the process ends.

Thereby, thereafter, every time an image (frame) is input, the estimation unit 6 calculates the affine parameter A _f using the individual face model, and further, based on the affine parameter A _f , The posture calculation unit 7 calculates head postures θx, θy, and θz.

<Effect>
As described above, the image processing apparatus 1 calculates the affine parameter _Af using the average face model based on the fixed feature points (face orientation) immediately after the process of estimating the head postures θx, θy, and θz is started. In parallel with this, learning of the individual face model using arbitrary feature points is executed, and when the accuracy of the individual face model is sufficiently improved, switching from the average face model to the individual face model is performed. Have been to do.

  Therefore, according to the image processing apparatus 1, no fixed feature point can be extracted regardless of who the driver is or the article (sunglasses, mask, hat, scarf, etc.) worn by the driver. However, the estimation of the head postures θx, θy, and θz can be obtained from the initial stage of the processing with a stable accuracy estimation result, and after switching from the average face model to the individual face model, the accuracy is high. Can be obtained.

  Further, in the image processing apparatus 1, when the observation data matrix x is factorized, a designation matrix h is introduced to simultaneously estimate which object each feature point is derived from.

  Therefore, according to the image processing apparatus 1, not only the feature point (head feature point) derived from the driver's head but also an object other than the driver's head (for example, outside the vehicle reflected in the vehicle or window) Even if a feature point (non-head feature point) derived from the scenery of the image is included, that is, even if the observation data matrix x is a mixed data matrix, the result of factorizing the observation data matrix x Therefore, the shape of the object can be estimated individually, and as a result, the shape of the driver's head can be estimated with high accuracy.

  Further, in the image processing apparatus 1, the feature point tracking unit 4 supplies information on the head feature points obtained by the individual face model learning unit 52. Arbitrary feature points can be selected based on the information of the head feature points, and unnecessary tracking processing can be executed based on the selection result, for example, by stopping tracking of non-head feature points. Can be prevented.

[Second Embodiment]
Next, a second embodiment will be described.

  In the present embodiment, only a part of the processing content in the individual face model learning unit 52 is different from that in the first embodiment, and thus this difference will be mainly described.

  Specifically, in the first embodiment, a so-called batch method is used in which factorization is performed in a lump after feature point data for the specified number of frames F has been obtained. A so-called online estimation method is executed in which θ, h, and z are sequentially estimated each time feature point data is acquired.

<Outline of online estimation method>
In the online estimation method, the free energy F _E shown in the equation (31) is replaced with an online free energy F _E λ in a form suitable for online estimation as shown in the equation (57).

However, the forgetting factor is λ (where 0 <λ <1), the current frame is f _c , the test distribution based on observations up to frame f is Equation (58), the sufficient statistics are Equation (59), and the forgetting factor is A coefficient for normalizing the influence is expressed by equation (60).

Comparing the equation (31) and the equation (57), the sufficient statistic r of the online free energy F _E λ is passed through f = 1, 2,..., Fc of the sufficient statistic r _f shown in the equation (61). It is replaced with a sum with attenuation.

Then, the sum of the sufficient statistics r _{f with} the attenuation coefficient can be calculated sequentially as shown in equation (62), and the normalization term is also calculated sequentially as shown in equation (63). Can do.

<Processing in the personal face model learning unit>
FIG. 9 is a flowchart showing the contents of processing executed by the individual face model learning unit 52.

  As in the case of the first embodiment, this process is activated by the input of an initialization command, and steps S310 to S340 are the same as steps S110 to S140 of the first embodiment, and thus description thereof is omitted.

  However, the specified number used in S130 may be a smaller number (for example, about a fraction to a fraction of a tenth) as in the first embodiment, and is 1 here.

Then, in S350, based on the observed data matrix x _f and the second trial distribution q (z, h) at S340 calculation result of (62) (63) using the formula, sufficient statistics << r >> _fc In the following S360, the calculation of the first test distribution q (θ), that is, the process as the M step of variational Bayes is executed, and the process proceeds to S170.

Specifically, the distribution parameter Θ _M shown in the equation (54) is changed to the sufficient statistic << r >> _fc calculated in S350, and the term of the sufficient statistic in the equations (47) to (52) (< Substituting into <>> fc term) and calculating.

In S370, the head feature point is extracted and notified to the feature point tracking unit 4 in the same manner as in S190. In the subsequent S380, the online free energy F _E λ is calculated according to the equation (57). In S390, as in S180. Then, on the condition that the calculated online free energy F _E λ is larger than a preset energy threshold, it is determined whether or not the termination condition is satisfied.

  If the end condition is not satisfied, the process returns to S330, and the processes of S330 to S380 are repeated. On the other hand, if the end condition is satisfied, the process proceeds to S400.

In S400, similarly to S200, and outputs the partial shape matrix based on the head feature point as an individual model face, the face model switch unit 53 as a face model C _a to be supplied to the estimating unit 6 selects an individual face model In this way, the switching signal is output, and this processing is terminated.

<Effect>
As described above, in the present embodiment, the processing for obtaining the first test distribution p (θ) and the second test distribution p (h, z) by executing the E step and the M step once for each frame. The factorization is performed using a so-called on-line variational Bayesian estimation method in which the results are multiplied and added by the attenuation coefficient η _fc so that the ratio decreases as the past data decreases.

  Therefore, according to the present embodiment, even if the number of image sequences to be used (the number of frames) is increased, the amount of processing increased by that is the amount of processing when the E step and the M step are executed once. As a result, since it increases only in an additive manner and does not increase exponentially, the processing load required for factorization (and thus the time required for processing) can be greatly reduced.

  As a result, an arithmetic unit mounted on an automobile generally has a low calculation capability, but even in such a case, it can be applied without any problem.

Further, in the present embodiment, the sufficient statistics << r >> _fc and the attenuation coefficient η _fc appearing in the calculation formula of the parameter that defines the first test distribution obtained in M steps are _simply used by using the calculation result in the previous frame. Therefore, the processing load required for factorization can be further reduced.

In this embodiment, if sufficient statistics << r >> _fc is obtained, the observation data matrix x _fc is not necessary, and the past observation data matrix is stored in order to cope with the increase / decrease of the feature points. Since it is not necessary, the storage capacity of the apparatus can be reduced.

[Experiment]
Processing is performed on the image processing apparatus 1 that executes the processing described in the second embodiment, with an input image series obtained by appropriately moving two objects (a quadrangular prism and a triangular prism) as shown in FIG. Thus, the shape matrix z was obtained. In addition, the vertex of each object was used as a feature point.

  FIG. 11 shows the transition of the estimation error (RMS error) of the shapes of two objects estimated from the shape matrix z obtained as a result of the processing, with respect to the number of observation frames. As a result, (b) is the result for the triangular prism.

  From FIG. 11, it can be seen that the estimation accuracy is low when the number of frames is small, but the estimation accuracy is improved as the number of frames increases, and that two objects can be correctly separated.

[Other Embodiments]
Although the embodiment of the present invention has been described above, the present invention is not limited to the above-described embodiment, and can be implemented in various modes without departing from the gist of the present invention.

  For example, in the above embodiment, an affine parameter is used to estimate the head posture. If the head posture can be estimated, for example, an extended Kalman filter is used without using the affine parameter. For example, a method of directly performing head estimation may be employed.

  In addition, as an end condition when terminating factorization, an energy threshold set based on the free energy obtained from the average face model is used. For example, the number of repetitions of the process is greater than a preset count threshold. A large value may be used as the end condition. In this case, since it is not necessary to calculate free energy to determine whether or not to end the processing, the processing load on the device can be further reduced, and the processing is always performed within a certain time. Can be terminated.

1 is a block diagram showing the overall configuration of an image processing apparatus to which the invention is applied. Explanatory drawing of a fixed feature point. Explanatory drawing of an arbitrary feature point. Explanatory drawing which shows the coordinate system which an image processing apparatus uses. The graph showing the state space model which a feature point tracking part and an estimation part handle. Explanatory drawing which shows the operation | movement outline | summary of time series Bayes estimation. Explanatory drawing which shows the graphic model which an individual face model learning part handles. The flowchart which shows the processing content of the personal face model learning part in 1st Embodiment. The flowchart which shows the processing content of the personal face model learning part in 2nd Embodiment. Explanatory drawing which shows the shape of the object used for experiment. The graph which shows the error of the shape of an object, and the shape obtained by experiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 2 ... Camera 3 ... Image capture device 4 ... Feature point tracking part 5 ... Face model supply part 6 ... Estimation part 7 ... Head posture calculation part 41 ... Fixed feature point tracking part 43 ... Arbitrary feature point tracking 51: Average face model storage unit 52 ... Individual face model learning unit 53 ... Face model switching unit

Claims (5)

Feature point extraction tracking that inputs an image series, extracts a plurality of feature points having a preset feature from each input image, and tracks the position of the feature point on the image plane for each of the feature points Means,
An observation data matrix generation means for generating an observation data matrix representing the position of the feature point according to the extraction and tracking results of the feature point extraction tracking means;
Decomposition means for factorizing the observation data matrix generated by the observation data matrix generation means to generate a shape matrix representing the three-dimensional shape of the object imaged in the image and a motion matrix representing the motion of the object; ,
In a three-dimensional shape restoration apparatus equipped with
The disassembling means includes
The observation data matrix is x, the shape matrix is z, the designation matrix for designating an object to which each of the feature points belongs, h, the motion matrix, the accuracy of measurement of the feature points, and the feature points for each object. A cubic is characterized in that θ, z, h are obtained by calculating a posteriori probability P (θ, z, h | x) from the observed data matrix x, where θ is a model parameter summarizing the probability of belonging. Original shape restoration device. The disassembling means includes
The test distribution q (θ, h, z), which is an approximate function of the posterior probability P (θ, z, h | x), is a first test distribution q (θ) and a second test distribution q (h, z). The first test distribution q () that minimizes the KL distance between the posterior probability P (θ, z, h | x) and the test distribution q (θ, h, z). 2. The three-dimensional shape restoration apparatus according to claim 1, wherein factorization calculation is executed stochastically by obtaining θ) and the second test distribution q (h, z). The observation data matrix generation means generates the observation data matrix x based on images for a preset number of frames,
The disassembling means includes
A first step of fixing the first test distribution q (θ) and estimating the second test distribution q (z, h), and fixing the second test distribution q (z, h) Estimating means for alternately and repeatedly executing the second step of estimating the first test distribution q (θ) using each other's estimation results;
Ending means for ending the estimating means when a preset ending condition is satisfied;
3. The three-dimensional shape restoration apparatus according to claim 2, wherein the shape matrix z and the designated matrix h are obtained from the second test distribution q (z, h) finally obtained. The observation data matrix generation means generates the observation data matrix x for each image of one frame,
The disassembling means includes
Each time the observation data matrix x is generated by the observation data matrix generation means, a first test distribution q (z, h) is estimated by fixing the first test distribution q (θ). Estimating means for executing the second step of estimating the first test distribution q (θ) by fixing the second test distribution q (z, h) using the estimation results;
Ending means for ending the estimating means and the updating means when a preset ending condition is satisfied;
3. The three-dimensional shape restoration apparatus according to claim 2, wherein the shape matrix z and the designation matrix h are obtained from the second test distribution q (z, h) sequentially obtained by the estimation means.   The three-dimensional shape restoration apparatus according to any one of claims 1 to 4, wherein the three-dimensional shape restoration apparatus is used by being mounted on an automobile.