JP3453769B2 - 3D information restoration device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a three-dimensional information restoring apparatus for restoring three-dimensional information from a two-dimensional moving image of an object in a three-dimensional space.

[0002]

2. Description of the Related Art When an object in a three-dimensional space is represented on a screen, it is displayed on a two-dimensional screen. FIG. 2 is a diagram illustrating a situation where an object is represented on a two-dimensional screen. The point represented by the X, Y, and Z coordinates is represented by an (x, y) point on the image screen. When the Z value representing the depth is overlapped on the image screen, a small Z value is represented by an image. Show on screen. The following method is used as a method for estimating the three-dimensional motion of an object from the moving image represented in this manner.

For example, assuming the rigidity of an object, its motion is represented by a translation velocity vector [t _x , t _y ,
t _z ] and the rotation speed vector [w _x , w _y , w _z ]. At this time, the position vector of the point on the two-dimensional image is [x, y], and the velocity vector field (optical flow) that is the time derivative thereof is [x ', y']. FIG. 3 is a view for explaining the concept of the optical flow and shows the optical flow when the cylinder is rotated. The restoration equation is as follows.

[0004]

(Equation 1)

By solving equation (1), the motion (translation speed vector and rotation speed vector) and shape (depth image Z) of the object are calculated.

[0006]

However, in this method, since the optical flow noise is directly detected at each point in the depth image Z, the estimation accuracy of the object shape is not sufficient. The present invention has been made in view of the above-described problems, and has as its object to provide a three-dimensional information restoration apparatus that reduces the influence of noise on depth coordinate values by using data in many frames. .

[0007]

In order to achieve the above object, a rigid body motion extraction unit 1 for calculating a time series of a translation speed and a rotation speed of an object contained in a time-series image, and a motion law which the object follows. And a parameter describing a time series of three Euler angles representing the posture of the object is obtained when the rotational speed is obtained, based on the motion law set by the motion law setting unit 2 and the motion law set by the motion law set unit 2. A parameter describing a time series of the position of the center of gravity of the object is calculated from the time series of the translation speed and the rotation speed according to the motion law set by the motion law setting unit 2 and the rotation motion parameter calculation unit 3 calculated from the series. And a translational motion parameter calculation unit 4 that calculates a time series of the position and orientation of the center of gravity of the object in the input image.

A rigid body motion extracting unit 1 for calculating a time series of a translation speed, a rotation speed and a depth coordinate image of an object included in the time-series image, and a motion-law setting unit for setting a motion law to be followed by the object. 2 and the motion law setting unit 2
A rotational motion equation is obtained from the motion rule set in the above, and a rotational motion parameter calculating unit 3 for calculating a parameter describing a time series of three Euler angles representing the posture of the object from the time series of the rotational speed; The translational motion parameter calculator 4 calculates a parameter describing the time series of the center of gravity position of the object from the time series of the translation speed and the rotation speed, and the time series of the depth coordinate image calculated by the rigid body motion extractor 1 From the time series of the Euler angles calculated by the rotation motion parameter calculation unit 3 and the time series of the center of gravity calculated by the translational motion parameter calculation means 4 to a rigid fixed coordinate system that is a coordinate system with the center of gravity of the object as the origin. A coordinate conversion unit 5 for conversion, and an object surface shape calculation unit 6 for smoothing the shape of the object represented by the rigid fixed coordinate system
It is provided with.

Further, the motion law setting unit 2 sets the motion law so that the force received by the object is expressed by a linear expression of displacement.

[0010]

In order to reduce the influence of noise on the depth coordinate value, data of many frames (time-series data of the shape of the object) may be used. Changes with the movement of the rigid body,
It is not always easy to associate depth coordinate values between frames (that is, to integrate time-series data of object shapes consistently).

In order to solve this problem, the present invention uses a coordinate system (rigid rigid coordinate system) with the origin of the center of gravity of a rigid body to correspond data between frames. In order to realize this, it is necessary to convert the depth coordinate value into a value in a rigid fixed coordinate system. For that purpose, the position of the center of gravity and the azimuth (posture) of the object must be known every moment. Integrating the translational velocity vector and the rotational velocity vector as they are is undesirable because errors accumulate. Therefore, the time series data of these quantities and the functional form of these quantities obtained by introducing the knowledge of the equation of motion are used. By comparison, the coefficients of these functional forms are calculated to obtain the equation of motion. Based on this equation of motion, the position and orientation of the center of gravity of the rigid body at each time are calculated. By using this result, the depth coordinate value of each point on the image at each time can be mapped to the rigid fixed coordinate system, and the depth coordinate values calculated for many frames are averaged on the rigid fixed coordinate system As a result, the shape of the rigid body can be accurately restored.

FIG. 4 is an explanatory diagram for consistently integrating the time series data of the object shape. When the object is represented by a rigid fixed coordinate system,
The translational motion does not appear because it is the coordinate system itself, but only the rotational motion. 1, 2, and 3 represent the principal axes of inertia, and the origin coincides with the origin of the rigid fixed coordinate system. The left side shows the rotational movement at each time. When these three are overlapped, they are on the right, but the three circles do not match. This mismatch amount is noise, and without this noise, the three circles should match. Therefore, the present invention reduces the influence of noise by averaging depth coordinate values on rigid body fixed coordinates, and restores the rigid body shape with high accuracy.

The rigid body motion extracting unit 1 calculates a time series of the translation speed and the rotation speed of the object included in the time-series image. The rotational motion parameter calculation unit 3 sets a motion method based on the set motion law, calculates this coefficient from a time series of rotational speed, and obtains three Euler angles θ, ψ, φ, Is calculated. The translational motion parameter calculation unit 4 calculates a motion equation according to the set motion rule, calculates the coefficient from a time series of the translation speed and the rotation speed, and calculates the position of the center of gravity of the object. Thereby, the position and orientation of the center of gravity of the object can be restored.

In addition to the rotation speed and the translation speed, the rigid body motion extraction unit 1 also obtains a time series of a depth coordinate image. The coordinate conversion unit 5 expresses the time series of the depth coordinate image in a rigid fixed coordinate system from the time series of the Euler angles calculated by the rotation motion parameter calculation unit 3 and the time series of the center of gravity calculated by the translation motion parameter calculation unit 4. . The object surface shape calculation unit 6 smoothes the depth coordinate image (which represents the shape of the surface of the object) represented by the rigid fixed coordinate system by giving a predetermined condition. This smoothing corresponds to taking the average value of the data including noise shown in FIG.

As a law of motion, the force received by the object is set so as to be expressed by a linear expression of displacement. This means that the object (rigid body) moves in the potential expressed by the quadratic equation,
This corresponds to the case where no moment of force is applied.
This makes it easy to formulate the equations of rotational motion and is quite realistic. In addition, since the coordinates of the center of gravity of the rigid body correspond to the case of performing a simple vibration or a constant acceleration movement, an actual movement can be well reproduced.

[0016]

Embodiments of the present invention will be described below with reference to the drawings.
I do. FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.
is there. FIG. 5 shows the relationship between the coordinates of the laboratory system and the rigid fixed system.
FIG. In FIG. 5, the laboratory system is a normal X, Y,
A coordinate system having Z coordinates, and a rigid fixed system is an object (rigid
) And its origin is the center of gravity G of the rigid body
To take. In the laboratory systemP= (P_x, P_y, P_z)^tRepresented by
Is that the rigid fixed systemP ₀ = (P₁, P _Two,
P_Three)^tIs represented by The coordinates of the center of gravity of the rigid bodyG= (G_x,
G_y, G_z)^tAnd the time for the rigid fixed system
If the matrix representing the inversion is R, the following equation holds. In addition,
Underlined characters of P and G represent vectors.

[0017] _{P = G + R P 0 ...} (2) where a rigid fixed-line shall have the same orientation as the laboratory system (if the laboratory system is right-handed rigid fixation system is also right-handed).

In FIG. 1, a rigid motion extraction unit 1 calculates a time series of a translation speed, a rotation speed and a depth coordinate image based on the origin of a laboratory system from a time series of an input image. The motion rule setting unit 2 sets a motion rule according to the rotational motion and the translational motion of the center of gravity. The rotational motion parameter calculation unit 3 calculates a rigid body rotational motion equation from the set motion law, and calculates a coefficient (parameter) of the motion equation from a time series of rotation speed. The translational motion parameter calculation unit 4 calculates a motion equation of the center of gravity of the rigid body from the set motion law,
The coefficient of the equation of motion is calculated from the time series of the rotation speed and the translation speed. The coordinate transformation unit 5 obtains R shown in FIG. 5 from a time series of three Euler angles representing the azimuth of the rigid body obtained from the rotational motion parameter calculation unit 3, and the position of the center of gravity of the rigid body obtained from the translational motion parameter calculation unit 4 Then, the depth coordinate values are converted into the rigid fixed system coordinates. The object surface shape calculation unit 6 converts the converted depth coordinate values into polar coordinates and obtains the shape of the rigid body surface so as to satisfy the smoothness condition.

The three-dimensional information restoring method performed in the embodiment will be described below. First, the overall flow will be described. First, it is shown that the magnitude f of the rotation speed vector w calculated from the optical flow follows the equation of motion of unknown coefficients, and the coefficients are determined by the least square method. Next, the equation of motion is solved, and a matrix R representing the rotation of the rigid fixed system is determined based on the equation. Next, the equation of motion of the center of gravity of the rigid body is set based on the law of motion, and the coefficient of the equation of motion is obtained by the least squares method using the time series of the translation speed and the rotational speed to calculate the motion of the center of gravity G of the rigid body. Thereafter, the depth coordinate value Z is converted into a rigid fixed system coordinate using the matrix R and the time series of the center of gravity G of the rigid body. Finally, the depth coordinate value Z is selected such that the evaluation function to which the smoothness condition is added is minimized to obtain the rigid body surface shape.

The details will be described below. It is assumed that a time series including the noise of the translation speed vector and the rotation speed vector is given. At this time, a method of obtaining various parameters representing an exact solution of the motion of the rigid body from these time series data, and also obtaining a dynamic feature amount of the object is considered. The motion of the rigid body in the three-dimensional space is described by a rotation matrix (or Euler angle) representing the position of the center of gravity and the orientation of the rigid body. Now, it is assumed that the rigid body moves in a potential that can be expressed by a quadratic equation and receives no moment of force. (This shows that the force received by the rigid body is expressed by a linear expression of displacement.) At this time, an exact solution of the general state of rotation of the rigid body is obtained, and parameters (moment of inertia, moment of inertia,
(Initial state).

Now, it is assumed that a rotation matrix representing the direction of the rigid body is R. That is, a point P represented by (P ₁ , P ₂ , P ₃ ) in a coordinate system fixed to a rigid body having the center of gravity as the origin is represented as follows in a laboratory system.

[0022]

(Equation 2)

(G _x , G _y , G _z ) are the coordinates of the center of gravity in the laboratory system.

R is expressed by the following equation using Euler angles.

[0025]

(Equation 3)

At this time, the rotational speed vector is expressed as follows in the laboratory system and the rigid body fixed system, respectively.

[0027]

(Equation 4)

[0028]

(Equation 5)

Assuming that the inertia moment symmetric matrix is I, the generality is maintained even if it is expressed by the following equation.

[0030]

(Equation 6)

That is, the direction of the principal axis of inertia is selected as the coordinate axis of the rigid fixed system. In this case, since the rigid body does not receive the moment of force, the angular momentum L = Iw and the energy E = (w, L)
/ 2 is a constant, which can be used as a parameter representing an exact solution of the equation of motion.

From the conservation of the angular momentum, the equation of the rotational motion of the rigid body (Euler's equation) is obtained.

[0033]

(Equation 7)

Assuming that the direction of the angular momentum is Z axis and the magnitude is M, the following equation is established. w _z = 2E / M

An exact solution for the rotational motion equation of the free rigid body can be written down using the Jacobi elliptic function. M ²
Although the behavior of the solution differs depending on whether −2EI ₂ is positive, negative or zero, a case where M ² −2EI ₂ > 0 will be described here. Then, the following equation holds.

[0036]

(Equation 8)

The amount required for the measurement is (w_x, W_y, W_z)so
Yes, (w₁, W_Two, W_Three) Can not be asked directly
However, by combining these exact solutions,
(W_x, W _y, W_z) Is calculated from
Can be w_z= 2E / E ... (10)

[0038]

(Equation 9)

[0039]

(Equation 10)

Where Φ = tan ^-1 (w _y / w _x )

In equation (11), if w ² _x + w ² _y → f (2EI ₃ −M ² ) (M ² -2EI ₁ ) / I ₁ I ₃ M ² → b is replaced, equation (11) becomes It is expressed like an equation. f = −α ² k ² sn ² (αt + β, k ² ) + b (13)

First, the plane on which the locus drawn by the vertex of the rotation speed vector is found from equation (10) is obtained. The normal direction is the Z axis (direction of angular momentum), and the distance from the origin of the plane is 2E / It turns out that it is M. Using this result, w _x ² +
it is possible to calculate the w _y ² and Φ, M ² -2 than (12)
The sign of EI ₂ can be determined. Finally, if the coefficients α, β, k ² , and b are determined so as to best fit the equation (13), the moment of inertia, initial position, and initial velocity can all be determined from these and substituted into the exact solution. become.

The function f = −α ² k ² sn ² (αt + β,
Since it is difficult to obtain the coefficient of k ² ) + b at a time, a coefficient other than β is first determined. The function f is (f ′) ² = −4f ³ + Af ² + Bf + C (14) where A = 4 {3b−α ² (1 + k ² )} B = 4 (3b ² −α ⁴ k ² ) −2b A C = 4b ³ -Ab ² -Bb, the least squares method is used to minimize the sum of squares of the left side minus the right side of equation (14).
Once B and C are determined, α, k and b can be derived from these.

First, b has a relational expression of 4b ³ -Ab ² -Bb-C = 0 (15).
Can be requested.

Further, α and k are obtained by using the following relationship: α ² (1 + k ² ) = 3b−A / 4 (16) α ⁴ k ² = 3b ² −Ab / 2−B / 4 (17) I can do things.

Finally, β may be selected by the least squares method so that the following equation best fits the linear equation αt + β.

[0047]

(Equation 11)

Since equation (14) itself includes a term of the form (f ′) ² , errors may accumulate in one direction. In this case, it is also possible to evaluate using f ″ + 6f ² −Af−B / 2 = 0 (19) obtained by differentiating the equation (14).

Since the force received by the rigid body is represented by a linear expression of displacement, the coordinates of the center of gravity (X _G , Y _G , Z _G ) of the rigid body are whether to perform a simple vibration or a uniform acceleration movement. As will be described shortly, the latter is considered to be the extreme case of the former, and thus only the former case will be described here. At this time, the motion of the center of gravity is expressed by the following equation.

[0050]

(Equation 12)

Here, α ₁ , α ₂ and α ₃ are vectors orthogonal to each other.

In the equation (20), α ₁ , α ₂ , α ₃ →
Taking the limits of ∞, β ₁ , β ₂ , β ₃ → 0, etc., this represents a constant acceleration motion. The translation speed vector based on the origin of the laboratory system is expressed by the following equation.

[0053]

(Equation 13)

In order to determine the parameters of equation (20), first, α, β, and δ are selected by the least squares method so as to best fit the derivative of equation (21). (X _G0 , Y _G0 , Z _G0 ) may be selected by the least square method so as to fit. Thus, the equation of motion of the center of gravity of the rigid body is obtained.

The parameters describing the motion of the rigid body have been obtained by the above-described method.
_i a is fixed to the rigid body as the origin centroid coordinate values (L _i,
M _i , N _i ) can be converted using the following equation.

[0056]

[Equation 14]

Here, R is the Euler angle shown by the equation (4).

The depth coordinate values (L _i , M _i , N _i ) are the coordinates of the surface of the rigid body, and represent the surface shape of the rigid body. Therefore, these rigid body coordinate values are further displayed in polar coordinate display r _i (θ
_i , φ _i ), and r (θ, φ) is selected so as to minimize the evaluation function of the following equation to which the smoothness condition is added, and the shape of the rigid body surface is obtained.

[0059]

(Equation 15)

[0060] (23) in the evaluation function of the equation, it is equivalent with the following equation taking values averaged over time r _i instead of integration with respect to time. When the formula (24) is used for calculation, the calculation amount and the memory amount may be smaller.

[0061]

(Equation 16)

Next, the case where the present embodiment is used will be described in comparison with theoretically obtained values. A simulation was performed using a theoretically obtained optical flow time series for a case where a rigid body having a spheroidal shape freely moves in a gravitational field. Regular white noise of various magnitudes was added to the flow to estimate the three-dimensional motion and shape of the rigid body. In order to verify the effectiveness of the present embodiment, the estimation results of the present embodiment and a conventional example using equation (1) are compared.

In this simulation, a time-series optical flow consisting of 100 frames was used, and the motion parameters and the depth coordinate values at the final time were compared between the two methods. However, the translation velocity vectors were compared with their magnitudes normalized to one. The average size of the flow was evaluated for each frame, and five levels of noise were added to the flow in proportion thereto. Each calculation was performed 100 times, and the relative error of the difference from the true value was evaluated. The size of the image is 0.281 × 0.281 in units of the focal length, and is represented by 200 × 200 pixels. Assuming that the inside of the spheroid is uniform, the moment of inertia was set to 1, 2, 2.6. When the rigid body is closest to the camera, the rigid body is almost inscribed on the edge of the image, and when the rigid body is farthest from the camera, the maximum diameter of the rigid body is almost half the length of one side of the image. Rigid body motion was set.

FIG. 6 shows simulation results. In the figure, A indicates the depth coordinate value according to the present embodiment, and B indicates the depth coordinate value according to the conventional method using equation (1). According to the present embodiment, the calculation error does not increase so much even if the noise increases, while the conventional method sharply increases.

[0065]

As is apparent from the above description, the present invention obtains the relationship between a laboratory system and a rigid fixed system using the equation of motion based on the motion parameters of the rigid body calculated from the optical flow, By averaging the depth coordinate values in the system, the shape of the rigid body can be accurately restored.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an embodiment of the present invention.

FIG. 2 is a diagram illustrating a two-dimensional image display.

FIG. 3 is a diagram illustrating an optical flow.

FIG. 4 is an explanatory diagram of integrating time-series data of an object shape in a consistent manner.

FIG. 5 is a diagram showing a relationship between a laboratory system and a rigid fixed system.

FIG. 6 is a diagram showing a simulation result of the present embodiment.

[Explanation of symbols]

1 Rigid body motion extraction unit 2 Motion law setting section 3 Rotational motion parameter calculation unit 4 Translational motion parameter calculator 5 Coordinate conversion unit 6 Object surface shape calculation unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI G06T 7/20 G06T 7/60 150C 7/60 150 G01B 11/24 K (56) References Norio Tagawa, Takashi Torii, Toshio Endo, Closed-form three-dimensional rigid body motion recovery method using optical flow, IEICE Technical Report, Japan Institute of Electronics, Information and Communication Engineers, March 19, 1991, Vol. 90 No. 490 (PRU90-139) pp. 23-29 Kenichi Kanaya, Image Understanding-Mathematics for 3D Recognition-, Japan, Morikita Publishing, November 20, 1991, 1st edition, pp. 79-82, 115-128 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06F 17/17 G01B 11/24 G01P 3/36 G06F 3/153 G06T 1/00 G06T 7/00-7 / 60 JICST file (JOIS)

Claims (3)

(57) [Claims] 1. A rigid body motion extraction unit that calculates a time series of a translation speed and a rotation speed of an object included in a time-series image, a motion law setting unit that sets a motion law to be followed by the object, and the motion obtains a rotary motion type from the set motion law the law setting unit, and the rotational motion parameter calculation unit configured to output calculate the parameters describing the time series of the three Euler angles from the time series of the rotational speed representative of the orientation of the object by motion rules set by the law of motion setting unit,
And a translation parameter calculating unit which exits calculate the parameters describing the time series of the center of gravity of the object from the time series of the rotational speed and the translational speed, with respect to the object of the input image, and the position of the center of gravity of the object A three-dimensional information restoration apparatus, which calculates a time series of postures. 2. A rigid body motion extraction unit for calculating a time series of a translation speed, a rotation speed, and a depth coordinate image of an object included in a time-series image, and a motion law setting unit for setting a motion law to be followed by the object. when obtains the rotational movement equation from the set motion law of the law of motion setting unit, the parameters describing the time series of the three Euler angles representing the attitude of the object from the time series of the rotational speed
And rotational motion parameter calculation unit configured to output calculated by the law of motion, when said translational speed of the rotational speed
A translation parameter calculation unit, the time series of depth coordinate image calculated in the rigid body motion extractor was calculated in the rotational motion parameter calculator Euler for calculating the parameters describing the time series of the center of gravity of the object from the series from the time series of the calculated position of the center of gravity of the time series and the translational motion parameter calculating section corner, and a coordinate conversion unit for converting the centroid of the object to the rigid fixed coordinate system is a coordinate system with the origin, the rigid body fixed coordinate system Time series of depth coordinate image represented by
Three-dimensional information reconstruction apparatus characterized by comprising a <br/> the object surface shape calculating unit for averaging gives a predetermined condition. Wherein the law of motion setting unit, three-dimensional information according to claim 1 or 2, wherein the force received by the said object is obtained by setting the law of motion as represented by a linear equation of the displacement Restoration device.