JP4702811B2 - Lattice point model deformation / movement method, apparatus and program - Google Patents

Description

  The present invention relates to a technique for deforming and moving a predetermined target in a still image in accordance with the movement of another moving image in the fields of numerical simulation, computer graphics (CG), digital editing, and the like.

  In the fields of numerical simulation, CG, digital editing, etc., various models are known as representation methods related to the deformation and movement of an object. In particular, it calculates the interaction expression between a fluid such as air or liquid and the object. Many challenges remain with respect to stability and computation time.

  In the numerical simulation model, while an advanced numerical model exists, in order to realize stable calculation, it is necessary to reduce the step width of the calculation time, and there is a problem that the calculation time is enormous. In addition, in the case of a model that does not have a fixed point and takes into account deformation and interaction while moving, the calculation tends to be unstable.

  Even in CG, a model with a high speed is known, but there is a problem that the degree of freedom is extremely small, such as a deformed model that does not involve movement (Non-Patent Document 1). In digital editing, a model that can be expressed easily is required.

  Many of these are models in which grid points are still connected, and there is a limit of expression that requires large-scale numerical computation. A simple expression (model) in the case of simultaneously including collision (Non-patent Document 1), movement, and deformation is hardly seen.

  There are various optical flow methods for estimating a motion field of an object from an image, but most of them are for an object such as a rigid body (Non-Patent Document 2). For this reason, Non-Patent Document 2 has a problem that almost no correct motion can be detected for an object whose luminance changes in a short cycle such as a fluid.

S. Schaefer, T. Mcphail, and J. Warren "Image Deformation Using Moving Least Squares" ACM Proc, SIGGRAPH pp. 533-540, 2006. Otsuka, Minegoshi, Adachi, Suzuki, "Estimation of optical flow based on motion trajectory in space and time", IEICE, D-II, 81, 9, pp.2061-2073, 1998.

  The object of the present invention is to use a common object that does not depend on the rigidity, elasticity, fluidity, etc. of the object without using mathematical equations when a predetermined object in a still image is deformed / moved in accordance with the movement of another moving image It is to be able to express the model.

In order to achieve the above object, a lattice point model deformation / movement method of the invention according to claim 1 is similar to a still image including an object to be deformed / moved by a lattice point model deformation / movement device and a similar object . Preparing a moving moving image, obtaining a motion field from the moving image by a motion estimation method, setting a plurality of grid points not connected to the object in the still image, and independently obtaining the grid points. By repeating the movement phase for moving according to the motion field and the deformation phase for returning each lattice point after movement to a similar relationship by virtual connection within the time range, the deformation and movement of the object are repeated. A lattice point model deformation / movement method that expresses both , wherein the deformation phase changes the position of each lattice point after the movement so that the displacement before and after the movement of the distance between the lattice points approaches zero. Each of the ingredients And processing for correcting along the connecting direction for the movement of the front and rear displacement angles lattice points formed is characterized in that it comprises a process modified to approach zero.
The lattice point model deformation / movement device of the invention according to claim 2 is a data input / accumulation means for inputting and accumulating a still image including an object to be deformed / moved and a moving image similar to the object; A grid point setting unit that sets a plurality of grid points that are not connected to the target in the still image, a motion field setting unit that obtains a motion field from the moving image by a motion estimation method, and each grid point independently Grid point moving means for deforming according to the obtained motion field and returning it to a similar relationship by initial connection by virtual connection, and the grid point moving means determines the position of each moved grid point as a grid The x component and the y component are corrected along the connecting direction so that the displacement before and after the movement of the distance between the points approaches zero, and the displacement before and after the movement of the angle formed by the lattice points is corrected to approach zero. That features To.
According to a third aspect of the present invention, in the lattice point model deforming / moving device according to the second aspect of the present invention, the lattice point model deforming / moving device includes display means for displaying the moving / deformed result of the target and the moving image in an overlapping manner.
According to a fourth aspect of the present invention, in the lattice point model deforming / moving device according to the second or third aspect, each of the lattice points has no connection relationship, and the virtual connection for returning to a similar relationship to the initial state and the lattice point model The model is characterized in that the release including deformation according to the motion field is repeated within the time width, and the model includes the interaction with the nearby fluid in the moving image.
According to a fifth aspect of the present invention, in the lattice point model deforming / moving device according to the second or third aspect, the motion field setting means is a means to which an optical flow model of a fluid system is applied.
The invention according to claim 6 is the lattice point model deforming / moving device according to claim 5, wherein the optical flow model is a model for obtaining a motion field in consideration of luminance fluctuation between image frames. Lattice point model transformation / movement device.
A lattice point model deformation / movement program according to claim 7 is a method of preparing a still image including a target to be deformed / moved and a moving image that moves similar to the target, and a motion estimation method based on the moving image. Determining a motion field by the step, setting a plurality of grid points that are not connected to the object in the still image, and a movement phase for independently moving each grid point according to the determined motion field A program for causing a lattice point model deformation / movement device to execute a step of repeating a deformation phase in which each lattice point after movement is moved so as to return to a similar relationship with the initial state by virtual connection , and a lattice point model deformation / movement device In the deformation phase, the position of each lattice point after the movement is determined for each of the x component and the y component so that the displacement before and after the movement of the distance between the lattice points approaches zero. And processing for correcting along the forming direction, the movement of the front and rear displacement angles lattice points formed is characterized in that it comprises a process modified to approach zero.

  According to the present invention, by repeating a movement phase in which each lattice point is independently moved in accordance with a motion field and a deformation phase in which each lattice point after movement is returned to a substantially initial relationship within a time width, Since a simple model expression including the interaction with the object can be realized, it is possible to describe a common model that does not depend on the target rigid body, elasticity, fluidity, etc., and to realize an interaction expression with the fluid expression. In addition, it becomes unnecessary to solve complicated mathematical equations. Furthermore, it is possible to realize a new flow of processing from one image to animation. Further, the increase in the calculation time can be suppressed regardless of the increase in the number of lattice points.

  The present invention prepares a still image including a target to be deformed and moved and a moving image similar to the target, and uses the motion estimation method such as an optical flow method based on a fluid model from the moving image. Determining a field, setting a plurality of grid points not connected to the object in the still image, and moving each grid point independently according to the determined motion field and each grid after movement By repeating the deformation phase for returning the points to a substantially initial relationship within a time width, both the deformation and movement of the object are expressed.

  An embodiment of the device configuration for realizing this is shown in FIG. This apparatus includes a data input / storage unit 101 for inputting and storing a still image including a target to be deformed / moved and a moving image similar to the target, and the target in the still image. A grid point setting unit 102 for setting a plurality of grid points having no connection relationship; a motion field setting unit 103 for obtaining a motion field from the moving image by a motion estimation method such as an optical flow method based on a fluid model; and each grid point Grid point moving means 104 that returns the image to a substantially initial relationship after being deformed independently according to the obtained motion field, and display means 105 that displays the moving / deformation result of the object and the moving image in an overlapping manner. .

  These means can be realized by a computer having an arithmetic processing unit, an external storage device, a memory, and the like, and the lattice point model deformation / movement processing of this embodiment is executed by a program. This program is stored in an external storage device of a computer, and can be recorded on a recording medium or provided via a network.

  The program includes a step of preparing a still image including a target to be deformed / moved and a moving image similar to the target, and a motion estimation method such as an optical flow method based on a fluid model from the moving image. Determining a motion field by the step, setting a plurality of grid points that are not connected to the object in the still image, and a movement phase for independently moving each grid point according to the determined motion field A step of repeating a deformation phase for moving each lattice point after movement so as to return to an almost initial relationship within a time width, and a step of displaying the moving / deformation result of the object and the moving image in an overlapping manner. .

  Here, a lattice point model having no connection relationship as described above is handled. That is, the virtual connection and release of the lattice points are repeatedly performed within the time width, and a simple model expression including the interaction with the neighboring fluid is realized. The object is two-dimensional or three-dimensional. Moreover, in order to obtain the motion field from a moving image, a fluid optical flow method is applied.

  FIG. 2A is an explanatory diagram of a lattice point model having no connection relationship in the present embodiment. Here, only four grid points 201 to 204 arranged on a specific target of a certain image are shown. It is assumed that each of the lattice points 201 to 204 has a virtual connection relationship with each other. This time is set to t and is called coupling ON.

  Next, assuming that a nearby flow velocity field is given as a motion field, each of the lattice points 201 to 204 is moved independently according to the velocity of the flow velocity field. This is referred to as a movement phase and is referred to as coupling OFF.

  Next, the lattice points are moved in the deformation phase in the connection ON so that the lattice points 201 to 204 return to the distance and angle between the lattice points at the initial stage. This time is t + 1. This coupling ON and coupling OFF process is repeated until the required time. In the deformation phase for returning to an almost initial state, for example, the lattice points 201 and 202 are adjusted for displacement as shown in FIG. Further, for example, with respect to the lattice points 201 to 203, the angle adjustment is performed so that the angle is almost the initial angle as shown in FIG. The above is the same for other lattice points.

  Here, a virtual time is taken in between the discretized times t and t + 1. The relative positional relationship between the lattice points is determined from the initial lattice point data. Since each lattice point requires two processes of movement and deformation by the flow field, the deformation phase is processed in a virtual time. After the start of deformation and movement, the apparent hardness and softness can be easily changed by variously adjusting the degree of deviation (error) from the initial value. When this error is small, it can be freely changed, such as a rigid body and when it is large, an elastic body. As an image, when multiple people move while holding hands, release their hands once, move according to the flow field at each person's location, and repeat the connection with the original combination opponent, Even so.

FIG. 2D shows details of the deformation phase. Assume that the lattice points 201, 202, and 204 are A, B, and C, respectively. At time t, they are at positions A _(t) , B _(t) , and C _(t) . That is, in the two-dimensional xy coordinates, A _(t) = (x _a , y _a ) ^t , B _(t) = (x _b , y _b ) ^t , C _(t) = (x _c , y _c ) ^t is there. L _{ab (t)} is the distance between A and B at time t, L _{bc (t)} is the distance between B and C at time t, and θ _{abc (t)} is A, B, at time t. This is the angle formed by the three points of C.

When this is time t + 1, each of the lattice points 201, 202, 204 becomes (A _(t) , B _(t) , C _(t) ) → (A _{(t + 1)} , B _{(t + 1)} , C _{(t + 1)} ), the distance between A and B changes from L _{ab (t)} → L _{ab (t + 1)} , and the distance between B and C becomes L _{bc (t)} → L It changes to _{bc (t + 1)} , and the angle formed by the three points A, B, and C changes from θ _{abc (t)} → θ _{abc (t + 1)} .

At this time, if the distance between A and B is shortened,
ΔL _ab = | L _{ab (t + 1)} −L _{ab (t)} | (1)
Only displaced and
If the distance between B and C is long,
ΔL _bc = | L _{bc (t + 1)} -L _{bc (t)} | (2)
Displace only. Correction is made along the connecting direction of A and B and B and C so that this displacement approaches zero.

で表される。 Next, the connection direction can be easily obtained by vector expression. That is, a unit vector is created from the coordinate values. First, the unit vector e _ab between A and B is

It is represented by

As described above, the position of the point A is corrected so as to be farther from the point B, but the point B is also connected to the point C.
x _a ← x _a −ΔL _ab · e _ab · 0.5
x _b ← x _b + ΔL _ab・ e _ab・ 0.5 (4)
It becomes. The coefficient 0.5 is for halving the amount of movement between the two points, and is arbitrary. X _a and x _{b on} the left side of Equation (4) are corrected positions. Since a plurality of lattice points are connected, iterative calculation is repeated, for example, about 10 times until | ΔL _ab | <ε ₁ , | ΔL _bc | <ε ₁ and smaller than a predetermined value ε ₁ . The same applies to _xc , and the same applies to the y component.

図２(e)のように、ＡからＡ’に修正する場合は、

となる。ただし、θ＝Δθ_abc／２である。このΔθ_abcも予め決めたε₂以下になるまで、反復的に補正を行う。 Next, assuming that the angle is further opened from a certain angle,
Δθabc = | θ _{abc (t + 1)} −θ _{abc (t)} | (5)
It becomes. For the angle correction, the rotation matrix R may be used.

As shown in FIG. 2 (e), when correcting from A to A ′,

It becomes. However, θ = Δθ _abc / 2. Correction is repeatedly performed until this Δθ _{abc is} also equal to or less than a predetermined ε ₂ .

  FIG. 3 is an explanatory diagram for setting an object to be deformed / moved within a still image. FIG. 3 (a) shows a still image including a plurality of fallen leaves. As shown in FIG. 3B, seven fallen leaves are artificially selected from the still image, mask images 301 to 307 of the fallen leaves are set as targets, and lattice points are arranged in the respective mask images 301 to 307. To do. FIG. 3 (c) shows an example in which lattice points 310 are arranged at random positions so that the distances between the mask images 304 are equal to each other. The same applies to the other mask images 301 to 302 and 304 to 307. If necessary, the lattice points may be artificially arranged at unequal intervals.

  FIG. 4 is an explanatory diagram of a method for estimating the motion field 403 from two consecutive images 401 and 402. The more complex the motion field is set, the more realistic the target object will be transformed. Therefore, it is preferable to perform motion estimation using an actual moving image such as a river flow.

An example of the method is an optical flow method. In the case of a fluid pattern, it is appropriate to apply a physical model corresponding to the pattern. A model for estimating the optical flow using two images is shown in Equation (8). I is a two-dimensional image, Δ is a spatial first derivative, It is a time first derivative, and (i, j) are coordinates on the image. Α _{1 to} α ₃ are coefficients for weighting the luminance fluctuation between frames. u is a two-dimensional vector of the estimated optical flow.

この式(9)から、線形連立１次方程式を求めて解くことで、画像上の動きベクトル、即ち、オプティカルフローが推定される。 In Equation (8), the optical flow is estimated from the necessary condition for ε _{i, j to} be minimum in a certain image region. That is, it results in a minimization problem of the objective function E for unknowns (u, v).

From this equation (9), a linear simultaneous linear equation is obtained and solved to estimate a motion vector on the image, that is, an optical flow.

で表されるのもであった。すなわち、式(8)のように、三角関数による画像フレーム間の輝度変動に対応したものではなく、画像フレーム間での輝度恒常性が仮定されていた。これに対し本実施例では、画像フレーム間の輝度変動に重み付けを行って動き場を求めているので、精度を高めることができる。 The conventional optical flow method (references: Xugang, Saburo Tsubame, “3D Vision”, Kyoritsu Shuppan, pages 112-113, 2001)

It was also represented by. That is, as in the equation (8), it does not correspond to the luminance fluctuation between image frames due to the trigonometric function, but the luminance constancy between the image frames is assumed. On the other hand, in this embodiment, since the motion field is obtained by weighting the luminance variation between the image frames, the accuracy can be improved.

  FIG. 5 shows experimental examples of deformation and movement for various objects. 5 (a) to 5 (c), the motion field is estimated at each time from the torrent 501 of the background river. A plurality of fallen leaves 502 were cut out from another image, and the expression of flowing each fallen leaf 502 into the torrent 501 was performed using the distance and angle between the initial lattice points of each fallen leaf 502 as a reference amount. As a result, each fallen leaf 502 flowed downstream while performing deformation according to the flow velocity of the torrent 501 on the spot.

  Next, an example in which a plurality of circular particles 503 are caused to flow in the turbulent flow 501 in FIGS. At that time, as shown in FIGS. 5 (g) to (i), the texture (particles 503) to be attached to the lattice model was made similar to the translucent red blood cells 504. Further, the torrent 501 was replaced so that the blood vessel 505 could be imaged in the background. As a result, the individual red blood cells 504 moved while deforming. Moreover, the expression which collides while flowing was also able to be done appropriately.

  FIG. 6 is an explanatory diagram of collision between objects. As described above, in this embodiment, deformation and movement can be expressed by the same single model. In addition to this, there is an advantage that the model can be easily extended for collision.

  Here, there are two objects 601 and 602 as examples, and it is assumed that a plurality of grid points 603 and 604 are allocated in advance. The centroids of the lattice points 603 and 604 constituting the objects 601 and 602 are obtained with the centroids at certain times as stars 605 and 606, respectively. For each object 601, 602, the star marks 605, 606 can be regarded as the lattice points in the object described so far. That is, by setting the distance between the two stars 605 and 606 as an initial value, when the objects 601 and 602 are deformed and moved, the distance between the two stars 605 and 606 is a certain distance or more. Each grid point of the object 601 and each grid point of the object 602 are translated so that the distance is equal to or greater than the distance at the time of collision. As described with reference to FIG. 2 (a), a collision is expressed by performing between time t and time t + 1.

  FIG. 7 shows the results of a performance evaluation comparison experiment between the conventional mathematical model and the method according to this embodiment. As a conventional mathematical model, the simplest expression is an elastic model based on the hook rule. The lattice points were connected by virtual springs, the elastic coefficient was set to 5.0, and the same number of lattice points and objects as in this example were configured. Experiments were performed to compare the calculation time with respect to the number of lattice points when the fallen leaves 502 were caused to flow in the torrent 501 described with reference to FIGS. 5 (a) to 5 (c) and flowed from upstream to downstream. In this embodiment, the calculation time hardly increased until the number of grid points was on the order of 10 to the 10th power. On the other hand, in the conventional method, the calculation time increased exponentially. This indicates that the calculation efficiency of this embodiment is high.

It is a block diagram of the Example of the apparatus structure of this invention. It is explanatory drawing of the lattice point model without a connection relation of a present Example. It is explanatory drawing which sets the object which deform | transforms in a certain still image. It is explanatory drawing which estimates a motion field from two continuous moving images. It is explanatory drawing of the experiment example of the deformation | transformation with respect to various objects, and a movement. It is explanatory drawing of the collision between objects. It is a characteristic view which shows the result of the performance evaluation comparison experiment with the conventional mathematical model and the method of a present Example.

Explanation of symbols

101: Data input / accumulation means, 102: Grid point setting means, 103: Motion field setting means, 104: Grid point moving means, 105: Display means 201-204: Grid points 301-307: Mask image of fallen leaves, 310: Grid points 401, 402: Image, 403: Motion field 501: Rapid flow, 502: Fallen leaves, 503: Particles, 504: Red blood cells, 505: Blood vessels 601, 602: Object, 603, 604: Grid, 605, 606: Center of gravity

Claims (7)

A still image including a target to be deformed / moved by a lattice point model deforming / moving device and a moving image similar to the target are prepared, a motion field is obtained from the moving image by a motion estimation method, and the still image A plurality of grid points that are not connected to the object in the object are set, and each grid point is moved according to the obtained motion field independently, and each grid point after the movement is virtually connected A lattice point model deformation / movement method that expresses both deformation and movement of the object by repeating a deformation phase that returns to a similar relationship with the initial time within a time width ,
The deformation phase includes a process of correcting the position of each lattice point after movement along the connecting direction for each of the x component and the y component so that the displacement before and after the movement of the distance between the lattice points approaches zero, A method of deforming and moving a lattice point model , comprising: correcting so that a displacement before and after movement of an angle formed by a point approaches zero . Data input / storage means for inputting and storing a still image including a target to be deformed / moved and a moving image similar to the target, and a plurality of grids not connected to the target in the still image Grid point setting means for setting points, motion field setting means for obtaining a motion field from the moving image by a motion estimation method, and virtual connection after deforming each grid point independently according to the obtained motion field Grid point moving means for returning to a similar relationship with the initial stage ,
The grid point moving means corrects the position of each grid point after movement along the connecting direction for each of the x component and the y component so that the displacement before and after the movement of the distance between the grid points approaches zero, An apparatus for deforming and moving a lattice point model, wherein the displacement before and after the movement of the angle formed by the lattice points is corrected to approach zero . In the lattice point model deformation / movement apparatus according to claim 2,
A lattice point model deforming / moving device comprising display means for displaying the moving / deforming result of the object and the moving image in an overlapping manner. In the lattice point model transformation / movement apparatus according to claim 2 or 3,
Each of the lattice points has no connection relationship, and repeats virtual connection that returns to a relationship similar to the initial state and release that deforms in accordance with the motion field within a time range, so that each of the lattice points interacts with neighboring fluids in the moving image. Lattice point model transformation / movement device characterized by being a model including action. In the lattice point model transformation / movement apparatus according to claim 2 or 3,
The lattice point model deforming / moving apparatus according to claim 1, wherein the motion field setting means is a means to which an optical flow model of a fluid system is applied. In the lattice point model deformation / movement device according to claim 5,
The lattice point model deforming / moving apparatus according to claim 1, wherein the optical flow model is a model for obtaining a motion field in consideration of a luminance variation between image frames. Preparing a still image including a target to be deformed / moved and a moving image similar to the target;
Obtaining a motion field from the video by a motion estimation method;
Setting a plurality of grid points not connected to the object in the still image;
A movement phase in which each grid point is independently moved in accordance with the obtained motion field and a deformation phase in which each grid point after movement is moved so as to return to a similar relationship by initial connection by virtual connection are included in the time width. Repeat the steps in
Is a program for causing the lattice point model transformation / movement device to execute
The deformation phase includes a process of correcting the position of each lattice point after movement along the connecting direction for each of the x component and the y component so that the displacement before and after the movement of the distance between the lattice points approaches zero, A lattice point model transformation / movement program characterized by including a process for correcting the displacement of the angle formed by the points before and after the movement to approach zero .

Images