JP6196597B2 - Physical motion estimation method and apparatus - Google Patents

Description

  The present invention relates to a method and apparatus for estimating the movement of a moving object, and more specifically, by using image processing to estimate the movement characteristics and status changes of a moving object in a monitoring image of a person, a car, a river, and the sea. Relates to a method and apparatus.

  In order to understand the situation of the environment in the real environment, it is one of the important means to analyze the time change of the movement and deformation of the remotely moving object through the camera image etc. which photographed the moving target object. Yes. On the other hand, among moving objects, there are many objects such as those that cannot be equipped with sensors, a large number of unspecified people, cars and clouds, natural phenomena such as waves, cells in the body, and red blood cells. In particular, the rigid system can follow the center of gravity, whereas the fluid system does not have a center of gravity. Therefore, it is a problem to accurately detect changes in the shape and texture of the surface. As for the rigid system, if a large number of moving bodies appear to be folded on the line of sight, attention is transferred to the flow as a whole rather than individual movements, so it is treated in the same way as a fluid system, called dynamic texture. There are many.

J.L.Barron, D.J.Fleet, and S.S.Beauchemin, “Performance of optical flow techniques”, IJCV, vol. 12, no. 1, pp. 43-77, 1994. Seiji Horikawa, [New] “Coastal Engineering”, University of Tokyo Press, 1981.

  The apparent motion vector on the estimated image is also referred to as an optical flow or a flow. Many models of optical flow can estimate the flow in a smooth flow, but for the discontinuous flow that occurs in bifurcation, passing, collision, etc. There is a problem that it is difficult to obtain an optical flow model that understands.

  Many methods for estimating an optical flow based on a fluid model have been shown. Specifically, there are Navier-Stokes (NS) equations and continuous equations that are generally used in numerical simulations. However, NS can derive continuous and discontinuous flow velocities as a computational solution, while the objective function minimization problem when estimating flow is discontinuous as a solution. The flow cannot be obtained. This is because, in the minimization problem, the condition for the solution to converge in the iterative calculation includes that the NS equation solution is as smooth as possible. The NS formula is a model that permits a turbulent flow up to a few thousand Ray Ray numbers (Re) indicating fluid characteristics. Therefore, it has been one of the problems to stably and accurately estimate a motion (optical flow) accompanied by a strong turbulent flow with Re of 5000 or more as seen in a natural phenomenon. Further, the constraint condition that the time-series change in luminance of an image that is widely used is constant has a problem that the application range is limited.

  Motion estimation of moving objects occupies an important position in computer vision (CV). As another basic motion detection method, a cross-correlation (CC) method (pattern matching method) used in the field of encoding typified by MPEG is known (for example, Non-Patent Document 1). In the CC method, similarity between two consecutive images is evaluated by a cross-correlation function or the like, and a similar region is regarded as a movement distance (vector) between images. In order to improve accuracy, the image is divided into a plurality of blocks, and the movement distance of each block is obtained. Here, the CC method is premised on that there is no luminance variation between similar images. However, in practice, it is known that since the apparent brightness of the surface of the moving object changes with deformation outdoors, it is not possible to simply associate images.

  Numerous constraint conditions have been proposed and used in the problem of estimating the motion (optical flow) of a moving object (see Non-Patent Document 1). Most of them relate to the smoothness of the flow, and the degree of smoothness is the same or fixed regardless of the difference in the application target, the image area, and the texture. Specifically, as a constraint condition, it is typical to perform spatial primary differentiation and secondary differentiation of the optical flow model. Therefore, conventionally, there has been a limit to improving the accuracy of motion estimation in an image. In particular, there was difficulty in discontinuous motion estimation.

  In order to solve such problems and problems, a first aspect of the present invention is a method for simultaneously estimating continuous and discontinuous movements of a moving object, the time series of the object moving by a data input unit A step of capturing an image, a step of storing image data of the time-series image in a data storage unit, and a step of estimating a model parameter of the moving object, wherein the model parameter is a time-series of luminance of the image It is estimated by an objective function of a mathematical model for waves that expresses the characteristics of wave phenomena including continuous and discontinuous motion, which integrates the wave generation equation that expresses various changes and the optical flow model that is a motion estimation model in image processing. The objective function includes a constraint condition based on a wave dispersion relational expression, and the estimating step includes the step of estimating the image data in a first optimization unit. A first optimization step for roughly estimating the model parameters in the framework of a minimization problem of the objective function by applying a robust function for mitigating the influence of noise to the objective function, and a second optimization A second optimization that optimizes the number of terms of the wave generation equation and a model parameter that is a size of an image area to which the objective function is applied based on a parameter statistic estimated by the first optimization calculation. And a convergence determination unit that obtains a frequency distribution related to the parameter based on the estimated model parameter, and compares the parameter with the statistical model set in advance. Determining similarity with the model, and determining that the similarity between the model parameter and the statistical model is high, If it is determined that the model parameter has reached a predetermined accuracy, the step of displaying the final result of the model parameter on the display unit, and it is determined that the similarity between the model parameter and the statistical model is low, the wave And changing the number of terms of the generation equation and the model parameter which is the size of the image area to which the objective function is applied, and then returning to the first optimization step.

  According to a second aspect of the present invention, there is provided the method according to the first aspect, wherein the model parameter includes vertical and horizontal components of an optical flow, a wave number, a frequency, and a phase. And

  A third aspect of the present invention is the first or second aspect, characterized in that the number of terms in the constraint condition expression changes in conjunction with the number of terms in the objective function.

  In addition, a fourth aspect of the present invention is any one of the first to third aspects, wherein the convergence determination unit uses, as a statistical distribution model, a feature relating to movement specialized for each moving object. It is characterized by.

  According to a fifth aspect of the present invention, there is provided an apparatus for simultaneously estimating continuous and discontinuous movements of a moving object, a data input unit for capturing a time-series image of the moving object, and an image of the time-series image. A data storage unit that stores data, and an optimization unit that estimates a model parameter of the moving object, the model parameter including a wave generation equation representing a time-series change in luminance of an image, and image processing It is estimated by the objective function of a mathematical model for waves that expresses the characteristics of wave phenomena including continuous and discontinuous motion, integrated with the optical flow model, which is a motion estimation model. The optimization unit includes a constraint condition, and the optimization unit applies a robust function for reducing the influence of noise of the image data to the objective function, thereby minimizing the objective function. A first optimization unit that roughly estimates the model parameters in a framework, and a model parameter that is the size of an image region to which the number of terms of the wave generation equation and an objective function are applied is estimated by a first optimal calculation A second optimization unit that optimizes based on a parameter statistic, and obtains a frequency distribution related to the parameter based on the estimated model parameter, and compares it with a statistical model set in advance. A convergence determination unit that determines the similarity between the model parameter and the statistical model by performing matching, and when determining that the similarity between the model parameter and the statistical model is low, the term of the wave generation equation And the model parameter, which is the size of the image area to which the objective function is applied, is changed, and the moving object model is moved to the first optimization unit. When the convergence determination unit and the convergence determination unit determine that the model parameter and the statistical model are highly similar, the final determination is made that the model parameter has reached a predetermined accuracy. And a display unit for displaying the result.

  The sixth aspect of the present invention is the fifth aspect, wherein the model parameter includes vertical and horizontal components of an optical flow, a wave number, a frequency, and a phase. .

  The seventh aspect of the present invention is the fifth or sixth aspect, wherein the number of terms in the constraint condition expression changes in conjunction with the number of terms in the objective function.

  In addition, an eighth aspect of the present invention is any one of the fifth to seventh aspects, wherein the convergence determination unit uses, as a statistical distribution model, a feature relating to a movement specialized for each moving object. It is characterized by.

  According to the present invention, in addition to continuous motion, such as a flow with a branch, a flow with a reflection at the time of collision, and a flow with a vortex rotation, which are not obtained by the conventional motion estimation method, a discontinuous motion is obtained. It is possible to estimate a motion in a time-series image having a flow that accompanies motion.

It is a block diagram which shows the structure of the motion estimation apparatus which concerns on one Embodiment of this invention. It is a flowchart which shows the flow of a process of the motion estimation method which concerns on one Embodiment of this invention. It is a figure which shows the concept regarding the method of matching based on a wave physical model from two continuous images ((a) and (b)) which copied the moving object. (A) and (b) are two continuous images to be moved, (c) is a diagram showing the basis of a flow vector in the image of (a), and (d) is a flow in the image of (b). It is an image which shows a vector. It is a figure which shows the image divided | segmented in the image of the moving object in this invention. (A) The image divided | segmented in the image of an object is shown, (b) is an image which shows the flow vector of a division part. It is a figure which shows the histogram distribution used in the convergence determination in this embodiment. (A) is a figure which shows the experimental data (a) of energy spectrum distribution of a bread schneider. (B) is a figure which shows histogram distribution when M is changed in flow estimation. It is a figure which shows a mode when a turbulent flow is image | photographed with the image of the ray noise of several thousand or more from the left to the right, and the flow collided with the obstacle in the middle. (A) shows motion estimation by the conventional method, and (b) shows motion estimation by the present invention. It is a figure which shows the motion estimation in a spiral time series image. (A) shows motion estimation in the conventional method, and (b) shows motion estimation in the present invention. It is a figure which shows the result of the estimation error evaluation of the flow by the conventional method and this invention method.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a motion estimation apparatus 100 according to an embodiment of the present invention. The motion estimation apparatus 100 includes an image data input unit 110 and an image data storage unit 120 that stores image data from the image data input unit 100. In addition, the motion estimation apparatus 100 uses the image data stored in the image data storage unit 120 and the fluid model for estimating the motion of the moving object, and performs model estimation of the fluid model to perform motion estimation. And a second optimization unit 140 that further optimizes the model parameters based on the model parameters estimated by the first optimization unit 130. . In addition, the motion estimation apparatus 100 includes a convergence determination unit 150 that determines whether the model parameter optimized by the second optimization unit 130 has reached a predetermined system, and a model parameter that has reached a predetermined accuracy. Parameter change for changing the value of the model parameter in order to perform motion estimation again when the model parameter optimized by the display unit 160 for displaying the value and the second optimization unit 130 does not reach a predetermined system Unit 170. The image data input unit 110 is connected to the camera 115. The camera 115 may be included in the motion estimation device 100 or may be connected to the outside.

  FIG. 2 is a flowchart showing a process flow of the motion estimation method according to the embodiment of the present invention. In this embodiment, first, in step 210, image data such as a time-series image or a moving image of an object obtained by acquisition from a camera or a sensor is captured from the data input unit 110. In step 220, the captured image data is stored in the data storage unit 120.

  Next, in step 230, a fluid model for estimating the motion of the moving object is defined. Here, the fluid model in the present embodiment is a mathematical model relating to a wave that expresses characteristics of a wave phenomenon including continuous and discontinuous motion, and a time-series change in the luminance of an image is a term of a plurality of trigonometric functions. The wave generation equation consisting of the sum and the optical flow model, which is a motion estimation model in image processing, are integrated and defined as an objective function. For the fluid model, first, a wave generation equation is defined in which a time-series change in luminance of an image is made up of the sum of a plurality of trigonometric terms. Next, the defined wave generation equation is applied to the optical flow model, and the constraint condition by the wave dispersion relational expression for improving the stability of the solution and the accuracy of the flow vector estimation, and the noise and outlier of the image data Define an objective function by applying a robust function that mitigates the effects.

  Next, in step 240, the first stage of optimization is performed. In the first stage optimization, the first optimization unit 130 applies the objective function to the image data captured in step 220, and the noise and outliers of the image data are applied according to the framework of the objective function minimization problem. The model parameters are estimated roughly. Specifically, the vertical and horizontal components of the flow vector of the optical flow model, as well as the wave number, frequency, phase, and apparent thickness of a wave in a fixed image region are estimated.

  After roughly estimating the model parameters in step 240, a second stage of optimization is performed in step 250. In the second stage optimization, the second optimization unit 140 estimates model parameters such as the number of wave generation equation terms and the size of the image area to which the objective function is applied in the first stage optimization. Further optimization is based on parameters, especially statistics such as frequency.

  In step 260, a frequency distribution for the parameter is obtained based on the estimated model parameter.

  Thereafter, in step 270, convergence determination is performed. In the convergence determination unit 150, the frequency distribution related to the parameter obtained in step 260 is compared with the statistical model set in advance, and the similarity between the estimated model parameter and the statistical model set in advance is determined. judge.

  If it is determined in step 270 that the similarity between the estimated model parameter and the statistical model is high, it is determined in step 280 that the model parameter has reached a predetermined accuracy, and the final result of the model parameter is displayed on the display unit. 160.

  On the other hand, when it is determined that the similarity between the model parameter estimated in step 270 and the statistical model is low, the model parameter such as the size of the image area and the number of terms is changed in step 290 and then step 240 is performed again. Returning to step 1, the first optimization and the second optimization are performed to optimize the model parameters. By performing the optimization calculation repeatedly, it is possible to simultaneously estimate flows including continuous and discontinuous waves.

  In the above, the model parameter includes the wave number, frequency, and phase in addition to the vertical and horizontal components of the optical flow. In addition, the number of terms in the constraint condition changes in conjunction with the number of terms in the objective function, and furthermore, the convergence determination unit can use a feature relating to movement specialized for each moving object as a statistical distribution model. .

  FIG. 3 is a diagram illustrating a concept regarding a method for performing association based on a wave physical model from two consecutive images in which a moving object is copied. In FIG. 3, the moving object is exemplified by waves on the water surface. In step 210 of FIG. 2, image data such as a time-series image or a moving image of an object obtained by acquisition from a camera or a sensor is captured from the data input unit. Among the captured images, two consecutive images are (a) (time t-1) and (b) (time t) in FIG. In step 220 in FIG. 2, the image data of FIGS. 3A and 3B, which are two consecutive images that have been captured, are stored in the data storage unit.

  Next, in step 230 of FIG. 2, a fluid model for estimating a motion vector (flow vector) of a moving object in the captured image is defined. The fluid model of the present embodiment is a mathematical model related to a wave that expresses characteristics of a wave phenomenon including continuous and discontinuous movements, and includes a mathematical model (wave generation equation) related to a wave physical phenomenon (see Non-Patent Document 2), It is an objective function integrated with an optical flow model, which is a motion estimation model in the framework of image processing or computer vision (CV). The moving object viewed from the camera line of view is often accompanied by deformation with time. In particular, in natural phenomena, clouds, waves, smoke, and the like are typical examples of moving objects with deformation. Since the apparent surface texture, contour line, and the like of these moving objects change with time, a model is required to associate the moving objects between images and to estimate a flow vector.

  Therefore, regarding the object to be moved or deformed in the image, the height (image brightness) of the apparent moving object at the two-dimensional position (x, y) and time t is set to H (x, y, t). In addition, waves in a certain image region approximate different wave numbers, frequencies, and phases as a linear overlap of various waves. Then, the luminance of the wave image at a certain time t is

It is expressed by the wave generation equation. Here, M is the number of terms of the linear sum. A _m is the amplitude of the m-th wave, (k _{x, m} , k _{y, m} ) is the wave number of the m-th wave, f _m is the frequency of the m-th wave, and θ _m is the m-th wave. Is the phase of the wave. A method for determining the number of terms M will be described later.

  In FIG. 3, when two continuous wave images (time t-1 and t) are input, in order to estimate a local flow vector in the wave image, the two images are associated with each other. . First, at time t−1, a plurality of bases are given in a partial region of the acquired wave image (FIG. 3A) (multiple bases: FIG. 3C). Next, in the wave image acquired at time t (FIG. 3B), the object at the position corresponding to the base in FIG. 3C moves and deforms with respect to FIG. Here, a vector between the base in FIG. 3A and a position corresponding to the base in FIG. 3B is regarded as an estimated flow vector (FIG. 3D). In the case of a fluid phenomenon such as a wave in FIG. 3, there is a problem that the association is disturbed due to the movement and deformation of the moving object. However, an objective function that takes into account that the luminance changes according to Equation (1) is added. As a result, the disturbance of association between images is alleviated. A local image region is also expressed by a mathematical model made up of a plurality of trigonometric function bases as shown in equation (1).

  Next, the framework of the optical flow method in CV will be described in order to incorporate the characteristics of the wave generation equation of equation (1) into the optical flow model. Assume that the luminance of one input image is defined by I (x, y, t), and the luminance change per unit time is not constant. That is,

It is. Subsequently, for the amount of change in luminance, using the mathematical model of Equation (1),

It is expressed. Here, the estimated flow vector is

far. “Tr” indicates transposition of a vector. On the left side of equation (3), when Taylor expansion is performed,

It becomes. Here, Ix, Iy, and It are first-order derivatives with respect to x, y, and t, respectively.

  From the above, when formula (3) is organized,

Get. here,

It is.

  In preparation for incorporating a mathematical model (wave generation equation) into the objective function described below, Equation (5) is defined as an error function of a quadratic form.

  It is known that the stability of the solution and the accuracy of the flow vector estimation can be improved by adding a constraint condition related to the flow vector in the estimation of the flow vector. Therefore, also in the present invention, the objective function is defined in consideration of the constraint condition regarding the flow vector. Note that the mathematical model of Equation (1) includes both wave continuity and discontinuity expressions. Therefore, a dispersion relational expression is introduced into the objective function as wave physical characteristics related to velocity. This dispersion relational expression indicates that the angular frequency increases as the wave number increases. The dispersion relation is

Defined by Here, ω is an angular frequency, k is a wave number, h is an apparent thickness, and g is gravity.

  Although it is difficult to find the apparent depth directly, in the first stage optimization described later, the flow vector is set so that the three parameters k, h, and g in Equation (7) are balanced. Will be estimated.

  For other basic variables, 1 / f is the period and L = 2π / k is the wavelength. The scalar speed c is expressed as ω = ck. In the present invention, in order to make tanh (x) easy to handle in Equation (7), Taylor expansion is used.

Approximated by However,

It is.

  Thus, equation (7) becomes

Can be defined. Where c is a flow vector,

Related. As described above, the following is obtained as a constraint on the new speed from the equations (8) and (9).

However,

here,

Is the Euclidean norm.

  In the first stage optimization (step 240 in FIG. 2), model parameters are estimated, but the data contains noise. Therefore, in order to reduce the influence of this outlier, the objective function is defined by applying a robust estimation method. In the present embodiment, a Lorentz function is selected as the robust function ρ. σ is called a shape parameter and is set to 5.0. In addition, the weighting coefficient is set to η for the influence of the expressions (6) and (10). In this embodiment, this η is optimized. Further, the two-dimensional image area is set to Ω. From the above, the final objective function of the present invention is

It becomes. However,

It is.

In the optimization of the first stage (step 240 in FIG. 2), a model is used in the framework of the minimization problem of the objective function (equation (11)) to which the robust function is applied in order to reduce the influence of noise and outliers of the image data. Estimate the parameters roughly. Specifically, eight unknowns are estimated using the objective function (Formula (11)) defined from Formula (6) and Formula (10). Here, the eight parameters estimated in the first optimization are u, v, k _x , k _y , f, θ, and h. For two consecutive images, a plurality of local parameters are calculated from corresponding constant image regions of two images at different times, and each is substituted into the objective function to minimize the objective function. By minimizing the objective function, the eight parameters are estimated in a well-balanced manner. Formula (11) can be solved as a minimization problem by giving initial values of M and Ω, and a flow vector can be estimated for each Ω from image data.

In the equation (11), ρ _img in the first term and ρ _{vel in} the second term are robust functions, respectively. Equation (11) can be solved even when η = 0. e ₀ and e _{1, m} have a plurality of bases composed of M terms. e ₀ FW has different wavenumbers within a certain image area

To include M section thickness _{h m} of the apparent _{and, e 1,} the number of terms _m also M. On the other hand, if the number of terms M is estimated,

The number of terms M also changes. The size of the image area was optimized as follows. In the first optimization, Equation (11) can be minimized by a conjugate gradient method (CG) given an initial value | Ω | and M. In order to improve convergence, σ in the robust function was decreased from a constant value along with the number of iterations. In the equation (11), the first derivative values are used for nine unknowns. The iterative calculation was continued until the iteration error was less than 0.001. In the first stage optimization, | Ω | = 10 × 10 pixels and M = 3 were given as initial values in the iterative calculation. The initial values of the nine unknowns were set to 0.

  In order to proceed with the calculation, the image I (x, y, t) was discretized in the calculation space. here,

I put it. However, n = 1, 2,...

Δx = Δy = Δt = 1.0
The partial differential values of x, y, t of I (x, y, t) are obtained by the finite difference method.

I put it. Except for M, all unknowns are real numbers. The size | Ω | pixels is smaller than one image, and the calculation was performed while overlapping Ω.

  In the second stage optimization (step 250 in FIG. 2), the model parameters such as the number of terms of the wave generation equation and the size of the image area to which the objective function is applied are determined by the frequency estimated in the first stage optimization. Optimize based on such statistics. In the second stage of optimization, for the optimization of M and | Ω |, the number of occurrence frequencies in the image of the frequency estimated by the first optimization is given.

  FIG. 4 is a diagram showing an image divided in an image of a moving object in the present invention. In FIG. 4, a hurricane image having a spiral is taken as an example of image division.

  When M is large, the number of terms of the polynomial in equation (1) increases. By increasing the number of terms in the polynomial, linear superposition of cos-sine bases can be performed, and the surface and texture of different images can be approximated widely. On the other hand, when the size | Ω | of the two-dimensional image region Ω increases, the flow vector becomes smoother. Therefore, in order to obtain local motion, it is necessary to reduce | Ω |. Therefore, the flow vector estimation accuracy is improved by optimizing the magnitudes of M and | Ω |. In the hurricane image swirling as shown in FIG. 4A, a plurality of small images | Ω | FIG. 4B shows a part of the estimated flow as an example.

  In the objective function of Equation (11), wave physical characteristics are included as two constraints, but M and | Ω | are optimized only by the objective function of Equation (11) (first stage optimization). Difficult to do. Therefore, in the present embodiment, a convergence determination (step 270 in FIG. 2) is performed for the estimated model parameter. Convergence determination is a frequency distribution of parameters based on estimated model parameters using a statistical model in which wave phenomena with various directions, sizes, and scales in the real environment are observed in the framework of wave physics. (Step 260 in FIG. 2) is performed by comparing with a preset statistical model. One such statistical model is the breadth Schneider energy spectrum distribution,

It is described. Here, H _* and T _* are constants, H _* is called a significant wave height, and T _* is called a significant period.

  FIG. 5 is a diagram showing a histogram distribution used in convergence determination in the present embodiment. FIG. 5A is a diagram showing experimental data (a) of the energy spectrum distribution of Bread Schneider. FIG. 5B is a diagram showing a histogram distribution when M is changed in flow estimation. As shown in FIG. 5A estimated from the equation (11), the energy spectrum distribution of the bread schneider is unimodal, the vertical axis indicates the dimensionless frequency, and the horizontal axis indicates the energy spectrum. The distribution band is wide and the central peak is slightly on the low frequency side. A plurality of parameters are estimated from the time-series image and the objective function of equation (11), but it is assumed that the energy changes for each frequency. For this, a histogram is created from the number of pixels obtained for each frequency in the image. This is obtained for each local image region of | Ω |. Subsequently, in order to obtain a global histogram, an average value was obtained for all | Ω |. At this time, normalization was performed so that the area of the average value of the histogram was 1.

  This makes it possible to compare the similarity between the Bread Schneider distribution of Expression (12) (FIG. 5A) and the histogram distribution estimated from Expression (11) (FIG. 5B). In FIG. 5B, the histogram distributions when M = 1 to 6 are changed are displayed in an overlapping manner. In FIG. 5B, the larger M is, the better the matching with the graph of FIG. At this time, | Ω | is an example of 10 × 10 pixels. In step 270 of FIG. 2, when it is determined that the similarity between the model parameter and the statistical model is high, that is, when the deviation between the model parameter and the statistical model is within a predetermined range, the predetermined accuracy is reached. It is determined that On the other hand, if it is determined that the similarity between the model parameter and the statistical model is low, that is, if the deviation between the model parameter and the statistical model is within a predetermined range, the size of the image area and the number of terms After changing the model parameters (step 290 in FIG. 2), the process returns to the first stage optimization (step 240 in FIG. 2) again to optimize the objective function. In this way, the similarity of the histogram distribution is obtained for all combinations by changing M and | Ω | within a certain range.

_{Assuming that} Est (f) _{M, | Ω |} is the histogram distribution estimated by Expression (11), the histogram distribution can be evaluated by the normalized correlation coefficient CR (•, •). That is,

However,

As CR (·, ·) approaches 1.0, B (f) and Est (f) _{M, | Ω |} have similar values. For each set of time-series images, M and | Ω | when the CR value is maximized are selected and optimized locally and globally, and the accuracy of flow estimation is improved.

  In the convergence determination, other statistical models can be applied. For a specific object, a statistical model can be created and applied from prior information (knowledge).

  6 and 7 show the results of motion (flow) estimation from time-series images according to the conventional method and the method of the present invention. Among these, FIG. 6 is a diagram showing a state when a turbulent flow is imaged with a high number of ray noises of several thousand or more from the left to the right, and the flow collides with an obstacle in the middle, (a) Motion estimation according to the conventional method, (b) shows motion estimation according to the present invention. In the motion estimation of the conventional method shown in FIG. 6A, a slightly smooth flow is estimated. However, in the motion estimation of the present invention shown in FIG. A continuous flow was obtained.

  As a next example, motion estimation in a spiral time-series image is shown in FIG. In the image of FIG. 7, (a) shows motion estimation in the conventional method, and (b) shows motion estimation in the present invention. The flow vector estimated by the motion estimation according to the present invention in FIG. 7B can obtain the vortex flow structure more clearly than the motion estimation of the conventional method shown in FIG. Since the central part of the vortex shows a locally discontinuous movement, it is considered that the characteristic could not be sufficiently captured under the smooth movement constraint condition of the conventional method shown in FIG. Like the texture of the vortex, the pattern of the surface frequency is different at each location. In the present invention, since the approximation by the number of terms of the trigonometric function changes in the optimization, the vortex center from the periphery It is thought that the flow was obtained.

  FIG. 8 shows the results of flow estimation error evaluation according to the conventional method and the method of the present invention. A time series image is generated by numerical simulation in the motion of vortex and discontinuity, and the average error of the conventional method (average error per pixel of correct flow and estimated flow) and comparison evaluation by the present invention are quantitative. The flow estimation error was improved. In addition, the correct optical flow for obtaining the estimation error is the one in which the motion information is 100% known in advance by numerical simulation, or the one in which an actual image is visually picked up in a specific narrow region. Yes.

  The present invention can estimate the movement of a moving object in fields such as image engineering, medical image engineering, biology, disaster prevention, fluid dynamics, civil engineering, oceanography, river science, and disaster monitoring.

110 Image data input unit 120 Image data storage unit 130 First optimization unit 140 Second optimization unit 150 Convergence determination unit 160 Display unit 170 Model parameter change unit

Claims (8)

A method for simultaneously estimating continuous and discontinuous movement of a moving object, comprising:
Capturing a time-series image of the moving object by the data input unit;
Storing the image data of the time-series image in a data storage unit;
A step of estimating a model parameter of the moving object, the model parameter integrating a wave generation equation representing a time-series change in luminance of an image and an optical flow model which is a motion estimation model in image processing; Is estimated by an objective function of a mathematical model relating to a wave expressing characteristics of a wave phenomenon including continuous and discontinuous motion, and the objective function includes a constraint condition by a wave dispersion relational expression, and the estimating step includes:
In the first optimization unit, a robust function for reducing the influence of noise in the image data is applied to the objective function to roughly estimate the model parameter in the framework of the objective function minimization problem Optimization steps,
In the second optimization unit, the number of terms of the wave generation equation and the model parameter that is the size of the image area to which the objective function is applied are optimized based on the parameter statistics estimated in the first optimal calculation A second optimization step that includes: and
A step of determining a similarity between the model parameter and the statistical model by obtaining a frequency distribution related to the parameter based on the estimated model parameter and performing comparison with a statistical model set in advance in a convergence determination unit; When,
When determining that the similarity between the model parameter and the statistical model is high, determining that the model parameter has reached a predetermined accuracy, and displaying the final result of the model parameter on a display unit;
When it is determined that the similarity between the model parameter and the statistical model is low, after changing the number of terms of the wave generation equation and the model parameter that is the size of the image area to which the objective function is applied, Returning to one optimization step.   The method of claim 1, wherein the model parameters include vertical and horizontal components of optical flow, wave number, frequency, and phase.   The method according to claim 1, wherein the number of terms in the constraint condition expression changes in conjunction with the number of terms in the objective function.   The method according to any one of claims 1 to 3, wherein the convergence determination unit uses, as a statistical distribution model, a feature related to movement specialized for each moving object. An apparatus for simultaneously estimating continuous and discontinuous movement of a moving object,
A data input unit for capturing a time-series image of a moving object;
A data storage unit for storing image data of the time-series images;
An optimization unit that estimates a model parameter of the moving object, the model parameter including a wave generation equation that represents a time-series change in luminance of an image, an optical flow model that is a motion estimation model in image processing, and Is estimated by an objective function of a mathematical model relating to a wave that expresses characteristics of a wave phenomenon including continuous and discontinuous motion, and the objective function includes a constraint condition by a wave dispersion relational expression, and the optimization unit Is
Applying a robust function for mitigating the influence of noise of image data to the objective function, and a first optimization unit that roughly estimates the model parameter in the framework of the objective function minimization problem;
A second optimization unit for optimizing model parameters, which are the number of terms of the wave generation equation and the size of the image area to which the objective function is applied, based on a parameter statistic estimated by the first optimal calculation; Including an optimization unit,
A convergence determination unit that determines a frequency distribution related to a parameter based on the estimated model parameter and determines similarity between the model parameter and the statistical model by comparing with a statistical model set in advance. When determining that the similarity between the model parameter and the statistical model is low, after changing the number of terms of the wave generation equation and the model parameter that is the size of the image area to which the objective function is applied, A convergence determination unit that causes a first optimization unit to estimate model parameters of the moving object;
In a convergence determination unit, when it is determined that the similarity between the model parameter and the statistical model is high, a display unit that determines that the model parameter has reached a predetermined accuracy and displays a final result;
The apparatus characterized by including.   6. The apparatus of claim 5, wherein the model parameters include vertical and horizontal components of optical flow, wave number, frequency, and phase.   The apparatus according to claim 5 or 6, wherein the number of terms in the constraint condition expression changes in conjunction with the number of terms in the objective function.   The apparatus according to any one of claims 5 to 7, wherein the convergence determination unit uses, as a statistical distribution model, a feature related to a movement specialized for each moving object.