JP6202879B2 - Rolling shutter distortion correction and image stabilization processing method - Google Patents

Description

  The present invention relates to a video stabilization process that simultaneously performs motion distortion deformation correction (rolling shutter distortion correction) and shake correction in a video shot by a camera using a CMOS sensor.

  A method for estimating motion of an image has been studied for a long time, and the method is roughly classified into a feature-based method and a region-based method. The feature-based method detects image features such as corner points and straight lines, and associates them with images (frames) to estimate the motion of the image. Processing takes a relatively large amount of processing time and cost.

  As described above, since the feature-based method requires a cost for the feature extraction processing and the subsequent association processing, development and commercialization of a shake stabilization device using the region-based method have been eagerly performed. However, all of them are intended for camera images acquired by a CCD sensor, do not support motion distortion deformation by a CMOS sensor, and cannot stabilize the shaking image of the CMOS.

  Conventionally, it is impossible to discriminate between intentional camera movement (for example, panning) and unintentional shaking (blurring) at the acquired video stage. Therefore, it was not possible to sufficiently remove only unnecessary shaking.

  In Patent Document 1 below, in a method of reading out video data from a CMOS sensor, distortion is reduced by devising simultaneous reading of a plurality of scanning lines that starts reading the next scanning line while reading a certain scanning line. An imaging method is disclosed.

  Further, Patent Document 2 below discloses a method for correcting distortion by video processing. However, this is a method using image feature points, and a technique for separately estimating both a translation component and a distortion component due to video shaking. The idea is disclosed.

JP2011-142592A JP2013-171155A

  Conventionally, neither feature-based video motion estimation nor region-based video motion estimation was compatible with motion sensor distortion correction using CMOS sensors, and it was not possible to stabilize shaking video. . For example, in situations where the subject in the screen moves faster than the scan speed during raster scanning, it is known that the image is acquired (motion distortion) so that the shape of the subject is distorted. There was no way.

  Further, conventionally, since it is impossible to discriminate between intentional camera movement (for example, panning) and unintentional shaking (blurring) at the acquired video stage, only unnecessary shaking is obtained from the acquired video from the mobile camera. Could not be removed sufficiently.

  In the present invention, motion distortion deformation caused by a rolling shutter is performed between two adjacent images (in order to be able to simultaneously perform motion distortion deformation and shake correction in an image captured by a camera using a CMOS sensor. It is described by two-dimensional four-parameter affine transformation (between two frames). Then, after optimally estimating the transformation matrix, the translation parameter is calculated by analytically decomposing.

  In this case, the estimation is performed by the region-based direct method using the gradient constraint condition without using any image feature or association (that is, without using the conventional feature-based method). In the case of a moving camera, the fluctuation component is removed by a recursive filter in which level adaptation is added to the recursive filter for the estimated time series change of the translation parameter, and the camera is moved (that is, the visual field image by panning or the like is removed). It is assumed that only shaking (blurring) in the video is corrected while holding (movement).

  Accordingly, the video stabilization processing method of the present invention performs a two-dimensional four-parameter affine transformation between two adjacent images on a motion image obtained from a rolling shutter, and transforms the transformation matrix of the video acquired from the CMOS sensor. It is characterized by calculating the translation parameter by analytically decomposing after estimation by a region-based direct method using gradient constraint conditions.

  The video stabilization processing method of the present invention is preferably characterized in that, in the case of a mobile camera, the fluctuation component is removed by a recursive filter taking the level adaptation into account for the time-series change of the estimated translation parameter. And

  In the video stabilization processing method of the present invention, it is more preferable that the analytical decomposition after estimating the transformation matrix by the region-based direct method using the gradient constraint condition is optimized by iterative updating using the linear solution method. It is an analytical decomposition of the affine transformation matrix.

  The video stabilization processing method of the present invention is more preferably characterized in that the optimization by iterative updating using a linear solution method is an arithmetic processing using a least square method.

  In addition, the image stabilization processing method of the present invention is more preferably characterized by simultaneously performing motion distortion deformation and shake correction in an image acquired from a CMOS sensor.

  The video stabilization processing method of the present invention is more preferably characterized in that no image feature or association is used when the transformation matrix is estimated by a region-based direct method using gradient constraint conditions.

  The image stabilization processing method of the present invention is more preferably characterized by correcting only the shaking in the image while maintaining the movement of the visual field image accompanying the movement of the camera.

  A program of the present invention is a program for causing a computer to execute any one of the above-described video stabilization processing methods.

  The storage medium of the present invention is a computer-readable storage medium storing the above-described program.

  The video stabilizer of the present invention is a video stabilizer that executes any one of the above-described video stabilization processing methods.

  According to the present invention, it is possible to simultaneously process motion distortion deformation and shake correction in an image taken by an imaging device using a CMOS sensor. In addition, in the case of an image including shaking of a moving camera such as a camera pan, it is possible to correct only shaking in the image while maintaining camera movement. In addition, even when transitioning from a moving camera to a fixed camera or vice versa, it is possible to simultaneously and faithfully perform correction processing for motion distortion deformation and shaking.

It is a block diagram explaining the movement distortion correction and stabilization process (blur correction process) of a CMOS camera image in the case of a fixed camera. It is a block diagram explaining the movement distortion correction and stabilization process (blur correction process) of a CMOS camera image in the case of a moving camera. It is a figure explaining the movement distortion by the sequential exposure (raster scan) of a CMOS camera, and when the vertical line moves to the right of the image (when the camera is panned to the left) and the resulting distortion image (upper stage) FIG. 6 is a diagram illustrating a case where a circle moves downward in an image (when the camera pans upward (also referred to as tilt)) and a distortion image (lower stage) obtained as a result. It is a figure explaining the motion distortion of a CMOS camera. A CMOS motion distortion image of a lattice image according to the direction of camera movement generated by simulation, where (a) illustrates the case where there is no camera movement, and (b) is a case where the image is deformed to the right when the camera is directed to the left. (C) illustrates the case where the image is deformed to the left when the camera is directed to the right, (d) illustrates the case where the image is stretched when the camera is directed upward, and (e) illustrates the case where the camera is The case where the image is shrunk when facing is described. FIG. 6 is a diagram for explaining the result of distortion correction by calculating a four-parameter affine transformation matrix using an image without camera movement as a reference image in the simulation image obtained by distortion-deforming the lattice image of FIG. It is a figure explaining an image (no camera motion), (b)-(e) is a figure explaining the correction result of each motion distortion image (b)-(e) by the motion of the camera of FIG. The image boundary remains black so that distortion correction can be easily understood. (A) is a figure which shows the addition average image of the translation distortion image row | line | column produced | generated using the translation parameter by the normal random number of average 0, standard deviation 5 and 3 pixels, respectively in the horizontal and vertical directions (30 frames), ( b) An addition average of image sequences of processing results obtained by performing distortion correction and stabilization processing using a result of estimating translation parameters between two adjacent images sequentially from the second frame using the first frame as a reference image. It is a figure which shows an image, and the addition average image after a process (b) looks clear with respect to the addition average image before a process (a) where the outline appears to overlap by distortion deformation and a shake, black of the image boundary vicinity The taste is due to an overrun by the correction process, and (c) is a diagram for explaining the trajectory of the translation parameter used to generate the distorted image sequence. It is a figure explaining a part of translation distortion image sequence which gave a translation parameter to an image of 2592 × 1944 pixels photographed with a digital camera (Canon IXY DIGITAL500 (registered trademark)) and cut out a part thereof. The image size is 640 × using the translation parameter (tx, ty) = (15, 3) serving as a reference in the horizontal and vertical directions, and a translation parameter obtained by adding normal random numbers each having an average of 0 and a standard deviation of 1 pixel. This shows the result of generating a 480-pixel translational distortion image sequence, which can be regarded as an image from a moving camera that moves in a straight line at a constant speed, in which the camera is panned up to the upper left in order from left to right. The original image sequence generated in this way, the middle row is the image row resulting from correction as a fixed camera, and the lower row is corrected as a moving camera. Is a diagram illustrating the result image sequence. It is a figure explaining the graph of the time-sequential change of the translation parameter between two adjacent images estimated from the translation distortion image sequence by a moving camera, The upper stage shows the horizontal direction and the lower stage has shown the perpendicular direction. It is a block diagram explaining the hierarchy motion estimation process in an Example.

  In the present invention, motion distortion deformation caused by a rolling shutter is performed between two adjacent images so that motion distortion deformation and shake correction in an image captured by a camera using a CMOS sensor can be performed simultaneously. It is described by a two-dimensional four-parameter affine transformation. Then, after optimally estimating the transformation matrix, the translation parameter is calculated by analytically decomposing.

  In this case, the estimation is performed by the region-based direct method using the gradient constraint condition without using any image feature or association. In the case of a moving camera, the camera component (ie, the movement of the visual field image) is removed by removing the shaking component with the cyclic filter that adds level adaptation to the cyclic filter for the estimated time-series change of the translation parameter. It is assumed that only shaking (that is, “blurring”) in the video is corrected while being held.

  As described above, in the present invention, in the case of a mobile camera, when the estimated translation parameter is used, it is possible to simultaneously perform motion distortion deformation and shake correction in an image captured by a camera using a CMOS sensor. In response to the sequence change, the shaking component is removed by a filter in which level adaptation is added to the recursive filter, and only the shaking in the video is corrected while maintaining the movement of the camera. Further, since the filter processing is based on level adaptation, it is possible to faithfully perform correction processing even in the case of transition from a moving camera to a fixed camera, or vice versa.

  Therefore, by modeling the motion distortion due to the rolling shutter (sequential exposure) mechanism of the CMOS sensor as a two-dimensional affine transformation, estimating the transformation matrix based on the gradient constraint condition, and analytically decomposing the estimated transformation matrix It is assumed that translational parameters that cause motion distortion are calculated, and that cyclic filter processing that takes into account level adaptation processing is applied to the time-series changes of the estimated translational parameters.

  The method of the present invention can be implemented by a hardware device that processes a baseband video signal, or by a software-based device that processes MXF files and a computer that executes the software. is there. In addition, if an apparatus that converts or reversely converts an MXF file into a baseband video signal is used, various other configurations can be realized.

FIG. 1 is a block diagram for explaining motion distortion correction and stabilization processing of a CMOS camera image in the case of a fixed camera. In FIG. 1, two images are obtained by motion estimation.


A four-parameter affine transformation matrix between and the translation parameters decomposed


To correct distortion (Distortion Correction).

Cumulative addition of translation parameters


Thus, the result of distortion correction is shake-corrected (Motion Stabilization). That is,


It is. Here, Z ⁻¹ represents one frame delay.

For mobile cameras,



Is not a cumulative addition of


Was directly smoothed by a low-pass filter (LPF).


Thus, as shown in FIG. 2, the distortion correction result is subjected to shake correction (Motion Stabilization). FIG. 2 is a block diagram illustrating motion distortion correction and stabilization processing of a CMOS camera image in the case of a mobile camera.

  If smoothed translation parameters are used, shake correction can be performed even in a moving camera such as a pan. When a recursive filter is used as a low-pass filter, only current and past data is used, so that no extra frame delay occurs, which is advantageous from the viewpoint of the delay amount of the entire process.

In addition, a weighting factor corresponding to a signal level difference is introduced into a cyclic filter applied to a time series of estimation results of translation parameters that cause motion distortion between two images. For the first-order Butterworth recursive filter, the horizontal translation parameter


Is calculated as follows (for the vertical translation parameter:


And it is sufficient).

here,

Also,


Is a parameter for adjusting an allowable range of signal level difference, and α ₀ , α ₁ , and β ₁ are first-order Butterworth recursive filter coefficients and are calculated as shown in [Equation 2] below.

here,


Is the sampling period,


Is the digital cutoff angular frequency.

  That is, in the present invention, it is possible to correct shaking and distortion at the same time, without extracting or associating feature points to estimate motion, and correcting only unnecessary shaking from a moving camera. In particular, the level adaptive filter for dealing with a moving camera has a great feature.

  If it is just a recursive filter, the change point when the camera changes from a moving state to a stationary state, or vice versa, the change point when changing from a stationary state to a moving state is dull. Although correct correction processing is not performed, if the above-described level adaptive filter described in the present invention is used, it is possible to perform faithful correction with respect to a motion / still state change.

(Rolling shutter distortion correction and image stabilization without using corresponding points)
In recent years, CMOS sensors have been widely used from low-priced mobile phone cameras to high-end digital single lens reflex cameras (DSLRs). Although the CMOS sensor can be reduced in price, reduced in power consumption, and large in size, it differs greatly from the conventional CCD sensor in that it acquires pixel data using a sequential exposure mechanism called a rolling shutter. This is the point at which motion distortion deformation is caused.

  The present invention performs a stabilizer process on a CMOS camera image by a rolling shutter mechanism. At this time, no image feature or association is used. A global motion between two adjacent images is described by two-dimensional affine transformation, and a linear solution is used for the estimation without performing nonlinear optimization. The translation parameters are calculated by analytically decomposing the affine transformation matrix optimized by the iteration of linear solution. An image simulation experiment is conducted to correct motion distortion and deformation included in the images of both the fixed camera and the moving camera, and to stabilize by removing shaking. In the case of a moving camera, the shaking component is removed by cyclic filter processing with respect to the estimated time-series change of the translation parameter, and only the shaking in the video is corrected while keeping the camera moving. Even when transitioning from a mobile camera to a fixed camera, a stabilization process faithful to the change is realized.

(CMOS motion distortion model)
The CMOS sensor has a shutter mechanism different from that of the CCD sensor. In the CCD sensor, all pixels are exposed simultaneously, but in the case of a CMOS sensor, sequential exposure by line scanning is used in order to achieve a small size and low cost. Therefore, if the camera motion is very large compared to the scanning time, the CMOS camera image is distorted according to the direction and type of camera motion due to the time difference between the first and last lines of the CMOS sensor. FIG. 3 shows how an object in the scene moves during the scanning time in such a rolling shutter mechanism. FIG. 3 is a diagram for explaining motion distortion due to sequential exposure of a CMOS camera. When the vertical line moves in the right direction of the image (when the camera is panned to the left) and the resulting distortion image (upper stage). FIG. 6 is a diagram illustrating a case where a circle moves downward in an image (when the camera pans upward (also referred to as tilt)) and a distortion image (lower stage) obtained as a result.

Here, it is assumed that when a CMOS camera whose image size is V × H moves, the feature point x of the object in the photographed scene is moved by the image movement u during one frame period. Assuming translational motion,


And the velocity v is obtained by dividing by one frame time Tf.

In the image _{I n} to the scanning starting at t = 0 from the image origin of upper left of the screen, the feature point xn is to move in un to one frame period, the pixel position xdn = (xdn, ydn) elapsed time until,


The CMOS distortion position xdn satisfies the following CMOS motion distortion model in which a fluctuation term due to motion distortion is added to the position xn when there is no distortion.

And approximating



From [Equation 8], solving for ydn:

Therefore, the distortion transformation by translation between (xn, yn) and (xdn, ydn) is as follows.

this,


And write


Therefore, the relationship between two adjacent images I _n and I _{n + 1} due to translational distortion is as follows (FIG. 4). FIG. 4 is a diagram for explaining the motion distortion of the CMOS camera.

here,


When the transformation matrix An, n + 1 is written as an element,


And this



And set each element of the matrix to be equal. Solving for txn, tyn, txn + 1, and tyn + 1 gives the following.

  FIG. 5 is a CMOS motion distortion image of a lattice image based on the direction of camera movement generated by simulation. FIG. 5A illustrates a case where there is no camera movement, and FIG. 5B illustrates an image when the camera faces left. Describes the case where the image is deformed to the right, (c) illustrates the case where the image is deformed to the left when the camera is directed to the right, (d) illustrates the case where the image is expanded when the camera is directed upward, e) illustrates a case where the image shrinks when the camera faces downward.

(CMOS motion estimation by two-dimensional affine transformation)
A transformation matrix An, n + 1 representing distortion deformation due to translational motion is a two-dimensional affine transformation, but its degree of freedom (the number of unknown parameters) is four. Therefore, this is referred to as 4-parameter affine transformation. Thus, CMOS motion estimation results in computing a four parameter affine transformation. A method for calculating a four-parameter affine transformation matrix is shown below.

From Equation 14, when the first image _{I n} An, the n + 1 4 parameter affine transformation, since overlapping the second image _{I n + 1,} the following relationship holds in the overlapping portion.

When the right side of [Equation 22] is first-order approximated by Taylor expansion,


In order to obtain a = (a2, a3, a5, a6), the following objective function J is minimized. Hereafter, abbreviated as xdn → x and ydn → y.

here,


It is. Σ represents the sum over all pixels in the overlapping area of the two images. [Equation 24] is a least square estimation of the gradient constraint condition. Differentiating J with each parameter, it becomes as follows.

Therefore, the following simultaneous equations may be solved.

The translation parameters txn, tyn, txn + 1, and tyn + 1 are calculated from [Equation 18] to [Equation 21] from the a2, a3, a5, and a6 calculated as described above.

  The txn, tyn, txn + 1, and tyn + 1 obtained in this way may be used as initial values, and may be optimized by, for example, the Gauss-Newton method. Direct optimization is cumbersome. Therefore, the result of optimal estimation of An, n + 1 by iterative updating is decomposed. The procedure for optimizing the least squares method by iteration is as follows.

Step 1: An initial value is given to generate a converted image of the first image, and J ← ∞ (a sufficiently large value).
Step 2. Calculate the spatiotemporal gradients Ix, Iy, It for each pixel according to [Equation 25] using the second image and the converted first image.
Step 3 Solve the following simultaneous equations.

Steps 4, a2, a3, a5, and a6 are updated as follows.

Step 5, a converted image of the first image with the updated parameter a is generated, and the residual J ′ = J (a) is calculated. If J ′ <= J and | J−J ′ | <ε (small threshold value), a is returned and the processing is terminated. Otherwise, return to step 2 as J ← J '.

  In practice, a hierarchical image is generated by thinning out by applying a Gaussian filter, and a hierarchical motion estimation process is performed in which a parameter estimated between the lowest resolution images is propagated to a motion estimation process between higher resolution images.

FIG. 10 is a block diagram illustrating the hierarchical motion estimation process in the embodiment. As shown in FIG. 10, adjacent motion distortion 2 images “I _n ” and “I _{n + 1} ” are input. Each is multiplied by a Gaussian filter Gσ to thin out the image size by half (↓ 2). The thinning process is repeated to reduce the image size to ¼. When the original image size is Level 0, the 1/2 image size is called Level 1 and the 1/4 image size is called Level 2.

4 gives the parameter initial value a of the affine transformation matrix parameters (0), performs correction processing by 4 parameter affine transformation I _n 1/4 image size (W). Then, a four-parameter affine transformation matrix is estimated with respect to In _{+ 1} of a _¼ image size (M). The result of adding the estimation result to a (0) is defined as a (1), and is used for Level1 processing with the next 1/2 image size.

The initial value a (0) of the four-parameter affine transformation matrix parameter may be a translation parameter, for example, by block matching in I _n and I _{n + 1} of the ¼ image size. In this case, a (0) = (a2, a3, a5, a6) = (0, tx (0), 0, ty (0)). Here, (tx (0), ty (0)) is a translation parameter by block matching.

1/4 image Level2 treated similarly by size, 1/2 image Level1 processing by size, 4 parameter affine transformation matrix parameters a (1) correction processing by 4 parameter affine transformation _{I n} 1/2 image size by (W )I do. Then, a four-parameter affine transformation matrix is estimated between I _{n + 1} of the ½ image size (M). The result of adding the estimation result to a (1) is set as a (2), and is used for Level0 processing with the next original image size.

Level1 similarly processed by the half image size, Level0 processing by the original image size, performing four-parameter affine transformation matrix parameters a (2) correction processing by 4 parameter affine transformation _{I n} of the original image size by a (W). Then, a 4-parameter affine transformation matrix is estimated between the original image size I _{n + 1} (M). The result of adding the estimation result to a (2) is a (3), which is the final estimation result of the 4-parameter affine transformation matrix, and the translation parameter is calculated by decomposing the resulting 4-parameter affine transformation matrix. Time series processing is performed on the obtained translation parameters, and finally the original image size I _{n + 1} is corrected and output by the four-parameter affine transformation matrix synthesized again by the translation parameters as a result.

(CMOS motion distortion correction and stabilization processing)
As described above, FIG. 1 is a block diagram for explaining motion distortion correction and stabilization processing of a CMOS camera image in the case of a fixed camera. In FIG. 1, a 4-parameter affine transformation matrix between two images I _n and I _{n + 1} is estimated by motion estimation, and distortion correction (Distortion Correction) is performed by a translation parameter tn = (txn, tyn) obtained by decomposing the matrix. ) Based on τn = (τxn, τyn) obtained by accumulating the translation parameters, the distortion correction result is subjected to shake stabilization (Motion Stabilization). That is,

It is. Z ⁻¹ represents one frame delay.

In the case of a moving camera, tn is not cumulatively added, but the result of distortion correction is shake-corrected (Motion Stabilization) by τn obtained by smoothing tn with a low-pass filter (LPF) (FIG. 2). In FIG. 2 described above, if smoothed translation parameters are used, shake correction can be performed even in a moving camera such as a pan. When a recursive filter is used as a low-pass filter, only current and past data is used, so that no extra frame delay occurs, which is advantageous from the viewpoint of the delay amount of the entire process. For a first order Butterworth recursive filter, its output is calculated as follows:

Here, ωac is an analog cutoff angular frequency prewarped by bilinear transformation, and when the digital cutoff angular frequency is ωc = 2πfc, the following is obtained.

Ts is a sampling period, and Ts = 1 / fs when fs is a sampling frequency.

  [Equation 31] is written as a vector in both the horizontal and vertical directions. If the cut-off frequencies are the same, the filter coefficients are the same, but different cut-off frequencies may be used.

  However, when transitioning from a moving camera to a fixed camera, the recursive filter becomes duller due to attenuation as the smoothing effect increases. This means that the stabilization process is performed as the moving camera for a while even though the moving camera is stationary. Or vice versa, the same applies to a transition from a fixed camera to a moving camera.

  Therefore, a weighting factor corresponding to the signal level difference is introduced into the cyclic filter so that the changing point is not dull due to the attenuation of the filter processing. For the first order Butterworth recursive filter, the horizontal translation parameter τxn is calculated as follows:

here,





σr is a parameter for adjusting an allowable range of signal level difference. The vertical translation parameters may be calculated as τx → τy and tx → ty.

  Actually, in the case of both the fixed camera and the moving camera, the distortion correction and the shake correction are collectively performed by four-parameter affine transformation in which the parallel parameters tn and τn are synthesized again. The pixel value at the pixel coordinates with subpixel accuracy by the conversion is calculated by interpolation using neighboring pixels.

(Image simulation experiment)

(1) Artificial Image Simulation Experiment FIG. 6 is a simulation image obtained by distorting and deforming the lattice image of FIG. 5, and calculating a distortion correction result by calculating a 4-parameter affine transformation matrix using an image without camera movement as a reference image. It is a figure explaining. In the CMOS motion distortion corrected image shown in FIG. 6, (a) is a diagram for explaining a reference image (no camera motion), and (b) to (e) are motion distortion images (b) due to the camera motion of FIG. It is a figure explaining the correction result of (e), and the image boundary is left black so that distortion correction can be easily understood.
From FIG. 6, it can be seen that distortion deformation due to movement in both the horizontal and vertical directions can be corrected.

(2) Real image sequence simulation experiment (in the case of a fixed camera)
A translation parameter is given to an image of 3008 × 2000 pixels captured by a digital camera (Nikon D40 (registered trademark)), and a part thereof is cut out to generate a translational distortion image. A translational distortion image sequence is generated using translation parameters based on normal random numbers having an average of 0, a standard deviation of 5 and 3 pixels in the horizontal and vertical directions, respectively. The generated image size is 640 × 480 pixels. This can be regarded as a fixed point monitoring video by a fixed camera.

  FIG. 7A is a diagram showing an addition average image of translational distortion image sequences generated in this way (30 frames). Using the first frame as a reference image, distortion correction and stabilization processing were performed using the results of estimating translation parameters between two adjacent images sequentially from the second frame. FIG. 7B is a diagram showing an addition average image of an image sequence as a processing result. Compared to the addition average image before the process that appears to have overlapping contours due to distortion and shaking, the addition average image after the process looks clear. This is because the blackness in the vicinity of the image boundary is cut out by the correction process. FIG. 7C is a diagram for explaining the locus of the translation parameter used for generating the distorted image sequence.

  The correction result is quantitatively evaluated based on the peak SN ratio of the square error image between two adjacent images. The peak SN ratio PSNR is obtained as follows from the average luminance value (average noise power) MSE and the maximum luminance value (maximum signal power) I2max of the square error image. In the experiment, Imax was set to an 8-bit maximum pixel value 255.

(3) Real image sequence simulation experiment (in the case of a moving camera)
FIG. 8 shows a part of a translation distortion image sequence generated by giving a translation parameter to an image of 2592 × 1944 pixels captured by a digital camera (Canon IXY DIGITAL 500 (registered trademark)) and cutting out a part of the image. . A translation distortion image sequence is generated by using translation parameters obtained by adding normal random numbers having an average of 0 and a standard deviation of 1 pixel to the translation parameters (tx, ty) = (15, 3) serving as the reference in the horizontal and vertical directions. To do. The generated image size is 640 × 480 pixels. In FIG. 8, the image can be regarded as an image from a moving camera that moves linearly at a constant speed, with the camera panning up to the upper left in order from left to right. The upper row in FIG. 8 is the original image sequence generated as described above, the middle row is the image row as a result of correction as a fixed camera, and the lower row is the image row as a result of correction as a moving camera.

  The result of distortion correction and stabilization processing performed as a fixed camera is completely stable with reference to the first frame. However, since the input is an image sequence from a moving camera, if the input frame moves greatly from the reference frame, there will be a region that is gradually cut out. It gets bigger.

  On the other hand, the results of the distortion correction and stabilization processing performed as a moving camera show that the correction processing follows the camera movement. Here, in order to remove unnecessary shaking components and smooth the trajectory of the camera, the first-order Butterworth cyclic level adaptive filter processing is performed on the time series change of the translation parameter estimated between two adjacent images, Each frame was corrected using the resulting translation parameters. The cut-off frequency in the first-order Butterworth cyclic level adaptive filter was 0.01 Hz in both the horizontal and vertical directions, and σr2 was 20 and 3, respectively.

  FIG. 9 is a diagram for explaining a graph of the time-series change of the translation parameter between two adjacent images estimated from the translation distortion image sequence by the moving camera, and the upper part indicates the horizontal direction and the lower part indicates the vertical direction. In FIG. 9, the translation parameter (IIR) obtained by smoothing the time series change (original) of the translation parameter by the first-order Butterworth cyclic low-pass filter (fc = 1 Hz), the first-order Butterworth cyclic level adaptive filter (fc = 0). .01 Hz, translation parameter (IIRbilital) obtained by smoothing σr2 by 20 and 3) in the horizontal and vertical directions, respectively. It can be seen that the translation parameter is smoothed by removing the high-frequency component, which is a fluctuation component, by the primary Butterworth cyclic level adaptive filter.

  Further, the translation parameter is set to (tx, ty) = (0, 0), and a normal random number is added in the same manner to continuously generate a translation distortion image sequence. That is, it is a case where after the camera moves, it becomes stationary and becomes a fixed camera. The translation parameter smoothed by the first-order Butterworth recursive filter with a cutoff frequency of 1 Hz at the transition point from the moving camera to the fixed camera in the 30th frame in FIG. 9 is a frame where the camera is stationary due to attenuation due to the smoothing action. Even if it exceeds, it will not become 0 immediately.

  On the other hand, the smoothing result by the first-order Butterworth recursive filter removes the shaking component from the moving camera and retains the change point, and a smoothing result close to 0 is obtained even after the camera is stationary. . It can be determined that the camera is sufficiently fixed by simple threshold processing. If the absolute value of the smoothed translation parameter is below a certain threshold value, it is determined that the camera is a fixed camera, and the correction parameter for the stabilization process is not the smoothed translation parameter, but the translation parameter from that frame. The result of cumulative addition may be used.

(Summary)
In the present invention, in order to simultaneously perform motion distortion deformation and shake correction in an image taken by a camera using a CMOS sensor, the motion distortion deformation caused by the rolling shutter is subjected to global two-dimensional four-parameter affine between adjacent images. The translation parameters were calculated by describing them by transformation, estimating the transformation matrix optimally, and then analyzing it analytically. At this time, the estimation was performed by directly processing the pixels without using any image feature or association.

  An image simulation experiment was performed to correct the motion distortion and deformation included in the images of both the fixed camera and the moving camera, and to stabilize by removing the shaking. In the case of a moving camera, the shaking component is removed by cyclic level adaptive filter processing with respect to the estimated translation parameter time-series change, and only the shaking in the video is corrected while maintaining the movement of the camera. When shifting from a moving camera to a fixed camera, we realized stabilization processing that is faithful to changes.

(Supplementary explanation)
The method of stabilizing the shaking of the camera image is called an optical or mechanical type that corrects the shaking by moving the camera lens or the camera itself based on the information from the acceleration sensor attached to the camera, and the image It is divided into what is called electronic by processing.

  These devices are generally referred to as video stabilizers. However, electronic video stabilizers based on image processing are advantageous in many respects such as a range in which shake can be corrected, downsizing and durability of the devices. In addition, electronic video stabilizers based on image processing can be processed later as post-processing of already acquired video, so that optical shake correction was not sufficient or was stored. It is possible to additionally process a past video or the like.

  For example, there is an example in which the global motion of the entire screen between two images is estimated by an optical flow to stabilize the camera image shake [Non-Patent Document 1].

Non-patent literature [1]
M. Irani, B. Rousso, and S. Peleg, Recovery of ego-motion using region alignment, IEEE Transactions on Pattern Analysis and Machine Intelligence,
19-3 (1997), 268-272.

  The stabilization of camera image shaking by image processing results in a problem of estimating the global motion of successive images.

  Image motion estimation is a fundamental problem in image processing and computer vision, and many studies have been conducted so far, but these can be broadly divided into region-based methods and feature-based methods.

  As a region-based method, block matching is used in the international standard MPEG of moving image compression encoding [Non-Patent Document 2], and optical flow [Non-Patent Documents 3 and 4] is often used in computer vision. In either case, grayscale pixels are directly processed.

Non-patent literature [2]
ISO / IEC-11172, Coding of moving pictures and associated audio for digital storage media up to 1.5 Mbits / s, 1993.

Non-patent literature [3]
BKP Horn and BG Schunck, Determining optical flow,
Artificial Intelligence, 17 (1981), 185-203.

Non-patent literature [4]
BD Lucas and T. Kanade, An iterative image registration technique with an application to stereo vision, Proceedings of the 1981 DARPA Image Understanding Workshop, April, 1981, 121-130.

  On the other hand, there is also processing performed in the frequency domain by frequency-converting an image, such as the phase correlation method [5].

Non-patent literature [5]
GA Thomas, Television motion measurement for DATV and other applications,
BBC R & D Reports RD1987 / 11, 1987.

  Some feature-based methods use image feature points such as corners and straight lines in the image. Kanazawa and Kanaya optimally calculated projective transformation between two images from feature points [6]. Matsunaga stabilized the swaying image by detecting the horizontal line in the image in order to remove the rotation and vertical movement of the image included in the image taken from the ship on the ocean [7].

Non-patent literature [6]
Satoshi Kanazawa, Kenichi Kanaya, Image mosaic generation by stepwise matching,
IEICE Transactions D-II, J86-D-II-6 (2003), 816-824.

Non-patent literature [7]
Matsunaga, Stabilization of Hull Motion Image by Horizon Detection,
Proceedings of the 15th Image Sensing Symposium (SSII2009), Yokohama (Pacifico Yokohama).

  Many of the conventional stabilizer processes are premised on a camera using a CCD sensor, but studies on stabilizer processes in CMOS sensors are also being conducted. Ringabil and Forssen [Non-patent Document 8] extracted the feature points in the video and tracked them after calibrating the internal parameters of the camera in advance in order to stabilize the camera video of the mobile phone. The camera motion was described by a three-dimensional rotation model, and nonlinear optimization was used for parameter estimation to minimize the reprojection error. Stabilization was performed by averaging the estimated parameters. Grundmann et al. [Non-Patent Document 9] divided a screen into blocks, calculated two-dimensional projection transformation between two adjacent images for each block, and corrected motion distortion by superimposing them. In order to calculate, the feature points in the video are still used.

Non-patent literature [8]
E. Ringaby and P.-E. Forssen, Efficient video rectication and stabilisation for cell-phones, International Journal of Computer Vision, 96-3 (2012), 335-352.

Non-patent literature [9]
M. Grundmann, V. Kwatra, D. Castro, and I. Essa, Calibration-free rolling shutter removal, Proceedings of IEEE Conference on Computational Photography (ICCP2012), April, 2012.

  The present invention performs a stabilizer process on a CMOS camera image by a rolling shutter mechanism. At this time, no image feature or association is used. A global motion between two adjacent images is described by two-dimensional affine transformation, and a linear solution is used for the estimation without performing nonlinear optimization. The translation parameters are calculated by analytically decomposing the affine transformation matrix optimized by the iteration of linear solution.

  An image simulation experiment was conducted, and it was confirmed that the motion distortion and deformation included in the images of both the fixed camera and the moving camera were corrected and the shake was removed and stabilized.

  In the case of a moving camera, the shaking component is removed by cyclic filter processing with respect to the estimated time-series change of the translation parameter, and only the shaking in the video is corrected while keeping the camera moving. Even when transitioning from a moving camera to a fixed camera, it is possible to realize stabilization processing that is faithful to changes.

  Further, in the present invention, since the method is based on video processing, it is possible to correct a stored video recorded with motion distortion or shaking taken by a camera using any CMOS sensor.

  In a conventionally known method, processing time and cost are required for extracting image feature points and associating feature points between two images. In the present invention, shake correction and distortion correction are simultaneously performed by directly processing pixel values without using any image features. The parallel component and the distortion component can be estimated and corrected together.

The present invention is suitable for video processing in general, particularly video monitoring and security. It can be used as a basis for estimating motion information of a video and performing processing such as a video stabilizer for performing motion compensation processing and frame rate conversion.

Claims (14)

Two-dimensional four-parameter affine transformation between adjacent two images of motion distortion caused by a rolling shutter is performed on an image acquired from a CMOS sensor, and the transformation matrix is obtained by a region-based direct method using gradient constraints. After estimation, the translation parameters are calculated by analytical decomposition ,
In the case of a mobile camera, a video stabilization processing method is characterized in that a shaking component is removed by a recursive filter in consideration of level adaptation with respect to a time-series change of the estimated translation parameter . The video stabilization processing method according to claim 1,
The video stabilization processing method characterized in that the two-dimensional four-parameter affine transformation between the two images is calculated by direct calculation using gradient information of a gray value between two pixels . In the video stabilization processing method according to claim 1 or 2,
The analytical decomposition after estimating the transformation matrix by a region-based direct method with gradient constraints is:
A video stabilization processing method characterized by the analytical decomposition of an affine transformation matrix optimized by iterative updating using a linear solution. The video stabilization processing method according to claim 3,
The video stabilization processing method, wherein the optimization by iterative updating using the linear solution method is an arithmetic processing using a least square method. In the video stabilization processing method according to any one of claims 1 to 4,
An image stabilization processing method characterized by simultaneously performing motion distortion deformation and shake correction in an image acquired from a CMOS sensor. The video stabilization processing method according to any one of claims 1 to 5,
An image stabilization processing method characterized in that no image feature or association is used when the transformation matrix is estimated by a region-based direct method using gradient constraint conditions. In the video stabilization processing method according to any one of claims 1 to 6,
An image stabilization processing method characterized by correcting only shaking in an image while maintaining movement of a visual field image accompanying movement of a camera.   A program for causing a computer to execute the video stabilization processing method according to any one of claims 1 to 7.   A computer-readable storage medium storing the program according to claim 8.   The video stabilizer which performs the image | video stabilization processing method as described in any one of Claims 1 thru | or 7. A video stabilizer according to claim 10, comprising:
Using the recursive filter as a low-pass filter and using only current and past data, processing is performed in real time without generating extra frame delay
A video stabilizer characterized by that. In the video stabilization processing method according to any one of claims 1 to 7,
The two-dimensional four-parameter affine transformation is obtained by modeling motion distortion deformation caused by the rolling shutter as a two-dimensional geometric transformation of an image.
And a video stabilization processing method. In the video stabilization processing method according to any one of claims 1 to 7,
In the estimation based on the region-based direct method using the gradient constraint condition, the transformation matrix is not subjected to feature point extraction processing and matching processing between the extracted feature points between the two images.
And a video stabilization processing method. In the video stabilization processing method according to any one of claims 1 to 7,
In the removal of the shaking component by the recursive filter, unnecessary motion due to blurring and significant motion of the camera are separated.
And a video stabilization processing method.