JP3209785B2 - Motion vector detection device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an improvement of a motion vector detecting device arranged in a moving picture signal coding apparatus and an image blur correcting apparatus.

[0002]

2. Description of the Related Art Conventionally, as a motion vector detecting method required for an image encoding apparatus and an image blur correcting apparatus, a spatio-temporal gradient method disclosed in U.S. Pat. No. 3,890,462 and Japanese Patent Publication No. 60-46878 has been proposed. Alternatively, a correlation method and a block matching method based on a correlation operation are known. Regarding the spatiotemporal gradient method, BKP, Horn et al.
elligence 17, pp. 185-203 (1981), and the matching operation is discussed in detail by Morio Onoe et al. in Information Processing Vol. 17, No. 7, p634-640 July 1976. I have.

The spatio-temporal gradient method is a method in which the amount of motion of an image is represented by "d / Δ" based on a luminance difference d between frames (or fields) and a luminance difference Δ between pixels in a screen. is there. This is based on the fact that the signal obtained from the camera is the time average of the field period, the edge becomes dull as the amount of motion of the image is large, and the luminance difference Δ between pixels becomes small, and the frame (or This is a result of normalizing the luminance difference d between the fields by the signal Δ.

On the other hand, the block matching method (template
The input image signal is divided into blocks of an appropriate size (for example, 8 pixels × 8 lines), and pixels in a certain range of a previous frame (or field) are divided into blocks. Is calculated, and a block of the frame (or field) before the sum of the absolute values of the differences is minimized is searched. The relative displacement of the block indicates the motion vector of the block.

[0005]

However, in the above-mentioned conventional example, when a plurality of different movements are present in the screen, a block straddling the boundary of the different movements is
Only unreliable motion vectors can be detected due to mismatching of corresponding points due to the appearance or occlusion of a subject, or different motions of other subjects, and coding is performed by an erroneous motion vector detected from the block. There is a problem that the encoding accuracy of the apparatus and the correction accuracy of the image blur correction apparatus are significantly deteriorated.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a motion vector detecting device capable of improving the accuracy of detecting a motion vector in view of the above points.

[0007]

According to the present invention, an input image is divided into a plurality of blocks, and a motion vector detecting means for detecting a motion vector for each of the blocks, and the same motion from a motion vector from the motion vector detecting means. Motion vector calculating means for calculating a motion vector value in the area by performing weighted averaging according to position information occupied in the area of each of the blocks, or the motion vector A motion in which an area having the same motion is extracted from the motion vector from the detecting means, and a motion vector value in the area is calculated by performing a weighted average according to positional information of each of the blocks in the input image. Vector calculation means.

[0008]

The motion vector calculating means calculates the position information of each block occupying the same moving area, or the position information of each block occupying the input image.
That is, the motion vector value is calculated by performing a weighting process according to the spatial or temporal adjacency of each block.

[0009]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, the present invention will be described in detail with reference to the illustrated embodiments.

FIG. 1 is a block diagram showing the configuration of a video camera incorporating an image blur correction device to which the first embodiment of the present invention is applied.

In FIG. 1, reference numeral 10 denotes a subject, reference numeral 20 denotes a photographing lens, reference numeral 30 denotes an image pickup device composed of, for example, a two-dimensional CCD,
Reference numeral 40 denotes a sample hold (S / H) circuit for holding an output signal from the image sensor 30 such as a two-dimensional CCD.
0 is an auto gain control (AGC) circuit, 60 is an analog-digital (A / D) converter, 70 is a color difference line-sequential signal from an image sensor 30 such as a two-dimensional CCD for two horizontal scanning periods. A delay circuit for delaying the color signal by 80, a color signal processing circuit 80, and a digital-analog (D
/ A) A converter 95 is a color signal output terminal.

100 is a low-pass filter (LPF) for removing a color signal mixed in the luminance signal, 110 is an enhancer for enhancing the high frequency component of the luminance signal, 120 is a gamma corrector, 130 is a motion vector detecting circuit, 140 is a microcomputer for determining the reliability of each block;
150 is a memory read control circuit, 160 is a memory for storing fields or frames, and 170 is the D of the luminance signal.
A / A converter 180 is a luminance signal output terminal.

Next, the operation of the video camera having the above configuration will be described.

The subject 10 is projected onto an image sensor 30 by a photographing lens 20 and is subjected to photoelectric conversion. The sample hold 40 holds the output signal of the image sensor 30, and the subsequent AGC circuit 50 automatically controls the gain of the signal.
The A / D converter 60 converts the output signal of the AGC circuit 50 into an A / D signal.
D-convert. The two horizontal scanning period delay circuit 70 converts the color difference line-sequential signal from the image sensor 30 into an “1H” delay signal and “0H”.
The signal is separated into a + 2H delay signal and sent to the luminance signal processing unit (the circuit after the LPF 100) and the color signal processing circuit 80, respectively.

A color signal processing circuit 80 generates a color signal. The color signal is converted into an analog signal by a subsequent D / A converter 90 and output from a color signal output terminal 95.

On the other hand, the signal sent to the luminance signal processing unit is
First, it is input to the LPF 100. This LPF 100
The carrier component is removed from the color difference sequential signal, and the luminance signal is separated. The next enhancer 110 performs a process of enhancing high-frequency components such as edges of a subject to improve image quality. Usually, the second derivative of the video signal is added to the original signal. The gamma corrector 120 prevents saturation in a highlight portion and extends a dynamic range. Motion vector detection circuit 130
May be configured by a circuit based on the spatiotemporal gradient method,
Similarly, it may be constituted by a circuit based on the matching operation, but in this embodiment, it is necessary that the detection method be capable of real-time processing. The memory 160 is a delay circuit for delaying the luminance signal for a predetermined time (one field time in this embodiment), stores the luminance signal one field before and calculates the current field with the space-time gradient method. , Or a correlation operation. The microcomputer 140 includes a motion vector detecting circuit 13.
First, from the distribution (optical flow) of the motion vector detected at 0, a block that outputs a motion vector due to a camera shake to be corrected is extracted. Weighting is performed in accordance with the weight, and the amount of motion of the image at that moment is calculated.
The memory read control circuit 150 is a microcomputer
The reading position of the memory 160 is controlled so that the final motion vector calculated by the data 140 is canceled (or corrected). The luminance signal read from the memory 160 is D / A
The signal is converted into an analog signal by the converter 170 and output from the luminance signal output terminal 180.

Next, how to evaluate the reliability of the motion vector performed in the microcomputer 140 will be described with reference to FIGS.
This will be described with reference to FIG.

FIG. 2 is a diagram showing a flow from input of an image signal to determination of an area to be image-stabilized at the moment (hereinafter, referred to as an image stabilization area).

FIG. 2A shows an input image, which is “8 × 6”.
Divided into blocks. FIG. 2B shows the result of detecting a motion vector for each block. Such a motion vector distribution diagram is generally called an optical flow diagram. In FIG. 2B, it can be seen that there are two largely moving regions in the center and the periphery of the screen. FIG. 2C shows the result of extracting the image stabilizing region from the optical flow diagram.

As a method of extracting the image stabilizing region from the motion vector, there is a time direction addition method of the motion vector for each block described in Japanese Patent Application No. 63-269554, which has already been filed.

FIG. 3 is a binary image stabilization area map (effective area) in which “1 value” is assigned to an extracted area and “0 value” is assigned to a non-target area.

Normally, by multiplying the detected motion vector by the image stabilization area map, an effective area is extracted, and the motion vectors in the effective area are simply averaged with uniform weights, thereby obtaining the motion of the image. Find the quantity.

FIG. 4 shows an example of a convolution mask to be applied to the image stabilizing area map as shown in FIG.

The meaning of the mask in FIG. 4 is that, when there is a block adjacent to the block of interest in the vertical and horizontal directions, a point 1 is given to the block of interest and When there is a block that touches the point, a point 0.5 is given, and the point becomes higher as the focused block is surrounded by other blocks. The point in the oblique direction is low based on the characteristic that the image signal generally has few oblique components. Note that the convolution is performed only for valid blocks (blocks in which the value “1” is inserted), or the image stabilization area map is AND-operated (logical AND operation) with the result of convolution over the entire screen. The value of the non-vibration-proof area (ineffective area) is kept at “0”.

FIG. 5 shows an example in which the convolution mask is expanded from “3 × 3” shown in FIG. 4 to “5 × 5”.

The size of the convolution mask and the weight coefficient (that is, the way of giving points) in the mask may be switched according to the photographing conditions.

FIG. 6 shows the result of convolution of the mask of FIG. 4 in the effective area of the image stabilization area map of FIG.

The coefficient values shown in FIG. 6 indicate the reliability of the motion vector detected from the corresponding block, and the motion vector may be averaged according to the weight.

The read position of the image from the memory is determined using the motion vector obtained by the above processing, and the image obtained by correcting the unpleasant camera shake component is output to the luminance signal output terminal 18.
Output from 0.

(Second Embodiment) In the first embodiment, weighted averaging of motion vectors is performed on the effective area of the image stabilization area map according to the result of convolution of a certain mask. However, in this embodiment, the reliability of the motion vector itself obtained in the first embodiment is evaluated again using the sum of the weighting coefficients obtained as a result of convolution of the mask, that is, the sum of the points. This can be used in combination with the first embodiment.

FIG. 7 shows the result of applying the convolution mask shown in FIG. 4 when it is determined that all “8 × 6” blocks in the screen are valid.

In this case, the sum of all the weight coefficients in FIG. 7 is "282", which is the upper limit of the sum of the weight coefficients. The larger the area of the image stabilizing area is, the higher the reliability of the motion vector detected from the area is. Therefore, the reliability of the motion vector can be evaluated again from the magnitude of the sum of the weight coefficients.

FIG. 8 shows a reliability evaluation function according to the sum of the weight coefficients.

When the total number of points is small, the extracted regions are small or scattered in a scattered manner, and there is a high possibility that a motion vector is erroneously detected.
The influence on the subsequent processing is suppressed by multiplying by a small coefficient less than “1”. If the number of blocks in the screen is “M × N”, the distribution of points is represented by 0 ≦ (total number of points) ≦ 7MN−4M−4N + 2.

The above processing is performed by the microcomputer shown in FIG.
This is performed by the data 140.

(Third Embodiment) In the first and second embodiments, the reliability is evaluated by using the information on the spatial arrangement of the block of interest as a clue.
In this embodiment, this concept is extended in the time axis direction.

In general, the temporal correlation of a moving image is high,
For example, when the subject is tracked and photographed, the position of the subject on the screen is strongly restricted to the position on the previous screen (the previous field), and the position of the subject is uncorrelated for each field. It rarely appears. That is, it is highly likely that the effective area on the current screen appears near the area considered as the effective area on the previous screen. Therefore, a map having a gentle weighting factor distribution is formed with the area determined to be an effective area on the current screen as a peak center, and a logical AND (AND) is calculated with the image stabilization area map obtained on the next screen. In addition, it is possible to evaluate reliability utilizing the correlation in the time direction of the position of a certain area on the screen.

The operation procedure will be described with reference to FIGS.

FIG. 9 shows a convolution mask to be applied to the image stabilization area map obtained on the current screen. Since it is considered that the displacement between the images of the position of the subject occurs at the same probability in every direction of 360 degrees, the weight coefficient distribution of the mask is also isotropic.

FIG. 10 shows the result of the convolution of the mask shown in FIG. 9 with the image stabilization area map shown in FIG.

The reliability of each block can be easily evaluated by performing an AND operation (logical AND operation) between the weight coefficient distribution shown in FIG. 10 and the image stabilization area map obtained on the next screen.

The above processing is performed by the microcomputer 140 shown in FIG.

According to each of the above-described embodiments, the reliability of a detected motion vector is evaluated based on the spatial or temporal adjacency (proximity) of each block, and a weighting process is performed according to the evaluation. Since it is applied to the vector, the accuracy of the moving image coding device and the image blur correction device, which has been significantly degraded by the unreliable motion vector detected from the block including the boundary where the different motion originally exists, Can be improved.

[0044]

As described above, according to the present invention,
The input image is divided into a plurality of blocks, a motion vector detecting means for detecting a motion vector for each block, and an area having the same motion is extracted from the motion vector from the motion vector detecting means. A motion vector calculating unit that calculates a motion vector value by performing weighted averaging according to position information occupying in the area of each of the blocks, or performing the same motion from a motion vector from the motion vector detecting unit Motion vector calculating means for extracting a region and calculating a motion vector value in the region by performing weighted averaging according to position information occupied in the input image of each of the blocks. Position information occupying in the region where
That is, the motion vector value is calculated by performing a weighting process according to the spatial or temporal adjacency of each block.

Therefore, it is possible to improve the detection accuracy of the motion vector.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a video camera incorporating an image shake correction apparatus to which a first embodiment of the present invention has been applied.

FIG. 2 is a diagram showing a flow from input of an image signal to determination of an image stabilizing area.

FIG. 3 is a diagram showing a binary map indicating an image stabilizing area according to the first embodiment of the present invention.

FIG. 4 is a diagram illustrating an example of a convolution mask applied to a vibration-proof region in FIG. 3;

FIG. 5 is a diagram showing another example of a convolution mask applied to the image stabilizing area of FIG. 3;

FIG. 6 is a distribution diagram of weighting coefficients representing reliability in the image stabilization area according to the first embodiment of the present invention.

FIG. 7 is a weight coefficient distribution diagram when the entire screen becomes a vibration-proof region in the second embodiment of the present invention.

FIG. 8 is a diagram illustrating a reliability evaluation function based on a sum of weighting coefficients according to a second embodiment of the present invention.

FIG. 9 is a diagram illustrating an example of a convolution mask applied to an image stabilizing area according to a third embodiment of the present invention.

FIG. 10 is a weight coefficient distribution diagram when the mask of FIG. 9 is composed on the image stabilization area map of FIG. 3;

[Description of sign]

 Reference Signs List 30 image pickup device 100 LPF 110 enhancer 120 gamma corrector 130 motion vector detection circuit 140 microcomputer 150 memory read control circuit 160 memory

──────────────────────────────────────────────────続 き Continued on the front page (58) Field surveyed (Int.Cl. ⁷ , DB name) G06T 7/20 G03B 17/00

Claims (2)

(57) [Claims] 1. An input image is divided into a plurality of blocks, a motion vector detecting means for detecting a motion vector for each block, and an area having the same motion is extracted from the motion vectors from the motion vector detecting means. A motion vector calculating means for calculating a motion vector value in the area by performing weighted averaging in accordance with positional information of each of the blocks in the area. 2. An input image is divided into a plurality of blocks, a motion vector detecting means for detecting a motion vector for each block, and an area having the same motion is extracted from a motion vector from the motion vector detecting means. A motion vector calculating means for calculating a motion vector value in the area by performing a weighted average according to position information of each of the blocks in the input image.