JP4688147B2 - Moving image processing device - Google Patents

Description

  The present invention relates to a moving image processing apparatus, and in particular, a moving image capable of efficiently and rapidly tracking a moving object in a moving image using not only decoding information of an encoded moving image but also code information. The present invention relates to a processing apparatus.

  As a method of tracking a moving object in a moving image, there are a method of tracking in a pixel region and a method of tracking in a code region. Background subtraction methods, KLT methods, and the like are known as methods for tracking a moving object in a pixel region. In the background difference method, the sum of absolute differences between the background image and the target image is used for the determination of moving object detection. In the KLT method, a moving object is detected by solving a constraint equation related to the gradient of an image.

  As a method for tracking a moving object in a code area, a method using code information itself of a compression-coded moving image has been proposed. In this case, various parameters and code information obtained at the time of compression coding are proposed. The process of moving object detection is achieved by directly operating. In the moving object detection process, only necessary information is selected and processed based on the motion prediction information of the code unit block in the image, and the moving object is tracked according to the detection result.

  However, the method of tracking a moving object in the pixel area is such that, when a moving image is encoded, when the method is applied, the code information of the encoded moving image is once decoded to the pixel region, Since the process is performed, there is a problem that the calculation load of the decoding process and the tracking process increases.

  On the other hand, the method of tracking a moving object in a code region depends on a motion vector determined to increase compression efficiency, and the motion in a moving image detected based on this motion vector does not always match the original motion. Therefore, there is a problem that the reliability of tracking is lacking. For example, since the influence of noise is large in a flat image region and the motion vector is often different from the original motion, the reliability of the motion detected based on the motion vector is particularly poor.

  For image understanding such as detection and tracking of a moving object, it is effective to analyze not only the pixel itself but also the feature quantity on the frequency. However, in order to analyze the content of the encoded moving image on the frequency, if the pixel region is once decoded and then converted to the frequency region again, the processing cost increases.

  An object of the present invention is to solve the above-described problem and use moving image information that can efficiently and rapidly track a moving object in a moving image by using code information itself of the encoded moving image as information on the frequency. An object is to provide an image processing apparatus.

In order to solve the above problems, the present invention provides an input for inputting a transmitted or accumulated encoded moving image in a moving image processing apparatus that tracks moving objects in the moving image using code information of the encoded moving image. using the parts, and a decoder for extracting and decoding of the code information from the input encoded moving image, to form a moving objects, and the code information a plurality of feature points of a region rich in change of the pixel a feature point selection section for selecting Te, by using the positional information and the decoded moving image of the feature point area unit smaller than the motion search in the encoded moving image, and estimating motion in a small region unit including a feature point It is basically characterized in that it comprises a motion estimator for estimating a motion of a moving object, the matching unit to track the moving object based on the motion of the moving object estimated by the motion estimation unit Te.

In the present invention, the moving object is detected and tracked using not only the decoding information of the encoded moving image but also the code information. At this time, a plurality of feature points in a region that forms a moving object and is rich in pixel changes are selected using code information , and the tracking target is limited to the feature points in the detection and tracking process of the moving object. Therefore , the processing cost can be reduced as compared with the method of tracking in the pixel area. Also, the processing cost can be reduced compared with the method of tracking in the code area. Furthermore, the motion of a moving object is estimated by estimating the motion in units of small areas including feature points, which is smaller than the unit of motion search in the encoded video using the position information of the feature points and the decoded video. Therefore, it is possible to track a moving object with high accuracy.

  The present invention will be described below with reference to the drawings. FIG. 1 is a functional block diagram of an embodiment of a moving image processing apparatus according to the present invention. The moving image processing apparatus according to the present embodiment includes an input unit 10, a decoding unit 11, a feature point selection unit 12, a motion estimation unit 13, a matching unit 14, and a display unit 15.

  Hereinafter, functions of each unit will be described in order. In the following description, it is assumed that the input moving image is an encoded moving image encoded according to the MPEG-1 video (ISO / IEC11172-2) which is an international standard. The present invention can also be applied to an encoded moving image, and does not limit the target encoded moving image.

  The encoded moving image is input to the decoding unit 11 via the input unit 10. The encoded moving image that is input may be transmitted from a remote place or read from a database inside or outside the apparatus.

  The decoding unit 11 decodes the input encoded moving image and extracts code information such as an encoding coefficient and a generated code amount from the encoded moving image. The extracted code information is sent to the feature point selection unit 12, and the decoded information is sent to the motion estimation unit 13.

  The feature point selection unit 12 receives the code information sent from the decoding unit 11, and forms a moving object that can easily match matches between frames so that the movement of the moving object can be accurately captured. Select.

  For example, a flat image region with little change is not suitable as a feature point because it is difficult to estimate an accurate motion when tracking. Also, an image area with a large noise or an image area with a random pixel value is not suitable as a feature point. On the contrary, an image region where a clear edge exists is suitable as a feature point because the position can be easily identified. In particular, an image region including a portion where a plurality of edges intersect, such as an object corner or an end point, is suitable as a feature point. The area of the moving object can be formed as a circumscribed area surrounding a plurality of feature points.

  Hereinafter, feature point selection focusing on edges will be described. FIG. 2 is a functional block diagram specifically illustrating the function of the feature point selection unit 12. The feature point selection unit 12 of this example sequentially executes three types of determinations of the texture determination 21, the distribution determination 22, and the determination 23 for the block boundary, and selects the feature points from the input code information.

  First, the texture determination 21 will be described. The texture determination 21 excludes a flat area and a noise area based on the texture of the coding coefficient, and sets an area having an appropriate edge as a feature point candidate. By determining the generated code amount and DCT coefficient, feature point candidates can be selected on the code area.

  For example, it is possible to determine a bit amount (generated code amount) that has been subjected to variable-length coding, and set a region in which this bit amount falls within a predetermined range as a feature point candidate. Further, it is possible to determine a length of 0 that continues to the highest order coefficient of the quantized DCT coefficient, and to select a region where this length falls within a predetermined range as a feature point candidate. It is also possible to determine the sum of the AC component absolute values of the DCT coefficient, and to set a region where the sum of the absolute values falls within a predetermined range as a feature point candidate. Since all the information that is the basis of these determinations is stored as code information, it can be easily obtained.

  The threshold used for the texture determination 21 is adaptive according to quantization information such as a quantization parameter, an encoding parameter such as whether the encoding coefficient is based on the frame DCT or the field DCT, and the like. It may be changed.

  Next, the distribution determination 22 will be described. The distribution determination 22 excludes a candidate having a simple shape from the feature point candidates selected in the texture determination 21, and selects a region having an appropriate complexity as a feature point. A region having an appropriate complexity can be selected on the code region based on, for example, a bias in the distribution of DCT coefficients.

  Specifically, the criterion is that both the absolute value sum of the horizontal AC component and the sum of the absolute AC component values of the DCT coefficient and the absolute sum of the other low frequency AC components exceed the threshold. Can be determined.

  Next, the determination 23 for the block boundary will be described. Since the texture determination 21 and the distribution determination 22 are determinations in units of blocks, it is not possible to determine feature points that straddle a plurality of blocks. For example, when an edge exists at a block boundary, the AC component of the DCT coefficient of the block is not a large coefficient. For this reason, such an edge cannot be determined in the determination using the DCT coefficient.

  Since the feature points that cannot be determined by the texture determination 21 and the distribution determination 22 are determined and selected, the block boundary determination 23 constitutes a block extending from a plurality of blocks to the block, and is different from the texture determination 21 and the distribution determination 22. Select feature points in a hierarchical block.

  For example, the DCT coefficients of four adjacent DCT coefficients X, Y, Z, and W are respectively xi, yi, zi, and wi (where i is the order in which the DCT coefficients are arranged horizontally and vertically) And a block that straddles four blocks with a 1/4 reduced image, a reduced block that straddles four blocks is obtained by equation (1). In Equation (1), IDCT (decoding processing) is approximated using DC components and three low-frequency AC components of each of the four blocks.

  Next, when a second-order Laplacian filter is separated into vector products as a high-pass filter used for edge detection, Equation (2) is obtained.

  When the vector of Expression (2) is applied to the block reduced by Expression (1), the convolution operation by the Laplacian filter is expressed by Expression (3) in the form of a matrix product. Summarizing the constants of equation (3) yields equation (4). Equation (4) can be used to derive the Laplacian filter results directly from the DCT coefficients.

If the criterion of the calculation result of Equation (4) is larger than the threshold, it is added to the feature point as an edge exists at the block boundary. The calculation of equation (4) can reduce the amount of calculation of the product of a constant matrix and an arbitrary vector (a, b, c, d) ^t (t represents transposition) by holding the calculation result in the middle. it can. FIG. 3 shows the calculation amount reduction by holding the calculation process and the calculation result in the middle.

For example, when the left two matrices are calculated to obtain the first element and the second element of the result matrix, a, b, c, and d may be set to x ₀ , x ₈ , z ₀ , and z ₈ , respectively. Other elements of the result matrix, and further each element obtained by the calculation of the result matrix and the right matrix can be obtained in the same manner by giving appropriate values to a, b, c, and d.

  In the above embodiment, the DCT coefficient of the DCT coefficient and the IDCT approximated by the three low-frequency AC components adjacent thereto and the high-pass filter are integrated to perform edge detection at the block boundary in the 1/4 reduced image. Edge detection can be performed similarly by approximating IDCT at other resolutions. In the determination 23 on the block boundary, since it is only necessary to determine the presence of an edge, the approximation of IDCT is arbitrary, and the IDCT can be approximated so that high-speed computation is prioritized.

  When the moving object to be tracked is known, it is possible to manually specify an area for selecting feature points from the beginning and limit the processing to the area. In this case, it is not necessary to specify the detailed shape of the moving object, and it is only necessary to specify the region with a rough shape such as a rectangle or a circle. This is because a plurality of feature points suitable for tracking are automatically selected from the designated region, and the smallest polygon surrounding the selected feature point is the initial region for moving object tracking.

  FIG. 4 is a functional block diagram specifically illustrating the function of the motion estimation unit 13. The motion estimation unit 13 has functions of a motion search 41 and a prediction error determination 42, determines a prediction error obtained during the search, and individually tracks a plurality of feature points forming the moving object, thereby moving the moving object. Is estimated. Since the MV (motion vector) is stored as the code information of the motion search MPEG, it may be possible to estimate the motion of the moving object from the weighted average of the MV in the area, but if the moving object is small, the macroblock If the number of (MB) is slightly increased or decreased, the motion may fluctuate greatly. Conversely, if the moving object is large, it may be affected by a low-reliability vector such as MV in the flat part. Since MV is determined to minimize the prediction error in units of MB (macroblock), it does not accurately represent the movement of feature points. Therefore, in the present invention, the motion is estimated again in units of small areas including feature points.

  The movement of the feature point is preferably estimated by determining a vector that minimizes the prediction error evaluation function from the viewpoint of edge preservation, noise resistance, and calculation load. Since the small region including the feature point is rich in change of pixels, it is possible to estimate an accurate motion even in a region that is much smaller than the MPEG MB. Also, by making the unit of motion estimation smaller than the MB of MPEG, not only the amount of calculation is reduced, but also the minimization direction and the gradient direction of the prediction error evaluation function are easily matched.

If the coordinates of the feature point are represented by <x> = (x, y), the coordinates <x> and the pixel value at time t is represented by I (<x>, t), the motion vector <v> = ( The prediction error evaluation function e (<v>) of v _x , v _y ) is defined by equation (5).

  Here, W represents a small region including feature points, and w (<x>) represents a weighting coefficient. Equation (5) can be simplified if the weighting factor w (<x>) is always set to 1, but the weighting factor can be set arbitrarily, such as the sum of the absolute values of the DCT coefficients AC component and the number of nonzero DCT coefficients. Is also possible.

An optimal motion vector <v _a > that minimizes the value of the prediction error evaluation function e (<v>) is obtained by iteratively repeating equations (6) to (9) using a certain initial vector <v ₀ >. be able to. Here, I _x and I _y are the differentials of I (<x>, t) in the x direction and y direction, respectively, and are expressed by equations (10) and (11). ΔI _k is a motion prediction error of I (<x>, t) and is expressed by Expression (12).

The end condition of the repeated iteration can be when the norm of Δ <v _k > becomes smaller than a predetermined threshold or when the number of repeated iterations reaches a predetermined number. Further, if MPEG MV is used as the initial vector <v ₀ >, the number of repeated iterations until convergence can be suppressed.

  Since a motion vector is affected by noise in a flat image region or the like, it is difficult to capture the original motion, but the flat image region or the like is excluded from the feature points in the feature point selection unit 12. However, since the true motion may be difficult to capture even at the feature point selected by the feature point selection unit 12, it is preferable to exclude the corresponding image area by the motion estimation unit 13. For example, a periodic edge region where a large DCT coefficient exists cannot be completely eliminated by the feature point selection unit 12, but this region easily induces an erroneous motion vector. In order to prevent this erroneous motion vector from being induced, not all feature points (edge regions) selected by the feature point selection unit 12 are feature points, but an error evaluation function obtained in the motion search process The edge region is classified based on the value, and the reliability as a feature point is increased.

  As described above, the motion vector is determined so as to minimize the value of the prediction error evaluation function defined by the sum of squares of pixel value differences or the sum of absolute values. There are a plurality of locations where the value of the prediction error evaluation function is minimized at regular intervals. Further, for example, in a simple edge region, the prediction error evaluation function takes the same value in the direction along the edge.

  In order to eliminate periodic edge regions and simple edge regions in which such erroneous motion vectors are likely to be induced, feature points are classified using motion vector information in the pixel regions. This classification can be achieved by looking at the distribution of prediction errors. For example, the value of the prediction error evaluation function at the location indicated by the motion vector may be compared with the value of the surrounding prediction error evaluation function.

  As a result of comparison, if only the location pointed to by the motion vector is sufficiently small, it can be classified as a feature point having a suitable complexity as a feature point, and if it is the same value at multiple locations, it is not suitable as a feature point Can be separated. For the comparison determination, for example, a location indicated by a motion vector and eight neighboring points can be used. Alternatively, a nearby point several pixels away can be used.

  The matching unit 14 tracks the moving object based on the motion of the feature points estimated by the motion estimation unit 13. If the monitoring camera is controlled by the matching unit 14, a monitoring system capable of detecting and tracking a moving object can be constructed. Specifically, the feature points for each frame are associated with each other using the motion vector <v> of the feature points, and a region surrounding the feature points is formed as a moving object, and the trajectory is tracked. The association of feature points between frames can be performed using the relative positional relationship between a plurality of feature points and the spatial distance between the positions of past feature points and predicted positions based on motion vectors <v>.

  Further, by performing association in consideration of color information for each feature point, it is possible to further reduce the possibility that the feature points are erroneously associated when some feature points are selected by mistake.

  The display unit 15 superimposes and displays the locus of the moving object tracked by the matching unit 14 on the decoded moving image. By looking at this display, it is possible to easily grasp how the moving object has moved.

1 is a functional block diagram of an embodiment of a moving image processing apparatus according to the present invention. It is a functional block diagram which shows the function of a feature point selection part. It is explanatory drawing which shows the calculation amount reduction by holding | maintaining the calculation process in the boundary determination, and the calculation result in the middle. It is a functional block diagram which shows the function of a motion estimation part.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 ... Input part, 11 ... Decoding part, 12 ... Feature point selection part, 13 ... Motion estimation part, 14 ... Matching part, 15 ... Display part, 21 ... Texture Determination, 22 ... distribution determination, 23 ... boundary determination, 41 ... motion search, 42 ... prediction error determination

Claims (23)

In a moving image processing apparatus for tracking a moving object in a moving image using code information of the encoded moving image,
An input unit for inputting transmitted or accumulated encoded moving images; and
A decoding unit that extracts and decodes code information from the input encoded moving image;
A feature point selection section for selecting moving objects is formed, and a plurality of feature points of a region rich in change of the pixel by using the code information,
A motion that estimates a motion of a moving object by estimating a motion in a small area unit including a feature point, which is smaller than a motion search area unit in an encoded video using the position information of the feature point and the decoded video. An estimation unit;
A moving image processing apparatus comprising: a matching unit that tracks a moving object based on the movement of the moving object estimated by the motion estimation unit. The moving image processing apparatus according to claim 1, further comprising a display unit that superimposes and displays the locus of the moving object tracked by the matching unit on the decoded moving image.   The moving image processing according to claim 1, wherein the input unit can specify a moving object region in the moving image, and the feature point selection unit first selects a feature point in the moving object region. apparatus.   The moving image processing apparatus according to claim 1, wherein the decoding unit includes means for extracting a coding coefficient and a generated code amount of the encoded moving image as code information.   The moving image processing apparatus according to claim 1, wherein the feature point selection unit includes means for associating feature points for each frame and setting a circumscribed area surrounding the feature points as a moving object area.   The moving image processing apparatus according to claim 4, wherein the feature point selection unit includes means for selecting a feature point by determining an evaluation of a texture and distribution of coding coefficients and a block boundary.   The feature point selection unit includes means for selecting feature point candidates by excluding a flat region and a noise region in a code region based on a texture of an encoding coefficient. A moving image processing apparatus.   The feature point selection unit uses the bit amount of the generated code amount, the number of 0s consecutive up to the highest order coefficient of the encoding coefficient, or the flat area in the encoding area by using the absolute value sum of the AC components of the encoding coefficient. The moving image processing apparatus according to claim 7, further comprising: determining a noise area.   The moving image processing apparatus according to claim 7, wherein the feature point selection unit adaptively changes a threshold value for determining a flat region and a noise region in a code region.   The moving image processing apparatus according to claim 9, wherein the threshold value is a numerical value depending on quantization information for each block.   The moving image according to claim 6, wherein the feature point selection unit includes means for selecting a feature point by excluding a region having a simple shape in the code region based on a distribution of coding coefficients. Image processing device.   The moving image processing apparatus according to claim 11, wherein the feature point selection unit determines a region having a simple shape by using a bias in a distribution of encoding coefficients.   The feature point selection unit has a simple shape using the sum of absolute values of AC components in the horizontal direction, the sum of absolute values of AC components in the vertical direction, and the sum of absolute values of other low-frequency AC components. The moving image processing apparatus according to claim 12, wherein an area having a difference is determined.   The moving image according to claim 6, wherein the feature point selection unit includes means for selecting a feature point in a code area in a block that straddles a plurality of block boundaries when determining an evaluation of a block boundary. Image processing device.   15. The feature point selection unit calculates a determination criterion using a constant matrix obtained in advance by synthesizing an approximate decoding process and a high-pass filter when determining an evaluation of a block boundary. The moving image processing apparatus according to 1.   The moving image processing apparatus according to claim 15, wherein the feature point selection unit includes means for reducing the number of computations by holding calculation results during the determination criterion calculation process.   The motion estimation unit includes a motion search unit that estimates the motion of the feature point from the position information of the feature point, and a determination unit that reselects the feature point from a prediction error obtained during the search using the decoded moving image. The moving image processing apparatus according to claim 1.   18. The moving image processing apparatus according to claim 17, wherein the motion search means searches for a motion vector so as to minimize the prediction error evaluation function in units of small regions including feature points.   The moving image processing apparatus according to claim 18, wherein the prediction error evaluation function includes a weighting factor arbitrarily set for a feature point.   20. The moving image processing apparatus according to claim 19, wherein the weighting coefficient can be an absolute value sum of AC components of coding coefficients or the number of non-zero coefficients.   The moving image processing apparatus according to claim 17, wherein the determination unit includes a unit that selects a feature point using a distribution of prediction errors.   The determination means includes means for selecting a feature point where a prediction error of a location indicated by a motion vector determined by motion search by the motion search means is sufficiently small and a neighborhood prediction error is relatively large. The moving image processing apparatus according to claim 21.   The matching unit uses a relative positional relationship between a plurality of feature points, a spatial distance between past feature points and predicted feature points, and feature points between frames using at least one of color information for each feature point. The moving image processing apparatus according to claim 1, further comprising: a unit that performs tracking to track a trajectory of a moving object.

Images