JP6584083B2 - Image processing apparatus and image processing program - Google Patents

Description

The present invention relates to an image processing technique for extracting a feature from an input image.

  In order to perform camera shake correction (anti-vibration) in the imaging apparatus, a motion vector indicating where the feature point extracted in the previous frame image of the video has moved in the subsequent frame image is detected, and an optical system is detected according to the motion vector The image pickup device is shifted or the cutout range of the video is shifted. The position of the feature point in the rear frame image is generally searched by template matching using a template image including the feature point extracted from the previous frame image.

  However, when a plurality of feature points are extracted from the entire previous frame image, the distribution of the plurality of feature points often becomes non-uniform. When motion vectors obtained for feature points that are unevenly distributed are used for camera shake correction, camera shake correction is performed in which motion vectors obtained in areas where feature points are concentrated, and the entire video However, it is difficult to perform good camera shake correction.

  In Patent Document 1, an input image is divided into a plurality of image regions (image blocks) in order to uniformly distribute feature points, and feature values representing the size of features are calculated for each pixel. Image processing for extracting a pixel having the largest feature value as a feature point is disclosed.

JP 2008-192060 A

  However, even in the image processing disclosed in Patent Document 1, if only a low feature value appears in the image region when extracting the feature point in the image region, the feature point cannot be extracted or template matching is not possible. Appropriate feature points may be extracted. As a result, correct motion vector detection and correct camera shake correction may not be possible.

  Note that in the image processing disclosed in Patent Document 1, when a feature point is not extracted in an image area, the motion vector calculation process is not performed in the image area. However, this makes the calculation of motion vectors non-uniform, and the meaning of dividing an input image into a plurality of image areas is reduced.

  The present invention provides an image processing apparatus capable of correctly extracting feature points from an input image with a uniform distribution.

An image processing apparatus according to an aspect of the present invention includes an extraction unit that performs an extraction process for extracting a feature portion in an input image, an analysis unit that performs image analysis on the input image, and the input image is divided into a plurality of image regions. Dividing means, and the analysis means performs the image analysis for each image region after the extraction process, and the extraction means determines whether to perform the extraction process again according to the result of the image analysis. When the extraction process is performed again, the feature extraction method in the extraction process is selected for each image area according to the result of the image analysis for each image area. To do.

  Note that an image pickup unit that picks up an image, the image processing device that extracts a feature portion of the input image generated by using the image pickup unit and calculates a motion vector of the feature portion, and vibration-proof using the motion vector An image pickup apparatus having an anti-vibration means for performing an operation also constitutes another aspect of the present invention.

An image processing program according to another aspect of the present invention is an image processing program that is a computer program that causes a computer to perform image processing, and the image processing is extraction processing that extracts a feature portion in an input image. And an analysis process for performing image analysis on the input image, and a process for dividing the input image into a plurality of image areas, and performing the image analysis for each of the image areas after the extraction process , Whether to perform the extraction process again according to a result is selected, and when the extraction process is performed again , the feature portion is extracted for each image area according to the result of the image analysis for each image area. you and selects the method.

  According to the present invention, the input image is divided into a plurality of image regions, and feature points are extracted for each image region using an extraction method according to the image analysis result. Correct feature points can be extracted.

1 is a block diagram illustrating a configuration of an image processing apparatus that is Embodiment 1 of the present invention. FIG. 6 is a diagram illustrating an example of dividing an input image according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration of a feature point extraction unit in the image processing apparatus according to the first embodiment. FIG. 3 is a block diagram illustrating a configuration of a template matching unit in the image processing apparatus according to the first embodiment. 3 is a flowchart illustrating processing performed by the image processing apparatus according to the first exemplary embodiment. 3 is a graph showing image analysis in Example 1. 5 is a flowchart illustrating processing performed by a feature point extraction unit according to the first embodiment. The figure which shows the contour line of the relationship between the eigenvalue and feature value by the difference in a feature evaluation formula. 5 is a flowchart illustrating processing performed by a template matching unit according to the first embodiment. FIG. 6 is a diagram for explaining template matching in the first embodiment. FIG. 3 is a block diagram illustrating a configuration of an image processing apparatus that is Embodiment 2 of the present invention. 9 is a flowchart illustrating processing performed by the image processing apparatus according to the second embodiment. The block diagram which shows the structure of the image processing apparatus which divided | segmented the image-analysis part shown in FIG. 14 is a flowchart showing processing performed by the image processing apparatus in FIG. 13. FIG. 6 is a block diagram illustrating a configuration of an imaging apparatus that is Embodiment 3 of the present invention.

  Embodiments of the present invention will be described below with reference to the drawings.

  FIG. 1 shows the configuration of an image processing apparatus that is Embodiment 1 of the present invention. The image processing apparatus 100 includes an image input unit 101, a grid division unit 102, an image analysis unit 103, a band pass filter unit 104, and a memory 105. Further, the image processing apparatus 100 includes a feature point extraction unit 106, a template matching unit 107, a reliability determination unit 108, and a motion vector detection unit 109.

  A frame image (input image) of a video signal generated by imaging is input to the image input unit 101. The image processing apparatus according to the present embodiment extracts feature portions (hereinafter referred to as feature points) in the frame image. FIG. 2A shows an example of a frame image. 201 is a main subject, and 202 is a background area. If processing for extracting feature points in the frame image is performed in this state, a plurality of feature points are concentrated and extracted around the main subject 201.

  The grid dividing unit (dividing unit) 102 divides the frame image into a plurality of image regions (hereinafter referred to as image blocks) by a grid 203 as shown in FIG. By extracting at least one feature point for each image block, it is possible to prevent the extracted feature points from being concentrated around the main subject 201.

  The image analysis unit (analysis unit) 103 performs image analysis processing on the contrast, the maximum pixel value, and the repetitive pattern for each image block divided by the grid division unit 102. Specifically, low contrast determination, pixel value maximum value protrusion determination, and repetitive pattern determination are performed. Details of these determinations will be described later.

  The band pass filter unit 104 performs a band pass filter process that cuts a high frequency component and a low frequency component of the image block by applying a band pass filter to each image block. The band pass filter unit 104 sets (selects) the frequency characteristics of the band pass filter used for the band pass filter processing for each image block in accordance with the result of the image analysis performed by the image analysis unit 103.

  The memory 105 temporarily stores (holds) a plurality of image blocks divided by one frame image or a plurality of frames for each frame image sequentially input in a frame cycle. When using the image block held in the memory 105 in the template matching unit 107, an image block (hereinafter also referred to as a band-pass image block) from which high-frequency and low-frequency components have been cut by the band-pass filter unit 104 is used. Retained.

  The feature point extraction unit (extraction unit) 106 extracts at least one feature point for each bandpass image block using the result of the image analysis by the image analysis unit 103 and the bandpass image block from the bandpass filter unit 104. Perform the process. Here, the extraction of the feature points is performed on a bandpass image block (hereinafter referred to as a previous frame image block) obtained from the previous frame image as the input image.

  A template matching unit (corresponding unit detection unit) 107 performs a template matching process. First, the template matching unit 107 creates a template image (hereinafter simply referred to as a template) by cutting out a predetermined small-size partial image for each feature point extracted by the feature point extraction unit 106 from the previous frame image block. Then, the correlation between this template and a candidate area which is a small-size image area in a band-pass image block (hereinafter referred to as a “back-frame image block”) that is divided and stored in the memory 105 after the subsequent-frame image (post-input image) The values are sequentially calculated while moving the candidate area. Corresponding points (corresponding portions) corresponding to the feature points are included in the candidate region having the minimum correlation value.

  A reliability determination unit (reliability determination unit) 108 performs low contrast determination of the candidate region, maximum value protrusion determination of the pixel value (correlation value), and repetition pattern determination using the correlation value calculated by the template matching unit 107. Do. When it is determined that the candidate area has low contrast, the maximum correlation value in the candidate area is low (low peak), or the image of the candidate area is a repetitive pattern, the reliability of the detected corresponding point is low Is determined.

  The motion vector detection unit (vector detection unit) 109 has the minimum value between the position of the feature point in the previous frame image block and the correlation value calculated by the template matching unit 107 in the subsequent frame image block (that is, the highest similarity). The position of the candidate area is high). Then, a motion vector is detected (calculated) using the position of the feature point and the position of the corresponding point. At this time, the motion vector detection unit 109 selects whether to detect a motion vector according to the reliability determination result input from the reliability determination unit 108.

  FIG. 3 shows the configuration of the feature point extraction unit 106 that performs feature point extraction processing. The feature point extraction unit 106 includes a feature filter unit 1061, a feature evaluation unit 1062, and a feature point determination unit 1063.

  The feature filter unit 1061 holds a plurality of types of feature point extraction filters (hereinafter also referred to as feature filters) such as a horizontal / vertical primary differential filter, a horizontal / vertical secondary differential filter, and a smoothing filter. Each feature filter can adjust (change) the parameter value. The feature filter unit 1061 changes (selects) the type of feature filter to be applied and its parameter value for each bandpass image block (previous frame image block) according to the result of image analysis from the image analysis unit 103. . Then, the selected feature filter and its parameter value are applied to the bandpass image block to perform a filter process.

  For each previous frame image block that has been filtered by the feature filter unit 1061, the feature evaluation unit 1062 has pixels having a large differential value in a plurality of directions such as an intersection of two edges and a maximum point of curvature on a curve (hereinafter, referred to as a pixel). ) (Referred to as a candidate point) is calculated by a feature evaluation formula. The feature evaluation unit 1062 prepares a plurality of types of feature evaluation formulas, which are calculation formulas for calculating feature values, and which feature is selected for each previous frame image block according to the result of image analysis in the image analysis unit 103. Select whether to use the evaluation formula.

  The feature point determination unit 1063 determines the candidate point (pixel) for which the largest feature value is calculated among the feature values calculated for each pixel by the feature evaluation unit 1062 as the feature point in the previous frame image block.

  FIG. 4 shows the configuration of the template matching unit 107 that performs template matching processing. The template matching unit 107 includes a correlation value calculation unit 1071 and a correlation value determination unit 1072.

  The correlation value calculation unit 1071 creates a template having the feature point extracted by the feature point extraction unit 106 as the central pixel for the previous frame image block as the reference image. Then, a correlation value between the template and a candidate area in a post-frame image block as a reference image is calculated.

  The correlation value determination unit 1072 calculates the minimum of the correlation values calculated by the correlation value calculation unit 1071, and the position where the minimum correlation value is obtained, that is, the subsequent frame corresponding to the feature point in the previous frame image block Outputs information about corresponding points in an image block.

  The flowchart of FIG. 5 shows the overall flow of image processing in the image processing apparatus 100 configured as described above. This process is executed according to an image processing program, which is a computer program, by the image processing apparatus 100 that is configured entirely by a computer.

  In this embodiment, as described above, the motion vector is detected between the previous frame image (block) and the subsequent frame image (block). However, the preceding and following frame images are not necessarily temporally adjacent frame images. Good. This is the same in the second embodiment described later.

  In step S501, the image input unit 101 captures the previous frame image and the rear frame image.

  In step S <b> 502, the grid dividing unit 102 divides the previous and next frame images into a plurality of image blocks, and causes the memory 105 to hold a plurality of previous frame image blocks and a plurality of subsequent frame image blocks.

  In step S503, the image analysis unit 103 performs the above-described three determinations, that is, the low contrast determination, the maximum pixel value protrusion determination, and the repetitive pattern determination as image analysis for each previous frame image block held in the memory 105. Do. The image analysis unit 103 uses the luminance value of the pixel as the pixel value of the image block, obtains the maximum value, the minimum value, the average value, and the maximum value of the luminance value, and uses them for the above three determinations.

  FIG. 6 shows the relationship between the pixel value and the above three determinations. FIG. 6A shows a case where three determinations give good results. In the low contrast determination, when the difference between the maximum value and the minimum value of the luminance values of the pixels in the image block is smaller than a predetermined threshold, it is determined that the image block has a low contrast. FIG. 6B shows a case where it is determined that the contrast is low due to the low contrast determination, and the difference between the maximum value and the minimum value of the pixel value is smaller than that in FIG.

  In the maximum value protrusion determination of the pixel value, it is determined how much the maximum value (peak) of the luminance value of the pixel in the image block stands out. Specifically, when the peak evaluation value obtained by dividing the difference between the maximum value and the average value of the luminance value by the difference between the maximum value and the minimum value of the luminance value is smaller than a predetermined threshold, the image block Is determined to be a low peak. On the other hand, when the peak evaluation value is larger than the threshold value, the image block is determined to be a high peak. FIG. 6C shows a case where the peak value is determined to be low by the maximum value protrusion determination of the pixel value, and the peak evaluation value is smaller than that in FIG.

  In the repetitive pattern determination, when the difference between the maximum luminance value and the maximum value in the image block is smaller than a predetermined threshold, it is determined that the image in the image block is a repetitive pattern. FIG. 6D shows a case where the repetition pattern is determined to be a repetition pattern, and the difference between the maximum pixel value and the maximum value is smaller than that in FIG.

  In this embodiment, the case where image analysis is performed using the luminance value of the pixel has been described. However, image analysis may be performed using color components. Further, the low contrast determination, the maximum pixel value protrusion determination, and the repetitive pattern determination may be performed by methods other than those described above as long as the states shown in FIGS. 6A to 6D can be determined. Further, it is not always necessary to perform the low contrast determination, the pixel value maximum value protrusion determination, and the repeated pattern determination, and at least one may be performed.

  In step S504, the bandpass filter unit 104 performs bandpass filter processing for each previous frame image block and each subsequent frame image block divided in step S502. As described above, the frequency characteristics of the bandpass filter are changed for each image block according to the result of the image analysis performed in step S503.

  Table 1 shows the relationship between the image analysis result and the frequency characteristics of the bandpass filter. As a result of image analysis, when it is determined that the image block has low contrast or low peak (hereinafter referred to as case 1), the bandpass filter unit 104 sets the passband of the bandpass filter to the high frequency side. Thereby, even when the image block has a low contrast or a low peak, the edge can be enhanced. However, in this case, noise is also emphasized, so the difference between the luminance value of the isolated point of the noise and the maximum value of the luminance value in the image block becomes a certain value or more by changing the passing edge on the low frequency side. In this way, parameters for noise suppression are set.

  As a result of the image analysis, when it is determined that the image in the image block is a repetitive pattern (hereinafter referred to as case 2), the bandpass filter unit 104 sets the passband of the bandpass filter to the low frequency side. As a result, the repetitive pattern existing in the image block becomes unclear, and the influence when there are a plurality of maximum brightness values in one image block or there is almost no difference between the maximum value and the maximum value is suppressed. Can do.

  In step S505, the feature point extraction unit 106 extracts feature points in each previous frame image block subjected to the bandpass filter processing in step S504. The flowchart of FIG. 7 shows the flow of feature point extraction processing performed by the feature point extraction unit 106.

  In step S701, the feature filter unit 1061 performs filter processing on the previous frame image block as the bandpass image block generated in step S504. At this time, the feature filter unit 1061 includes a feature filter (horizontal / vertical primary differential filter, horizontal / vertical secondary differential filter, smoothing filter, etc.) used in the filter processing according to the result of the image analysis in step S503. Change the parameter value.

  Table 1 shows the relationship between image analysis results, feature filters, and parameter values. In case 1 described above, since the change in the pixel value in the image block is gentle, the feature filter unit 1061 selects the secondary differential filter as the horizontal and vertical differential filters, respectively. As a result, a sharper output can be obtained as compared with the case where the first-order differential filter is used. However, as described above, in case 1, noise is also emphasized by setting the frequency characteristics of the bandpass filter to the high frequency side. For this reason, an image with much noise such as a high-sensitivity image may be further reduced with a smoothing filter in addition to noise suppression with the band-pass filter. In this embodiment, a Gaussian filter is used as the secondary differential filter. The filter coefficient G (x, y) of the Gaussian filter is expressed by the following equation (1).

The smaller the value of σ in Equation (1), the smaller the effect of smoothing by the secondary differential filter, and the larger the value of σ, the greater the effect. The feature filter unit 1061 adjusts the filter coefficient G (x, y) (that is, σ) of Expression (1) as the parameter value according to the result of the image analysis.

  Further, in the case 2 described above, the feature filter unit 1061 selects the primary differential filter as the horizontal and vertical differential filters, respectively. As a result, an unclear output is obtained as compared with the case of using a second-order differential filter, and the influence of the repeated pattern is suppressed. Further, in this case 2, the feature filter unit 1061 selects the Gaussian filter having the filter coefficient expressed by the equation (1) as a smoothing filter, and applies the smoothing to the previous frame image block after applying the first-order differential filter. Apply a filter. Then, the filter coefficient of Expression (1) is adjusted as the parameter value of the smoothing filter according to the result of the image analysis.

  In this embodiment, a Gaussian filter is used as the smoothing filter, but other smoothing filters such as a moving average filter may be used.

  In this embodiment, the frequency characteristics of the bandpass filter used in the bandpass filter process in step S504 and the feature filter and parameter values used in the feature point extraction process in step S505 are the same as the image analysis performed in step S503. Select according to the result. However, as shown in FIGS. 13 and 14, respectively, the image analysis unit 1031 for the feature point extraction process and the step of performing image analysis (S5031), the image analysis unit 103 for the bandpass filter process and the image analysis are performed. It may be provided separately from the step of performing (S503). That is, image analysis may be performed on the bandpass image block generated in step S504, and the feature filter and parameter value may be selected according to the result.

  In step S702, the feature evaluation unit 1062 calculates a feature value using the feature evaluation formula for the above-described candidate points (pixels) in the previous frame image block subjected to the filtering process in step S701. As described above, the feature evaluation unit 1062 selects a feature evaluation formula to be used according to the result of the image analysis in step S503 from a plurality of types of feature evaluation formulas.

  In this embodiment, the feature evaluation unit 1062 prepares a Harris corner detector feature evaluation formula and a Shi and Tomasi feature evaluation formula.

  The feature evaluation unit 1062 creates an autocorrelation matrix H expressed by the following equation (2) from the pixel values as a result of applying the horizontal and vertical differential filters in step S701.

In Expression (2), Ix represents a pixel value as a result of applying the horizontal differential filter, and Iy represents a pixel value as a result of applying the vertical differential filter. * Represents the convolution of the Gaussian filter G.
The characteristic evaluation formula of the Harris detector is shown in the following formula (3).

In Expression (3), α is a constant, and a value of 0.04 to 0.15 is considered to be good experimentally. det represents the determinant of the autocorrelation matrix H, and tr represents the sum of the main diagonal components of the autocorrelation matrix H.

  Further, Shi and Tomasi's characteristic evaluation formula is shown in the following formula (4).

In equation (4), the smaller eigenvalue of eigenvalues λ1 and λ2 of autocorrelation matrix H in equation (2) is used as the feature value.

  Table 1 shows the relationship between the image analysis result and the feature evaluation formula to be selected. Further, in FIG. 8, the contour lines of the relationship between the eigenvalues and the characteristic values of the equations (3) and (4) are shown. The portion displayed as “corner” in FIG. 8 is a candidate point. As shown in the figure, since the expression (3) has a smaller area displayed as “corner” than the expression (4), by using the expression (3), the calculation of the feature value more strictly, that is, the feature point Is determined.

  In the case 1 described above, since the change in the pixel value in the image block is gentle, the expression (4) (Shi and Tomasi) with a gentle feature point determination is selected. On the other hand, in the case 2, the expression (3) (Harris corner) for selecting the feature point more strictly is selected.

  In the present embodiment, the Harris corner detector and Shi and Tomasi feature evaluation formulas are used to calculate the feature values. However, other feature evaluation formulas such as a harmonic average may be used.

  In step S703, the feature point determination unit 1063 determines the candidate point (pixel) for which the maximum value is calculated among the feature values calculated for each candidate point in the previous frame image block in step S702 as the feature point.

  Returning to FIG. 5, in step S506, the template matching unit 107 creates a template centered on the feature point extracted from the previous frame image block in step S505. Then, as described above, the correlation value between the template and the candidate area in the subsequent frame image block is sequentially calculated while moving the candidate area.

  The flowchart of FIG. 9 shows the flow of template matching processing performed by the template matching unit 107.

  In step S901, the correlation value calculation unit 1071 calculates a correlation value between the template and the candidate area of the subsequent frame image block. FIG. 10 shows an outline of template matching.

  FIG. 10A shows four front frame image blocks as reference images, and FIG. 10B shows four rear frame image blocks as reference images. In FIG. 10A, 1001 (black circle) indicates a feature point, and 1002 (frame) indicates a template. In FIGS. 10A and 10B, reference numeral 1003 denotes a grid that divides the preceding and following frame images into four image blocks. Further, in FIG. 10B, 1004 (solid line frame) indicates a search range for searching for the corresponding point 1007 in the reference image, and 1005 (dotted line frame) indicates a candidate area for which a correlation value is calculated.

  As shown in FIG. 10A, the correlation value calculation unit 1071 first creates a template 1002 by cutting out a small-size partial image centered on a feature point in the reference image. Next, the correlation value calculation unit 1071 moves the candidate area 1005 within the template block 1002 and the search area 1004 in the reference image shown in FIG. 10B, while correlating the template with the candidate area 1005 at each position. Is calculated. Note that calculating the correlation value by moving the candidate area over the entire area of the reference image requires an enormous amount of computation. Therefore, in this embodiment, the search for calculating the correlation value in a part of the reference image is performed. A range 1004 is set. Although the position and size of the search range 1004 are not limited, the search range 1004 needs to include corresponding points corresponding to feature points in the reference image.

  In this embodiment, a case where a sum of absolute differences (hereinafter abbreviated as SAD) is used as a correlation value will be described. A formula for calculating SAD is shown in Formula (5).

In equation (5), f (i, j) represents the luminance value at the coordinates (i, j) in the template 1002. G (i, j) represents a luminance value at the coordinates (i, j) in the candidate area 1005 in which the correlation value is calculated in the search range 1004.

  In SAD, a correlation value S_SAD is obtained by calculating an absolute value of a difference between these luminance values f (i, j) and g (i, j) and obtaining a sum thereof. The smaller the correlation value S_SAD, the smaller the difference in brightness value between the template 1002 and the candidate area 1005, that is, the texture in the template 1002 and the candidate area 1005 is similar.

  In the present embodiment, the case where SAD is used as the correlation value has been described. However, other correlation values such as sum of squared differences (SSD) and normalized cross correlation (NCC) may be used.

  In step S902, the correlation value determination unit 1072 outputs information on the position of the candidate area 1005 where the correlation value calculated in step S901 is the minimum, that is, the position of the corresponding point in the subsequent frame image block.

  Returning to FIG. 5 again, in step S507, the reliability determination unit 108 uses the correlation value calculated in step S506 as the pixel value, and obtains the maximum value, the minimum value, the average value, and the minimum value of the correlation value. Then, using the maximum value, the minimum value, the average value, and the minimum value of the correlation values, as the determination regarding the reliability of the corresponding point, the low contrast determination, the pixel value maximum value protrusion determination, and the repeated pattern determination are performed.

  The contents of these three determinations as reliability determinations for corresponding point detection are basically the same as the three determinations in the image analysis described with reference to FIGS. However, since the similarity between the template and the candidate region is higher as the correlation value is smaller, the maximum pixel value in FIGS. 6A to 6D represents the minimum value in the correlation value, and the minimum pixel value is the correlation value. The value represents the maximum value. Further, the maximum value of the pixel value represents the minimum value in the correlation value.

  FIG. 6A shows a case where the above three determinations are good, that is, a result that the reliability of the detected corresponding point is high is obtained. FIG. 6B shows a case where the difference between the maximum value and the minimum value of the correlation value is smaller than a predetermined threshold, that is, a case where it is determined that the candidate region has low contrast. FIG. 6C shows a case where the evaluation value obtained by dividing the difference between the minimum value and the average value of the correlation value by the difference between the maximum value and the minimum value of the correlation value is smaller than a predetermined threshold, that is, the candidate area is low. The case where it is determined to be a peak is shown. In addition, when the said evaluation value is larger than a threshold value, it determines with a candidate area | region being a high peak. FIG. 6D shows a case where the difference between the minimum value and the minimum value of the correlation values in the candidate area is smaller than a predetermined threshold, that is, the image of the candidate area is determined to be a repetitive pattern. . As described above, the determination that the candidate area has a low contrast, the determination that the candidate area has a low peak, and the determination that the image of the candidate area is a repetitive pattern are the reliability of the detected corresponding point, In other words, it is synonymous with the determination that the reliability of the feature point and the corresponding point is low.

  In FIG. 5, in step S508, the motion vector detection unit 109 corresponds to the feature point that is the object of the reliability determination when the reliability determination result in step S507 is a determination result that the reliability is low. A motion vector is detected using the points. In other words, a vector having a feature point position as a start point and a corresponding point position as an end point is detected. When it is determined that the reliability determination result in step S507 is low contrast, low peak, or a repetitive pattern, the motion vector detection unit 109 performs motion using the feature points and corresponding points that are targets of reliability determination. Vector detection is not performed.

  According to the present embodiment, since the input image (previous frame image) is divided into a plurality of image blocks and the feature points are extracted using the extraction method according to the image analysis result for each image block, Feature points can be extracted accurately with a uniform distribution. In addition, when the corresponding points in the reference image (back frame image) corresponding to the feature points in the base image (previous frame image) are obtained, the motion vector is detected if the reliability of the feature points and the corresponding points is low. Therefore, the reliability of the detected motion vector can be improved.

  FIG. 11 shows the configuration of an image processing apparatus 1100 that is Embodiment 2 of the present invention. In the present embodiment, the same reference numerals as those in the first embodiment are assigned to components common to the first embodiment (FIG. 1).

  An image processing apparatus 1100 according to the present exemplary embodiment includes an image analysis unit 1101 instead of the image analysis unit 103 illustrated in FIG. In the image processing apparatus 1100 according to the present embodiment, the feature evaluation unit 1102 included in the feature extraction unit 106 in the first embodiment is arranged between the image analysis unit 1101 and the template matching unit 107 as shown in FIG. Is arranged.

  In the image processing apparatus 100 according to the first embodiment, the image analysis unit 103 is disposed in front of the bandpass filter unit 104 and the feature extraction unit 106. As a result, the luminance value was used as the pixel value in the image analysis, and the result of the image analysis was used to select the frequency characteristics of the bandpass filter, the feature filter, its parameter value, and the feature evaluation formula. On the other hand, in the image processing apparatus 1100 of this embodiment, the image analysis unit 1101 is arranged after the band pass filter unit 104 and the feature extraction unit 106. Thereby, image analysis is performed using the feature value obtained by the feature extraction unit 106. As a result of the image analysis, when the determination of low contrast, low peak, or repetitive pattern is made, the bandpass filter unit 104 and the feature evaluation unit 1102 display the frequency characteristics of the bandpass filter, the feature filter, its parameter value, and the feature evaluation formula Select again (change). Then, bandpass filter processing and feature point extraction processing are performed again.

  The flowchart of FIG. 12 shows the overall flow of image processing in the image processing apparatus 1100 of the present embodiment. This process is executed according to an image processing program, which is a computer program, by an image processing apparatus 1100 that is entirely configured by a computer. Note that steps in FIG. 12 that are common to the steps shown in FIG. 5 are given the same step numbers as in FIG. 5 and description thereof is omitted.

  In step S1201 after step S501 and step S502, the bandpass filter unit 104 performs bandpass filter processing for each previous frame image block and each subsequent frame image block divided in step S502. In the first main step, since image analysis has not been performed yet, the bandpass filter unit 104 sets the frequency characteristic of the bandpass filter to the default frequency characteristic. In the present embodiment, the default frequency characteristic is set so that the middle band is the pass band.

  In the second and subsequent steps, the band-pass filter unit 104 determines the frequency characteristics of the band-pass filter for each previous frame image block and each subsequent frame image block according to the result of image analysis performed in step S1203 described later. select. The relationship between the result of image analysis and the frequency characteristics of the bandpass filter is the same as in the first embodiment.

  In step S1202, the feature point extraction unit 106 performs a filtering process using a feature filter on each previous frame image block subjected to the bandpass filter processing in step S1201, and further uses a feature evaluation formula for the feature point candidate points. To calculate the feature value. Since image analysis has not yet been performed in the first main step, the feature point extraction unit 106 sets a default feature filter, its parameter value, and a feature evaluation formula. Also in this step, the feature point extraction unit 106 determines the candidate point having the maximum feature value as the feature point of the previous frame image block, as in the first embodiment.

  In this embodiment, a first-order differential filter is adopted as a horizontal and vertical differential filter by default, and a Gaussian filter in which σ in the filter coefficient G (x, y) shown in Expression (1) is 1 is used as a smoothing filter. Adopt by default. The Harris corner detector characteristic evaluation formula (formula (3)) described in the first embodiment is adopted as a default.

  In the second and subsequent steps, the feature point extraction unit 106 selects a feature filter, its parameter value, and a feature evaluation formula for each previous frame image block in accordance with the result of image analysis performed in step S1203 described later. It changes for every grid from the result of the image analysis performed in below-mentioned step S1203. The relationship between the result of image analysis, the feature filter, its parameter value, and the feature evaluation formula is the same as in the first embodiment.

  In step S1203, the image analysis unit 1101 performs low contrast determination, pixel value maximum value protrusion determination, and repetitive pattern determination described in the first embodiment as image analysis for each previous frame image block. At this time, the image analysis unit 103 uses the feature value calculated in step S1202 as the pixel value of the image block, obtains the maximum value, the minimum value, the average value, and the local maximum value of the feature value, and uses them for the above three determinations. .

  FIG. 6A shows a case where the above three determinations are good, that is, a result that the reliability of the feature points detected in the image block is high is obtained. FIG. 6B shows a case where the difference between the maximum value and the minimum value of the feature value is smaller than a predetermined threshold value, that is, a case where the image block from which the feature point is extracted is determined to have low contrast. Yes. FIG. 6C shows a case where the evaluation value obtained by dividing the difference between the maximum value and the average value of the feature values by the difference between the maximum value and the minimum value of the feature values is smaller than a predetermined threshold, that is, the image block is low. The case where it is determined to be a peak is shown. When the evaluation value is larger than the threshold value, the image block from which the feature point is extracted is determined to be a high peak. FIG. 6D shows a case where the difference between the maximum feature value and the maximum value is smaller than a predetermined threshold, that is, the case where the image of the image block from which the feature point is extracted is determined to be a repetitive pattern. Show. As described above, the determination that the image block from which the feature point is extracted has a low contrast, the determination that the image block has a low peak, and the determination that the image of the image block is a repetitive pattern are extracted. This is synonymous with the determination that the reliability of the selected feature point is low.

  In step S1204, the feature evaluation unit 1102 selects (determines) whether or not to perform the bandpass filter process and the feature point extraction process again according to the determination result of the reliability of the feature point in step S1203. If the above-described determination of low contrast, low peak, and repetitive pattern (determination that the reliability of the feature point is low) is made, the process returns to step S1201 and step S1202. In step S1201, the bandpass filter unit 104 changes the frequency characteristics of the bandpass filter according to the result of the image analysis. Then, a band pass filter having a frequency characteristic after the change is applied to the previous frame image block and the corresponding subsequent frame image block, which have been targets for feature point reliability determination, and the band pass filter process is performed again. I do. In step S1202, the feature point extraction unit 106 changes the feature filter, its parameter value, and the feature evaluation formula according to the result of the image analysis. Then, feature point extraction processing is performed on the previous frame image block that has been subjected to the pan-pass filter processing by the pan-pass filter having the changed frequency characteristics, using the changed feature filter, parameter value, and feature evaluation formula.

  If no determination of low contrast, low peak, or repetitive pattern has been made in step S1204 (determination that the reliability of the feature point is high), the process proceeds to steps S506 to S508. Then, this process ends.

  In this embodiment, the input image (previous frame image) is divided into a plurality of image blocks, and feature points are extracted for each image block. If it is determined that the reliability of the feature point is low as a result of the subsequent image analysis for each image block, the feature point is extracted again using an extraction method according to the image analysis result (reliability). For this reason, it is possible to correctly extract feature points from the input image with a uniform distribution.

  Moreover, in the first embodiment described above, the frequency characteristics of the bandpass filter, the feature filter, its parameter value, and the feature evaluation value are selected and fixed according to the result of one image analysis. In this method, the process proceeds until the reliability determination unit 108 determines that the reliability of the feature point and the corresponding point is low, and as a result, the process is performed without calculating the motion vector using the feature point and the corresponding point. Ends.

  In contrast, in this embodiment, image analysis is performed at the time when the feature value is calculated, and the bandpass filter processing is performed while changing the frequency characteristic of the bandpass filter, the feature filter, etc. until the reliability of the feature point is ensured. And the feature point extraction process is performed. Thereby, when it is determined by the reliability determination unit 108 that the reliability of the feature points and the corresponding points is low, that is, the number of image blocks in which no motion vector is detected can be reduced as compared with the first embodiment.

  FIG. 15 shows an imaging apparatus 500 such as a digital still camera or a video camera including the image processing apparatus described in the first embodiment or the second embodiment (indicated by reference numeral 300 in FIG. 15).

The imaging apparatus 500 uses an imaging element (imaging means) 502 such as a CMOS sensor or a CCD sensor that captures (photoelectric conversion) a subject image formed by the imaging optical system 501 and an output signal from the imaging element 502. And a video signal generation unit 503 that generates a signal. The video signal is recorded on the recording medium 504 or displayed on the monitor 505. Note that the imaging optical system 501 may be provided integrally with the imaging device 500 or may be provided so as to be replaceable with respect to the imaging device 500.
The image processing apparatus 300 in the imaging apparatus 500 performs the image processing described in the first embodiment or the second embodiment using the frame image of the video signal generated by the video signal generation unit 503 as an input image, thereby obtaining a motion vector. To detect. In the present embodiment, the detected motion vector corresponds to a video shake caused by a hand shake of a user who is picking up an image by holding the image pickup apparatus 500 by hand.

  The imaging apparatus 500 is also provided with an electronic image stabilization unit (anti-vibration unit) 506 that performs a so-called electronic image stabilization operation. The electronic image stabilization unit 506 uses the motion vector information detected by the image processing apparatus 300 to determine a range to be cut out for recording or display from the video signal (continuous frame image) generated by the video signal generation unit 503. Shift. This reduces (corrects) the shake of the recorded video or display video. Note that the user can arbitrarily select whether or not the electronic image stabilization unit 506 performs the electronic image stabilization operation.

In the video signal generated by the video signal generation unit 503, a so-called optical image stabilization operation for shifting the optical element in the imaging optical system 501 and the imaging element 502 is performed using the detected motion vector information. Image blur may be reduced.
(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  Each embodiment described above is only a representative example, and various modifications and changes can be made to each embodiment in carrying out the present invention.

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 102 Grid division part 103 Image analysis part 106 Feature point extraction part 1001 Feature point

Claims (8)

Extraction means for performing extraction processing for extracting a feature portion in the input image;
Analyzing means for performing image analysis on the input image;
Dividing means for dividing the input image into a plurality of image regions,
The analysis means performs the image analysis for each of the image regions after the extraction process ,
The extraction means selects whether or not to perform the extraction process again according to the result of the image analysis, and when performing the extraction process again , according to the result of the image analysis for each image area, An image processing apparatus, wherein a method for extracting the feature portion in the extraction processing is selected for each image region.   The extraction means selects at least one of a type of a filter used for the extraction process, a parameter value of the filter, and a type of an arithmetic expression for calculating a feature value in the image area as the extraction method. The image processing apparatus according to claim 1, wherein:   The image processing apparatus according to claim 1, wherein the analysis unit performs an analysis on at least one of contrast, a maximum pixel value, and a repetitive pattern as the image analysis.   The image processing apparatus according to any one of claims 1 to 3, wherein the analysis unit changes the extraction method according to a reliability of the feature obtained by the image analysis. A corresponding part detecting means for detecting a corresponding part corresponding to the characteristic part in another input image generated after the input image;
The image processing apparatus according to any one of 4 claims 1, characterized in that it comprises a vector detecting means for detecting a motion vector using the corresponding portion and the feature. Having a reliability determination means for determining the reliability of the feature part or the corresponding part;
The image processing apparatus according to claim 5, wherein the vector detection unit selects whether or not to detect the motion vector according to the reliability. Imaging means for imaging;
The image processing apparatus according to claim 5 or 6 , wherein a feature part of an input image generated using the imaging unit is extracted and a motion vector of the feature part is calculated.
An image pickup apparatus comprising: an image stabilization unit that performs an image stabilization operation using the motion vector. An image processing program that is a computer program that causes a computer to perform image processing,
The image processing is
An extraction process for extracting features in the input image;
Analysis processing for performing image analysis on the input image;
Dividing the input image into a plurality of image regions,
Performing the image analysis for each image region after the extraction process ;
Whether to perform the extraction process again according to the result of the image analysis, and when performing the extraction process again , depending on the result of the image analysis for each image area, An image processing program characterized by selecting a feature extraction method.