JP7580914B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to image processing technology for detecting image movement.

Conventionally, in order to provide a realistic video experience, stereo cameras capable of capturing images for stereoscopic display have been provided. When detecting a motion vector using image data captured by a stereo camera, since the image data contains images of almost the same subject, it is inefficient to detect the motion vector in the entire image data. Patent Document 1 discloses a technology that can solve this problem by devising a template matching method. In template matching, one of two frame images in an image is used as the original image and the other is used as the reference image. Then, a rectangular area placed on the original image is used as the template block, and a rectangular area placed on the reference image is used as the search area, and a correlation with the distribution of pixel values in the template block is obtained at each position of the search area. Here, when the stereo camera captures a left eye image and a right eye image, there is an overlapping area between the search area in the left eye image and the search area in the right eye image. In the technology of Patent Document 1, template matching is performed on only one of the left eye image or the right eye image for the overlapping search area, thereby reducing the computation cost.

JP 2000-165909 A

As mentioned above, in the method of Patent Document 1, in order to reduce computational costs, template matching is performed on only one of the left eye image or the right eye image for a search area that overlaps between the left eye image and the right eye image. However, depending on the movement of the camera caused by camera shake or the like, the image in the template block may move in different directions in the left eye image and the right eye image. In this case, template matching cannot be performed correctly, and as a result, motion vectors cannot be detected correctly, resulting in reduced motion vector detection performance.

Therefore, the present invention aims to reduce computational costs as well as improve motion vector detection performance.

The image processing device of the present invention comprises an acquisition means for acquiring a plurality of images having parallax due to a plurality of optical systems, an extraction means for extracting feature points from the images, a detection means for detecting a motion vector based on the feature points , a reflection means for reflecting the feature points extracted from a first image of the plurality of images on the feature points extracted from a second image of the plurality of images while taking into account the parallax, and a camera shake detection means for detecting camera shake in the rotational direction of an imaging device that captured the plurality of images having parallax due to the plurality of optical systems, wherein the detection means detects the motion vector for the second image based on feature points in which the feature points of the first image are reflected, and the camera shake detection means detects camera shake in the rotational direction of the imaging device using the motion vector detected based on the feature points of the first image and the motion vector detected based on the feature points of the second image in which the feature points of the first image are reflected .

The present invention makes it possible to reduce computational costs and also improve motion vector detection performance.

FIG. 1 illustrates an example of the configuration of a digital camera according to an embodiment. FIG. 4 is a diagram illustrating an example of the configuration of a motion vector detection unit. 1A and 1B are explanatory diagrams of a grid arrangement, feature points, and a template matching area. 13 is a flowchart showing a process performed by a motion vector detection unit. FIG. 4 illustrates an example of the configuration of a feature point extraction unit. 13 is a diagram showing the relationship between pixel values and judgments in an accuracy judgment unit. FIG. 13 is a flowchart showing a process of a tracking target feature point determining unit. 11A and 11B are diagrams illustrating an example of motion vector distribution when camera shake occurs in a rotational direction; FIG. 1 is a diagram showing examples of two image circles. FIG. 13 is a diagram showing an example of motion vector distribution of two image circle images. FIG. 11 is a diagram used to explain feature point reflection processing. 13 is a flowchart of a feature point reflection process. 13 is a flowchart of a feature point number setting process. FIG. 13 is a diagram showing an example of feature point distribution when the number of feature points is appropriate. 13 is a timing chart relating to motion vector detection in the case of 11 points. FIG. 13 is a diagram showing an example of feature point distribution when the number of feature points is inappropriate. 13 is a timing chart relating to motion vector detection in the case of 16 points.

Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings.
First Embodiment
1 is a diagram showing a schematic functional block configuration of a digital camera as an application example of an image processing device according to a first embodiment of the present invention. The digital camera of this embodiment is an imaging device (hereinafter, referred to as a stereo camera) capable of acquiring data of two parallax images having parallax corresponding to the left and right eyes, which are captured by, for example, two optical systems with wide angles of view corresponding to the left and right eyes. Note that in this embodiment, a configuration in which two optical systems are arranged in parallel is given as an example, but the present invention is not limited to this, and may be configured by three or more optical systems.

In FIG. 1, the first imaging optical unit 101 and the second imaging optical unit 131 each include a lens and an aperture, and the two are arranged in parallel to form a pair of optical systems. When taking a photograph, the first imaging optical unit 101 and the second imaging optical unit 131 perform focus adjustment and exposure adjustment, and form an image circle of the two optical systems on the image sensor 102. A specific example of the image circle of the two optical systems on the image sensor 102 will be described later using FIG. 9.

The image sensor 102 has a photoelectric conversion function that converts an optical image into an electrical signal (analog image signal), and is composed of a CCD or CMOS sensor. The A/D converter 103 converts the analog image signal from the image sensor 102 into digital image data (hereinafter, image data will be referred to simply as image, where appropriate).

DRAM 107 is a volatile memory that stores data and has sufficient storage capacity to store a predetermined number of still images, a predetermined amount of moving images, audio data, and other data, as well as constants and programs for the operation of CPU 112. DRAM is an abbreviation for Dynamic Random Access Memory (Dynamic Random Access Memory). Since the contents of DRAM are lost over time after writing, it is necessary to refresh it at regular intervals to maintain the memory. Memory control unit 106 writes data to and reads data from DRAM 107 in response to instructions from CPU 112 or data transfer unit 105.

The non-volatile memory control unit 108 writes and reads data to the ROM 109, which is a non-volatile memory, in response to instructions from the CPU 112. The ROM 109 is an electrically erasable and recordable memory, and an EEPROM or the like is used. The ROM 109 stores constants, programs, etc. for the operation of the CPU 112.

The bus 114 is a system bus, and the bus 115 is an image data bus.
The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that perform data transfer. DMA is an abbreviation for Direct Memory Access. DMA is one of the data transfer methods within a computer system, and is a method of directly transferring data between peripheral devices and main memory (RAM) without going through the CPU. DMAC is also an abbreviation for DMA controller. A DMA controller is a controller that controls direct data transfer between memories and between memories and hardware.

The CPU 112 is composed of a microcomputer that controls the entire stereo camera, and issues operational instructions to each functional block to execute various control processes. The CPU 112 controls the image processing unit 104, data transfer unit 105, memory control unit 106, non-volatile memory control unit 108, display control unit 110, operation unit 113, image sensor 102, etc. via the bus 114. The microcomputer in the CPU 112 executes programs recorded in the ROM 109 to realize each process according to this embodiment.

The display unit 111 is made up of a liquid crystal monitor or the like, and is controlled by the display control unit 110 to display various image data and the like on the display unit 111 .
The operation unit 113 includes switches, buttons, and the like that are operated by the user, and is used for operations such as turning the power on/off and the shutter on/off.

The image processing unit 104 is composed of a buffer memory and other components that perform various types of image processing, such as white balance adjustment, color interpolation, reduction/enlargement, filtering, encoding, and decoding, and further includes a motion vector detection unit 121 and a subject detection unit 122. The image processing unit 104 also performs various types of image processing on image data captured in the image circles of the two optical systems, taking into account the optical center shift, and can generate image data for stereoscopic display.

The subject detection unit 122 detects the main subject, such as the face or object of the subject, from the captured image using a known method. The effective range of the subject's coordinates is expressed, for example, by the start coordinates in the upper left corner of the image and the end coordinates in the lower right corner, and the subject's coordinate information is stored in the DRAM 107 via the data transfer unit 105 and memory control unit 106.

The motion vector detection unit 121 detects a motion vector based on the image data.
Here, as a technique for detecting a motion vector, a technique is known in which feature points are extracted and template blocks are arranged at the extracted feature points in order to improve the detection accuracy of the motion vector. However, when feature points are extracted from the entire image, the distribution of feature points is often uneven. If a motion vector obtained based on uneven feature points is used for, for example, image stabilization, the image stabilization will be dominated by the area where feature points are concentrated. Therefore, in order to distribute feature points uniformly, there is a technique in which an image is divided into a grid, a feature value representing the size of the feature is calculated for each pixel, and the pixel with the largest feature value in each grid is extracted as the feature point. The motion vector detection unit 121 of this embodiment performs template matching processing using a rectangular search area and a template area of a preset size centered on the feature point extracted in this way.

Fig. 2 is a block diagram showing an example of the configuration of the motion vector detection unit 121. Fig. 3 is a diagram showing the relationship between the grid arrangement, feature points, and template matching area read by the motion vector detection unit 121. Fig. 4 is a flowchart showing the flow of processing by each block of the motion vector detection unit 121 shown in Fig. 2.

The motion vector detection unit 121 tracks feature points between multiple frame images and detects motion vectors. The motion vector detection unit 121 takes one of two consecutive frame images on the time axis as an original image (image of the current frame) and the other as a reference image (image of the previous frame), and performs correlation value calculation processing on pixel values within a rectangular area of the reference image centered on the feature point and a rectangular area of the original image. The motion vector detection unit 121 then tracks the subject based on the motion vector detected by the correlation value calculation processing.

The configuration and operation of the motion vector detection unit 121 will be described below with reference to FIG. 2 and subsequent figures.
First, the relationship between the grid arrangement, the feature points, and the template matching area shown in FIG. 3 will be described. As shown in FIG. 3, the motion vector detection unit 121 divides a frame image into a plurality of grids. The grids are arranged in a predetermined size as feature point arrangement grids 302 and peripheral grids 301 (grids with dot patterns in FIG. 3) in a predetermined number in the horizontal and vertical directions. The motion vector detection unit 121 extracts one feature point 303 from each feature point arrangement grid 302. The example in FIG. 3 shows an example in which a feature point 303 indicated by coordinates (x, y) is extracted from a feature point arrangement grid 302 whose start coordinates are (Xs, Ys) and whose end coordinates are (Xe, Ye). The peripheral grids 301 are arranged around a group of feature point arrangement grids made up of the feature point arrangement grids 302. The peripheral grids 301 are an area in which feature point extraction is not performed but which is used in template matching. The motion vector detection unit 121 performs template matching processing by setting a rectangular search area 305 and a template area 304 of a predetermined size in the reference image of the previous frame, centered on the coordinates (x, y) of a feature point 303 extracted from the original image, which is the current frame.

Next, an overview of the processing in each block of the motion vector detection unit 121 will be described with reference to the flowchart of FIG.
In step S401, the first RDDMAC 221 included in the data transfer unit 105 reads input image data 241 of a current frame to be subjected to motion vector detection from the DRAM 107 via the bus 115. The RDDMAC is a controller that controls direct data reading from memory to hardware. The amount of data read at this time is set to an amount according to the sizes of the feature point arrangement grid 302 and the surrounding grid 301 set as shown in FIG. 3. The input image data 241 is data after various image processes are performed in the image processing unit 104. The image data 241 read in units of grids by the first RDDMAC 221 is output to the matching image generation unit 201 and the feature point extraction unit 202. The first RDDMAC 221 also outputs grid coordinate information 252 indicating the coordinate position of the read grid to the tracking target feature point determination unit 205 described later. After the process of step S401, the motion vector detection unit 121 performs the process of step S402, the process of the following step S403, and the process of step S404 in parallel.

When proceeding to step S402, the matching image generation unit 201 generates a matching image to be used for template matching during motion vector detection, and outputs it to the WRDMAC 231. The WRDMAC is a controller that controls the direct writing of data from the hardware to memory. The matching image generation unit 201 also has a band-pass filter circuit, which cuts out high-frequency and low-frequency components of the image signal that are not required for template matching processing.

Next, in step S403, the WRDMAC 231 of the data transfer unit 105 performs processing to write out the matching image data 242 input to the DRAM 107 via the bus 115. The DRAM 107 also stores matching image data 243 generated in the previous frame, which precedes the current frame on the time axis. After step S403, the processing of the motion vector detection unit 121 proceeds to step S405.

In step S404, the feature point extraction unit 202 or the feature point reflection unit 207 calculates new feature points for the current frame.
Here, a description will be given of the feature point extraction process in the feature point extraction unit 202. Fig. 5 is a block diagram showing an example of the configuration of the feature point extraction unit 202. The feature point extraction unit 202 is configured to include a feature filter unit 501, a feature evaluation unit 502, and a feature point determination unit 503.

The feature filter unit 501 is configured with multiple filters, such as a bandpass filter, a horizontal differential filter, a vertical differential filter, and a smoothing filter. The feature filter unit 501 first cuts out unnecessary high-frequency and low-frequency components from the image data using a bandpass filter. Next, the feature filter unit 501 further performs horizontal differential filtering using a horizontal differential filter and vertical differential filtering using a vertical differential filter on the data after bandpass filtering. The feature filter unit 501 then outputs the image data after smoothing filtering on each of the image data after horizontal differential filtering and the image data after vertical differential filtering to the feature evaluation unit 502.

The feature evaluation unit 502 calculates feature points from the image data in grid units after filtering by the feature filter unit 501. For example, the feature evaluation unit 502 performs a calculation using a predetermined feature evaluation formula and calculates, as feature values, points where the differential values around a pixel are large in multiple directions, such as the intersection of two edges or curved points with maximum curvature for each pixel.

Hereinafter, an example of calculation of feature points in the feature evaluation unit 502 will be described, taking the so-called Shi and Tomasi method as an example.
First, the feature evaluation unit 502 uses the data filtered by the above-mentioned horizontal differential filter and the data filtered by the vertical differential filter to create an autocorrelation matrix H according to the following formula (1): In the formula, Ix represents the result of applying the horizontal differential filter, and Iy represents the result of applying the vertical differential filter, and formula (1) is a formula for convolving a Gaussian filter G to create the autocorrelation matrix H.

Then, the feature evaluation unit 502 calculates the feature value by the Shi and Tomasi feature evaluation formula shown in the following formula (2). Note that formula (2) indicates that the smaller eigenvalue of λ1 and λ2 of the autocorrelation matrix H in formula (1) is set as the feature value.

Shi and Tomasi = min(λ1, λ2) Equation (2)

The feature value calculated for each pixel in this manner by the feature evaluation unit 502 is sent to a feature point determination unit 503 .
The feature point determination unit 503 determines, for each grid, the pixel having the largest feature value among the feature values calculated by the feature evaluation unit 502 for each pixel in the grid as the feature point in that grid. In the case of this embodiment, the feature point information is expressed as relative coordinates (PX, PY) with the upper left corner of the grid being coordinates (0, 0). The feature point coordinates may be expressed as absolute coordinates in the frame image. The feature point information determined by the feature point determination unit 503 (hereinafter referred to as feature point coordinate information) is stored in an internal memory provided in the feature point extraction unit 202. This internal memory has a capacity sufficient to store the information of each feature point obtained from the previous frame and the feature point coordinate information obtained from the current frame.

When the matching processing unit 203 starts template matching processing of the corresponding grid, the feature point extraction unit 202 reads out feature point coordinate information 251 calculated in the previous frame from the internal memory and outputs it to the tracked feature point determination unit 205.
The tracked feature point determination unit 205 outputs the feature point coordinate information 251 sent from the feature point extraction unit 202 as tracked feature point coordinate information 257 to the second RDDMAC 222 included in the data transfer unit 105 .

Next, the feature point reflection process in the feature point reflection unit 207 will be described.
Here, the image captured by the first imaging optical section 101 is called a first image, and the image captured by the second imaging optical section 131 is called a second image.
The feature point reflecting unit 207 reflects the feature points extracted from the first image by the feature point extracting unit 202 onto the feature points of the second image in such a way that the parallax is taken into consideration. If the parallax is u, the distance to the subject is Z, the base line distance between the first imaging optical unit 101 and the second imaging optical unit 131 is d, and the focal length of these optical units is f, the parallax u can be calculated by the following formula (3).

u=(d×f)/Z Formula (3)

The feature point reflection unit 207 further adds the parallax u to any feature point coordinate (x, y) in the first image to obtain the feature point coordinate (x+u, y) in the second image with the parallax added. The current frame feature points that are the source of reflection are stored in the SRAM 206, and new feature points that have been subjected to the feature point reflection process described above are stored in the internal memory of the feature point reflection unit 207.

Then, when the matching processing unit 203 starts template matching processing of the corresponding grid, the feature point reflection unit 207 reads out from the internal memory the feature point coordinate information 259 that has been subjected to the feature point reflection processing, and outputs it to the tracked target feature point determination unit 205.
The tracked feature point determination unit 205 outputs the feature point coordinate information 259 sent from the feature point reflection unit 207 to the second RDDMAC 222 included in the data transfer unit 105 as tracked feature point coordinate information 257 in the second image.
A specific description of how new feature points are calculated for images captured by the stereo camera through the feature point reflection process described above will be given later.

Returning to the flowchart of FIG.
After the processes of steps S404 and S403, which are performed in parallel as described above, the motion vector detection unit 121 advances the process to step S405.
In step S405, the second RDDMAC 222 reads out data of a rectangular area of a preset size centered on the tracking target feature point coordinate information 257 calculated in the previous frame as described above. That is, data of the search area 305 in FIG. 3 is read out from the matching image data 242 generated in the current frame, while data of the template area 304 is read out from the matching image data 243 generated in the previous frame. Then, the read out search area image data 253 and template area image data 254 are sent to the matching processing unit 203. However, in the first template matching process, since the tracking target feature point coordinates cannot be calculated from the previous frame, the process is performed using all new feature point coordinates calculated in the current frame. After step S405, the process of the motion vector detection unit 121 proceeds to step S406.

When the process proceeds to step S406, the matching processing unit 203 calculates a correlation value using the input search area image data 253 and template area image data 254, and calculates a vector value based on the correlation value. After step S406, the process of the motion vector detection unit 121 proceeds to step S407.

In this embodiment, the correlation value is calculated using the sum of absolute differences (hereinafter abbreviated as SAD). Equation (4) is the calculation formula for SAD.

In formula (4), f(i,j) represents the pixel value of the coordinates (i,j) in the template area image data 254, and g(i,j) represents each pixel value in the target area for correlation value calculation (hereinafter referred to as the correlation value calculation area) in the search area image data 253. The correlation value calculation area is the same size as the area in the template area image data 254. The matching processing unit 203 calculates the absolute value of the difference between each pixel value f(i,j) and g(i,j) in these areas and calculates the correlation value S_SAD by calculating the sum of these values. Here, the smaller the correlation value S_SAD, the smaller the difference in luminance values between the two areas, that is, the more similar the textures are between the template area image data 254 and the correlation value calculation area. In this embodiment, SAD is used as an example of a correlation value, but this is not limited thereto, and other correlation values such as the sum of squared differences (SSD) and normalized cross correlation (NCC) may be used.

Then, the matching processing unit 203 finds the position of the minimum correlation value as the motion vector value of the grid. The matching processing unit 203 outputs the vector value and correlation value information 255 calculated as described above to the accuracy determination unit 204.

Next, proceeding to step S407, the accuracy determination unit 204 uses the correlation values from the vector values and correlation value information 255 calculated in the process of step S406 to determine the maximum, minimum, average, and minimum values of the correlation values. Then, using the maximum, minimum, average, and minimum values of these correlation values, the accuracy determination unit 204 performs processes of low contrast determination, maximum pixel value protrusion determination, and repetitive pattern determination, as described below. After step S407, the process of the motion vector detection unit 121 proceeds to step S408.

Figures 6(a) to 6(d) are graphs used to explain the relationship between the maximum, minimum, average, and minimum values of the correlation value and examples of judgments using them. However, in this embodiment, the smaller the correlation value, the higher the similarity, so in Figures 6(a) to 6(d), the maximum pixel value represents the minimum correlation value, the minimum pixel value represents the maximum correlation value, and the maximum pixel value represents the minimum correlation value.

FIG. 6A shows an example in which each determination result is good.
FIG. 6B shows an example of a low contrast determination. The low contrast determination is a process for determining whether the correlation value calculation region has low contrast. In the low contrast determination in this embodiment, when the difference between the maximum and minimum correlation values in the correlation value calculation region is smaller than a predetermined threshold value set in advance, the correlation value calculation region is determined to have low contrast. In the case of FIG. 6B, the difference between the maximum and minimum pixel values is smaller than that in FIG. 6A, and the difference between the maximum and minimum correlation values in the correlation value calculation region is also smaller than a predetermined threshold value. Therefore, in the case of FIG. 6B, the accuracy determination unit 204 determines that the correlation value calculation region has low contrast because the difference between the maximum and minimum correlation values in the correlation value calculation region is smaller than a predetermined threshold value.

Figure 6 (c) shows an example where a pixel value is determined to be a low peak in the maximum value protrusion judgment. The pixel value maximum value protrusion judgment is a process for judging how prominent the minimum value of the correlation value in the correlation value calculation area is. In the pixel value maximum value protrusion judgment in this embodiment, if the value of the division result of the difference between the maximum and average pixel values and the difference between the maximum and minimum pixel values is smaller than a predetermined threshold value set in advance, the correlation value calculation area is determined to be a low peak. On the other hand, if the value of the division result is larger than the predetermined threshold value, the correlation value calculation area is determined to be a high peak. In the case of Figure 6 (c), the value of the division result of the difference between the maximum and average pixel values and the difference between the maximum and minimum pixel values is smaller than the threshold value compared to Figure 6 (a). Therefore, in the case of Figure 6 (c), the accuracy judgment unit 204 judges that the correlation value calculation area is a low peak because the value of the division result is smaller than the predetermined threshold value.

Figure 6(d) shows an example where a repeating pattern is determined by the repeating pattern determination. Repeating pattern determination is a process for determining whether or not pixel values are fluctuating repeatedly. In the repeating pattern determination of this embodiment, a repeating pattern is determined when the difference between the maximum and maximum pixel values in the correlation value calculation area is smaller than a predetermined threshold value that is set in advance. In the case of Figure 6(d), the difference between the maximum and maximum pixel values is smaller than the predetermined threshold value compared to Figure 6(a). Therefore, in the case of Figure 6(d), the accuracy determination unit 204 determines that a repeating pattern exists because the difference between the maximum and maximum pixel values in the correlation value calculation area is smaller than the predetermined threshold value.

Return to the explanation of the flowchart in FIG. 4. After step S407, the motion vector detection unit 121 proceeds to the process of step S408. When proceeding to step S408, the precision determination unit 204 writes the above-mentioned low contrast, low peak, and repeat pattern determination information and vector information 256 to the SRAM 206. As a result, each piece of information is stored in the storage area 244 of the SRAM 206. Note that SRAM is an abbreviation for static random access memory (static RAM). Among semiconductor storage devices (semiconductor memories), SRAM is one of the volatile memories whose records are lost when the power is turned off, and has the characteristic that when data is written to the element, the contents are not lost while the power is on even if a refresh operation is not performed. After step S408, the motion vector detection unit 121 proceeds to the process of step S409.

In step S409, the tracked feature point determination unit 205 calculates tracked feature points to be used in the template matching process of the next frame.
7 is a flowchart showing the flow of processing in the tracked feature point determining unit 205. Below, an overview of the tracked feature point determination processing in the tracked feature point determining unit 205 will be described with reference to the flowchart in FIG.

First, in the process of step S701, the tracked feature point determination unit 205 acquires vector values and the aforementioned judgment information 258 from the SRAM 206. The tracked feature point determination unit 205 calculates the tracked feature point of the current frame using the acquired vector values, and further calculates the coordinates of the tracked feature point of the next frame by adding the vector value calculated in the template matching process centered on the coordinates of the tracked feature point. After step S701, the tracked feature point determination unit 205 proceeds to step S702.

In step S702, the tracked feature point determination unit 205 determines whether the tracked feature point calculated in step S701 is a valid tracked feature point that can be used for template matching processing of the next frame, based on the aforementioned determination information acquired from the SRAM 206. If the tracked feature point is determined to be low contrast in the low contrast determination, or a low peak in the maximum pixel value protrusion determination, or a repeating pattern in the repeating pattern determination, the tracked feature point determination unit 205 determines that the tracked feature point is not valid (NG determination). On the other hand, if the tracked feature point is not low contrast, low peak, or a repeating pattern, the tracked feature point determination unit 205 determines that the tracked feature point is valid (OK determination). If the tracked feature point determination unit 205 determines that the tracked feature point is NG (NO) in step S702, the process proceeds to step S704, whereas if the tracked feature point determination unit 205 determines that the tracked feature point is OK (YES), the process proceeds to step S703.

When the process proceeds to step S703, the tracked feature point determination unit 205 determines whether the tracked feature point coordinates calculated in step S701 are within the grid. Specifically, the tracked feature point determination unit 205 determines that the tracked feature point coordinates are OK when they are inside the feature point arrangement grid 302. In the case of the example in FIG. 3 described above, the tracked feature point determination unit 205 determines that the tracked feature point coordinates are OK when Xs≦x≦Xe and Ys≦y≦Ye are satisfied based on the start coordinates (Xs, Ys) and end coordinates (Xe, Ye) of the feature point arrangement grid 302 and the coordinates (x, y) of the feature point 303. On the other hand, the tracked feature point determination unit 205 determines that the tracked feature point coordinates are not OK when they are not inside the feature point arrangement grid 302. That is, the tracked feature point determination unit 205 determines that the result is NG when any one of x<Xs, Xe<x, y<Ys, and Ye<y is satisfied. If the tracked feature point determination unit 205 determines that the result is OK, the process proceeds to step S703, and if the tracked feature point determination unit 205 determines that the result is NG, the process proceeds to step S704.

When the process proceeds to step S704, the tracked feature point determination unit 205 deletes the tracked feature point coordinates corresponding to the coordinate position determined as NG in step S702 or step S703. The tracked feature point determination unit 205 then performs processing to replace the deleted tracked feature point coordinates with the new feature point coordinates calculated in step S404. After step S704, the tracked feature point determination unit 205 advances the process to step S705.

In step S705, the tracked feature point determination unit 205 determines whether or not all the tracked feature point coordinates have been calculated. Here, the number of feature points is set by the feature point number setting unit 208. If the tracked feature point determination unit 205 determines in step S705 that all the tracked feature point coordinates have not been calculated (NO), the process returns to step S701 and performs the above-mentioned process. The processes from step S701 to step S705 are repeated until it is determined in step S705 that all the tracked feature point coordinates have been calculated (YES). On the other hand, if the tracked feature point determination unit 205 determines in step S705 that all the tracked feature point coordinates have been calculated (YES), the process of the flowchart in FIG. 7 ends. This completes the process of each block of the motion vector detection unit 121 based on the process of each step in the flowchart in FIG. 4.

Next, the details of the calculation process of new feature points by the feature point reflection process performed by the feature point reflection unit 207 in step S404 of the flowchart shown in FIG. 4 will be described with reference to FIGS.
Fig. 8 is a diagram showing an example of distribution of motion vectors in an image when, for example, camera shake occurs in which the camera rotates in an axis direction (roll direction) about the midpoint between the first imaging optical unit 101 and the second imaging optical unit 131, among the camera movements caused by camera shake and the like. Note that camera shake generally includes components of camera movement in the yaw direction and pitch direction in addition to the roll direction, but Fig. 8 omits the motion vectors in the yaw direction and pitch direction. In the following, an explanation will be given by taking as an example only camera shake in the axis direction at the midpoint between the two imaging optical units.

As shown in Figure 8, when camera shake occurs in the rotational direction, each motion vector 802 of image data 801 is distributed in a point-symmetrical circular fashion. In the example of image data 801 in Figure 8, motion vector 802 points upward on the left side and downward on the right side. This distribution of rotational motion vectors can be considered to be similar for images captured from a stereo camera. In other words, in the case of images captured from a stereo camera, the motion vectors are considered to be distributed in a point-symmetrical circular fashion corresponding to the midpoint of the two imaging optical sections.

Figure 9 is a diagram showing an example of two image circles 901, 902 formed on the imaging surface of the image sensor 102 by two optical systems, the first imaging optical unit 101 and the second imaging optical unit 131, as in the stereo camera of this embodiment. The image sensor 102 captures images of the two image circles 901, 902 to obtain data of two parallax images having parallax in which the same subject 905 is captured. In the following description, the image corresponding to the image circle 902 on the left side is called the left eye image, and the image corresponding to the image circle 901 on the right side is called the right eye image.

Figure 10 shows an example of the distribution of feature points and motion vectors for image data 1001 consisting of a left eye image 1002 and a right eye image 1003 captured from a stereo camera when camera shake occurs in the rotational direction around an axis at the midpoint between two imaging optical units. Note that Figure 10 also shows a feature point arrangement grid and surrounding grid, as in Figure 3.

The motion vector detection unit 121 of this embodiment extracts feature points in all feature point arrangement grids of the left eye image 1002. On the other hand, the motion vector detection unit 121 extracts feature points from some of the feature point arrangement grids for the right eye image 1003, and reflects some of the feature points extracted from the left eye image 1002 in the feature points of the right eye image 1003. The motion vector detection unit 121 then detects motion vectors for each of the feature points of the left eye image 1002 and the right eye image 1003. That is, the motion vector detection unit 121 is used for detecting the motion vectors of the subject movement factors and camera shake factors for the left eye image 1002, and for detecting only the motion vectors due to feature points suitable for camera shake detection in the rotational direction for the right eye image 1003. In other words, in this embodiment, the motion vector detection unit 121 basically performs vector detection using the left eye image 1002, and uses the right eye image 1003 to assist in camera shake detection in the rotational direction. Details regarding which feature point arrangement grid is to be used to extract feature points from the right eye image 1003, and which feature points of the left eye image 1002 are to be reflected in the feature points of the right eye image 1003, will be described later.

If, for example, motion vector detection were performed using only the left eye image 1002, it would be impossible to detect rotational camera shake. This is because, in the case of FIG. 10, all of the motion vectors in the left eye image 1002 point upward, making it impossible to determine whether the camera shake is rotational or upward translational. In this case, if the motion vector in the right eye image points downward, for example, it can be determined that the camera shake is rotational, and if it points upward, it can be determined that the camera shake is upward translational. Therefore, if there is at least one motion vector in the right eye image 1003, it becomes possible to determine whether the camera shake is rotational or translational.

For example, increasing the number of extracted feature points can improve the accuracy of camera shake detection in the rotational direction, but in that case, the amount of processing required for motion vector detection increases, and the calculation costs rise. For this reason, in this embodiment, the number of feature points handled in the right eye image is limited (restricted in number), for example, to two. In this way, in this embodiment, by reducing the number of feature points handled in the right eye image 1003 compared to the left eye image 1002, it is possible to reduce the amount of processing required for motion vector detection and reduce calculation costs.

On the other hand, if the feature points of the right eye image 1003 are feature points derived from the subject, when the motion vector of the subject is larger than the motion vector of the camera shake, the motion vector of the camera shake cannot be confirmed, and camera shake detection in the rotational direction cannot be performed correctly. Therefore, the motion vector detection unit 121 of this embodiment reflects feature points extracted from a feature point arrangement grid in which no subject exists in the left eye image 1002 to the right eye image 1003. In other words, in this embodiment, of the feature points extracted from the left eye image 1002, feature points of the subject are not reflected in the feature points of the right eye image 1003.

Figure 11 is a diagram showing an example of the distribution of feature points in image data 1101 consisting of a left eye image 1102 and a right eye image 1103 captured by a stereo camera, and an example of a subject 1105. Note that, like Figure 3, Figure 11 also shows a feature point arrangement grid and a surrounding grid. Figure 11 shows an example of processing in which, for image data 1101 consisting of a left eye image 1102 and a right eye image 1103 in which a subject 1105 exists, the feature points of the left eye image 1102 are reflected in the image data of the right eye image 1103.

12 is a flowchart showing the flow of the feature point reflection process performed by the motion vector detection unit 121 of this embodiment. The feature point reflection process as shown in FIG. 11 will be described below with reference to the flowchart shown in FIG.
First, in step S1201, the motion vector detection unit 121 performs processing depending on whether the area for which feature points are to be newly calculated is a left-eye image or a right-eye image. If the area for which feature points are to be newly calculated is a left-eye image, the motion vector detection unit 121 proceeds to processing for extracting feature points from the left-eye image, whereas if the area for which feature points are to be newly calculated is a right-eye image, the motion vector detection unit 121 proceeds to processing for reflecting feature points from the left-eye image to the right-eye image.

When the area for calculating new feature points is the left eye image and the process proceeds to step S1202, the feature point extraction unit 202 extracts feature points from the left eye image. Specifically, as in step S704 described above, the feature point extraction unit 202 performs a process of replacing the feature point arrangement grid at the position where the deleted tracked feature point was first calculated as a new feature point with the new feature point coordinates of the current frame calculated at the corresponding position.

On the other hand, when the area for which new feature points are to be calculated is the right eye image and the process proceeds to step S1203, the feature point extraction unit 202 assigns a number Gn representing the priority order reflecting the feature points to the feature point arrangement grid of the right eye image. Here, in the case of this embodiment, feature points are not arranged in all feature point arrangement grids in the right eye image, but are limited to a portion. Also, in the case of this embodiment, the number of feature points in the right eye image is determined based on the processing speed for calculating the motion vector and the accuracy of camera shake detection in the rotational direction.

Here, if the minimum number of feature points that can detect camera shake in the rotational direction with sufficient accuracy is Nmin, and the maximum number of feature points that can calculate a motion vector at a processing speed sufficient to maintain real-time performance is Nmax, the number of feature points N must satisfy Nmin≦N≦Nmax. To reduce the calculation cost, N=Nmin may be set. In the example of FIG. 11, the number of feature points in the right eye image is described as two. For this reason, in step S1203, the feature point extraction unit 202 assigns a number Gn=1-Gmax to the feature point arrangement grid of the right eye image, which represents the priority of reflecting the feature points in the feature point reflection unit 207 described later. Note that the number Gn=1 indicates the highest priority, and Gn=Gmax indicates the lowest priority. In the example of FIG. 11, the feature point arrangement grid with the lowest priority is Gmax=9. In other words, it is considered that the greater the distance from the midpoint between the left eye image and the right eye image, the greater the motion vector value of the camera shake factor in the rotational direction. For this reason, in this embodiment, a higher priority is set for grids that are farther away from the midpoint between the left eye image and the right eye image. That is, in the example of FIG. 11, it is considered that the motion vector value of the cause of camera shake in the rotational direction will be a large value in the feature point arrangement grid on the right side of the figure in the right eye image. For this reason, in this embodiment, the priority is set to be higher for the feature point arrangement grid on the right side of the right eye image (Gn=1 in the example of FIG. 11). In this embodiment, the feature point extraction unit 202 first initializes the priority number Gn to 1, and processing is performed in order starting from the feature point arrangement grid with the highest priority.

Next, in step S1204, the feature point extraction unit 202 determines whether or not a feature point already exists in the grid of interest. That is, in the present embodiment, since a maximum of one feature point is arranged in one feature point arrangement grid, the feature point extraction unit 202 divides the process according to whether or not a feature point already exists in the feature point arrangement grid corresponding to the priority number Gn in the right eye image. If the feature point extraction unit 202 determines that a feature point already exists in the grid of interest, the process proceeds to step S1211, increments the number Gn (G=Gn+1), and then returns to step S1204. For example, in the case of the image data 1101 in FIG. 11, the feature point 1104 already exists in the feature point arrangement grid corresponding to Gn=3 in the right eye image 1103, so that the calculation process of the feature point is performed in a grid other than Gn=3. On the other hand, if the feature point extraction unit 202 determines that a feature point does not exist in the grid of interest, the process proceeds to step S1205.

In step S1205, the feature point extraction unit 202 determines whether or not a subject exists in the grid of interest. That is, the feature point extraction unit 202 divides the process according to whether or not a subject detected by the subject detection unit 122 in FIG. 1 exists in the feature point arrangement grid corresponding to the number Gn in the left eye image, which is the source of reflection of the feature points of the right eye image. If the feature point extraction unit 202 determines that a subject does not exist, the process proceeds to step S1210 performed by the feature point reflection unit 207. On the other hand, if it is determined that a subject exists, the feature point extraction unit 202 proceeds to the process after step S1206. That is, if a subject exists, the camera shake detection in the rotation direction is not performed correctly, so the feature point extraction unit 202 performs feature point calculation processing for the feature point arrangement grids of other numbers Gn. In the case of the image data 1101 in FIG. 11, the subject 1105 exists in the grids Gn=1 and Gn=2, so it is necessary to calculate feature points in grids other than Gn=1 and Gn=2.

When the process proceeds to step S1206, the feature point extraction unit 202 determines whether or not Gn=Gmax has been achieved. If the feature point extraction unit 202 determines that Gn=Gmax has been achieved, the process proceeds to step S1209, which is performed by the feature point reflection unit 207. On the other hand, if the feature point extraction unit 202 determines that Gn=Gmax has not been achieved, the process proceeds to step S1207.

When the process proceeds to step S1207, the feature point extraction unit 202 determines whether this step has been passed for the first time in this feature point extraction process. If the feature point extraction unit 202 has already passed this step, the process proceeds to step S1211, sets Gn=Gn+1, and returns to step S1204. On the other hand, if the feature point extraction unit 202 determines that this step has been passed for the first time, the process proceeds to step S1208.

When the process proceeds to step S1208, the feature point extraction unit 202 retains the value of number Gn, proceeds to step S1201, sets Gn = Gn + 1, and then returns to step S1204.

When it is determined in step S1206 that Gn=Gmax and the process proceeds to step S1209, the feature point reflection unit 207 performs processing to reflect feature points from the left eye image to the right eye image based on the value of number Gn held in step S1208. The processing in step S1209 is a procedure to be taken when an appropriate feature point to be reflected from the left eye image to the right eye image cannot be found, and although the feature point is within the feature point arrangement grid in which the subject is detected, the feature point is reflected using data with a high priority. After step S1209, the feature point reflection unit 207 proceeds to step S1210.

When proceeding to step S1210, the feature point reflection unit 207 reflects the feature points of the left eye image to the feature points of the right eye image based on the value of number Gn, taking into account the parallax between the left eye image and the right eye image. In the case of image data 1101 in FIG. 11, an example is shown in which feature points are reflected when Gn=4.

Then, as described above, the tracking target feature point determination unit 205 and the matching processing unit 203 perform processing to calculate a motion vector, which enables the motion vector detection unit 121 to detect camera shake in the rotational direction.

As described above, the image processing device according to the first embodiment detects the motion vector of the cause of the camera shake in the rotational direction by reflecting some of the feature points of the left eye image in the feature points of the right eye image, thereby making it possible to reduce the computational cost while still detecting the camera shake in the rotational direction.

In FIG. 11, the priority order numbers are assigned so that the right side of the right eye image has a higher priority, but the priority order can be freely determined. For example, in image data captured from a stereo camera, image distortion is smaller near the center of each of the left eye image and right eye image (a region within a specified range from the center coordinates or a rectangular region of a specified width and height). Therefore, the priority order may be set higher (preferentially set) near the center of each image. In addition, to prevent feature points from concentrating in a certain location and causing bias in the obtained motion vectors, the priority order may be set so that the feature points are evenly scattered throughout the entire image. This allows the feature points to be reflected so that the distance between each feature point in the right eye image is evenly distributed, for example.

Second Embodiment
Next, an image processing device according to a second embodiment of the present invention will be described. Note that the configuration of the image processing device according to the second embodiment is the same as that of Fig. 1 and Fig. 2 described above, and therefore illustration and description thereof will be omitted. The processing in the image processing device of the second embodiment is also generally similar to that described in Fig. 3 to Fig. 12, but whereas the number of feature points reflected from the left eye image to the right eye image is fixed in the first embodiment, the image processing device of the second embodiment dynamically controls the number of feature points over time.

Hereinafter, a feature point number setting process for dynamically controlling the number of feature points over time in the image processing apparatus of the second embodiment will be described with reference to FIGS.
FIG. 13 is a flowchart showing the flow of the feature point number setting process in the image processing apparatus of the second embodiment.

First, in step S1301, the rotational direction camera shake detection unit 209 detects camera shake in the rotational direction using the motion vector data stored in the SRAM 206, and sends information on the amount of camera shake in the rotational direction to the feature point number setting unit 208.

The feature point number setting unit 208 compares the amount of camera shake in the rotational direction detected in the current frame by the rotational direction camera shake detection unit 209 with the amount of camera shake in the rotational direction detected in the previous frame. The feature point number setting unit 208 then determines whether the amount of camera shake in the rotational direction has increased in the current frame compared to the previous frame. If the feature point number setting unit 208 determines that the amount of camera shake in the rotational direction has increased in the current frame compared to the previous frame, the process proceeds to step S1304 and subsequent steps, which are steps for reducing the number of feature points in order to reduce calculation costs. On the other hand, if the feature point number setting unit 208 determines that the amount of camera shake in the rotational direction has not increased in the current frame compared to the previous frame, the process proceeds to step S1302 and subsequent steps, which are steps for increasing the number of feature points in order to improve the accuracy of camera shake detection in the rotational direction.

When the process proceeds to step S1302, the feature point number setting unit 208 determines whether the number of feature points N is equal to the minimum number of feature points Nmin at which camera shake in the rotational direction can be detected. If N=Nmin, the feature point number setting unit 208 terminates the feature point number setting process according to the flowchart in FIG. 13 because the number of feature points N cannot be further reduced. On the other hand, if the feature point number setting unit 208 determines that the number of feature points N is not equal to the minimum number Nmin, the process proceeds to step S1303.

When the process proceeds to step S1303, the feature point number setting unit 208 reduces the feature point number N to the extent that the feature point number N does not become less than the minimum number of points Nmin (N<Nmin). After step S1303, the feature point number setting unit 208 ends the processing of the flowchart in FIG. 13.

When the process proceeds to step S1304, the feature point number setting unit 208 determines whether the number of feature points N is equal to the maximum number of feature points Nmax for which a motion vector can be calculated while maintaining real-time performance. If N=Nmax, the feature point number setting unit 208 ends the process of the flowchart in FIG. 13 because the number of feature points N cannot be increased further. On the other hand, if the feature point number setting unit 208 determines that the number of feature points N is not equal to the maximum number of points Nmax, the process proceeds to step S1305.

When the process proceeds to step S1305, the feature point number setting unit 208 increases the feature point number N within a range in which the feature point number N does not exceed the maximum number of points Nmax (N>Nmax). After step S1305, the feature point number setting unit 208 ends the processing of the flowchart in FIG. 13.

Here, how the maximum score Nmax is determined will be described with reference to FIGS.
Fig. 14 is a diagram showing an example of distribution of feature points for image data 1401 consisting of a left eye image 1402 and a right eye image 1403 captured by a stereo camera, similar to the example of Fig. 10 described above. Note that arrows representing motion vectors are not shown in Fig. 14. Fig. 14 also shows a feature point arrangement grid and surrounding grid, similar to Fig. 3. In the example of Fig. 14, nine feature points are arranged in the left eye image 1402, and two feature points are arranged in the right eye image 1403, for a total of 11 feature points.

Figure 15 is a timing chart relating to motion vector detection when there are 11 feature points as illustrated in Figure 14. As shown in Figure 15, in the motion vector detection unit 121, during one frame, a matching image is generated from the image data output from the image sensor 102, and template matching is performed sequentially according to the number of feature points N. Here, the time 1505 of one frame is RVD, the time 1506 required until template matching for the first feature point begins is T0, and the processing time 1507 required to perform one template matching operation calculated in advance is TT. In this case, these times must satisfy the relationship TDV>T0>T0+NxTT.

As shown in FIG. 15, when the number of feature points is 11, the processing time (T0 + N × TT) for detecting motion vectors for these 11 feature points falls within the time TDV of one frame shown as VD in FIG. 15. From this, it can be seen that when the number of feature points is 11, motion vectors can be calculated while maintaining real-time performance.

Fig. 16 is a diagram showing an example of the distribution of feature points for image data 1601 consisting of a left eye image 1602 and a right eye image 1603 captured by a stereo camera. In the example of Fig. 16, a total of 16 feature points are arranged, with nine feature points arranged in the left eye image 1602 and seven feature points arranged in the right eye image 1603. Note that Fig. 16 also shows a feature point arrangement grid and surrounding grid, etc., in the same way as Fig. 3.

17 is a timing chart for motion vector detection when there are 16 feature points as shown in FIG. 16. In FIG. 17, the time for one frame is time 1705, the time until template matching for the first feature point starts is time 1706, and the processing time required for executing one template matching is time 1707. In the motion vector detection unit 121, as described above, a matching image is generated from the image data output from the image sensor 102 during one frame, and template matching is performed sequentially according to the number of feature points N. When there are 16 feature points, the processing time (T0+N×TT) for detecting motion vectors for the 16 feature points does not fall within the time TDV for one frame shown by VD in FIG. 17, and therefore it is understood that the motion vector cannot be calculated while maintaining real-time performance. In other words, the number of feature points N of 16 points is not appropriate. And, from the example of FIG. 17, it can be said that the maximum number Nmax of feature points for which a motion vector can be calculated while maintaining real-time performance is 15 points.

As described above, the image processing device according to the second embodiment dynamically controls the number of feature points reflected from the left eye image to the right eye image over time. As a result, according to the second embodiment, it is possible to reduce the computational cost under optimal conditions based on the processing speed for calculating the motion vector and the accuracy of camera shake detection in the rotational direction.

As described above, in the image processing device of the first and second embodiments, the computational cost consumed for template matching can be reduced by limiting the number of feature points without limiting the search area. As a result, the image processing device of the first and second embodiments can inexpensively detect camera shake in the rotational direction for image data captured by the stereo camera, and can improve the performance of motion vector detection.

In the first and second embodiments, an example was given in which feature points were reflected from a left eye image to a right eye image, but feature points may also be reflected from a right eye image to a left eye image. Furthermore, in the first and second embodiments, the processing is not limited to the case of two images, a left eye image and a right eye image, and similar processing may be performed in the case of multiple images.

The present invention has been described in detail above based on preferred embodiments, but the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the invention are also included in the present invention. Parts of the above-mentioned embodiments may be combined as appropriate.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The above-described embodiments are merely examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted in a limiting manner based on these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main characteristics.

101: First imaging optical unit, 131: Second imaging optical unit, 102: Image sensor, 104: Image processing unit, 112: CPU, 121: Motion vector detection unit, 202: Feature point extraction unit, 207: Feature point reflection unit, 208: Feature point number setting unit, 209: Rotational direction shake detection unit

Claims (12)

An acquisition means for acquiring a plurality of images having parallax by a plurality of optical systems;
An extraction means for extracting feature points from the image;
A detection means for detecting a motion vector based on the feature points;
a reflecting means for reflecting the feature points extracted from a first image of the plurality of images on the feature points extracted from a second image of the plurality of images while taking into account the parallax;
a camera shake detection unit that detects camera shake in a rotational direction of an imaging device that captures the plurality of images having parallax due to the plurality of optical systems;
having
the detection means detects the motion vector for the second image based on feature points reflecting feature points of the first image;
The image processing device is characterized in that the camera shake detection means detects camera shake in the rotational direction of the imaging device using a motion vector detected based on the feature points of the first image and a motion vector detected based on feature points of the second image in which the feature points of the first image are reflected . 2. The image processing apparatus according to claim 1 , wherein said reflecting means limits the number of feature points of said second image that reflect the feature points of said first image. 3. The image processing apparatus according to claim 2 , wherein said reflecting means determines the number of said feature points to be limited based on a processing speed at which said detecting means detects said motion vector. 3. The image processing apparatus according to claim 2 , wherein said reflecting means determines the number of said feature points to be limited based on the accuracy of the camera shake detection in the rotational direction. 5. The image processing device according to claim 2, wherein the reflecting means reflects, among the feature points extracted from the first image, feature points suitable for detecting camera shake in the rotational direction of the imaging device, on the feature points of the second image. a subject detection means for detecting a subject from the image,
6. The image processing apparatus according to claim 1, wherein the camera shake detection means detects camera shake in the rotational direction based on a motion vector detected without using feature points of the subject. 6. The image processing device according to claim 1, wherein the reflection means sets a priority for each of the second image divided into a plurality of regions, and reflects the feature points extracted from the first image on the feature points of the regions of the second image based on the priority set for each region. 8. The image processing apparatus according to claim 7 , wherein the reflecting means sets a higher priority order for the area located farther away from the midpoint between the first image and the second image. 8. The image processing apparatus according to claim 7 , wherein the reflecting means reflects the feature points preferentially in an area within a predetermined range from the center of the second image. 10. The image processing device according to claim 1, wherein the detection means detects the motion vector by calculating a correlation value between pixel values within a rectangular area of the reference image centered on the feature point and a rectangular area of the original image, using one of two consecutive images on a time axis as an original image and the other as a reference image. An image processing method executed by an image processing device, comprising:
An acquisition step of acquiring a plurality of images having parallax by a plurality of optical systems;
An extraction step of extracting feature points from the image;
A detection means for detecting a motion vector based on the feature points;
an application step of applying the feature points extracted from a first image of the plurality of images to the feature points extracted from a second image of the plurality of images while taking into account the parallax;
a camera shake detection step of detecting a camera shake in a rotational direction of an imaging device that captures the plurality of images having parallax by the plurality of optical systems;
having
In the detection step, the motion vector is detected for the second image based on the feature points reflecting the feature points of the first image,
an image processing method characterized in that, in the camera shake detection process, camera shake in the rotational direction of the imaging device is detected using a motion vector detected based on the feature points of the first image and a motion vector detected based on feature points in which the feature points of the first image are reflected in the feature points of the second image . A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 10 .

Images