JP6016546B2 - Imaging device, control method thereof, and control program - Google Patents

Description

  The present invention relates to an imaging apparatus, a control method thereof, and a control program, and in particular, control for changing the size (size) of an image obtained as a result of imaging according to the imaging state of the imaging apparatus and the state of a subject. About.

  Generally, when a moving image is shot with an imaging device such as a digital camera or a digital video camera, the position of a stationary subject may move in the next frame due to camera shake. Moving a stationary subject caused by such camera shake makes it difficult to view a moving image after shooting.

  For this reason, the motion vector of the next frame image is detected for one frame image, and camera shake correction is performed so that the position of the stationary subject does not move. On the other hand, when the amount of camera shake increases when detecting a motion vector, the vector amount of the motion vector exceeds the predetermined search range, and the motion vector cannot be accurately detected for each region (block) in the image. Sometimes.

  In order to solve such a problem, that is, in order to substantially widen the search range, the number of pixels in the search range is reduced by thinning out at a ratio of 1 pixel to 2 pixels in the horizontal direction or the vertical direction in the image. The search range is substantially expanded (see Patent Document 1).

  Further, a search image is generated with a plurality of resizing magnifications, a motion vector detection process is performed on each search image, and the search range is expanded by combining the results (see Patent Document 2). ).

JP 2007-194773 A JP 2008-287648 A

  However, the technique described in Patent Document 1 has a problem that the detection accuracy is lowered depending on the shooting situation as a result of the image being reduced to a predetermined size. Furthermore, in the method described in Patent Document 1, if the size of an image obtained as a result of shooting increases, it cannot be said that the image is sufficiently reduced even if thinning processing is performed, and motion vector detection cannot be performed accurately. There is a problem.

  In the method described in Patent Document 2, a plurality of resized images are generated, and a motion vector detection process is required for each of these images. For this reason, there is a problem that the time required for processing and the amount of resources used at that time increase.

  Accordingly, an object of the present invention is to provide an imaging apparatus, a control method thereof, and a control program capable of always detecting a motion vector of a subject accurately and correcting camera shake without increasing processing time. .

To achieve the above object, an imaging apparatus according to the present invention is an imaging apparatus for determining a motion vector of the object from the image to obtain an image corresponding to an optical image by imaging an object, a front Kiga image Based on the resizing image, a resizing unit that determines a resizing factor to be resized, a resizing unit that resizes the image according to the resizing factor determined by the scaling factor determining unit, and obtains a resized image. have a motion vector detecting means for determining the motion vector, the magnification determining means, when determining the resizing ratio in accordance with a feature quantity indicating the feature of the object, the feature of the feature amount is a predetermined Until the feature amount is increased, the reduction ratio as the resizing magnification is reduced as the feature amount increases, and the feature amount exceeds the predetermined feature amount. Characterized in that by increasing the reduction rate with increasing of the feature quantity.

Control method according to the present invention is a control method of an imaging device to obtain an image corresponding to the optical image by imaging a subject obtaining the motion vector of the object from the image, the resize ratio to resize the front Kiga image A scaling factor determination step, a resizing step for resizing the image in accordance with the resizing magnification determined in the scaling factor determination step to obtain a resized image, and obtaining the motion vector based on the resized image. have a motion vector detection step, and in the magnification determining step, the resizing ratio determined in accordance with the feature quantity indicating the feature of the object, until an increase in feature quantity the feature amount is a predetermined said feature As the amount increases, the reduction ratio that is the resizing magnification is reduced, and when the feature amount exceeds the predetermined feature amount, Characterized in that to increase the reduction rate with increasing symptoms amount.

A control program according to the present invention is a control program used in an imaging apparatus that captures an image of a subject, obtains an image corresponding to an optical image, and obtains a motion vector of the object from the image. a magnification determining step of determining a resize ratio to resize the front Kiga image, a resized to obtain the resizing processed image by resizing the image in accordance with the resizing factor determined by the magnification determining step, the resizing process A motion vector detection step for obtaining the motion vector based on a completed image, and in the magnification determination step, the resizing magnification is determined according to a feature amount indicating a feature of the subject, and the feature amount is determined in advance. The reduction ratio that is the resizing magnification as the feature amount increases until the feature amount increases. Small, and the feature amount is equal to or to increase the reduction rate with increasing the said characteristic quantity with a predetermined excess of feature amount.

  According to the present invention, it is possible to always detect a motion vector of a subject accurately and perform camera shake correction without increasing processing time.

It is a block diagram which shows the structure about an example of the imaging device by embodiment of this invention. It is a figure which shows an example of a structure of the resizing control part shown in FIG. 2A and 2B are diagrams for explaining processing in the resizing control unit illustrated in FIG. 1, in which FIG. 1A illustrates a characteristic of a first reduction ratio generation unit, and FIG. 2B illustrates a characteristic of a second reduction ratio generation unit; (C) is a diagram showing the characteristics of the third reduction rate generation unit, (d) is a diagram showing the characteristics of the fourth reduction rate generation unit, and (e) is a diagram showing the characteristics of the fifth reduction rate generation unit. FIG. 2A and 2B are diagrams for explaining motion vector detection processing in the motion vector generation unit illustrated in FIG. 1, in which FIG. 1A illustrates a configuration of the motion vector generation unit, and FIG. 2B illustrates a camera read from the frame memory illustrated in FIG. 1. The figure which shows a signal, (c) is a figure for demonstrating the camera signal read from a buffer memory.

  Hereinafter, an example of an imaging apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of an example of an imaging apparatus according to an embodiment of the present invention.

  The illustrated imaging apparatus (hereinafter simply referred to as a camera) includes an optical system 1, and the optical system 1 includes, for example, a zoom lens, a focus lens, and a diaphragm (both not shown). The optical system 1 is controlled by the microcomputer 8 and changes the positions of the zoom lens and the focus lens according to the focal length information and the focus position information output from the microcomputer 8, for example.

  The optical image that has passed through the optical system 1 is formed on the image sensor 2. Then, the image sensor 2 outputs an electrical signal (analog image signal) corresponding to the optical image. At this time, the microcomputer 8 controls the exposure time of the image sensor 2 in accordance with the shutter speed information (shutter speed).

  The analog image signal is given to the analog front end unit (AFE) 3. The AFE 3 has a CDS (Correlated Double Sampling) circuit and an A / D converter. The AFE 3 converts an analog image signal into a digital image signal (referred to as a camera signal).

  The camera signal processing unit 4 performs predetermined signal processing on the camera signal and outputs it as image data. Here, the predetermined signal processing is, for example, processing such as aperture correction, gamma correction, and white balance processing. Further, the camera signal processing unit 4 can receive the motion vector value from the motion vector generation unit 13 and cut out an image so as to cancel the camera motion indicated by the motion vector value, thereby performing camera shake correction.

  The camera includes a camera shake detection unit 5, and the camera shake detection unit 5 detects the movement (blur) of the camera. Then, the camera shake detection unit 5 outputs a camera shake detection signal to the resizing control unit 9. Note that the camera shake detection unit 5 includes, for example, a gyro sensor, and detects a shake by the gyro sensor.

  The AF evaluation value generation unit 6 receives the camera signal and generates an AF evaluation value according to the camera signal. Here, the AF evaluation value generation unit 6 extracts a contrast component in the camera signal and outputs it as an AF evaluation value.

  The feature extraction unit 7 receives the camera signal and extracts the feature amount of the subject image from the image indicated by the camera signal. Here, the feature extraction unit 7 performs high-pass filter processing on the camera signal to obtain a flat portion and a repetitive pattern that are likely to be erroneously determined in the motion vector detection processing in the camera signal, and calculates the absolute value sum as the feature amount. Output as data.

  The microcomputer 8 controls the entire camera. Here, the microcomputer 8 outputs focal distance information and lens drive information indicating the focus position to the optical system 1 and outputs shutter speed information to the image sensor 2. The microcomputer 8 outputs focal length information and shutter speed information to the resizing control unit 9 and outputs access control information of the frame memory 10 to the memory control unit 11.

  The resize control unit 9 is provided with a camera shake detection signal from the camera shake detection unit 5 and an AF evaluation value from the AF evaluation value generation unit 6. Further, the feature size data indicating the feature amount of the subject is supplied from the feature extraction unit 7 to the resizing control unit 9. As will be described later, the resizing control unit 9 determines an image reduction ratio (resizing magnification) according to a camera shake detection signal, an AF evaluation value, and feature amount data. At this time, the resizing control unit 9 calculates the amount of blurring that appears in the image according to the camera shake detection signal (for example, angular velocity signal) given from the camera shake detection unit 5 and the focal length.

  The resizing unit 10 resizes the camera signal under the control of the resizing control unit 9. The frame memory 11 temporarily stores a resized camera signal (resized camera signal) under the control of the memory control unit 12. The camera signal stored in the frame memory 11 is read by the memory control unit 12 and given to the motion vector generation unit 13.

  The motion vector generation unit 13 generates a motion vector according to the camera signal stored in the frame memory 11 and the resized camera signal that is the output of the resizing unit 10.

  FIG. 2 is a diagram for explaining an example of the configuration of the resizing control unit 9 shown in FIG.

  The resizing control unit 9 includes first to fifth reduction rate generation units 91 to 95 and a reduction rate determination unit 96. A camera shake detection signal is input from the camera shake detection unit 5 to the first reduction ratio generation unit 91. Then, the first reduction rate generation unit 91 obtains the first reduction rate S91 according to the camera shake detection signal.

  An AF evaluation value is input from the AF evaluation value generation unit 6 to the second reduction ratio generation unit 92. Then, the second reduction rate generation unit 92 obtains a second reduction rate S92 according to the AF evaluation value. The feature amount data is input from the feature extraction unit 7 to the third reduction rate generation unit 93. Then, the third reduction rate generation unit 93 obtains a third reduction rate S93 according to the feature amount data.

  Focal length information that is a control signal output from the microcomputer 8 is input to the fourth reduction ratio generation unit 94. Then, the fourth reduction rate generation unit 94 obtains a fourth reduction rate S94 according to the focal length information. The fifth reduction ratio generation unit 95 receives shutter speed information that is a control signal output from the microcomputer 8. The fifth reduction rate generation unit 95 obtains a fifth reduction rate S95 according to the shutter speed information.

  FIG. 3 is a diagram for explaining processing in the resize control unit 9 shown in FIG. FIG. 3A is a diagram illustrating the characteristics of the first reduction rate generation unit 91, and FIG. 3B is a diagram illustrating the characteristics of the second reduction rate generation unit 92. FIG. 3C is a diagram illustrating the characteristics of the third reduction rate generation unit 93, and FIG. 3D is a diagram illustrating the characteristics of the fourth reduction rate generation unit 94. Further, FIG. 3E is a diagram showing the characteristics of the fifth reduction rate generation unit 95.

  In FIG. 3A, the horizontal axis indicates the amount of camera movement indicated by the camera shake detection signal, and the vertical axis indicates the reduction ratio. When the amount of camera motion indicated by the camera shake detection signal increases, it is necessary to perform motion vector detection processing on the entire image, so the first reduction rate generation unit 91 increases the first reduction rate S91.

  On the other hand, when the amount of camera motion indicated by the camera shake detection signal is small, it is necessary to perform motion vector detection processing for fine motion, and therefore the first reduction rate generation unit 91 reduces the first reduction rate S91. .

  Note that the characteristics shown in FIG. 3A are examples, and the first reduction rate generation unit 91 only needs to have a characteristic of increasing the first reduction rate S91 as the amount of camera movement increases. .

  In FIG. 3B, the horizontal axis indicates the AF evaluation value, and the vertical axis indicates the reduction rate. The smaller the AF evaluation value, the closer to the out-of-focus state, and the lower the contrast of the entire image. For this reason, the second reduction ratio generation unit 92 increases the second reduction ratio S92 to increase the contrast so that the feature amount in the motion vector detection process is easily generated.

  On the other hand, the larger the AF evaluation value, the closer to the in-focus state, and the higher the contrast of the entire image. For this reason, the second reduction rate generation unit 92 lowers the second reduction rate S92 to improve the accuracy of motion vector detection for fine motion.

  The characteristic shown in FIG. 3B is an example, and the second reduction ratio generation unit 92 only needs to have a characteristic of increasing the second reduction ratio S92 as the AF evaluation value decreases.

  In FIG. 3C, the horizontal axis indicates the feature amount data, and the vertical axis indicates the reduction rate. When the feature amount is small, the contrast of the entire image is low. For this reason, the third reduction rate generation unit 93 increases the third reduction rate S93 to increase the contrast so that the feature amount in the motion vector detection process is easily generated.

  When the feature amount is large, the contrast of the entire image is high and contains a lot of high frequency components. Therefore, there is a high possibility that an erroneous determination occurs in the motion vector detection process. For this reason, the third reduction rate generation unit 93 increases the third reduction rate S93 to reduce high frequency components and improve the accuracy of the motion vector detection process.

  When the feature amount is located in the middle, the contrast of the image is suitable for the motion vector detection process. Therefore, the third reduction rate generation unit 93 decreases the third reduction rate S93 so as to improve the accuracy of the motion vector detection process for fine motion.

  In this way, the third reduction rate generation unit 93 reduces the reduction rate as the feature amount increases until the feature amount increases to the predetermined feature amount, and when the feature amount exceeds the predetermined feature amount. , It has the characteristic of increasing the reduction ratio as the feature amount increases.

  Note that the characteristic shown in FIG. 3C is an example, and the characteristic of the third reduction ratio generation unit 93 may draw a downwardly convex parabola according to the feature amount data.

  In FIG. 3D, the horizontal axis represents the focal length, and the vertical axis represents the reduction ratio. The closer the focal length is to the tele side, the higher the possibility that the image will move due to camera shake. For this reason, since it is necessary to perform motion vector detection processing on the entire image, the fourth reduction ratio generation unit 94 increases the fourth reduction ratio S94.

  On the other hand, the wider the focal length, the lower the possibility that the image will move due to camera shake. For this reason, since it is necessary to perform motion vector detection processing for fine motion, the fourth reduction rate generation unit 94 decreases the fourth reduction rate S94.

  The characteristic shown in FIG. 3D is an example, and the fourth reduction ratio generation unit 94 has a characteristic that increases the fourth reduction ratio S94 as the focal distance goes to the tele side. Just do it.

  In FIG. 3E, the horizontal axis indicates the shutter speed, and the vertical axis indicates the reduction rate. The slower the shutter speed, the higher the possibility that motion blur will occur in the subject image and the lower the contrast of the image. For this reason, the fifth reduction ratio generation unit 95 increases the fifth reduction ratio S95 to increase the contrast so that the feature amount in the motion vector detection process is easily generated.

  On the other hand, the higher the shutter speed, the lower the possibility of motion blur in the subject image and the higher the image contrast. For this reason, the fifth reduction rate generation unit 95 reduces the fifth reduction rate S95 to maintain the contrast and improve the accuracy of the motion vector detection processing for fine motion.

  The characteristic shown in FIG. 3E is an example, and the fifth reduction ratio generation unit 95 has a characteristic that increases the fourth reduction ratio S94 as the shutter speed decreases. Good.

  The reduction ratio determination unit 96 inputs the first to fifth reduction ratios S91 to S95. Then, the reduction rate determination unit 96 determines the reduction rate S96 using the resizing unit 10 from the first to fifth reduction rates S91 to S95. In the illustrated example, the reduction rate determination unit 96 selects the maximum value from the first to fifth reduction rates S91 to S95 and outputs it as the reduction rate S96.

  Note that the reduction rate determination unit 96 may not select the maximum value, and may output, for example, an average value of the first to fifth reduction rates S91 to S95 as the reduction rate S96.

  The resizing unit 10 performs resizing processing of the camera signal using a bicubic method or the like according to the reduction rate S96, and outputs the camera signal after the resizing processing to the frame memory 11.

  The motion vector generation unit 13 performs a motion vector detection process on the image in accordance with the resized camera signal output from the resizing unit 10 and the camera signal read from the frame memory 11, and obtains the motion vector value. Output to the camera signal processing unit 4.

  FIG. 4 is a diagram for explaining motion vector detection processing in the motion vector generation unit 13 shown in FIG. FIG. 4A is a diagram showing a configuration of the motion vector generation unit 13, and FIG. 4B is a diagram showing a camera signal read from the frame memory 11 shown in FIG. FIG. 4C is a diagram for explaining a camera signal read from the buffer memory.

  The motion vector generation unit 13 includes a buffer memory 131 and a representative point matching unit 132. The frame memory 11 has a capacity to hold at least two representative point frames. The frame memory 11 holds a representative point data image obtained by resampling the resized camera signal as an input signal in the horizontal and vertical directions with a predetermined pixel interval. The representative point data image as an input signal is also held in the buffer memory 131.

  When the representative point data image is read from the frame memory 11, the rectangular area Ys is represented as a representative point pixel group Ys for each area of a predetermined size, as indicated by the rectangular area Ys in FIG. Read sequentially from upper left to lower right.

  On the other hand, when the representative point data image is read from the buffer memory 131, the image is divided by a predetermined size as shown by a rectangular area in FIG. Yt is sequentially read from the partition numbers B1 to B12.

  First, the representative point matching unit 132 calculates the difference between the representative point pixel group Yt and the representative point pixel group Ys with respect to the partition number B1, and obtains the readout position of the representative point pixel group Ys having the smallest absolute value. Then, the representative point matching unit 132 outputs the motion vector Id of the partition number B1 according to the readout position of the representative point pixel group Ys having the smallest absolute value and the position of the partition number B1 of the representative point pixel group Yt.

  When the matching operation for the partition number B1 is completed, the representative point matching unit 132 reads the representative point pixel group Yt for the partition number B2 from the buffer memory 131, and the representative point pixel group Ys (the frame memory 11 of the frame memory 11) read from the frame memory 11. A difference from the representative point pixel group Ys) sequentially read from the upper left to the lower right is obtained. Then, similarly to the case of the block number B1, the representative point matching unit 132 outputs the motion vector Id of the block number B2.

  Similarly, the motion vector generation unit 13 outputs the motion vector Id for the section numbers B3 to B12. As a result, the motion vector generation unit 13 outputs all motion vectors from the partition numbers B1 to B12 as motion vector values. Then, the camera signal processing unit 4 selects or calculates a representative motion vector value of the image from the obtained 12 motion vector values, and determines the cutout position of the image so as to cancel the camera motion. .

  As described above, in the embodiment of the present invention, since the size of the image used in the motion vector detection process is changed according to the shooting state of the camera, the motion vector detection process suitable for the shooting state of the camera is performed. be able to.

  As is apparent from the above description, in the example shown in FIG. 1, the microcomputer 8 and the resizing control unit 9 function as a magnification determining unit, and the resizing unit 10 functions as a resizing unit. The motion vector generation unit 13, the frame memory 11, the microcomputer 8, and the memory control unit 12 function as motion vector detection means.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the imaging apparatus. Further, a program having the functions of the above-described embodiments may be used as a control program, and the control program may be executed by a computer included in the imaging apparatus. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a magnification determination step, a resizing step, and a motion vector detection step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

4 Camera signal processing unit 5 Camera shake detection unit 6 AF evaluation value generation unit 7 Feature extraction unit 8 Microcomputer 9 Resize control unit 10 Resize unit 11 Frame memory 12 Memory control unit 13 Motion vector generation unit

Claims (7)

An imaging device that captures an image of a subject, obtains an image corresponding to an optical image, and obtains a motion vector of the subject from the image,
A magnification determining means for determining a resize ratio to resize the front Kiga image,
Resizing means for resizing the image according to the resizing magnification determined by the magnification determining means to obtain a resized image;
Have a motion vector detecting means for determining the motion vector based on the resized processed image,
In the case where the resizing magnification is determined according to the feature amount indicating the feature of the subject, the resizing magnification is increased as the feature amount increases until the feature amount increases to a predetermined feature amount. An image pickup apparatus , wherein the reduction rate is reduced, and when the feature amount exceeds the predetermined feature amount, the reduction rate is increased as the feature amount increases . It said magnification determining means, in addition to the feature quantity, further, with a focal length of the imaging device, according to claim 1, wherein the focal length is equal to or increasing the reduction rate is the resizing ratio as large The imaging device described in 1. It said magnification determining means, in addition to the feature quantity, further, the use of shutter speed of the imaging device, according to claim 1, wherein the shutter speed to increase the reduction ratio is the resizing ratio as slower The imaging device described in 1. The magnification determining means uses an AF evaluation value indicating the contrast of the image in addition to the feature amount, and increases the reduction ratio as the resizing magnification as the AF evaluation value decreases. Item 2. The imaging device according to Item 1 . The magnification determination unit uses a camera shake amount indicating movement of the imaging apparatus in addition to the feature amount, and increases the reduction ratio as the resizing magnification as the camera shake amount increases. The imaging apparatus according to 1 . A method for controlling an imaging apparatus that captures an image of a subject to obtain an image corresponding to an optical image and obtains a motion vector of the subject from the image,
A magnification determining step of determining a resize ratio to resize the front Kiga image,
A resizing step of resizing the image according to the resizing magnification determined in the magnification determining step to obtain a resized image;
Have a motion vector detection step of obtaining the motion vector based on the resized processed image,
In the magnification determination step, the resizing magnification is determined according to a feature amount indicating the feature of the subject, and the resizing magnification is increased as the feature amount increases until the feature amount increases to a predetermined feature amount. A control method characterized by reducing a reduction rate and increasing the reduction rate as the feature amount increases when the feature amount exceeds the predetermined feature amount . A control program used in an imaging device that captures an image of a subject to obtain an image corresponding to an optical image and obtains a motion vector of the subject from the image,
In the computer provided in the imaging device,
A magnification determining step of determining a resize ratio to resize the front Kiga image,
A resizing step of resizing the image according to the resizing magnification determined in the magnification determining step to obtain a resized image;
Performing a motion vector detection step for obtaining the motion vector based on the resized image ,
In the magnification determination step, the resizing magnification is determined according to a feature amount indicating the feature of the subject, and the resizing magnification is increased as the feature amount increases until the feature amount increases to a predetermined feature amount. A control program characterized by reducing a reduction rate and increasing the reduction rate as the feature amount increases when the feature amount exceeds the predetermined feature amount .

Images