JP7324066B2 - IMAGE PROCESSING APPARATUS, CONTROL METHOD THEREOF, AND IMAGING APPARATUS - Google Patents

Description

The present invention relates to an image processing device, its control method, and an imaging device, and more particularly to a moving object detection technique.

A technique for detecting a subject moving in a direction opposite to the general moving direction of the subject based on a motion vector between frame images of a moving image is known (Patent Document 1). Also, there is known a technique of determining an area where there is a large difference between the movement of the entire screen and the local movement as the area of the main subject (Patent Document 2).

JP 2015-194915 A JP 2015-111746 A

The technique described in Japanese Patent Laid-Open No. 2002-200002 detects a moving object moving in the opposite direction to a large number of moving objects based on a motion vector whose angular difference is equal to or greater than a predetermined value with respect to the motion vector representing the moving direction of the large number of moving objects. Detecting. Therefore, when shooting a scene in which the background and the subject (moving object) move in the same direction at different speeds, such as when shooting a moving object while panning, the method described in Patent Document 1 can be used to It cannot be separated from the subject.

On the other hand, in Patent Document 2, since the main subject is determined based on the difference between the motion of the entire image and the difference between the local motion, the background and the main subject are moving in the same direction at different speeds. However, the main subject can be identified. However, the main subject intended by the user is not necessarily a large subject that has a large difference in motion from the motion of the entire screen.

SUMMARY OF THE INVENTION The present invention has been made in view of such problems in the prior art, and an object of the present invention is to provide an image processing apparatus capable of detecting a moving object moving in the same direction as the background, a control method thereof, and an imaging apparatus. Let it be one.

The above objects are provided by motion vector detection means for detecting a plurality of motion vectors between a plurality of images, calculation means for calculating a background vector representing the motion of the background based on the motion vectors, and each of the plurality of motion vectors. A moving object detecting means for detecting a motion vector of a moving object from a plurality of motion vectors based on the magnitude of the Euclidean distance to a background vector; a clustering means for clustering the plurality of motion vectors to generate clusters; and a clustering means. selecting means for selecting a background cluster from the obtained clusters; motion detecting means for detecting motion of the image processing apparatus ; estimates the motion direction of the background from the motion of the image processing device detected by the motion detection means, and selects the background cluster from the clusters generated by the clustering means using the motion direction of the background. achieved by the device.

ADVANTAGE OF THE INVENTION According to this invention, the image processing apparatus which can detect the moving body which moves in the same direction as a background, its control method, and an imaging device can be provided.

1 is a block diagram showing a functional configuration example of a digital camera according to an embodiment of the invention; FIG. FIG. 2 is a block diagram showing a functional configuration example related to moving object detection of an image processing unit according to the embodiment; Flowchart relating to moving object detection processing according to the embodiment Schematic diagram for explaining the moving object detection method in the embodiment Flowchart for an example of background cluster selection operation in an embodiment 4 is a flow chart of another example of background cluster selection operation in an embodiment Schematic diagram for explaining moving object detection using a distance map in the embodiment FIG. 11 is a block diagram showing a functional configuration example related to moving object detection of an image processing unit according to the second embodiment; Flowchart regarding the operation of the image processing unit in the second embodiment Diagram relating to clustering of motion vectors in the second embodiment Flowchart relating to main subject determination processing in the second embodiment

The invention will now be described in detail on the basis of its exemplary embodiments with reference to the accompanying drawings. It should be noted that the described embodiments are merely examples and do not limit the scope of the present invention. For example, an embodiment in which the present invention is applied to a digital camera will be described below. However, a digital camera is only one example of an image processing device to which the present invention can be applied. The present invention can be implemented in any electronic device. Such electronic devices include, but are not limited to, imaging devices such as digital cameras and digital video cameras, as well as personal computers, tablet terminals, mobile phones, game machines, drive recorders, robots, and drones. Note that the present invention does not require a photographing function, and can be implemented by an electronic device capable of acquiring images photographed in chronological order, such as moving images.

(Configuration of imaging device)
FIG. 1 is a block diagram showing an example functional configuration of a digital camera 100 according to the first embodiment. The digital camera 100 can shoot and record moving and still images. Each functional block within the digital camera 100 is communicably connected to each other via a bus 160 . The operation of the digital camera 100 is realized by one or more programmable processors of the main control unit 151 reading, for example, a program stored in the ROM 155 into the RAM 154 and executing the program to control each functional block.

The taking lens 101 (lens unit) includes a fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, a focus lens 131, a zoom motor (ZM) 112, an aperture motor (AM) 104, and a focus motor ( FM) 132. A fixed 1st group lens 102, a zoom lens 111, an aperture 103, a fixed 3rd group lens 121, and a focus lens 131 constitute a photographing optical system. Although the lenses 102, 111, 121, and 131 are illustrated as one lens for convenience, they may each be composed of a plurality of lenses. Also, the photographing lens 101 may be configured as an interchangeable lens that can be removed from the imaging device 100 .

The aperture control unit 105 controls the operation of the aperture motor 104 according to the command from the main control unit 151 to change the aperture diameter of the aperture 103 .
The zoom control unit 113 controls the operation of the zoom motor 112 according to commands from the main control unit 151 to change the focal length (angle of view) of the photographing lens 101 .

The focus control unit 133 calculates the defocus amount and the defocus direction of the photographing lens 101 based on the phase difference between the pair of focus detection signals obtained from the image sensor 141, for example. The focus control unit 133 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 132 . The focus control unit 133 controls the operation of the focus motor 132 based on the drive amount and the drive direction, and drives the focus lens 131 to control the focus state of the photographing lens 101 .

In this manner, the focus control unit 133 performs automatic focus detection (AF) using the phase difference detection method. may be executed. Alternatively, phase difference detection AF may be performed using a focus detection signal obtained from an AF sensor provided separately from the imaging device 141 . In the AF operation of the focus control section 133, the focus detection area can be set to the area of the main subject detected by the image processing section 152, which will be described later.

A subject image formed on the imaging plane of the imaging element 141 by the imaging lens 101 is converted into an electric signal (image signal) by a photoelectric conversion element of each of a plurality of pixels arranged on the imaging element 141 . In this embodiment, m pixels in the horizontal direction and n pixels in the vertical direction (where n and m are plural) are arranged in a matrix in the image sensor 141, and each pixel has two photoelectric conversion elements (photoelectric conversion regions). ) is provided. Signal reading from the imaging device 141 is controlled by the sensor control section 143 according to instructions from the main control section 151 .

The two photoelectric conversion regions of each pixel are called region A and region B. An image composed of a group of image signals read out from region A of each pixel is called A image, and a group of image signals read out from region B of each pixel. is called a B-image. An image obtained by adding the A image and the B image pixel by pixel is called an A+B image. The A and B images form a parallax image pair. A+B images are used for display and recording. Further, the A image and the B image are used to generate a focus detection signal used in phase difference detection AF and to generate a distance map.

An image signal read out from the imaging element 141 is supplied to the signal processing section 142 . The signal processing unit 142 applies signal processing such as noise reduction processing, A/D conversion processing, and automatic gain control processing to the image signal, and outputs the image signal to the sensor control unit 143 as image data. The sensor control unit 143 stores the image data received from the signal processing unit 142 in a RAM (random access memory) 154 .

When recording the image data stored in the RAM 154, the main control unit 151 adds, for example, a predetermined header to the image data to generate a data file according to the recording format. At this time, the main control unit 151 encodes the image data in the compression/decompression unit 153 as necessary, and stores the encoded data in the data file. The main controller 151 records the generated data file in a recording medium 157 such as a memory card.

When displaying the image data stored in the RAM 154, the main control unit 151 scales the image data in the image processing unit 152 so as to fit the display size of the display unit 150, and then uses the RAM 154 as a video memory. Write to the area (VRAM area). The display unit 150 reads image data for display from the VRAM area of the RAM 154, and displays it on a display device such as an LCD or an organic EL display. The display unit 150 also displays the detection result of the main subject (moving object) detected by the image processing unit 152 (the frame indicating the main subject area, etc.).

The digital camera 100 causes the display unit 150 to function as an electronic viewfinder (EVF) by immediately displaying the captured moving image on the display unit 150 during moving image capturing (shooting standby state or moving image recording). A moving image and its frame images displayed when the display unit 150 functions as an EVF are called a live view image or a through image. Further, when a still image is captured, the digital camera 100 displays the last captured still image on the display unit 150 for a certain period of time so that the user can confirm the result of the capturing. These display operations are also realized under the control of the main control unit 151 .

A compression/decompression unit 153 encodes and decodes image data. For example, when recording still images and moving images, image data and audio data are encoded by a predetermined encoding method. Also, when reproducing a still image data file or a moving image data file recorded on the recording medium 157 , the compression/decompression unit 153 decodes the encoded data and stores it in the RAM 154 .

The RAM 154 is used as system memory, video memory, buffer memory, etc. for executing programs.
The ROM 155 stores programs that can be executed by the processor of the main control unit 151, various setting values, unique information of the digital camera 100, GUI data, and the like. The ROM 155 may be electrically rewritable.

The operation unit 156 is a general term for a group of input devices such as switches, buttons, keys, and touch panel for the user to input instructions to the digital camera 100 . An input through the operation unit 156 is detected by the main control unit 151 through the bus 160, and the main control unit 151 controls each unit in order to realize operations according to the input.

The main control unit 151 has one or more programmable processors such as a CPU and an MPU, and controls each unit by reading a program stored in the ROM 155 into the RAM 154 and executing it, thereby realizing the functions of the digital camera 100 . The main control unit 151 also executes AE processing for automatically determining exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on subject brightness information. Information on subject brightness can be acquired from the image processing unit 152, for example. The main control unit 151 can also determine exposure conditions based on luminance information of a specific subject area such as a person's face.

The main control unit 151 fixes the aperture 103 during moving image shooting, and controls the exposure based on the electronic shutter speed (accumulation time) and the magnitude of the gain. The main controller 151 notifies the sensor controller 143 of the determined accumulation time and gain. The sensor control unit 143 controls the operation of the imaging device 141 so that shooting is performed according to the notified exposure conditions.

The distance map generator 161 (distance detector) generates a distance map using image data stored in the RAM 154, for example. For example, the luminance value of a pixel in a distance map represents the subject distance, and is sometimes called a depth map distance image, a depth image, or the like. A distance map can be generated by a known method. For example, the distance map generation unit 161 obtains the defocus amount (the amount of deviation from the in-focus position of the focus lens and the direction of deviation) at each pixel position from the amount of image deviation of the parallax images (images A and B described above). be able to. Since the defocus amount represents the amount of defocus based on the current subject distance, the defocus amount can be regarded as distance information. Of course, the in-focus position of the focus lens may be obtained based on the defocus amount, and the object distance corresponding to the in-focus position may be obtained. In addition, since the amount of image shift and the amount of defocus correspond one-to-one in a normal imaging optical system, it is possible to use the distribution of the amount of image shift as a distance map to perform processing according to the depth. Note that parallax images may be acquired by using the imaging device 100 as a multi-view camera such as a stereo camera, or parallax images may be acquired from a storage medium or an external device.

Also, the distance map can be generated without using parallax images. By finding the focus lens position that maximizes the contrast evaluation value for each pixel, the subject distance can be obtained for each pixel. In addition, from the image data obtained by shooting the same scene multiple times with different focal distances and the point spread function (PSF) of the optical system, distance information for each pixel is calculated based on the correlation between the amount of blur and the distance. can also be asked for. The distance map generator 161 may generate a distance map for the entire image, or may generate a distance map only for a partial area of the image that is necessary for detecting a moving object. Distance map generator 161 stores the generated distance map in RAM 154 . The distance map is referred to by the image processing unit 152 . Note that the distance map generation unit 161 may generate the distance map by obtaining the subject distance for each small area (pixel block) instead of obtaining the subject distance for each pixel.

Furthermore, the distance map generation unit 161 can calculate the reliability for each area of the distance map and store it together with the distance map. There is no particular limitation on the method of calculating the reliability. For example, when generating a distance map using parallax images, in order to obtain an image shift amount between parallax images, a correlation amount (similarity) such as SAD is calculated while changing the relative shift amount, and the correlation is The amount of shift that maximizes (minimizes the amount of correlation) is detected as the amount of image deviation. It is considered that the larger the difference between the calculated average value and the maximum value of the correlation amount, the higher the reliability of the detected image shift amount (defocus amount). Therefore, the difference between the average value and the maximum value of correlation amounts calculated at each pixel position can be used as the pixel position reliability. Note that the distance map generation unit 161 may generate the distance map by obtaining the subject distance and its reliability for each small area (pixel block) instead of obtaining it for each pixel.

The motion detection unit 162 is composed of, for example, an orientation sensor such as a gyro, an acceleration sensor, and an electronic compass, and measures changes in the position and orientation of the digital camera 100 . In this embodiment, as an example, the optical axis of the photographing lens 101 is the roll axis, the axis perpendicular to the roll axis and parallel to the longitudinal direction of the imaging element is the pitch axis, and the axis perpendicular to the roll and pitch axes is the yaw axis. Angular velocities around the yaw and pitch axes are detected as attitude changes. The motion detector 162 saves the detected motion in the RAM 154 . Information on the motion detected by the motion detection unit 162 is referred to by the image processing unit 152 .

The image processing unit 152 applies predetermined image processing to the image data accumulated in the RAM 154 . Image processing applied by the image processing unit 152 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation (demosaicing) processing, and gamma correction processing, as well as signal format conversion processing and scaling processing. . Further, the image processing unit 152 executes moving object detection processing, which will be described later, and selects a moving object as the main subject. In the moving object detection process, the image processing unit 152 can use the distance map generated by the distance map generating unit 161 and the motion detected by the motion detecting unit 162 for main subject discrimination processing based on moving object information.

Information related to the determined area of the main subject may be used for other image processing (for example, white balance adjustment processing, subject brightness information generation processing, etc.). When the focus control unit 133 performs contrast detection AF, the image processing unit 152 can generate a contrast evaluation value and supply it to the focus control unit 133 . The image processing unit 152 stores the processed image data, information about the area of the main subject, and the like in the RAM 154 .

(Moving object detection processing)
FIG. 2 is a schematic diagram showing the image processing unit 152 with functional blocks specialized for moving object detection processing, in order to explain the moving object detection processing in this embodiment. Each of the functional blocks shown in FIG. 2 may be implemented as a separate hardware circuit, or may be implemented by a programmable processor included in the image processing unit 152 reading a program into memory and executing it.

FIG. 3 is a flowchart of moving object detection processing performed by the image processing unit 152 .
In S<b>201 , the image input unit 501 acquires two frames of input images captured at different times from the RAM 154 . The current frame may be acquired from the sensor control unit 143 and the previous frame (past frame) may be acquired from the RAM 154 . The image input unit 501 supplies the acquired input image to the motion vector detection unit 502 and the saliency calculation unit 507 .

In S<b>202 , the motion vector detection unit 502 detects a plurality of motion vectors between the two frames of input images supplied from the image input unit 501 . Motion vectors can be detected using any known technique. In this embodiment, template matching is used as an example to detect motion vectors.

That is, the motion vector detection unit 502 generates a plurality of partial images by horizontally and vertically dividing an image of one frame (referred to as frame t-1) captured earlier among the images of the two frames. do. Then, the motion vector detection unit 502 performs template matching using each partial image as a template and an image of one frame captured later (assumed to be frame t) as a reference image. Explore in the reference image. Then, the motion vector detection unit 502 creates a vector whose starting point is the coordinates of the central point of the partial image and whose ending point is the coordinates of the central point of the area found by the search that has the highest degree of similarity to the partial image. be a motion vector. In this manner, the motion vector detection unit 502 detects a motion vector for each partial image of frame t-1.

を始点、それらに対応するフレームt上の点

を終点とするベクトルv_iを以下のように算出する。

なお、等号の上に三角形が付された記号は、左辺を右辺によって定義することを意味する。また、ベクトルの第一成分を画像の横方向、第二成分を画像の縦方向とする（以下同様）。動きベクトル検出部５０２は、検出した動きベクトルv_iと、動きベクトルv_iの終点e_iとを関連づけてＲＡＭ１５４に保存する。 FIG. 4(a) shows an example of frame t-1, and FIG. 4(b) shows an example of frame t. In FIG. 4(b), some motion vectors detected by template matching are indicated by arrows. ing. point on frame t-1

, and their corresponding points on frame t

is calculated as _follows .

Note that the symbol with a triangle above the equal sign means that the left side is defined by the right side. Also, let the first component of the vector be the horizontal direction of the image and the second component be the vertical direction of the image (the same applies hereinafter). The motion vector detection unit 502 associates the detected motion vector _vi with the end point _ei of the motion vector _vi and stores them in the RAM 154 .

In S203, the clustering unit 503 clusters the motion vector group _vi detected in S202. Any known clustering method such as the K-Means method or Affinity Propagation can be used for clustering. FIG. 4(c) shows an example of clustering results using Affinity Propagation, which is a clustering method capable of automatically determining the number of clusters. When using a clustering method such as the K-Means method that cannot automatically determine the number of clusters, clustering is performed after determining the number of clusters by some method.

In S204, the background cluster selection unit 504 selects a background cluster made up of background region vectors from among the clusters obtained by the clustering in S203 (FIG. 4(c)). A method of selecting the background cluster will be described later. If the background cluster does not exist or cannot be selected, the background cluster selection unit 504 sets the background cluster selection result to no background cluster. The background cluster selection unit 504 supplies the background cluster selection result to the background vector calculation unit 505 .

In S205, the background vector calculation unit 505 determines whether or not a background cluster exists. If it is determined that a background cluster exists, the process proceeds to S206. do. The background vector calculation unit 505 can determine that the background cluster does not exist when the background cluster selection result indicates no background cluster.

In S<b>206 , the background vector calculation unit 505 calculates the background vector b representing the motion of the background from the motion vectors belonging to the background cluster selected by the background cluster selection unit 504 . Here, as an example, the average vector of the motion vectors belonging to the background cluster is calculated as the background vector b, but the invention is not limited to this.

なお、dist_m（すなわち、背景ベクトルbと動きベクトルv_iとの最大のユークリッド距離）が所定の閾値未満である場合、動体選択部５０６は動体が検出されないものと判定して動体検出処理を終了する。この場合、動体検出結果は、動体検出無しとなる。一方、dist_mが閾値以上であれば、動体選択部５０６は、動きベクトルv_mの終点の画像座標e_mをフレームtに対する動体検出位置として出力し、動体検出処理を終了する。 In S207, the moving object selection unit 506 calculates the Euclidean distance between all motion vectors detected by the motion vector detection unit 502 and the background vector obtained in S206. Then, the moving object selection unit 506 selects the motion vector that maximizes the calculated Euclidean distance. Specifically, if dist _{i is} the Euclidean distance between the motion vector v _i and the background vector b, and m is the index of the motion vector to be selected, the moving object selection unit 506 performs the following equations (2) and (3): Determine the motion vector index m.

Note that when dist _m (that is, the maximum Euclidean distance between the background vector b and the motion vector v _i ) is less than a predetermined threshold, the moving object selection unit 506 determines that no moving object is detected, and terminates the moving object detection process. do. In this case, the moving object detection result is no moving object detection. On the other hand, if dist _m is equal to or greater than the threshold, the moving object selection unit 506 outputs the image coordinates _em of the end point of the motion vector v _m as the moving object detection position for the frame t, and ends the moving object detection process.

The detected moving object detection position can be used to control the operation of the imaging device. For example, by setting the focus detection area so as to include the moving object detection position, it is possible to control the focus detection of the lens unit so as to focus on the moving object detection position. Also, automatic exposure control can be performed so that the moving object detection position is properly exposed.

(Background cluster selection process example 1)
Here, an example of the background cluster selection process performed by the background cluster selection unit 504 in S204 will be described using the flowchart of FIG. In this example, the background cluster selection unit 504 selects, as the background cluster, the cluster in which the motion vectors forming the cluster exist in the largest possible range in the image. This can be said to select the cluster having the widest distribution range of detection positions of the motion vectors forming the cluster as the background cluster.

ここで、n_kはクラスタkを構成する動きベクトルの総数であり、

はそれぞれ、クラスタkを構成する動きベクトルの終点のｘ座標ex_iおよびｙ座標ey_iの平均値である。 In S<b>301 , the background cluster selection unit 504 calculates the variance of the start point coordinates or the end point coordinates of the motion vectors forming the clusters for each cluster generated by the clustering unit 503 . The background cluster selection unit 504 calculates, for example, the x-coordinate variance and the y-coordinate variance of the coordinates, and adds them together to calculate the variance value of the coordinates. In this embodiment, the background cluster selection unit 504 calculates the variance var _k of cluster k by simply adding the variance of the x-coordinate and the variance of the y-coordinate as shown in the following equation (4). and the variance of the y-coordinate may be added with a weight according to the aspect ratio of the image.

where n _k is the total number of motion vectors that make up cluster k,

are the average values of the x-coordinates ex _i and y-coordinates ey _i of the end points of the motion vectors forming the cluster k.

In S302, the background cluster selection unit 504 selects the maximum value var _K of the variance var _k calculated in S301 and the corresponding cluster K, and stores them in the internal memory, for example.

In S303, the background cluster selection unit 504 determines whether or not the maximum variance value var _K stored in S302 is greater than or equal to a predetermined threshold.

In S304, the background cluster selection unit 504 selects the cluster K corresponding to the maximum variance value var _K stored in S302 as the background cluster, and ends the background cluster selection process. Here, the cluster _K corresponding to the maximum variance var K was selected as the background cluster.

In S305, the background cluster selection unit 504 determines that there is no background cluster, and ends the background cluster selection process.

(Background cluster selection process example 2)
Another example of the background cluster selection process performed by the background cluster selection unit 504 in S204 will be described using the flowchart of FIG. In this example, the background cluster selection unit 504 selects background clusters based on the movement of the digital camera 100 . By considering the movement of the digital camera, stable background cluster selection can be achieved.

In S<b>401 , the motion input unit 508 acquires motion information (here, angular velocities around the yaw axis and pitch axis) of the digital camera 100 from the motion detection unit 162 and supplies it to the background cluster selection unit 504 .

で定義されるgに平行な方向と推定する。なお、ここで説明した方法は単なる一例であり、ロール軸周りの角速度を考慮したり、シフト方向の速度を考慮したりして背景の動き方向を推定してもよい。 In S<b>402 , the background cluster selection unit 504 estimates the motion direction of the background from the motion information received from the motion input unit 508 . In this embodiment, the background cluster selection unit 504 selects the moving direction of the background as yaw for the angular velocity around the yaw axis and pitch for the angular velocity around the pitch axis.

is assumed to be the direction parallel to g defined by Note that the method described here is merely an example, and the direction of movement of the background may be estimated by considering the angular velocity around the roll axis or the velocity in the shift direction.

In S403, the background cluster selection unit 504 calculates the angle formed by the cluster center vector of each cluster and g in Equation (5). Here, the cluster center vector is a representative vector of the motion vectors v _i forming each cluster, and may be an average vector, for example.

In S404, the background cluster selection unit 504 selects the minimum value θ _K of the angle calculated in S403 and the corresponding cluster K, and stores them in the internal memory, for example.

In S405, the background cluster selection unit 504 determines whether or not θ _K selected in S404 is less than a predetermined threshold.

In S406, the background cluster selection unit 504 selects the cluster K selected in S404 as the background cluster, and ends the background cluster selection process.

In S407, the background cluster selection unit 504 determines that there is no background cluster, and ends the background cluster selection process.

(Another example 1 of moving object detection processing)
Next, another example of the moving object detection process in S207 will be described. In this example, the moving object selection unit 506 applies moving object detection processing only to visually conspicuous regions (salient regions) in the image, thereby making it possible to select regions that are more likely to be a subject.

The salience calculation unit 507 calculates the salience of each end point coordinate _ei of the motion vector detected by the motion vector detection unit 502 for the frame t supplied from the image input unit 501 . The saliency calculation unit 507 calculates, for example, Laurent Itti, Christof Koch, and Ernst Niebur, ``A Model of Saliency-Based Visual Attention for Rapid Scene Analysis'', IEEE Transactions on Pattern Analysis and Machine Intelligence archive Volume 20 Issue 11, November 1998 Pages. Saliency can be calculated using any known saliency calculation method, such as saliency described in pp. 1254-1259. The salience calculation unit 507 stores the calculated salience in the RAM 154 in association with the motion vector.

In S207, the moving object selection unit 506 limits the motion vectors for calculating the Euclidean distance to the background vector b to motion vectors having a degree of salience equal to or greater than a predetermined threshold. As a result, moving object detection can be applied by limiting it to the salient region.

(Another example 2 of moving object detection processing)
Next, still another example of the moving object detection processing in S207 will be described. In this example, the moving object selection unit 506 preferentially applies moving object detection to an area close to the digital camera 100 . As a result, when there are a plurality of moving objects, the moving object in front can be preferentially detected as the main subject.

The distance map input unit 509 acquires the distance map generated for the frame t by the distance map generation unit 161 and supplies it to the moving object selection unit 506 .

In S207, the moving object selection unit 506 limits the motion vectors for calculating the Euclidean distance from the background vector b to motion vectors whose object distance corresponding to the end point coordinates _ei is less than a threshold. As a result, moving object detection can be applied by limiting it to an area close to the digital camera 100 .

Alternatively, the moving object selection unit 506 may set the moving object detection position _ei , which has the smallest subject distance corresponding to the end point coordinates _ei , among the motion vectors whose Euclidean distance to the background vector b is equal to or greater than a threshold.

The latter example will be described with reference to FIG. FIGS. 7A and 7B schematically show frame images containing a person and a dog as moving objects. For any frame image, the motion vector relating to the person and the dog has the Euclidean distance from the background vector b equal to or greater than the threshold, and can be assumed to be the motion vector of a moving object. Among the motion vectors whose Euclidean distance is equal to or greater than the threshold, e _i with the smallest object distance corresponding to the end point coordinates e _i is used as the moving object detection position. In b) the end point of the dog's motion vector is selected as the motion detection position.

(another example of background vector calculation)
In the above configuration, motion vectors are clustered to select a background cluster, and the background vector b is calculated from the motion vectors forming the background cluster. However, the movement of the digital camera 100 and the focal length (angle of view) of the imaging lens 101 may be used to calculate the background vector without clustering. This makes it possible to remove the computational load associated with the clustering of motion vectors. In addition, this method is useful in a scene where the background area has little color change and it is difficult to detect the motion vector.

The background vector calculation unit 505 acquires the focal length f [mm] of the photographing lens 101 (photographing optical system) when the frame t was photographed through the zoom control unit 113 . The motion input unit 508 also acquires motion information (here, angular velocities around the yaw axis and pitch axis) of the digital camera 100 from the motion detection unit 162 and supplies it to the background vector calculation unit 505 .

ここで、Ａは撮像素子１４１上の実距離[mm]を画像座標系の距離[pixel]に変換するための係数であり、デジタルカメラ１００の構造と画像サイズで決定される撮像素子の情報であり、例えばＲＯＭ１５５に予め定め記憶されている。Ｔはフレームt-1とフレームtの撮影間隔[sec]である。 The background vector calculation unit 505 can calculate the background vector b according to Equation (6) below.

Here, A is a coefficient for converting the actual distance [mm] on the image sensor 141 to the distance [pixel] in the image coordinate system, and is the information of the image sensor determined by the structure of the digital camera 100 and the image size. Yes, it is stored in advance in the ROM 155, for example. T is the imaging interval [sec] between frame t-1 and frame t.

The background vector calculation unit 505 supplies the calculated background vector b to the moving object selection unit 506 . The processing of the moving object selection unit 506 may be as described above.

As described above, according to this embodiment, even a moving object moving in the same direction as the background is detected by selecting the motion vector of the moving object based on the Euclidean distance between each motion vector and the background vector. can do.

(Second embodiment)
Next, a second embodiment of the invention will be described. The present embodiment relates to an image processing apparatus and a control method thereof for discriminating a main subject intended by a user with high probability when discriminating a main subject using motion information. Below, the same reference numerals are used for configurations and operations that are similar or similar to those of the first embodiment, and overlapping descriptions are omitted.

FIG. 8 is a schematic diagram showing the image processing unit 152 with functional blocks specialized for moving object detection processing, in order to explain the moving object detection processing in this embodiment. Each of the functional blocks shown in FIG. 8 may be implemented as a separate hardware circuit, or may be implemented by a programmable processor included in the image processing unit 152 reading a program into memory and executing it.

The image input unit 501 acquires two frames of input images captured at different times from the RAM 154 . The current frame may be acquired from the sensor control unit 143 and the previous frame (past frame) may be acquired from the RAM 154 .

A motion vector detection unit 502 detects a plurality of motion vectors between input images acquired by the image input unit 501 . Motion vectors can be detected using any known technique. In this embodiment, template matching is used as an example to detect motion vectors.

A clustering unit 503 classifies a plurality of motion vectors detected by the motion vector detection unit 502 into at least the following three clusters.
- Background cluster: a motion vector of the entire image (for example, a vector based on the motion of the imaging device 100)
・Main subject cluster: motion vector related to the main subject in the previous frame ・Main subject candidate cluster: vector with a large distance from the motion vector belonging to the background cluster (vector with large motion in real space)

By classifying into at least these three clusters, it is possible to realize determination that considers the current motion information of the main subject in main subject determination processing based on motion. Therefore, it is possible to increase the probability of correctly determining the main subject intended by the user. For example, if the current main subject is stationary in real space, it is possible to determine another moving subject as the new main subject. Alternatively, if the current main subject is moving in real space, it is possible not to change the determination result of the main subject even if another moving object exists.

A main subject discriminating unit 802 discriminates a main subject area in the current frame based on the result of clustering by the clustering unit 503 . Information on the main subject area determined by the main subject determination unit 802 can be used for various processes such as automatic exposure control and automatic focus detection. Also, the subject tracking unit 801 executes subject tracking processing for searching for the main subject area in the input image of the next and subsequent frames by matching processing using the main subject area determined by the main subject determining unit 802 as a template. The subject tracking result (information about the main subject area in the image) by the subject tracking unit 801 is also supplied to the clustering unit 503 .

In addition to the input image acquired by the image input unit 501, other information that can be acquired by the imaging device 100 may be used for main subject determination processing based on motion information. For example, motion information detected by the motion detection unit 162 can be input to the clustering unit 503 through the motion input unit 508 and used to estimate the background cluster. Further, the distance map obtained by the distance map generation unit 161 can be input to the main subject determination unit 802 through the distance map input unit 509 and used for main subject determination processing.

(Main subject discrimination processing)
FIG. 9 is a flowchart of main subject determination processing performed by the image processing unit 152 .
In S901, the image input unit 501 acquires two frames of input images captured at different times.
(Motion vector detection)
In S<b>902 , the motion vector detection unit 502 detects a plurality of motion vectors between the two frames of input images acquired by the image input unit 501 . The motion vector detection unit 502 generates a plurality of partial images by horizontally and vertically dividing the first frame image (frame t-1) of the two frame images. Then, the motion vector detection unit 502 performs template matching using each partial image as a template and an image of one frame captured later (assumed to be frame t) as a reference image. Explore in the reference image. Then, the motion vector detection unit 502 creates a vector whose starting point is the coordinates of the central point of the partial image and whose ending point is the coordinates of the central point of the area found by the search that has the highest degree of similarity to the partial image. be a motion vector. In this manner, the motion vector detection unit 502 detects a motion vector for each partial image of frame t-1 in the same manner as in the first embodiment. Then, the motion vector detection unit 502 associates the detected motion vector _vi with the end point _ei of the motion vector _vi and stores them in the RAM 154 .

(Clustering)
S 903 to S 906 relate to clustering processing of motion vectors by the clustering unit 503 .
In S903, the clustering unit 503 clusters the motion vector group _vi detected in S902. As in the first embodiment, any known clustering method such as the K-Means method or Affinity Propagation can be used for clustering.

FIG. 10A schematically shows an image of the current frame and motion vectors detected between the current frame and the previous frame. FIG. 10(c) shows an example of clustering results using Affinity Propagation, which is a clustering method capable of automatically determining the number of clusters, for these motion vectors. When using a clustering method such as the K-Means method that cannot automatically determine the number of clusters, clustering is performed after determining the number of clusters by some method.

(select background cluster)
In S904, the clustering unit 503 selects a background cluster composed of vectors of background regions (FIG. 10(d)) among the clusters obtained by the clustering in S903 (FIG. 10(c)). Background clusters can be selected by the method described with reference to FIGS. 5 and 6 in the first embodiment.

(Selection of main subject cluster)
Returning to FIG. 9, in S905, the clustering unit 503 selects a main subject cluster composed of vectors of the main subject area from among the clusters obtained in S903. The main subject area is before the previous frame and has been determined by the main subject determination unit 802 . Then, the subject tracking unit 801 searches for the main subject area in the image of the current frame acquired by the image input unit 501, for example, by template matching using the most recently determined main subject area as a template.

Specifically, the subject tracking unit 801 sequentially calculates the degree of similarity between the partial area of the current frame image and the main subject area (template) while changing the position of the partial area, thereby finding the area with the highest similarity to the template. is searched in the image of the current frame. The degree of similarity may be, for example, the amount of correlation. Then, when the similarity of the partial region with the highest similarity exceeds a predetermined threshold, the subject tracking unit 801 detects the partial region in the same image as the template (that is, the main subject region in the image of the current frame). Determine that there is.

FIG. 10B shows an area 501 determined to be the main subject area by the subject tracking unit 801 . The clustering unit 503 acquires information on the main subject area from the subject tracking unit 801, and assigns a motion vector including the end point _ei in the main subject area to the main subject cluster (FIG. 10(d)). If the subject tracking unit 801 cannot obtain the subject tracking result, the clustering unit 503 may use, for example, an image area specified by the user as the main subject area.

(Selection of main subject candidate cluster)
In S906, the clustering unit 503 selects, from among the clusters obtained in S903, main subject candidate clusters composed of motion vectors indicating new main subject area candidates. First, the clustering unit 503 calculates a background vector b representing the motion of the background from the motion vectors belonging to the background cluster selected in S905. Here, as in the process performed in S206 by the background cluster selection unit 504 of the first embodiment, the average vector of the motion vectors belonging to the background cluster is calculated as the background vector b (FIG. 10(d)). It is not limited to this.

The clustering unit 503 then calculates the Euclidean distance between all motion vectors detected by the motion vector detection unit 502 and the background vector b. Then, the clustering unit 503 selects the motion vector that maximizes the calculated Euclidean distance. This process is the same as the process performed in S207 by the moving object selection unit 506 of the first embodiment. Specifically, the clustering unit 503 determines the motion vector index m according to equations (2) and (3). When a plurality of motion vectors with the maximum Euclidean distance are detected, a plurality of indices m are also determined. Note that when dist _m (that is, the maximum Euclidean distance between the background vector b and the motion vector v _i ) is less than a predetermined threshold, the clustering unit 503 determines that there is no motion vector to be assigned to the main subject candidate cluster. On the other hand, if dist _m is equal to or greater than the threshold, the clustering unit 503 assigns the motion vector v _m to the main subject candidate cluster (FIG. 10(d)).

(Main subject discrimination)
In S907, the main subject determination unit 802 determines the main subject based on the motion vectors belonging to the three clusters of background, main subject, and main subject candidate. FIG. 11 is a flowchart regarding details of the main subject determination processing in S907.

In S1101, the main subject determination unit 802 calculates the Euclidean distance between the representative vector of motion vectors belonging to the main subject cluster (eg, average vector) and the representative vector of motion vectors belonging to the background cluster (eg, background vector b). Then, the main subject determination unit 802 determines whether or not the calculated Euclidean distance is less than a predetermined threshold. Proceed to. Here, the fact that the distance is less than the threshold indicates that the movement of the main subject in the real space is small. Note that the main subject determination unit 802 advances the process to S1102 if there is no motion vector belonging to the main subject cluster, and to S1104 if there is no motion vector belonging to the background cluster.

In S1102, the main subject determination unit 802 determines whether or not a motion vector satisfying a predetermined condition exists in the main subject candidate cluster. In this embodiment, since only one type of motion vector constitutes a main subject candidate cluster, the main subject determination unit 802 determines whether a motion vector belonging to a main subject candidate cluster satisfies a predetermined condition. good too.

In S1103, the main subject discriminating unit 802 discriminates, as a new main subject area, the partial area in the current frame image in which the motion vector (here, motion vector v _m ) belonging to the main subject candidate cluster is detected. That is, the main subject determination unit 802 determines to change the main subject. Note that if there are a plurality of partial areas in the current frame image for which motion vectors belonging to main subject candidate clusters have been detected, the main subject determination unit 802 selects a group of partial areas adjacent to one or more other partial areas as the main subject. You may discriminate|determine with an area|region.

Here, the predetermined condition in S1102 is to determine whether or not the main subject intended by the user is more likely to be another subject corresponding to the motion vector belonging to the main subject candidate cluster than the current main subject. It is a condition for Then, when it is determined that the predetermined condition is satisfied, the main subject determination unit 802 estimates that the main subject intended by the user is another subject corresponding to the motion vector belonging to the main subject candidate cluster, to change Then, the main subject determination unit 802 notifies the subject tracking unit 801 of information on the new main subject area. In response to the notification, the subject tracking unit 801 updates the template used for tracking processing to a template based on the new main subject area.

For example, when framing the digital camera 100 to place the subject in the center of the screen, the subject is likely to be the main subject intended by the user. By setting a predetermined condition that a motion vector belonging to a main subject candidate cluster is a motion vector in a direction approaching the center of the image, when such a camera operation is performed on another subject, the main subject You can change the area. As a result, the main subject intended by the user can be determined with high probability.

Also, there is a high possibility that the subject approaching the digital camera 100 is the main subject. Therefore, by making it a predetermined condition that a motion vector belonging to a main subject candidate cluster is a motion vector in a direction approaching the digital camera 100, when another subject is approaching the digital camera 100, the main subject region can be changed. Also in this case, the main subject intended by the user can be discriminated with good probability.

It should be noted that whether or not the motion vector is in the direction of approaching the camera can be determined using the distance map acquired by the distance map input unit 509 . A motion vector in the direction of approaching the digital camera 100 is a motion vector in a direction in which the distance of another subject becomes smaller. A motion vector belonging to the main subject candidate cluster whose end point is closer to the start point can be determined to be a motion vector in the direction of approaching the digital camera 100 .

Note that the predetermined condition described here is merely an example, and any other condition for determining that the subject corresponding to the motion vector belonging to the main subject candidate cluster is likely to be the main subject intended by the user. You can set the conditions. Also, a plurality of predetermined conditions may exist. In this case, it is possible to change the main subject area if it is determined that even one of a plurality of conditions is satisfied, or to change the main subject area if it is determined that all of the plurality of conditions are satisfied. .

In S1104, the main subject determination unit 802 determines to maintain (not change) the main subject area. For example, when the Euclidean distance between the representative vector of the motion vectors belonging to the main subject cluster and the representative vector of the motion vectors belonging to the background cluster is equal to or greater than a threshold, it is considered that the main subject is moving significantly. In this case, the current main subject is not changed. Further, when a moving subject is not detected, the dist _m described above is less than the threshold, and there is no motion vector assigned to the main subject candidate cluster, so the current main subject area is maintained.

As described above, in this embodiment, when the main subject is determined based on the motion vector detected between frames, the motion vector of the current main subject and another subject is taken into consideration. Then, for example, when it is assumed that the main subject intended by the user is not the current main subject but another subject from the motion vector of another subject, the main subject is changed. Therefore, it is possible to increase the possibility of determining the main subject that matches the user's intention.

In this embodiment, the main subject candidate cluster is a cluster composed only of motion vectors having the maximum Euclidean distance from the background vector b. However, other methods may be used to define the main subject candidate clusters. For example, main subject candidate clusters are selected from motion vectors having the largest Euclidean distance from the background vector b and the difference in magnitude and direction from the motion vector having the largest Euclidean distance from the background vector b. It may be a configured cluster.

Also, in S1101, if the Euclidean distance between the representative vector of the motion vectors belonging to the main subject cluster and the representative vector of the motion vectors belonging to the background cluster is less than the threshold, the process proceeds to S1102. However, even if the Euclidean distance is less than the threshold, if it is determined that the main subject is approaching based on the distance map, the process may proceed to S1104 without proceeding to S1102.

(Other embodiments)
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

Moreover, the above-described embodiments are merely specific examples for the purpose of helping understanding of the present invention, and are not intended to limit the present invention to the above-described embodiments in any sense. All embodiments falling within the scope defined by the claims are included in the present invention.

DESCRIPTION OF SYMBOLS 100... Imaging apparatus 101... Lens unit 141... Imaging element 142... Imaging signal processing part 143... Imaging control part 151... Main control part 152... Image processing part 154... RAM 155... ROM 161... distance map generator, 162 motion detector

Claims (19)

An image processing device,
motion vector detection means for detecting a plurality of motion vectors between a plurality of images;
calculation means for calculating a background vector representing the motion of the background based on the motion vector;
moving object detection means for detecting a motion vector of a moving object from the plurality of motion vectors based on the Euclidean distance between each of the plurality of motion vectors and the background vector;
clustering means for clustering the plurality of motion vectors to generate clusters;
a selection means for selecting a background cluster from the clusters generated by the clustering means;
and motion detection means for detecting motion of the image processing device,
The calculation means calculates the background vector from motion vectors forming the background cluster ,
The selecting means estimates the direction of motion of the background from the motion of the image processing device detected by the motion detecting means, and selects the background cluster from the clusters generated by the clustering means using the direction of motion of the background. ,
An image processing apparatus characterized by: An image processing device,
motion detection means for detecting motion of the image processing device;
motion vector detection means for detecting a plurality of motion vectors between a plurality of images;
Acquisition means for acquiring information about the focal length of the imaging optical system and the imaging device that captured the plurality of images;
a calculation means for calculating a background vector representing the movement of the background from the movement of the image processing device, the focal length, the information of the image sensor, and the photographing intervals of the plurality of images;
moving object detection means for detecting a motion vector of a moving object from the plurality of motion vectors based on the Euclidean distance between each of the plurality of motion vectors and the background vector;
An image processing device comprising : further comprising saliency calculation means for calculating the salience of the image at the end point coordinates of the motion vector;
3. The moving object detection means according to claim 1, wherein said moving object detection means detects the motion vector of said moving object from among said plurality of motion vectors, the motion vectors whose saliency of the image at the coordinates of the end point is equal to or greater than a threshold. Image processing device. further comprising distance detection means for detecting a subject distance at a pixel position;
The moving object detection means detects the motion vector of the moving object from among the plurality of motion vectors, the motion vector of which the subject distance at the pixel position of the end point coordinates is less than a threshold.
3. The image processing apparatus according to claim 1, wherein: further comprising distance detection means for detecting a subject distance at a pixel position;
The moving object detection means detects, from among the plurality of motion vectors, a motion vector having the Euclidean distance equal to or greater than a threshold and having a minimum object distance at a pixel position of end point coordinates as the motion vector of the moving object. do,
3. The image processing apparatus according to claim 1, wherein: An image processing device according to any one of claims 1 to 5;
an imaging device that captures the plurality of images;
a control means for performing focus detection and/or exposure control based on the motion vector of the moving object detected by the image processing device;
An imaging device characterized by comprising: A control method for an image processing device,
detecting multiple motion vectors between multiple images;
calculating a background vector representing the motion of the background based on the plurality of motion vectors;
Detecting a motion vector of a moving object from the plurality of motion vectors based on the magnitude of the Euclidean distance between each of the plurality of motion vectors and the background vector;
clustering the plurality of motion vectors to generate clusters;
selecting a background cluster from the clusters;
detecting motion of the image processing device;
the calculating includes calculating the background vector from motion vectors that make up the background cluster;
Said selecting is
estimating the motion direction of the background from the motion of the image processing device;
selecting the background cluster from the clusters using the background motion direction;
A control method for an image processing apparatus, characterized by: A control method for an image processing device,
detecting motion of the image processing device;
detecting multiple motion vectors between multiple images;
Acquiring information about the focal length of the imaging optical system and the imaging device that captured the plurality of images;
calculating a background vector representing the movement of the background from the movement of the image processing device, the focal length, the information of the imaging element, and the photographing intervals of the plurality of images;
Detecting a motion vector of a moving object from the plurality of motion vectors based on the magnitude of the Euclidean distance between each of the plurality of motion vectors and the background vector;
A control method for an image processing device, comprising : A program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 1 to 5. vector detection means for detecting a plurality of motion vectors between a plurality of images;
determining means for determining an area of the main subject based on the plurality of motion vectors;
The determination means is
Detecting a motion vector related to a subject other than the current main subject from among the plurality of motion vectors;
calculating a background vector representative of the motion vectors relating to the background among the plurality of motion vectors;
Detecting, from among the plurality of motion vectors, a motion vector having a Euclidean distance to the background vector equal to or greater than a threshold value as a motion vector relating to the another subject;
determining the another subject as a new main subject;
An image processing apparatus characterized by: The discriminating means does not change the current main subject when the Euclidean distance between the motion vector relating to the current main subject and the background vector, among the plurality of motion vectors, is equal to or greater than a threshold. The image processing apparatus according to claim 10. The determination means does not change the current main subject when it is determined that the current main subject is approaching based on the motion vector relating to the current main subject among the plurality of motion vectors. 12. The image processing apparatus according to claim 10, wherein: further comprising clustering means for clustering the plurality of motion vectors to generate clusters;
The determination means is
Selecting, from among the clusters generated by the clustering means, a cluster having the widest distribution range of detection positions of motion vectors constituting the cluster as a background cluster;
calculating the background vector from the motion vectors that make up the background cluster;
13. The image processing apparatus according to any one of claims 10 to 12, characterized by: further comprising clustering means for clustering the plurality of motion vectors to generate clusters;
The determination means is
Selecting, from among the clusters generated by the clustering means, a cluster having the largest variance of the coordinates of the start point or the end point of the motion vectors forming the cluster as a background cluster;
calculating the background vector from the motion vectors that make up the background cluster;
13. The image processing apparatus according to any one of claims 10 to 12, characterized by: clustering means for clustering the plurality of motion vectors to generate clusters;
and motion detection means for detecting motion of the image processing device,
The determination means is
estimating the motion direction of the background from the motion of the image processing device detected by the motion detection means;
selecting a background cluster from the clusters generated by the clustering means using the motion direction of the background;
calculating the background vector from the motion vectors that make up the background cluster;
13. The image processing apparatus according to any one of claims 10 to 12, characterized by: 16. The image according to claim 15, wherein said determining means selects, from among the clusters generated by said clustering means, a cluster having a minimum angle between a representative vector and a moving direction of said background as said background cluster. processing equipment. an image processing device according to any one of claims 10 to 16;
an imaging device that captures the plurality of images;
a control means for performing focus detection and/or exposure control based on the motion vector of the moving object detected by the image processing device;
An imaging device characterized by comprising: detecting multiple motion vectors between multiple images;
determining a region of a main subject based on the plurality of motion vectors;
The determining is
Detecting a motion vector related to a subject other than the current main subject among the plurality of motion vectors;
Calculating a background vector representing a motion vector related to the background among the plurality of motion vectors;
Among the plurality of motion vectors, a motion vector having a magnitude of Euclidean distance to the background vector equal to or greater than a threshold is detected as a motion vector related to the another subject, and the another subject is determined as a new main subject. A control method for an image processing device, comprising: A program for causing a computer to function as each unit included in the image processing apparatus according to any one of claims 10 to 16.

Images