JP7710293B2 - Image blur correction device, control method thereof, and program - Google Patents

Description

The present invention relates to a technology for correcting image blur in an imaging device.

In imaging devices such as digital cameras, images may become blurred due to "hand shake" by the user holding the camera body. Images may also become blurred due to "subject shake," which occurs when a subject, such as a person, moves and changes the subject position.

Methods for detecting "camera shake" include using an angular velocity sensor attached to the imaging device, and using the motion vector of a stationary object (background) in the captured image. On the other hand, a method for detecting "subject shake" includes detecting a subject in the captured image and using the motion vector of the subject.

There are two methods for correcting image blur caused by camera shake or subject movement: optical blur correction and electronic blur correction. With optical blur correction, the correction lens or image sensor in the optical system is moved in response to the shake, and the image formed on the light-receiving surface of the image sensor is moved in a way that counteracts the shake, thereby correcting the image blur. With electronic blur correction, image processing is performed on the captured image to artificially correct the shake.

"Camera shake" and "subject shake" generally have different amounts of shake (the direction and magnitude of the shake), so it is not possible to completely correct both. Therefore, for example, Patent Document 1 discloses a technology that corrects "subject shake" when the subject's face is detected, and corrects "camera shake" when the subject's face is not detected.

JP 2007-201534 A

Whether to correct "camera shake" or "subject shake" is thought to depend on the user's intention (target of focus) in the shooting scene. For example, if the user is focusing on the background, it is desirable to correct "camera shake," which is the shaking of the entire screen. On the other hand, if the user is focusing on the main subject, it is desirable to correct "subject shake." For this reason, it is necessary to appropriately control the target of shake correction according to the user's intention, which changes along with the shooting scene.

However, the technology disclosed in Patent Document 1 switches the shake correction target depending on whether or not a face is detected, regardless of the user's intention, which causes the problem that the image stabilization effect suited to the user's intention cannot be obtained depending on the scene. Furthermore, when the technology in Patent Document 1 is applied to moving images, there is a problem that an unnatural shift in the image occurs when switching, even if it is very small, because the shake correction target is switched in a binary manner depending on whether or not a face is detected.

The present invention has been made in consideration of the above-mentioned problems, and its purpose is to provide an image stabilization device that can achieve the image stabilization effect desired by the user.

an image blur correction device according to the present invention comprising: subject detection means for detecting a specific subject in an input image and outputting subject information; camera information acquisition means for acquiring camera information necessary for estimating a shooting situation; estimation means for estimating an object of interest in the image using the subject information and the camera information; first motion detection means for detecting background movement and subject movement in the input image; conversion means for converting the background movement and subject movement detected by the first motion detection means into a first blur correction amount for correcting background blur and a second blur correction amount for correcting subject blur, respectively; and correction amount calculation means for combining the first blur correction amount and the second blur correction amount based on the object of interest estimated by the estimation means to generate a final blur correction amount, wherein the camera information includes at least any of information indicating a shutter speed, information indicating an AF area, information indicating a focal length, information indicating a user operation, information indicating a detection result of an inertial sensor, information indicating a distance to the subject, and information indicating a detection result of a user's line of sight .

The present invention makes it possible to provide an image stabilization device that can achieve the image stabilization effect desired by the user.

1 is a block diagram showing the configuration of an image capturing apparatus according to a first embodiment of the present invention. 4 is a flowchart illustrating a process performed in the first embodiment. FIG. 4 is a diagram showing the relationship between camera information and an object of interest. FIG. 4 is a diagram for explaining template matching. FIG. 13 is a diagram for explaining a correlation value map. FIG. FIG. 13 is a block diagram showing the configuration of an image capturing apparatus according to a second embodiment. 10 is a flowchart illustrating a process performed in a second embodiment. FIG. 13 is a diagram for explaining a method of expressing a correlation value map. 11A and 11B are diagrams for explaining a correlation value index representing the reliability of a motion vector. 5A and 5B are diagrams for explaining a method of calculating the reliability of a motion vector. 11A and 11B are diagrams for explaining an example in which the degree of separation of motion is low; 5A and 5B are diagrams for explaining a method of calculating a degree of separation of motion. FIG. 13 is a block diagram showing the configuration of an imaging apparatus according to a third embodiment. 13 is a flowchart illustrating a process performed in a third embodiment. 5A to 5C are diagrams illustrating display items relating to image stabilization. 5A to 5C are diagrams illustrating a method of displaying display items relating to image stabilization. FIG. 13 is a block diagram showing the configuration of an image pickup apparatus according to a fourth embodiment of the present invention. 13 is a flowchart illustrating a process performed in the fourth embodiment. 5A and 5B are diagrams for explaining an operation for determining a coefficient and a gain of a high-pass filter. A diagram showing a table for resetting the coefficients and gains of a high-pass filter. A diagram showing a table for resetting the coefficients and gains of a high-pass filter.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

First Embodiment
FIG. 1 is a diagram showing the arrangement of an image pickup apparatus according to a first embodiment of the present invention.

In FIG. 1, the subject image formed by the optical system 101 is converted into an image signal by the image sensor 102, and development processing such as white balance processing, color (luminance and color difference signal) conversion processing, and gamma correction processing is performed by the development processing unit 103, and the image data is output as image data (input image data). The image memory 104 temporarily stores and holds the image data developed by the development processing unit 103 for one frame or multiple frames.

The motion vector detection unit 105 detects a motion vector from the image data input from the development processing unit 103 and the image memory 104. The motion separation unit 106 separates the motion vector detected by the motion vector detection unit 105 into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject. The blur correction amount conversion unit 107 converts the first motion vector and second motion vector obtained by the motion separation unit 106 into a first blur correction amount and a second blur correction amount, respectively.

The subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103 and outputs subject information. Subject information includes, for example, the subject area (position and size of the subject), the subject's human-likeness, the subject's movement, etc.

The camera information acquisition unit 109 acquires camera information necessary to estimate the shooting conditions. Camera information includes, for example, the shooting mode, shutter speed, AF area information, focal length, user instruction information, inertial sensor information, depth information, and line of sight information.

The target object estimation unit 110 estimates the target object in the image using the subject information obtained by the subject detection unit 108 and the camera information obtained by the camera information acquisition unit 109.

The blur correction amount calculation unit 111 generates a final blur correction amount by combining the first blur correction amount and the second blur correction amount obtained by the blur correction amount conversion unit 107 based on the information on the target of interest obtained by the target of interest estimation unit 110.

The generated shake correction amount is output to a shift mechanism that shifts the correction lens in the optical system 101, which is the optical shake correction means, and/or the image sensor 102 in a direction perpendicular to the optical axis, and optical shake correction is performed. Note that the present invention is not limited to optical shake correction, and the shake correction amount may be output to an electronic shake correction means (not shown) to perform electronic shake correction.

Next, the image stabilization operation of the imaging device 100 configured as described above will be explained using the flowchart shown in FIG.

In step S201, the subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103, and outputs subject information. As subject information, subject area information (position and size of the subject), information on the subject's human-likeness, and subject movement information are used. A typical example of a specific subject is a human face. A publicly known face detection method may be used as the detection method. Publicly known face detection techniques include a method that uses knowledge about the face (skin color information, features such as the eyes, nose, and mouth), and a method that uses a learning algorithm such as a neural network to configure a classifier for face detection. To improve detection accuracy, it is common to combine these methods to perform face detection.

When multiple human faces are present in an image, the face that is considered to be the most dominant is selected and output. As a selection method, for example, the face that is larger, closer to the center of the image, and has a higher reliability in face detection may be selected first. Alternatively, the user may be allowed to select from multiple candidates.

By using the above method, the subject area and the human-like nature of the subject can be obtained as subject information. In addition, the movement of the subject can be obtained by calculating the amount of displacement of the center of gravity of the subject area between successive frame images, for example.

In step S202, the camera information acquisition unit 109 acquires camera information necessary to estimate the shooting conditions. The camera information to be used includes shooting mode information, shutter speed information, AF area information, focal length information, user instruction information, inertial sensor information, depth information, and line of sight information.

Shooting mode information, shutter speed information, AF area information, focal length information, and user instruction information can be obtained by reading the values set by the user on the camera. User instruction information is information that indicates the degree of attention to the background or subject, which will be described later, and can be set directly by the user on the camera.

Inertial sensor information is obtained by an angular velocity sensor or acceleration sensor installed in the camera, and is information that indicates the position and orientation of the camera. Depth information is detected, for example, by using a distance measurement sensor, or from captured image data using the well-known SfM (Structure from Motion). Gaze information is obtained by a well-known method such as the corneal reflex method, and is information that indicates the area in the image where the gaze is located.

In step S203, the focus estimation unit 110 estimates whether the photographer is focusing on the background or the subject based on the subject information detected in step S201 and the camera information acquired in step S202.

A method for estimating an object of interest based on subject information and camera information is described below. First, for each piece of information, a background degree, which indicates the degree to which the object of interest is the background, and a subject degree, which indicates the degree to which the object of interest is the subject, are calculated. Here, the background degree and subject degree are expressed as a numerical value that sums to 1. Note that it is also possible to calculate only one of the background degree or subject degree.

First, we will explain the subject information. Regarding the subject's position, the closer the subject is to the center of the screen, the more likely it is that the subject was the focus of the shot, so the subject degree is set high (e.g., 0.8) and the background degree is set low (e.g., 0.2).

Regarding the size of the subject, the larger the subject, the more likely it is that the subject was the focus of the photo, so the subject degree is set high (e.g., 0.8) and the background degree is set low (e.g., 0.2).

Regarding the subject's human-likeness, the more the subject resembles a person, the more likely it is that the subject was the focus of the photo, so the subject degree is set high (e.g., 0.7) and the background degree is set low (e.g., 0.3).

Regarding the subject's movement, the smaller the subject's movement, the more likely it is that the camera is being held to capture the subject, and the more likely it is that the subject is being photographed with a focus on it, so the subject degree is set high (e.g., 0.6) and the background degree is set low (e.g., 0.4).

Next, the camera information will be explained. Regarding the shooting mode, for example, in portrait mode, it is highly likely that the image was shot with a focus on a person (= subject), so the subject degree is set high (e.g. 0.9) and the background degree is set low (e.g. 0.1). On the other hand, in landscape mode, it is highly likely that the image was shot with a focus on the landscape, so the subject degree is set low (e.g. 0.1) and the background degree is set high (e.g. 0.9). In this way, the shooting mode can determine the background degree and subject degree by assuming the most likely shooting situation depending on the mode.

As for the shutter speed, the faster the shutter speed, the more likely it is that the image is being shot with a focus on a fast-moving subject, so the subject degree is set high (e.g., 0.7) and the background degree is set low (e.g., 0.3).

For AF area information, the wider the AF area, the more likely it is that the background is being focused on when taking the picture, so the subject degree is set low (e.g. 0.3) and the background degree is set high (e.g. 0.7). Also, when face recognition AF or eye AF is set, the more likely it is that the subject, such as a person or animal, is being focused on when taking the picture, so the subject degree is set high (e.g. 0.9) and the background degree is set low (e.g. 0.1).

When combined with subject information, the gaze information is used to determine, for example, if the area in the image where the gaze is located is the subject area, there is a high possibility that the subject was the focus of the photograph, so the subject degree is increased (e.g., 0.9) and the background degree is decreased (e.g., 0.1).

It is difficult to understand the photographer's intentions using focal length and depth information alone. Therefore, we will explain an example of a method that combines the two to estimate the subject of interest.

If the focal length f [mm] and the distance to the subject (depth information) d [mm] are given, and the size of the subject on the imaging surface is X [mm], the actual size of the subject Y [mm] can be calculated using the following formula (1).

Y=(d/f)・X (1)
If the actual size of the subject is known, the photographer's intention can be understood from the relationship between the size of the subject on the image plane and the focal length. For example, if the actual size of the subject is small, but the size of the subject on the image plane is large and the focal length is long, the photographer is paying close attention to the subject. Therefore, the smaller the actual size of the subject is, the larger the size of the subject on the image plane is, and the longer the focal length is, the higher the subject degree is and the lower the background degree is.

In this way, in this embodiment, by combining information from multiple cameras, it is possible to understand the photographer's intentions, which cannot be understood from information from a single camera.

It is also difficult to understand the photographer's intentions from inertial sensor information alone. Therefore, we will explain an example of a method for estimating the target of interest by combining inertial sensor information and subject movement information. When the photographer's target of interest is the subject, the camera is moved to keep the subject in a fixed position on the screen, so the subject's movement is relatively small compared to the camera movement.

Therefore, the smaller the amount of subject movement is compared to the amount of camera movement between frame images obtained from the inertial sensor information, the more likely it is that the camera is being held to capture the subject. As a result, it is highly likely that the subject is being focused on when taking the picture, so the subject degree is set high and the background degree is set low.

This concept can also be applied to panning. When panning, the photographer shakes the camera widely to keep a moving subject in the center, resulting in large background movements and small subject movements. Therefore, when panning, the subject degree is automatically increased based on inertial sensor information and subject movement information.

The relationship between the subject information and camera information and the target object described above is summarized in the table in Figure 3. The numbers in parentheses indicate an example of the background degree/subject degree described above.

When multiple pieces of information are available for estimating the target object, the background degree and subject degree obtained for each piece of information are weighted and added together to calculate the final background degree and subject degree. The weights can be set, for example, based on the reliability of each information source.

For example, the information sources are the shooting mode, the subject's human-likeness, the subject's size, the subject's movement, and the shutter speed, and the background degrees obtained from each are Da, Db, Dc, Dd, and De, and the weights are Ga, Gb, Gc, Gd, and Ge. The final background degree D can be calculated using the following formula (2).

D=Ga×Da+Gb×Db+Gc×Dc+Gd×Dd+Ge×De (2)
For example, the weights are Ga=0.3, Gb=0.2, Gc=0.2, Gd=0.1, and Ge=0.2. Here, the weights are normalized so that the sum is 1. If the shooting mode is landscape mode (Da=0.9), the subject is highly human-like (Db=0.3), the subject is small in size (Dc=0.8), the subject moves little (Dd=0.4), and the shutter speed is slow (De=0.7), then D=0.6 is obtained from formula (2). The final subject degree can be calculated in the same way. If the certainty of each information source is the same or unknown, all the weights should be set to the same.

When using user instruction information as camera information, for example, the degree of attention to the background and subject can be determined without estimation by having the user indicate the degree of attention to the background and subject. The degree of attention may be indicated directly as a numerical value, or may be indicated qualitatively, such as strength. If there are multiple subjects, one of them may be selectable. The background degree and subject degree may also be set independently in the horizontal and vertical directions.

In step S204, the motion vector detection unit 105 detects a motion vector by template matching for the image input from the development processing unit 103 and the image memory 104.

Figure 4 is a diagram showing an overview of template matching. Figure 4(a) shows a base image, which is one of two vector detection images, and Figure 4(b) shows the other, a reference image. Here, a frame image stored in image memory 104 is used as the base image, and image data directly input from development processing unit 103 is used as the reference image, to calculate a motion vector from a past frame image to a current frame image. Note that the base image and reference image may be swapped, in which case this means that a motion vector is calculated from the current frame image to a past frame image.

The motion vector detection unit 105 places a template area 301 in the base image and a search area 302 in the reference image, and calculates the correlation value between the template area 301 and the search area 302.

The template regions 301 may be positioned arbitrarily, and may be positioned around multiple fixed coordinates determined within the screen, or around the coordinates of feature points detected by a known method. The subject information obtained in step S201 may also be used to determine the coordinates to be positioned so that the number of template regions positioned in the subject region and other regions is equal. The search region may be positioned with a predetermined size so as to equally encompass the template regions on all four sides.

In this embodiment, the Sum of Absolute Difference (hereinafter abbreviated as SAD) is used as the method for calculating the correlation value. The calculation formula for SAD is shown in Equation (3).

In equation (3), f(i,j) indicates the luminance value at the coordinates (i,j) in the template region 301. Furthermore, g(i,j) indicates the luminance value at each coordinate in the region (hereafter referred to as the correlation value calculation region) 303 in the search region 302 for which the correlation value is to be calculated. In SAD, the absolute value of the difference between the luminance values f(i,j) and g(i,j) in both regions 302 and 303 is calculated, and the correlation value S_SAD is obtained by calculating the sum of these values. The smaller the correlation value S_SAD, the higher the similarity in texture between the template region 301 and the correlation value calculation region 303.

Note that methods other than SAD may be used to calculate the correlation value, such as sum of squared differences (SSD) or normalized cross correlation (NCC).

The motion vector detection unit 105 calculates correlation values by moving the correlation value calculation area 303 throughout the entire search area 302. In this way, a correlation value map such as that shown in FIG. 5 is created for the search area 302.

Figure 5(a) shows a correlation value map calculated in the coordinate system of the search region 302, with the X-axis and Y-axis representing the correlation value map coordinates and the Z-axis representing the magnitude of the correlation value at each coordinate. Figure 5(b) shows the contour lines of Figure 5(a).

In Figures 5(a) and 5(b), the correlation value is smallest for minimum value 401, and it can be determined that the area in search area 302 where minimum value 401 was calculated has a texture that is very similar to that of template area 301. 402 represents the second minimum value, and 403 represents the third minimum value, which means that a texture next to minimum value 401 exists.

In this way, the motion vector detection unit 105 calculates the correlation value between the template area 301 and the search area 302, and determines the position of the correlation value calculation area 303 where this value is smallest. This makes it possible to identify the destination on the reference image of the template area 301 on the standard image. Then, it is possible to detect a motion vector whose direction and magnitude are the direction to the destination on the reference image and the amount of movement based on the position of the template area on the standard image.

In step S205, the motion separation unit 106 separates the motion vector detected in step S204 into a first motion vector (hereafter referred to as a background vector) representing the motion of the background, and a second motion vector (hereafter referred to as a subject vector) representing the motion of the subject.

An example of a separation method will be described. First, using the subject information obtained in step S201, a first separation process is performed with the motion vector detected at a point outside the subject region as the background vector and the motion vector detected at a point inside the subject region as the subject vector.

If no subject is detected, separation based on the depth information obtained in step S202 may be considered. In that case, the first separation process may be performed using, for example, a motion vector detected at a point in an area farther than a predetermined depth in the image as a background vector, and a motion vector detected at a point in an area closer than the predetermined depth as a subject area. If no subject is detected and there is no depth difference in the screen, all are treated as background vectors and the separation process ends.

The separation process may be completed with only the first separation process, but such "separation based on subject area" may result in incorrect classification of motion vectors detected at points near the boundary between the background and the subject. In addition, if the subject area is incorrect due to a problem with the accuracy of subject detection, the motion vector may also be incorrectly classified. For this reason, it is preferable to further perform "separation based on the amount of motion vector" as a second separation process. Here, for example, the well-known k-means method can be used.

In the k-means method, the number of clusters k to be classified and the initial value Vk of the center of gravity of each cluster must be determined in advance. In this case, the number of clusters is set to k = 2 in order to separate into background clusters and subject clusters. If there are multiple subjects, the number of clusters may be changed according to the number of subjects. Also, a cluster may be provided to separate the motion vectors that belong neither to the background nor to the subject.

The results of the first separation process are used for the initial value of the center of gravity of each cluster. Specifically, for the background vectors obtained in the first separation process, histograms are generated for the amount of movement in the X direction and the amount of movement in the Y direction, and the most frequent values V1x and V1y of each histogram are found. Since this most frequent value is the representative value of the background vectors, it is set as the initial value V1 = (V1x, V1y) of the center of gravity of the background cluster. Using a similar method, the representative values V2x and V2y of the subject vectors obtained in the first separation process are found, and the initial value V2 = (V2x, V2y) of the center of gravity of the subject cluster is set.

In addition, the initial value of the background cluster may be determined using the inertial sensor information obtained in step S202 instead of the background vector. In this case, the angular velocities in the yaw and pitch directions obtained from the angular velocity sensor are integrated and converted into angles θy and θp, respectively, and then converted into the amount of displacement on the imaging surface. If the focal length is f, then V1x = f tan(θy) and V1y = f tan(θp) are calculated.

In addition, the initial value of the object cluster may be the displacement of the center of gravity between frames of the object region obtained in step S201, instead of the object vector. In this case, it is assumed that the same object has been detected between frames.

As described above, the number of clusters and the initial value of the center of gravity of each cluster are determined, and the next step is to calculate the distance between each motion vector data and the center of gravity of each cluster. Then, each motion vector data is reassigned to a cluster by assuming that it belongs to the cluster with the closest distance.

If the cluster assignment for all vector data remains unchanged in the above process, or if the amount of change falls below a preset threshold, it is determined that the process has converged and the second separation process is terminated. Otherwise, the process repeats the above-mentioned process of recalculating the centroid from the newly assigned cluster, calculating the distance between each motion vector data and the centroid of each cluster, and reassigning the clusters.

Figure 6 shows an example of the results of separating motion vectors as described above.

Figure 6(a) is a plot of the X-direction displacement of the motion vector on the horizontal axis and the Y-direction displacement on the vertical axis. □ represents the initial value of the center of gravity of the background cluster, and ☆ represents the initial value of the center of gravity of the subject cluster. Those separated as background vectors are indicated with ○, and those separated as subject vectors are indicated with △.

Figure 6(b) shows the starting points (feature point positions) of the background vectors and subject vectors in Figure 6(a) plotted on the image. Figure 6(b) shows that the background and main subject can be correctly separated.

In step S206, the blur correction amount conversion unit 107 converts the background vector and the subject vector obtained in step S205 into a first blur correction amount (hereinafter referred to as the camera shake correction amount) and a second blur correction amount (hereinafter referred to as the subject blur correction amount), respectively.

As a method for converting a motion vector into a blur correction amount, for example, there is a method using a histogram as mentioned above. A histogram is generated for the amount of movement in the X direction and the amount of movement in the Y direction for the motion vector, and the most frequent value of each histogram is calculated. Since this most frequent value is a representative value of the blur occurring between frames, by taking the inverse sign of this most frequent value, a blur correction amount that cancels out the blur can be obtained.

Another conversion method is to use the correspondence between feature points between frames obtained from motion vectors. From the correspondence between feature points between frames, a projective transformation matrix (or affine transformation matrix) is calculated using a known method. The calculated projective transformation matrix represents the blur that occurs between frames, so by calculating the inverse matrix of this matrix, it is possible to obtain a blur correction amount that cancels out the blur.

The method using histograms can only correct the translational component of blur, but has the characteristic that it can obtain a relatively stable amount of correction even when the number of motion vectors is small. On the other hand, the method using the correspondence between feature points can correct the rotational and tilt components of blur in addition to the translational component, but has the characteristic that it cannot obtain an accurate amount of correction when the number of motion vectors is small. Therefore, it is possible to change which method to use depending on the number of motion vectors.

Here, a background vector is used to obtain the first blur correction amount, but if an angular velocity sensor is included, it is not necessary to use a background vector. As explained in the section on how to calculate the initial value of the background cluster, angular velocity information obtained from the angular velocity sensor may be converted into the amount of displacement on the imaging surface and used.

In step S207, the shake correction amount calculation unit 111 generates a final shake correction amount by combining the camera shake correction amount and the subject shake correction amount obtained in step S206 based on the target object obtained in step S203.

An example of a method for combining the camera shake compensation amount and the subject shake compensation amount will be described below. The background degree is represented as A, and the subject degree is represented as 1-A. If the camera shake compensation amount and the subject shake compensation amount calculated in step S206 are H0 and H1, respectively, the final combined shake compensation amount H can be calculated using the following formula.

H=(A×H0)+{(1-A)×H1} (4)
According to formula (4), when the photographer focuses 100% on the background, A=1, so H=H0, and camera shake is corrected 100%. Conversely, when the photographer focuses 100% on the subject, A=0, so H=H1, and subject shake is corrected 100%. And when the photographer focuses 50% on the background and 50% on the subject, A=0.5, so camera shake is corrected 50% and subject shake is corrected 50%. In this way, the correction ratio of camera shake and subject shake is controlled according to the background degree and subject degree. Note that, like the background degree and subject degree, the correction ratio of camera shake and subject shake may be set independently in the horizontal and vertical directions.

In the final step S208, the imaging device 100 determines whether or not processing has been completed up to the final frame. If processing has been completed up to the final frame, the processing ends, and if processing has not been completed, the processing returns to step S201.

As described above, in this embodiment, the photographer can obtain the shake correction effect desired by controlling the correction ratio between camera shake and subject shake according to the subject of the photographer's attention.

The technology described in this embodiment can be applied to both video and still images. In the case of video, it is possible to continuously switch between camera shake correction and subject shake correction depending on the scene, and it is possible to suppress discomfort such as slight image shifts caused by switching. In the case of still images, it is possible to generate images in which camera shake and subject shake are suppressed in an appropriate balance. As mentioned above, it can also be applied to a special type of still image shooting called panning.

Second Embodiment
Fig. 7 is a diagram showing the configuration of an image pickup device 600 according to a second embodiment of the present invention. In Fig. 7, parts common to the components shown in Fig. 1 are given the same reference numerals as in Fig. 1, and description thereof will be omitted. In addition to the configuration shown in Fig. 1, the image pickup device of this embodiment has a reliability calculation unit 601 and a separation degree calculation unit 602. In this embodiment, only parts that perform processing different from the first embodiment will be described.

In the first embodiment, the amount of blur correction was calculated according to the degree of attention given to the background and the main subject. However, depending on the shooting scene, motion detection and motion separation may be difficult even when the degree of attention is high. For example, it is difficult to detect motion in a low-contrast sky, or in a soft body or fluid whose shape changes drastically between frames. Furthermore, if the amount of movement differs within the same subject, or there is a depth difference, it is difficult to separate the movements of the background and the main subject. In such cases, if the amount of blur correction is calculated simply based on the degree of attention, it is thought that blur correction will not be performed correctly and will become unstable.

Therefore, in this embodiment, we will explain a method for performing blur correction more stably by taking into account the reliability of motion detection and the degree of motion separation in addition to the degree of attention to the background and main subject.

This embodiment differs from the first embodiment in that the motion separation unit 106a uses the output results of the reliability calculation unit 601, and the blur correction amount calculation unit 111a uses the output results of the reliability calculation unit 601 and the separation degree calculation unit 602.

The reliability calculation unit 601 calculates the reliability of the motion vector input from the motion vector detection unit 105. The separation calculation unit 602 calculates the separation of the background vector and the subject vector input from the motion separation unit 106a.

Figure 8 is a flowchart showing the image stabilization operation in the second embodiment. In Figure 8, steps common to those shown in Figure 2 are given the same reference numerals as in Figure 2, and descriptions thereof will be omitted.

In step S701, the reliability calculation unit 601 calculates the reliability of the motion vector obtained in step S204.

A two-dimensional correlation value map is used to calculate the reliability of a motion vector. In the two-dimensional correlation value map of FIG. 5(b), the correlation values are arranged in raster order as indicated by arrow 404, and shown in one dimension in FIG. 9. The vertical axis of FIG. 9 is the correlation value, and the horizontal axis is the pixel address that is uniquely determined by the X and Y coordinates of the correlation value map. Hereafter, the representation in FIG. 9 will be used to calculate the reliability of a motion vector. Note that point 801 indicates the position corresponding to the minimum value in FIG. 5.

Figure 10 shows an example of a correlation value index that indicates the reliability of a motion vector. The horizontal axis of Figure 10 shows pixel addresses, and the vertical axis shows correlation values.

In Figure 10(a), the difference Da between the minimum and maximum correlation values is used as an index. Da represents the range of the correlation value map, and if Da is small, it is considered that the texture contrast is low, indicating low reliability.

In FIG. 10(b), the ratio Db (=B/A) of the difference A between the minimum and maximum correlation values and the difference B between the minimum and average correlation values is used as an index. Db represents the steepness of the correlation value peak, and when Db is small, it is considered that the similarity between the template region and the search region is low, indicating low reliability.

In FIG. 10(c), the difference Dc between the minimum value of the correlation value and the second minimum value is used as an index. Here, points 901, 902, and 903 correspond to correlation values 401, 402, and 403 in FIG. 5, respectively. Therefore, FIG. 10(c) means to check whether there is a minimum value similar to the minimum of the correlation value in the contour lines in FIG. 5(b).

Dc represents the periodicity of the correlation value map. If Dc is small, it is possible that the texture is a repeating pattern or edge, indicating low reliability. Note that here, the minimum value and the second minimum value were selected, but other minimum values may be selected as long as it is possible to determine the periodicity of the correlation value map.

In FIG. 10(d), the minimum correlation value Dd is used as the index. If Dd is large, the similarity between the template region and the search region is considered to be low, indicating low reliability. Since Dd and reliability are inversely proportional, the reciprocal of Dd (1/Dd) is used as the index.

The correlation value index described above can be used as the reliability as it is, but the correlation value index and the reliability may be associated as shown in FIG. 11. The horizontal axis of FIG. 11 indicates the correlation value index (any of Da, Db, Dc, or 1/Dd described above), and the vertical axis indicates the reliability. In this example, two thresholds T1 and T2 are set, and the reliability is set to 0 if the reliability is below the threshold T1, and 1 if the reliability is above the threshold T2. The threshold may be changed for each correlation value index. In addition, the correlation value index and the reliability may be associated linearly or nonlinearly in the section between the thresholds T1 and T2. In the following description, the reliability obtained from each correlation value index is expressed as Ra, Rb, Rc, and Rd. Here, the relationship is Ra=f(Da), Rb=f(Db), Rc=f(Dc), and Rd=f(Dd).

The final reliability R of the motion vector can be calculated by combining Ra, Rb, Rc, and Rd. Here, we will explain the combination method using weighted addition and logical operations.

In the combination by weighted addition, if the weights of Ra, Rb, Rc, and Rd are Wa, Wb, Wc, and Wd, respectively, the reliability R is calculated as shown in formula (5).

R=Wa×Ra+Wb×Rb+Wc×Rc+Wd×Rd (5)
For example, the weights are Wa = 0.4, Wb = 0.3, Wc = 0.2, and Wd = 0.1. If all reliabilities are sufficiently high and Ra = Rb = Rc = Rd = 1, then equation (5) gives R = 1.0. If Ra = 0.6, Rb = 0.5, Rc = 0.7, and Rd = 0.7, then equation (5) gives R = 0.6.

In combination using logical operations, if the thresholds for Ra, Rb, Rc, and Rd are Ta, Tb, Tc, and Td, respectively, the reliability R is calculated, for example, using logical product as shown in formula (6).

R=(Ra≧Ta)∧(Rb≧Tb)∧(Rc≧Tc)∧(Rd≧Td) (6)
Here, ^ is a symbol representing a logical product. If Ra ≥ Ta, Rb ≥ Tb, Rc ≥ Tc, and Rd ≥ Td are all satisfied, then R = 1 (high reliability), and otherwise R = 0 (low reliability).

It is also possible to use logical sum as in equation (7).

R=(Ra<Ta)↓(Rb<Tb)↓(Rc<Tc)↓(Rd<Td) (7)
Here, ↓ is a symbol representing a negative logical sum. When none of Ra<Ta, Rb<Tb, Rc<Tc, and Rd<Td is satisfied, R=1 (high reliability), and otherwise R=0 (low reliability).

In step S702, the motion separation unit 106a separates the motion vector detected in step S204 into a background vector and a subject vector, using the reliability calculated in step S701.

The difference with step S205 is that first, of all motion vectors, those with a reliability lower than a predetermined threshold are removed. Motion vectors with low reliability are likely to be falsely detected, increasing the possibility of erroneous motion separation, and are therefore not used. After removing the motion vectors with low reliability, motion separation processing is performed in exactly the same way as in step S205.

In step S703, the separation degree calculation unit 602 uses the background vector and the subject vector separated in step S702 to calculate a separation degree U that indicates the degree of separation between the two vectors. The above-mentioned FIG. 6 shows an example of a state in which the separation degree is high. On the other hand, FIG. 12 shows an example of a state in which the separation degree is low.

As an example of a method for calculating the degree of separation, the variance S of the coordinates of the starting point (=feature point) of the subject vector can be used. When the separated subject vector captures a subject with a certain movement, the variance S has the property of remaining constant over time. Therefore, for example, as shown in Figure 13, the degree of separation U can be calculated so that the larger the inter-frame displacement δS of the variance S is, the lower the degree of separation U becomes.

The horizontal axis of Figure 13 indicates the amount of displacement δS of the variance S between frames, and the vertical axis indicates the degree of separation U. In this example, two thresholds T1 and T2 are set. When δS is small and equal to or less than threshold T1, it is considered that the movement of a specific subject has been separated from the background, as described above, and the degree of separation U is set to 1. Conversely, when δS is large and equal to or greater than threshold T2, it is considered that the movement of a specific subject has not been separated from the background, and the degree of separation U is set to 0. In the section between thresholds T1 and T2, δS and the degree of separation U may be associated linearly or nonlinearly.

In step S704, the blur correction amount calculation unit 111a generates a final blur correction amount by combining the camera shake correction amount and subject blur correction amount obtained in step S206 based on the target object obtained in step S203, the reliability of the motion vector detection obtained in step S701, and the degree of motion separation obtained in step S703.

The difference from step S207 is that the reliability of motion detection and the degree of separation of motion are reflected as the blending ratio of the shake correction amount. The sums of the reliability of the background vector and the subject vector are represented as RA and RB, respectively, and the degree of separation is represented as U. As an example of the reflection method, a method of calculating the shake correction amount as shown in the following formula (8) can be considered.
H=A・RA・(1-U)・H0/{A・RA・(1-U)+(1-A)・RB・U}
+(1-A)・RB・U・H1/{A・RA・(1−U)+(1−A)・RB・U}
(8)
According to formula (8), the higher the reliability RA of the background vector is compared to the reliability RB of the subject vector, the higher the ratio of background blur correction. Also, the higher the degree of separation U, the higher the ratio of subject blur correction.

As described above, in this embodiment, by taking into account the "reliability of motion vector detection" and the "separability of motion" in addition to the "degree of attention to the background and main subject," it is possible to obtain the effect of performing blur correction more stably than in the first embodiment.

Third Embodiment
Fig. 14 is a diagram showing the configuration of an image pickup apparatus 1300 according to a third embodiment of the present invention. In Fig. 14, parts common to the components shown in Fig. 1 are given the same reference numerals as in Fig. 1, and description thereof will be omitted. In addition to the configuration shown in Fig. 1, the image pickup apparatus of this embodiment has a blur correction effect determination unit 1301 and a display unit 1302. In this embodiment, only parts that perform processing different from the first embodiment will be described.

In the first embodiment, the correction ratio between camera shake and subject shake was controlled according to the subject of the photographer's attention. However, when the amount of camera shake and subject shake is roughly equal, the photographer cannot know which type of shake correction is working. Cameras these days come equipped with a camera shake correction function as standard, but many still do not have a subject shake correction function. For this reason, there are many photographers who are familiar with camera shake correction but not with subject shake correction. It is very important to accurately communicate to such photographers the situations in which subject shake correction is working.

Therefore, in this embodiment, we will explain a method for determining a situation in which the effect of subject shake correction is prominent and notifying the photographer of that situation.

This embodiment differs from the first embodiment in that the blur correction effect determination unit 1301 determines the effect of subject blur correction using the output of the focus target estimation unit 110 and the blur correction amount conversion unit 107, and the display unit 1302 notifies the photographer of the determination result.

The blur correction effect determination unit 1301 determines the effect of subject blur correction based on the information on the target obtained by the target estimation unit 110 and the camera shake correction amount and subject blur correction amount obtained from the blur correction amount conversion unit 107. The display unit 1302 displays the blur correction effect determination result input from the blur correction effect determination unit 1301 together with the image data obtained from the development processing unit 103.

Figure 15 is a flowchart showing the processing operation of the third embodiment. In Figure 15, steps common to those shown in Figure 2 are given the same reference numerals as in Figure 2 and descriptions thereof are omitted.

In step S1401, the image stabilization effectiveness determination unit 1301 determines the effectiveness of the object shake correction using the object of interest obtained in step S203 and/or the camera shake correction amount and object shake correction amount obtained in step S206.

As a method of determining the effectiveness of subject blur correction, for example, the higher the subject degree, which indicates the degree to which the object of interest is a subject, and the larger the absolute value of the difference between the amount of camera shake correction and the amount of subject blur correction, the greater the effect of subject blur correction is determined to be. Note that the subject degree or the absolute value of the difference does not have to have a linear relationship with the degree of correction effectiveness. If the subject degree is equal to or greater than a predetermined value, it may be determined that the effect of subject blur correction is greater than if the subject degree is less than the predetermined value, or the effect of subject blur correction may be determined to be greater if the absolute value of the difference between the amount of camera shake correction and the amount of subject blur correction is equal to or greater than a predetermined value.

In addition, the reliability of the subject information, the reliability of the subject vector, the degree of separation between the background vector and the subject vector, and the upper limit of the blur correction amount may be taken into account as information for determining the effectiveness of subject blur correction.

The reliability of the subject information is the reliability of the subject information obtained from the subject detection unit 108. If the specific subject is a person's face, the reliability of the face detection can be used. If the reliability of the subject information is low, there is a possibility that subject blur correction will not work properly. For this reason, it is determined that the effect of subject blur correction is greater when the reliability of the subject information is high than when the reliability is low.

The reliability of the subject vector can be obtained from the reliability calculation unit 601, as described in the second embodiment. If the reliability of the subject vector is low, there is a possibility that subject blur correction will not work properly. Therefore, it is determined that the effect of subject blur correction is greater when the reliability of the subject vector is high than when the reliability is low.

The degree of separation between the background vector and the subject vector can be obtained from the separation calculation unit 602, as described in the second embodiment. If the degree of separation between the background vector and the subject vector is low, subject blur correction may not work properly. For this reason, it is determined that the effect of subject blur correction is greater when the degree of separation between the background vector and the subject vector is high than when the degree of separation is low.

The upper limit of the blur correction amount is the upper limit of the amount of blur that can be corrected by the optical blur correction means or electronic blur correction means described above. In the case of optical blur correction means, the upper limit is determined by the movable range of the correction lens or image sensor in the optical system. In the case of electronic blur correction, the upper limit is determined by the size of the surplus pixel area provided on the periphery of the image output area. If the subject blur correction amount exceeds the upper limit of the blur correction amount, subject blur correction may not work correctly. For this reason, it is determined that the effect of subject blur correction is greater when the value obtained by subtracting the subject blur correction amount from the upper limit of the blur correction amount is larger than when the value is smaller.

The effectiveness of subject blur correction is determined based on the multiple pieces of information shown above. The method of making a determination from multiple pieces of information is similar to the method of estimating the target of interest using equation (2) described above, so a detailed explanation will be omitted. Finally, if the value indicating the effectiveness of subject blur correction is greater than a predetermined threshold, it is determined that subject blur correction is effective, and if it is equal to or less than the predetermined threshold, it is determined that subject blur correction is ineffective. In this case, in order to prevent the determination result from switching frequently, it is preferable to switch the determination result when a state in which the value remains greater or smaller than the predetermined threshold continues for more than a predetermined time.

Next, in step S1402, the display unit 1302 displays the image data obtained from the development processing unit 103 as well as the judgment result of the image stabilization effect obtained in step S1401. What is displayed is display items such as text and icons, and there are no particular limitations.

Figure 16 shows an example of a display item corresponding to the judgment result of the shake correction effect. Figure 16 (a) shows an icon indicating that image stabilization is working (hereafter referred to as the image stabilization icon), and Figure 16 (b) shows an icon indicating that subject shake correction is working (hereafter referred to as the subject shake correction icon).

For example, if it is determined in step S1401 that subject blur correction is ineffective, the display unit 1402 displays the image stabilization icon in FIG. 16(a) at a specific location on the screen. On the other hand, if it is determined in step S1402 that subject blur correction is effective, the display unit 1402 displays the subject blur correction icon in FIG. 16(b) at a specific location on the screen. By switching the display items in this way, the photographer can accurately distinguish between situations where image stabilization is effective and situations where subject blur correction is ineffective.

The following explains how to display the display items. As image stabilization corrects blurring of the entire background, the image stabilization icon is generally displayed in a fixed position on the screen. However, subject stabilization corrects blurring of a specific subject, and there is not necessarily only one subject on the screen. For this reason, the subject stabilization icon requires a different display method than the normal image stabilization icon.

Figure 17 shows an example of how to display a subject blur correction icon. In Figure 17, three people (1601, 1602, 1603) are present on the screen as subjects. Let us now consider the case where the subject of subject blur correction (blur correction target) is person 1602 in the center of the screen.

In FIG. 17(a), a subject blur correction icon 1604 is displayed near a person 1602 who is the target of subject blur correction. By dynamically changing the display position of the subject blur correction icon in this way, it is possible to inform the photographer that the subject blur of the person 1602 is being corrected.

In FIG. 17(b), a subject shake correction icon 1604 is displayed in a fixed position in the upper left of the screen. A detection frame 1605 for a person 1602 is displayed for the person 1602 who is the target of subject shake correction. By combining the subject shake correction icon and the subject detection frame in this way, it is possible to inform the photographer that the subject shake of the person 1602 has been corrected. Note that the subject detection frame may surround a part of the subject, such as the face, or may surround the entire subject.

In FIG. 17(c), the detection frame for person 1601 is indicated by 1606, the detection frame for person 1602 by 1607, and the detection frame for person 1603 by 1608. In this case, only the detection frame 1607 for person 1602, who is the target of subject shake correction, is displayed with a different design from the other detection frames 1606 and 1608. In this example, detection frame 1607 is distinguished from the other detection frames by drawing a double line. Alternatively, detection frame 1607 alone may be distinguished from the other detection frames by changing its color, shape, or size. By changing the design of the subject detection frame in this way, it is possible to inform the photographer that the subject shake of person 1602 has been corrected.

As described above, this embodiment has the added advantage over the first embodiment of being able to accurately notify the photographer of situations in which the effects of subject shake correction are most pronounced.

(Fourth embodiment)
In this embodiment, an embodiment in which both effective use of the correction stroke and blur correction performance are improved will be described.

Figure 18 is a diagram showing the configuration of an imaging device according to a fourth embodiment of the present invention. A description of the configuration common to the imaging device according to the first embodiment shown in Figure 1 will be omitted.

In FIG. 18, the inertial sensor 112 is a sensor that detects the shake of the imaging device 100, and an angular velocity sensor or the like is used. The output processing unit 113 converts the output signal of the inertial sensor (hereinafter referred to as the angular velocity sensor) 112 into a third shake correction amount. The shake correction amount calculation unit 111 generates an optical shake correction amount and an electronic shake correction amount using the first and second shake correction amounts obtained by the shake correction amount conversion unit 107 and the third shake correction amount obtained by the output processing unit 113 based on the information of the target of interest obtained by the target of interest estimation unit 110. The generated optical shake correction amount is output to a shift mechanism that shifts the correction lens in the optical system 101, which is an optical shake correction means, and/or the image sensor 102 in a direction perpendicular to the optical axis, and optical shake correction is performed. The generated electronic shake correction amount is also output to the electronic shake correction unit 114, and electronic shake correction is performed to control the position of the subject to be near a predetermined position.

Next, the image stabilization operation in the imaging device 100 configured as described above will be described using the flowchart shown in FIG. 19. Steps S201 to S205 and step S208 are the same as in the first embodiment, so only the flow of these steps will be described.

In step S201, the subject detection unit 108 detects a specific subject from the image data input from the development processing unit 103 and outputs subject information.

In step S202, the camera information acquisition unit 109 acquires the camera information necessary to estimate the shooting conditions.

In step S203, the focus estimation unit 110 estimates whether the photographer is focusing on the background or the subject based on the subject information detected in step S201 and the camera information acquired in step S202.

In step S204, the motion vector detection unit 105 detects a motion vector by template matching for the image input from the development processing unit 103 and the image memory 104.

In step S205, the motion separation unit 106 separates the motion vector detected in step S204 into a background vector and a subject vector.

In step S1906, the blur correction amount conversion unit 107 converts the background vector into a first blur correction amount (hereinafter referred to as the camera shake correction amount) and the subject vector into a second blur correction amount (hereinafter referred to as the subject blur correction amount) in the same manner as in step S206 in FIG. 2. At the same time, the output processing unit 113 converts the output signal of the angular velocity sensor 112 into a third blur correction amount. Note that the output processing unit 113 performs integration processing on the output signal of the angular velocity sensor 112 after removing the offset component, and converts the angular velocity information into a displacement amount on the imaging surface. Then, by taking the inverse sign, it is converted into a third blur correction amount that cancels out the blur.

In step S1907, the shake correction amount calculation unit 111 calculates the camera shake correction amount and the subject shake correction amount from the first, second, and third shake correction amounts obtained in step S1906 based on the target obtained in step S203. Furthermore, the camera shake correction amount and the subject shake correction amount are combined to generate a shake correction amount.

The subject shake compensation amount is the second shake compensation amount calculated from the subject vector. The camera shake compensation amount is calculated using the first shake compensation amount calculated from the background vector and the third shake compensation amount calculated from information from the angular velocity sensor 112. This method generally uses frequency division.

The third blur correction amount has a short sample period and is advantageous for detecting high-frequency blur, but detection errors due to offset errors and drift of the angular velocity sensor occur in the low-frequency range. These errors become larger at high temperatures. The first blur correction amount has a long sample period and is disadvantageous for detecting high-frequency blur, but detection errors due to offset errors and drift do not occur. However, in dark places, the signal-to-noise ratio of the image sensor 102 decreases, making errors more likely to occur.

Therefore, filtering is performed using a low-pass filter and a high-pass filter with a cutoff frequency of about 1 Hz. The image stabilization amount is determined by adding the output obtained by applying the first image stabilization amount to the low-pass filter and the output obtained by applying the third image stabilization amount to the high-pass filter. In addition, in high temperatures, the image stabilization amount may be determined by applying the output obtained by applying the first image stabilization amount to the low-pass filter, and in dark places, the image stabilization amount may be determined by applying the output obtained by applying the third image stabilization amount to the high-pass filter.

The method for combining the camera shake compensation amount and the subject shake compensation amount in step S1907 is the same as that in step S207 in FIG. 2, so a detailed description will be omitted. For example, the final combined shake compensation amount H can be obtained using the above-mentioned formula (4).

In step S1909, the combined blur correction amount H calculated in step S1907 is allocated to the correction lens, the shift mechanism of the image sensor, and image cropping, and blur correction control is performed to correct blur that occurs in the image. This blur correction control is performed by outputting a blur correction control signal to the correction lens and the shift mechanism of the image sensor to drive them. At the same time, a control signal is output to the electronic blur correction unit 114, and electronic blur correction is performed by executing control of the image cropping position (electronic blur correction control) that crops out a part of the image signal output from the development processing unit 103 and generates a new image signal. The output image from the image sensor 102 is then sent to image signal recording means and image display means (not shown).

In the final step S208, the imaging device 100 determines whether or not processing has been completed up to the final frame. If processing has been completed up to the final frame, the processing ends, and if processing has not been completed, the processing returns to step S201.

Here, we further explain the blur correction control in step S1909.

Even if it is possible to correct blurring caused by normal camera shake or subject movement, if the photographer's panning is delayed in relation to the rapid movement of the subject, or if the main subject (the subject to be corrected) is temporarily hidden by another subject, the main subject may move significantly on the screen. In such cases, if you try to keep the subject to be corrected (the main subject) in a specified position on the screen, the correction stroke will be insufficient, and it will not be possible to correct the blurring caused by the subject to be corrected.

Therefore, in this embodiment, in order to make it possible to both resolve the insufficient correction stroke and correct subject blur, a correction limit is not simply imposed when the end of the correction stroke is reached. Instead, if the main subject (subject to be corrected) moves significantly, this movement is tolerated to a certain extent, and a filter with a high-pass effect is applied to the correction amount according to the remaining amount of correction stroke, creating a correction signal that focuses on correcting high-frequency blur. In addition, the output gain is made variable.

The method for generating the shake correction control signal is explained below. Note that in the following explanation, for ease of understanding, an example will be given in which shake correction is performed on only one axis, but in reality, there are multiple correction axes, and shake correction is performed by performing similar control on each axis. It is also possible to assign each correction means to each axis. As an example, a method may be used in which all shake correction in the roll direction is performed by the shift mechanism of the image sensor, and shake correction in the pitch and yaw directions is performed by a correction lens.

The method for generating the blur correction control signal will be explained using Figure 20.

In step S2001, it is determined whether a predetermined time has passed since the coefficient Fc and gain β of the high-pass filter to be applied to the shake correction amount were calculated. This is performed by checking whether the difference between the time when the coefficient Fc and gain β were set in step S2012 or S2013 described later and the current time is equal to or greater than a predetermined time. If the predetermined time has passed, proceed to step S2002, and if it is less than the predetermined time, proceed to step S2005.

In step S2002, the time when the coefficient Fc and gain β were set is reset, and then the amount of blur correction to be performed by the correction lens, the shift mechanism of the image sensor 102, and the electronic blur correction unit 114 is calculated (allocation process of the blur correction amount H is performed).

When calculating the amount of correction shared by each mechanism, settings are made so that even if the amount of correction becomes large, the stroke for blur correction is insufficient and correction is not impossible. Specifically, the coefficient Fcc and gain βc of the high-pass filter for the amount of correction by controlling the image cropping position, as well as the coefficient Fcls of the high-pass filter for the third amount of blur correction calculated from information from the angular velocity sensor 112, and the gains βl and βs of the correction lens and shift mechanism are set.

When the blur correction amount H is small (Case 1), the image extraction position is fixed at the center, and blur correction is performed by driving the correction lens and shift mechanism. This is because the optical performance near the center of the screen is superior to that of the periphery, and there is almost no shading, so an image with higher resolution and no brightness difference between the left and right can be obtained.

When the amount of blur correction H increases and approaches the maximum amount that can be corrected by the correction lens and shift mechanism (Case 2), the image extraction position is controlled to reduce the amount of correction by the correction lens and shift mechanism, and to avoid insufficient stroke of blur correction. This enables good blur correction.

When the shake correction amount H is further increased (Case 3), the high-pass filter coefficient Fcc and gain βc are set for the correction amount due to control of the image crop position. Also, the high-pass filter coefficient Fcls and gains βl and βs for the third shake correction amount calculated from the information of the angular velocity sensor 112 are set. This reduces the sensitivity of the shake correction to low-frequency vibration, avoids insufficient shake correction stroke, and enables good shake correction.

In this embodiment, the high-pass filter coefficient Fcc and gain βc for the correction amount due to control of the image crop position, and the high-pass filter coefficient Fcls and gains βl and βs for the third blur correction amount are switched according to each other's signal, but in step S2002, they are first set to optimal values independently. Below, a method for setting the high-pass filter coefficients Fcc, Fcls and gains βc, βl, and βs independently to optimal values in step S2002 will be described.

The third blur correction amount calculated from the correction amount by controlling the image cropping position and the information from the angular velocity sensor 112 is high-pass filtered, and the effect can be changed by changing the coefficients Fcc and Fcls of this high-pass filter. In addition, it is possible to set a limit on the control level by multiplying the output by gains βc, βl, and βs with a maximum value of 100%.

The correction amount Hl due to control of the correction lens, the correction amount Hs due to control of the shift mechanism of the image sensor 102, and the correction amount Hc due to control of the image cropping position of the electronic image stabilization unit 114 are calculated from the correction amount H as follows:

(Case 1)
If H≦(Hlmax+Hsmax)*αls, Hl=H×Hlmax/(Hlmax+Hsmax)
Hs=H×Hsmax/(Hlmax+Hsmax)
(Case 2)
If H>(Hlmax+Hsmax)*αls and H≦(Hlmax+Hsmax)*αls+Hcmax*αc, then Hc=H-(Hlmax+Hsmax)*αls
Hl=Hlmax*αls
Hs=Hsmax*αls
(Case 3)
If H>(Hlmax+Hsmax)*αls+Hcmax*αc, Hc=Hcmax*αc+Hover*Hcmax/Hmax
Hl=Hlmax*αls/2+Hover*Hlmax/Hmax
Hs=Hsmax*αls/2+Hover*Hsmax/Hmax
however,
Hover=H-{(Hlmax+Hsmax)*αls+Hcmax*αc}
Hmax=Hlmax+Hsmax+Hcmax
Here, Hlmax, Hsmax, and Hcmax are the maximum values of the correction amount Hl by controlling the correction lens, the correction amount Hs by controlling the shift mechanism of the image sensor, and the correction amount Hc by controlling the image clipping position, i.e., the correction stroke. αls and αc are ratios to the correction stroke, and are predetermined values used for different cases in allocating the final shake correction amount and setting the correction value.

In (Case 3), the coefficient Fcc of the high-pass filter of the correction amount Hc by controlling the image crop position, the coefficient Fcls of the high-pass filter of the third blur correction amount, the gain βl of the correction amount Hl, the gain βs of the correction amount Hs, and the gain βc of the correction amount Hc are set according to the correction amount Hl, the correction amount Hs, and the correction amount Hc. Specifically, they are set as follows.

When Hover is less than Pe1(%) of [Hmax-{(Hlmax+Hsmax)*αls+Hcmax*αc}], Fcc=Fcc1(Hz)>Fcc0(Hz), βc=βc1(%)<100%
Fcls=Fcls0 (Hz), βl=βs=100%
When Hover is [Hmax-{(Hlmax+Hsmax)*αls+Hcmax*αc}] Pe1(%) or more and less than Pe2(%), Fcc=Fcc2(Hz)>Fcc1(Hz), βc=βc2(%)<βc1(%)
Fcls=Fcls2(Hz)>Fcls1(Hz), βl=βs=βls2(%)<100%
When Hover is equal to or greater than Pe2(%) of [Hmax-{(Hlmax+Hsmax)*αls+Hcmax*αc}], Fcc=Fcc3(Hz)>Fcc2(Hz), βc=βc3(%)<βc2(%)
Fcls=Fcls3(Hz)>Fcls1(Hz), βl=βs=βls3(%)<βls2(%)
where Fcc is the high-pass filter coefficient of the correction amount by controlling the image cropping position, βc is the gain of the correction amount Hc, Fcls is the high-pass filter coefficient of the third blur correction amount, βl and βs are the gains of the correction amount Hl and the correction amount Hs. Note that Fcc0 (Hz) is the minimum value of the high-pass filter coefficient Fcc, and Fcls0 (Hz) is the minimum value of the high-pass filter coefficient Fcls.

Note that the coefficients and gains of the high-pass filter in (Case 1) and (Case 2) are Fcc = Fcc0Hz (minimum value), βc = 100%, Fcls = Fcls0 (Hz), and βl = βs = 100%.

Then, the process proceeds to step S2003, where it is determined whether the image capture device 100 is in a panning state. If a large shake occurring at the start or first half of panning is detected from the output of the angular velocity sensor 112 or the vector detection result of the motion vector detection unit 105, it is determined that the image capture device 100 is in a panning state, and the process proceeds to step S2004. If panning is not detected, the process proceeds to step S2006.

In step S2004, the value of the coefficient Fcc of the high-pass filter for the amount of correction due to control of the image cropping position is increased so that the proportion of the amount of correction for higher frequencies becomes greater. For example, if the value before the change is Fcc0 (Hz), it is changed to Fcc1 (Hz), if it is Fcc1 (Hz), it is changed to Fcc2 (Hz), if it is Fcc2 (Hz), it is changed to Fcc3 (Hz), and if it is Fcc3 (Hz), it is not changed.

At the same time, the value of the gain βc of the correction amount by controlling the image cropping position is lowered. If the value before the change is βc = 100%, it is changed to βc1(%), if it is βc1(%), it is changed to βc2(%), if it is βc2(%), it is changed to βc3(%), and if it is βc3(%), it is not changed.

This reduces the effect of sudden increases in the signal from the angular velocity sensor 112 caused by panning, etc.

In addition, the signal from the angular velocity sensor 112 reflects the movement of the camera, such as panning, and is therefore delayed compared to the motion vector that reflects the movement of the subject. In order to reduce the effect of this delay during high-speed camera movement, the coefficient Fcls for the third correction amount, the gain βl of the correction amount Hl by the correction lens, and the gain βs of the correction amount Hs by the shift mechanism are reset according to FIG. 21 based on the values of the coefficient Fcc and gain βc for the correction amount by control of the image cropping position that are set at that time.

If the coefficient Fcc of the correction amount due to control of the image crop position is Fcc1 (Hz) (the gain in this case is βc1), the coefficient Fcls of the high-pass filter for the third blur correction amount selects either Fcls0 or the higher cutoff frequency set at that time. The gain βl of the correction amount Hl due to control of the correction lens and the gain βs of the correction amount Hs due to control of the shift mechanism select either 100% or the lower set value at that time. The same applies to the other cases.

Then, the process proceeds to step S2005, where a correction control signal is output to the correction lens of the optical system 101, the shift mechanism of the image sensor 102, and the electronic blur correction unit 114 according to the set value, and blur correction is performed.

On the other hand, if the process proceeds from step S2003 to S2006, it is determined whether panning is about to end, and if so, the process proceeds to step S2007, and if not, the process proceeds to step S2010. The determination of whether panning is about to end is made by observing the output of the angular velocity sensor 112 and the vector detection results in chronological order.

If a large shake occurring at the start or first half of panning is detected from the output of the angular velocity sensor 112 or the vector detection results, and then the absolute value of the signal indicating panning decreases to a predetermined value or less, and the slope is less than the predetermined slope, it is determined that panning is about to end. Conversely, if it is greater than the predetermined value, it is determined that panning is not about to end. The predetermined absolute value and the predetermined slope value may be predetermined values, or may be calculated each time from the signal that detects panning. The predetermined absolute value may be a predetermined multiple (less than 1) of the maximum value of the signal during panning, and the predetermined slope value may be a predetermined multiple (less than 1) of the average slope up to the maximum value of the signal during panning.

In step S2007, the coefficient Fcls of the third correction amount is lowered to correct lower frequency shake. For example, if the value of Fcls before the change is Fcls3 (Hz), it is changed to Fcls2 (Hz), if it is Fcls2 (Hz), it is changed to Fcls1 (Hz), and if it is Fcls0 (Hz), it is not changed.

At the same time, the gain βl of the correction amount Hl caused by controlling the correction lens and the gain βs of the correction amount Hs caused by controlling the shift mechanism are increased. If the values of βl and βs before the change are βl3(%) and βs3(%), they are changed to βl2(%) and βs2(%), if they are βl2(%) and βs2(%), they are changed to βl = βs = 100%, and if βl and βs are 100%, no change is made. This makes it possible to correct low-frequency blur even near the end of panning.

Moreover, since detecting a motion vector requires one or more frames of images, the detection takes time. To reduce this effect during slow camera movement, such as tracking a subject with small movements, the coefficient Fcc and gain βc of the correction amount due to control of the image cut-out position are reset according to FIG. 22 based on the values of Fcls for the third correction amount set at that time, the gain βl of the correction amount Hl due to control of the correction lens, and the gain βs of the correction amount Hs due to control of the shift mechanism.

If the coefficient Fcls of the high-pass filter for the third blur correction amount is Fcls2 (Hz) (in this case, the gains βl and βs of the correction lens and shift mechanism are βls2), the correction amount Fcc due to control of the image crop position selects the lower cutoff frequency of Fcc2 or the setting value at that time. In that case, the gain βc selects the higher of βc2 or the setting value at that time. The same applies to other cases.

On the other hand, if it is determined in step S2006 that panning is not about to end, the vector detection results are accumulated in step S2010. Note that the vector accumulation results are reset in step S2004.

Then, the process proceeds to step S2011, where it is determined whether the absolute value of the vector integrated value (electronic shake correction amount) has decreased and is smaller than a predetermined value. If the condition is met, the process proceeds to step S2012, and if not, the process proceeds to step S2013. However, if the predetermined time has not elapsed since the start of the vector integration, the process proceeds to step S2005 without performing the following process. This is because an integrated value that has not yet elapsed the predetermined time since the start of integration is unreliable.

In step S2012, the coefficient Fcc and gain βc of the high-pass filter of the correction amount due to control of the image cut-out position are set based on the coefficient Fcls of the high-pass filter of the third correction amount. At that time, the coefficient Fcls of the third correction amount is set lower than the coefficient Fcc of the correction amount due to control of the image cut-out position.

If the absolute value of this vector integrated value (electronic image stabilization amount) is small and decreasing, it is considered that the camera is able to track the movement of the subject, so a setting that allows image stabilization over a wide frequency range is used.

First, the coefficient Fcls is compared with the coefficient Fcc, and if the coefficient Fcls is not lower than the coefficient Fcc, the following changes are made to the coefficient Fcc so that it is lower. However, if the coefficient Fcls is already Fcls0 (Hz), which is the lowest frequency, no changes are made.

If Fcls ≧ Fcc,
If Fcls=Fcls0, then Fcc=Fcc1
If Fcls=Fcls1, then Fcc=Fcc2
If Fcls = Fcls2, then Fcc = Fcc3
If Fcls = Fcls3, then Fcc = Fcc3
Change it to:

Then, the gain βc is calculated according to the changed coefficient Fcc as follows:
If Fcc=Fcc1, then βc=βc1
If Fcc = Fcc2, then βc = βc2
If Fcc = Fcc3, then βc = βc3
Change it to:

Then, in step S2013, it is determined whether the absolute value of the vector integrated value (electronic shake correction amount) has increased and is greater than a predetermined value. If this condition is met, the process proceeds to step S2014; if not, the process proceeds to step S2005.

In step S2014, the coefficient Fcls of the third correction amount and the gains βl and βs of the correction lens and shift mechanism are set based on the coefficient Fcc of the correction amount due to control of the image cut-out position. At that time, the coefficient Fcls of the third correction amount is set higher than the coefficient Fcc of the correction amount due to control of the image cut-out position.

If the absolute value of this vector integrated value (electronic image stabilization amount) has increased significantly, it is believed that the camera is not able to keep up with the movement of the subject, so the proportion of high frequency correction amount is increased.

First, the coefficient Fcls is compared with the coefficient Fcc, and if the coefficient Fcls is not higher than the coefficient Fcc, the following changes are made so that it is higher. However, if Fcls is already the lowest frequency, Fcls0 (Hz), no changes are made.

If Fcls≦Fcc,
If Fcc=Fcc0 or Fcc=Fcc1, then Fcls=Fcls2
If Fcc=Fcc2, then Fcls=Fcls3
If Fcc=Fcc3, then Fcls=Fcls3
Change it to:

Then, the gains βl and βs are calculated according to the changed Fcls.
If Fcls = Fcls2, then βl = βs = βls2
If Fcls = Fcls3, then βl = βs = βls3
Change it to:

The viewfinder display may be changed to warn the photographer according to the gain βc of the correction amount due to control of the image crop position. An LED or the like in the viewfinder may be lit or blinked to warn that the correction stroke margin is decreasing, or the display around the periphery of the viewfinder may be made semi-transparent, and the semi-transparent parts may be made darker as the correction stroke margin decreases.

It is also possible to improve blur correction performance by observing the phase difference between the output of the angular velocity sensor 112 and the vector detection result in advance. Since a vector requires one or more frames to detect, a detection delay occurs, so the amount of delay is calculated in advance. The calculated and measured values of the camera shake and vector delay are related to each other from the output of the angular velocity sensor. This makes it possible to calculate a vector integrated value (electronic blur correction amount) that takes into account the detection delay. This makes it possible to more accurately set the coefficient Fcc and gain of the correction amount by controlling the image cut-out position using the vector integrated value (electronic blur correction amount), as well as the coefficient Fcls of the third correction amount, and the gains βl and βs of the correction lens and shift mechanism.

The signals with the coefficient Fc and gain β calculated in this way are output to the respective correction mechanisms, and blur correction is performed.

As described above, in this embodiment, the photographer can obtain the shake correction effect desired by controlling the correction ratio between camera shake and subject shake according to the subject of the photographer's attention.

In addition, the coefficient and gain of the high-pass filter for the amount of correction by controlling the image crop position and the third blur correction amount are switched according to each signal. This makes it possible to avoid a lack of stroke for blur correction and to simultaneously prevent the main subject from going out of frame and perform blur correction, which has a large impact on image degradation.

The technology described in this embodiment can be applied to both video and still images. In the case of video, it is possible to continuously switch between camera shake correction and subject shake correction depending on the scene, and it is possible to suppress discomfort such as slight image shifts caused by switching. In the case of still images, it is possible to generate images in which camera shake and subject shake are suppressed in an appropriate balance. It can also be applied to a special type of still image shooting called panning.

Other Embodiments
The present invention can also be realized by a process in which a program for realizing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for realizing one or more of the functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: imaging device, 101: optical system, 102: imaging element, 103: development processing unit, 104: image memory, 105: motion vector detection unit, 106: motion separation unit, 107: blur correction amount conversion unit, 108: subject detection unit, 109: camera information acquisition unit, 110: focus target estimation unit, 111: blur correction amount calculation unit, 601: reliability calculation unit, 602: separation degree calculation unit, 1301: blur correction effect determination unit, 1302: display unit

Claims (36)

a subject detection means for detecting a specific subject from an input image and outputting subject information;
A camera information acquisition means for acquiring camera information necessary for estimating a photographing situation;
an estimation means for estimating an object of interest in an image using the subject information and the camera information;
a first motion detection means for detecting a motion of a background and a motion of a subject in an input image;
a conversion means for converting the background motion and the subject motion detected by the first motion detection means into a first blur compensation amount for compensating for background blur and a second blur compensation amount for compensating for subject blur, respectively;
a correction amount calculation means for combining the first blur correction amount and the second blur correction amount based on the target object estimated by the estimation means to generate a final blur correction amount;
Equipped with
the camera information includes at least any one of information indicating a shutter speed, information indicating an AF area, information indicating a focal length, information indicating a user operation, information indicating a detection result of an inertial sensor, information indicating a distance to a subject, and information indicating a detection result of a user's line of sight. The image stabilization device according to claim 1, characterized in that the first motion detection means detects a motion vector for an input image and separates the motion vector into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject, and the conversion means converts the first motion vector and the second motion vector into the first shake correction amount and the second shake correction amount, respectively. The image stabilization device according to claim 1, further comprising an acquisition means for acquiring a signal from an angular velocity sensor that detects shake of the imaging device, the first motion detection means detecting a motion vector for an input image and separating the motion vector into a first motion vector representing the motion of the background and a second motion vector representing the motion of the subject, and the conversion means converting the shake detected by the angular velocity sensor and the second motion vector into the first shake compensation amount and the second shake compensation amount, respectively. a reliability calculation means for calculating a reliability of the motion vector; and a separation calculation means for calculating a separation between the first motion vector and the second motion vector,
4. The image blur correction device according to claim 2, wherein the correction amount calculation means generates a final blur correction amount by combining the first blur correction amount and the second blur correction amount based on the target object estimated by the estimation means, the reliability of the motion vector calculated by the reliability calculation means, and the separation between the first motion vector and the second motion vector calculated by the separation calculation means. The image stabilization device according to any one of claims 1 to 4, characterized in that the subject information includes at least one of the position and size of the subject, the likeness of the subject to a human, and the movement of the subject. The image stabilization device according to any one of claims 1 to 5, characterized in that the estimation means calculates at least one of a background degree indicating the degree to which the target object is a background and a subject degree indicating the degree to which the target object is a subject, based on at least one of the subject information and the camera information. The image stabilization device according to claim 6, characterized in that the correction amount calculation means generates a final blur correction amount by performing a weighted addition of the first blur correction amount and the second blur correction amount based on at least one of the background degree and the subject degree. The image stabilization device according to claim 7, characterized in that the correction amount calculation means increases the weight of the first image stabilization amount as the background degree increases. The image stabilization device according to claim 7 or 8, characterized in that the correction amount calculation means increases the weight of the second image stabilization amount as the subject degree increases. The image stabilization device according to claim 4, characterized in that the correction amount calculation means generates a final blur correction amount by performing a weighted addition of the first blur correction amount and the second blur correction amount based on the reliability of the motion vector. The image stabilization device according to claim 10, characterized in that the correction amount calculation means increases the weight of the first blur correction amount as the reliability of the first motion vector becomes higher than the reliability of the second motion vector. The image stabilization device according to claim 4, characterized in that the correction amount calculation means generates a final blur correction amount by performing a weighted addition of the first blur correction amount and the second blur correction amount based on the degree of separation between the first motion vector and the second motion vector. The image stabilization device according to claim 12, characterized in that the correction amount calculation means increases the weight of the second blur correction amount as the degree of separation between the first motion vector and the second motion vector increases. The image stabilization device according to any one of claims 2 to 4, characterized in that the first motion detection means separates the motion vector into the first motion vector and a second motion vector based on the subject information and the amount of the motion vector. a determination means for determining an effect of blur correction based on the second blur correction amount;
a display unit that displays a display item corresponding to the determination result by the determination unit together with the input image,
3. The image blur correction device according to claim 2, wherein the determination unit determines an effect of blur correction based on the second blur correction amount, based on at least any of the target, the first blur correction amount and the second blur correction amount, reliability of the object information, reliability of the second motion vector, a degree of separation between the first motion vector and the second motion vector, and an upper limit of the blur correction amount. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the shake correction based on the second shake correction amount is higher when the degree of the object of interest being a subject is a first value than when the degree is a second value lower than the first value. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the stabilization based on the second stabilization amount is higher when the absolute value of the difference between the first stabilization amount and the second stabilization amount is a third value than when the absolute value of the difference is a fourth value smaller than the third value. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the shake correction based on the second shake correction amount is greater when the reliability of the subject information is high than when the reliability is low. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the stabilization based on the second stabilization amount is higher when the reliability of the second motion vector is high than when the reliability is low. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the blur correction based on the second blur correction amount is higher when the degree of separation between the first motion vector and the second motion vector is high than when the degree of separation is low. The image stabilization device according to claim 15, characterized in that the determination means determines that the effect of the shake correction based on the second shake correction amount is higher when the value obtained by subtracting the second shake correction amount from the upper limit of the shake correction amount is a fifth value than when the value is a sixth value smaller than the fifth value. The image stabilization device according to claim 15, characterized in that the determination means determines that the image stabilization based on the second image stabilization amount is effective when a state in which the effect of the image stabilization based on the second image stabilization amount is higher than a predetermined threshold continues for a predetermined time or more. The image stabilization device according to claim 15, characterized in that the display means displays a first display item corresponding to the shake correction based on the first shake correction amount when the determination means determines that the shake correction based on the second shake correction amount is ineffective, and displays a second display item corresponding to the shake correction based on the second shake correction amount when the determination means determines that the shake correction based on the second shake correction amount is effective. The image stabilization device according to claim 23, characterized in that the display means displays the second display item in the vicinity of the subject area that is the target of image stabilization. The image stabilization device according to claim 23, characterized in that the display means notifies the photographer of the subject for which stabilization is to be performed by using a second display item and a frame surrounding the subject. The image stabilization device according to claim 23, characterized in that the display means notifies the photographer of the subject to be subjected to shake correction by surrounding the subject with a frame and changing at least one of the color, shape, and size of the frame of the subject to be subjected to shake correction so as to be different from that of other subjects. a second motion detection unit for detecting a motion of the image stabilization device,
the conversion means further converts the movement of the image stabilization device detected by the second movement detection means into a third image stabilization amount for correcting blur;
the correction amount calculation means calculates the final blur correction amount from the first blur correction amount, the second blur correction amount, and the third blur correction amount based on the target estimated by the estimation means, and
27. The image stabilization device according to claim 1, wherein a filter process is performed on the outputs of the first motion detection means and the second motion detection means, and a characteristic or an output gain of the filter is set in accordance with each output. The image stabilization device according to claim 27, characterized in that, when the second motion detection means determines that the speed of movement of the image stabilization device is equal to or greater than a first predetermined value, the correction amount calculation means performs at least one of increasing the cutoff frequency of a filter for the output of the second motion detection means or decreasing the output gain based on the output of the first motion detection means. The image stabilization device according to claim 27, characterized in that, when the second motion detection means determines that the speed of movement of the image stabilization device is equal to or less than a second predetermined value, the correction amount calculation means performs at least one of lowering the cutoff frequency of a filter for the output of the first motion detection means or increasing the output gain based on the output of the second motion detection means. The image stabilization device according to claim 27, characterized in that the correction amount calculation means makes the cutoff frequency of the filter for the output of the second motion detection means higher than the cutoff frequency of the filter for the output of the first motion detection means when the absolute value of the output of the first motion detection means increases and is equal to or greater than a predetermined value. The image stabilization device according to claim 27, characterized in that the correction amount calculation means, when the absolute value of the output of the first motion detection means decreases and is equal to or smaller than a predetermined value, makes the cutoff frequency of the filter for the output of the second motion detection means lower than the cutoff frequency of the filter for the output of the first motion detection means. a subject detection means for detecting a specific subject from an input image and outputting subject information;
A camera information acquisition means for acquiring camera information necessary for estimating a photographing situation;
a first motion detection means for detecting a motion of a background and a motion of a subject in an input image;
a conversion means for converting the background motion and the subject motion detected by the first motion detection means into a first blur compensation amount for compensating for background blur and a second blur compensation amount for compensating for subject blur, respectively;
a correction amount calculation means for combining the first motion compensation amount and the second motion compensation amount to generate a final motion compensation amount;
and a display control means for controlling a display means to display, together with the input image, a display item according to a subject blur compensation effect and a background blur compensation effect based on at least any one of a difference between the first blur compensation amount and the second blur compensation amount, a difference between an upper limit of the blur compensation amount and the second blur compensation amount, a reliability of the subject information, a reliability of a second motion vector representing a movement of the specific subject , and a degree of separation between the first motion vector and the second motion vector representing a movement of the background. 33. The image stabilization apparatus according to claim 1, further comprising an image capturing unit that captures an image of a subject and generates an input image. a subject detection step of detecting a specific subject from an input image and outputting subject information;
A camera information acquisition step of acquiring camera information necessary for estimating a shooting situation;
an estimation step of estimating an object of interest in the image using the subject information and the camera information;
a motion detection step for detecting background motion and subject motion in an input image;
a conversion step of converting the background motion and the subject motion detected in the motion detection step into a first blur compensation amount for compensating for background blur and a second blur compensation amount for compensating for subject blur, respectively;
a correction amount calculation step of combining the first blur correction amount and the second blur correction amount based on the target object estimated in the estimation step to generate a final blur correction amount;
having
the camera information includes at least any one of information indicating a shutter speed, information indicating an AF area, information indicating a focal length, information indicating a user operation, information indicating a detection result of an inertial sensor, information indicating a distance to a subject, and information indicating a detection result of a user's line of sight. a subject detection step of detecting a specific subject from an input image and outputting subject information;
A camera information acquisition step of acquiring camera information necessary for estimating a shooting situation;
a motion detection step for detecting background motion and subject motion in an input image;
a conversion step of converting the background motion and the subject motion detected in the motion detection step into a first blur compensation amount for compensating for background blur and a second blur compensation amount for compensating for subject blur, respectively;
a correction amount calculation step of combining the first blur correction amount and the second blur correction amount to generate a final blur correction amount;
and a display control step of controlling a display means to display, together with the input image, a display item according to a subject blur compensation effect and a background blur compensation effect based on at least any one of a difference between the first blur compensation amount and the second blur compensation amount, a difference between an upper limit of the blur compensation amount and the second blur compensation amount, a reliability of the subject information , a reliability of a second motion vector representing a movement of the specific subject, and a degree of separation between the first motion vector and the second motion vector representing a movement of the background. A program for causing a computer to function as each of the means of the image stabilization device according to claim 1 or 32.