JP7770855B2 - Control device, imaging device, control method, and program - Google Patents

Description

The present invention relates to a control device, an imaging device, a control method, and a program.

Shake that occurs in the subject of a captured image (subject blur) can be caused by movement of the imaging device, such as camera shake, or by the movement of the subject. To correct subject blur and track the subject intended by the photographer (main subject) to provide a stable composition, it is necessary to detect the movement of the main subject separately from the movement of other subjects such as the background, and correct the movement so that the main subject remains captured in approximately the same position on the screen.

Patent Document 1 discloses a method for easily performing panning by driving a correction unit to correct image blur that occurs in a captured image based on a shake detection signal that indicates shake applied to the imaging device and the motion vector of the subject. Patent Document 2 discloses a method for reducing the flickering sensation that occurs when the AF frame is moved by changing the drawing position of the AF frame and the position of the focus detection area with different characteristics.

Japanese Patent Application Laid-Open No. 2018-36366 Japanese Patent Application Laid-Open No. 2010-204585 Patent No. 3143173

In the method of Patent Document 1, the position of the motion vector detection area is set according to the focus position, so if it becomes difficult to track the correction target, there is a possibility that an inappropriate subject (area) will be selected as the target for image stabilization operation. In the method of Patent Document 2, the area that should actually be controlled may not match the changed area. As a result, the methods of Patent Documents 1 and 2 are unable to capture smooth, high-quality images in scenes with multiple subjects.

The present invention aims to provide a control device, imaging device, control method, and program that can capture smooth, high-quality images in scenes with multiple subjects.

A control device according to one aspect of the present invention includes a subject detection means for detecting main subject information, a motion vector detection means for detecting motion vector information in an image, a blur detection means for detecting blur information using an inertial sensor , and a control means for determining whether or not to make a blur correction control area different from a focus control area based on the main subject information, the blur information, and the motion vector information.

Other objects and features of the present invention are described in the following embodiments.

The present invention provides a control device, imaging device, control method, and program that can capture smooth, high-quality images in scenes containing multiple subjects.

FIG. 2 is a block diagram of an imaging system according to each embodiment. 10 is a flowchart of a main subject detection process in each embodiment. 10 is a flowchart of a photographing process in each embodiment. 4 is a flowchart of a shake correction process according to the first embodiment. FIG. 4 is an explanatory diagram relating to setting of a vector detection frame in the first embodiment. 10 is a flowchart of a shake correction process according to a second embodiment.

Embodiments of the present invention will be described in detail below with reference to the drawings.

(First embodiment)
First, an image capturing apparatus according to a first embodiment of the present invention will be described with reference to Fig. 1. Fig. 1 is a block diagram of an image capturing system 100. The image capturing system 100 includes an image capturing apparatus (camera body) 1 and an interchangeable lens (lens device) 31 that is detachable from the image capturing apparatus 1.

Reference numeral 2 denotes a lens mount for mounting an interchangeable lens 31, 3 a sensor (image sensor) that photoelectrically converts the subject image (optical image) that passes through the imaging optical system, 4 an imaging circuit that generates a predetermined image signal by performing various image processing on the electrical signal photoelectrically converted by sensor 3, 5 an A/D conversion circuit that converts the analog image signal generated by imaging circuit 6 into a digital image signal, 6 a VRAM (memory) such as a buffer memory that temporarily stores the digital image signal output from A/D conversion circuit 7, and 7 a D/A conversion circuit that reads the image signal stored in VRAM 6, converts it into an analog signal, and converts it into an image signal suitable for playback output. Reference numeral 8 denotes an image display device such as an LCD (liquid crystal display device) that displays the image signal output from the D/A conversion circuit 7, and 10 denotes a storage memory made up of a semiconductor memory or the like that stores image data. 9 denotes a compression/expansion circuit having a compression circuit and an expansion circuit. The compression circuit reads the image signal temporarily stored in the VRAM 6 and performs compression and encoding processes on the image data to make it in a format suitable for storage in the storage memory 10. The expansion circuit performs decoding and expansion processes on the image data stored in the storage memory 10 to make it in an optimal format for playback and display, etc.

Reference numeral 11 denotes an AE processing circuit that performs automatic exposure (AE) processing on the output signal from the A/D conversion circuit 7. Reference numeral 12 denotes an AF processing circuit that generates an AF evaluation value and detects defocus amount information for performing automatic focus adjustment (AF) processing on the output signal from the A/D conversion circuit 5. Reference numeral 14 denotes a shake detection sensor that detects movement of the imaging device 1, such as camera shake, and is composed of inertial sensors such as a gyro sensor and an accelerometer, and detects multi-axial shake by using multiple inertial sensors. Reference numeral 13 denotes a shake detection circuit that processes the signal from the shake detection sensor 14.

Reference numeral 15 denotes a CPU (control means) with built-in memory for calculations that controls the imaging device 1; 16 denotes a TG (timing generator) that generates predetermined timing signals; 17 denotes a sensor driver; and 18 denotes an operation switch (operation SW) consisting of various switch groups. Reference numeral 19 denotes an EEPROM, an electrically rewritable read-only memory that pre-stores programs for various controls and data used to perform various operations. Reference numeral 20 denotes a battery; 21 denotes a communication driver for communicating with the interchangeable lens 31; and 22 denotes an LED (display element) that displays warnings and other information. Reference numeral 25 denotes a sensor movement motor for moving the sensor 3 in the horizontal and vertical directions; and 24 denotes a sensor movement control circuit for controlling the movement of the sensor movement motor 25. Reference numeral 27 denotes a motion vector detection circuit that uses output signals from the A/D conversion circuit 7 to detect the motion vector of the subject. Reference numeral 26 denotes a main subject detection circuit that uses output signals from the motion vector detection circuit 27, the A/D conversion circuit 7, and the CPU 15 to perform main subject detection. Reference numeral 28 denotes an image transformation and clipping circuit that performs image processing such as rotating, enlarging, or trimming (cutting) the image. Reference numeral 23 denotes a speaker for issuing various notifications and warnings.

Reference numeral 32 denotes a motion compensation lens, 33 denotes a focus lens, 34 denotes a diaphragm (light intensity adjustment means) that controls the amount of light passing through the imaging optical system consisting of the motion compensation lens, focus lens 33, etc., and 35 denotes a communication driver for communicating with the imaging device 1. 36 denotes a control circuit that controls an diaphragm drive motor (not shown) that drives the diaphragm 34, a focus drive motor (not shown) that drives the focus lens 33, and a motion compensation lens drive motor (not shown) that drives the motion compensation lens 32. 37 denotes an EEPROM, an electrically rewritable read-only memory that pre-stores data used to perform various operations. The interchangeable lens 31 has an imaging optical system including the motion compensation lens 32, focus lens 33, and diaphragm 34, and is equipped with a zoom ring (not shown) that changes the focal length of the interchangeable lens 31, a manual focus ring that adjusts the focus, etc.

Note that the storage memory, which is a storage medium for image data, etc., is not limited to fixed semiconductor memory such as flash memory, or semiconductor memory such as card-shaped or stick-shaped card-type flash memory that is detachable from the device. Various types of storage media, such as hard disks and floppy disks, can also be used.

The operation switches 18 include a main power switch that starts up the imaging device 1 and supplies power, a release switch that starts video shooting (recording) operations, a playback switch that starts playback operations, an exposure compensation amount adjustment dial, an exposure time adjustment dial, and an aperture value adjustment dial. The release switch is a two-stage switch with a first stroke (SW1) that generates a command signal to start AE processing and AF processing, which are performed prior to shooting, and a second stroke (SW2) that generates a command signal to start the actual exposure operation.

Next, the operation of the imaging system 100 will be described. First, a subject light beam (subject image) passing through the interchangeable lens 31, the light intensity of which has been adjusted, is focused on the light-receiving surface of the sensor 3. The subject image is converted into an electrical signal by photoelectric conversion processing by the sensor 3 and output to the imaging circuit 4. The imaging circuit 4 performs various signal processing on the input signal to generate a predetermined image signal. This image signal is output to the A/D conversion circuit 5 and converted into a digital signal (image data), which is then temporarily stored in the VRAM 6. The image data stored in the VRAM 6 is output to the D/A conversion circuit 7 and converted into an analog signal, which is then converted into an image signal in a format suitable for display, and then displayed as an image on the LCD 8. Meanwhile, the image data stored in the VRAM 6 is also output to the compression/expansion circuit 9. After compression processing by the compression circuit in the compression/expansion circuit 9, the image data is converted into image data in a format suitable for storage and stored in the storage memory 10.

Furthermore, for example, when a playback switch (not shown) among the operation switches 18 is operated and turned on, playback operation begins. At this time, the image data stored in compressed form in the storage memory 10 is output to the compression/expansion circuit 9, where it is subjected to decoding and expansion processes, etc., before being output to the VRAM 6 and temporarily stored. Furthermore, this image data is output to the D/A conversion circuit 7 and converted into an analog signal, which is then converted into an image signal in a form suitable for display, and then displayed as an image on the LCD 8.

Meanwhile, TG 16 outputs a predetermined timing signal to CPU 15, imaging circuit 4, and sensor driver 17, and CPU 15 performs various controls in synchronization with this timing signal. Furthermore, imaging circuit 4 receives the timing signal from TG 16 and performs various image processing such as color signal separation in synchronization with this. Furthermore, sensor driver 17 receives the timing signal from TG 16 and drives sensor 3 in synchronization with this.

On the other hand, the image data digitized by the A/D conversion circuit 5 is also output to the AE processing circuit 11, AF processing circuit 12, motion vector detection circuit 27, main subject detection circuit 26, and image transformation and cutting circuit 28, in addition to the VRAM 6.

The AE processing circuit 11 receives the input digital image signal, calculates an AE evaluation value according to the brightness of the subject, and outputs the AE evaluation value to the CPU 15. Based on the AE evaluation value, the CPU 15 calculates the exposure time of the sensor 3 and the aperture value of the diaphragm 34, and sends these to the interchangeable lens 31 via the communication driver 21. In the interchangeable lens 31, the control circuit 36 performs diaphragm drive processing and adjusts the aperture value of the diaphragm 34 so that it is appropriate.

The AF processing circuit (defocus amount detection means) 12 performs image correction on the image signal acquired by the sensor 3, which has imaging pixels for focus adjustment, and performs correlation calculations on the corrected image signal to detect defocus amount information. The CPU 15 determines the amount and direction of drive of the focus lens 33 and sends this to the interchangeable lens 31 via the communication driver 21. The interchangeable lens performs drive processing on the focus lens 33, enabling AF control to achieve a focused state.

The motion vector detection circuit 27 calculates the motion vector of the subject based on the input digital image signal (base image) and the digital image signal of the previous frame (reference image) in accordance with the regions divided by instructions from the CPU 15. That is, it calculates the difference between the base image and the reference image while shifting the reference image by a specified number of pixels in the horizontal and vertical directions, and determines the pixel shift amount that produces the highest degree of correlation (smallest difference amount) as the motion amount of the subject in that region, and the direction of the horizontal and vertical pixel shift at that time as the direction of motion. In this way, the motion vector of the subject within the region between frames is calculated. Details are described in Patent Document 3, etc., so further explanation is omitted here.

The main subject detection circuit 26 acts as a main subject detection means and detects the position of the main subject within the screen (main subject information) as follows. First, the main subject detection circuit 26 determines whether the main subject area has been specified by the photographer. For example, it determines that the main subject area is an area where the AF point has been specified by the photographer, or an area where the photographer has touched the LCD 8 for a predetermined period of time if the LCD 8 is equipped with a touch panel.

The main subject detection circuit 26 also searches the image for features that characterize the face, such as the pupils and eyebrows, based on the output signal from the A/D conversion circuit 5. From this positional relationship, the main subject detection circuit 26 then detects the position of the person's face in the image, and further determines the size and tilt of the face from the positional relationship, such as the spacing between the features that characterize the face.

The main subject detection circuit 26 also detects the position of a person on the screen by searching the image for areas with shapes that characterize a person, such as the head or torso, based on the output signal from the A/D conversion circuit 5, and evaluating their positional relationships. For example, if a shape close to a circle is detected, and a first rectangular shape exists below it, and a second rectangle with a shorter side than the first rectangle exists, and the two are adjacent to each other, it can be determined that a person is present.

The main subject detection circuit 26 also detects clusters of similar color and brightness based on the output signal from the A/D conversion circuit 5, detects their size and position on the screen, and estimates the likelihood that they are the main subject from the results. For example, if a cluster of similar color and brightness with a size equal to or larger than a predetermined size is present near the center of the screen, it is determined to be likely to be the main subject. Then, from among those whose size and position on the screen meet predetermined conditions, the area with the highest likelihood of being the main subject, calculated from the center of gravity coordinates with the center of the screen as the origin and the size of the cluster, is determined to be the main subject area.

Furthermore, main subject detection circuit 26 acquires the distance or defocus amount for each AF point within the screen from the processing result of AF processing circuit 12. Then, using the processing result of motion vector detection circuit 27 and the result of blur detection circuit 13, it detects a subject that is moving in real space and that the photographer is trying to capture within the screen.
Main subject detection circuit 26 also obtains information such as the processing results of AE processing circuit 11, AF point information set by the photographer including that set by touching the LCD screen, shooting mode, shutter speed, and aperture value (F-number) from CPU 15. Main subject detection circuit 26 comprehensively detects multiple main subject areas from the detection results obtained in this manner, and transmits the ranking results to the CPU.

In this embodiment, if the main subject detected by the main subject detection circuit 26 is highly reliable, that main subject is selected as the target for shake correction control. The motion vector of that main subject is then detected by the motion vector detection circuit 27, and the detected motion vector is used to correct any shake that occurs in the main subject. If multiple main subjects are detected, the main subject with the highest rank is selected as the main subject, and shake correction is performed in the same way. Conversely, if there are multiple subjects with the same rank and the main subject cannot be determined, or if the main subject is temporarily not detected, the main subject detected by the main subject detection circuit 26 is not selected as the target for shake correction control, and instead another subject, such as the background, is selected as the target for shake correction.

Next, the main subject detection process in this embodiment will be described with reference to FIG. 2. FIG. 2 is a flowchart of the main subject detection process. Each step in FIG. 2 is executed primarily by the main subject detection circuit 26 based on instructions from the CPU 15. In this process, the earlier a main subject is detected, the higher the priority of that main subject.

When processing begins, all main subject detection flags are first turned off. Then, in step S201, the main subject detection circuit 26 determines whether a main subject area (AF focus point) has been specified by the photographer. This is done by examining the AF point information sent from the CPU 15 and checking whether an AF point has been specified by the photographer via a menu or touch on the LCD screen. If a main subject area has been specified, the process proceeds to step S202. In step S202, the main subject detection circuit 26 determines the AF point specified by operating the operation switch 18 or the main subject area selected by the photographer by touching the LCD screen, as well as an area nearby where a similar subject exists, as the main subject area.

On the other hand, if a main subject area has not been specified, the process proceeds to step S203. In step S203, the main subject detection circuit 26 determines whether a face has been detected. If a face has been detected, the face detection flag is turned on, and then in step S221 it is determined whether the detected face can be considered to be the same subject as the main subject up to the previous frame. If it can be considered to be the same subject, the process proceeds to step S204, and the detected face area is designated as the main subject area. On the other hand, if it cannot be considered to be the same subject, the process proceeds to step S205. Note that the determination of whether it can be considered to be the same subject can be made using the function of the main subject detection circuit 26 to track subjects that are presumed to be the same.

In step S205, the main subject detection circuit 26 determines whether a person has been detected. If a person has been detected, the person detection flag is turned on, and then in step S222 it is determined whether the detected person can be considered the same subject as the person that was the main subject up until the previous frame. If the person can be considered the same subject, the process proceeds to step S206, where the detected person area is set as the main subject area. On the other hand, if the person cannot be considered the same subject, the process proceeds to step S207.

In step S207, the main subject detection circuit 26 determines whether a moving object (a subject moving in real space that the photographer is trying to capture within the frame) is present (whether a moving object has been detected). If a moving object is present, the moving object detection flag is turned on, and then in step S222 it is determined whether the detected moving object can be considered the same subject as the main subject up to the previous frame. If it can be considered the same subject, proceed to step S208, and the moving object is designated as the main subject area. On the other hand, if it cannot be considered the same subject, proceed to step S209. The determination of whether a moving object is present is made using the output signals of the shake detection circuit 13, motion vector detection circuit 27, and AF processing circuit 12.

If the output signal from the shake detection circuit 13 is small (if all detection values for multiple axes are below a predetermined value), i.e., if the photographer is not intentionally moving the imaging device 1, the motion vector detection circuit 27 determines whether there is an area where the amount of movement of the motion vector detected is greater than or equal to a predetermined value. If there is an area where the amount of movement of the motion vector is greater than or equal to the predetermined value, that area is determined to be the main subject area.

On the other hand, if the output signal from the shake detection circuit 13 is large (if any of the detection values for multiple axes is equal to or greater than a predetermined value), i.e., if the photographer is intentionally moving the imaging device 1, the area exhibiting movement equivalent to this intentional movement is determined to be the main subject area. In other words, if there is an area where the amount of movement of the motion vector detected by the motion vector detection circuit 27 is equal to or less than a predetermined value, that area is determined to be the main subject area.

If a moving object cannot be detected using processing using the output signal of the motion vector detection circuit 27, a determination is made as to whether there is a moving object moving in the distance direction (optical axis direction). This can be done by checking whether there are any AF points whose distance or defocus amount obtained from the AF processing circuit 12 changes in the same direction over time (for example, AF points whose distance decreases over five consecutive frames). The changes in distance or defocus amount for each frame are checked for all AF points obtained from the AF processing circuit 12, and AF points whose distance changes in the same direction by a specified amount over a specified number of frames or more are extracted. These AF points are considered to be areas containing moving objects, and are merged if adjacent. If there are multiple AF point areas whose distance changes in the same direction by a specified amount over a specified number of frames or more, the measurement base area closest to the center of the screen is selected.

In step S209, the main subject detection circuit 26 determines whether there is any area that can be considered a main subject area with a high degree of main subject likelihood among the clusters of similar color and brightness. If there is an area that can be considered a main subject area, the main subject detection flag is turned on, and then in step S224 it is determined whether the main subject in the detected main subject area can be considered the same subject as the main subject up until the previous frame. If it can be considered the same subject, proceed to step S210 and designate that area as the main subject area. The degree of main subject likelihood is determined based on the position and size on the screen of clusters of similar color and brightness. Detected clusters that do not touch two of the four sides of the screen are selected, and of these, those that are a predetermined size or larger are considered to be clusters with a high degree of main subject likelihood. If there are multiple such clusters, the one with the center of gravity closest to the center of the screen is selected.

If the subject cannot be considered to be the same as the main subject up to the previous frame, proceed to step S225. In step S225, a detection flag is checked to determine whether a main subject candidate has been detected in the processing up to that point. If any detection flag is on, the main subject detection circuit 26 determines that a main subject candidate has been detected, and proceeds to step S226.

In step S226, the main subject detection circuit 26 selects the detection area with the highest priority among those whose detection flags are on, and designates that area as the main subject area. If a face has been detected, the face detection area is selected; if a face has not been detected but a person has been detected, the person detection area is selected; if a moving object has been detected without a face or person being detected, the moving object area is selected; and if neither a face nor person has been detected, the detection area determined to have a high likelihood of being the main subject is selected as the main subject area.

If the detection flag is not on, no main subject candidate has been detected in the processing up to that point, and processing proceeds to step S211. In step S211, the main subject detection circuit 26 checks the processing results of the AF processing circuit 12 to see if there is a subject with a high proportion of AF points within the screen that are AF-enabled and at a different distance. If the condition is met, processing proceeds to step S212, and the AF point that shows the closest AF result among the multiple AF points is determined to be the main subject area. In this case, since the entire screen is not the subject, as in landscape photography, and it is likely to be a commemorative photo with the landscape as the background, the closest subject is determined to be the main subject area. On the other hand, if the condition is not met, in step S213, the area to perform AF is determined based on the AE processing results, shooting mode, shutter speed, aperture value, and flash ON/OFF information obtained from the CPU 15.

First, based on the shooting mode, the area to perform AF is determined as shown in Table 1 below.

If the above conditions are not met, the main subject area is determined as shown in Table 2 below.

If multiple faces, people, or moving objects are detected, the main subject area is determined based on their detected position and size (faces) or their detected position (people/moving objects). Main subjects with different priorities may also be detected, and the main subject area is determined according to the priorities. Initially, the main subject area is determined in this way, but from the next frame onwards, priority is given to the same subject.

The main subject detection circuit 26 has the function of tracking a subject that is estimated to be the same as the subject detected as the main subject in the previous frame. With this function, if the subject detected as the main subject in the above procedure is estimated to be the same as the main subject up to the previous frame, that subject is set as the main subject with the highest priority. Even if there is a main subject that was detected earlier in the above procedure, the main subject that is estimated to be the same is given priority. Therefore, blur correction is performed on this main subject that is estimated to be the same. Note that this function is only performed when an arbitrary AF point is specified (steps S204, S206, S208, S210 in Figure 2). If the closest subject is set as the main subject area (steps S212, S213 in Figure 2), no determination is made as to whether the subject is estimated to be the same as the subject.

Furthermore, if the photographer explicitly specifies an AF point using a menu or the like to specify a main subject, the subject at the specified AF point will be considered the main subject. Therefore, a new main subject is determined to have been detected when the photographer explicitly specifies a main subject (step S202), or when a main subject that is no longer estimated to be the same subject and a higher-ranked main subject is detected at the same time (step S212). Here, a case where a main subject is no longer estimated to be the same subject and a higher-ranked main subject is detected at the same time would occur if the main subject had moved significantly up until then and its position on the screen changed significantly without the photographer intentionally tracking that subject.

The image transformation and cropping circuit 28 then uses the output signals from the main subject detection circuit 26 and motion vector detection circuit 27 to correct the changes in accordance with the up/down/left/right movement of the main subject on the image calculated by the CPU 15 and information about the rotation of the imaging device 1. To achieve this, image processing such as transformations such as image rotation and cropping of portions of the image is performed. For example, the position of the face of a person detected as the main subject on the screen may change significantly due to the person's movement or the photographer's camera shake, or the main subject may move diagonally. If such shaking or movement of the main subject in the horizontal, vertical, or diagonal directions occurs between frames of a video, the main subject may not be in the position on the screen desired by the photographer, or unnatural movement may occur. If such phenomena occur frequently between frames, recorded video images will be very difficult to watch.

The motion vector representing the horizontal and vertical movement of the main subject between frames is detected by the motion vector detection circuit 27, and the CPU 15 calculates information for correcting the image from the motion vector. The image is then transformed and corrected by the image transformation and cutout circuit 28 according to the calculated amount of correction, and the generated image is recorded in a specified area of the VRAM 6.

Next, the shooting process of the imaging device 1 will be described with reference to Figure 3. Figure 3 is a flowchart of the shooting process. Each step in Figure 3 is executed by each section of the imaging device 1 based mainly on commands from the CPU 15. When the main power switch of the imaging device 1 is on and the operating mode of the imaging device 1 is in shooting (recording) mode, the shooting process sequence is executed.

First, in step S301, the CPU 15 performs initialization processing, such as initializing variables used in processing and moving drive members to their initial positions, and then determines whether an interchangeable lens 31 is attached. If an interchangeable lens 31 is attached, the CPU 15 acquires information about the image stabilization lens 32, focus lens 33, aperture 34, focal length, etc. On the other hand, if an interchangeable lens 31 is not attached, the process proceeds to step S302 without acquiring interchangeable lens information.

In step S302, the CPU 15 displays the image that has passed through the interchangeable lens 31 and formed on the sensor 3 as an image on the LCD. That is, the subject image formed on the sensor 3 is photoelectrically converted by the sensor 3 into an electrical signal, which is then output to the imaging circuit 4. The input signal undergoes various signal processing to generate a predetermined image signal, which is then output to the A/D conversion circuit 5, converted into a digital signal (image data), and temporarily stored in the VRAM 6. The image data stored in the VRAM 6 is output to the D/A conversion circuit 7, converted into an analog signal, and converted into an image signal in a form suitable for display, which is then displayed as an image on the LCD 8. Note that if the video recording mode is selected and the image stabilization setting is ON, an image that has undergone image stabilization processing (image stabilization control) may also be displayed on the LCD 8.

Next, in step S303, the CPU 15 determines whether the camera is set to video recording mode or still image capture mode. If the camera is set to video recording mode, the process proceeds to step S304. On the other hand, if the camera is set to still image capture mode, the process proceeds to step S321.

In step S321, the CPU 15 checks the state of the release switch. If the photographer operates the release switch and the CPU 15 checks that SW1 (the first stroke of the release switch) is in the ON state, the process proceeds to step S322. If the photographer has set the camera to perform shake correction processing, the CPU 15 performs shake correction processing. This processing can be performed using the shake correction processing used in conventional still image shooting, as described in Patent Document 2, for example, and therefore a detailed description of the processing will be omitted.

Next, in step S323, the CPU 15 executes AF and AE processing, drives the focus lens 33 to the in-focus position, and determines the aperture value, exposure time, etc. for still image capture. Next, in step S324, the CPU 15 superimposes the AE processing results (high brightness/low brightness warnings, etc.) and AF processing results (AF success/failure, etc.) on the image formed on the sensor 3 and displays them as an image on the LCD. The AE and AF processing results may also be notified to the photographer by flashing the LED 22 or by emitting a focus or out-of-focus sound from the speaker.

Next, in step S325, the CPU 15 checks SW2 (the first stroke of the release switch). If SW2 is on, the process proceeds to step S326, where the CPU 15 executes exposure processing. If the photographer has also set image stabilization during exposure processing, the CPU 15 executes image stabilization processing in the same manner as in step S322. After exposure processing is complete, the process proceeds to step S310.

On the other hand, if video recording mode was selected in step S303, the process proceeds to step S304, where the CPU 15 executes AF and AE processing. The CPU 15 drives the focus lens 33 to the in-focus position, determines the aperture value and exposure time, and controls the drive of the aperture 34 and the exposure time (accumulation time) of the sensor 3.

Next, in step S305, the CPU 15 determines whether the photographer has set up shake correction processing. If shake correction processing has been set up (IS is on), the CPU 15 performs shake correction processing in step S306. On the other hand, if shake correction processing has not been set up (IS is off), the process proceeds to step S307. Details of the processing in step S306 will be described later.

In step S307, the CPU 15 displays the image on the LCD 8 with the AE processing results (high brightness/low brightness warning, etc.) and AF processing results (AF success/failure, etc.) superimposed. If image stabilization processing has been set, this image is the image that has been subjected to image stabilization processing created in step S306; if image stabilization processing has not been set, this is the image that was read out in step S302.

Next, in step S308, the CPU 15 determines whether a video recording instruction has been issued by the photographer. If a video recording instruction has been issued, the process proceeds to step S309, where the CPU 15 executes video recording processing, and then proceeds to step S310. This video recording processing is performed by recording the image with the shake correction processing created in the processing of step S306, if shake correction processing has been set. On the other hand, if shake correction processing has not been set, the image to be displayed in step S307 (the image read out in step S302) is recorded.

If a video recording instruction has not been issued, proceed to step S310. In step S310, the CPU 15 determines whether the state of the main power switch or playback switch has changed, or whether the lens has been replaced. If the state of any switch has changed, or the lens has been replaced, the process ends. Otherwise, return to step S302.

The shake correction process of step S306 will now be described with reference to Figures 4 and 5. Figure 4 is a flowchart of the shake correction process in this embodiment. Figure 5 is an explanatory diagram related to setting the vector detection frame. The steps in Figure 4 are mainly executed by the CPU 15, or by various components such as the main subject detection circuit 26 and motion vector detection circuit 27 based on instructions from the CPU 15.

First, in step S401, main subject detection circuit 26 determines whether a subject presumed to be the same as the subject previously detected as the main subject has been detected. If a subject presumed to be the same as the subject previously detected as the main subject has not been detected, processing proceeds to step S402. On the other hand, if a subject presumed to be the same as the subject detected as the main subject in the previous frame has been detected, that is, if it is determined that a tracking state is occurring, processing proceeds to step S421. This determination is made using the function of main subject detection circuit 26 to track a subject presumed to be the same as the subject detected as the main subject in the previous frame. That is, if a main subject area is determined in steps S204, S206, S208, or S210 of FIG. 2, it is determined that a subject presumed to be the same as the subject previously detected as the main subject has been detected. Otherwise, that is, if a main subject area is determined in steps S202, S226, S212, or S213 of FIG. 2, it is determined that no main subject has been detected.

In step S402, the main subject detection circuit 26 determines whether a new main subject has been detected. If a main subject area has been determined in steps S202 and S226 of FIG. 2, it determines that a new main subject has been detected, and proceeds to step S403. On the other hand, if a main subject area has been determined in steps S212 and S213 of FIG. 2, it determines that a new main subject has not been detected, and proceeds to step S422.

In step S403, the main subject detection circuit 26 determines whether the subject (focus control area) targeted for focus control has been designated by the photographer. If the main subject area was determined in step S202 of FIG. 2, it is determined that the photographer has designated it, and the process proceeds to step S421. If the main subject area was determined in any other step of FIG. 2, it is determined that the photographer has not designated it, and the process proceeds to step S404.

In step S404, the main subject detection circuit 26 determines whether the photographer is tracking the subject by panning or the like. If the subject is being tracked, proceed to step S421. On the other hand, if the subject is not being tracked, proceed to step S405. The determination of whether the subject is being tracked is made using the output signal of the shake detection circuit 13 and the output signal of the motion vector detection circuit 27. That is, if the output signal of the shake detection circuit 13 is large and the movement of the main subject on the sensor 3 calculated by the motion vector detection circuit 27 is small, it is determined that the photographer is moving the imaging device 1 so that the desired main subject does not move on the sensor 3 and is tracking the subject.

In step S405, the main subject detection circuit 26 uses the face recognition function to determine whether the newly detected person is the same person. If it is determined that the person is the same person, the process proceeds to step S421. On the other hand, if it is determined that the person is not the same person, the process proceeds to step S406.

In step S406, the main subject detection circuit 26 determines whether multiple subjects exist and the main subject cannot be determined. If the main subject cannot be determined, proceed to step S422. On the other hand, if the main subject can be determined, proceed to step S407. This determination is made using the size and number of subject areas detected by the main subject detection circuit 26. The main subject detection circuit 26 detects subjects such as human faces, people themselves, and moving objects, and may detect multiple subjects. If multiple subjects are detected with the same priority, the main subject area is determined based on the detected position and size (faces) or the detected position (people/moving objects). In this case, if there is no clear difference in the detected position and size, the main subject cannot be clearly determined from the multiple subjects.

Therefore, if the difference in size of the detected faces is smaller than a predetermined ratio, i.e., if the size of the second-place face relative to the size of the first-place face is equal to or greater than a predetermined ratio (Cr1) (if the ratio of the sizes of multiple subjects is equal to or greater than a first predetermined ratio), the main subject cannot be clearly determined. In this case, it is determined that multiple subjects exist and the main subject cannot be determined.

Similarly, if the size of the second-placed face is equal to or greater than a predetermined ratio (Cr2: Cr2 < Cr1) and the difference between the sensor center position of the first-placed face and the position of the second-placed face is smaller than a predetermined value (Cd1), the main subject cannot be clearly determined. In other words, if the ratio of the sizes of multiple subjects is equal to or greater than a second predetermined ratio and the difference in the positions of the multiple subjects is smaller than a first predetermined value, it is determined that the main subject cannot be clearly determined.

Similarly, if the difference between the position of the first detected subject (person or moving object) with the sensor center as the origin and the position of the second detected subject is smaller than a predetermined value (Cd2) (if the difference in position between multiple subjects is smaller than a second predetermined value), the main subject cannot be clearly determined. In this case, too, it is determined that multiple subjects exist and the main subject cannot be determined.

In step S407, the motion vector detection circuit 27 determines whether a subject with different movement on the sensor 3 has intersected with the previous main subject (old subject). If the subject has intersected, proceed to step S422. On the other hand, if the subject has not intersected, proceed to step S408. This determination is made using the detection results of the motion vector detection circuit 27. The motion vector detection circuit 27 detects motion vectors, i.e., the amount of movement on the sensor 3, for the multiple subjects detected by the main subject detection circuit 26. If multiple subjects are detected, motion vectors are detected for the multiple subjects in order of rank.

Therefore, the motion vectors and positions of each of the multiple subjects from several frames before to the current frame are recorded. If the difference in the positions of two of the multiple subjects is smaller than a predetermined value (Cd3) and the difference in the motion vectors of those subjects between the frames immediately before is greater than a predetermined value (Cv1), it is determined that a subject with different movement on sensor 3 has intersected with the previous main subject. In other words, if the difference in the positions of the multiple subjects is smaller than a third predetermined value and the difference in motion vectors is greater than a first predetermined vector difference, it is determined that the subjects have intersected.

In step S408, the main subject detection circuit 26 determines whether the position of the detected main subject on sensor 3 is at an edge position (outside a certain range from the center of sensor 3). If the position of the main subject is outside the certain range from the center of sensor 3, proceed to step S422. On the other hand, if the position of the main subject is within the certain range from the center of sensor 3, proceed to step S409. This determination is made if the position of the main subject on sensor 3 is away from the center position of sensor 3 by more than a predetermined value (Cp1), it is determined to be outside the certain range, and proceed to step S422. In other words, if the position of the most recently detected main subject is outside a first predetermined range from the center position, proceed to step S422.

In step S409, main subject detection circuit 26 determines whether the previous main subject is temporarily no longer detected. If a main subject is not detected near the previous detection position of the main subject, and main subject detection circuit 26 detects a new main subject, but that new subject is now further away and the size of the main subject area is significantly different, it is determined that the main subject is temporarily no longer detected, and processing proceeds to step S423. On the other hand, if a main subject is detected, processing proceeds to step S410. This determination assumes that the previous main subject is temporarily no longer detected because it has been temporarily hidden by another subject.

The main subject detection circuit 26 may fail to detect a face if the face is hidden by another subject. As a countermeasure, even if the main subject detection circuit 26 does not detect a face, the CPU 15 will determine the main subject by assuming that the face is present at the previous face detection position for several frames. If a new face (someone else's face) is detected during this process while the previous main subject is still hidden, the main subject detection circuit 26 will notify the CPU 15 that the other person's face is the main subject.

Therefore, when a new main subject is notified, the size and detection position of the face detected up to that point are compared with those of the newly detected face. If the face size ratio (Rf = previous face size / newly detected face size) is outside a predetermined range (Crf1 < Rf < Crf2), it is determined that the face has temporarily stopped being detected. Alternatively, if the difference Df between the detection positions of the previously detected face and the newly detected face is greater than a predetermined value (Cd4), it is determined that the face has temporarily stopped being detected. In other words, if the size ratio of multiple subjects is less than or equal to the first comparison value or greater than the second comparison value, or if the difference in position between multiple subjects is greater than a fourth predetermined value, it is determined that the main subject has temporarily stopped being detected, and the process proceeds to step S423. The same applies to people, etc. Note that after a provisional period of several frames has elapsed, it is determined that this condition is not met, and the process proceeds to step S410.

In step S410, the main subject detection circuit 26 determines whether all of the detected multiple subjects are within a predetermined defocus range (whether the difference in defocus amount between the multiple subjects is less than a predetermined value). If all of the multiple subjects are within the predetermined defocus range, the process proceeds to step S423. On the other hand, if all of the multiple subjects are not within the predetermined defocus range (if the multiple subjects are within a first predetermined defocus range or a first distance range), the process proceeds to step S423. This determination is made using the distance or defocus amount obtained from the AF processing circuit 12.

For multiple subjects detected by the main subject detection circuit 26, the distance or defocus amount of the area where the subject is located is obtained from the AF processing circuit 12. If all main subject areas are found to be within the predetermined range using this value, they are determined to be within the predetermined defocus range, and processing proceeds to step S423. On the other hand, if any area is not within the predetermined range, processing proceeds to step S421. This can be done by finding the defocus amount (or distance) of the nearest and farthest areas and comparing the difference with the predetermined range (distance range). The predetermined range is determined based on depth.

In step S421, the CPU 15 sets a vector detection frame in the motion vector detection circuit 27, using the main subject region detected by the main subject detection circuit 26 as the target region for blur correction (blur correction control region). For example, as shown in FIG. 5(A), the entire image is divided into 9x7 regions, and the regions that partially overlap with the region included in the main subject region (the gray areas in FIG. 5(A)) are set as detection frames for detecting vectors to correct subject blur. Vectors are then detected in the vector detection frames of each region. The vectors are then clustered to separate the vectors in the vector detection frames within the main subject frame into those caused by subject movement and those that are not, and the amount of subject blur that has occurred is detected by detecting motion vectors caused by subject movement.

In step S422, the background is set as the target area for blur correction (blur correction control area), and a vector detection frame in the motion vector detection circuit 27 is set in the area excluding the main subject area detected by the main subject detection circuit 26. For example, as shown in Figure 5(A), the entire image is divided into 9x7 areas, and the area excluding the gray parts is set as the detection frame for detecting vectors to correct blur. Vectors are then detected in the vector detection frame of each area. Vector clustering is then performed to separate the vectors in the vector detection frame excluding the main subject frame into background blur caused by camera shake and other vectors, and the amount of background blur caused by camera shake is detected by detecting the background motion vectors.

In step S423, a vector detection frame is set in the motion vector detection circuit 27, with the area where the previous main subject (old main subject) was present as the target area for blur correction. However, because the main subject detection circuit 26 is temporarily not detecting the previous main subject or has detected another subject as the main subject, it is also necessary to detect the motion vectors of the background and other subjects. In this case, for example, multiple subjects, multiple subjects and the background, the previously detected subject (old subject), or the old subject and the background are set as the blur correction control area. Therefore, a vector detection frame in the motion vector detection circuit 27 is set to cover the entire screen. Three areas are then set: an area including the previous subject (old subject) obtained from the main subject detection circuit 26, an area including the newly detected subject (new subject), and an area excluding these, and a vector detection frame in the motion vector detection circuit 27 is set for each of them.

For example, as shown in Figure 5(B), the entire image is divided into 9x7 regions. Three regions are defined: a region that partially overlaps with the region included in the old main subject region (the gray region in Figure 5(B)), a region that partially overlaps with the region included in the new main subject region (the white region in Figure 5(B)), and a region excluding both. These serve as detection frames for detecting vectors for blur correction. Next, vector detection frames are detected for each region. Vector clustering is then performed to separate the vectors in the vector detection frames within the previous and new main subject region frames into those caused by subject movement and those not, and motion vectors caused by subject movement are detected. This detects the amount of subject blur that has occurred. Furthermore, the vectors in the vector detection frames excluding the two subject frames are separated into those caused by background blur due to camera shake and those not, and the amount of background blur that has occurred is detected.

Thereafter, the amount of blur to be corrected is calculated from the three amounts of blur thus obtained in the following manner.
(1) If the difference between the three obtained amounts of blur is within a predetermined value (Cb1), the average of the three amounts of blur is calculated and set as the amount of blur to be corrected.
(2) If the difference between the amount of blur of the main subject and the amount of blur of the background is within a predetermined value (Cb1), the average of the two amounts of blur is calculated and set as the amount of blur to be corrected.
(3) If the difference between the amount of blur of the main subject up to that point and the amount of blur of the new main subject is within a predetermined value (Cb1), the average of the two amounts of blur is calculated and set as the amount of blur to be corrected.
(4) If the amount of blur of the main subject up to that point is greater than a predetermined value (Cs1), the amount of blur of the main subject up to that point is set as the amount of blur to be corrected.
(5) If the amount of blur in the background is greater than a predetermined value (Cs1), the amount of blur in the background is set as the amount of blur to be corrected.
(6) In all other cases, the amount of blur of the main subject up to that point is set as the amount of blur to be corrected.

Then, in step S424, the CPU 15 performs shake correction processing. First, the CPU 15 calculates the correction amount (shake correction amount) for correcting the shake occurring in the captured image from the amount of shake to be corrected calculated in steps S421 to S423. Then, the image transformation and cropping circuit 28 is used to crop the image based on the calculated correction amount, thereby generating a shake-corrected image.

In this embodiment, main subject detection circuit 26 functions as a main subject detection means that detects the position of the main subject (main subject information). Motion vector detection circuit 27 functions as a motion vector detection means that detects the movement of the subject (motion vector information). Shake detection sensor 14 and shake detection circuit 13 function as a shake detection means that detects the movement of the imaging device 1 (shake information). CPU 15 functions as a calculation means that calculates the amount of correction used to correct shake that occurs in the subject of the captured image. Operation switch 18, touch panel-equipped LCD 8, and CPU 15 function as a means for specifying the target area for focus control (focus control area).

The CPU 15 also functions as a determination means (control means) that determines whether or not to make the shake correction control area different from the focus control area based on main subject information, blur information, and motion vector information. The CPU 15 also has the function of determining the target area for shake correction control based on the determination result of the determination means. This determines whether or not the subject (area on the screen) to be targeted for shake correction control should be a separate subject from the target for focus control based on the subject detection results, the defocus amount of each area, and the subject movement, and determines the target for shake correction control based on the result.

If the previous subject was being tracked, that subject will continue to be the control target. When a new main subject is detected, the subject that is the target of focus control will be the same subject that is the target of shake correction control if the subject that is the target of focus control has been specified by the photographer. Alternatively, the output signal from the shake detection means is large and the movement of the subject calculated from the motion vector detection means and shake detection means is small (i.e., when it can be determined that the photographer is tracking the subject by moving the imaging device so that it does not move on the sensor). In other words, this is the case when the output signal from the shake detection means is larger than the first output signal and the motion vector information is smaller than the first motion vector. Alternatively, this is the case when a newly detected main subject, such as a face recognition function, can be considered the same as the previous main subject. This causes the same subject to be the target of shake correction control as the target of focus control, resulting in a good image that is in focus and has little subject blur.

When a new main subject is detected, the subject of image stabilization control is not the same as the subject of focus control, for example, in the following cases (1) to (5).
(1) When it is not possible to determine the main subject from multiple subjects.
(2) When it can be determined that a different motion has intersected from the motion of the subject calculated by the motion vector detection means and the blur detection means.
(3) When the main subject is detected near the edge of the screen.
(4) When the main subject area detection means temporarily fails to detect the subject.
(5) When the difference in defocus amount between the detected multiple subjects is equal to or less than a predetermined value.

Since focus control is always required, a single control target must be determined. However, with image stabilization control, it is possible to perform image stabilization processing (camera stabilization) on the background without determining a specific subject to control. Therefore, if the subject detected as the main subject is not one for which the photographer wants to correct subject shake, the subject to image stabilization control can be set to a different subject from the subject to focus control. This makes it possible to provide smooth, high-quality images even in scenes with multiple mixed subjects.

Second Embodiment
Next, a second embodiment of the present invention will be described with reference to Fig. 6. This embodiment is the same as the first embodiment except for the processing in step S423 in Fig. 4. Fig. 6 is a flowchart of the image stabilization processing in this embodiment. The steps in Fig. 6 are mainly executed by the CPU 15 or by various components such as the main subject detection circuit 26 based on instructions from the CPU 15.

First, in step S601, the CPU 15 determines whether the old and new main subjects are within the depth of field for the aperture value used during video recording. If the old and new main subjects are within the depth of field, the process proceeds to step S608, where the area where the old main subject is present is designated as the target area for image stabilization control. On the other hand, if the old and new main subjects are outside the depth of field, the process proceeds to step S602. That is, the CPU 15 obtains from the AF processing circuit 12 the defocus amounts for the area where the old main subject, detected by the main subject detection circuit 26, is present and the area where the new main subject, detected by the main subject detection circuit 26, is present. The CPU 15 then determines whether the difference between these values is within the depth of field. The depth of field is calculated by multiplying the aperture value (aperture value information) obtained by communicating with the interchangeable lens 31 via the communication driver 21 by a predetermined allowable circle of confusion.

In step S602, the CPU 15 determines whether a focusing operation (driving the focus lens 33) on a new main subject has started. If a focusing operation on a new main subject has started, the process proceeds to step S603. On the other hand, if a focusing operation on a new main subject has not started, the process proceeds to step S608.

In step S603, the CPU 15 communicates with the interchangeable lens 31 via the communication driver 21 and determines, from the drive position of the focus lens 33 obtained, whether the area in which the old main subject existed is still within the depth of field. If the area in which the old main subject existed is within the depth of field, the process proceeds to step S608. On the other hand, if the area in which the old main subject existed is outside the depth of field, the process proceeds to step S604.

In step S604, the CPU 15 determines whether the focusing operation on the new main subject has finished. If the focusing operation on the new main subject has finished, the process proceeds to step S605. On the other hand, if the focusing operation on the new main subject has not finished, the process proceeds to step S608. If the new main subject is not yet in focus, even if the area where the new main subject exists is set as the target area for image stabilization control, it will not be possible to obtain a good image that is in focus and has little subject blur. Therefore, by setting the area where the previous main subject exists as the target area for image stabilization control, a continuity of image is provided.

In step S605, the CPU 15 determines whether the aperture value of the interchangeable lens 31 can be changed. If the aperture value can be changed, the process proceeds to step S606. In step S606, the CPU 15 narrows the aperture 34 of the interchangeable lens 31. Next, in step S607, the CPU 15 determines whether narrowing the aperture 34 makes it possible to bring the old main subject within the depth of field. If it is possible to bring the old main subject within the depth of field, the process proceeds to step S608. On the other hand, if it is not possible to bring the old main subject within the depth of field, the process proceeds to step S609. In step S609, the CPU 15 designates the area where the new main subject exists as the target area for image stabilization control. Note that if it is not possible to change the aperture value of the interchangeable lens 31 in step S605, the process proceeds to step S609, and designates the area where the new main subject exists as the target area for image stabilization control.

In this embodiment, the main subject detection circuit 26 serves as the main subject detection means for detecting the position of the main subject (main subject information). The CPU 15 has the function of determining the shake correction control area according to the determination result of the determination means. As a result, if there is a possibility that the subject detected as the main subject is not necessarily the subject for which the photographer wants to correct subject shake, or if the previous main subject is within the depth, the subject targeted for shake correction control is set to be different from the subject targeted for focus control. This makes it possible to provide smooth, high-quality images even in scenes with a mixture of multiple subjects.

In each embodiment, an interchangeable lens imaging device has been described as an example, but the invention can also be applied to image stabilization in digital video cameras and integrated lens digital cameras. In each embodiment, the CPU is responsible for the functions of the calculation means for calculating the correction amount, the means for designating the target area for focus control, the determination means for determining whether the image stabilization control area should be different from the focus control area, and the function for determining the image stabilization control area in accordance with the determination result of the determination means. However, dedicated circuits for performing these functions may also be provided.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

In each embodiment, the subject (area on the screen) targeted for image stabilization control is switched between the same subject as the subject targeted for focus control or a different subject based on predetermined conditions. Therefore, each embodiment can provide a control device, imaging device, control method, and program that can capture smooth, high-quality images in scenes with multiple subjects.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the invention.

The shake correction process in each embodiment is electronic image stabilization (image stabilization by trimming), but is not limited to this. For example, shake correction can also be performed by optical image stabilization, such as by moving the correction lens (image stabilization lens) of the imaging optical system in a direction perpendicular to the optical axis, or by moving the sensor 3 in a direction perpendicular to the optical axis.

1. Imaging device (control device)
13 Shake detection circuit (shake detection means)
14 Shake detection sensor (shake detection means)
15 CPU (control means)
26 Main subject detection circuit (main subject detection means)
27 Motion vector detection circuit (motion vector detection means)

Claims (18)

a subject detection means for detecting main subject information;
a motion vector detection means for detecting motion vector information in an image;
a shake detection means for detecting shake information using an inertial sensor ;
and control means for determining whether or not to make a blur correction control area different from a focus control area based on the main subject information, the blur information, and the motion vector information. The control device described in claim 1, characterized in that the control means determines the blur correction control area based on the main subject information, the blur information, and the motion vector information. 3. The control device according to claim 1, wherein the control means makes the image stabilization control area different from the focus control area when a ratio of the size of the second object to the size of the first object is equal to or greater than a first predetermined ratio. 3. The control device according to claim 1 , wherein the control means makes the image stabilization control area different from the focus control area when a ratio of a size of the second subject to a size of the first subject is equal to or greater than a second predetermined ratio and a difference in position of the second subject to the first subject is smaller than a first predetermined value. 3. The control device according to claim 1, wherein the control means makes the image stabilization control area different from the focus control area when a difference in position of the second object relative to the first object is smaller than a second predetermined value. 3. The control device according to claim 1, wherein the control means makes the image stabilization control area different from the focus control area when a difference in position of the second object relative to the first object is smaller than a third predetermined value and a difference in motion vector is equal to or greater than a first predetermined vector difference. A control device according to claim 1 or 2, characterized in that the control means differentiates the image stabilization control area from the focus control area when the position of the most recently detected main subject is outside a first predetermined range from the center position. 3. The control device according to claim 1, wherein the control means makes the image stabilization control area different from the focus control area when a ratio of the size of the second object to the size of the first object is equal to or smaller than a first comparison value or equal to or larger than a second comparison value, or when a difference in position of the plurality of objects is larger than a fourth predetermined value. Further, the apparatus includes a defocus amount detection means for detecting defocus amount information,
3. The control device according to claim 1, wherein the control means makes the image stabilization control area different from the focus control area when a plurality of subjects are within a first predetermined defocus range or a first distance range. A control device according to any one of claims 1 to 9, characterized in that the control means, when determining that the image stabilization control area should be different from the focus control area, sets the background as the image stabilization control area. A control device according to any one of claims 1 to 10, characterized in that, when the control means determines that the image stabilization control area should be different from the focus control area, the control means sets the image stabilization control area to multiple subjects, the multiple subjects and the background, a previously detected subject, or a previously detected subject and the background. A control device according to any one of claims 1 to 11, characterized in that the control means, when a subject is specified by the photographer, makes the image stabilization control area the same as the focus control area. A control device according to any one of claims 1 to 12, characterized in that the control means makes the image stabilization control area the same as the focus control area when the output signal of the shake detection means is greater than the first output signal and the motion vector information is smaller than the first motion vector. Further, the apparatus includes a defocus amount detection means for detecting defocus amount information,
14. The control device according to claim 1, wherein the control means determines whether to make the image stabilization control area different from the focus control area based on the main subject information, the defocus amount information, and the aperture value information. The control means
calculating a shake correction amount based on the shake information;
15. The control device according to claim 1, wherein the image stabilization control in the image stabilization control region is performed by electronic image stabilization or optical image stabilization based on the image stabilization amount. An imaging element;
An imaging device comprising: the control device according to claim 1 . detecting main subject information;
Detecting motion vector information in the image;
detecting shake information using an inertial sensor ;
and determining whether or not to make a blur correction control area different from a focus control area based on the main subject information, the blur information, and the motion vector information. A program causing a computer to execute the control method described in claim 17.