JP7594367B2 - Image capture device and control method thereof - Google Patents

Description

The present invention relates to an imaging device and a control method thereof.

Conventionally, imaging devices have been proposed that have a subject tracking control function that automatically tracks a subject so as to frame the subject at a specified position within the shooting screen.

Patent Document 1 discloses an imaging device that employs an optical tracking method, in which a blur correction mechanism for correcting image blur is driven to track the movement of a subject within the shooting screen, thereby tracking the subject so that it is positioned at a predetermined position within the shooting screen. On the other hand, an electronic tracking method is known as a subject tracking method that is different from the optical tracking method. When the electronic tracking method is applied, the imaging device achieves subject tracking by cutting out an image so that the subject's position within the shooting screen is positioned at a predetermined position in accordance with the subject's movement within the shooting screen. The cut-out image is also displayed in live view.

JP 2010-93362 A

In an imaging device equipped with a subject tracking control function, the imaging device automatically tracks the subject so that the subject is positioned at a predetermined position within the shooting screen. This makes it easy for the photographer to check the subject displayed in live view on a liquid crystal display (LCD) or the like before taking the picture, for example.

In particular, when subject tracking control is performed using an electronic tracking method, the size of the subject image displayed in live view is enlarged by using the cropped image, making it even easier to confirm the subject.

However, when an image is cropped so that the subject on the screen is positioned at a specific position, the angle of view of the resulting image becomes narrow. For example, when taking continuous shots of a moving subject, live view is displayed for a short period of time between each shot (hereinafter referred to as "inter-frame live view display"). The photographer needs to follow the subject (framing operation) while checking the inter-frame live view display. In this case, if a cropped image with a narrow angle of view is displayed as the inter-frame live view display, it is difficult to confirm the surroundings of the subject, which results in the problem of difficult framing operations.

The present invention was made in consideration of the above problems, and aims to provide an easy-to-view live view display when automatically tracking a subject by cropping an image.

In order to achieve the above object, an imaging device of the present invention comprises an imaging means for outputting an image obtained by capturing an image, a detection means for detecting a position of a subject to be tracked in the image, a setting means for setting a partial region including the subject in the image based on the position of the subject, a generation means for generating a tracking image having one of two different configurations using the image and the partial region depending on whether the state of the imaging device is a first state or a second state, a display means for displaying the tracking image generated by the generation means, and a recording means for recording an image, wherein the setting means performs a pre-setting operation so as to place the subject at a predetermined target position of the partial region. The partial area is set, and the first state is a state in which subject tracking is instructed and live view display is performed before image recording is instructed, or a state in which subject tracking is instructed and image shooting and recording are being performed, and when the state of the imaging device is in the first state, the generation means enlarges the image of the partial area to the same size as the image to generate a first tracking image, and in the first state, the recording means records the first tracking image generated by the generation means, and when the state of the imaging device is in the second state, the generation means superimposes information indicating the partial area on the image to generate a second tracking image .

The present invention makes it possible to provide an easy-to-view live view display when automatically tracking a subject by cropping an image.

FIG. 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. 4 is a flowchart showing a subject tracking control process in the embodiment. 4 is a flowchart showing a shooting process in the present embodiment. 5A to 5C are diagrams for explaining cut-out areas in the present embodiment. 5A to 5C are diagrams illustrating generation of a first tracking image in the present embodiment. 5A to 5C are diagrams illustrating generation of a second tracking image in the present embodiment.

The following embodiments are described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe multiple features, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicate explanations are omitted.

FIG. 1 is a diagram showing the configuration of an image pickup apparatus according to the present invention.
The imaging device 100 is, for example, a digital single-lens reflex camera, a digital compact camera, a digital video camera, etc. Furthermore, the present invention can also be applied to a camera-equipped mobile terminal, a surveillance camera, a Web camera, etc. The imaging device 100 of this embodiment has a tracking control function of an electronic tracking method that tracks a subject by cutting out the subject area from a captured image.

The imaging device 100 also has a shake correction mechanism that corrects image blur caused by vibrations applied to the device. In this embodiment, the CPU 101 realizes the shake correction function by driving the optical correction system 122. Note that the correction of image blur in either the vertical or horizontal direction of the image will be described, and the correction of image blur in the other direction will not be described as it is controlled in a similar manner.

The shake detection unit 102 is composed of an angular velocity sensor such as a vibration gyroscope, and detects the amount of shake applied to the imaging device 100 due to hand shake, body shaking, etc. as an angular velocity signal, and supplies the angular velocity signal to the A/D converter 103. The A/D converter 103 digitizes the angular velocity signal from the shake detection unit 102 and supplies it to the DC component removal filter 104 inside the CPU 101 as angular velocity data.

The DC component removal filter 104 cuts off low frequency components contained in the angular velocity data from the A/D converter 103 and outputs high frequency components. The DC component removal filter 104 is, for example, a high pass filter having a function of changing characteristics in any frequency band. Note that the DC component removal filter 104 may be configured to subtract, instead of a high pass filter, a signal that has passed through a low pass filter that cuts off high frequency component signals from the output of the A/D converter 103, from the output of the A/D converter 103.
The focal length calculator 105 calculates the focal length of the imaging optical system 121 , and corrects the output of the DC component removal filter 104 so that the output becomes an optimum value for driving the optical correction system 122 .

The integrator 106 has the ability to change its characteristics in any frequency band. The integrator 106 integrates the output from the focal length calculation unit 105 and calculates the drive amount of the optical correction system 122. The optical correction data output control unit 107 limits the output of the integrator 106 and supplies it to the subtractor 108 so that the optical correction system 122 is driven within its movable range.

The optical correction system position detection unit 109 is composed of a magnet and, for example, a Hall sensor provided in a position facing the magnet, and detects the amount of movement of the optical correction system 122 in a direction perpendicular to the optical axis, and outputs the detection value to the A/D converter 110. The A/D converter 110 A/D converts the output from the optical correction system position detection unit 109, and supplies the obtained position data of the optical correction system 122 to the subtractor 108.

The subtractor 108 subtracts the position data output from the A/D converter 110 from the output of the optical correction data output control unit 107, and supplies the resulting deviation data, i.e., the correction amount data for optical image stabilization, to the control filter calculation unit 117.

The control filter calculation unit 117 is composed of an amplifier that amplifies the correction amount data output from the subtractor 108 with a predetermined gain, and a phase compensation filter. The correction amount data undergoes signal processing by the amplifier and phase compensation filter in the control filter calculation unit 117, and is output to the pulse width modulation unit 118.

The pulse width modulation unit 118 modulates the data supplied through the control filter calculation unit 117 into a PWM waveform that changes the duty ratio of the pulse wave and supplies it to the motor drive unit 119. The motor 120 is, for example, a voice coil type motor for driving the optical correction system 122, and is driven by the motor drive unit 119 to move the optical correction system 122 in a direction perpendicular to the optical axis. This forms a feedback control system that causes the amount of movement of the optical correction system 122 in a direction perpendicular to the optical axis to follow the output of the optical correction data output control unit 107.

The imaging optical system 121 performs operations such as zooming and focusing, and focuses the incident subject image on the imaging element 123. The optical correction system 122 is a correction system capable of optically correcting shake, which deflects the optical axis by moving in a direction perpendicular to the optical axis, and is, for example, a correction lens (shift lens). By driving the shift lens, an image in which image blur (movement of the subject on the imaging surface) caused by vibration applied to the device has been corrected is formed on the imaging element 123. The imaging element 123 converts the subject image formed through the optical system consisting of the imaging optical system 121 and the optical correction system 122 into an image signal and outputs it. The signal processing unit 124 generates image data conforming to, for example, the NTSC format based on the image signal output from the imaging element 123, and supplies it to the captured image memory 125.

The subject detection unit 126 detects the subject to be tracked from the image data stored in the captured image memory 125, and detects the position of the detected subject in the image. The position of the subject detected here is the position coordinate in the image. Various methods have been proposed for detecting the subject, and any of these methods may be used. The position of the subject may be the center or center of gravity of the area of the detected subject, or, if a face is detected, the center or center of gravity of the face. The subject to be tracked is not limited to the stored image data. For example, the subject may be specified by the operation unit 136 while the photographer watches the image continuously displayed in live view on the screen of the display device 135 at any time.

The cut-out region determination unit 127 sets a partial region including the subject to be tracked detected by the subject detection unit 126 as a cut-out region. At that time, the cut-out region is determined so that the subject is positioned near a target position in the cut-out region (for example, the center position of the cut-out region).

The target position is not limited to the center of the shooting screen. For example, the photographer can operate the operation unit 136 while watching the image continuously displayed in live view on the screen of the display device 135 at any time, thereby specifying any position on the screen of the display device 135 as the target position.

The tracking image generation unit 128 cuts out a cut-out area from the captured image based on the coordinates of the cut-out area determined by the cut-out area determination unit 127. Furthermore, the tracking image generation unit 128 performs enlargement processing on the image of the cut-out area to make it the same size as the original captured image, and performs positioning processing on the image of the cut-out area to position it at the center of the captured screen to create a tracking image.

The image selection control unit 129 determines whether the captured image stored in the captured image memory 125 or the tracking image created by the tracking image generation unit 128 is to be displayed on the display device 135 or recorded on the recording medium 132, depending on the state of the imaging device 100. Then, depending on the determination, the captured image stored in the captured image memory 125 or the tracking image created by the tracking image generation unit 128 is stored in either the recorded image memory 130 or the displayed image memory 133.

The recording control unit 131 reads out an image to be recorded from the recorded image memory 130, and records the read image on a recording medium 132. The recording medium 132 is an information recording medium such as a magnetic recording medium, such as a hard disk, or a semiconductor memory.
The display control unit 134 reads an image to be displayed from the display image memory 133, and displays the read image on the display device 135. The display device 135 displays an image using a liquid crystal display element (LCD) or the like.

The operation unit 136 includes zoom operation members such as a zoom lever or a zoom button, a release switch that issues an instruction to start shooting, and a touch panel or operation switch that specifies a subject or sets the imaging device 100. Operation signals from the operation unit 136 are input to the CPU 101.

Next, the subject tracking control in this embodiment will be described with reference to FIG.
2 is a flowchart showing a subject tracking control process performed by the image capture device 100 according to this embodiment. Each step in FIG.

When the imaging device 100 is powered on, it performs various initialization processes and then goes into a standby state for an operation instruction from the user.
First, in S 201 , a captured image is acquired by photoelectrically converting light incident via the imaging optical system 121 and the optical correction system 122 by the imaging element 123 , and the captured image is stored in the captured image memory 125 .

Next, in S202, it is determined whether or not the user has operated the operation unit 136 to instruct execution of the subject tracking mode. If execution of the subject tracking mode has been instructed, the process proceeds to S203, and if execution of the subject tracking mode has not been instructed, the process proceeds to S204.

In S203, a subject to be tracked is selected, and it is determined whether or not it is possible to start tracking the subject. If a subject is selected and it is possible to start tracking the subject, the process proceeds to S205. On the other hand, if it is not possible to start tracking the subject, the process proceeds to S204.

In S204, the captured image acquired in S201 is displayed on the display device 135. In other words, if it is determined in S202 that there is no instruction to execute the subject tracking mode, or if it is determined in S203 that subject tracking cannot be started, the captured image acquired in S201 is displayed as is on the display device 135.

In S205, the subject detection unit 126 detects the subject and its position (hereinafter referred to as the "subject position") in the captured image acquired in S201. The subject position detected here is the position coordinates in the captured image.

In S206, the cut-out region determination unit 127 determines a cut-out region from the detected subject position. The cut-out region is determined to be a region in which the subject position detected in S205 can be positioned at a target position (e.g., the center position of the cut-out region) in a preset cut-out region.

Here, the cropping area in this embodiment will be described with reference to FIG. 4. FIG. 4(a) is a diagram showing an example of a captured image 401 acquired in S201. In the captured image 401, a subject 402 is a subject that is the target of tracking control, and the position of the subject 402 in the captured image 401 is the subject position detected in S205.

Figure 4(b) shows an example of a cutout region 403 for Figure 4(a) when the target position in the pre-set cutout region is the center position of the cutout region. In other words, the cutout region 403 is determined so that the subject 402 is placed at the center position of the cutout region 403.

In the example shown in FIG. 4(b), the target position in the preset cut-out area is the center position of the cut-out area, but the present invention is not limited to this and may be set to any position. For example, the target position in the preset cut-out area may be set to the left end in the horizontal direction and the center in the vertical direction of the cut-out area. FIG. 4(c) shows cut-out area 403' in the example shown in FIG. 4(a) when the target position in the preset cut-out area is the left end in the horizontal direction and the center in the vertical direction of the cut-out area. In this case, cut-out area 403' is determined so that subject 402 is placed at the left end in the horizontal direction and the center in the vertical direction of cut-out area 403'.

In S207, the tracking image generation unit 128 generates a first tracking image based on the cut-out area determined in S206. Here, the generation of the first tracking image in this embodiment will be described with reference to FIG. 5. Note that, here, as shown in FIG. 4(b), a case will be described as an example in which a cut-out area 403 is set in S206 for the captured image 401 acquired in S201.

Figure 5 (a) shows an example of a first tracking image 501 generated from Figure 4 (b). In other words, the first tracking image 501 is an image in which the cutout area 403 is enlarged to the same size as the original captured image 401. As a result, the subject 402 is positioned at the center position of the first tracking image 501. However, in this case, the size of the subject 402 in the first tracking image 501 is larger than that of the subject 402 in the captured image 401.

In the example shown in FIG. 5(a), the cutout area 403 is enlarged to the same size as the original captured image 401 to create the first tracking image 501, but the size of the cutout area 403 may not be changed and the image may be configured with the cutout area 403 positioned at the center of the image. FIG. 5(b) shows an example of a first tracking image 502 generated when the size of the cutout area 403 is not changed. In other words, the cutout area 403 is positioned at the center of the image without being enlarged, and an image with single-color data positioned around it is synthesized, and this is used as the first tracking image 502. As a result, the subject 402 is positioned at the center of the first tracking image 502.

In this case, the size of the subject 402 in the first tracking image 502 is equal to the size of the subject 402 in the captured image 401. In this way, since the size of the subject 402 in the captured image 401 and the size of the subject in the first tracking image 502 are equal, it is possible to reduce the discomfort felt by the user due to the change in the size of the subject when, for example, the captured image 401 and the first tracking image 502 are continuously displayed in live view.

In S208, the image selection control unit 129 stores the first tracking image generated in S207 in the display image memory 133, and the display control unit 134 displays the stored first tracking image on the display device 135. In other words, if it is determined in S202 that there is an instruction to execute the subject tracking mode, and it is determined in S203 that subject tracking can be started, the first tracking image generated in S207 is displayed on the display device 135.

In S209, it is determined whether or not a shooting instruction has been issued from the user. If a shooting instruction has been issued from the user, the process proceeds to S210, and if no shooting instruction has been issued, the process returns to S201.
In S210, a photographing process is performed, which will be described with reference to FIG.

3 is a flowchart illustrating an example of a photographing process performed by the image capturing apparatus 100 according to this embodiment.
When the image capturing process is started, the image capturing apparatus 100 performs image capturing preparation operations such as AF (autofocus) processing and AE (autoexposure) processing in S300. Then, in S301, a captured image is acquired by photoelectrically converting light incident via the image capturing optical system 121 by the image capturing element 123.

In S302, the subject detection unit 126 detects the subject position in the captured image acquired in S301. Note that the method for detecting the subject position in S302 is the same as in S205.

In S303, the cut-out area determination unit 127 determines the cut-out area from the subject position detected in S302. Note that the method for determining the cut-out area in S303 is the same as in S206.

In S304, the tracking image generation unit 128 generates a tracking image for recording based on the cut-out area determined in S303. The method of generating the tracking image for recording in S304 is the same as the method described in S207. In this embodiment, the tracking image for recording in S304 is an image in S207 in which the cut-out area is enlarged to the same size as the original captured image, as shown in FIG. 5(a). However, for example, as shown in FIG. 5(b), the cut-out area may be positioned in the center of the image without changing the size of the cut-out area.

In S305, the image selection control unit 129 stores the first tracking image generated in S304 in the recording image memory 130, and the recording control unit 131 records the stored first tracking image on the recording medium 132, and the shooting process ends.

In the present embodiment, even in the shooting process, the subject position is detected from the captured image in S302, and then the cutout area is determined in S303. However, the shooting process may be configured not to perform these steps, and instead to generate a tracking image for recording in S304 based on the cutout area determined immediately before the shooting process.

When the photographing process is completed as described above, the process proceeds to S221 in FIG.
In S211, it is determined whether or not the user continues to give a shooting instruction. If the user continues to give a shooting instruction, the process proceeds to S212. On the other hand, if the user does not give a shooting instruction, the process returns to S201.

After the photographing process, in S212, a captured image generated via the image sensor 123 and the imaging optical system 121 is acquired.
In S213, the subject detection unit 126 detects the subject position in the captured image acquired in S212 by the same subject position detection method as in S205.

In S214, the clipping region determination unit 127 determines a clipping region from the subject position detected in S213 by the same clipping region determination method as in S206.
In S215, the tracking image generating unit 128 generates a second tracking image based on the cut-out area determined in S214.

Here, generation of the second tracking image in this embodiment will be described with reference to FIG.
6A is a diagram showing an example of a captured image 601 acquired in S212. In the captured image 601, a subject 602 is a subject that is a target of tracking control, and the position of the subject 602 in the captured image 601 is the subject position detected in S213.

Figure 6 (b) shows an example of a second tracking image 603 generated from Figure 6 (a) when the target position in the cut-out area is the center position of the cut-out area. As shown in Figure 6 (b), the second tracking image 603 is an image in which a frame 604 indicating the range of the cut-out area determined in S214 is superimposed on a captured image 601. The second tracking image 603 allows the user to grasp the range set as the cut-out area as the position of the frame 604 within the same image as the captured image 601. Furthermore, the subject 602 in the second tracking image 603 is the same as the subject 602 in the captured image 601 in both position and size within the image.

In this embodiment, the second tracking image is an image in which a frame indicating the range of the cut-out area determined in S214 is superimposed on the captured image. However, the second tracking image is not limited to a frame as long as the information can indicate the range of the cut-out area. Also, for example, if information indicating the range of the cut-out area is not necessary as the second tracking image, the captured image itself may be used as the second tracking image. In other words, the captured image 601 in FIG. 6(a) is used as the second tracking image.

In S216, the image selection control unit 129 stores the second tracking image generated in S215 in the display image memory 133, and the display control unit 134 displays the stored tracking image on the display device 135.

In S217, it is determined whether the conditions for shooting are met and shooting processing can be performed. If it is determined that shooting processing can be performed, the process returns to S210. On the other hand, if it is determined that shooting processing cannot be performed, the process returns to S212. Note that the conditions for shooting are, for example, whether or not information for performing AF processing and AE processing obtained from the captured image acquired in S212 has been acquired the required number of times. However, the conditions for shooting are not limited to this, and if there are other requirements necessary for performing shooting processing, those may also be used as conditions.

In other words, if the user continues to give a shooting instruction in S211, the second tracking image generated in S215 will be displayed on the display device 135 until the next shooting process is executed.

As described above, according to this embodiment, when subject tracking is possible, the live view display before the shooting process is performed displays a first tracking image in which the subject is located at a preset target position. At that time, by using an image in which the cropped area is enlarged to the same size as the original captured image as the first tracking image, it is possible to provide the user with a live view display that allows them to easily check the subject.

Furthermore, when performing continuous shooting, which is a continuous shooting process, a second tracking image is displayed as a live view display between frames between each shooting, in which information such as a frame indicating a cutout area for positioning the subject at a preset target position is superimposed on the captured image. In other words, the second tracking image only displays the cutout area in a superimposed manner, and does not actually cut out the subject. This makes it possible to provide a live view display between frames that allows the user to easily perform framing operations during continuous shooting by displaying a second tracking image with a wider angle of view than the first tracking image while performing subject tracking control during continuous shooting.
<Other embodiments>
The present invention may be applied to a system made up of a plurality of devices, or to an apparatus made up of a single device.

The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-mentioned embodiments to a system or device via a network or storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions.

The invention is not limited to the above-described embodiment, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention.

100: Imaging device, 101: CPU, 126: Subject detection unit, 127: Cutout area determination unit, 128: Tracking image generation unit, 129: Image selection control unit, 130: Recorded image memory, 131: Recording control unit, 132: Recording medium, 133: Display image memory, 134: Display control unit, 135: Display device

Claims (6)

1. An imaging device, comprising:
An imaging means for outputting an image obtained by photographing;
a detection means for detecting a position of a subject to be tracked in the image;
a setting means for setting a partial region including the subject within the image based on a position of the subject;
a generating means for generating a tracking image having one of two different configurations using the image and the partial region depending on whether the state of the imaging device is a first state or a second state ;
A display means for displaying the tracking image generated by the generation means;
A recording means for recording an image,
the setting means sets the partial area so as to place the subject at a predetermined target position of the partial area ;
The first state is a state in which a subject tracking command is issued and a live view display is being performed before an image recording command is issued, or a state in which a subject tracking command is issued and an image is being photographed and recorded,
When the state of the imaging device is the first state, the generating means enlarges the image of the partial region to the same size as the image to generate a first tracking image;
In the first state, the recording means records the first tracking image generated by the generating means,
When the state of the imaging device is the second state, the generating means generates a second tracking image by superimposing information indicating the partial region on the image.
1. An imaging device comprising: The imaging device according to claim 1 , wherein the information indicating the partial region is a frame indicating the partial region. 3. The imaging apparatus according to claim 1, wherein the second state is a state in which, when subject tracking is instructed and continuous shooting is instructed, a live view display is performed during continuous shooting. A control method for an imaging device, comprising:
an imaging step in which the imaging means outputs an image obtained by imaging;
a detection step in which a detection means detects a position of a subject to be tracked in the image;
a setting step of setting a partial region including the subject in the image based on a position of the subject by a setting means;
A generation step in which a generation means generates a tracking image having one of two different configurations using the image and the partial region depending on whether the state of the imaging device is a first state or a second state ;
a display step in which a display means displays the tracking image generated in the generation step;
The recording means has a recording step of recording an image ,
In the setting step, the partial region is set so that the subject is disposed at a predetermined target position of the partial region ;
The first state is a state in which a subject tracking command is issued and a live view display is being performed before an image recording command is issued, or a state in which a subject tracking command is issued and an image is being photographed and recorded,
When the state of the imaging device is the first state, in the generating step, an image of the partial region is enlarged to the same size as the image to generate a first tracking image;
In the first state, the recording step records the first tracking image generated in the generating step,
When the state of the imaging device is the second state, the generating step generates a second tracking image by superimposing information indicating the partial region on the image.
23. A method for controlling an imaging apparatus comprising the steps of: A program for causing a computer to execute each step of the control method according to claim 4 . A computer-readable storage medium storing the program according to claim 5 .

Images