JP6797955B2 - Image processing device and control method of image processing device, imaging device, program - Google Patents

Description

The present invention relates to a recording control technique for image and distance information.

There is a technique for acquiring one still image that suits the user's taste by selecting a favorite image from the continuously acquired moving images and performing various image processing after the fact. Patent Document 1 discloses a technique for determining whether or not a subject is in a predetermined state with respect to continuously captured images, and relatively improving the recorded image quality of the images in the predetermined state. Further, Patent Document 2 discloses a technique for acquiring a high-quality image by not performing frame compression when the shooting mode is changed to the super-resolution mode. Further, as in Patent Document 3, a technique for adjusting the blur of a captured image by image processing by generating distance distribution information as additional information in addition to the captured image is disclosed.

Japanese Unexamined Patent Publication No. 2011-166391 JP-A-2007-189665 JP-A-2015-119416

However, various image processes include processes that require highly accurate contour extraction of the subject. There are background blurring processing that electronically blurs the background, and area-specific gradation processing that obtains a high-quality dynamic range compressed image by separately performing gradation correction processing on the main subject and other areas. In order to realize these technologies, not only the frame image data is recorded, but also a distance map showing the distance distribution within the angle of view (at least the information that shows the relative relationship of the distances) is acquired. It is necessary. Further, in the area-specific gradation processing, it is necessary to record the RAW image before the image processing.

On the other hand, in the form of recording the data of the distance map or the RAW image for all the frames of the moving image in order to perform the image processing, there is a problem that the recording capacity becomes enormous and the burden on the user increases. Therefore, it is also very important to minimize the amount of frame image data when acquiring information for post-processing.
An object of the present invention is to provide an image processing apparatus capable of acquiring captured images and additional information, which can perform highly convenient image recording while suppressing the recording capacity.

The apparatus of one embodiment of the present invention is an image processing apparatus including a recording means for acquiring a plurality of image data and recording them on a recording medium, and is an acquisition means for acquiring depth distribution information of a subject corresponding to the image data. A first mode in which the image data and the depth distribution information corresponding to the image data are recorded on the recording medium by the recording means, and the image data is recorded by the recording means without recording the depth distribution information. The control means includes a control means for switching the second mode of recording on the recording medium and performing the recording process of the plurality of image data, and the control means has a frame rate for capturing the plurality of image data set by the user. An image processing apparatus characterized in that the recording process of a plurality of image data is performed by switching between the first mode and the second mode at a cycle corresponding to the above.

According to the present invention, it is possible to perform highly convenient image recording while suppressing the recording capacity.

It is a block diagram which shows the structure of the image pickup apparatus which concerns on 1st Embodiment of this invention. It is a block diagram of the recording mode control part in 1st Embodiment. It is a flowchart which shows the process of the recording mode control part in 1st Embodiment. It is explanatory drawing which shows an example of the main subject area detection result. It is a figure which illustrates the distance histogram of the main subject area and the background area. It is a figure which shows an example of the optical model which shows the object distance and the image distance. It is a figure which shows the relationship between the defocus amount and the object surface distance by a graph. It is a block diagram of the image processing part in 1st Embodiment. It is a flowchart of the background image blurring processing in 1st Embodiment. It is a figure which shows the screen example which informs that the blur processing of a background image is possible. It is a figure explaining the process of the distance map shaping part. It is a figure explaining the process of the focus subject extraction part. It is a figure explaining the blurring process. It is a block diagram of the recording mode control part in 2nd Embodiment of this invention. It is a flowchart which shows the process of the recording mode control part in 2nd Embodiment. It is a figure explaining the process of the threshold value calculation part. It is a block diagram of the image processing part in 2nd Embodiment. It is a flowchart of the gradation correction processing for each area in 2nd Embodiment. It is a figure which shows the screen example which informs that the gradation correction processing for each area is possible. It is a figure explaining the gradation characteristic calculation method in 2nd Embodiment. It is a figure explaining the synthesis process. It is a block diagram which shows the structure of the image pickup apparatus which concerns on 3rd Embodiment of this invention. It is a flowchart which shows the process of the record information adjustment part in 3rd Embodiment. It is a figure explaining the main subject score calculation process. It is a figure explaining the calculation process of the main subject score threshold value. It is a block diagram which shows the structure of the image pickup apparatus which concerns on 4th Embodiment of this invention. It is a flowchart which shows the process of the exposure condition control part in 4th Embodiment. It is a figure explaining the recording format of an image and a distance map.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. In each embodiment, an image processing apparatus that performs image processing accompanied by high-performance area extraction processing later according to a user's instruction will be described.

[First Embodiment]
In the first embodiment of the present invention, it is assumed that the image processing apparatus blurs the background image of a plurality of recorded continuous images for an arbitrary frame selected by the user later. To do. Therefore, in the present embodiment, an image processing device capable of switching between a mode in which depth distribution information as additional information is recorded and a mode in which depth distribution information is not recorded for a plurality of captured images is provided. FIG. 1 is a block diagram illustrating a configuration applicable to an image pickup apparatus as an example of the image processing apparatus of the present embodiment.
The imaging optical system 101 includes a lens group such as a zoom lens and a focus lens, an aperture adjusting device, and a shutter device. The imaging optical system 101 adjusts the magnification, focus position, and amount of light of the subject image reaching the imaging unit 102. The imaging unit 102 includes a CCD (charge-coupled device) image sensor, a CMOS (complementary metal oxide semiconductor) image sensor, and the like, and converts the luminous flux from the subject that has passed through the imaging optical system 101 into an electric signal by photoelectric conversion. The A (analog) / D (digital) conversion unit 103 converts the input analog electric signal into a digital image signal.

The recording mode control unit 104 has a plurality of recording modes and controls the amount of information of the image to be recorded in the recording unit 109. The image processing unit 105 performs various processing on the image signal read from the recording unit 109 in addition to the image signal output from the A / D conversion unit 103. For example, processing such as distortion and noise correction processing due to the imaging optical system, demosaiking processing, white balance adjustment, color conversion processing, and gamma correction is executed. In addition to the predetermined image processing, the image processing unit 105 also performs the background blurring process assumed in the present embodiment. In the present embodiment, among the image processing by the image processing unit 105, an image in which at least a part of the image processing other than the correction processing of distortion and noise caused by the imaging optical system is not performed is defined as a RAW image. In this embodiment, an example in which the image processing unit 105 performs the reproduction processing will be described, but a reproduction processing unit may be provided separately from the image processing unit 105.

The system control unit 106 is a control center that controls the operation control of the entire image pickup apparatus, and includes a CPU (central processing unit), a memory, and the like. The system control unit 106 controls drive control of the image pickup optical system 101 and the image pickup unit 102, predetermined image processing in the image processing unit 105, and the like according to an operation instruction from the operation unit 107 by the user.

The display unit 108 is composed of a liquid crystal display, an organic EL (Electro Luminescence) display, or the like, and displays an image according to an image signal generated by the imaging unit 102 or an image signal read from the recording unit 109. The recording unit 109 performs a process of recording an image signal on a recording medium. According to the recording settings of each of the still image and the moving image, the recording process is performed in the encoding format for still images (for example, JPEG) and the encoding format for moving images (for example, H.264, H.265, etc.). In addition, when the still image and the moving image are set to be recorded as RAW images before image processing, low compression such as uncompression or lossless compression, which causes less deterioration in image quality than when recording image data after image processing. It is compressed and recorded at a rate. The recording medium is an information recording medium using, for example, a package containing a rotating recording body such as a memory card or a magneto-optical disk on which a semiconductor memory is mounted. The recording medium is removable from the imaging device.

The bus 110 is used to send and receive signals between the recording mode control unit 104, the image processing unit 105, the system control unit 106, the display unit 108, and the recording unit 109.
Hereinafter, the processing flow in the present embodiment will be described with a focus on the recording mode control unit 104 and the image processing unit 105. The recording mode control unit 104 is a processing block characteristic of the present embodiment. Further, the image processing unit 105 is a processing block that performs blurring processing of the background image in response to an operation instruction after shooting.

The operation of the recording mode control unit 104 will be described with reference to FIGS. 2 and 3.
FIG. 2 is a block diagram of the recording mode control unit 104. The main subject area detection unit 201 acquires the data of the input image and detects the main subject area. The main subject area is, for example, an image area of a subject selected from a plurality of subjects. The distance map calculation unit 202 acquires the data of the input image and calculates the distance map (depth distribution information). The main subject distance calculation unit 203 acquires the information of the main subject area detected by the main subject area detection unit 201 and the distance map calculated by the distance map calculation unit 202, and calculates the main subject distance. The main subject distance is the distance from the image pickup device to the main subject. The background distance calculation unit 204 calculates the background distance from the information of the background area and the distance map. The background area is an area other than the main subject, and the background distance is the distance from the image pickup device to the background. The threshold value calculation unit 205 acquires the main subject distance and calculates a threshold value for the distance difference between the main subject and the background. The recording mode determination unit 206 acquires the main subject distance, the background distance, and the threshold value, compares the difference between the main subject distance and the background distance, and the threshold value, and performs the recording mode determination process. The recording mode includes at least a first mode for recording image data and distance information and a second mode for recording image data. In the present embodiment, control for switching between the first and second modes at the time of recording each frame will be described for a moving image in which the image data acquired by imaging is used as each frame, but the control is not limited to this. For example, in still image imaging, the present invention also includes an embodiment in which the first and second modes are switched in each imaging. That is, the control for switching between the first mode and the second mode may be sufficient for the plurality of acquired image data. Details of the processing of each part will be described later.

FIG. 3 is a flowchart illustrating the processing of the recording mode control unit 104. In this flowchart, it is assumed that a moving image is being acquired, and frame images are continuously input at a predetermined frame rate from the time when the user instructs the start of shooting to the time when the user instructs the end of the shooting operation. To do.

First, in S301, the main subject area detection unit 201 detects the main subject area. A specific example will be described with reference to FIG. FIG. 4 shows an example of the detected main subject area, and the main subject area is detected by using the face detection frame. In addition, when the main subject is not a person, the subject area is detected by general object detection, and the present invention is not limited to a specific detection method. Further, as shown in FIG. 4, a region that is not the main subject region in the entire region of the image is defined as a background region.

Next, in S302, the distance map calculation unit 202 calculates the distance map. As a method of calculating the distance map, for example, as described in Patent Document 3, the light from the subject is divided into pupils to generate a plurality of viewpoint images (parallax images), the amount of parallax is calculated, and the depth of the subject is calculated. There is a way to get distribution information. The depth distribution information of the subject is data showing the distance (subject distance) from the camera as an imaging means to the subject as an absolute value as a distance value, or data showing a relative distance relationship (image depth) in the image data (image depth). Distribution of parallax amount, distribution of defocus amount, etc.) is included. In the present embodiment, the following description will be given as the data represented by the distance value, but when the distribution of the parallax amount or the defocus amount is used as the depth distribution information, the distance value is replaced with the parallax amount and the defocus amount, respectively, and each process is performed. It shall be made.
Regarding the depth distribution information, there are various embodiments as information corresponding to the depth in the depth direction (depth direction) of each subject in the captured image. That is, the information indicated by the data corresponding to the depth of the subject directly represents the subject distance from the image pickup device to the subject in the image, or the relative relationship between the distance (subject distance) and the depth of the subject in the image. Any information may be used. For example, the image pickup unit 102 is controlled to change the focusing position, and a plurality of captured image data captured are acquired. Depth distribution information can be acquired from the focusing region of each captured image data and the focusing position information of the captured image data. In addition to this, when the image sensor of the image pickup unit 102 has a pupil-divided pixel configuration, depth distribution information for each pixel can be acquired from the phase difference of a pair of image signals. Specifically, the image sensor converts a pair of light fluxes that pass through different pupil region regions of the image pickup optical system into an electric signal, and converts the paired image data into a plurality of photoelectric conversion units. Output from. The image shift amount of each region is calculated by the correlation calculation between the paired image data, and the image shift map showing the distribution of the image shift amount is calculated. Alternatively, the image shift amount is further converted into the defocus amount, and a defocus map showing the distribution of the defocus amount (distribution on the two-dimensional plane of the captured image) is generated. When this defocus amount is converted into the subject distance based on the conditions of the imaging optical system and the image sensor, distance map data representing the distribution of the subject distance can be obtained. It is possible to acquire image shift map data, defocus map data, or distance map data of subject distance converted from the defocus amount.
In addition, the subject distance from the image pickup device to the subject in the image is directly measured by using the TOF (Time Of Flight) method in which the delay time from the projection of the light to the subject to the reception of the reflected light is measured to measure the distance to the subject. May be acquired. In the TOF method, pulsed light is projected onto a subject (object) by a light projecting means, the reflected light is received by the imaging unit 102, and the flight time (delay time) of the pulsed light is measured to measure the subject distance (delay time). Measure the distance to the object) and acquire the depth distribution information.
In the present embodiment, the following description will be given as the data represented by the distance value, but when the distribution of the parallax amount or the defocus amount is used as the depth distribution information, the distance value is replaced with the parallax amount and the defocus amount, respectively, and each process is performed. It shall be made.

In S303, the main subject distance calculation unit 203 calculates the main subject distance (parallax amount and defocus amount corresponding to the main subject) using the main subject area information and the distance map. In S304, the background distance calculation unit 204 calculates the background distance using the background area information and the distance map. A specific example of the processing of S303 and S304 will be described with reference to FIG. FIG. 5 is a diagram showing a method of calculating the main subject distance and the background distance. The horizontal axis represents the distance in the optical axis direction of the image pickup device, and the vertical axis represents the frequency (frequency) of the distance histogram.

First, the main subject distance calculation unit 203 acquires a distance histogram in the main subject area shown in FIG. 5A, and sets the distance corresponding to the peak value at which the frequency becomes the maximum value as the main subject distance. FIG. 5A shows an example in which two peak values exist. The reason why the peak value exists in addition to the main subject distance is that the boundary of the main subject area is defined by a rectangular frame, so that a part of the background image is included in the main subject area.

Next, the background distance calculation unit 204 acquires a distance histogram in the background region shown in FIG. 5 (B), and sets the distance corresponding to the peak value at which the frequency becomes the maximum value as the background distance. FIG. 5B shows an example in which three peak values exist.

In S305, the threshold value calculation unit 205 calculates the threshold value for the distance difference between the main subject and the background. The threshold value is determined using an optical model from the value of the main subject distance. This will be described with reference to FIG. The optical model of FIG. 6 shows an object distance and an image distance when the imaging optical system 101 is approximated as one lens. With respect to the lens with the focal length (denoted as f), the side where the object exists (the left side of the lens in FIG. 6) is the object surface side, and the side where the light from the object is imaged through the lens (the lens in FIG. 6). The right side) is defined as the image plane side. Furthermore, assuming that the main subject is in focus, the following distances are defined.
・ Main subject distance: Dist_Obj
・ Background distance: Dist_Back
・ Main subject image distance: Img_Obj
The main subject image distance is the distance at which the light from the main subject is imaged on the image plane side. Further, the position on the image plane side where the light from the main subject is imaged is defined as the focus plane, and the displacement amount from the focus plane to the position where the background light is imaged is defined as the defocus amount def. Here, regarding the sign of the defocus amount def, the direction away from the object surface side with respect to the lens is positive. Therefore, the defocus amount at the position where the background light, which is located farther from the main subject on the object surface, is formed, has a negative value. Qualitatively, the larger the absolute value of the defocus amount, the greater the degree of scattering of the background image on the focus plane, and the greater the blurring of the background.

In FIG. 6, the following equations (1) and (2) are established from the lens formula.
Here, when the main subject image distance Img_obj is deleted from the equations (1) and (2) and transformed into the equation for the background distance Dist_Back, the following equation (3) is obtained.
From the equation (3), the background distance Dist_Back can be regarded as a function of the defocus amount def when the main subject distance Dist_Obj and the focal length f are uniquely determined. Therefore, in the present embodiment, an example of mainly calculating the subject distance as a distance map is shown, but the object can also be achieved by the defocus amount and the parallax amount.

FIG. 7 is a graph created based on the equation (3) exemplifying the relationship between the defocus amount (horizontal axis) and the object surface distance (vertical axis). Two examples are shown when the focal length is 50 mm and the main subject distance Dist_obj is 3 m (3000 mm) and 5 m (5000 mm). Since it is assumed that the main subject is in focus, the defocus amount is 0 when the object distance is equal to Dist_obj. Assuming that the allowable defocus amount that can be regarded as the background is sufficiently blurred is -0.2 mm, the corresponding object distance is about 3911 mm when Dist_obj = 3 m. Therefore, the threshold value for the distance difference between the subject and the background is | 3911-3000 | = 911 mm, which is calculated to be about 90 cm.
The above is the description of the process performed by the threshold value calculation unit 205 in S305.

Next, in S306, the recording mode determination unit 206 compares the difference between the main subject distance and the background distance with the threshold value acquired from the threshold value calculation unit 205. When the difference between the main subject distance and the background distance is within the threshold value, the recording mode determination unit 206 determines that the background image is not sufficiently blurred and may require electronic blurring processing later in S307. Proceed to, and the recording process is performed in the high quality mode, which is the first mode. In the high quality mode, the process of recording the distance map in addition to the data of the compressed frame is usually executed. The process of blurring the background using the information recorded in the high quality mode will be described later. On the other hand, when the difference between the main subject distance and the background distance is larger than the threshold value, the recording mode determination unit 206 considers that the background image is sufficiently blurred. That is, the recording mode determination unit 206 determines that the electronic blurring process is unnecessary, proceeds to S308, and performs the recording process in the normal mode, which is the second mode. In the normal mode, the process of recording only the data of the normal compressed frame is executed. The predetermined signal processing associated with the frame compression processing and the development processing shall be performed in the image processing unit 105. The processing by the recording mode control unit 104 is performed at the time of shooting. Further, the image recorded in the high quality mode is not limited to the normal compressed frame, and may be a RAW image before image processing. Here, the RAW image may be compressed by a reversible compression method at the time of recording.

Next, the background image blurring process performed according to the user's instruction after recording will be described. This processing is mainly performed by the image processing unit 105, but the operation unit 107 and the system control unit 106 that receive the user's instruction are also involved. Since this process is performed after recording, it will be referred to as "post-blurring process" below. The post-blurring process will be described with reference to FIGS. 8 and 9.

FIG. 8 is a processing block diagram of the image processing unit 105 relating to the post-blurring process. The distance map shaping unit 801 performs the distance map shaping process. The shaping process of the distance map will be described later. The focus subject extraction unit 802 extracts an image of the focus subject. The in-focus subject is the main subject that is in focus. The blur processing unit 803 performs blur processing of the background image and outputs the frame image data after the processing. Details of the processing of each part will be described later.

FIG. 9 is a flowchart of the post-blurring process.
First, in S901, the imaging device receives an instruction for blurring the background image from the user. Since the frame that can be post-blurred is only the frame recorded in the high quality mode, it is necessary to inform the user whether or not the frame can be processed. FIG. 10 shows a display example in which the display unit 108 notifies (notifies) the user that the post-blurring process is possible. Display and input processing is performed to ask for an instruction as to whether or not to perform blurring processing of the background image. When the user chooses to perform the background image blurring process during video reproduction, the system control unit 106 receives an operation instruction from the operation unit 107 and instructs the image processing unit 105 to perform the processing after S902.

In S902, the distance map shaping unit 801 acquires the frame image and the distance map data, and performs the distance map shaping process. FIG. 11 is a diagram illustrating a shaping process of the distance map. FIG. 11A exemplifies an input frame image, FIG. 11B exemplifies an input distance map before the shaping process, and FIG. 11C exemplifies an output distance map after the shaping process.

The subject contour of the input distance map shown in FIG. 11B may be less accurate than the subject contour in the input frame image shown in FIG. 11A. Reasons for this include the influence of perspective competition at the boundary of the subject when calculating the distance, and the fact that the distance map is calculated by lowering the resolution of the frame image. High-precision contour extraction is required for blurring the background image. Therefore, a process of matching the subject contour of the distance map with the subject contour of the frame image is required, and this process is called a shaping process. Of course, the distance map after the shaping process is highly versatile and can be used in processes that require a distance map other than the background image blurring process.

The shaping process is performed by bilateral filtering. In the bilateral filter processing, the shaping image is used as a frame image (see FIG. 11A), and the filter result (denoted as Jp) at the pixel position p of interest is represented by the following equation (4).
Jp = (1 / Kp) ΣI1q ・ f (| p−q |) ・ g (| I2p−I2q |) ・ ・ ・ (4)
The meaning of each symbol in the formula (4) is as follows.
q: Peripheral pixel position Ω: Integration target area centered on the pixel position p of interest Σ: Integration of the q ∈ Ω range I1q: Distance map signal value at the peripheral pixel position q f (| p−q |): Pixel position of interest p Gaussian function centered on I2p: Pixel value of the shaping image at the pixel position p of interest I2q: Pixel value of the shaping image at the peripheral pixel position q g (| I2p-I2q |): Pixel value of the shaping image I2p Gaussian function centered on Kp: Normalization coefficient, which is the integrated value of f · g weights.
In equation (4), the closer the pixel position p of interest to the peripheral pixel position q, the larger the f-number. The smaller the difference between I2p of the pixel position p of interest and I2q of the peripheral pixel position q, that is, the closer the pixel values of the pixel of interest and the peripheral pixels are in the shaping image, the more the g weight (smoothing weight) of the peripheral pixels becomes. growing. The output obtained by weighting and adding the signal value I1q of the input distance map with the f · g weight is the signal value Jp of the output distance map after shaping.

FIG. 11 (D) represents the profile 1100 pf at position x of the frame image of FIG. 11 (A). The acquisition position of the profile 1100 pf is shown in line 1100 of FIG. 11 (A). Profile 1100pf is a step shape that changes at position xa. Further, FIG. 11 (E) shows the profile at the position x of the distance map of FIGS. 11 (B) and 11 (C). The acquisition position of the profile 1101pf is shown in line 1101 of FIG. 11 (B), and the acquisition position of the profile 1102 pf is shown in line 1102 of FIG. 11 (C). The position of line 1101 and the position of line 1102 are the same.

In the input distance map shown in FIG. 11B, the signal value indicating the distance of the subject protrudes outside the contour of the subject image. The change in the profile 1101 pf shown by the broken line in FIG. 11 (E) deviates from the position xa in which the profile 1100 pf in FIG. 11 (D) changes. The shaping process by the bilateral filter is executed, and the profile 1102pf corresponding to the shaped distance map of FIG. 11C is obtained. The changing position of the profile 1102pf coincides with the changing position xa of the profile 1100pf in FIG. 11 (D), and the shape matches the contour of the subject image. That is, the profile 1102pf is a step shape that greatly changes at the position xa.

In S903 of FIG. 9, the focus subject extraction unit 802 extracts the focus subject. A specific description will be given with reference to FIG. FIG. 12A exemplifies an input frame image, and FIG. 12B exemplifies a distance map after shaping. FIG. 12C exemplifies the extraction characteristics, and FIG. 12D exemplifies the extraction result of the focused subject.

The extraction characteristics of FIG. 12C are applied to the shaped distance map shown in FIG. 12B. The horizontal axis of FIG. 12C represents the distance of the subject, and the vertical axis represents the output value of the extraction result. The extraction characteristic is a characteristic that outputs the maximum value only for a distance within a predetermined focus plane distance range, and outputs zero or a minimum value for other distances. The extraction result output by applying the extraction characteristics is shown in FIG. 12 (D). Only the main subject that is in focus is extracted.

In S904 of FIG. 9, the blur processing unit 803 performs blur processing of the background image. The blurring process will be described with reference to FIG. FIG. 13 (A) shows an example of the shape of the kernel of the blur filter, and FIG. 13 (B) illustrates an image extracted from the focus plane. FIG. 13C exemplifies a blur filter for the position of interest.

In order to simulate the round blur caused by the lens, the shape of the kernel of the blur filter is substantially circular as shown in FIG. 13 (A), and the filter weight (weighting coefficient value) is constant. In this embodiment, the filter size is 5 × 5. The blur filter is applied to each image of the input frame image. At that time, the blur processing unit 803 refers to the focus surface extraction image and controls so that the filter processing is performed only on the background portion. FIG. 13B illustrates a case where the position of interest P is the position on the right side of the subject. In this case, the blur filter performs a filter operation on the position shown by the symbol ◯ in FIG. 13 (C). The position shown by the X symbol is a position belonging to the image area of the focused subject. If these positions are included in the filter calculation target, the contour portions are mixed and the image quality may deteriorate. Therefore, control is performed so that only the background portion is filtered. By the above processing, an image obtained by blurring only the background area other than the focused subject can be obtained.

Post-adjustment is performed in S905 of FIG. This process is a process of adjusting the blur intensity when the blur intensity does not match the user's preference as a result of confirming whether or not the blur intensity matches the user's preference. As a result, it is possible to obtain a blurred image of the degree desired by the user. The intensity of blurring can be adjusted by changing the number of TAPs in the filter kernel shown in FIG. 13 (A). The post-blurring process executed by the image processing unit 105 is performed according to a user's instruction after shooting. Finally, the image data generated by the image processing unit 105 is recorded on the recording medium by the recording unit 109.

In the present embodiment, it is dynamically switched whether or not additional information for region extraction is acquired at the same time according to the distance difference between the main subject and the background. Therefore, it is possible to reduce the burden on the user due to the enormous amount of recording, and to acquire the information for the post-blurring process in advance. By recording information for highly accurate area extraction only for frames that are likely to require post-processing, it is possible to perform image recording with high user benefits while reducing the recording capacity. .. Here, in the present embodiment, when the distance difference between the main subject and the background is large, only the captured image is recorded as the normal mode, but the method of responding to the distance difference between the main subject and the background is not limited to this. When the distance difference between the main subject and the background is larger than the threshold value, the background blurring process is desired to emphasize the distance difference, so recording is performed in the high quality mode.

In the present embodiment, based on the distance difference between the main subject and the background, it is determined whether or not additional information is acquired from the comparison result between the distance difference and the threshold value. Without being limited to this, for example, depth information may be acquired from the F value, and the depth information may be used as an element of determination. Further, in the present embodiment, as a distance map calculation method, the distance is calculated from the parallax of the pupil split image. Without being limited to this, the distance may be acquired by using, for example, a contrast AF (autofocus) evaluation value or the like. These matters are the same in the embodiments described later.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In the first embodiment, it is assumed that the recorded image is subjected to post-blurring processing. In the second embodiment, for example, it is assumed that a frame image in which the main subject is darkened by backlight is subjected to gradation correction processing for each region later. When the main subject darkened by backlight and the other background area are processed with the same gradation conversion characteristics, the dark part of the background becomes extremely bright, resulting in an unnatural image. The significance of performing gradation correction for each region is to correct the main subject region and the background region in the image with different gradation characteristics to suppress unnaturalness. On the other hand, this process requires an advanced area extraction process.

In the processing in this embodiment, the operation of the recording mode control unit 104 and the operation of the image processing unit 105 related to post-processing are different from those in the first embodiment. Hereinafter, the parts where the processing is different from that of the first embodiment will be mainly described, and the same configurations as those of the first embodiment will be described in detail by using the already used reference numerals. The omission of such a description is the same in the embodiments described later.

The operation of the recording mode control unit 104 will be described with reference to FIGS. 14 and 15. FIG. 14 is a processing block diagram of the recording mode control unit 104 in the second embodiment. The differences from the configuration shown in FIG. 2 are as follows.

The main subject Bv value calculation unit 1402 acquires the information of the main subject area from the main subject area detection unit 201 and calculates the Bv value of the main subject. The Bv value is an exposure value representing a brightness difference with respect to a target brightness value in the region of interest. The background Bv value calculation unit 1403 acquires the information of the background area and calculates the background Bv value. The exposure step calculation unit 1404 acquires each Bv value of the main subject and the background, and calculates the exposure step between the main subject and the background. The threshold value calculation unit 1405 calculates the threshold value for the exposure step from the Bv value of the main subject. The recording mode determination unit 1406 acquires the exposure level difference between the main subject and the background calculated by the exposure level difference calculation unit 1404, and the threshold value calculated by the threshold value calculation unit 1405. The recording mode determination unit 1406 compares the exposure step with the threshold value and determines the recording mode. Details of the processing of each part will be described later.

FIG. 15 is a flowchart illustrating the processing of the recording mode control unit 104.
First, in S1501, the main subject area detection unit 201 detects the main subject area in the image. In the next S1502, the main subject Bv value calculation unit 1402 calculates the Bv value of the main subject. When the Bv value of the main subject is expressed as Bv_obj, this is calculated by the following equation (5).
Bv_obj = log2 (Y_obj / Y_obj_target) ・ ・ ・ (5)
Log2 in Eq. (5) is a logarithmic function with 2 as the base. Y_obj is a representative luminance value of the main subject, and is calculated as an average value of the luminance values of the main subject area. Y_obj_target is the target luminance value of the main subject area. The target luminance value is a luminance value regarded as proper exposure, and is a predetermined value. The target luminance value may be changed depending on whether or not the main subject is a person. From equation (5), the darker the brightness of the main subject, the smaller the Bv value. For example, when the representative luminance value of the main subject is 1/2 of the target luminance value, the Bv value is -1. When the representative luminance value of the main subject is twice the target luminance value, the Bv value becomes +1.

In S1503, the background Bv value calculation unit 1403 calculates the background Bv value. When the background Bv value is expressed as Bv_back, this is expressed by the following equation (6).
Bv_back = log2 (Y_back / Y_back_target) ・ ・ ・ (6)
Y_back in the equation (6) is a representative luminance value of the background, and is calculated by the average value of the luminance values of the background region and the like. Y_back_target is the target luminance value of the background area.

In the next S1504, the exposure step calculation unit 1404 calculates the exposure step (denoted as delta_Bv) between the main subject area and the background area by the following equation (7).
delta_Bv = | Bv_obj --Bv_back | ・ ・ ・ (7)
As can be seen from the equations (5) to (7), the exposure step increases as the background is bright and the main subject is dark, or vice versa, that is, the wider the D range (dynamic range) when focusing on the main subject. The value of delta_Bv increases. The narrower the D range, the smaller the value of delta_Bv. In this embodiment, an exposure step is used as an evaluation value indicating the D range of the frame.

Next, in S1505, the threshold value calculation unit 1405 calculates the threshold value for the exposure step from the main subject Bv value. An example of threshold calculation shown in FIG. 16 will be described. In the graph shown in FIG. 16, the horizontal axis represents the main subject Bv value, and the vertical axis represents the exposure step threshold. In the example of FIG. 16, the threshold value corresponding to Bv_min is TH_min, and the threshold value when the main subject Bv value is zero or more is TH_max. An example in which linear interpolation is performed by a linear equation in a section from Bv_min to zero is shown, but higher-order interpolation processing of a second order or higher may be performed.

As shown in FIG. 16, qualitatively, when the main subject Bv value is a negative value, that is, when the main subject is darker than the proper exposure, it becomes necessary to perform the gradation correction process later. Therefore, the threshold value for the exposure step is set small. On the other hand, when the main subject Bv value is near zero, the main subject is close to the proper exposure. Therefore, the need for gradation correction processing is reduced, and a large threshold value is set. Further, when the main subject Bv value is a positive value, the main subject is too bright. In this case, the threshold value is set high (TH_max) in order to exclude the gradation correction main processing.

The recording mode determination unit 1406 in S1506 compares the threshold value calculated by the threshold value calculation unit 1405 with the exposure step, and determines the recording mode. When the exposure step is equal to or greater than the threshold value, the recording mode determination unit 1406 determines that there is a high possibility that gradation correction will be required later, and proceeds to S1507 to perform recording processing in the high quality mode. In the high quality mode, the data of the frame image, the distance map, and the RAW image are recorded. The reason for recording the RAW image before image processing is to perform gradation correction before nonlinear processing such as gamma conversion. On the other hand, when the exposure step is less than the threshold value, the recording mode determination unit 1406 determines that it is not necessary to perform gradation correction later, and proceeds to S1508 to perform recording processing in the normal mode. The processing of the recording mode control unit 104 is performed at the time of shooting.

Next, a region-specific gradation correction process performed according to a user's instruction after recording will be described. This processing is mainly performed by the image processing unit 105, but the operation unit 107 and the system control unit 106 that receive the user's instruction are also involved. Hereinafter, the area-specific gradation correction processing performed after recording is referred to as "post-correction processing". The post-correction process will be described with reference to FIGS. 17 and 18.

FIG. 17 is a processing block diagram of the image processing unit 105 relating to the post-correction processing. The differences from the configuration shown in FIG. 8 are as follows.
The first gradation characteristic calculation unit 1703 acquires frame image data and calculates the first gradation characteristic. The first gradation characteristic is the gradation characteristic of the main subject area. Further, the second gradation characteristic calculation unit 1704 acquires the frame image data and calculates the second gradation characteristic. The second gradation characteristic is the gradation characteristic of the background region. The first gradation correction unit 1705 uses the first gradation characteristic to perform gradation correction processing on the input frame image. The second gradation correction unit 1706 uses the second gradation characteristic to perform gradation correction processing on the input frame image. The compositing unit 1707 acquires the information of the focused subject from the focused subject extraction unit 802, the first image in which the first gradation correction unit 1705 has performed gradation correction, and the second gradation correction unit 1706 perform gradation correction. The second image is acquired and the compositing process is performed. Details of the processing of each part will be described later.

FIG. 18 is a flowchart illustrating the post-correction process.
First, in S1801, the imaging device receives an instruction for post-correction processing from the user. FIG. 19 shows a display example, and the display unit 108 shows the user that the post-correction processing is possible for the high-quality recording frame capable of the post-correction processing. In addition, display and input processing for asking the user to instruct whether or not to perform post-correction processing is performed. When the user selects to perform gradation correction processing after shooting during video reproduction, the system control unit 106 receives an instruction from the operation unit 107 and instructs the image processing unit 105 to perform processing after S1802. ..

In S1802, the distance map shaping unit 801 performs the distance map shaping process, and in S1803, the focus subject extraction unit 802 extracts the focus subject. In S1804, the first gradation characteristic calculation unit 1703 calculates the first gradation characteristic. In S1805, the second gradation characteristic calculation unit 1704 calculates the second gradation characteristic. The calculation process of the first gradation characteristic and the second gradation characteristic will be described with reference to FIG. FIG. 20A illustrates an input frame image and shows a main subject area and a background area in the image. That is, the main subject area detection unit 201 detects the main subject area and the background area in the input frame image separately. Separate gradation correction processing is performed on the main subject area and the background area. First, a uniform gain process is performed on the main subject area. The reason for using uniform gain processing is that when this function is applied, there is a high possibility that the main subject is uniformly darkened due to backlight, shade, or the like. On the other hand, for the background area, gain processing for each brightness is performed. The reason is that subjects having various brightnesss generally exist in the background area and the D range is wide. Gradation compression is executed so as not to impair the gradation of each region. The gain characteristics are not limited to the above concept, and the characteristics of any shape can be taken.

FIG. 20B illustrates the characteristics of the gain (vertical axis) with respect to the input luminance (horizontal axis) in the main subject area. The gain takes a constant value (denoted as Gain_obj). The gain Gain_obj is calculated by the following equation (8) in order to correct the average luminance value Y_obj of the main subject area to be the target luminance value Y_obj_target for proper exposure.
Gain_obj = Y_obj_target / Y_obj ・ ・ ・ (8)

FIG. 20C is a solid line graph showing the characteristics of the input / output luminance when the processing is performed with the gain characteristics shown in FIG. 20B. That is, this characteristic is the first gradation characteristic. The dotted line shows the case where the ratio of the input luminance value and the output luminance value is 1: 1. The slope of the graph line of the first gradation characteristic is larger than the slope of the graph line shown by the dotted line.

FIG. 20D exemplifies a representative luminance value in the luminance histogram of the background region. The horizontal axis represents the input brightness, and the vertical axis represents the frequency (frequency). FIG. 20E illustrates the gain characteristic with respect to the input luminance in the background region. The gain changes according to the input brightness. The second gradation characteristic calculation unit 1704 calculates the representative luminance values of the dark portion side and the bright portion side of the input luminance before calculating the gain characteristic. In the calculation process, as shown in FIG. 20D, a luminance histogram of the background region is acquired. The representative luminance value Y_back_low is a representative luminance value of the dark portion calculated when the number of pixels of a predetermined ratio is counted from the minimum luminance. The representative luminance value Y_back_high is a representative luminance value of the bright part calculated when the number of pixels of a predetermined ratio is counted from the maximum luminance. The maximum gain (denoted as Gain_back) for the background area is calculated by the following equation (9) so that the average luminance value Y_back of the background region becomes the target luminance value Y_back_target for proper exposure.
Gain_back = Y_back_target / Y_back ・ ・ ・ (9)

In the gain characteristic shown in FIG. 20 (E), Gain_back is a constant value in the section where the input luminance value is Y_back_low or less, and the minimum gain amount is 1 in the section where the input luminance value is Y_back_high or more. In the section between T_back_low and Y_back_high, it is a characteristic that monotonically decreases according to the input luminance value. FIG. 20 (F) is a solid line graph showing the characteristics of the input / output luminance when the processing is performed with the gain characteristics shown in FIG. 20 (D). This characteristic is the second gradation characteristic. The solid line graph showing the second gradation characteristic has a polygonal line shape protruding upward at Y_back_low, and the dotted line graph line (the ratio of the input luminance value to the output luminance value is 1: 1 in the section above Y_back_high). The shape matches the case).

In S1806 of FIG. 18, the first gradation correction unit 1705 uses the first gradation characteristic to perform gradation correction processing of the input frame image. In S1807, the second gradation correction unit 1706 uses the second gradation characteristic to perform gradation correction processing of the input frame image. These processes are processes for converting the luminance value of the input frame image with the gradation conversion characteristics illustrated in FIGS. 20 (C) and 20 (F), respectively. In the next S1808, the synthesizing unit 1707 synthesizes the two images that have undergone the gradation correction process. The processing of the synthesis unit 1707 will be described with reference to FIG.

FIG. 21 (A) illustrates an input frame image, and FIG. 21 (B) illustrates a focus plane image extracted by the focus subject extraction unit 802. In the compositing process, an image in which gradation correction is performed with the first gradation characteristic is output for the focus plane region in the image shown in the white region in FIG. 21 (B). Further, for the non-focus plane region (background region) in the image shown by the black region in FIG. 21B, an image in which gradation correction is performed by the second gradation characteristic is output. When this processing is performed, the image after the composition processing becomes the image shown in FIG. 21 (C) with respect to the input frame image of FIG. 21 (A). FIG. 21C shows an image in which gradation correction processing is performed on a main subject region, which is a focus plane, and a background region, which is a non-focus plane region, with different gradation conversion characteristics. Finally, the data of the image to which the gradation correction for each region is performed is recorded on the recording medium by the recording unit 109.

In the present embodiment, it is dynamically switched whether or not to acquire additional information for area extraction at the same time according to the exposure step (difference in brightness) between the main subject area and the background area. Therefore, it is possible to reduce the burden on the user due to the enormous amount of recording, and to acquire the information for the post-correction processing (gradation correction). In this embodiment, when the exposure step between the main subject area and the background area is larger than the threshold value, the image and the depth distribution information are recorded in the high quality mode. However, the present invention is not limited to this, and even if the subject is recorded in the high quality mode, for example, when the exposure step between the main subject area and the background area is smaller than the threshold value, the subject is photographed with appropriate brightness than when it is larger. Good. At this time, when the exposure step is larger than the threshold value, the image is recorded without recording the depth distribution information in the normal mode.

[Third Embodiment]
Next, a third embodiment of the present invention will be described. The present embodiment is premised on the post-blurring process described in the first embodiment, and an object thereof is to further reduce the recording capacity particularly at the time of recording information. In the first embodiment, information for post-processing is acquired for, for example, a frame determined that the background image is not sufficiently blurred based on the distance difference between the main subject and the background. However, depending on the shooting scene, the distance between the main subject and the background may always be short. In such a case, information for post-processing is acquired over almost all frames. As a result, the recording capacity of the captured image may become enormous. Therefore, in the present embodiment, a process of further narrowing down the frames to be recorded in the high quality mode will be described in order to maintain an appropriate recording capacity. That is, in the present embodiment, the determination of whether it is necessary to reduce the recording capacity and the determination of the recording mode by the score determination of the subject are executed by combining a part or all of them with the first and second embodiments, respectively. It can be done.

FIG. 22 is a block diagram showing a configuration applicable to the imaging device of the present embodiment. The difference from the configuration shown in FIG. 1 is that the recording information adjusting unit 2210 is added. FIG. 23 is a flowchart showing the processing of the recording information adjusting unit 2210. The recording information adjusting unit 2210 will be described with reference to FIG. 23. The following processing is executed after the recording unit 109 records a series of moving image data in the recording mode controlled by the recording mode control unit 104.

In S2301, the recording information adjusting unit 2210 acquires the recording capacity (denoted as MEM) of the image, and calculates the maximum recording capacity (denoted as MEM_MAX) in S2302. The maximum recording capacity MEM_MAX is calculated by the following formula (10).
MEM_MAX = MEM_FRAME × NUM_FRAME × k1 × k2 ・ ・ ・ (10)
In equation (10), MEM_FRAME is the recording amount per frame, and in this case, inter-frame compression and intra-frame compression are not performed. NUM_FRAME is the total number of frames of the moving image to be processed. k1 is a predetermined compression rate and takes a value of 1 or less. The actual compression rate changes depending on the shooting scene, but here it is set to a predetermined compression rate. k2 is an allowable value of the capacity increase rate by recording additional information, and takes a value of 1 or more.

In the next S2303, the recording information adjusting unit 2210 compares the recording capacity MEM with the maximum recording capacity MEM_MAX. If MEM is less than or equal to MEM_MAX, the processing ends without adjusting the recorded information. If the MEM is larger than MEM_MAX, the recording information adjusting unit 2210 determines that it is necessary to adjust the recorded information, and proceeds to S2304.

In the present embodiment, when it is determined that it is necessary to reduce the recording capacity, a process of further narrowing down the frames to be recorded in the high quality mode is performed based on the size information and the position information of the main subject. The reason is that the frame in which the user wants to blur the background after shooting and record it as a still image is a frame in which the main subject is captured in good condition. Specifically, the frame acquired by the narrowing down process is a frame in which the image area of the main subject exists near the center of the captured image and the size of the main subject is large.

In S2304 of FIG. 23, the recording information adjusting unit 2210 acquires the size information and the position information of the main subject, and in the next S2305, calculates the main subject score based on the size information and the position information of the main subject. It is assumed that the processing after S2304 is performed on all the recorded frames. The processing of S2304 and S2305 will be described with reference to FIG. 24. FIG. 24A illustrates an input frame image. The coordinates of the position of the center of gravity of the main subject area are expressed as (X, Y), the height of the main subject area is expressed as Height, and the width of the main subject area is expressed as Width.

In S2304, a process of extracting the main subject area in a rectangular shape from the input frame image shown in FIG. 24A and acquiring the width width and the height height is executed. Further, the coordinates (X, Y) of the center of gravity of the extracted main subject area are acquired. From the acquired information, the normalized size (denoted as Size) corresponding to the area is calculated by the following equation (11).
Size = Width x Height / Size_all ・ ・ ・ (11)

Size_all in Eq. (11) is a normalization coefficient for normalizing so as not to depend on the image size. For example, let Size_all be the area of the entire image. In this case, Size indicates the ratio of the area of the main subject area to the area of the entire image. When the coordinates of the center position of the image are expressed as (Xc, Yc), the normalized distance (denoted as Dist) from (Xc, Yc) to the coordinates of the center of gravity (X, Y) of the main subject area is as follows (denoted as Dist). 12) Calculated by equation.

R in Eq. (12) is a normalization coefficient corresponding to the distance from the center position of the image to the edge of the image. That is, the normalized distance Dist indicates the ratio of the distance difference between the coordinates (Xc, Yc) and (X, Y) to the distance from the center position of the image to the edge of the image.

Next, in S2305, the main subject score is calculated. This will be described with reference to FIGS. 24 (B) and 24 (C). FIG. 24B illustrates the calculation characteristics of the distance score. The horizontal axis represents the normalized distance Dist, and the vertical axis represents the distance score (denoted as Score_Dist). FIG. 24C illustrates the calculation characteristics of the size score. The horizontal axis represents the normalized size Size, and the vertical axis represents the size score (denoted as Score_Size).

In the present embodiment, first, the distance score Score_Dist and the size score Score_Size are calculated from the calculated normalized distance information and the normalized size information according to the characteristics shown in FIGS. 24 (B) and 24 (C), respectively. Regarding the characteristic of the distance score Score_Dist shown in FIG. 24 (B), the closer the image of the subject is to the central portion of the image, the larger the score, so that the characteristic is monotonically decreasing with respect to Dist. FIG. 24B exemplifies the characteristic of linear interpolation between two points by a linear equation. In the range where the Dist value is smaller than the first threshold value D1 for the normalized distance Dist, the distance score Score_Dist is constant. Further, in the range where the Dist value is larger than the second threshold value D2 for the normalized distance Dist, the distance score Score_Dist is constant. When the Dist value is equal to or greater than the first threshold value and equal to or less than the second threshold value, the value of the distance score Score_Dist decreases linearly as the Dist value increases.

Regarding the characteristic of the size score Score_Size shown in FIG. 24C, the score increases as the image size of the subject increases, so that the characteristic increases monotonically with respect to Size. FIG. 24C exemplifies a characteristic in which two points are linearly interpolated by a linear equation. The size score Score_Size is constant in the range where the Size value is smaller than the first threshold value S1 for the normalized size Size. Further, in the range where the Size value is larger than the second threshold value S2 with respect to the normalized size Size, the size score Score_Size is constant. When the Size value is equal to or greater than the first threshold value and equal to or less than the second threshold value, the value of the size score Score_Size increases linearly as the Size value increases.
The characteristics shown in FIGS. 24 (B) and 24 (C) are examples, and the interpolation process may be performed by setting three or more points.

Next, from the calculated distance score Score_Dist and size score Score_Size, the main subject score (denoted as Score) is calculated by the following equation (13).
Score = w_d × Score_Dist + w_s × Score_Size ・ ・ ・ (13)
In equation (13), w_d and w_s are arbitrary weighting coefficients, respectively.

In S2306 of FIG. 23, the recording information adjusting unit 2210 narrows down high-quality recording frames. In S2305, the main subject score Score is calculated over all frames. The recording information adjusting unit 2210 compares the main subject score Score with a predetermined threshold value (denoted as TH_Score). The recording process in the high quality mode is executed for the frame whose main subject score Score value exceeds the threshold value TH_Score. Frames whose main subject score Score value is less than or equal to the threshold value are in the normal mode, and additional information is deleted and is not recorded. A specific description will be given with reference to FIG. The horizontal axis represents the threshold value TH_Score, and the vertical axis represents the recording capacity MEM. As the threshold value TH_Score increases, the number of high-quality frame images (number of frames) decreases, so that the recording capacity MEM decreases. Let TH_Score_min be the maximum threshold value at which the recording capacity MEM falls below the upper limit value MEM_MAX calculated in S2302. By narrowing down high-quality frames using the threshold value TH_Score_min, it is possible to acquire moving images while keeping the recording capacity of moving images below MEM_MAX.

In the present embodiment, the increase in the recording capacity of the moving image can be suppressed by the processing of the recording information adjusting unit 2210. In this embodiment, the position information and the size information of the main subject area are used as indexes for narrowing down the high-quality frames.
Further, as another embodiment, the degree of scene change may be used as another index for switching the recording mode. The degree of scene change is an index showing the magnitude of change in the scene when the image changes between different frames (for example, the current frame and the previous frame). The degree of scene change is calculated by aligning a plurality of time-series images, for example, the current frame and the previous frame, and calculating the difference between the images. When there is almost no change in the image between two frames that are continuous in time, a process of thinning out additional information in the high quality mode is executed. When there is a large change in the image between two frames that are continuous in time, a process of recording additional information related to both frames is executed.
In the present embodiment, the determination based on the subject score and the determination based on the degree of scene change described above are performed when it is determined that the recording capacity needs to be reduced, but the determination is not limited to this. For example, the recording mode may be switched using the subject score or the degree of scene change as a method for determining a high-quality frame without detecting or determining the recording capacity.

[Fourth Embodiment]
Next, a fourth embodiment of the present invention will be described. In the present embodiment, the post-blurring process described in the first embodiment is premised, and it is an object of the present invention to reduce the motion blur of each frame. A control for changing the shutter speed and frame rate when recording in the high quality mode according to the movement of the main subject will be described.

With respect to the frame for which a high-quality still image is generated by post-processing, it is preferable that the subject does not move and the subject is stationary. In the exposure control when capturing a moving image, the shutter speed is controlled so that the movement of the subject image does not appear to be discontinuous when the frame image is continuously viewed. That is, control is performed so that the shutter speed does not become too high. On the other hand, when one frame of the image of the moving image frame captured in this way is viewed as a still image, there is a possibility that motion blurring may occur depending on the moving speed of the subject. Therefore, it is an object of the present embodiment to suppress the motion blur of the subject with respect to the exposure control of the frame image acquired in the high quality mode, particularly the shutter speed.

FIG. 26 is a block diagram showing a configuration applicable to the imaging device of the present embodiment. The difference from the configuration shown in FIG. 1 is that the exposure condition control unit 2606 is provided. FIG. 27 is a flowchart showing the processing of the exposure condition control unit 2606. The exposure condition control unit 2606 will be described with reference to FIG. 27. The following processing is performed after the processing of the recording mode control unit 104, and is performed every frame or at a predetermined frame interval.

First, in S2701, a determination process is performed as to whether or not the recording mode is a high quality mode. When the recording mode is the normal mode, the process proceeds to S2705, and the shooting operation is performed with the exposure of the moving image in the normal mode also for the next frame one frame time after the current frame. On the other hand, when the recording mode is the high quality mode, the process proceeds to S2702, and the exposure control is performed according to the movement of the main subject. In S2702, the exposure condition control unit 2606 calculates the motion vector of the main subject. The motion vector calculation method is performed by a known pattern matching process.

In the next S2703, the exposure condition control unit 2606 calculates the shutter speed of the next frame. The shutter speed of the next frame is expressed as Tv, and the magnitude of the motion vector calculated in S2702 is expressed as v (unit: pixel). The frame interval is expressed as T_frame (1/60 seconds when the frame rate is 60 fps). The shutter speed Tv is calculated from v and T_frame by the following equation (14).
Tv = T_frame / v ・ ・ ・ (14)
Equation (14) means that Tv is calculated as a shutter speed at which the amount of movement of the main subject image is 1 pixel within the shooting time. In other words, Tv is the shutter speed at which motion blur is within one pixel. For example, let T_frame be 1/60 second and v be 6 pixels. In this case, the shutter speed corresponding to the movement of one pixel of the main subject image is Tv = 1/360 seconds. Therefore, according to the object of this embodiment, a value smaller than Tv calculated by Eq. (14) may be used as the shutter speed.

In S2704, the exposure condition control unit 2606 determines other exposure conditions and the frame rate. The other exposure conditions are, specifically, the sensitivity and the aperture value. When the aperture value is changed, the depth of field changes and the continuity with the front and rear frames is lost. Therefore, in the present embodiment, control is performed to keep the exposure constant by changing the sensitivity for the change in the shutter speed Tv. Further, as the shutter speed Tv increases, control is performed to change the frame rate. For example, when the Tv value is 1/120 second or less, the process of changing the frame rate from 60 fps to 120 fps is executed. This process has the effect of making it difficult to miss the decisive moment of a fast-moving subject. Finally, the exposure condition control unit 2606 controls to perform the imaging process of the next frame after feeding back and reflecting the exposure conditions determined in S2704 or S2705 to the control of the imaging optical system 101 and the imaging unit 102. ..

In the present embodiment, the processing of the exposure condition control unit 2606 enables shooting at an optimum shutter speed according to the speed of the subject (moving object). In addition, when viewing as a moving image, the subject is electronically subjected to the amount of movement of the subject in order to avoid the movement of the moving object appearing discontinuous due to the large value of the shutter speed Tv related to the image frame. Processing such as adding blur to the image is performed.

Further, in the first, second, third and fourth embodiments described above, the recording mode is basically determined for each frame and switched for control, but the present invention is not limited to this. For the purpose of reducing the amount of recorded data, control may be performed by periodically switching the recording mode, such as manually or automatically recording in a high quality mode every predetermined frame.
When the setting is performed manually, for example, if the predetermined number of frames is set to 5 frames by a user operation via the operation unit 107, 1 frame is recorded in 5 frames, and the depth distribution information is recorded together with the image as a high quality mode. Alternatively, the recording cycle in the high quality mode may be determined according to the frame rate of imaging set by the user.
When the setting is automatically performed, at least one of determinations such as the distance difference between the main subject and the background, the exposure step, the subject score, and the degree of scene change in each of the above-described embodiments is periodically performed. Then, the cycle of recording in the high quality mode may be determined, and control may be performed so that recording in the high quality mode is performed in that cycle until the next determination.

<Recording format pattern in each embodiment>
In each of the above-described embodiments, any of the following formats may be used for recording the plurality of captured images (frames) and the depth distribution information corresponding to some of the frames.
FIG. 28 shows an image of each pattern of a plurality of images (frames) captured and a format for recording depth distribution information corresponding to some of the frames. That is, as the recording format, as shown in FIG. 28 (A), a plurality of images 1, 2 and 3 sequentially captured and acquired, and distance maps 1 and 3 corresponding to the images 1 and 3 are all separate files. It is recorded as. In this case, it is preferable to record the information relating the image and the distance map (or the information that there is no corresponding distance map) in the header of each image file, and record the information of the corresponding image on the distance map side as well.

Further, as shown in FIG. 28B, each image is associated (may be encoded) as one moving image file as a continuous image, and a plurality of distances are synchronized with this moving image file. Each map may be recorded individually as a distance map file. The time code of the corresponding moving image is recorded in the distance map, and it can be read out and used as needed in association with the image when editing the moving image including cutting out the still image.

Further, as shown in FIG. 28C, each image is associated (may be encoded) as one moving image file as a continuous image, and a plurality of distances are synchronized with this moving image file. The map may be recorded as a separate file. In this case, the time code synchronized with the time code of each frame of the moving image may be recorded in the corresponding distance map and recorded as one file. Compared with the form of FIG. 28B, the distance map is also made into one file so that it can be easily handled as a pair with the moving image file, and if necessary, the distance maps can be encoded by using a known coding technique. As a result, the amount of data can be expected to be reduced.

Further, as shown in FIG. 28 (D), the image and the corresponding distance map before or after the image may be connected and recorded, and one moving image file may be formed and recorded as a whole. In this format, the distance Map, which is the depth distribution information corresponding to the image to which the recording process has been performed, is recorded as the metadata of the image data before or after the image data. This format has advantages such as being able to be handled by one file and being easy to access because the distance map corresponding to the image is also adjacent.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

As described above, according to the present invention, it is possible to perform highly convenient image recording while suppressing the recording capacity.

101 ... Imaging optical system 102 ... Imaging unit 104 ... Recording mode control unit 105 ... Image processing unit 106 ... System control unit 109 ... Recording unit 2210 ... Recording information adjustment unit 2606・ ・ ・ Exposure condition control unit

Claims (28)

An image processing device including a recording means for acquiring a plurality of image data and recording them on a recording medium.
An acquisition method for acquiring depth distribution information of a subject corresponding to image data,
A first mode in which the image data and the depth distribution information corresponding to the image data are recorded on the recording medium by the recording means, and the image data is recorded by the recording means without recording the depth distribution information. A control means for switching between a second mode for recording on a medium and performing recording processing of the plurality of image data is provided.
The control means switches between the first mode and the second mode at a cycle corresponding to the frame rate of imaging of the plurality of image data set by the user to perform the recording process of the plurality of image data. An image processing device characterized by. The control means switches between the first mode and the second mode in the recording of the plurality of image data to perform recording processing, and the plurality of image data are used as moving images in the recording medium by the recording means. The image processing apparatus according to claim 1, wherein the image processing apparatus is recorded. 2. The recording means according to claim 2, wherein the depth distribution information corresponding to the plurality of image data and the image data recorded in the first mode is recorded in one moving image file. The image processing apparatus described. The recording means records the depth distribution information corresponding to the image data recorded in the first mode in the moving image file before or after the image data as metadata of the image data. The image processing apparatus according to claim 3. The control means acquires the depth distribution information including the depth information of the main subject and the background among a plurality of subjects, and the difference between the depth information of the main subject and the depth information of the background included in the depth distribution information is within the threshold value. The image processing apparatus according to claim 1, wherein the recording process is performed in the first mode. The image processing apparatus according to claim 5, further comprising a threshold value calculating means for calculating the threshold value from the focal length of the imaging optical system, the depth information of the main subject included in the depth distribution information, and the allowable defocus amount. .. 5. The control means is characterized in that it determines whether or not to perform the recording process in the first mode based on the comparison result between the difference and the threshold value and the F value of the imaging optical system. The image processing apparatus according to 6. The claim is characterized in that the control means calculates an evaluation value indicating a dynamic range of a frame and compares the evaluation value with a threshold value to determine whether or not to perform recording processing in the first mode. Item 1. The image processing apparatus according to item 1. As the evaluation value, the control means calculates an exposure step between the main subject area and the background area related to the main subject among the plurality of subjects, and when the exposure step is equal to or more than the threshold value, in the first mode. The image processing apparatus according to claim 8, wherein the recording process is performed, and when the exposure step is smaller than the threshold value, the recording process is performed in the second mode. An extraction means for extracting information on a main subject area related to a main subject among a plurality of subjects from the image data and depth distribution information recorded in the first mode.
The image processing apparatus according to claim 1, further comprising an image processing means that acquires information on a main subject area extracted by the extraction means and performs image processing of the image data. The image processing apparatus according to claim 10, wherein the image processing means determines a background area in the image from the information of the main subject area and performs blurring processing of the background image. An extraction means for extracting information on a main subject area related to a main subject among a plurality of subjects from the image data and depth distribution information recorded in the first mode.
An image processing means for acquiring information on the main subject area extracted by the extraction means and performing image processing on the image data is further provided.
The image processing means is characterized in that the main subject area and the background area in the image are determined from the information of the main subject area, and different gradation correction processing is performed on the main subject area and the background area. The image processing apparatus according to 1. Further provided with an adjusting means for adjusting the recording amount by acquiring the recording capacity of the moving image and narrowing down the frames to be recorded in the first mode when the recording capacity is equal to or more than the threshold value.
The image processing apparatus according to claim 1, wherein the adjusting means controls to record the narrowed-down image data of the frame and the depth distribution information. The control means compares a score calculated from at least one of the position information and size information of the main subject among the plurality of subjects with the threshold value, and in the frame where the score is larger than the threshold value, in the first mode. The image processing apparatus according to claim 1, wherein the recording process is performed, and the recording process is performed in the second mode in a frame whose score is equal to or less than the threshold value. The image processing according to claim 1, wherein the control means detects a change in a scene from a difference between images, and when the change in the scene is detected, performs recording processing in the first mode. apparatus. The image processing apparatus according to claim 1, wherein the control means performs recording processing in the first mode for each predetermined frame in a plurality of time-series images acquired by the acquisition means. A first aspect of the present invention, wherein the control means notifies that the moving image recorded in the first mode is the image recorded in the first mode when the moving image is reproduced. The image processing apparatus according to any one of 16. The depth distribution information is obtained by an image shift map based on the amount of parallax of a plurality of viewpoint images, a defocus map based on the amount of defocus for each area, a distance map showing the relative distance relationship of each subject in the image data, and the TOF method. The image processing device according to any one of claims 1 to 17, wherein the image processing device is any one of the distance information indicating the distance relationship from the acquired image pickup device to each subject. The acquisition means is characterized in that an image shift map, which is the depth distribution information corresponding to the image data, is calculated and acquired based on the parallax amount of the paired parallax images corresponding to the acquired image data. The image processing apparatus according to any one of claims 1 to 18. The claim is characterized in that the acquisition means calculates and acquires a defocus map which is the depth distribution information corresponding to the image data based on the defocus amount for each area of the acquired image data. The image processing apparatus according to any one of 1 to 18. The acquisition means obtains the relative distance relationship of each subject, which is the depth distribution information corresponding to the image data, based on the defocus amount for each region of the acquired image data and the image pickup optical system or the image sensor. The image processing apparatus according to any one of claims 1 to 18, wherein the image processing apparatus is calculated and acquired. The acquisition means uses the TOF method of measuring the delay time from the projection of light to the subject to the reception of reflected light and measuring the distance to the subject from the image pickup device which is the depth distribution information corresponding to the image data. The image processing apparatus according to any one of claims 1 to 18, wherein the subject distance to each subject is acquired. The image processing apparatus according to any one of claims 1 to 22, wherein the control means controls to record data of a RAW image before image processing in the first mode. The image processing apparatus according to claim 23, wherein the RAW image is an image that has not been subjected to image processing including demosaiking processing, white balance adjustment, color conversion processing, or gamma correction. The image processing apparatus according to any one of claims 1 to 24,
An imaging device including an imaging means for imaging a subject. An exposure condition control means for controlling an exposure condition when an image of a subject is imaged by the image pickup means and image data is generated is provided.
The exposure condition control means acquires the amount of movement of the subject in the image of the frame in the first mode, and determines the shutter speed and frame rate related to the imaging of the next frame of the frame from the amount of movement and the frame interval. The imaging apparatus according to claim 25, wherein one or more of them is determined. A control method for an image processing apparatus including a recording means for acquiring a plurality of image data and recording them on a recording medium.
The acquisition process to acquire the depth distribution information of the subject corresponding to the image data,
A first mode in which the image data and the depth distribution information corresponding to the image data are recorded on the recording medium by the recording means, and the image data is recorded by the recording means without recording the depth distribution information. It has a control step of switching between a second mode of recording on a medium and performing recording processing of the plurality of image data.
In the control step, the first mode and the second mode are switched at a cycle corresponding to the frame rate of imaging of the plurality of image data set by the user to perform the recording process of the plurality of image data. A control method characterized by. A program for causing a computer of an image processing apparatus to execute each step according to claim 27.