JP6817855B2 - Image processing equipment, imaging equipment, image processing methods, and programs - Google Patents

Description

The present invention relates to an image processing device, an imaging device, an image processing method, and a program.

An imaging device capable of simultaneously capturing a plurality of viewpoint images by recording a luminous flux that has passed through different pupil regions of a photographing lens is known. Patent Document 1 discloses an image pickup device using a two-dimensional image pickup element in which one microlens and a plurality of photoelectric conversion units are formed for one pixel. The divided photoelectric conversion unit is configured to receive light from different pupil region regions of the exit pupil of the photographing lens via one microlens, and performs pupil division. Patent Document 1 discloses that parallax is reduced by synthesizing a plurality of viewpoint images obtained from these divided photoelectric conversion units.

Japanese Unexamined Patent Publication No. 2012-147046

Since each pupil region is narrower than the entire exit pupil region, the aperture value corresponding to each viewpoint image is larger (darker) than the aperture value corresponding to the entire exit pupil. Further, the aperture value of each viewpoint image may differ depending on the image height due to the pupil deviation between the exit pupil of the photographing lens and the entrance pupil of the image sensor. However, the technique of Patent Document 1 does not consider the difference in aperture value when synthesizing a plurality of viewpoint images.

The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for adjusting the depth of field of a composite image by utilizing a difference in aperture value.

In order to solve the above problems, the present invention provides an acquisition means for acquiring a plurality of viewpoint images corresponding to a plurality of pupil partial regions in which the ejection pupil of the imaging optical system is divided, and a depth for specifying the depth of field. The designated means, the determination means for determining the aperture value of each viewpoint image in the partial region based on the optical information of the plurality of viewpoint images, and the composition based on the designated depth of field and the determined aperture value. Provided is an image processing apparatus comprising: a generation means for generating a composite image having the specified depth of field by synthesizing the partial regions of the plurality of viewpoint images at a ratio.

In addition, other features of the present invention will be further clarified by the accompanying drawings and the description in the form for carrying out the following invention.

According to the present invention, it is possible to adjust the depth of field of a composite image by utilizing the difference in aperture value.

The block diagram which shows the functional structure example of the digital camera 100 as an example of an image processing apparatus. The figure which shows the detailed structure of the image processing unit 125. The figure explaining the arrangement of the pixel and the sub-pixel of the image pickup device 107 which concerns on 1st Embodiment. (A) A plan view of the pixel 200G of FIG. 3 seen from the light receiving surface side (+ z side) of the image sensor 107, and (b) a cross-sectional view of the aa cross section of FIG. 4 (a) seen from the −y side. FIG. 4 is a diagram showing a correspondence relationship between a cross-sectional view of the aa section of the pixel 200G shown in FIG. 4A as viewed from the + y side and an exit pupil surface of the imaging optical system. The figure which shows the example of the light intensity distribution when the light is incident on the microlens 305 formed in the pixel 200G. The figure which shows the example of the pupil intensity distribution. The figure explaining the principle of the depth change processing. The figure explaining the pupil deviation in the peripheral image height of the image sensor 107. The figure explaining the eclipse in the peripheral image height of the image sensor 107. The figure explaining the conjugation relationship between the exit pupil surface of the photographing optical system and the photoelectric conversion part of the image sensor 107 arranged near zero image height. A flowchart of a series of operations related to changing the depth of a captured image. The flowchart which shows the detail of the depth change image processing (S104). The flowchart which shows the detail of the development process (S208). The figure which shows the example of the depth change UI. The figure which shows the example of the relationship between the slider bar operation and the composite ratio of the 1st viewpoint image and the 2nd viewpoint image. The figure which shows the example of the relationship between the image height and the effective F value of the 1st viewpoint image and the 2nd viewpoint image. The figure explaining the arrangement of the pixel and the sub-pixel of the image pickup device 107 which concerns on 2nd Embodiment. (A) A plan view of the pixel 200G of FIG. 18 viewed from the light receiving surface side (+ z side) of the image sensor 107, and (b) a cross-sectional view of the aa section of FIG. 19A seen from the −y side.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The technical scope of the present invention is determined by the scope of claims, and is not limited by the following individual embodiments. Also, not all combinations of features described in the embodiments are essential to the present invention. It is also possible to appropriately combine the features described in the separate embodiments.

In the following embodiment, as an example of the image processing device, a digital camera capable of acquiring a plurality of viewpoint images by pupil division will be used for description. However, the image processing device is not limited to the digital camera, and may be any device capable of processing the acquired plurality of viewpoint images. Such devices include, for example, mobile phones, game consoles, tablet terminals, personal computers, clock-type and eyeglass-type information terminals, surveillance systems, in-vehicle systems, medical systems such as endoscopes, and robots capable of providing images. Etc. are included. Further, in the following embodiment, an arbitrary device transmits a viewpoint image and operation contents to a server device having a processing function such as a processor on the Internet or a local network, and performs a part or all of the processing on the viewpoint image. It may include a configuration that runs on a server device. In this case, any device may include a configuration in which the processing result is received from the server device and displayed. The server device may include a virtual machine.

[First Embodiment]
(Overall configuration of digital camera 100)
FIG. 1 is a block diagram showing a functional configuration example of a digital camera 100 as an example of an image processing device. One or more of the functional blocks shown in FIG. 1 may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or MPU executing software. You may. It may also be realized by a combination of software and hardware. Therefore, in the following description, the same hardware can be realized as the main body even if different functional blocks are described as the main body of operation.

The first lens group 101 includes, for example, a zoom lens that constitutes the imaging optical system, is arranged at the tip of the imaging optical system, and is held so as to be able to move forward and backward in the optical axis direction. The shutter 102 includes a diaphragm, and by adjusting the aperture diameter thereof, the amount of light incident on the image sensor 107 at the time of photographing is adjusted. Further, the shutter 102 functions as a shutter for adjusting the exposure time when shooting a still image. The shutter 102 and the second lens group 103 constituting the imaging optical system integrally advance and retreat in the optical axis direction, and by interlocking with the advancing and retreating operation of the first lens group 101, a scaling action (zoom function) is performed. Make up.

The third lens group 105 includes, for example, a focal lens that constitutes an imaging optical system, and adjusts the focus by advancing and retreating in the optical axis direction. The optical element 106 includes an optical low-pass filter to reduce false colors and moire of captured images. The image pickup device 107 includes, for example, an image pickup device composed of a CMOS photo sensor and peripheral circuits, and is arranged on an imaging surface of an imaging optical system.

The zoom actuator 111 includes a driving device for generating an advance / retreat operation of the first lens group 101 to the third lens group 103, and by rotating a cam cylinder (not shown), the first lens group 101 to the third lens group 103 Is driven forward and backward in the optical axis direction. The aperture shutter actuator 112 includes a drive device that generates the operation of the shutter 102, and controls the aperture diameter of the shutter 102 and the shutter operation according to the control of the aperture shutter drive unit 128. The focus actuator 114 includes a driving device that causes the third lens group 105 to move forward and backward, and drives the third lens group 105 forward and backward in the optical axis direction to adjust the focus.

The lighting device 115 includes an electronic flash for illuminating a subject at the time of shooting, and is, for example, a flash lighting device using a xenon tube or a lighting device including an LED that continuously emits light. The auxiliary light emitting unit 116 includes an AF auxiliary light emitting device, projects an image of a mask having a predetermined aperture pattern onto the field of view through a projection lens, and focuses on a dark subject or a low-contrast subject. Improve detection capability.

The control unit 121 includes a CPU (or MPU), a ROM, and a RAM, and controls each part of the entire digital camera 100 by expanding and executing a program stored in the ROM in the RAM, and performs AF, photographing, and image processing. , Perform a series of operations such as recording. Further, the control unit 121 may include an A / D converter, a D / A converter, a communication interface circuit, and the like. Further, the control unit 121 includes a function as a display control unit that controls the display content displayed on the display unit 131, and may instead execute the process executed by the image processing unit 125.

The electronic flash control unit 122 includes a control circuit or a control module, and controls the lighting of the lighting device 115 in synchronization with the photographing operation. The auxiliary light driving unit 123 controls the lighting of the auxiliary light emitting unit 116 in synchronization with the focus detection operation. The image sensor driving unit 124 controls the image pickup operation of the image sensor 107, and A / D-converts the acquired image signal and transmits it to the control unit 121. The image processing unit 125 performs processing such as γ conversion, color interpolation, and JPEG compression of the image acquired by the image sensor 107.

The focus drive unit 126, the aperture shutter drive unit 128, and the zoom drive unit 129 each include a control circuit or a control module. The focus drive unit 126 controls the focus actuator 114 based on the focus detection result. Further, the aperture shutter drive unit 128 controls the aperture shutter actuator 112 at a predetermined timing of the shooting operation. Further, the zoom drive unit 129 controls the zoom actuator 111 according to the zoom operation of the photographer.

The display unit 131 includes a display device such as an LCD, and displays, for example, information on a shooting mode of the camera, a preview image before shooting and a confirmation image after shooting, a display image of a focused state at the time of focus detection, and the like. The operation unit 132 includes a group of switches for operating the digital camera 100, and includes, for example, a power switch, a release (shooting trigger) switch, a zoom operation switch, a shooting mode selection switch, and the like. When the operation unit 132 transmits the input user operation to the control unit 121, the control unit 121 controls each unit of the digital camera 100 in order to execute the operation corresponding to the user operation. The recording medium 133 includes, for example, a removable flash memory, and records a captured image.

The communication unit 134 includes a communication circuit or a module, and establishes communication with an external device (for example, a server installed outside) by using a communication method conforming to a predetermined standard. The communication unit 134 uploads and downloads image data with an external device, receives the result of a predetermined process performed by the external device on the uploaded image data, and the like.

(Structure of image processing unit 125)
Next, the detailed configuration of the image processing unit 125 will be described with reference to FIG. The image acquisition unit 151 stores the image data read from the recording medium 133. The image data is image data composed of an image (also referred to as an A + B image) obtained by synthesizing a first viewpoint image and a second viewpoint image, which will be described later, and a first viewpoint image.

The subtraction unit 152 generates a second viewpoint image by subtracting the first viewpoint image from the A + B image. The shading processing unit 153 that corrects the peripheral light falloff holds in the digital camera 100 in advance the change in the light amount due to the image heights of the first viewpoint image and the second viewpoint image as peripheral light falloff information. Alternatively, the shading processing unit 153 calculates and corrects the light intensity ratio between the A + B image and the first viewpoint image and the A + B image and the second viewpoint image. The operation information acquisition unit 154 receives the area specified by the user and the adjustment value of the depth change changed by the user, and supplies the area operated by the user and the adjustment value to the depth change processing unit 155.

The depth change processing unit 155 changes the addition ratio (weighting) of the first viewpoint image and the second viewpoint image to generate a composite image in which the depth of field is changed. Although the details will be described later, an image with an enlarged depth of field can be generated by the processing of the depth change processing unit 155.

Further, the image processing unit 125 performs development processing according to the configuration of the white balance unit 156, the demoizing unit 157, the gamma conversion unit 158, and the color adjusting unit 159 described below. The white balance unit 156 performs a white balance process. Specifically, gain is applied to each color of R, G, and B so that R, G, and B in the white region have the same color. By performing this white balance processing before the demo-saiking processing, it is possible to prevent the saturation from becoming higher than the saturation of the false color due to color cast or the like when calculating the saturation, and to prevent erroneous judgment. it can.

The demosaiking unit 157 generates a color image in which R, G, and B color image data are aligned in all pixels by interpolating the color mosaic image data of two of the three primary colors missing in each pixel. To do. The demoizing unit 157 performs interpolation on the pixel of interest using pixels around the pixel of interest. After that, color image data of the three primary colors R, G, and B as the interpolation processing result is generated for each pixel.

The gamma conversion unit 158 applies a gamma correction process to the color image data of each pixel to generate color image data matched to, for example, the display characteristics of the display unit 131. The color adjustment unit 159 applies various color adjustment processes such as noise reduction, saturation enhancement, hue correction, and edge enhancement, which are processes for improving the appearance of the image, to the color image data.

The compression unit 160 compresses the color-adjusted color image data by a method compliant with a predetermined compression method such as PEG, and reduces the data size of the color image data at the time of recording. The output unit 161 outputs the color image data, the compressed image data, or the display data for the user interface described above.

The optical information acquisition unit 162 holds in advance the exit pupil distance of the imaging optical system, the set pupil distance of the image sensor 107, the shooting aperture value, the peripheral light falloff information, etc., which will be described later, or receives them from the Internet. To get by. Further, the optical information acquisition unit 162 also holds in advance the optical information of the image plane movement amount conversion coefficient for calculating the image plane movement amount from the image shift amount of each viewpoint image, or receives it from the Internet. To get by. The optical information is used by the depth change processing unit 155.

(Structure of image sensor 107)
The arrangement of the pixels and the sub-pixels of the image pickup device 107 according to the present embodiment will be described with reference to FIG. In FIG. 3, the pixel array arranged in a two-dimensional shape is shown in the range of 4 columns × 4 rows, and the sub-pixel array included in each pixel is shown in the range of 8 columns × 4 rows.

Of the pixel arrangement shown in FIG. 3, in the pixel group of 2 columns × 2 rows, the pixel 200R having the spectral sensitivity of R (red) is in the upper left position, and the spectral sensitivity of G (green) is in the upper right and lower left positions. It has a pixel 200G and a pixel 200B having a spectral sensitivity of B (blue) at a lower right position. Further, each pixel has a sub-pixel 201 and a sub-pixel 202 arranged in two columns × one row.

By arranging a large number of pixels of 4 columns × 4 rows (sub-pixels of 8 columns × 4 rows) shown in FIG. 3 in a plane, it is possible to acquire a captured image (or a viewpoint image). In the image sensor 107, for example, the period P in which pixels are arranged is 4 μm, the number of pixels N is 5575 columns in width × 3725 rows in length = about 20.75 million pixels, the column direction period PS of sub-pixels is 2 μm, and the number of sub-pixels NS is horizontal. 11150 columns x 3725 rows = about 41.5 million pixels.

The structure of the pixel 200G shown in FIG. 3 will be described in more detail with reference to FIG. FIG. 4A is a plan view of the pixel 200G viewed from the light receiving surface side (+ z side) of the image sensor 107, and FIG. 4B is a cross section of aa of FIG. 4A from the −y side. The cross-sectional views seen are shown respectively.

The pixel 200G is configured to have a photoelectric conversion unit 301 and a photoelectric conversion unit 302 that are NH-divided (divided into two) in the x direction and NV-divided (divided into one) in the y direction. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 correspond to the sub-pixel 201 and the sub-pixel 202, respectively.

Further, the pixel 200G has a microlens 305 for condensing incident light on the light receiving side (+ z direction) of the pixel, and the light flux incident through the microlens 305 is the photoelectric conversion unit 301 or the photoelectric conversion unit. It is configured to receive light by 302. The photoelectric conversion unit 301 and the photoelectric conversion unit 302 may be a pin structure photodiode in which an intrinsic layer is sandwiched between a p-type layer and an n-type layer, or if necessary, the intrinsic layer is omitted and a pn junction is formed. It may be a photodiode. The color filter 306 is arranged between the microlens 305 and the photoelectric conversion unit 301 and the photoelectric conversion unit 302, and allows light of a predetermined frequency to pass therethrough. FIG. 4B shows an example in which one color filter 306 is provided for each pixel 200G, but if necessary, a color filter having a different spectral transmittance may be provided for each sub-pixel, or a color filter may be provided. The filter may be omitted.

In the photoelectric conversion unit 301 and the photoelectric conversion unit 302, electrons and holes are pair-produced according to the amount of light received, and further separated by a depletion layer. After that, negatively charged electrons are accumulated in the n-type layer, and holes are output to the outside of the image sensor 107 through the p-type layer 300 connected to a constant voltage source (not shown). The electrons accumulated in the photoelectric conversion unit 301 and the n-type layer of the photoelectric conversion unit 302 are transferred to the capacitance unit (FD) via the transfer gate and converted into a voltage signal.

(Relationship between the pixel structure of the image sensor 107 and pupil division)
Next, with reference to FIG. 5, the correspondence between the pixel structure of the image sensor 107 shown in FIG. 4 and the pupil division will be described. FIG. 5 shows the correspondence between the cross-sectional view of the aa section of the pixel 200G shown in FIG. 4A as viewed from the + y side and the exit pupil surface of the imaging optical system. In FIG. 5, the x-axis and the y-axis of the cross-sectional view of the pixel 200G are inverted with respect to FIG. 4B in order to correspond to the coordinate axes of the exit pupil surface.

The pupil region 501 of the sub-pixel 201 represents a pupil region that can receive light from the sub-pixel 201. The pupil region 501 of the sub-pixel 201 is generally in a conjugated relationship with the light receiving surface of the photoelectric conversion unit 301 whose center of gravity is eccentric to the + X side on the pupil surface and whose center of gravity is eccentric in the −x direction due to the microlens. It has become.

On the other hand, the pupil region 502 of the sub-pixel 202 represents a pupil region that can be received by the sub-pixel 202. The pupil region 502 of the sub-pixel 202 has a center of gravity eccentric to the −X side on the pupil surface and is generally in a conjugated relationship with the light receiving surface of the photoelectric conversion unit 302 whose center of gravity is eccentric in the + x direction due to the microlens. It has become. The pupil region 500 is a pupil region in which the entire pixel 200G can receive light when the photoelectric conversion unit 301 and the photoelectric conversion unit 302 (sub-pixel 201 and sub-pixel 202) are all combined.

With reference to FIG. 6, the state when light is incident on the pixel 200G having the above-described configuration will be described more specifically. FIG. 6 shows an example of the light intensity distribution when light is incident on the microlens 305 formed in the pixel 200G. FIG. 6A shows a light intensity distribution in a cross section parallel to the microlens optical axis, and FIG. 6B shows a light intensity distribution in a cross section perpendicular to the microlens optical axis at the microlens focal position. ing. In FIG. 6, H indicates the convex surface of the microlens 305, f indicates the focal length of the microlens, nFΔ indicates the movable range of the focal position due to refocus, and φ indicates the maximum angle of the incident luminous flux.

The incident light is focused on the focal position by the microlens 305, but the diameter of the focused spot is not smaller than the diffraction limit Δ due to the influence of diffraction due to the wave nature of the light, for example, FIGS. 6A and 6A. As shown in (b), the size is finite. For example, when the size of the light receiving surface of the photoelectric conversion unit 301 is about 1 to 2 μm, the focusing spot of the microlens is also about 1 μm. Therefore, the pupil region 501 (the pupil region 502 in the photoelectric conversion unit 302), which has a conjugate relationship with the light receiving surface of the photoelectric conversion unit 301 via the microlens 305, is clearly divided into pupils due to diffraction blur. It becomes a light receiving rate distribution (pupil intensity distribution).

The pupil intensity distribution in the pixel 200G is as shown in FIG. 7 when the horizontal axis is the pupil coordinates and the vertical axis is the light receiving rate. The pupil intensity distribution 701 (solid line) is an example of the pupil intensity distribution along the X axis of the pupil region 501 in FIG. 5, and the pupil intensity distribution 702 (broken line) is along the X axis of the pupil region 502. Examples of pupil intensity distribution are shown respectively. The pupil region 501 and the pupil region 502 have a gentle peak of pupil intensity at different pupil positions, indicating that the light passing through the microlens 305 is gently divided into pupils.

Further, the correspondence between the pixels having different positions of the image sensor 107 and the pupil division is as shown in FIG. 8A. The light fluxes that have passed through the different pupil region regions (pupil region 501 and pupil region 502) pass through the imaging surface 600 and are incident on each pixel of the image pickup device 107 at different angles. Then, the light is received by the sub-pixel 201 (photoelectric conversion unit 301) and the sub-pixel 202 (photoelectric conversion unit 302) of each pixel divided into 2 × 1. That is, the image pickup device 107 arranges a plurality of pixels provided with a plurality of sub-pixels configured to receive light flux passing through different pupil region regions of the imaging optical system.

By using the image sensor 107 having such a configuration, the light receiving signals of the sub-pixels 201 of each pixel are collected to generate a first viewpoint image, and the light receiving signals of the sub pixels 202 of each pixel are collected to obtain a second viewpoint image. Can be generated. That is, a plurality of viewpoint images can be generated for each different pupil region from the input image acquired by each pixel of the image sensor 107. In the present embodiment, since the first viewpoint image and the second viewpoint image are images of the Bayer array, respectively, the demosaiking process is applied to the first viewpoint image and the second viewpoint image as necessary. May be good. Further, by adding and reading the signals of the sub-pixel 201 and the sub-pixel 202 for each pixel of the image sensor 107, it is possible to generate an captured image having a resolution of N effective pixels. In this embodiment, an example of using an captured image generated from a plurality of viewpoint images (first viewpoint image and second viewpoint image) will be described. In addition, although FIG. 4A shows an example in which the pupil region is divided into two in the horizontal direction, the pupil division may be performed in the vertical direction depending on the method of dividing the sub-pixels.

(Depth change processing)
Next, the depth change processing according to the present embodiment, which is executed by the depth change processing unit 155, will be described. The depth change process is a process for expanding the depth of field.

8 (a) to 8 (c) show the principle of the depth change processing. In this figure, the image sensor 107 (not shown in FIGS. 8A to 8C) is arranged on the imaging surface 600, and the exit pupils of the imaging optical system are located in the pupil region 501 and the pupil region 502. It is divided into two.

FIG. 8A shows an in-focus image P1 of the main subject q1. Further, FIGS. 8 (b) and 8 (c) show the example shown in FIG. 8 (a) as a light flux passing through the pupil region 501 of the imaging optical system and a light flux passing through the pupil region 502. , Each is shown separately. In FIG. 8B, the luminous flux from the main subject q1 passes through the pupil portion region 501 and is imaged on the image P1 in the focused state. Each light flux is received by the sub-pixels 201 of different pixels of the image sensor 107, and a first viewpoint image is generated.

On the other hand, in FIG. 8C, the luminous flux from the main subject q1 passes through the pupil portion region 502 and is imaged on the image P1 in the focused state. Each light flux is received by the sub-pixel 202 of each pixel of the image sensor 107, and a second viewpoint image is generated.

Next, a process in which the depth change processing unit 155 changes the addition ratio of the first viewpoint image and the second viewpoint image to generate a composite image will be described. The depth change processing unit 155 is input with the above-mentioned first viewpoint image and second viewpoint image. As the first step, the pixel value of the first viewpoint image at the coordinates (j, i) is expressed as A (j, i), and the pixel value of the second viewpoint image at the coordinates (j, i) is expressed as B (j, i). , The depth change processing unit 155 sets the predetermined area R = [j1, j2] × [i1, i2] for changing the depth and the boundary width σ of the predetermined area, and then sets the predetermined area R according to the equation (1). The table function T (j, i) corresponding to the boundary width σ of the predetermined region is calculated.

The table function T (j, i) is a function that becomes 1 inside the predetermined region R and 0 outside the predetermined region R, and changes substantially continuously from 1 to 0 at the boundary width σ of the predetermined region R. To do. The depth change processing unit 155 may make the predetermined region a circular shape or any other shape, if necessary, or may set a plurality of predetermined regions and a plurality of boundary widths.

As a second step, the depth change processing unit 155 uses the actual coefficient w (-1 ≦ w ≦ 1) and follows the equation (2A), and the first weight coefficient Wa (1 ≦ w ≦ 1) of the first viewpoint image A (j, i) Calculate j and i). Further, the depth change processing unit 155 calculates the second weighting coefficient Wb (j, i) of the second viewpoint image B (j, i) according to the equation (2B).

As a third step, the depth change processing unit 155 includes the first viewpoint image A (j, i), the second viewpoint image B (j, i), the first weighting coefficient Wa (j, i), and the second. The output image I (j, i) is generated according to the equation (3) using the weighting coefficient Wb (j, i).

In this way, the depth change processing unit 155 uses the weighting coefficient that continuously changes according to the area of the output image, multiplies the weighting coefficient for each of the plurality of viewpoint images, synthesizes the output image, and changes the depth. Generate.

A method of generating an output image in which the weighting coefficient (that is, the addition ratio) is changed according to the user's specification will be described later, but the user may specify a predetermined area for performing the depth change process.

(Depth change processing for pupil deviation)
Next, the pupil shift in the peripheral image height of the image sensor 107 will be described. 9 (a) to 9 (c) show the relationship between the pupil portion regions 501 and 502 that the sub-pixel 201 and the sub-pixel 202 of each pixel receive light, and the exit pupil 400 of the imaging optical system.

FIG. 9A shows a case where the exit pupil distance Dl of the imaging optical system and the set pupil distance Ds of the image sensor 107 are the same. In this case, the exit pupil 400 of the imaging optical system is divided into pupils substantially evenly by the pupil region 501 and the pupil region 502 in both the central image height and the peripheral image height.

On the other hand, FIG. 9B shows a case where the exit pupil distance Dl of the imaging optical system is shorter than the set pupil distance Ds of the image sensor 107. In this case, at the peripheral image height, the exit pupil 400 of the imaging optical system is non-uniformly divided by the pupil region 501 and the pupil region 502. In the example of FIG. 9B, the effective aperture value of the first viewpoint image corresponding to the pupil portion region 501 is smaller (brighter) than the effective aperture value of the second viewpoint image corresponding to the pupil portion region 502. On the opposite side of the image height (not shown), on the contrary, the effective aperture value of the first viewpoint image corresponding to the pupil region 501 is larger than the effective aperture value of the second viewpoint image corresponding to the pupil region 502 (dark). ) Value.

FIG. 9C shows a case where the exit pupil distance Dl of the imaging optical system is longer than the set pupil distance Ds of the image sensor 107. Also in this case, at the peripheral image height, the exit pupil 400 of the imaging optical system is non-uniformly divided by the pupil region 501 and the pupil region 502. In the example of FIG. 9C, the effective aperture value of the first viewpoint image corresponding to the pupil portion region 501 is larger (darker) than the effective aperture value of the second viewpoint image corresponding to the pupil portion region 502. On the opposite side of the image height (not shown), on the contrary, the effective aperture value of the first viewpoint image corresponding to the pupil region 501 is smaller than the effective aperture value of the second viewpoint image corresponding to the pupil region 502 (bright). ) Value.

That is, as the pupil division becomes non-uniform at the peripheral image height due to the pupil shift, the effective aperture values of the first viewpoint image and the second viewpoint image also become non-uniform. For this reason, the degree of blurring of either the first viewpoint image or the second viewpoint image becomes large, and the way of spreading the other blur becomes small.

The actual aperture value (effective aperture value) differs depending on the position of each image sensor. As shown in FIG. 10A, when eclipse occurs due to a single frame, the aperture value does not change for each position of each image sensor. However, as shown in FIG. 10B, the actual lens has a plurality of frames, and the actual aperture value changes depending on the result of the plurality of frame eclipses. It is generally known that when eclipse occurs due to this lens, the amount of light drops (peripheral light amount drop) toward the image height at the edge of the image sensor. Since this peripheral light falloff is a value unique to each lens, there are many cameras having a function of storing peripheral light falloff information for each lens and correcting it after shooting. By using this peripheral light falloff information, it can be seen that eclipse due to the lens frame occurs as much as the light falloff. Therefore, the effective aperture value for each image height of the image sensor can be predicted by using the peripheral illumination falloff information.

The effective aperture value F at the peripheral light falloff information V (x, y) at the image height (x, y) of a certain image sensor can be calculated by the following equation (4) using the photographing aperture value F.

Here, f (V (x, y)) is a function using V (x, y). Since the optimum function form of f (V (x, y)) differs depending on the shape of the keratin by the lens, it is necessary to use a function suitable for each lens. The following shows a case where the shape of the kerare is close to a circle (Equation (5)) and a case where the shape is close to an ellipse (Equation (6)).

The above equation is an abbreviated form, but with the aim of improving accuracy, multiple functional forms can be considered. The effective aperture value can be calculated using the peripheral illumination drop information as shown in the equation (4), and the functional form of this case does not have to be limited to the equations (5) and (6).

From the above, by calculating the aperture value from the peripheral light falloff information of the first viewpoint image and the second viewpoint image, it is possible to obtain the magnitude relationship of the aperture value of each viewpoint image.

(Depth of field change processing)
Next, with reference to FIG. 9 again, the depth of field change processing by the depth change processing unit 155 will be described. As described above, in FIG. 9A, the image that has passed through the pupil region 501 is the first viewpoint image, and the image that has passed through the pupil region 502 is the second viewpoint image. As is clear from FIG. 9, each viewpoint image is an image obtained by passing through half of the original pupil region. Therefore, in the case of the pupil division region divided into two in the horizontal direction, the aperture diameter in the horizontal direction is large. It will be halved. Therefore, the depth of field in the horizontal direction is quadrupled. On the other hand, in the present embodiment, since the pupil is not divided in the vertical direction, there is no change in the depth of field in the vertical direction. Therefore, the first-viewpoint image or the second-viewpoint image has a depth of field that is twice as long and horizontal as the depth of field of the image (A + B image) obtained by combining the first-viewpoint image and the second-viewpoint image. The image has a depth.

In this way, the depth change processing unit 155 generates an image in which the depth of field is changed by changing the addition ratio of the first viewpoint image or the second viewpoint image to other than 1: 1 and generating a composite image. can do. As the area for changing the depth, a predetermined area may be processed according to the user's specification. When the composite image generated from the viewpoint image is output from the depth change processing unit 155, the above-mentioned development processing is applied, and the composite image is output from the image processing unit 125 as an image to which the development processing is applied.

(Image plane movement conversion coefficient)
FIG. 11 is a diagram illustrating a conjugate relationship between the exit pupil surface of the photographing optical system and the photoelectric conversion unit of the image sensor 107 arranged near zero image height (that is, near the center of the image surface) in the digital camera 100. is there. The photoelectric conversion unit in the image sensor 107 and the exit pupil surface of the photographing optical system are designed to have a conjugate relationship by an on-chip microlens. The exit pupil of the photographing optical system generally coincides with the surface on which the iris diaphragm for adjusting the amount of light is placed. On the other hand, the photographing optical system of the present embodiment is a zoom lens having a scaling function, but depending on the optical type, the distance and size of the exit pupil from the image plane change when the scaling operation is performed. The photographing optical system in FIG. 11 shows a state in which the focal length is between the wide-angle end and the telephoto end, that is, Middle. Assuming this is the standard exit pupil distance Zep, the optimum design of the eccentricity parameter according to the shape of the on-chip microlens and the image height (X, Y coordinates) is made.

FIG. 11A shows a lens barrel member 101b that holds the first lens group 101 and the first lens group 101, and a lens barrel member 105b that holds the third lens group 105 and the third lens group 105. .. Further, the shutter 102, the opening plate 102a that defines the opening diameter of the shutter 102 when the aperture is opened, and the aperture blade 102b for adjusting the opening diameter of the shutter 102 when the aperture is closed are also shown. The lens barrel member 101b, the aperture plate 102a, the diaphragm blade 102b, and the lens barrel member 105b, which act as limiting members for the light flux passing through the photographing optical system, show an optical virtual image when observed from the image plane. .. Further, the synthetic aperture in the vicinity of the shutter 102 is defined as the exit pupil of the lens, and the distance from the image plane is defined as Zep as described above.

The pixels for photoelectrically converting the subject image are composed of the photoelectric conversion units 301 and 302, the wiring layers 303a to 303c, the color filter 306, and the microlens 305 from the bottom layer. Then, the two photoelectric conversion units are projected onto the exit pupil surface of the photographing optical system by the microlens 305. In other words, the exit pupil of the photographing optical system is projected onto the surface of the photoelectric conversion unit via the microlens 305.

FIG. 11B shows a projected image of the photoelectric conversion unit on the exit pupil surface of the photographing optical system, and the projected images on the photoelectric conversion units 301 and 302 are EP1a and EP1b, respectively. EP1a corresponds to the first viewpoint image and EP1b corresponds to the projected image of the second viewpoint image.

Here, when the shutter 102 is open (for example, F2.8), the outermost part of the light flux passing through the photographing optical system is indicated by L (F2.8), but the projected images EP1a and EP1b are cut off at the aperture opening. There is nothing. On the other hand, when the shutter 102 has a small aperture (for example, F5.6), the outermost part of the luminous flux passing through the photographing optical system is indicated by L (F5.6), but the outside of the projected images EP1a and EP1b is the aperture opening. Be done. However, at the center of the image plane, the eclipsed states of the projected images EP1a and EP1d are symmetrical with respect to the optical axis, and the amount of light received by the photoelectric conversion units 301 and 302 is equal.

11 (c) shows the light receiving characteristics in the AA cross section of FIG. 11 (b), the horizontal axis represents the horizontal coordinates on the exit pupil surface of the photographing optical system, and the vertical axis represents the light receiving efficiency of the photoelectric conversion unit. In FIG. 11A, it was explained that the photoelectric conversion unit arranged in the pixel has a conjugate relationship with the exit pupil of the photographing optical system by the microlens. This means that only the luminous flux passing through the common region of the exit pupil TL2 (FIG. 11 (b)) of the photographing optical system and the projected image EP of the photoelectric conversion unit on the exit pupil surface reaches the photoelectric conversion unit. .. Therefore, the projected image corresponds to a pixel-specific aperture diaphragm arranged on the exit pupil surface of the photographing optical system, and the vertical axis of FIG. 11C is the transmittance distribution of each aperture diaphragm. This transmittance distribution can be regarded as the luminous flux light receiving efficiency of the photoelectric conversion unit. The distribution characteristic of this luminous flux light receiving efficiency will be referred to as "pupil intensity distribution" for convenience.

Here, if the projection performance by the microlens is geometrically optics-free, the pupil intensity distribution is a step function having only one of zero and one value. However, since the size of each pixel is as small as several μm, the sharpness of the image of the photoelectric conversion unit projected on the exit pupil surface is lowered by the diffraction of light. Further, since the microlens is usually a spherical lens, the sharpness of the projected image is lowered due to spherical aberration. Therefore, the pupil intensity distribution of each pixel also causes blurring, and as shown in FIG. 11C, the shoulder portions at both ends are blunted and the hem is pulled.

Next, the relationship between the pupil intensity distribution and the focus detection characteristic will be described. In the pair of pupil intensity distributions in the X-axis direction, the amount of separation of the center of gravity of each portion cut out in the exit pupil range of the photographing optical system corresponds to the baseline length in the phase difference type focus detection system. Here, the baseline length is defined by an angle θ (unit is radian) obtained by dividing the amount of separation of the center of gravity (unit: mm) on the pupil surface of the photographing optical system by the pupil distance (unit: mm). Then, assuming that the amount of image shift of the 1: 2 image at the time of focus detection is u (unit is mm) and the amount of defocus at that time is dz (unit is mm), these relationships are related to the following equations (7) and (unit: mm). It is represented by 8).

θ × dz = u ・ ・ ・ (7)
dz = u × (1 / θ) = u × K ・ ・ ・ (8)
Here, K is a coefficient defined by the reciprocal of the baseline angle θ.

The above K is the image plane movement amount conversion coefficient of this embodiment. θ takes a different value for each F number of the photographing optical system, and in FIG. 11C, the baseline angles of F2.8 and F5.6 are indicated by θ (F2.8) and θ (F5.6). .. According to the equation (7), the larger the baseline angle θ, the larger the amount of deviation of the focus detection image with respect to the unit defocus amount, and the higher the focus detection accuracy.

In the same manner as described above, the baseline length of the projected image EP1a + EP1d (A + B image), which is a projection image obtained by adding the photoelectric conversion units 301 and 302, and EP1a (first viewpoint image), which is a projection image of the photoelectric conversion unit 301, is photographed. It is defined by the angle θ1 (unit is radian) obtained by dividing the amount of separation of the center of gravity (unit: mm) on the pupil surface of the system by the pupil distance (unit: mm). Then, assuming that the amount of image shift of the 1: 2 image at the time of focus detection is u (unit is mm) and the amount of defocus at that time is dz1 (unit is mm), these relationships are related to the following equations (9) and (unit: mm). It is represented by 10).

θ1 × dz1 = u ・ ・ ・ (9)
dz1 = u × (1 / θ1) = u × K1 ・ ・ ・ (10)
Here, K1 is a coefficient defined by the reciprocal of the baseline angle θ1.

K1 is an image plane movement amount conversion coefficient of the A + B image and the first viewpoint image, and is used as the first image plane movement amount conversion coefficient.

Further, the baseline length of the projected image EP1a + EP1d (A + B image) which is the projection image obtained by adding the photoelectric conversion units 301 and 302 and the EP1b (second viewpoint image) which is the projection image of the photoelectric conversion unit 302 is set to the pupil of the photographing optical system. It is defined by the angle θ2 (unit is radian) obtained by dividing the amount of separation of the center of gravity (unit: mm) on the plane by the pupil distance (unit: mm). Then, assuming that the amount of image shift of the 1: 2 image at the time of focus detection is u (unit is mm) and the amount of defocus at that time is dz2 (unit is mm), these relationships are related to the following equations (11) and (unit: mm). It is represented by 12).

θ2 × dz2 = u ・ ・ ・ (11)
dz2 = u × (1 / θ2) = u × K2 ・ ・ ・ (12)
Here, K2 is a coefficient defined by the reciprocal of the baseline angle θ2.

K2 is the image plane movement amount conversion coefficient of the A + B image and the second viewpoint image, and is used as the second image plane movement amount conversion coefficient.

From the above, the image plane movement amount conversion coefficient is the reciprocal of the baseline angle θ, and the baseline angle corresponds to the F value. The smaller the baseline angle, the larger the F value. Here, the first image plane movement amount conversion coefficient K1 which is the image plane movement amount conversion coefficient of the A + B image and the first viewpoint image, and the first image plane movement amount conversion coefficient K1 which is the image plane movement amount conversion coefficient of the A + B image and the second viewpoint image. Comparing the values of the image plane movement amount conversion coefficient K2 means comparing the aperture values. Assuming that the aperture value of the first viewpoint image is F1 and the aperture value of the second viewpoint image is F2, the magnitude relationship is F1> F2 when the magnitude relationship K1> K2 and F1 <F2 when K1 <K2. From the above, by comparing the magnitude relationship between the image plane movement conversion coefficient K1 of the A + B image and the first viewpoint image and the image plane movement conversion coefficient K2 of the A + B image and the second viewpoint image, the first viewpoint image can be obtained. It is possible to obtain the magnitude relationship of the aperture value of the second viewpoint image.

The image plane movement conversion coefficient can be calculated if the information on the pupil, the aperture value, the exit pupil distance, and the image height of the image sensor described above is known. The image plane movement amount conversion coefficient can be calculated in advance and stored in the digital camera, or can be used by calculating in the digital camera 100.

(A series of operations related to the operation of changing the depth of the captured image)
Next, with reference to FIG. 12, a series of operations related to changing the depth of the captured image will be described. The processing of the flowchart of FIG. 12 is started, for example, when the release switch or the like included in the operation unit 132 is pressed by the user. Further, unless otherwise specified, the control unit 121 expands the program stored in the ROM (not shown) into the work area of the RAM and executes the processing of each step of this flowchart, and each unit such as the image processing unit 125 and the like. It is realized by controlling.

In S101, the image sensor 107 takes an image according to the instruction of the control unit 121. In S102, the image sensor 107 outputs the viewpoint image data. Specifically, the image sensor 107 outputs the above-mentioned A + B image and the first viewpoint image (A image) as viewpoint image data in one file format. In addition, the recording medium 133 temporarily stores the viewpoint image data output from the image sensor 107.

In S103, the image processing unit 125 reads the viewpoint image data in response to the instruction of the control unit 121. For example, the image processing unit 125 acquires the viewpoint image data stored in the recording medium 133 by using the image acquisition unit 151. At this time, the image processing unit 125 generates a B image from the A + B image, and for example, a first viewpoint image (A image) which is an image of the left viewpoint and a second viewpoint image (A image) which is an image of the right viewpoint ( B image) and is acquired.

In S104, the control unit 121 controls the operation unit 132 and the image processing unit 125 to perform depth change image processing described later, that is, depth change with respect to the captured image. When the control unit 121 finishes the depth change image processing, the control unit 121 ends the processing of this flowchart.

(A series of operations related to depth change image processing)
Next, a series of operations related to the depth change image processing in S104 will be described with reference to the flowchart shown in FIG. In the following description, operations in the user interface (depth change UI) for changing the depth of field will be shown.

In S201, the control unit 121 causes the display unit 131 to display the user interface (UI) provided with the depth change UI and the captured image.

In S202, the control unit 121 determines whether or not to change the depth based on the user operation input via the operation unit 132. When the input user operation indicates that the depth is changed, the control unit 121 advances the process to S203. On the other hand, if the input user operation does not indicate that the depth is changed, the control unit 121 returns the process to the main flowchart (FIG. 12).

In S203, the control unit 121 (area designating means) further acquires a user operation for operating the UI for selecting an area for which the depth is to be changed from the captured image via the operation unit 132. Here, an example of the depth changing UI displayed on the display unit 131 is shown in FIG. In the example of FIG. 15A, the control unit 121 displays an image in a part of the area 1000 constituting the UI. When the user selects an area (process target area) whose depth is to be changed, the process proceeds to S204.

In S204, the control unit 121 further acquires a user operation for operating the depth change UI via the operation unit 132. In the depth change UI, the slider 1001 and the slider bar 1002 are arranged in the horizontal direction. The user moves the slider 1001 to the position where the depth is to be changed, and the control unit 121 acquires the operation information (depth designation information) via the operation unit 132.

In S205, the control unit 121 determines whether or not the specified depth is greater than a predetermined value. If the specified depth is greater than a predetermined value, the process proceeds to S206, otherwise the process proceeds to S207. The predetermined value is the depth of field corresponding to the smaller (brighter) aperture value of the aperture value of the first viewpoint image and the aperture value of the second viewpoint image. Which viewpoint image has the smaller aperture value depends on the exit pupil distance of the imaging optical system, the set pupil distance of the image sensor 107, the image height for changing the shooting aperture value and depth, and the peripheral illumination drop information. It can be judged by the image plane movement amount conversion coefficient of the viewpoint image. The information necessary for making this determination may be stored in the digital camera 100 in advance, or may be received by the digital camera 100 via the Internet.

In S206, the control unit 121 identifies the viewpoint image having the larger depth (aperture value). Then, the control unit 121 increases the ratio of the viewpoint image having the larger depth (aperture value), synthesizes a partial region of the viewpoint image (the region designated in S203), and generates a depth change image. As described above, the exit pupil distance of the imaging optical system, the set pupil distance of the image pickup element 107, the image height for changing the shooting aperture value and the depth, the peripheral illumination falloff information, and the image plane movement amount conversion coefficient of each viewpoint image are used. The magnitude relationship between the aperture value of the first viewpoint image and the aperture value of the second viewpoint image changes. When the specified depth is larger than the predetermined value, in order to obtain the image of the specified depth, it is necessary to mainly synthesize the viewpoint image with the larger aperture value. It should be noted that synthesizing the specific viewpoint image "mainly" means that the composition is performed at the composition ratio in which the ratio of the specific viewpoint image is the largest.

In S207, the control unit 121 identifies the viewpoint image having the smaller depth (aperture value). Then, the control unit 121 increases the ratio of the viewpoint image having the smaller depth (aperture value), synthesizes a partial region of the viewpoint image (the region designated in S203), and generates a depth-changed image. When the depth change is not more than a predetermined value, an image having a specified depth can be obtained regardless of whether the first viewpoint image or the second viewpoint image is mainly combined. Therefore, the processing of S205 and S207 may be omitted, and the digital camera 100 may be configured to perform the processing of S207 regardless of the specified depth. However, if the processing of S205 and S207 is provided, the S / N ratio of the depth-changed image can be improved. This is because the viewpoint image having a smaller depth (aperture value) has a larger amount of light than the viewpoint image having a larger depth (aperture value), and therefore generally has less noise. Further, the predetermined value may be a depth of field equal to or less than the depth of field corresponding to the smaller (brighter) aperture value (first depth of field), and the subject corresponding to the smaller (brighter) aperture value. It does not necessarily have to match the depth of field. In this case, even if the specified depth is within the range that can be realized by synthesizing the viewpoint image mainly with the viewpoint image having the smaller aperture value, if the specified depth exceeds the predetermined value, the aperture is stopped. The viewpoint image is mainly combined with the viewpoint image having the larger value.

In S208, the control unit 121 performs a developing process. The development process will be described later with reference to FIG. In S209, the control unit 121 displays an image, ends the depth change image processing, and returns the processing to the main flowchart (FIG. 12).

FIG. 16 is a diagram showing an example of the relationship between the slider bar operation and the composition ratio of the first viewpoint image and the second viewpoint image. In FIG. 16, the vertical axis shows the ratio of each viewpoint image, and the horizontal axis shows the scale of the slider bar. As for the scale of the slider bar, "0" corresponds to the depth of the A + B image, and "10" corresponds to the depth of the viewpoint image which is the larger (deeper) of the first viewpoint image and the second viewpoint image. Correspond. The dotted line corresponds to the first viewpoint image, and the solid line corresponds to the second viewpoint image. FIG. 16A shows a case where the depth (aperture value) of the first viewpoint image and the second viewpoint image is the same, and FIG. 16B shows the depth (aperture value) of the second viewpoint image is higher than that of the first viewpoint image. Value) is large.

In the case of FIG. 16A, there is no substantial difference in the adjustable depth range and the S / N ratio regardless of whether the first viewpoint image or the second viewpoint image is mainly combined. Therefore, the control unit 121 changes the depth by continuously increasing the ratio of either the first viewpoint image or the second viewpoint image as the value of the scale of the slider bar increases.

In the case of FIG. 16B, the control unit 121 changes the depth by mainly synthesizing the first viewpoint image until the designated depth reaches a predetermined value. Here, it is assumed that the scale "5" corresponds to the predetermined value of the depth (that is, the aperture value of the first viewpoint image), but the value of the scale corresponding to the predetermined value of the depth is the aperture value of the first viewpoint image. It changes according to the ratio between the aperture value and the aperture value of the second viewpoint image. When the designated depth exceeds a predetermined value, the control unit 121 changes the depth by mainly synthesizing the second viewpoint image.

The depth (F value) that can be achieved by synthesizing the first viewpoint image and the second viewpoint image can be calculated from the peripheral light falloff information. The F-numbers of the A + B image, the first viewpoint image, and the second viewpoint image are represented as Fab, F1, and F2, respectively, and the amount of light is represented as Yab, Y1, and Y2, respectively. The amount of light is a function of the horizontal image height x and the vertical image height y, and the control unit 121 calculates the coefficient of the light amount function for each of the A + B image, the first viewpoint image, and the second viewpoint image using the shading processing unit 153. To do.

From the above, the F-numbers of the first viewpoint image and the second viewpoint image are given by the following equation (13).

Assuming that the F value when the ratio of the first viewpoint image is α (0 ≦ α ≦ 1) is F (α), the following equation (14) is obtained.

The control unit 121 calculates α such that F (α) = (designated depth (F value)) based on the equation (14). Thereby, the ratio of each of the first viewpoint image and the second viewpoint image can be obtained. Then, the control unit 121 calculates the weighting coefficients Wa and Wb of the first viewpoint image and the second viewpoint image, respectively, according to the above-mentioned formulas (2A) and (2B). In addition, in the formula (2A) and the formula (2B), w = 1-2α. Then, the control unit 121 can obtain an image of a specified depth by generating a composite image according to the above-mentioned equation (3).

As described above, the depth can be changed as shown in FIG. 16 by calculating the F value according to the composite ratio of the first viewpoint image and the second viewpoint image.

By the way, as described above, the F value (effective F value) of each viewpoint image changes according to the image height. Therefore, in the present embodiment, as described in detail below, when the depth change area specified in S203 of FIG. 13 exceeds a predetermined size (threshold value), the control unit 121 divides the depth change area into a predetermined size or less. , S205 to S207 are performed for each of the divided regions.

FIG. 17 is a diagram showing an example of the relationship between the image height and the effective F value of the first viewpoint image and the second viewpoint image. In FIG. 17, the vertical axis represents the effective F value and the horizontal axis represents the horizontal image height. The solid line corresponds to the first viewpoint image, and the dotted line corresponds to the second viewpoint image. The example of FIG. 17 is an example in which the depths of the first viewpoint image and the second viewpoint image are different for each image height.

In the example of FIG. 17, consider a case where the user designates an area of a predetermined size or less (below a threshold value) (for example, horizontal 5% and vertical 5% of the total image height) as a depth changing area. In this case, the control unit 121 uniformly determines the ratio of the first viewpoint image and the second viewpoint image with respect to the designated area. As a result, the depth can be changed with substantially the same effective F value for the entire depth change area.

When the user designates a large area (for example, 80% horizontal and 80% vertical of the total image height) as the depth changing area, the control unit 121 divides the designated area for each image height. Then, the control unit 121 determines the ratio of the first viewpoint image and the second viewpoint image for each divided area so that the entire designated area has substantially the same effective F value. In the example of FIG. 17, if the specified depth is equal to or less than a predetermined value, the control unit 121 increases the ratio of the first viewpoint image or the second viewpoint image in the center image height, and increases the ratio of the first viewpoint image in the right image height. The ratio of the second viewpoint image is increased at the left image height.

(A series of operations related to development processing)
Next, the development process in S208 will be described with reference to FIG. In S301, the control unit 121 applies a gain to each color of R, G, and B so that R, G, and B in the white region have the same color, and performs white balance processing. In S302, the control unit 121 performs a demosiking process. Specifically, the control unit 121 interpolates the input image in each specified direction, and then selects the direction, so that the interpolation processing result for each pixel is the three primary colors of R, G, and B. Generate a color image signal.

In S303, the control unit 121 performs gamma processing. In S304, the control unit 121 performs various color adjustment processes such as noise reduction, saturation enhancement, hue correction, and edge enhancement for improving the appearance of the image. In S305, the control unit 121 compresses the color-adjusted color image signal in S304 by a predetermined method such as JPEG to generate compressed image data. In S306, the control unit 121 records the image data generated in S305 on the recording medium 133. After that, the process returns to the flowchart of FIG.

As described above, according to the first embodiment, the digital camera 100 synthesizes the first viewpoint image and the second viewpoint image at a synthesis ratio according to the depth specified by the user. As described above, the digital camera 100 of the present embodiment adjusts the depth of field of the composite image by utilizing the difference in the aperture value.

[Second Embodiment]
In the first embodiment, as shown in FIG. 3, each pixel of the image pickup device 107 has been described as being divided into two in the horizontal direction. However, the form of pixel division is not limited to that shown in FIG. In the second embodiment, a configuration in which each pixel of the image pickup device 107 is divided into two in the horizontal direction and the vertical direction will be described. Other configurations of the digital camera 100 are substantially the same as those of the first embodiment. Hereinafter, the points different from the first embodiment will be mainly described.

FIG. 18 is a diagram illustrating an arrangement of pixels and sub-pixels of the image sensor 107 according to the second embodiment. FIG. 18 shows, for the image sensor 107 of the present embodiment, the pixel arrangement is shown in the range of 4 columns × 4 rows, and the sub-pixel arrangement is shown in the range of 8 columns × 8 rows.

In the present embodiment, in the pixel group 200 of 2 columns × 2 rows shown in FIG. 18, the pixel 200R having the spectral sensitivity of R (red) is on the upper left, and the pixel 200G having the spectral sensitivity of G (green) is on the upper right and lower left. Pixels 200B having a spectral sensitivity of B (blue) are arranged at the lower right. Further, each pixel is composed of sub-pixels 221 to sub-pixels 224 arranged in 2 columns × 2 rows.

The image pickup device 107 arranges a large number of pixels of 4 columns × 4 rows (sub-pixels of 8 columns × 8 rows) shown in FIG. 18 on a surface to enable acquisition of an image pickup image (or a viewpoint image). In the image sensor 107, for example, the period P in which pixels are arranged is 4 μm, the number of pixels N is 5575 columns in width × 3725 rows in length = about 20.75 million pixels, the period PS of sub-pixels is 2 μm, and the number of sub-pixels NS is 11150 columns in width. × Vertical 7450 lines = about 83 million pixels.

The structure of the pixel 200G shown in FIG. 18 will be described in more detail with reference to FIG. 19 (a) is a plan view of the pixel 200G viewed from the light receiving surface side (+ z side) of the image sensor 107, and FIG. 19 (b) is a cross-sectional view taken along the line aa of FIG. 19 (a) from the −y side. The cross-sectional views seen are shown respectively. As shown in FIG. 19A, in the pixel 200G of the present embodiment, the photoelectric conversion units 301 to 304 which are NH-divided (divided into two) in the x direction and NV-divided (divided into two) in the y direction are formed. The photoelectric conversion units 301 to 304 correspond to the sub-pixels 221 to 224, respectively.

In the present embodiment, the first viewpoint image is generated by collecting the light receiving signals of the sub-pixel 201 of each pixel. Similarly, the second viewpoint image is the light receiving signal of the sub pixel 202 of each pixel, the third viewpoint image is the light receiving signal of the sub pixel 203 of each pixel, and the fourth viewpoint image is the light receiving signal of the sub pixel 204 of each pixel. The received signals are collected and generated respectively. In the present embodiment, the first-viewpoint image to the fourth-viewpoint image are images of the Bayer array, respectively, and even if the first-viewpoint image to the fourth-viewpoint image are demosized as necessary. Good.

With j and i as integers, the j-th position in the row direction and the i-th position in the column direction of the image sensor 107 are (j, i), and the first viewpoint image of the pixel at the position (j, i) is A (j, i). Let the second viewpoint image be B (j, i), the third viewpoint image be C (j, i), and the fourth viewpoint image be D (j, i). At this time, the captured image I is I (j, i) = A (j, i) + B (j, i) + C (j, i) + D (j, i).

(Depth change processing)
The depth change processing unit 155 calculates the weighting coefficient of each viewpoint image according to the equations (15A) to (15D) with the actual coefficient w (-1 ≦ w ≦ 1).

Here, Wa (j, i) is the first weighting coefficient of the first viewpoint image A (j, i), and Wb (j, i) is the second weighting coefficient of the second viewpoint image B (j, i). .. Further, Wc (j, i) is the third weighting coefficient of the third viewpoint image C (j, i), and Wd (j, i) is the fourth weighting coefficient of the fourth viewpoint image D (j, i). Further, Wa + Wb + Wc + Wd = 0.

The depth change processing unit 155 generates an output image I (j, i) according to the equation (16) from the weighting coefficient corresponding to each viewpoint image.

The method for determining Wa, Wb, Wc, and Wd is the same as the method for determining Wa and Wb in the first embodiment, and the equation (14) may be extended to the case of four viewpoint images.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 ... Digital camera, 107 ... Image sensor, 121 ... Control unit, 125 ... Image processing unit, 131 ... Display unit, 132 ... Operation unit, 133 ... Recording medium

Claims (9)

An acquisition means for acquiring a plurality of viewpoint images corresponding to a plurality of pupil region regions in which the exit pupil of the imaging optical system is divided, and
Depth of field specification means to specify the depth of field and
A determination means for determining the aperture value of each viewpoint image in a partial region based on the optical information of the plurality of viewpoint images, and
A composite image of the specified depth of field is generated by synthesizing the partial regions of the plurality of viewpoint images with a composite ratio based on the specified depth of field and the determined aperture value. Generation means and
An image processing device characterized by comprising. The generation means
From the plurality of viewpoint images, the first viewpoint image having the smallest aperture value in the partial region is identified.
When the designated depth of field is the first depth of field that is equal to or less than the depth of field of the first viewpoint image in the partial region, the ratio of the first viewpoint image is the largest. The image processing apparatus according to claim 1, wherein the partial regions of the plurality of viewpoint images are combined at a composition ratio. The image processing apparatus according to claim 2, wherein the first depth of field is equal to the depth of field of the first viewpoint image in the partial region. Area designation means to specify the processing target area and
When the size of the processing target area is equal to or less than the threshold value, the processing target area is controlled to be used as the partial area, and when the size of the processing target area is larger than the threshold value, the processing target area is set to a size equal to or less than the threshold value. A control means for controlling each of the plurality of divided regions divided into the above to be used as the partial region.
The image processing apparatus according to any one of claims 1 to 3, further comprising. The image processing apparatus according to any one of claims 1 to 4, wherein the optical information of the plurality of viewpoint images includes image plane movement amount conversion coefficients of the plurality of viewpoint images. The image processing apparatus according to any one of claims 1 to 4, wherein the optical information of the plurality of viewpoint images includes peripheral light falloff information of the plurality of viewpoint images. The image processing apparatus according to any one of claims 1 to 6.
An image sensor in which a plurality of pixels including a plurality of sub-pixels, each of which receives a light flux corresponding to a different pupil region of the exit pupil, are arranged.
An imaging device characterized by comprising. An image processing method executed by an image processing device.
An acquisition process for acquiring a plurality of viewpoint images corresponding to a plurality of pupil region regions in which the exit pupil of the imaging optical system is divided, and
The depth specification process that specifies the depth of field and
A determination step of determining the aperture value of each viewpoint image in a partial region based on the optical information of the plurality of viewpoint images, and
A composite image of the specified depth of field is generated by synthesizing the partial regions of the plurality of viewpoint images with a composite ratio based on the specified depth of field and the determined aperture value. Generation process and
An image processing method characterized by comprising. A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 6.