JP6802633B2 - Image sensor, image sensor, image generation method, and program - Google Patents

Description

The present invention relates to an image sensor, an image sensor, an image generation method, and a program.

In recent years, a device for performing focus detection by obtaining a plurality of images having different viewpoints with an image sensor has been proposed. In addition, proposals have been made to achieve both color imaging and focus detection. Patent Document 1 discloses an imaging device that simultaneously acquires two-dimensional intensity distribution of light in a subject space and angle information of light rays, that is, images having different viewpoints. Here, the two-dimensional intensity distribution of light and the angle information of light rays are collectively referred to as light ray space information (or light field information). By acquiring the ray space information, it is possible to obtain three-dimensional information on the subject space. In the above-mentioned imaging device, a microlens is provided so as to straddle a plurality of pixels, and a color filter is provided for each pixel.

As another configuration for achieving both color imaging and focus detection, Patent Document 2 provides an aperture biased in the pupil division direction with respect to a microlens provided for each pixel, and between pixels of the same color having the smallest adjacent color arrangement. An image sensor in which the apertures have a symmetrical relationship is disclosed.

Japanese Unexamined Patent Publication No. 2007-4471 JP-A-2007-158597

However, in the image pickup apparatus of Patent Document 1, a plurality of pixels exist under one microlens, and the microlens acts as a spatial low-pass filter, so that the resolution tends to deteriorate. Further, since the color interpolation processing is performed under one microlens, inappropriate interpolation may be performed when the color is out of focus.

Further, since Patent Document 2 focuses only on focus detection, only the concept of symmetry with the pupil division direction is disclosed. Therefore, the technique of Patent Document 2 is not easy to develop into an imaging device capable of acquiring ray space information. Further, since the image pickup apparatus of Patent Document 2 evenly refers to pixel signals having the same color and a symmetric relation to interpolate the missing color information, improper interpolation may be performed when the image is out of focus.

The present invention has been made in view of such a situation, and an object of the present invention is to provide a technique for enabling color interpolation processing for suppressing image quality deterioration according to a focus state.

In order to solve the above problems, in the present invention, a plurality of pixels each including a photoelectric conversion unit, microlenses arranged in pixel units, and vertical n _c x horizontal _mc (n _c and _mc are 2 or more). A color filter arranged in pixel units by repeating the minimum color arrangement of), and a mask means having an aperture and a light-shielding portion arranged in pixel units between the photoelectric conversion unit and the microlens. , wherein the mask means is vertical and the number of horizontal pattern when a n _d and m _{d (n d} and m _d is an integer of 2 or more) each, the color filters of the same color in the open position And the combination of the same opening position appears by repeating the minimum unit array of vertical (n _c * _nd * a) x horizontal ( _mc * m _d * b) (a and b are natural numbers), and is specified in the minimum unit array. pixel color filters of all the pixels, looking contains each color at the same rate as the minimum color sequence, for any pattern of open position of the (n d _{* m d)} also corresponding to the pattern corresponding to the opening position of the Provided is an imaging device characterized in that is configured to be present in any column and in any row of the smallest unit array .

In addition, other features of the present invention will be further clarified by the accompanying drawings and the description in the form for carrying out the following invention.

According to the present invention, it is possible to execute a color interpolation process that suppresses image quality deterioration according to a focus state.

The figure explaining the image generation processing and the pattern of the color filter and the pupil dividing means in the image sensor 6. (A) A central sectional view of the image pickup apparatus 1 and the lens unit 2, and (b) a block diagram showing an electrical configuration of the image pickup apparatus 1 and the lens unit 2. (A) A diagram schematically showing the relationship between the image sensor 6 and the microlens 20, (b) a schematic diagram showing the correspondence between the pixels of the image sensor 6 and the pupil dividing means, and (c) the pupil having a specific pixel by the pupil dividing means. The figure which shows that it is associated with an area. The figure explaining the relationship between the subject and the image acquired by the image sensor 6. It is a figure explaining the method of performing phase detection using the image sensor shown in FIG. 1 (b). The figure which shows the example of the image sensor which does not satisfy the arrangement condition. The figure which shows the example of the image sensor which satisfies the arrangement condition. The figure which shows the example of the image sensor which satisfies the arrangement condition (a) and (b), and the figure which shows the example of the image sensor which does not satisfy the arrangement condition (c) and (d). The figure which shows the example of the image sensor which satisfies the arrangement condition.

Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. The technical scope of the present invention is determined by the scope of claims, and is not limited by the following individual embodiments. Also, not all combinations of features described in the embodiments are essential to the present invention.

[First Embodiment]
FIG. 2A is a central sectional view of the image pickup apparatus 1 and the lens unit 2, and FIG. 2B is a block diagram showing an electrical configuration of the image pickup apparatus 1 and the lens unit 2. In FIGS. 2 (a) and 2 (b), the same reference numerals indicate the same components.

First, refer to FIG. 2 (a). 1 is an imaging device, 2 is a lens unit mounted on the imaging device 1, 3 is a photographing optical system composed of a plurality of lenses, 4 is an optical axis of the photographing optical system 3, 6 is an imaging element, and 9 is. Each display unit is shown. Further, 11 is an electrical contact of the image pickup device 1 and the lens unit 2, 12 is a lens system control unit provided in the lens unit 2, 14 is a quick return mechanism, and 16 is a finder display unit provided in the image pickup device 1. Are shown respectively.

Next, refer to FIG. 2 (b). The camera system including the camera 1 and the lens unit 2 includes an imaging system, an image processing system, a recording / reproducing system, and a control system. The image pickup system includes a photographing optical system 3 and an image pickup element 6. The image processing system includes an image processing unit 7. The recording / playback system includes a memory unit 8 and a display unit 9. The control system includes a camera system control unit 5, an operation detection unit 10, a lens system control unit 12, and a lens drive unit 13. The lens driving unit 13 can drive a focus lens, a blur correction lens, an aperture, and the like.

The image pickup system is an optical processing system that forms an image of light from an object on the image pickup surface of the image pickup device 6 via the photographing optical system 3. Microlenses are arranged in a grid pattern on the surface of the image sensor 6 to form a so-called microlens array (MLA). Details of the functions and arrangement of the MLA will be described later with reference to FIG. As will be described later, since the phase difference information can be obtained from the image sensor 6, the subject image can be formed in the vicinity of the image sensor 6 by appropriately adjusting the photographing optical system 3 based on this information.

The image processing unit 7 has an A / D converter, a white balance adjustment circuit, a gamma correction circuit, an interpolation calculation circuit, and the like inside, and can generate an image for recording. The image processing unit 7 performs color interpolation processing (demosizing processing) from the signals of the Bayer array to generate a color image.

The memory unit 8 includes a processing circuit necessary for recording in addition to the memory. The memory unit 8 outputs data to the memory, and also generates and stores an image to be output to the display unit 9. In addition, the memory unit 8 compresses images, moving images, sounds, and the like using a predetermined method.

The camera system control unit 5 generates and outputs a timing signal or the like at the time of imaging. Further, the camera system control unit 5 controls the imaging system, the image processing system, and the recording / reproducing system, respectively, according to an external operation. For example, when the operation detection unit 10 detects the pressing of the shutter release button (not shown), the camera system control unit 5 controls the drive of the image sensor 6, the operation of the image processing unit 7, the compression processing of the memory unit 8, and the like. .. Further, the camera system control unit 5 controls the information display by the display unit 9 (liquid crystal monitor or the like). Further, the display unit 9 is a touch panel and is connected to the operation detection unit 10.

The adjustment operation of the photographing optical system 3 by the control system will be described. An image processing unit 7 is connected to the camera system control unit 5, and an appropriate focus position and aperture position are obtained based on the signal from the image sensor 6. The camera system control unit 5 issues a command to the lens system control unit 12 via the electrical contact 11, and the lens system control unit 12 appropriately controls the lens drive unit 13. Further, a camera shake detection sensor (not shown) is connected to the lens system control unit 12, and in the mode of performing camera shake correction, a blur correction lens is appropriately applied via the lens drive unit 13 based on the signal of the camera shake detection sensor. To control.

The image pickup apparatus 1 shown in FIG. 2 can also reproduce an image. When the reproduction mode is set by the signal from the operation detection unit 10, the image pickup apparatus 1 displays the acquired image on the display unit 9. When presenting a plurality of images having different viewpoints, the images are reconstructed (so-called developed) at appropriate focus positions and aperture values and displayed.

FIG. 3 is an explanatory diagram of the photographing optical system 3. The photographing optical system 3 of the present embodiment is configured so that a subject image can be acquired from a plurality of viewpoints. In the following description, a configuration for acquiring a subject image from a plurality of viewpoints is referred to as a pupil dividing means. The pupil dividing means is configured by arranging the MLA in the vicinity of the image plane of the photographing optical system 3 and arranging the mask layer in the image sensor 6.

FIG. 3A is a diagram schematically showing the relationship between the image sensor 6 and the microlens 20. FIG. 3B is a schematic diagram showing the correspondence between the pixels of the image sensor 6 and the pupil dividing means. FIG. 3C is a diagram showing that the pixel is associated with a specific pupil region by the pupil dividing means.

As shown in FIG. 3A, a microlens 20 is provided for each pixel on the image sensor 6, and the front principal point of the microlens 20 is located near the image plane of the photographing optical system 3. Have been placed. The microlens 20 is paired with a mask layer described later to form a pupil dividing means. FIG. 3A shows a state in which the microlens 20 is viewed from the side of the image pickup apparatus 1 and from the front. When viewed from the front of the image pickup device 1, the MLA is arranged so as to cover the image pickup device 6. In FIG. 3A, the microlens 20 is shown large in order to make it easy to see, but in reality, each microlens 20 has only the same size as a pixel.

FIG. 3B is a partially enlarged view of the image pickup apparatus 1 of FIG. 3A from the front. The grid-like frame shown in FIG. 3B shows the pixels of the image sensor 6. Further, as will be described in detail later with reference to FIG. 3C, the mask layers constituting the pupil dividing means are arranged in a predetermined arrangement.

FIG. 3C is a diagram for explaining the correspondence between the cross-sectional view of the image sensor 6 cut along the cross-sectional symbol of FIG. 3B and the exit pupil surface. Actually, when the direction is aligned with the cross-sectional view shown in the lower part of FIG. 3C, the exit pupil surface in the upper part of FIG. 3C is in the vertical direction of the paper surface in FIG. 3C, but it is projected for explanation. The direction is changing. Further, in FIG. 3C, one-dimensional projection / signal processing will be described for the sake of simplicity. In an actual imaging device, the configuration of FIG. 3C is expanded two-dimensionally.

In the image sensor 6 shown in FIG. 3C, a photoelectric conversion unit 31 that generates a pixel signal is formed inside the silicon substrate 30. The signal charge generated by the photoelectric conversion unit 31 is output to the outside via the floating diffusion unit (not shown), the first electrode 41, and the second electrode 42. An interlayer insulating film 32 is formed between the photoelectric conversion unit 31 and the electrode 41, and an interlayer insulating film 33 is formed between the electrode 41 and the electrode 42. Further, an interlayer insulating film 34 is formed on the light incident side of the electrode 42, and a passivation film 35 and a flattening layer 36 are further formed. A color filter layer 37, a flattening layer 38, and a microlens 20 are formed on the light incident side of the flattening layer 36. Here, the power of the microlens 20 is set so that the exit pupil of the lens of the photographing optical system 3 and the photoelectric conversion unit 31 are substantially conjugated. Further, in the pixel located in the center of the image sensor 6, the microlens 20 is arranged in the center of the pixel, and in the pixel located in the periphery of the image sensor 6, the microlens 20 is located on the optical axis 4 side of the photographing optical system 3. Displaced and arranged.

The subject light transmitted through the photographing optical system 3 is focused in the vicinity of the image sensor 6. Further, the light that reaches each pixel of the image sensor 6 is refracted by the microlens 20 and focused on the photoelectric conversion unit 31. Further, in order to detect the focus of the photographing optical system 3, a part of the electrode 41 is configured to cover the photoelectric conversion unit 31. Since the electrode 41 blocks a part of the light flux reaching from the pupil, the layer of the electrode 41 is called a mask layer. The mask layer has openings and shading portions and is regularly arranged according to a predetermined arrangement. In the lower sectional view of FIG. 3C, the pixel shown on the left has an opening on the left side with respect to the optical axis, and the pixel shown on the right has an opening on the right side with respect to the optical axis. That is, the image sensor 6 has a plurality of types of openings whose positions differ depending on the pixel. As a result, each pixel of the image sensor 6 is configured to capture only a part of the luminous flux of the exit pupil, and as a result, capture only the incident light from a specific angle.

In FIG. 3 (c), consider some regions 51-1, 51-2, 53-1, and 51-4 of the exit pupil 50. In particular, attention is paid to the luminous flux from the regions 51-3 and 51-4. Rays from the ends of regions 51-3 and 51-4 are illustrated with reference numerals 63 and 64, respectively. As is clear from FIG. 3C, for the pixel shown on the left, which has an opening on the left side with respect to the optical axis in the mask layer (electrode 41), the light ray 63 passes through the photoelectric conversion unit 31. Although it reaches, the light beam 64 is blocked and does not reach the photoelectric conversion unit 31. On the other hand, for the pixel shown on the right, which is open on the right side with respect to the optical axis in the mask layer (electrode 41), the light ray 63 is blocked and does not reach the photoelectric conversion unit 31, but the light ray 64 passes through. Then, it reaches the photoelectric conversion unit 31. That is, the microlens 20 and the mask layer (electrode 41) have a structure that captures only the light flux that has passed through a specific region of the exit pupil, and serves as a pupil dividing means.

FIG. 4 is a diagram for explaining the relationship between the subject and the image acquired by the image sensor 6. In FIG. 4A, 54, 55, and 56 indicate points on the exit pupil 50, 70 and 71 indicate the image plane, 80 indicates the subject, and 81, 82, and 83 indicate points on the subject, respectively. ing. Further, 90-a and 91-a indicate subject images obtained on the image planes 70 and 71, respectively. The point 54 on the exit pupil 50 indicates the point where the exit pupil 50 and the optical axis 4 intersect, and the points 55 and 56 indicate points symmetrically eccentric with respect to the point 54. On the subject 80, the point 82 indicates the boundary between white and black, the point 81 indicates the white end, and the point 83 indicates the black end. 81-1 is a light ray passing from the point 81 on the subject 80 to the point 55 on the exit pupil 50, and 81-2 is a light ray passing from the point 81 on the subject 80 to the point 54 on the exit pupil 50. Reference numeral 3 denotes a light ray passing from the point 81 on the subject 80 to the point 56 on the exit pupil 50, respectively. In the notation 81-1, the front of the hyphen corresponds to the point on the object, and the number after the hyphen corresponds to the point on the exit pupil 50. The numbers behind the hyphen correspond to 1 for point 55, 2 for point 54, and 3 for point 56. Rays 82-1, 82-2, 82-3, 83-1, 83-2, and 83-3 were defined and illustrated using this notation. The ray 82-2 coincides with the optical axis 4.

The following description will be given assuming that the subject 80 is imaged on the image plane 70. Since the image plane 70 is in a so-called in-focus state, the light flux emitted from a certain point on the subject 80 and passed through the photographing optical system 3 intersects at one point. That is, the rays 81-1, 81-2, and 81-3 from the point 81 intersect on the image plane 70. Similarly, the image of the set of the rays 82-1, 82-2, and 82-3 from the other points 82 and 83 on the subject 80 and the set of the rays 83-1, 83-2, and 83-3. Cross on surface 70.

Consider the images 90-a and 91-a of the subject 80 acquired on the image planes 70 and 71. Since the image plane 70 is in focus, an inverted image of the subject 80 can be obtained. As described above, it is clear from the fact that the light rays emitted from the points on the subject 80 intersect at one point on the image plane 70. On the other hand, since the image plane 71 is out of focus, the image is blurred and observed. Only the light rays of the white part on the subject 80 are received only in the area sandwiched between the light rays 82-1 and 81-3, and only the light rays of the black part on the subject 80 are received by the light rays 83-1 and 81-3. Only the area sandwiched between 82-3. The region sandwiched between the light rays 82-3 and 82-1 between these two regions has a different color depending on the location where it has passed through the exit pupil 50, and this is spatially convoluted (integrated) to cause blurring. Observed.

4 (b) and 4 (c) are diagrams for explaining the state of the subject 80 when only a part of the luminous flux of the exit pupil 50 is considered. 4 (b) shows the luminous flux passing between the points 54 and 55 of the exit pupil 50 shown in FIG. 4 (a), and FIG. 4 (c) shows the exit pupil 50 shown in FIG. 4 (a). This is a case where the luminous flux passing between the points 54 and 56 of the above is considered. As described above with reference to FIG. 3, the image pickup device 6 of the present embodiment includes the pupil dividing means, and can receive only a part of the luminous flux of the exit pupil 50. 4 (b) and 4 (c) show the state of the subject image at that time.

In FIGS. 4 (b) and 4 (c), the same items as those in FIG. 4 (a) are numbered the same. In FIG. 4B, 90-b and 91-b show subject images obtained on the image planes 70 and 71, respectively. Further, 92 indicates a point corresponding to the point 82 on the subject 80 in the subject image 90-b, and 93 indicates a point corresponding to the point 82 on the subject 80 in the subject image 91-b. Further, in FIG. 4 (c), 90-c and 91-c show subject images obtained on the image planes 70 and 71, respectively. Further, 94 indicates a point corresponding to the point 82 on the subject 80 in the subject image 90-c, and 95 indicates a point corresponding to the point 82 on the subject 80 in the subject image 91-c.

As described in FIG. 4A, since the image plane 70 is a focused plane, a light ray from a certain point on the subject 80 is at the same point regardless of which point on the exit pupil 50 passes through. Intersect. In other words, this indicates that the same image can be obtained regardless of the passing region at the exit pupil 50. That is, the subject images 90-b and 90-c obtained on the image plane 70 in FIGS. 4 (b) and 4 (c) are equivalent. Therefore, the points 82 on the subject and the corresponding points 92 and 94 also match.

On the other hand, since the image plane 71 is out of focus, the image is observed out of phase depending on the passing region of the exit pupil 50. That is, while the point 93 corresponding to the point 82 on the subject 80 in FIG. 4B exists on the right side with respect to the optical axis 4, it corresponds to the point 82 on the subject 80 in FIG. 4C. The point 95 exists on the left side with respect to the optical axis 4. In FIGS. 4 (a) to 4 (c), only the image plane 71 far from the subject 80 is considered with respect to the in-focus image plane 70, but when considering the image plane on the side closer to the subject 80, the phase is considered. Is observed on the contrary. Using this principle, it is possible to know the focus state in each area on the screen. In the present embodiment, a signal suitable for such phase detection is generated to detect the phase.

Also, pay attention to the blur in FIGS. 4 (b) and 4 (c). As shown in FIGS. 4 (b) and 4 (c), even when only a part of the luminous flux of the exit pupil 50 is received, blurring occurs on the image plane 71 which is out of focus. On the other hand, paying attention to the degree of blurring, it can be seen that the blurring is less than that in FIG. 4A. This is because it receives only a part of the luminous flux of the exit pupil 50, so that it is in a so-called focused state and blurring is suppressed.

4 (d) and 4 (e) are graphs schematically showing signals on image planes 70 and 71, respectively. In FIGS. 4 (d) and 4 (e), the horizontal axis indicates the position on the image planes 70 and 71, and corresponds to the pixel. The vertical axis shows the level of the signal obtained by the photoelectric conversion unit of each pixel. In FIGS. 4 (d) and 4 (e), the position indicated by “◯” corresponds to the signal obtained in FIG. 4 (b), and the position indicated by “Δ” is shown in FIG. 4 (c). Corresponds to the obtained signal.

In FIG. 4D showing the signal of the image plane 70, the pixels adjacent to each other on the image plane 70 correspond to the positions adjacent to each other on the subject 80. Therefore, the line segment 96 connecting the signal 90-b indicated by "○" and the signal 90-c indicated by "Δ" in FIG. 4D forms a smooth line and is very very. Contains high frequency components (edges are neatly formed).

On the other hand, in FIG. 4E showing the signal of the image plane 71, even the pixels adjacent to each other on the image plane 70 correspond to the positions separated on the subject 80. Therefore, the line segment 96 connecting the signal 91-b indicated by “◯” and the signal 91-c indicated by “Δ” in FIG. 4 (e) cannot form a smooth line. , Generates a signal near the Nyquist frequency that does not originally exist. At this time, considering the line segment 98 connecting only the signals 91-b indicated by "○" and the line segment 97 connecting only the signals 91-c indicated by "△", the curve becomes smooth. There is. Further, the high frequency component is cut as compared with FIG. 4 (d). This shows the blur shown in FIGS. 4 (b) and 4 (c). Further, the line segment 98 connecting only the signals 91-b indicated by "○" and the line segment 97 connecting only the signals 91-c indicated by "Δ" have similar shapes and are laterally displaced. It can be seen that there is (a phase difference has occurred).

As is clear from FIGS. 4 (d) and 4 (e), in the image plane in focus as shown in FIG. 4 (d), the images that have passed through different regions of the exit pupil 50 are directly subjected to interpolation processing or the like. It can be used. On the other hand, in the image plane that is out of focus as shown in FIG. 4 (e), the images that have passed through different regions of the exit pupil 50 cannot be used as they are for interpolation processing or the like. As the limit of the size of the out-of-focus, it is considered that about twice the size of the pixel is appropriate. This is because if there is a deviation larger than two pixels, as shown in FIG. 4 (e), the edge portion is overtaken due to the phase shift, and it is considered that a signal near the Nyquist frequency that does not originally exist is generated. .. The specific contents of the interpolation process will be described later with reference to FIG.

The image generation process and the pattern of the color filter and the pupil dividing means in the image sensor 6 will be described with reference to FIG. FIG. 1A is a flowchart showing the flow of image generation processing. FIG. 1B is a diagram illustrating a pattern of a color filter and pupil dividing means in the image sensor 6. FIG. 1C is a conceptual diagram of a process for discarding viewpoint information. FIG. 1D is a diagram showing an array of pixels when focusing on an image from a specific viewpoint. FIG. 1 (e) is a conceptual diagram of color interpolation processing based on the viewpoint signal of FIG. 1 (d).

Unless otherwise specified, the processing of each step of FIG. 1A is realized by the camera system control unit 5 executing a control program to control each unit of the image pickup apparatus 1. When the image sensor 6 acquires the subject image before the color interpolation processing and supplies it to the image processing unit 7, the processing of this flowchart starts.

In S110, the camera system control unit 5 calculates the phase difference. The basic principle of phase detection (phase difference detection processing) is as described with reference to FIG. 4, but a specific signal generation method for phase detection will be described later with reference to FIG.

In S120, the camera system control unit 5 determines whether or not the phase difference is less than the threshold value. If the phase difference is less than the threshold value, the camera system control unit 5 advances the process to S130, and if the phase difference is greater than or equal to the threshold value, the camera system control unit 5 advances the process to S150.

In S130, the camera system control unit 5 discards the viewpoint information. As described with reference to FIG. 4, when the phase difference is less than the threshold value and the focus is on, the adjacent pixels on the image plane are adjacent on the subject 80 regardless of the passing region of the exit pupil 50. There is. Therefore, it is possible to obtain a signal with an increased resolution by discarding the viewpoint information. Specific processing will be described later with reference to FIGS. 1 (b) and 1 (c).

In S140, the camera system control unit 5 performs color interpolation processing. The color interpolation process referred to here is a process of interpolating a color signal lacking in a mosaic pattern when a color image is obtained by using a color filter arranged in a Bayer array or the like (also called demosaicing or demosaiking). Is. This produces a complete color image (ie, a color image containing all types of colors in the array of color filters).

In S150, the camera system control unit 5 generates each viewpoint image. As described above with reference to FIGS. 3 and 4, the image pickup device 6 of the present embodiment can acquire only the luminous flux in a specific region of the exit pupil 50 by the pupil dividing means. This is equivalent to acquiring images from different perspectives. That is, if only the pixels that capture the luminous flux that has passed through the same region with the exit pupil 50 are arranged, an image of the subject observed from that position can be obtained. This is called a viewpoint image. A plurality of viewpoint images are generated, but the number varies depending on the pattern of the pupil dividing means. Some patterns will be illustrated later. In S150, the camera system control unit 5 extracts only pixels having the same aperture pattern to generate a viewpoint image. Specific processing will be described later with reference to FIGS. 1 (b) and 1 (d).

In S160, the camera system control unit 5 performs color interpolation processing using the viewpoint image. In the case of S140, the color interpolation processing is performed regardless of the difference in the viewpoint after the viewpoint information is discarded, but in the case of S160, the color interpolation processing is performed without using the signals of different viewpoints. Specific processing will be described later with reference to FIG. 1 (e).

In S170, the camera system control unit 5 synthesizes a viewpoint image. That is, after color interpolation processing is performed on each of the plurality of viewpoint images in S160, composition processing such as addition is performed in S170 as necessary. Noise can be reduced by performing addition. On the other hand, S170 can be omitted if it is unnecessary.

In S180, the camera system control unit 5 displays and records the image generated in S140 or S170.

The pattern of the color filter and the pupil dividing means will be described with reference to FIG. 1 (b). In FIG. 1B, the square circumscribing the circle indicates a pixel. Further, the circle indicates a microlens provided in the pixel. The small squares provided in the pixels indicate the openings of the mask layer. As described with reference to FIG. 3, the pupil dividing means is formed by the microlens and the mask layer, and as shown in FIG. 1B, in the image pickup device 6 of the present embodiment, all the pixels are ejected. It is formed to acquire the luminous flux in a part of the pupil.

In FIG. 1B, the cross-hatched pixels have a red (R) color filter. Pixels with downward-sloping hatching have a green (G) color filter. Pixels with downward-sloping hatching have a blue (B) color filter. As is clear from FIG. 1B, the color filters form a so-called Bayer array (minimum color array is 2x2) and are provided for each pixel. In FIG. 1 (b), the blocks of this minimum color arrangement are designated by reference numerals 101, 102, 103, and 104. As described above, the blocks 101, 102, 103, and 104 have four pixels, and the color filters are arranged according to the Bayer array.

In FIG. 1B, the pupil dividing means has a more complicated arrangement than the color filter. This will be described below. The pupil dividing means of FIG. 1B has four patterns of 2x2. That is, focusing on the block 101, R1 has an opening at the upper left, Gr2 at the upper right, Gb3 at the lower left, and B4 at the lower right. Since there are two phases in the vertical and horizontal directions (ie, the numbers are both in the vertical and horizontal patterns of the opening positions), the minimum split array is 2x2. Blocks 101, 102, 103, and 104 each have a different pattern. In FIG. 1B, characters such as R1 written in the pixel indicate a color filter and a pupil division pattern. The color filter corresponds to the English letter in front, R indicates red, Gr indicates green on the R line, Gb indicates green on the B line, and B indicates blue. On the other hand, the pupil division pattern (hereinafter, also referred to as "opening pattern") corresponds to the number behind, 1 corresponds to the upper left opening, 2 corresponds to the upper right opening, 3 corresponds to the lower left opening, and 4 corresponds to the lower right opening. There is.

The block 101 and the block 102 are adjacent to each other in the horizontal direction. In the example of FIG. 1 (b), for two blocks having the smallest color arrangement adjacent to each other in the horizontal direction, the aperture pattern at a specific pixel position of one block is symmetrical in the block and the pixel positions are symmetrical within the block. It is configured to appear in. That is, in the block 101, the upper left pixel position in the block has a pattern of the upper left opening with the number 1, but in the block 102, the upper left pixel position symmetrical in the block has the same upper left opening pattern. As a result, in the block 102, the pattern 1 of the upper left opening corresponds to Gr. Similarly for the other pixels, the block 102 is composed of R2, Gr1, Gb4, and B3 by selecting the opening pattern so as to be symmetrical in the block.

The block 101 and the block 103 are adjacent to each other in the vertical direction. In the example of FIG. 1B, for two blocks having the smallest color arrangement vertically adjacent to each other, the aperture pattern at a specific pixel position of one block is vertically symmetrical within the block, and the pixel position of the other block is vertically symmetrical. It is configured to appear in. That is, in the block 101, the upper left pixel position in the block has a pattern of the upper left opening with the number 1, but in the block 103, the block 103 has a pattern of the upper left opening having the same pixel position in the lower left symmetrically in the block. As a result, in the block 103, the pattern 1 of the upper left opening corresponds to Gb. Similarly for the other pixels, the block 103 is composed of R3, Gr4, Gb1 and B2 by changing the opening pattern so as to be vertically symmetrical within the block.

Similarly, the block 104 is configured to have a vertically symmetrical relationship with the block 102. At this time, the block 104 automatically has a symmetrical relationship with the block 103. As a result, block 104 is composed of R4, Gr3, Gb2, and B1. By repeating this, the pupil division pattern of the entire image sensor 6 is determined. Here, considering the adjacent minimum color arrangements such as block 101 and block 102, block 101 and block 103, and the like, it can be seen that the arrangement start patterns of the pupil dividing means are different. In the block 101, the upper left pixel in the block is the pattern 1 in which the upper left is open, and in the block 102, the upper left pixel in the block is the pattern 2 in which the upper right is open. It can be seen that the arrangement start patterns of the pupil dividing means are also different for the other adjacent blocks.

By repeating the above operation, it can be seen that the same array is repeated in units of 16 pixels of 4x4 composed of blocks 101, 102, 103, and 104. This size is defined as the smallest unit array. That is, when a specific minimum unit array is compared with another minimum unit array, the two pixels corresponding to each other in the minimum unit array have the same combination of color and aperture position. The minimum unit array is determined by the rules of phase change of the minimum color array, the minimum divided array, and the minimum divided array. The rule of the phase change of the minimum divided array means, for example, in the example of FIG. 1B, the rule that the horizontal direction of the adjacent minimum color array is symmetrical and the vertical direction is vertically symmetrical. In the case of FIG. 1B, the minimum color array = 2x2, the minimum divided array = 2x2, and the minimum unit array = 4x4.

FIG. 1 (c) is a diagram corresponding to processing (S130 and S140 in FIG. 1 (a)) when the phase difference is less than the threshold value. As described with reference to FIG. 4, when the phase difference is less than the threshold value, the images that have passed through different regions on the exit pupil surface can be used as they are for color interpolation processing and the like. Therefore, the camera system control unit 5 discards information (viewpoint information) related to pupil division. In FIG. 1 (c), the small square provided in the pixel and the numerical portion of the character written in the pixel are deleted with respect to FIG. 1 (b). This corresponds to discarding the information (viewpoint information) related to pupil division. As is clear from FIG. 1 (c), this is an image sensor having a normal Bayer array, and color interpolation is performed by performing well-known image processing. That is, when the phase difference is less than the threshold value, all kinds of colors (here, RGB) of the minimum color array are obtained by performing color interpolation processing on the signals obtained by the pixels corresponding to the opening positions of all patterns. A color image containing is generated.

On the other hand, FIGS. 1 (d) and 1 (e) are diagrams corresponding to the processing (S150 and S160 in FIG. 1A) when the phase difference is equal to or larger than the threshold value. As described with reference to FIG. 4, when the phase difference is equal to or larger than the threshold value, the images that have passed through different regions on the exit pupil surface cannot be used as they are for interpolation processing or the like. However, it is possible to perform interpolation processing on an image that has passed through the same region on the exit pupil surface. Therefore, as shown in FIG. 1D, the camera system control unit 5 extracts the signal of the image that has passed through the same region on the exit pupil surface. In FIG. 1 (d), only the pixel with the subscript 1 attached and the upper left opening is shown in FIG. 1 (b). As is clear from comparing FIGS. 1 (c) and 1 (d), sampling is sparse in FIG. 1 (d). Therefore, a high frequency component cannot be obtained. However, when the phase difference is equal to or larger than the threshold value, the image is out of focus, and the high frequency component is optically cut off. Therefore, there is little disadvantage associated with performing the process of FIG. 1 (d). Focusing on the blocks 101, 102, 103, and 104 of FIG. 1 (d), R, Gr, Gb, and B are present, respectively. The values of R, Gr, Gb, and B are taken as representative values of blocks 101, 102, 103, and 104, respectively. FIG. 1 (e) is a conceptual diagram of this representative value. Actually, the pixel R1 in the block 101 captures only a part of the luminous flux in the block 101. However, in the case of out-of-focus, the image appears to be spread out, so that the value corresponding to a part of the luminous flux may be a value representing the block 101. Looking at FIG. 1 (e), this is an image sensor having a normal Bayer array, and color interpolation is performed by performing well-known image processing. That is, when the phase difference is equal to or greater than the threshold value, all kinds of colors in the minimum color array (here, by performing color interpolation processing on the signal obtained by the pixel corresponding to the specific opening position (here, upper left). Then, a color image including RGB) is generated.

In FIGS. 1 (d) and 1 (e), the pixel with the subscript 1 attached and the upper left opening is focused on, but the same processing is performed by paying attention to other pupil division patterns. In the example of FIG. 1 (b), since the pupil dividing means has four patterns of 2x2, the processes shown in FIGS. 1 (d) and 1 (e) are performed for each of the four patterns. After that, the processing results of each pattern are synthesized in S170 of FIG. 1 (a). In the synthesis of S170, since the phase difference of each pupil division pattern is known, addition processing or the like is performed after the phase matching is performed using this. Even if all four patterns are not combined, if at least two patterns are combined, the above-mentioned effects such as noise reduction can be obtained to some extent. Generally speaking, in S170 of FIG. 1A, the camera system control unit 5 performs color interpolation processing on the first color image generated by performing color interpolation processing on the first pattern and color interpolation processing on the second pattern. Is combined with the second color image generated by performing.

The processing when the phase difference is equal to or larger than the threshold value is not limited to that shown in FIGS. 1 (d) and 1 (e). Other methods include, for example, a method of rearranging an image and then color interpolation (the method described in Japanese Patent Application Laid-Open No. 2009-21683), a method of rearranging an image on a focus plane, and then color interpolation. Can be mentioned.

Next, with reference to FIG. 5, a method of performing phase detection using the image pickup device shown in FIG. 1B will be described. 5 (a) and 5 (c) show a state in which a signal of a specific pupil division pattern is extracted. 5 (b) and 5 (d) show a state in which the signals of FIGS. 5 (a) and 5 (c) are converted into luminance signals for phase detection, respectively.

FIG. 5A is a diagram in which only the pattern 1 having the upper left opening is extracted from the pupil division patterns. Focusing on blocks 101, 102, 103, and 104, R1, Gr1, Gb1, and B1 are present, respectively. For phase detection, it is desirable to use signals of various wavelengths (colors) instead of using signals of a specific wavelength (color). Therefore, the camera system control unit 5 weights and adds these signals and converts them into the luminance signal Y1. Considering a simple S / N, all signals may have the same weight, but if it is desired to pay attention to a specific wavelength according to chromatic aberration and the characteristics of the developed image, the weight may be changed.

As shown in FIG. 5B, Y1 created by adding the signals of blocks 101, 102, 103, and 104 is treated as a luminance signal corresponding to the region 111 including the blocks 101, 102, 103, and 104. be able to. The camera system control unit 5 uses this luminance signal to perform phase detection. That is, in the example of FIG. 5B, the pitch of the signal for phase detection is four times the pitch of the original pixel.

Next, consider extracting another pupil division pattern. FIG. 5C is a diagram in which only the pattern 2 in which the upper right is open is extracted from the pupil division patterns. Focusing on blocks 101, 102, 103, and 104, Gr2, R2, B2, and Gb2 are present, respectively. As in the case of the pattern 1, the camera system control unit 5 weights and adds these signals and converts them into the luminance signal Y2.

As shown in FIG. 5D, Y2 created by adding the signals of blocks 101, 102, 103, and 104 is treated as a luminance signal corresponding to the region 112 including the blocks 101, 102, 103, and 104. be able to.

The camera system control unit 5 generates image signals from different angles whose pupils are divided by a plurality of types of openings as shown in FIGS. 5 (b) and 5 (d), and correlates them with an appropriate length (field of view length). The phase difference can be detected by performing an operation known as an operation.

The configuration of the image sensor 6 according to the present embodiment is not limited to that shown in FIG. 1 (b), and is arbitrary as long as the following two conditions (collectively referred to as "arrangement conditions") are satisfied. Can be adopted.

Condition 1: Vertical minimal color sequence _{n c} x lateral _m c _{(n c} and _{m c} is an integer of 2 or more), vertically minimum division sequence _{n d} x lateral _m d _{(n d} and _{m d} is an integer of 2 or more) In the case of, the minimum unit array is vertical (n _c * n _d * a) x horizontal ( _mc * m _d * b) (a and b are natural numbers).

Condition 2: The color filter of all pixels corresponding to a specific aperture position (aperture pattern) in the minimum unit array includes all kinds of colors in the minimum color array.

Hereinafter, an example of an image sensor that satisfies the arrangement condition and an example of an image sensor that does not satisfy the arrangement condition will be described with reference to FIGS. 6 to 9.

FIG. 6 is a diagram showing an example of an image sensor that does not satisfy the arrangement condition. The image pickup devices of FIGS. 6 (a) and 6 (c) both have a minimum color array = 2x2 and a minimum divided array = 2x2.

In the case of FIG. 6A, unlike FIG. 1B, the arrangement of the pupil division patterns does not change between the minimum division arrangements. The minimum color arrangement is 2x2 as in FIG. 1 (b). Therefore, in the case of FIG. 6A, the minimum unit array is 2x2. In this case, the processing (S150 and S160 in FIG. 1A) when the phase difference is equal to or larger than the threshold value cannot be appropriately performed. As shown in FIG. 6B, if only the pattern 1 signal is extracted, only the R (red) signal can be obtained. This is because the lengths of the minimum color array and the minimum division array are the same, so that only the same color filter corresponds to the same pupil division pattern. As a result, even if the color interpolation processing is performed, only a single color image can be obtained, and an RGB color image cannot be obtained.

Next, consider the arrangement of FIG. 6 (c). This is an example of naturally extending the idea of Patent Document 2 to a two-dimensional array. Regarding the pupil division pattern, the blocks 141 and 142 adjacent in the horizontal direction are symmetrically configured, and the blocks 141 and 143 adjacent in the vertical direction are also symmetrically configured. Since the focus detection based on the phase difference is considered in Patent Document 2, the baseline length direction is limited to one direction. Therefore, the condition that the adjacent blocks are symmetrical is added. In the case of FIG. 6C, the minimum unit array is 4x4. However, when the signal is extracted by paying attention to a specific aperture pattern, it is not possible to obtain all kinds of colors in the minimum unit arrangement. That is, as shown in FIG. 6D, when the signal is extracted by paying attention to the pattern 1, R1, Gr1, Gr1, and R1 are extracted from the blocks 141, 142, 143, and 144, respectively. The image of this pattern is not in a Bayer array and there is no B (blue). Therefore, an appropriate RGB color image cannot be obtained.

FIG. 7 is a diagram showing an example of an image sensor that satisfies the arrangement condition. In the case of FIG. 7A, the minimum color array = 2x2, the minimum divided array = 2x2, and the minimum unit array = 8x8. That is, the image sensor of FIG. 7 (a) has a larger minimum unit array than the image sensor of FIG. 1 (b). FIG. 7B is a diagram in which pixels of a specific pupil division pattern are extracted in FIG. 7A. In the example of FIG. 7A, the color filters are arranged in pixel units, the arrangement start patterns of the pupil dividing means are different between the adjacent minimum color arrays, and the ratio is the same as the minimum color array in the minimum unit array. It can be seen that the pixels are arranged. This will be described in detail. The color filter is shown using the same notation as in FIG. 1 (b). It can be clearly seen that each pixel has a Bayer array. Since the color filter is provided for each pixel, a high-definition image can be obtained by discarding the parallax information when the color filter is in focus. As described above, the image sensor of FIG. 7A can generate a higher image quality image by satisfying the above-mentioned condition 2 and at the same time satisfying the following condition 2'which further embodies the condition 2.

Condition 2': The color filter of all pixels corresponding to a specific aperture position (aperture pattern) in the minimum unit array includes each color in the same ratio as the minimum color array.

Pay attention to blocks 201 and 202 of the minimum color array. Block 201 and block 202 are adjacent to each other in the horizontal direction. The pupil division pattern of the adjacent minimum color array blocks is vertically symmetrical within the block. That is, in the block 201, the pattern of the upper left opening with the number 1 corresponding to R, which is located in the upper left position in the block, is located in the lower left position in the vertically symmetrical block in the block 202. As a result, pattern 1 of the upper left opening corresponds to Gb. Similarly, the block 202 is composed of R3, Gr4, Gb1 and B2 by changing the opening pattern so that the other patterns are vertically symmetrical in the block.

Next, pay attention to the blocks 201 and 203 of the minimum color array. Block 201 and block 203 are vertically adjacent to each other. The pupil division pattern of the adjacent minimum color array blocks is symmetrical within the block. That is, in the block 201, the pattern of the upper left opening with the number 1 which is located at the upper left position in the block and corresponds to R is located at the upper right side in the symmetrical block. As a result, pattern 1 of the upper left opening corresponds to Gr. Similarly, the block 203 is composed of R2, Gr1, Gb4, and B3 by changing the opening pattern so that the other patterns are symmetrical in the block.

Similarly, the block 204 may be symmetrical with the block 202. At this time, the block 204 is automatically vertically symmetrical with the block 203. Further, in the arrangement of FIG. 7A, a device is devised to increase the size of the minimum unit arrangement. Consider an intermediate block 211 that includes blocks 201, 202, 203, and 204 as shown in FIG. 7B. Further, consider the intermediate blocks 212, 213, and 214 so that they are of the same size and adjacent to each other. In the example of FIG. 1B, the intermediate block 211 and the intermediate block 212 have the same arrangement, but in the example of FIG. 7A, the minimum color arrangement blocks 201, 202, 203, and 204 are set to 1. It is considered as one mass and symmetrically rearranged between adjacent intermediate blocks.

Pay attention to the intermediate block 211 and the intermediate block 212. The intermediate block 211 and the intermediate block 212 are adjacent to each other in the horizontal direction. In this adjacent intermediate block, the block arrangement of the minimum color array is vertically symmetrical within the intermediate block.

Pay attention to the intermediate block 211 and the intermediate block 213. The intermediate block 211 and the intermediate block 213 are adjacent to each other in the vertical direction. In this adjacent intermediate block, the block arrangement of the minimum color array is symmetrical within the intermediate block.

In the same manner thereafter, the block arrangement of the minimum color array is arranged vertically symmetrically between the intermediate blocks adjacent in the horizontal direction, and the block arrangement of the minimum color array is arranged symmetrically between the intermediate blocks adjacent in the vertical direction. As a result, the arrangement is as shown in FIGS. 7A and 7B. As a result, the smallest unit array is 8x8 in size, including intermediate blocks 211, 212, 213, and 214.

FIG. 7B is a diagram in which only the pixels of the specific pupil division pattern (pattern 1) are extracted, and it can be seen that this also has a Bayer arrangement. That is, it is possible to output an image from a specific viewpoint in RGB color. Another advantage of the pixel arrangement shown in FIGS. 7 (a) and 7 (b) is that the pixels corresponding to a specific viewpoint are present in every column / row. This reduces the possibility of failing to catch the thin line.

FIG. 7 (c) shows an example in which the aperture ratio of the mask for pupil division is changed, which is equivalent to the arrangement shown in FIG. 1 (b). FIG. 7 (d) is a diagram in which pixels of a specific pupil division pattern are extracted in FIG. 7 (c). FIG. 7 (e) illustrates an operation for obtaining a signal equivalent to that shown in FIG. 1 (b) from the opening shown in FIG. 7 (c). In the case of FIG. 7C, the minimum color array = 2x2, the minimum divided array = 2x2, and the minimum unit array = 4x4.

In FIG. 7C, the color filter is provided for each pixel, and when the color filter is in focus, a high-definition image can be obtained by discarding the parallax information. Further, as is clear from FIG. 7D in which the signal of the specific pupil division pattern is extracted, it is also possible to output the image from the specific viewpoint in RGB color.

In the example of FIG. 1B, for the sake of clarity, an example in which 1/4 of the pixels are open is shown. On the other hand, if only 1/4 is opened, the 3/4 signal is shielded by the mask layer and is wasted. Therefore, in the example of FIG. 7C, 3/4 is opened to shield 1/4 from light. That is, FIG. 7 (c) has a complementary color relationship with that of FIG. 1 (b). This is illustrated by the large white area, which is different from the example of FIG. 1 (b). Opening the 3/4 is advantageous in terms of S / N. On the other hand, it is not possible to explicitly obtain a signal with a split pupil. Therefore, by performing the calculation as shown in FIG. 7 (e), a signal equivalent to that in the case of FIG. 1 (b) can be obtained. That is, as shown in FIG. 7 (e), all aperture patterns are added and multiplied by an appropriate gain (1/3 in FIG. 7 (e)) to obtain a full aperture signal. By reducing the pattern (pattern 4 in FIG. 7E) that blocks the pupil region to be extracted from this region, only the signal in a specific region can be obtained. This calculation may be performed according to the application, such as when it is necessary to output an image of a specific viewpoint.

8 (a) and 8 (b) are diagrams showing an example of an image sensor satisfying the arrangement condition. In the case of FIG. 8A, the minimum color array = 2x2, the minimum divided array = 4x4, and the minimum unit array = 8x8. Since there are a total of 16 pupil division patterns, they are entered numerically in the pixels. Regarding the color, the same notation (hatching pattern) as in FIG. 1 (b) was used, but the English characters in the pixels were not shown. FIG. 8B is a diagram in which pixels of a specific pupil division pattern are extracted in FIG. 8A.

In FIG. 8A, the color filters are arranged in pixel units, the arrangement start patterns of the pupil dividing means are different between the adjacent minimum color arrays, and the pixels are arranged in the minimum unit array at the same ratio as the minimum color array. You can see that they are arranged. The color filter is shown using the same notation as in FIG. 1 (b), but it can be clearly seen that the Bayer array is arranged for each pixel. Since the color filter is provided for each pixel, a high-definition image can be obtained by discarding the parallax information when the color filter is in focus.

In FIG. 8A, consider blocks 161, 162, 163, and 164 of the smallest split array. Focus on blocks 161 and blocks 162 of the smallest split array. Blocks 161 and 162 are laterally adjacent to each other. The pupil division pattern of the adjacent minimum division array blocks is vertically symmetrical within the block. That is, in the block 161, the pattern 1 of the opening closest to the upper left, which is located at the upper left position in the block and has the number 1 corresponding to R, is located in the lower left position in the vertically symmetrical block in the block 162. As a result, the pattern 1 of the opening closest to the upper left corresponds to Gb. Similarly, the other patterns were changed so that they were vertically symmetrical within the block.

Next, pay attention to blocks 161 and blocks 163 of the minimum split array. Blocks 161 and 163 are vertically adjacent to each other. The pupil division pattern of the adjacent minimum division array blocks is symmetrical within the block. That is, in the block 161, the pattern 1 of the upper left opening with the number 1 corresponding to R, which is located in the upper left position in the block, is located in the upper right position in the symmetrical block in the block 163. As a result, pattern 1 of the upper left opening corresponds to Gr. Similarly, the other patterns were changed so that they were symmetrical in the block.

Similarly, the block 164 may be symmetrical with the block 162. At this time, the block 164 automatically becomes vertically symmetrical with the block 163. By repeating this, an array having the same effect as that described in FIGS. 1 (b) and 7 can be obtained. As is clear from FIG. 8 (b), focusing on the specific pupil division pattern (pattern 1) in the array of FIG. 8 (a), R, Gb, Gr, from blocks 161, 162, 163, and 164, respectively, The signal of B is obtained. These signals have the same ratio as the minimum color arrangement, and the arrangement is also a Bayer arrangement, maintaining the arrangement of the minimum color arrangement. As a result, an image of a specific viewpoint can be obtained in RGB color.

Next, FIG. 8C will be described. In the case of FIG. 8C, the minimum color array = 2x2, the minimum divided array = 4x4, and the minimum unit array = 4x4. The example of FIG. 8C can be considered to apply the example of FIG. 6A to the minimum divided array 4x4. That is, in this sequence, when the phase difference is less than the threshold value, the same effect as in FIG. 1B can be obtained by discarding the parallax information. On the other hand, when the signal is extracted by paying attention to the pattern 1, an RGB color image cannot be obtained. FIG. 8D is an example in which only the signal of pattern 1 is extracted, and only the signal of R (red) can be obtained.

FIG. 9 is a diagram showing an example of an image sensor that satisfies the arrangement condition. In FIG. 9A, the minimum color array = 2x2, the minimum divided array = 3x3, and the minimum unit array = 6x6. FIG. 9B is a diagram illustrating an arrangement of pixels when focusing on an image from a specific viewpoint. FIG. 9C is a diagram corresponding to FIG. 3C, and is a diagram for explaining the correspondence between the cross-sectional view of the pixel and the exit pupil surface.
In the example of FIG. 9A, a high-definition image can be obtained by discarding the parallax information when the image is in focus. Further, even when a specific division pattern is extracted, a Bayer array signal can be obtained as shown in FIG. 9B, and an RGB color image can be obtained.

However, inconvenience occurs when focus detection based on the phase difference is performed using the signal output by the image sensor shown in FIG. 9A. 9 (c) is a diagram corresponding to FIG. 3 (c), and structures having the same function are numbered the same. What is noted here is the case where the exit pupil 50 is narrowed down. The narrowed pupil is shown by reference numeral 302 in FIG. 9 (c) and by reference numeral 301 in FIG. 3 (c). In the example of FIG. 3 in which the pupils are divided by even numbers, light rays are incident on a plurality of pixels even when the exit pupils are narrowed down. On the other hand, in the example of FIG. 9 in which the pupils are divided by an odd number, when the diaphragm is stopped down to reference numeral 302, the light beam is incident only on the central pixel. This means that the phase difference information cannot be obtained. That is, it is shown that the focus cannot be detected by the signal from the image sensor in the situation where the aperture is stopped down when shooting a moving image outdoors. To avoid this, the pupil should be divided into even numbers. However, the image sensor of FIG. 9A can be used without any problem when the aperture amount is limited or when the phase difference detection process is performed by another known method.

As described above, according to the present embodiment, an image pickup device that satisfies the arrangement condition is provided. This makes it possible to execute color interpolation processing that suppresses image quality deterioration according to the focus state.

[Other Embodiments]
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1 ... Camera, 2 ... Lens, 3 ... Shooting optical system, 4 ... Optical axis, 5 ... Camera system control unit, 6 ... Imaging element, 7 ... Image processing unit, 8 ... Memory unit, 9 ... Display unit, 10 ... Operation Detection unit, 11 ... electrical contact, 12 ... lens system control unit, 13 ... lens drive unit

Claims (9)

Multiple pixels, each containing a photoelectric conversion unit,
Microlenses arranged in pixel units and
A color filter arranged in pixel units by repeating a minimum color array of vertical n _c x horizontal _mc (n _c and _mc are integers of 2 or more), and
A mask means having an aperture and a light-shielding portion arranged in pixel units between the photoelectric conversion unit and the microlens.
With
The masking means
When the vertical and horizontal n _d and m _d are the number of patterns of the open position _{(n d} and m _d is an integer of 2 or more) as a combination of color filters and the same open position of the same color vertically ( It appears by repeating the smallest unit array of n _c * n _d * a) x horizontal ( _mc * m _d * b) (a and b are natural numbers).
Color filters of all the pixels corresponding to the particular open position in the minimum unit sequence, see contains each color at the same rate as the minimum color sequence,
For any of the patterns of the opening positions of ( _nd * m _d ), the imaging is characterized in that the pixels corresponding to the pattern are configured to be present in any column and every row of the minimum unit array. element. Said mask means, for any pattern of open position also (n d _{* m d),} which column of two or more pixels the minimum unit sequences, each corresponding to the color filters of different colors as well as corresponding to the pattern The image pickup device according to claim 1, wherein the image pickup device is configured to be present in any row . The image pickup device according to claim 1 or 2, wherein the minimum color arrangement is a Bayer arrangement. An image pickup apparatus comprising the image pickup device according to any one of claims 1 to 3. Phase difference detecting means for detecting the phase difference of the subject image corresponding to the incident light from a plurality of different pupil regions of the photographing optical system, and
A color interpolation means for generating a color image including all kinds of colors of the minimum color array by performing color interpolation processing on a signal obtained by at least a part of the plurality of pixels.
With more
The color interpolation means is characterized in that when the phase difference is equal to or larger than a threshold value, the color image is generated by performing the color interpolation processing on the signal obtained by the pixels corresponding to the specific aperture position. The imaging device according to claim 4. The color interpolation means is used when the phase difference is equal to or greater than a threshold value.
A second color image is generated by performing the color interpolation processing on the signal obtained by the pixels corresponding to the second aperture position different from the specific aperture position.
The imaging apparatus according to claim 5, wherein the color image and the second color image are combined. The color interpolation means is characterized in that when the phase difference is less than the threshold value, the color image is generated by performing the color interpolation processing on the signal obtained by the pixels corresponding to the aperture positions of all patterns. The imaging device according to claim 5 or 6. An image generation method executed by an image pickup apparatus including the image pickup device according to any one of claims 1 to 3.
A phase difference detection step that detects the phase difference of the subject image corresponding to the incident light from a plurality of different pupil regions of the photographing optical system, and
A color interpolation step of generating a color image including all kinds of colors of the minimum color array by performing color interpolation processing on a signal obtained by at least a part of the plurality of pixels.
With
The color interpolation step is characterized in that when the phase difference is equal to or larger than a threshold value, the color image is generated by performing the color interpolation processing on the signal obtained by the pixels corresponding to the specific aperture position. Image generation method to be performed. A program for causing a computer to function as each means of the imaging device according to any one of claims 5 to 7.