JP6541503B2 - Image pickup apparatus, image processing apparatus, image processing method and program - Google Patents

Description

  The present invention relates to an imaging device, an image processing device, an image processing method, and a program.

  In an imaging apparatus using a single-plate sensor in which a color filter such as a Bayer array is arranged, color signals that can be generated for each pixel are different at the time of imaging, and aliasing (distortion) occurs due to the array of color filters. Image quality may be impaired. Further, there has been proposed an imaging apparatus capable of dividing an exit pupil of a photographing lens into a plurality of regions and simultaneously photographing a plurality of parallax images according to the divided pupil regions. For example, Patent Document 1 includes one microlens and a plurality of photoelectric conversion units (hereinafter, referred to as sub-pixels) for each pixel, and a sub-pixel includes one of RGB color filters 2. An imaging device using a two-dimensional imaging device is proposed.

Patent No. 5359465 gazette

  However, in the configuration of Patent Document 1, depending on the in-focus state at the time of imaging, a shift in the position at which the sub-pixel in the pixel receives a light beam (hereinafter referred to as "gravity shift") occurs. At the time of demosaicing, rattling of the edge due to center-of-gravity shift occurs. Further, the center-of-gravity shift amount differs depending on the in-focus state at the time of imaging, and it is difficult to determine the local in-focus state of the captured image and appropriately correct the center-of-gravity shift according to the in-focus state.

  The present invention provides an imaging device, an image processing device, and an image processing device capable of generating a high quality image signal by a simple method based on signals from a plurality of sub-pixels receiving light from different pupil regions. It is an exemplary object to provide methods and programs.

The imaging device of the present invention has a pixel array consisting of a plurality of pixels, and color filters of a plurality of colors regularly arranged on the pixel array, and each pixel is a different pupil area of the imaging optical system. And a plurality of images for generating an output image on the pixel array, and an imaging device including a plurality of photoelectric conversion units for respectively receiving the luminous fluxes passing through and photoelectrically converting an object image formed by the imaging optical system. A generation position is set, and for each image generation position, a plurality of photoelectric conversion units to be used are selected from the plurality of pixels, and the plurality of photoelectric conversion units are selected based on signals from the selected plurality of photoelectric conversion units. Image processing means for setting all the color information of the colors, wherein the image processing means is a position where the selected photoelectric conversion unit receives the light flux for each image generation position; Among the pixels corresponding to the selected photoelectric conversion unit A weight for canceling the deviation amount between the position, the color information set by adding to weighted signals from said selected photoelectric conversion unit, in the pixel array, said plurality of image generating position And a center position of the plurality of pixels .

  According to the present invention, an imaging device and an image processing device capable of generating a high quality image signal by a simple method based on signals from a plurality of sub-pixels receiving light from different pupil regions. An image processing method and program can be provided.

It is a block diagram showing an example of composition of an imaging device of the present invention. It is a schematic diagram for demonstrating the pixel array of the image pick-up element shown in FIG. It is a schematic diagram which shows the divided | segmented pupil area | region in the imaging optical system shown in FIG. FIG. 4 is a schematic view showing how a light beam passing through a divided pupil region shown in FIG. 3 is incident; It is a block diagram of the color signal generation processing means in the camera signal processing circuit shown in FIG. FIG. 5 is a schematic view showing sampling positions of sub-pixels and sampling positions of an output image. FIG. 6 is a schematic view showing a G sub-pixel and a sampling position of an output image. It is the model which showed the sub pixel of R, and the sampling position of an output image. FIG. 7 is a schematic view showing a B sub-pixel and a sampling position of an output image. It is a schematic diagram explaining the operation | movement of gravity center shift correction processing of R, B of this invention. It is a schematic diagram which shows that the pixel signal correct | amended by this invention is produced | generated.

  FIG. 1 is a block diagram showing an example of the configuration of an imaging device according to an embodiment of the present invention. The imaging device 100 may be a camera body in a single-lens reflex camera or a non-reflex camera (mirrorless camera), or may be a lens integrated type. The imaging device 100 may be a digital still camera, a digital video camera, a television camera, or the like.

  In FIG. 1, an imaging apparatus 100 includes an imaging optical system 101, an imaging element 102, a camera signal processing circuit 103, a frame memory 104, a drive control unit 105 of the imaging optical system 101, a display unit 106, a recording unit 107, and an operation system. 108, system control means 109 is provided. The camera signal processing circuit 103, the frame memory 104, the display unit 106, the recording unit 107, and the operation system 108 are mutually connected via a bus line 110.

  The imaging optical system 101 forms a subject image. That is, the imaging optical system 101 includes optical elements such as a focus lens, a zoom lens, an aperture, and a camera shake correction lens, and captures incident light (image light) from a subject and forms an image on the imaging surface of the imaging element 102 . Further, the optical elements of the imaging optical system 101 are driven and controlled based on a control signal from the drive control unit 105. The drive control unit 105 is controlled by the system control unit 109. The focus lens is moved in the optical axis direction to perform focusing. In focusing, the imaging device 102 may be moved in the optical axis direction. The drive means of the focus lens and the imaging device 102 can be configured by an actuator such as a DC motor, a stepping motor, or a voice coil motor. The drive control means 105 controls the drive of the drive means.

  The imaging device 102 is an imaging device including a two-dimensional CMOS photosensor and a peripheral circuit, and is disposed on the imaging surface of the imaging optical system. The imaging element 102 converts the light amount of incident light focused on the imaging surface by the imaging optical system 101 into an electrical signal in pixel units and outputs the electrical signal as a pixel signal. As described above, the image sensor 102 photoelectrically converts the subject image formed by the imaging optical system 101, and the detailed structure thereof will be described later.

  The camera signal processing circuit 103 temporarily stores, for example, an image signal of one frame output from the imaging element 102 in the frame memory 104, and performs various signal processing (white balance, gamma) on the stored image signal. Perform correction, demosaicing, etc. The camera signal processing circuit 103 may be composed of a digital signal processor (DSP). Demosaicing is generally performed by interpolating the missing color information (brightness information, pixel value information) from the signal obtained from the image pickup device to obtain a plurality of colors set in the color filter for each pixel. Color interpolation processing to set color information for all.

  FIG. 5 is a block diagram showing a configuration of color signal generation processing means (demosaicing means) 1030 which the camera signal processing circuit 103 has. The color signal generation processing unit 1030 sets a plurality of image generation positions in the pixel array, selects a plurality of sub-pixels to be used from the plurality of pixels for each image generation position, and selects a plurality of selected pixels. All color information is set based on the signal from the sub-pixel. The image generation position is a position set in the pixel array to generate an output image, and may be hereinafter referred to as "the output image sampling position". Conventionally, the sampling position of the output image has been the center position of each pixel.

  When the present invention is applied to an image processing apparatus, the image processing apparatus comprises means for setting the plurality of image generation positions, means for selecting the sub-pixels, and the selected sub-pixels. And means for setting all color information based on the signal. When the present invention is applied to an image processing method, the image processing method comprises the steps of setting the plurality of image generation positions, selecting the sub-pixels, and the selected sub-pixels. Setting all color information based on the signal.

  The color signal generation processing unit 1030 includes an R signal interpolation processing unit 1031, a B signal interpolation processing unit 1032, and a G signal interpolation processing unit 1033 inside.

  The image processing method (signal processing method) of the present embodiment may be performed inside the imaging apparatus 100, or a dedicated image processing apparatus (signal processing apparatus) or a program of the image processing method of the present embodiment is installed. It may be performed outside the imaging device 100, such as a personal computer. Such an image processing apparatus, an image processing method, and a program also constitute one aspect of the present invention. At this time, each step of the image processing method may be executed by software by a computer or may be executed by dedicated hardware.

  Returning to FIG. 1, the display unit 106 includes a panel type display unit such as a liquid crystal display unit, and displays a moving image or a still image captured by the image sensor 102. The image processing method of the present invention may be applied when the display unit 106 generates a live view image for sequentially displaying the image processed by the image processing means. The recording unit 107 records a moving image or a still image captured by the imaging element 102 on a storage medium such as an SD card.

  The operation system 108 includes a button, a dial, a lever, a touch panel, and the like which issue various operation commands in accordance with the operation by the user. The operation system 108 includes a focus selection switch. The focus selection switch (focus selection means) can select the manual focus (MF) mode, the auto focus (automatic focus adjustment, AF) mode, and the defocus mode. In the MF mode, the user operates a focus ring (not shown) to move the focus lens and perform manual focus adjustment. In the AF mode, focus detection is performed using a phase difference detection method, a contrast detection method, or the like, and the focus lens is automatically moved to the in-focus position to perform focus adjustment. In the defocus mode, the focus lens is automatically moved to a position defocused from the in-focus position by a predetermined amount. In the defocused state, the effects of the present embodiment described later can be obtained, but due to manufacturing errors and the like, the focus lens often does not completely coincide with the focus position even in the AF mode, and the effects of the present embodiment described later can be obtained. There is a case.

  The system control unit 109 is a control unit that controls the operation of each unit in the imaging apparatus 100.

  Next, the configuration of the image sensor 102 and acquisition processing of the pupil-divided image will be described. The imaging device 102 includes a pixel array formed of a plurality of pixels, color filters of a plurality of colors regularly arranged on the pixel array, and a plurality of microlenses. Each pixel is provided with sub-pixels as a plurality of photoelectric conversion units that respectively receive light fluxes passing through different pupil regions of the imaging optical system 101 and photoelectrically convert an object image formed by the imaging optical system 101. .

  FIG. 2 shows a part of the pixel array of the imaging device 102. As shown in FIG. FIG. 2A is a schematic view of the image pickup surface of the image pickup device 102 as viewed from above, and the plurality of pixels 200, 201, 202, and 203 are signals of the same color component corresponding to one pixel of the output image. Generate The pixel outputs an image signal in a sampling period of ΔX in the horizontal direction and in the vertical direction. ΔX is an interval larger than the subpixel interval (ΔX / 2). 300 and 302 represent center positions of corresponding pixels.

  One micro lens is disposed in each of the pixels 200, 201, 202, and 203, and FIG. 2 shows the micro lens as a circle. Each pixel has a total of four sub-pixels, two each in the horizontal and vertical directions. With such a configuration, by reading out the signal picked up for each sub-pixel, a plurality of areas (shown by broken lines) obtained by dividing the exit pupil of the imaging optical system 101 shown by the solid line in FIG. It is possible to simultaneously capture parallax images according to C and D. A color filter of any of R, G, and B is disposed in the sub-pixel, and a color filter of the same color is disposed in the sub-pixel associated with the common microlens. For example, color filters of the Bayer arrangement shown in FIG. 2 (B) are arranged.

  Next, with reference to FIG. 4, the sampling position of the subject image sampled by the sub-pixel at the time of imaging will be described. FIGS. 4A and 4B are schematic diagrams showing that light beams 400 (solid line) and 401 (broken line) passing through the divided pupil regions are incident on the imaging surface 30 of the imaging element 102. The luminous fluxes 400 and 401 indicate luminous fluxes passing through the pupil regions A and D, respectively.

  Here, when light from a subject forms an image on the imaging surface 30 of the imaging device 102, as shown in FIG. 4A, the light beams 400 and 401 that have entered the imaging device 102 through the divided pupil regions. Are gathered at one point of the imaging plane 30. Therefore, as shown in FIG. 4C, the centers of gravity of the sub-pixels corresponding to the divided pupil areas A, B, C, and D acquired by the imaging device 102 coincide with each other. The directions E in FIGS. 4A and 4C correspond to each other, and the directions E in FIGS. 4B and 4D correspond to each other. The positions where the sub-pixels Ra, Rb, Rc, and Rd in the pixel 200 shown in FIG. 2B receive the light flux coincide with the center position 300 of the pixel 200 in FIG. 4A. The positions at which the sub-pixels Ba, Bb, Bc, and Bd in the pixel 203 shown in FIG. 2B receive the luminous flux coincide with the center position 302 of the pixel 203 in FIG. 4A. As described above, the position where the sub-pixel generates the image may be referred to as "sampling position" or "center of gravity position".

  However, in practice, due to a control error or the like of the imaging optical system 101, imaging is often performed at a position deviated from the in-focus position. In this case, the luminous fluxes passing through the divided pupil regions and incident on the image sensor 102 do not converge at one point as shown in FIG. 4 (B).

  In this case, as shown in FIG. 4D, the position of the center of gravity of the sub-pixel deviates from the position of the center of gravity of the pixel. For example, the positions 311 to 314 at which the sub-pixels Ra, Rb, Rc, and Rd receive light beams have deviation amounts within ΔX / 2 in the vertical direction and the horizontal direction from the center position 300 of the pixels. Hereinafter, the amount of deviation between the position of the center of gravity of the sub-pixel and the position of the center of the pixel may be referred to as a “phase difference”.

  As described above, when the subject image is acquired by pupil division, when the image is not completely in focus at the time of imaging, the sub-pixels are individually read to obtain an interval (for example, ΔX / 4) finer than the pixel interval ΔX. Makes it possible to sample the subject image. The camera signal processing circuit 103 generates (sets) all RGB color signals at positions different from the center of the pixel, using the signal oversampled by the sub-pixel.

  The camera signal processing circuit 103 uses all of the signals oversampled by the sub-pixels to generate all of R, G, and B at positions different from the center of gravity of the pixel (for example, positions 301 shown in FIGS. 4C and 4D). Generate a color signal of The amount of deviation between the position of the center of gravity of the sub-pixel and the center of the pixel differs depending on the in-focus state at the time of imaging. Each of the R signal interpolation processing 1031, the B signal interpolation processing 1032, and the G signal interpolation processing 1033 has gravity center shift correction means, and generates (sets) RGB color signals while performing gravity center shift correction.

  FIG. 6 is a schematic view showing sampling positions of sub-pixels output from the imaging device 102 and sampling positions of an output image generated by the color signal generation processing unit 1030 using the sub-pixels. For example, the pixel 250 at the upper left corner of FIG. 6 includes four sub-pixels 251 to 254, and the barycenter positions of the sub-pixels 251 to 254 are 261 to 264.

  In FIG. 6, the sampling position indicated by a black circle is the sampling position of the sub-pixel output from the imaging element 102, and the sampling position indicated by a white circle is a sampling position of the output image. Since one of the R, G, and B color filters is disposed in each of the sub-pixels, the obtainable color signals differ depending on the position. At the positions of the white circles, RGB color interpolation processing is performed by the R signal interpolation processing unit 1031, the B signal interpolation processing unit 1032, and the G signal interpolation processing unit 1033, using the color information of the adjacent subpixels.

  Here, the arrangement of the color filters arranged in the adjacent subpixels differs depending on the sampling position (the position of the white circle) P00, P01, P10 and P11 of the output image. For example, the arrangement of color filters in a 4 × 4 pixel sub-pixel area centering on P00 is R, G, G, in the order of upper left, upper right, lower left, lower right when 2 × 2 is one unit. It becomes B. The arrangement of color filters in 4 × 4 pixel sub-pixel area centering on P01 is G, R, B, G and in the order of upper left, upper right, lower left, lower right when 2 × 2 is one unit. Become. The arrangement of color filters in a 4 × 4 pixel sub-pixel area centering on P10 is G, B, R, G, and in the order of upper left, upper right, lower left, lower right when 2 × 2 is one unit. Become. The arrangement of color filters in a 4 × 4 pixel sub-pixel area centering on P11 is B, G, G, R and in the order of upper left, upper right, lower left, lower right when 2 × 2 is one unit. Become.

  FIG. 7 is a schematic view showing a sub-pixel in which a G color filter is arranged, which is referred to by the G signal interpolation processing means 1033, and a sampling position of an output image to be generated.

  FIG. 7A shows an example in the case of the sampling position P00 or P11 of the output image. For example, in the case of generating a G signal at a position 600 of the intersection of a pixel having subpixels 603 to 606 and a pixel having subpixels 607 to 610 which is diagonally adjacent and has G color information, The color information of adjacent G sub-pixels 605 and 608 is averaged. The sub-pixel 608 has a phase difference corresponding to the in-focus state at the time of imaging with respect to the barycentric position of the sub-pixel 607 to 610, and the sub-pixel 605 includes the sub-pixel 603 to 606 A phase difference corresponding to the in-focus state at the time of imaging occurs with respect to the barycentric position of. The distance between the position 600 and the position of the center of gravity of the sub-pixels 605 and 608 also changes due to the sampling phase of the sub-pixel according to the in-focus state. However, the pixel centroid of the generated G signal can be made to coincide with the position 600 by averaging the sub-pixels 605 and 608.

  FIG. 7B shows an example in the case of the sampling position P01 or P10 of the output image. For example, in the case of generating the G signal at the position 620, color information of diagonally adjacent G sub-pixels 625 and 628 is averaged. The sub-pixel 625 has a phase difference corresponding to the in-focus state at the time of imaging with respect to the center of gravity of the pixel consisting of the sub-pixels 622 to 625, and the sub-pixel 628 has a pixel consisting of the sub-pixels 628 to 631 A phase difference corresponding to the in-focus state at the time of imaging occurs with respect to the barycentric position of. The distance between the position 620 and the gravity center position of the sub-pixels 625 and 628 also changes due to the sampling phase of the sub-pixel according to the in-focus state. However, by averaging the sub-pixels 625 and 628, the pixel gravity center of the generated G signal can be made to coincide with the position 620.

  Thus, when generating a G signal, a pair of subpixels of the same distance in the diagonal direction of the sampling position of the output image is selected.

  FIG. 8 is a schematic view showing R subpixels used by the R signal interpolation processing 1031 and sampling positions of an output image. In FIG. 8, positions 700, 701, 702, and 703 correspond to sampling positions P00, P01, P10, and P11 of the output image, respectively. The R signals at the positions 700, 701, 702, and 703 are generated by interpolation processing with reference to nearby R sub-pixels.

  In the signals of the R sub-pixels 710 to 713, a phase difference between the sub-pixels occurs in accordance with the in-focus state at the time of imaging. In order to perform interpolation with reference to the R sub-pixels in the vicinity so that the center of gravity coincides with the position 700, the in-focus state in the vicinity of the position 700 is accurately detected, and the gravity center shift amount caused by the in-focus state is calculated. Although it is necessary to calculate a weighting factor to be corrected, this is complicated. Therefore, the R signal interpolation processing unit 1031 generates an image signal at the target position without performing the gravity center shift correction using the determination of the local focusing state and the determination result by performing the gravity center shift correction process. Do.

  Next, center-of-gravity shift correction processing performed by the R signal interpolation processing unit 1031 will be described with reference to FIGS. 10 and 11. In the present embodiment, the camera signal processing circuit 103 outputs weights from the selected sub-pixels to the weights for canceling the phase difference of the selected sub-pixels with respect to the sampling position of each output image. Set color information by weighting and adding to. The cancellation of the phase difference needs to be performed in the pupil division direction. In this embodiment, since the pupil division is performed in the horizontal direction and the vertical direction, it is desirable to cancel the phase difference in each of the horizontal direction and the vertical direction. On the other hand, when pupil division is performed only in one of the vertical direction and the horizontal direction, the phase difference may be canceled in one of the horizontal direction and the vertical direction.

  In FIG. 10, four pixels arranged so as to be separated by one pixel in the horizontal direction and the vertical direction are drawn. The upper left pixel R1 has sub-pixels 710 to 713, the upper right pixel R2 has sub-pixels 714 to 717, the lower left pixel R3 has sub-pixels 718 to 721, and the lower right pixel R4 has sub-pixels It has 722-725.

  In FIG. 10, referring to the sub-pixels 710, 711, 712, 713, 714, 716, 718, 719, 722 in which the R color filter is disposed, the position 700 (first image generation) of the lower right corner of the pixel R1. Generate the R signal at

  First, the R signals at the position 730 are generated by performing weighted addition of the sub-pixels 710, 711, and 714 at a ratio of 1: 2: 1 by the gravity center shift correction process in the horizontal direction in the R signal interpolation process 1031. That is, the color information of the upper left sub pixel 710 of the pixel R1, the color information of the upper right sub pixel 711 and the color information of the upper left sub pixel 714 of the pixel R2 are weighted and added at a ratio of 1: 2: 1. As a result, the first color information is set at the first position 730 where the first line connecting the barycentric positions of the sub-pixels 711 and 714 intersects the outline of the pixel R1.

  Similarly, the R signal at position 731 is generated by weighted addition of the sub-pixels 712, 713, 716 at a ratio of 1: 2: 1. That is, the color information of the lower left sub pixel 712 of the pixel R1, the color information of the lower right sub pixel 713, and the color information of the lower left sub pixel 716 of the pixel R2 are weighted and added at a ratio of 1: 2: 1. As a result, the second color information is set at the second position 731 where the second line connecting the barycentric positions of the sub-pixels 712 and 713 intersects the outline of the pixel R2.

  Similarly, the R signal at position 732 is generated by weighted addition of the sub-pixels 718, 719, 722 at a ratio of 1: 2: 1. That is, the color information of the upper left sub pixel 718 of the pixel R3, the color information of the upper right sub pixel 719, and the color information of the upper left sub pixel 722 of the pixel R4 are weighted and added at a ratio of 1: 2: 1. As a result, third color information is set at a third position 732 where the third line connecting the center-of-gravity positions of the sub-pixels 718 and 719 intersects the outline of the pixel R3.

  Next, the R signal at position 700 is weighted by adding the R signals at positions 730, 731, and 732 at a ratio of 1: 2: 1 by the gravity center shift correction process in the vertical direction in the R signal interpolation processing 1031. Generate That is, the R signal is generated at the position 731 by weighted addition of the first color information, the second color information, and the third color information at a ratio of 1: 2: 1.

  When each of the central positions of the pixels R1 and R2 is vertically projected onto the first line, the position 730 internally divides the distance between the central positions of the pixels R1 and R2 projected onto the first line into 1: 3. Do. When each of the central positions of the pixels R1 and R2 is projected on the second line in the vertical direction, the position 731 internally divides the distance between the central positions of the pixels R1 and R2 projected onto the second line into 1: 3. Do. If the central positions of the pixels R3 and R4 are projected on the third line in the vertical direction, the position 732 divides the distance between the central positions of the pixels R3 and R4 projected on the third line into 1: 3 Do.

  Assuming that the in-focus states in the vicinity of the pixels R1 to R4 are the same, it is possible to consider that the amounts of phase shift between the respective barycentric positions of the pixels R1 to R4 and the sub-pixels are equal to the direction. Therefore, by performing the above-described weighted addition, it is possible to generate a signal at the target position (for example, the position 700) while correcting the gravity center shift according to the in-focus state at the time of imaging.

With reference to FIG. 11, the function of the gravity center shift correction when the R pixels are generated at the position 730 by weighted addition of the sub-pixels 710, 711, 714 at a ratio of 1: 2: 1 will be described. In FIG. 11, the barycentric positions at the time of imaging of the sub-pixels 710, 711, 714, and 715 are positions indicated by black circles. At the barycentric positions of the sub-pixels 710 and 711, phase differences of −Δd and + Δd are generated with respect to the barycentric position 740 in the horizontal direction of the pixel R 1. Further, at the barycentric positions of the sub-pixels 714 and 715, phase differences of −Δd and + Δd are generated with respect to the barycentric position 741 in the horizontal direction of the pixel R2.
Here, assuming that the signals obtained from the sub-pixels 710, 711, and 714 are p0, p1, and p2, respectively, and the signal generated at the position 730 is p1 ', the signal p1' is expressed by Equation (1). Generated on
p1 '= (p0 + p1 * 2 + p2) / 4 (1)
Further, the horizontal coordinates (x-coordinate) of the barycentric position of the signals p1 ′, p0, p1 and p2 can be expressed as x (p1 ′), x (p0), x (p1) and x (p2). By the calculation of (1), the center of gravity of the signal p1 'can be expressed by equation (2).
x (p1 ') = (x (p0) + x (p1) * 2 + x (p2)) / 4 (2)
Here, assuming that the distance between the sub-pixels is ΔX and the x-coordinate (= x (R1)) of the barycentric position 740 of the pixel R1 is 0, the x-coordinate x (R2) of the barycentric position 741 of the pixel R2 is 4ΔX. Moreover, since the phase difference according to the focusing state at the time of imaging is generated in the signals p0, p1 and p2, respectively, the x-coordinate can be expressed by Equations (3) to (5).
x (p0) = 0-Δd (3)
x (p1) = 0 + Δd (4)
x (p2) = 4ΔX-Δd (5)
Substituting the equations (3) to (5) into the equation (2), the center of gravity of the signal p1 'is expressed by the following equation.
x (p1 ') (-. DELTA.d + (. DELTA.d) * 2 + 4.DELTA.X-.DELTA.d) /4=.DELTA.X (6)
That is, the component of the center-of-gravity shift Δd generated due to the in-focus state at the time of imaging is offset, and a signal at a desired position away from the horizontal center of gravity position 730 of the pixel R1 by ΔX can be generated.

  The center-of-gravity shift correction shown in Formula (6) is also performed in the horizontal direction with respect to the positions 731 and 732. Thereafter, with reference to the signals at the positions 730, 731, and 732, the R-signal is finally generated at the position 700 by performing the center-of-gravity shift correction processing similar to the equation (6) in the vertical direction. Further, the R signal interpolation processing similar to the position 700 is also performed for the positions 701, 702, and 703, and a signal of a correct pixel position is generated, which is generated by the focusing state at the time of imaging.

  FIG. 9 is a schematic view showing a B sub-pixel used by the B signal interpolation processing means 1032 and a sampling position of an output image. In FIG. 9, positions 800, 801, 802 and 803 correspond to sampling positions P00, P01, P10 and P11 of the output image, respectively. The B signals at the positions 800, 801, 802 and 803 are generated by interpolation processing with reference to nearby B sub-pixels.

  Also in the signals of the B sub-pixels 810 to 813, a phase difference between the sub-pixels occurs in accordance with the in-focus state at the time of imaging. In order to perform interpolation with reference to the B sub-pixels in the vicinity so that the center of gravity coincides with the position 800, the in-focus state in the vicinity of the position 800 is accurately detected, and the center-of-gravity shift amount due to the in-focus state is calculated. Although it is necessary to calculate a weighting factor to be corrected, this is complicated. Therefore, the B signal interpolation processing unit 1032 performs the simple gravity center offset correction processing to determine the in-focus state locally and to correct the image signal at the target position without performing the gravity center offset correction using the determination result. Generate The processing content of the B signal interpolation processing unit 1032 is the same as that of the R signal interpolation processing unit 1031, so the description will be omitted.

  As described above, the camera signal processing circuit 103 corrects the deviation of the position of the center of gravity occurring in the sub-pixel by a simple method due to the in-focus state at the time of imaging. An image signal can be generated. A high quality image signal can be output by oversampling and imaging a subject image. Note that 1: 2: 1 can be changed according to the number of sub-pixels included in the pixel and the number of sub-pixels selected for the sampling position of the output image.

  The present invention can be applied to applications such as imaging devices such as digital cameras and image processing devices. In particular, the present invention is suitable for an imaging device for simultaneously capturing a plurality of parallax images according to divided pupil regions and generating an image signal.

DESCRIPTION OF SYMBOLS 100 ... Imaging device, 101 ... Imaging optical system, 103 ... Camera signal processing circuit (image processing means), 250 ... Pixel, 251-254 ... Sub pixel (photoelectric conversion part), P00, P01, P10, P11 ... Output image Sampling position (image generation position)

Claims (9)

It has a pixel array consisting of a plurality of pixels and color filters of a plurality of colors regularly arranged on the pixel array, and each pixel receives a light flux that has passed through different pupil regions of the imaging optical system. And an imaging device including a plurality of photoelectric conversion units for photoelectrically converting an object image formed by the imaging optical system.
A plurality of image generation positions for generating an output image are set in the pixel array, and for each of the image generation positions, a plurality of photoelectric conversion units to be used from the plurality of pixels are selected and selected. An image processing unit that sets all color information of the plurality of colors based on signals from the plurality of photoelectric conversion units;
Have
The image processing means cancels the amount of deviation between the position at which the selected photoelectric conversion unit receives the light flux and the center position of the pixel corresponding to the selected photoelectric conversion unit for each image generation position. Setting the color information by weighting and adding a weighting for the selection to the signal from the selected photoelectric conversion unit ,
In the pixel array, the plurality of image generation positions are offset from the center positions of the plurality of pixels . The color filters are arranged in a Bayer pattern, and each pixel includes a 2 × 2 photoelectric conversion unit.
When setting the color information of R and B in the Bayer array, the image processing means sets an upper left pixel among four pixels arranged apart by one pixel in the horizontal direction and the vertical direction in the pixel array. Set the position of the lower right corner as the first image generation position,
The image processing means
The color information of the top left photoelectric conversion unit of the top left pixel, the color information of the top right photoelectric conversion unit of the top left pixel, and the color information of the top left photoelectric conversion unit of the top right pixel among the four pixels By performing weighted addition at a ratio of 2: 1, a first line connecting positions where the photoelectric conversion unit at the upper left and the photoelectric conversion unit at the upper right of the upper left pixel receive the light flux and the upper left Setting first color information at a first position where pixel outlines intersect;
The color information of the lower left photoelectric conversion unit of the upper left pixel, the color information of the lower right photoelectric conversion unit of the upper left pixel, and the color information of the lower left photoelectric conversion unit of the upper right pixel are 1: 2: 1. Of the upper left pixel and the lower right one of the photoelectric conversion units at the lower left and the lower left, respectively, of the upper left pixel and the outline of the upper left pixel. Set second color information at a second position where
Color information of the photoelectric conversion unit at the upper left of the lower left pixel among the four pixels, color information of the photoelectric conversion unit at the upper right of the lower left pixel, and photoelectric conversion of the upper left of the lower right pixel among the four pixels A weighted connection of the color information of the part at a ratio of 1: 2: 1 to connect the positions where the photoelectric conversion units at the upper left and the photoelectric conversion units at the upper right of the lower left pixel receive the light flux Setting third color information at a third position where the line of lines and the outline of the lower left pixel intersect;
The color information is set at the first image generation position by weighted addition of the first color information, the second color information, and the third color information at a ratio of 1: 2: 1. The imaging device according to claim 1, wherein: When the center position of the upper left pixel and the center position of the upper right pixel are vertically projected onto the first line, the first position corresponds to the upper left pixel projected onto the first line. Internally divide the distance between the center position of and the center position of the upper right pixel to 1: 3,
When the center position of the upper left pixel and the center position of the upper right pixel are vertically projected onto the second line, the second position corresponds to the upper left pixel projected onto the second line. Internally divide the distance between the center position of and the center position of the upper right pixel to 1: 3,
When the center position of the lower left pixel and the center position of the lower right pixel are vertically projected onto the third line, the third position is the lower left projected onto the third line. The image pickup apparatus according to claim 2, wherein a distance between a center position of a pixel and a center position of the lower right pixel is internally divided into 1: 3. The color filters are arranged in a Bayer pattern, and each pixel includes a 2 × 2 photoelectric conversion unit.
The image processing means, when setting G color information of the Bayer array, sets an intersection point of two pixels having G color information adjacent in the diagonal direction as a first image generation position, The color information of the first image generation position is set by averaging the color information of two photoelectric conversion units adjacent in the diagonal direction via the image generation position of 1. The imaging device according to 1.   2. The imaging according to claim 1, wherein the image processing means weights the weights so as to offset the deviation amount in at least one of the horizontal direction and the vertical direction with respect to each image generation position. apparatus. A driving unit for driving the imaging optical system or the imaging device;
Control means for controlling the drive means to pick up an image at a position deviated from the in-focus position;
The imaging device according to any one of claims 1 to 5 , characterized in that It has a pixel array consisting of a plurality of pixels and color filters of a plurality of colors regularly arranged on the pixel array, and each pixel receives a light flux that has passed through different pupil regions of the imaging optical system. An image processing apparatus for processing a signal obtained from an image pickup device including a plurality of photoelectric conversion units for photoelectrically converting an object image formed by the imaging optical system;
Means for setting a plurality of image generation positions for generating an output image in the pixel array;
A unit for selecting a plurality of photoelectric conversion units to be used from the plurality of pixels for each image generation position;
A unit configured to set all color information of the plurality of colors based on signals from the selected plurality of photoelectric conversion units;
Have
The means for setting the color information is a shift between a position where the selected photoelectric conversion unit receives the light flux and the center position of the pixel corresponding to the selected photoelectric conversion unit with respect to each image generation position. Weight for canceling the amount is added to the signal from the selected photoelectric conversion unit ,
In the pixel array, the plurality of image generation positions are offset from the center positions of the plurality of pixels . It has a pixel array consisting of a plurality of pixels and color filters of a plurality of colors regularly arranged on the pixel array, and each pixel receives a light flux that has passed through different pupil regions of the imaging optical system. An image processing method for processing a signal obtained from an image pickup element provided with a plurality of photoelectric conversion units for photoelectrically converting an object image formed by the imaging optical system;
Setting a plurality of image generation positions for generating an output image in the pixel array;
Selecting a plurality of photoelectric conversion units to be used from the plurality of pixels for each image generation position;
Setting all color information of the plurality of colors based on signals from the selected plurality of photoelectric conversion units;
Have
In the step of setting the color information, for each image generation position, a shift between a position where the selected photoelectric conversion unit receives the luminous flux and a center position of a pixel corresponding to the selected photoelectric conversion unit. Weight for canceling the amount is added to the signal from the selected photoelectric conversion unit ,
2. The image processing method according to claim 1 , wherein in the pixel array, the plurality of image generation positions are shifted from center positions of the plurality of pixels . The program for making a computer perform the image processing method of Claim 8 .