JP6955543B2 - Image sensor and image sensor - Google Patents

Description

The present invention relates to an image pickup device and an image pickup apparatus.

In recent years, image pickup devices using CMOS image pickup devices and the like have become more sophisticated and multifunctional in order to meet various needs. As the number of pixels and high-speed imaging of CMOS image sensors are increasing, there is an increasing demand for a method capable of reading pixel signals at higher speed.

For example, as a method for performing high-speed reading, as described in Patent Document 1, a method in which an analog / digital conversion circuit (hereinafter, column ADC) is arranged for each column and digital output is performed has become widespread in recent years. .. By introducing the row ADC, it becomes possible to digitally transmit the pixel signal to the outside of the image sensor, and with the improvement of the digital signal transmission technology, high-speed reading becomes possible.

On the other hand, as an example of multi-functionalization, for example, an imaging device capable of acquiring not only light intensity distribution but also light incident direction and distance information has been proposed. Patent Document 2 discloses an image pickup device capable of focusing detection using a signal obtained from the image pickup device. By dividing the photodiode (hereinafter referred to as PD) corresponding to one microlens into two, each PD is configured to receive light from different pupil surfaces of the photographing lens. Then, the focus is detected by comparing the outputs of the two PDs. Further, a normal captured image can be obtained by adding the output signals from the two PDs constituting the unit pixel.

Japanese Unexamined Patent Publication No. 2005-278135 Japanese Patent No. 3774597

However, when focusing is detected in an image sensor as disclosed in Patent Document 2, it is necessary to read out all PD signals, so that the time required to read out PD signals becomes long and the frame rate becomes high. There is a problem that it goes down. Even if the signal read time is increased by the read method using the row ADC as in Patent Document 1, it is expected that the number of pixels and the frame rate will be further increased in the future, and further reduction of the signal read time is desired. Is done.

The present invention has been made in view of the above-mentioned problems, and an object of the present invention is to provide an image sensor capable of acquiring both captured image data and focus detection information at high speed within one frame. be.

The imaging device according to the present invention includes a pixel unit in which a plurality of unit pixels each having one microlens and a plurality of photoelectric conversion units are arranged in a matrix, and a signal output from the unit pixels of the entire pixel unit. A signal holding unit that holds a signal, and a signal processing unit that processes a signal held by the signal holding unit, and performs signal processing for generating an captured image on the signal held by the signal holding unit. Processing by the signal processing unit and a signal processing unit having a signal processing unit and a correlation calculation unit that performs a correlation calculation between signals of a plurality of photoelectric conversion units of each unit pixel held by the signal holding unit. It is characterized by including an output unit for outputting the signal.

Further, the image pickup apparatus according to the present invention includes an image pickup element, an image processing means for processing a signal read from the image pickup element to generate an image, and processing the read signal to perform focus detection. From a pixel unit in which a plurality of unit pixels each having a focus detecting means and each having a microlens and a plurality of photoelectric conversion units are arranged in a matrix, and a unit pixel of the entire pixel unit. A signal holding unit that holds the output signal and a signal processing unit that processes the signal held by the signal holding unit, and signal processing for generating an captured image on the signal held by the signal holding unit. A signal processing unit having a signal processing unit for captured images and a correlation calculation unit that performs a correlation calculation between signals of a plurality of photoelectric conversion units of each of the unit pixels held by the signal holding unit, and the signal. It is characterized by having an output unit that outputs a signal processed by the processing unit.

Further, the imaging device according to the present invention comprises a pixel unit in which a plurality of unit pixels are arranged in a matrix, a signal holding unit that holds a signal output from the unit pixels of the entire pixel unit, and the signal holding unit. A signal processing unit that processes the held signal, and is held by the signal holding unit and a signal processing unit for an image capture image that performs signal processing for generating an image to be captured by the signal held by the signal holding unit. It is characterized by including a signal processing unit having a phase difference information generation unit that generates phase difference information using the signal, and an output unit that outputs a signal processed by the signal processing unit.

Further, the image pickup apparatus according to the present invention includes an image pickup element, an image processing means for processing a signal read from the image pickup element to generate an image, and processing the read signal to perform focus detection. The image pickup element includes a focus detecting means, a pixel unit in which a plurality of unit pixels are arranged in a matrix, a signal holding unit that holds a signal output from the unit pixels of the entire pixel unit, and the signal holding unit. A signal processing unit that processes the signal held by the unit, the image processing unit that performs signal processing to generate an image to the signal held by the signal holding unit, and the signal holding unit. It is characterized by having a signal processing unit having a phase difference information generation unit that generates phase difference information using the held signal, and an output unit that outputs a signal processed by the signal processing unit.

According to the present invention, it is possible to provide an image pickup device capable of acquiring both captured image data and focus detection information at high speed within one frame.

The block diagram which shows the structure of the image pickup apparatus which concerns on 1st Embodiment of this invention. The figure explaining the structure of the image sensor of 1st Embodiment. A conceptual diagram in which a luminous flux emitted from an exit pupil of a photographing lens is incident on a unit pixel. The block diagram which shows the structure of the image sensor of 1st Embodiment. The figure explaining the pixel circuit and the readout circuit of the image pickup element of 1st Embodiment. The timing chart which shows the signal reading operation of the image pickup device of 1st Embodiment. The figure explaining the structure of the signal processing part in the image pickup device of 1st Embodiment. The figure which showed the signal output area for focus detection in the pixel area of 2nd Embodiment. The figure explaining the structure of the image sensor of 3rd Embodiment. The figure explaining the pixel circuit and the readout circuit of the image sensor of the 3rd Embodiment. The whole block diagram of the image sensor of 4th Embodiment. The whole block diagram of the image sensor of 4th Embodiment. The whole block diagram of the image sensor of 5th Embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First Embodiment)
FIG. 1 is a block diagram showing a configuration of an image pickup device having an image pickup device according to the first embodiment of the present invention. The image sensor 100 includes a light receiving unit 102, a reading unit 103, a control unit 104, a signal processing unit 105, and an output unit 106. A plurality of unit pixels are arranged in a matrix in the light receiving unit 102, and receive an optical image formed by the photographing lens 101. The configuration of the light receiving unit 102 will be described later. The reading unit 103 receives the control signal of the control unit 104, A / D converts the pixel signal output from the light receiving unit 102, and sends it to the signal processing unit 105.

The signal processing unit 105 performs arithmetic processing such as signal addition, subtraction, and multiplication on the A / D converted image signal, and selection processing of a signal to be output from the image pickup element 100 to the outside via the output unit 106. I do. In addition, the signal processing unit 105 also performs various corrections such as reference level adjustment and rearrangement of data. These processes are executed in response to a control signal from the control unit 104. The details of the signal processing unit 105 will be described later, but the signal processing unit 105 performs signal processing for captured images and signal processing for focus detection on the pixel signal obtained from the light receiving unit 102, and outputs them. Send to unit 106. The output unit 106 outputs the pixel signal processed by the signal processing unit 105 to the outside of the image sensor 100.

The image processing unit 107 receives a signal for an captured image from the output unit 106 of the image pickup element 100, and performs image processing such as defect pixel correction, noise reduction, color conversion, white balance correction, and image correction, resolution conversion processing, and an image. Performs compression processing and the like to generate still images and moving images. The phase difference detection unit 108 receives a focus detection signal from the output unit 106 and calculates a phase difference evaluation value for performing focus detection.

The overall control / calculation unit 109 comprehensively drives and controls the image sensor 100 and the entire image sensor. The display unit 110 displays an image after shooting, a live view image, various setting screens, and the like. The recording unit 111 and the memory unit 112 are recording media such as a non-volatile memory or a memory card that record and hold an image signal or the like output from the overall control / calculation unit 109. The operation unit 113 receives a user's command by an operation member provided in the image pickup apparatus, and inputs the command to the overall control / calculation circuit 106. The lens control unit 114 calculates optical system drive information based on the phase difference evaluation value calculated by the phase difference detection unit 108, and controls the focus lens position of the photographing lens 101.

Next, the relationship between the photographing lens 101, the microlens array, and the light receiving unit 102 of the image pickup element 100 in the image pickup apparatus of the present embodiment, the definition of pixels, and the principle of focus detection by the pupil division method will be described.

FIG. 2 is a view of the light receiving unit 102 of the image sensor 100 and the microlens array observed from the optical axis direction (Z direction). In the present embodiment, each microlens 202 forming the microlens array is defined as one pixel, and this is defined as a unit pixel 200. Further, two photoelectric conversion units made of a photodiode (PD) are arranged side by side in the X-axis direction with respect to one microlens 202, and are defined as photoelectric conversion units 201A and 201B, respectively.

FIG. 3 shows a state in which light that has passed through the photographing lens 101 passes through one microlens 202 and is received by the unit pixel 200 of the light receiving unit 102 of the image sensor 100 in a direction perpendicular to the optical axis (Z axis). It is a figure observed from (Y-axis direction). 302 and 303 represent the exit pupil of the photographing lens. 301 is a lens diaphragm. The light that has passed through the exit pupil is incident on the unit pixel 200 about the optical axis. As shown in FIG. 3, the luminous flux passing through the pupil region 302 passes through the microlens 202 and is received by the photoelectric conversion unit 201A, and the luminous flux passing through the pupil region 303 is received by the photoelectric conversion unit 201B through the microlens 202. Therefore, the photoelectric conversion units 201A and 201B receive light in different regions of the exit pupil of the photographing lens, respectively.

The signal of the photoelectric conversion unit 201A that divides the light from the photographing lens 101 into pupils is acquired from a plurality of unit pixels 200 arranged in the X-axis direction, and the subject image composed of these output signal groups is defined as the A image. Similarly, the signal of the photoelectric conversion unit 201B that divides the pupil is acquired from a plurality of unit pixels 200 arranged in the X-axis direction, and the subject image composed of these output signal groups is defined as the B image.

A correlation calculation is performed on the A image and the B image, and the amount of image shift (pupil division phase difference) is detected. Further, by multiplying the amount of image shift by the focal position of the photographing lens 101 and a conversion coefficient determined by the optical system, the focal position corresponding to an arbitrary subject position on the screen can be calculated. By controlling the focus position of the photographing lens 101 based on the focus position information calculated here, the imaging surface phase difference AF (autofocus) becomes possible. Further, by combining the A image signal and the B image signal with the A + B image signal, this A + B image signal can be used for a normal captured image.

Next, the configurations of the light receiving unit 102 and the reading unit 103 of the image sensor 100 will be described with reference to FIGS. 4 and 5. FIG. 4 is a block diagram showing a configuration example of a light receiving unit 102 and a reading unit 103 of the image sensor 100.

The light receiving unit 102 has a pixel unit 401 and a drive circuit unit 402. A plurality of unit pixels 200 are arranged in a matrix in the pixel unit 401 in the horizontal direction (row direction) and the vertical direction (column direction). In FIG. 4, a total of 6 unit pixels 200 are shown in 2 rows and 3 columns, but in reality, millions and tens of millions of unit pixels 200 are arranged. The drive circuit unit 402 includes a power supply circuit for driving the pixel unit 401, a TG, a scanning circuit, and the like. The pixel signal driven and read by the drive circuit unit 402 is output to the reading unit 103. The drive circuit unit 402 is driven under the control of the control unit 104 of FIG. A pixel signal from the pixel unit 401 is input to the reading unit 103, and analog-to-digital conversion (A / D conversion) is performed. The reading unit 103 is configured to include a plurality of reading circuits, for example, one reading circuit in one row.

FIG. 5 is a diagram showing an example of a read-out circuit 509 constituting the unit pixel 200 of the image sensor 100 and the read-out unit 103. In the unit pixel 200, the transfer switch 502A is connected to the photoelectric conversion unit 201A composed of the photodiode (PD), and the transfer switch 502B is connected to the photoelectric conversion unit 201B. The electric charges generated by the photoelectric conversion units 201A and 201B are transferred to the common floating diffusion unit (FD) 504 via the transfer switches 502A and 502B, respectively, and are temporarily stored. When the selection switch 506 is turned on, the electric charge transferred to the FD 504 is output to the column output line 507 as a voltage corresponding to the electric charge via the amplification MOS transistor (SF) 505 forming the source follower amplifier. A current source 508 is connected to the column output line 507.

The reset switch 503 resets the potential of the FD504 and the potentials of the photoelectric conversion units 201A and 201B to VDD via the transfer switches 502A and 502B. The transfer switches 502A and 502B, the reset switch 503, and the selection switch 506 are controlled by the control signals PTXA, PTXB, PRESS, and PSEL via signal lines connected to the peripheral drive circuit units 402, respectively.

Next, the circuit configuration of the read circuit 509 will be described. The amplifier 510 amplifies the signal appearing on the column output line 507, and the capacitance 512 is used to hold the signal voltage. Writing to the capacitance 512 is controlled by a switch 511 that is turned on and off by the control signal PSH. Vslope, which is a reference voltage supplied from a slope voltage generation circuit (not shown), is input to one input of the comparator 513, and the output of the amplifier 510 written in the capacitance 512 is input to the other input. .. The comparator 513 compares the output of the amplifier 510 with the reference voltage Vslope, and outputs either a low level or a high level depending on the magnitude relationship. Specifically, when Vslope is small with respect to the output of the amplifier 510, a low level is output, and when it is large, a high level is output. The CLK (clock) starts to move at the same time as the transition of the reference voltage Vslope starts, and the counter 514 counts up corresponding to the CLK when the output of the comparator 513 is at a high level, and counts at the same time when the output of the comparator 513 is inverted to a low level. Stop the signal.

The memory 516 holds a digital signal obtained by AD-converting the reset level signal of the FD 504 (hereinafter, "N signal"), and the memory 517 stores the signals of the photoelectric conversion unit 201A and the photoelectric conversion unit 201B as the N signal of the FD 504. A digital signal obtained by AD-converting a signal superimposed on the signal (hereinafter referred to as “S signal”) is held. Which of the memories 516 and 517 is written is sorted by the switch 515. For the signals held in the memories 516 and 517, the difference obtained by subtracting the N signal from the S signal is calculated by the CDS circuit 518. Then, under the control of the drive circuit unit 402, the signal is output to the signal processing unit 105 via the digital signal output line 519.

One read circuit 509 is arranged for each column of pixels, and the pixel signal is read in row units. In this case, the selection switch 506 is controlled in units of rows, and the pixel signals of the selected rows are collectively output to the column output line 507. The larger the number of the read circuits 509, the faster the pixel signal of the pixel unit 401 can be read out to the signal processing unit 105.

FIG. 6 is a timing chart showing an example of a charge reading operation from the unit pixel 200 of the image sensor 100 having the circuit configuration shown in FIG. The timing of each drive pulse, the reference voltage Vslope, CLK, and the horizontal scanning signal are schematically shown. In addition, the potential Vl of the column output line at each timing is also shown.

Prior to reading the signal from the photoelectric conversion unit 201A, the signal line PRESS of the reset switch 503 becomes Hi (t600). As a result, the gate of the SF (source follower amplifier) 505 is reset to the reset power supply voltage. At time t601, the control signal PSEL is set to Hi, and SF505 is set to the operating state. Then, the reset of the FD504 is released by setting the control signal PRESS to Lo at t602. The potential of the FD 504 at this time is read out to the column output line 507 as a reset signal level (N signal) and input to the read circuit 509.

By turning the switch 511 on and off with the control signals PSH as Hi and Lo at time t603 and t604, the N signal read out on the column output line 507 is amplified by the amplifier 510 with a desired gain and then held in the capacitance 512. Will be done. The potential of the N signal held in the capacitance 512 is input to one of the comparators 513. After the switch 511 is turned off at time t604, the reference voltage Vslope is reduced from the initial value with time by a slope voltage generation circuit (not shown) from time t605 to t607. A CLK (clock) is supplied to the counter 514 at the same time as the transition of the reference voltage Vslope is started. The value of the counter 514 increases according to the number of CLKs. Then, when the reference voltage Vslope input to the comparator 513 reaches the same level as the N signal, the output COMP of the comparator 513 becomes low level, and at the same time, the operation of the counter 514 also stops (time t606). The value when the operation of the counter 514 is stopped becomes the value obtained by AD-converting the N signal, the counter 514 and the memory 516 are connected by the switch 515, and the digital value of the N signal is held in the N signal memory 516. Will be done.

Next, at times t607 and t608 after the digitized N signal is held in the N signal memory 516, the control signals PTXA are sequentially set as Hi and Lo, and the optical charge accumulated in the photoelectric conversion unit 201A is transferred to the FD504. Then, the potential fluctuation of the FD 504 according to the amount of electric charge is read out as a signal level (light component + reset noise component (N signal)) on the column output line 507 and input to the read circuit 509. The input signal (S (A) + N) is amplified by the amplifier 510 with a desired gain, and then the control signal PSH is sequentially set to Hi and Lo at time t609 and t610, and the switch 511 is turned on and off at the timing of turning the switch 511 on and off to the capacitance 512. Be retained. The potential held in the capacitance 512 is input to one of the comparators 513. After the switch 511 is turned off at time t610, the reference voltage Vslope is reduced from the initial value with time by the slope voltage generation circuit from time t611 to t613. With the start of the transition of the reference voltage Vslope, CLK is supplied to the counter 514. The value of the counter 514 increases according to the number of CLKs. Then, when the reference voltage Vslope input to the comparator 513 reaches the same level as the S signal, the output COMP of the comparator 513 becomes low level, and at the same time, the operation of the counter 514 also stops (time t612). The value when the operation of the counter 514 is stopped is the value obtained by AD-converting the S (A) + N signal. Then, the counter 514 and the memory 517 are connected by the switch 515, and the digital value of the S (A) + N signal is held in the S signal memory 517. The differential signal level (optical component) is calculated by the CDS circuit 518 from the signals held in the memory 516 and the memory 517, and the S (A) signal from which the reset noise component is removed is obtained. The S (A) signal is sequentially sent to the signal processing unit 105 under the control of the control unit 104.

The above is the operation of reading the signal from the photoelectric conversion unit 201A of the unit pixel 200. Similarly, when reading the signal from the other photoelectric conversion unit 201B of the unit pixel 200, the signal may be driven according to the timing chart of FIG. However, in this case, at t607 and t608, PTXB is sequentially Hi and Lo instead of the control signal PTXA. That is, when the t600 to t613 in FIG. 6 are driven for the first time, the control signal PTXA is Hi and Lo to read the pixel signal S (A), and then the control signal PTXB is Hi and Lo for the second time to read the pixel signal S ( By reading B), the reading of one line is completed. By repeating this for all lines, the reading of the pixel signals S (A) and S (B) of all the pixels is completed.

FIG. 7 is a diagram showing a configuration example of the signal processing unit 105 and the output unit 106 of the image sensor 100. The signal processing unit 105 includes a captured image signal processing unit 701 and a focus detection signal processing unit 702. The pixel signal output from the reading unit 103 is input to the signal processing unit 105 via the digital signal output line 519. The input signal is processed according to the control from the control unit 104. The captured image signal processing unit 701 and the focus detection signal processing unit 702 are each provided with a memory (not shown).

The captured image signal processing unit 701 calculates a captured image signal from the read signal. That is, the photoelectric conversion unit 201A of the unit pixel 200 and the pixel signals S (A) and S (B) of the photoelectric conversion unit 201B are received and mixed, and the S (A + B) signal is calculated. Then, the pixel signal S (A + B) is sent to the output unit 106 via the captured image signal output line 703. In the captured image signal processing unit 701, the pixel signals S (A) and S (B) of the two photoelectric conversion units are mixed and processed and output from the output unit 106 to the outside of the image sensor 100, whereby the image sensor 100 The amount of signal transmission to the outside can be reduced. The calculation of the pixel signals S (A) and S (B) becomes possible when both the signals of the unit pixels are aligned. The previously read pixel signal S (A) is held in the memory, and when the pixel signal S (B) is read and input to the captured image signal processing unit 701, S (A) + S are sequentially obtained. (B) is calculated and output from the output unit 106.

The captured image signal processing unit may further mix and average the signals of the unit pixels 200. For example, in a pixel portion having a general configuration in which a color filter having a Bayer array of red (R), green (G), and blue (B) is provided, signals of adjacent pixels of the same color are mixed and averaged and output. If it is sent to unit 106, the amount of signal transmission can be further reduced. Further, instead of outputting the signals of all the pixel units to the output unit 106, the signals of only the necessary region may be output. These processes are controlled by the control unit 104.

Subsequently, the processing of the focus detection signal processing unit 702 will be described. The focus detection signal processing unit 702 calculates and outputs a focus detection signal from the read signal. As described above, the pixel signals S (A) and S (B) are required for the phase difference detection, respectively. However, if the pixel signals S (A) and S (B) of all the pixels of the pixel unit 401 are output from the output unit 106 to the outside of the image sensor 100, the amount of signal transmission becomes enormous, which is not desirable for high-speed reading.

Therefore, the focus detection signal processing unit 702 performs arithmetic processing to reduce the signal amount and output the signal from the output unit 106. For example, the pixel signals S (A) and S (B) are added as bayers to calculate the luminance value Y, and the luminance signals Y (A) and Y (B) are output. The focus detection calculation may be performed after converting the signal into a Y value, and the signal transmission amount can be reduced to 1/4 by calculating the Y value before outputting from the image sensor 100. Since the calculation of the Y value requires a signal for each bayer, the pixel signal input to the focus detection signal processing unit 702 is held in the memory until the signals required for the calculation are prepared. That is, since the signal of the RG line is read after the signal of the RG line is read, the pixel signals S (A) and S (B) of the RG line are stored in the memory. When the signals in the GB lines are read out, the luminance signals Y (A) and Y (B) are sequentially calculated and output from the output unit 106 via the signal line 704.

Further, the focus detection signal processing unit 702 may perform a correlation calculation, and the calculated value may be output from the signal line 704 to the output unit 106. The phase difference detection using the correlation calculation can be carried out by a known method. If only the correlation calculation value is output, the amount of output signal can be significantly reduced, although it depends on the number of region divisions at the time of the calculation.

As described above, in the image sensor 100 including the image sensor signal processing unit 701 and the focus detection signal processing unit 702, signal processing is performed to output only necessary signals to the outside of the image sensor 100. As a result, the amount of signal transmission can be reduced, and both the captured image data and the focus detection information can be obtained at high speed.

In the present embodiment, each of the captured image signal processing unit 701 and the focus detection signal processing unit 702 is provided with a memory. However, even if a memory is provided in front of these stages and signals required for calculations in each processing unit are prepared, signals are sent to the captured image signal processing unit 701 and the focus detection signal processing unit 702. good.

Further, in the description of the present embodiment, the luminance signals Y (A) and Y (B) are output as the focus detection signals, but only the luminance signal Y (A) may be output from the output unit 106. .. Specifically, the captured image signal processing unit 701 outputs a captured image signal, that is, a pixel signal S (A + B). Therefore, after being output to the outside of the image sensor 100, the phase difference detection unit 108 or the like calculates the luminance signal Y (A + B) from the pixel signal S (A + B), subtracts the luminance signal Y (A), and performs the luminance signal Y (A). A focus detection signal may be obtained by calculating the signal Y (B). In this way, the signal transmission amount can be further reduced by outputting only the luminance signal Y (A) from the focus detection signal processing unit 702.

For example, when the number of pixels is 20 million pixels, it is necessary to output the pixel signals S (A) and S (B) for all the pixels, that is, 40 million pieces of data when the signal processing unit is not provided. On the other hand, when the Y value is calculated and output as a focus detection signal by the image sensor provided with the signal processing unit of the present embodiment, 20 million pieces of data for the captured image and 20 million/4 = 5 million for focus detection. It can be seen that the amount of signal transmission is reduced because individual data are output. As a result, high-speed reading becomes possible. Further, when the focus detection signal is a correlation calculation value, it is clear that the signal transmission amount is further reduced.

Further, in the timing chart of FIG. 6, it is also possible to simultaneously control the control signals PTXA and PTXB at t607 and t608 to obtain the signal of the unit pixel 200 in which the charges of the photoelectric conversion unit 201A and the photoelectric conversion unit 201B are mixed. .. Specifically, after reading the signal of the photoelectric conversion unit 201A according to the timing chart of FIG. 6, the pixel signal S (A + B) is obtained by simultaneously controlling Hi and Lo of the control signals PTXA and PTXB and reading the signal. Is possible. In this case, since the reset signal is read out once, higher speed reading is possible.

When the pixel signals S (A) and S (A + B) are read from the pixels, the focus detection signal processing unit 702 performs a process of subtracting the pixel signal S (A) from the pixel signal S (A + B). For example, the pixel signal S (B) can be obtained. Alternatively, the focus detection processing unit 702 may process and output only the pixel signal S (A), and the phase difference detection unit 108 may calculate the pixel signal S (B) or the luminance signal Y (B).

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, the focus detection signal is output in the entire area of the pixel portion, but if the focus detection signal is selected and output only in the necessary area, the speed can be further increased. can do.

FIG. 8 is a diagram showing an example of a focus detection signal output region in the pixel region. The shaded area outputs the focus detection signal and the captured image signal, and the other areas output only the captured image signal. For example, as in the example shown in FIG. 8A, the focus detection signal of only the target region in a wide range of pixels is discretely (selectively) output. This makes it possible to obtain focus detection information for the entire pixel region, while suppressing the amount of signals output to the outside of the image sensor 100. Further, in the case of the example shown in FIG. 8B, detailed focus detection information can be obtained for a part of the region, and the amount of signal output to the outside of the image sensor 100 can be suppressed. .. The selection of these output areas is controlled by the control unit 104. The focus detection signal processing unit 702 reads only the signal in the output target area from the memory and performs calculation.

The output of the focus detection signal in a part of the region as shown in FIG. 8 may be a Y value signal or a correlation calculation result, but may be a pixel signal S (A). Although the amount of signal to be transmitted is larger than that of the Y value signal and the result of the correlation calculation, since only the necessary area is output, the amount of signal transmission can be suppressed. It can also be realized with a relatively small-scale signal processing circuit.

The focus detection signal processing unit 702 may also mix and average the focus detection signals. In this case, the pixel signals S (A) and the pixel signals S (B) are mixed and averaged.

As described above, in the image sensor 100 including the image sensor signal processing unit 701 and the focus detection signal processing unit 702, signal processing is performed to output only necessary signals to the outside of the image sensor 100. As a result, the transmission amount of the focus detection signal can be reduced, and both the captured image data and the focus detection information can be obtained at high speed and efficiently.

There is a method of shortening the reading time of one frame by limiting the pixels used for focus detection in this way. Normally, only the line used for the focus detection process reads out the signals of the two photoelectric conversion units in the unit pixel, and the line not used for the focus detection process mixes the signals of the two photoelectric conversion units to generate an image. By reading only, the increase in read time is suppressed. In this case, the individual output signals of the two photoelectric conversion units read out for focus detection can be mixed and used as pixel signals for captured images. However, since the signal reading method and the method of mixing the output signals of the two photoelectric conversion units differ between the line used for the focus detection process and the line not used for the focus detection process, a difference occurs in the noise level and the like, and the resulting image pickup is obtained. There is a problem that the image is deteriorated. However, by providing the focus detection signal processing unit as in the present embodiment, all the signals from the pixel unit can be read out at the same read timing, and the pixels to be output by the focus detection signal processing unit 702 can be selected. Therefore, the amount of noise of the pixel signal S (A + B) used for the captured image does not differ depending on the region, and a high-quality captured image can be obtained.

(Third Embodiment)
Next, a third embodiment of the present invention will be described. In the third embodiment, the configuration of the unit pixel 200 of the pixel unit 402 is different. FIG. 9 is a view of the light receiving unit 102 of the image sensor 100 and the microlens array observed from the optical axis direction (Z direction). Four photoelectric conversion units 901A, 901B, 901C, and 901D are arranged with respect to one microlens 202. In this way, by having a total of four photoelectric conversion units, two in each of the X-axis direction and the Y-axis direction, it is possible to receive the light in the exit pupil region divided into four. The signal reading method and the processing of the signal processing unit 105 in the image sensor 100 including the pixel unit 401 in which each of such unit pixels has four photoelectric conversion units will be described.

FIG. 10 is a schematic view showing an example of the configuration of the unit pixel 900 and the reading unit 103. In the configuration of FIG. 10, each photoelectric conversion unit is provided with a readout circuit. That is, the pixel signal of the photoelectric conversion unit 901A is read out by the reading circuit 1001A. Similarly, the photoelectric conversion unit 901B is read by the read circuit 1001B, the photoelectric conversion unit 901C is read by the read circuit 1001C, and the photoelectric conversion unit 901D is read by the read circuit 1001D. Since the signal reading operation from the photoelectric conversion unit can be performed by a method substantially the same as the driving method described with reference to FIGS. 5 and 6, the description thereof will be omitted.

The processing of the signal processing unit 105 for the signal read from each photoelectric conversion unit will be described. The captured image signal processing unit 701 calculates a captured image signal from the read signal. That is, the pixels are mixed by receiving the pixel signals S (A), S (B), S (C), and S (D) of the plurality of photoelectric conversion units 901A, 901B, 901C, and 901D of the unit pixel 900. The signal S (A + B + C + D) is calculated. Then, the pixel signal S (A + B + C + D) is sent to the output unit 106 via the captured image signal output line 703. The captured image signal processing unit 701 can reduce the amount of signal transmission to the outside of the image sensor 100 by mixing and processing the signals of a plurality of photoelectric conversion units and outputting the signals from the output unit 106 to the outside of the image sensor 100. can. The effect is even greater in an image sensor that includes four photoelectric conversion units, such as the unit pixel 900.

Subsequently, the processing of the focus detection signal processing unit 702 will be described. In the case of an image sensor having four photoelectric conversion units per unit pixel 900 as shown in FIG. 9, it is necessary to individually read the signals of the four photoelectric conversion units for focus detection, and the amount of signal transmission is enormous. Therefore, it is not desirable for high-speed reading. As described in the first embodiment, it is preferable to calculate and output the Y value, or to output only the correlation calculation result. If the signal is output only in the required area, the amount of signal transmission can be further reduced.

Further, in the case of a configuration in which the unit pixel 900 is provided with a 2x2 photoelectric conversion unit as shown in FIG. 9, it is possible to detect the phase difference in the vertical direction in addition to the horizontal direction. For example, the pixel signals S (A) and S (C) are mixed and processed, and the pixel signals S (B) and S (D) are mixed and output. In this case, from the obtained focus detection signal, it is possible to perform a phase difference type focus detection in which the pupil is divided in the left-right direction. Further, when the pixel signals S (A) and S (B) are mixed and processed, and the pixel signals S (C) and S (D) are mixed and output, the obtained focus detection signals are up and down. It is possible to detect the focus by the phase difference method in which the pupils are divided in the direction. By switching and outputting these according to the subject, it is possible to accurately detect the focus on each subject of vertical stripes and horizontal stripes. Further, the output pattern may be changed depending on the pixel area.

Further, the focus detection signal processing unit 702 may perform a correlation calculation using the obtained mixed signals (for example, S (A + C) and S (B + D)) and output only the result. If the focus detection signal processing unit 702 performs correlation calculation processing, the amount of signal transmission output from the image sensor 100 can be reduced. Further, the signal transmission amount can be suppressed even if the correlation calculation in both the left-right direction and the up-down direction is performed and output in the same region.

As described above, in the image sensor provided with the multi-divided pixels, by providing the focus detection signal processing unit 702 in the image sensor, an increase in the amount of signals output from the image sensor is suppressed, and the captured image data and the focus are performed at high speed. Detection information can be obtained. Further, since the phase difference information in the horizontal direction and the vertical direction can be acquired, the focus can be detected with high accuracy. In the present embodiment, an image pickup element having four photoelectric conversion units per unit pixel has been described as an example, but a configuration having more photoelectric conversion units may be used. In order to obtain more parallax, only the necessary PD signal may be output, or the signal may be added and output in the diagonal direction.

(Fourth Embodiment)
Since the signal processing unit 105 of the image pickup device 100 as described in the first and second embodiments is a large-scale circuit, the image pickup device 100 is likely to have a large area as a whole. Therefore, in the present embodiment, the configuration of the image pickup device 100 that suppresses the increase in the area will be described.

11 and 12 are block diagrams of the image sensor 100 according to the third embodiment. The image pickup device of this embodiment has a configuration (multilayer structure) in which a pixel region chip 1101 and a signal processing chip 1102 are laminated. The wiring between the chips is electrically connected by using a micro bump or the like by a known substrate lamination technique.

The pixel region chip 1101 includes a pixel unit 401 in which unit pixels 200 having a plurality of photoelectric conversion units are arranged in a matrix, a drive circuit unit 402, and a reading unit 103. The drive circuit unit 402 sends a drive signal to the pixels of the pixel unit 401. In FIG. 11, the unit pixel 200 has two photoelectric conversion units, but the number of photoelectric conversion units is not limited to this.

The read-out unit 103 is configured to include a large number of read-out circuits 509, for example, one read-out circuit 509 per pixel row, and reads out the pixel signal of the pixel unit 401. The read pixel signal is vertically or horizontally selected under the control of the drive circuit unit 402, and is sequentially transferred to the signal processing unit 105.

The signal processing chip 1102 includes a control unit 104, a signal processing unit 105, and an output unit 106. The signal processing unit 105 has a signal processing unit 701 for captured images and a signal processing unit 702 for focus detection, processes the pixel signal read from the reading unit 103, and transmits the pixel signal read from the reading unit 103 to the image sensor 100 via the output unit 106. Output to the outside. Since the signal processing in the signal processing unit 105 is the same as the processing described in the first and second embodiments, the description thereof will be omitted. As shown in FIG. 12, the image pickup device 100 has a configuration in which the pixel region chip 1101 and the signal processing chip 1102 are laminated and integrated.

As described above, since the image pickup elements have a laminated structure, a sufficient area can be secured in the signal processing unit 105, and a large-scale circuit can be mounted. The signal processing unit 105 can reduce the amount of signal transmission by performing signal processing that outputs only the necessary signals to the outside of the image pickup element 100, and can obtain both the captured image data and the focus detection information at high speed. Become.

(Fifth Embodiment)
In an image sensor composed of pixel units including a plurality of photoelectric conversion units per unit pixel as described in the first and second embodiments, it is preferable that the number of readout circuits 509 is large. For example, in the case of a configuration including one readout circuit per unit pixel or a configuration including one readout circuit per photoelectric conversion unit, the pixel signal can be read out at the same time for all pixels and A / D conversion can be performed. , Faster reading is possible. In this case, an area is required for arranging the readout circuit, and it is desirable that the image sensor has a laminated structure.

FIG. 13 is a diagram showing the configuration of the image sensor 100 according to the fifth embodiment. The image pickup device of this embodiment has a configuration in which a pixel region chip 1301, a readout circuit chip 1302, and a signal processing chip 1303 are laminated. The wiring between the chips is electrically connected by using a micro bump or the like by a known substrate lamination technique.

The pixel region chip 1301 includes a pixel unit 401 and a drive circuit unit 402 in which unit pixels 200 having a plurality of photoelectric conversion units are arranged in a matrix. The drive circuit unit 402 sends a drive signal to the pixels of the pixel unit 401. In FIG. 13, the unit pixel 200 includes two photoelectric conversion units, but the number of photoelectric conversion units is not limited to two.

The readout circuit chip 1302 includes a readout unit 103, a vertical selection circuit 1304, and a horizontal selection circuit 1305. The reading unit 103 has a large number of reading circuits 509 and reads the pixel signal of the pixel unit 401. The read pixel signal is sequentially transferred to the signal processing unit 105 under the control of the vertical selection circuit 1304 and the horizontal selection circuit 1305.

The signal processing chip 1303 includes a control unit 104, a signal processing unit 105, and an output unit 106. The signal processing unit 105 has a signal processing unit 701 for captured images and a signal processing unit 702 for focus detection, and the pixel signal read by the reading unit 103 is processed by the signal processing unit 105 and passed through the output unit 106. Is output to the outside of the image pickup element 100. Since the signal processing in the signal processing unit 105 is the same as the processing described in the first and second embodiments, the description thereof will be omitted.

The image pickup device 100 has a configuration in which a pixel region chip 1301, a readout circuit chip 1302, and a signal processing chip 1303 are laminated and integrated. By the way, in the case of a configuration including, for example, one readout circuit for each photoelectric conversion unit as in the configuration of the present embodiment, the time for the pixel signal to be read out by the signal processing unit 105 is dramatically increased. For example, assuming that the time required for reading one photoelectric conversion unit is α, in the case of an image sensor provided with one reading circuit per column, it takes time for αx rows to read the pixel signal for one frame. On the other hand, when one readout circuit is provided for each photoelectric conversion unit, one frame of pixel signals is read out in α time. However, in this case, there is a concern that the signal processing unit 105 that receives the pixel signal receives the signal and the signal processing is rate-determining. However, as in the present embodiment, since the signal processing unit 105 of the image pickup device 100 having a laminated structure can have a large area, it is possible to provide a large number of transmission lines for signals from the readout circuit chip 1302, and the speed is high. The pixel signal can be sent to the signal processing unit 105. Further, since a large number of signal processing circuits can be mounted on the signal processing unit 105, parallel processing becomes possible and the signal processing time is shortened.

As described above, since the image sensor has a laminated structure, a sufficient area can be secured for the reading unit and the signal processing unit. The image pickup device of the present embodiment also speeds up signal reading from the pixel section, and further reduces the amount of signal transmission by performing signal processing in the signal processing section 105 to output only necessary signals to the outside of the image pickup device. Therefore, both the captured image data and the focus detection information can be obtained at high speed.

(Other embodiments)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100: Image sensor, 101: Photograph lens, 102: Light receiving unit, 103: Read unit, 104: Control unit, 105: Signal processing unit, 106: Output unit, 107: Image processing unit, 108: Phase difference detection unit, 109 : Overall control / calculation unit

Claims (34)

A pixel unit in which a plurality of unit pixels, each having one microlens and a plurality of photoelectric conversion units, are arranged in a matrix, and a pixel unit.
A signal holding unit that holds a signal output from a unit pixel of the entire pixel unit, and a signal holding unit.
A signal processing unit that processes a signal held by the signal holding unit, and a signal processing unit for an image capture image that performs signal processing for generating an image captured from the signal held by the signal holding unit, and the signal. A signal processing unit having a correlation calculation unit that performs a correlation calculation between signals of a plurality of photoelectric conversion units of each of the unit pixels held by the holding unit, and a signal processing unit.
An output unit that outputs the signal processed by the signal processing unit, and an output unit.
An image pickup device characterized by comprising. The correlation calculation unit selects a signal of the unit pixel in a target region in the pixel unit from the signal of the entire pixel unit held by the signal holding unit, and a plurality of signals of each of the selected unit pixels. performs correlation operation between the photoelectric conversion portion of the signal, the imaging device according to claim 1, wherein the outputting the calculated value by the correlation calculation. The image pickup device according to claim 2 , wherein the correlation calculation unit selectively mixes signals from a plurality of photoelectric conversion units for each of the selected unit pixels. The image pickup device according to any one of claims 1 to 3 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel. .. Any one of claims 1 to 3 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel for each column. The image pickup device according to. The method according to any one of claims 1 to 5 , wherein the image sensor signal processing unit includes a compression processing means for compressing and outputting the signals of the plurality of photoelectric conversion units of each unit pixel. Image sensor. The image pickup device according to claim 6 , wherein the compression processing means mixes and averages the signals of the plurality of photoelectric conversion units of each unit pixel. The image pickup device according to any one of claims 1 to 7 , further comprising a multilayer structure formed of a plurality of semiconductor substrates, wherein the pixel portion and the signal processing portion are formed on different substrates. .. It includes an image pickup element, an image processing means for processing a signal read from the image pickup element to generate an image, and a focus detection means for processing the read signal to perform focus detection.
The image sensor
A pixel unit in which a plurality of unit pixels, each having one microlens and a plurality of photoelectric conversion units, are arranged in a matrix, and a pixel unit.
A signal holding unit that holds a signal output from a unit pixel of the entire pixel unit, and a signal holding unit.
A signal processing unit that processes a signal held by the signal holding unit, and a signal processing unit for an image capture image that performs signal processing for generating an image captured from the signal held by the signal holding unit, and the signal. A signal processing unit having a correlation calculation unit that performs a correlation calculation between signals of a plurality of photoelectric conversion units of each of the unit pixels held by the holding unit, and a signal processing unit.
An output unit that outputs the signal processed by the signal processing unit, and an output unit.
An imaging device characterized by having. The correlation calculation unit selects a signal of the unit pixel in a target region in the pixel unit from the signal of the entire pixel unit held by the signal holding unit, and a plurality of each of the selected unit pixels. The imaging device according to claim 9 , wherein a correlation calculation is performed between the signals of the photoelectric conversion unit, and a value calculated by the correlation calculation is output. The imaging device according to claim 10 , wherein the correlation calculation unit selectively mixes signals of a plurality of photoelectric conversion units for each of the selected unit pixels. The imaging apparatus according to any one of claims 9 to 11 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel. .. Any one of claims 9 to 11 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel for each column. The imaging apparatus according to. The method according to any one of claims 9 to 13 , wherein the captured image signal processing unit has a compression processing means for compressing and outputting the signals of the plurality of photoelectric conversion units of each unit pixel. Imaging device. The imaging device according to claim 14 , wherein the compression processing means mixes and averages the signals of the plurality of photoelectric conversion units of each unit pixel. Any one of claims 9 to 15 , wherein the image pickup device has a multilayer structure formed of a plurality of semiconductor substrates, and the pixel portion and the signal processing portion are formed on different substrates. The imaging device according to. A pixel part in which a plurality of unit pixels are arranged in a matrix and
A signal holding unit that holds a signal output from a unit pixel of the entire pixel unit, and a signal holding unit.
A signal processing unit that processes a signal held by the signal holding unit, and a signal processing unit for an image capture image that performs signal processing for generating an image captured from the signal held by the signal holding unit, and the signal. A signal processing unit having a phase difference information generation unit that generates phase difference information using the signal held by the holding unit, and a signal processing unit.
An output unit that outputs the signal processed by the signal processing unit, and an output unit.
An image pickup device characterized by comprising. The image pickup device according to claim 17 , wherein each of the unit pixels has one microlens and a plurality of photoelectric conversion units. The phase difference information generation unit selects a signal of the unit pixel in a target region in the pixel unit from the signal of the entire pixel unit held by the signal holding unit, and each of the selected unit pixels. The image pickup device according to claim 17 or 18 , wherein phase difference information is generated using signals of a plurality of photoelectric conversion units, and the generated phase difference information is output. The image pickup device according to claim 19 , wherein the phase difference information generation unit selectively mixes signals of a plurality of photoelectric conversion units for each of the selected unit pixels. The image pickup device according to any one of claims 18 to 20 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel. .. Any one of claims 18 to 20 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel for each column. The image pickup device according to. The method according to any one of claims 18 to 22 , wherein the image sensor signal processing unit has a compression processing means for compressing and outputting the signals of the plurality of photoelectric conversion units of each unit pixel. Image sensor. The image pickup device according to claim 23 , wherein the compression processing means mixes and averages the signals of the plurality of photoelectric conversion units of each unit pixel. The image pickup device according to any one of claims 17 to 24 , which has a multilayer structure formed of a plurality of semiconductor substrates, and the pixel portion and the signal processing portion are formed on different substrates. .. It includes an image pickup element, an image processing means for processing a signal read from the image pickup element to generate an image, and a focus detection means for processing the read signal to perform focus detection.
The image sensor
A pixel part in which a plurality of unit pixels are arranged in a matrix and
A signal holding unit that holds a signal output from a unit pixel of the entire pixel unit, and a signal holding unit.
A signal processing unit that processes a signal held by the signal holding unit, and a signal processing unit for an image capture image that performs signal processing for generating an image captured from the signal held by the signal holding unit, and the signal. A signal processing unit having a phase difference information generation unit that generates phase difference information using the signal held by the holding unit, and a signal processing unit.
An output unit that outputs the signal processed by the signal processing unit, and an output unit.
An imaging device characterized by having. The imaging device according to claim 26 , wherein each of the unit pixels has one microlens and a plurality of photoelectric conversion units. The phase difference information generation unit selects the signal of the unit pixel in the target region in the pixel unit from the signal of the entire pixel unit held by the signal holding unit, and each of the selected unit pixels. The imaging apparatus according to claim 26 or 27 , wherein phase difference information is generated using signals of a plurality of photoelectric conversion units, and the generated phase difference information is output. 28. The imaging apparatus according to claim 28 , wherein the phase difference information generation unit selectively mixes signals from a plurality of photoelectric conversion units for each of the selected unit pixels. The imaging apparatus according to any one of claims 27 to 29 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel. .. Any one of claims 27 to 29 , wherein the signal holding unit has a plurality of holding circuits for holding signals output from the photoelectric conversion unit of each unit pixel for each column. The imaging apparatus according to. The method according to any one of claims 27 to 31 , wherein the captured image signal processing unit has a compression processing means for compressing and outputting the signals of the plurality of photoelectric conversion units of each unit pixel. Imaging device. The imaging device according to claim 32 , wherein the compression processing means mixes and averages the signals of the plurality of photoelectric conversion units of each unit pixel. Any one of claims 26 to 33 , wherein the image pickup device has a multilayer structure formed of a plurality of semiconductor substrates, and the pixel portion and the signal processing portion are formed on different substrates. The imaging device according to.

Images