JP6851744B2 - Signal processing equipment, signal processing methods, programs and storage media - Google Patents

Description

The present invention relates to a technique for interpolating a signal read from a defective photoelectric conversion unit of an image pickup device having a plurality of pixels having a plurality of photoelectric conversion units.

An image sensor having a plurality of pixels having a plurality of photoelectric conversion units for one microlens is known. In each pixel of the image pickup device, a plurality of out read part of the photoelectric conversion unit and Racing No. Toka other photoelectric conversion unit of the photoelectric conversion unit, by obtaining the phase difference between the signals out read, Distance information to the subject can be obtained. If there is a defect in some of the photoelectric conversion units of the plurality of photoelectric conversion units, or if there is a defect in the circuit connected to some of the photoelectric conversion units, the signal read from the photoelectric conversion unit is read. Cannot be used for phase difference detection as it is.

Therefore, in Patent Document 1, in an image sensor provided with a plurality of pixels having a plurality of photoelectric conversion units for one microlens, a photoelectric conversion unit of peripheral pixels is used for a signal of an abnormal photoelectric conversion unit. A method of interpolating using a signal is disclosed.

JP-A-2015-56710

However, in the interpolation method described in Patent Document 1, the signal of the photoelectric conversion unit included in the pixel corresponding to the same color filter as the pixel including the photoelectric conversion unit that outputs an abnormal signal (hereinafter referred to as defective pixel). Therefore, the degree of freedom of pixels that can be used for interpolation is low. For example, in the case of an image sensor configured in a Bayer array, only every other pixel corresponding to the same color filter exists in the horizontal and vertical directions. Therefore, in the interpolation method described in Patent Document 1, the signal read from the photoelectric conversion unit of the pixels adjacent in the horizontal and vertical directions cannot be used when performing the interpolation.

When the signal used for interpolation is a signal read from a spatially close position, the reproducibility of the high frequency component at the time of interpolation is improved. Therefore, although it depends on the pattern of the subject, in order to further improve the reproducibility of the high frequency component during interpolation, the position of the pixel including the photoelectric conversion unit that refers to the signal during interpolation is the photoelectric conversion of the interpolation target. It is desirable that the position is not limited to the position of the pixel having the same color as the pixel including the part.

Further, it is conceivable that the signal of the photoelectric conversion unit itself that reads out the signal used for interpolation may not be normal or may be saturated. Alternatively, depending on the pattern of the subject, even a photoelectric conversion unit included in nearby pixels may output a signal related to a subject different from the photoelectric conversion unit to be interpolated.

Therefore, in order to further improve the accuracy of the signal obtained by interpolation, the degree of freedom of the position of the pixel including the photoelectric conversion unit that refers to the signal at the time of interpolation is improved, and the photoelectric conversion unit that refers to the signal depending on the situation is improved. It is desirable to give an appropriate priority to the signal read from.
An object of the present invention is to provide a device capable of interpolating a signal with high accuracy for a photoelectric conversion unit that outputs an abnormal signal.

The present invention is configured to have a plurality of pixels, and each pixel has one microlens, and at least a first photoelectric conversion region and a second photoelectric conversion region that receive light through the microlenses. In a signal processing device that processes a signal obtained from the image pickup element, when there is an abnormality in the signal from the first photoelectric conversion region in the first pixel among the plurality of pixels, the above-mentioned It has an interpolation means for performing a first interpolation for interpolating a signal in a first photoelectric conversion region having an abnormality, and the interpolation means with respect to a signal from a second photoelectric conversion region in the first pixel. It is configured to interpolate the signal from the abnormal first photoelectric conversion region by multiplying it by a predetermined coefficient, and the coefficient is within at least one peripheral pixel around the first pixel. factor der determined based on the ratio of the second first signal of the photoelectric conversion region in the peripheral pixel signal and the at least one photoelectric conversion region is, the interpolation unit, when a predetermined condition is satisfied, the first When a pixel adjacent to one pixel and having a color different from that of the first pixel is selected as the peripheral pixel and the predetermined condition is not satisfied, the pixel is not adjacent to the first pixel and is described as described above. Provided is a signal processing apparatus characterized in that a pixel having the same color as the first pixel is selected as the peripheral pixel.

According to the present invention, in a signal processing device that processes a signal from an image pickup device having a plurality of pixels having a plurality of photoelectric conversion units for one microlens, the photoelectric conversion unit that outputs an abnormal signal it is possible to interpolate a signal with high Te accuracy.

It is a block diagram which shows the structure of the image pickup apparatus in embodiment of this invention. It is a figure for demonstrating the structure of the image pickup device in embodiment of this invention. It is an equivalent circuit diagram which shows a part of the pixel array in embodiment of this invention. It is a figure which shows the timing of reading a signal in embodiment of this invention. It is a figure which shows the signal interpolation of the pixel which has the color filter of the hue of two PD and G in the embodiment of this invention. It is a figure which shows the signal interpolation of the pixel which has two PD and R hue color filters in embodiment of this invention. It is a figure which shows the signal interpolation of the pixel which has 4 PD and G hue color filter in embodiment of this invention. It is a flowchart for demonstrating embodiment of this invention.

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. The following embodiments are merely examples for carrying out the present invention, and the present invention is not limited to the illustrated configuration.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a block diagram showing a configuration of an image pickup apparatus as an example of the signal processing apparatus according to the embodiment of the present invention. In the figure, the image sensor 100 photoelectrically converts an optical image of a subject formed by the image pickup optical system into an electric signal. The image sensor 100 is controlled by a CPU (Central Processing Unit) 103 or the like, which will be described later, and captures a still image or a moving image. Further, as will be described later, the image pickup device is covered with a color filter composed of a plurality of colors. The AFE (analog front end) 101 digitally converts the analog image signal output from the image sensor 100 in response to gain adjustment and a predetermined quantization bit. The TG (timing generator) 102 controls the drive timing of the image sensor 100 and the AFE 101. In the present embodiment, the AFE 101 and the TG 102 are arranged outside the image sensor 100, but they may be built in the image sensor.

As described above, the CPU 103 executes a computer program for controlling each part of the image sensor. The operation unit 104 sets the image pickup command, the image pickup condition, and the like on the CPU 103. The display unit 105 is a display that displays captured still images, moving images, menus, and the like. The RAM 106 has a function of an image data storage means for storing image data digitally converted by the AFE 101 and image data processed by the image processing unit 108 described later, and a function of a work memory when the CPU 103 operates. .. In the present embodiment, these functions are performed by using the RAM 106, but other memories may be applied as long as the access speed is sufficiently high and there is no problem in operation. It is possible. The ROM 107 functions as a storage medium for storing a computer program loaded and executed by the CPU 103 to control the operation of each unit. Also, information such as the saturation level of the photoelectric conversion unit position and the signal of defect definitive to the image sensor 100, from the factory, is stored in ROM 107. Here, the photoelectric conversion unit defective referred not only that there is a defect in the configuration itself of the photoelectric conversion unit, including those where there is a defect in the circuit connected to the photoelectric conversion unit. That is, resulting in a photoelectric conversion unit which signal output is considered not normal, and that the photoelectric conversion unit defective. Images processing unit 108 performs processing of correction and compression such as a captured still image or a moving image. It also has a function of separating A image data and B image data, which will be described later, an image correction function, and a function of generating still images and moving images.

The AF calculation unit 109 calculates the driving amount of the focus lens by using the result of the correlation calculation output from the correlation calculation unit 120. The flash memory 110 is a removable flash memory for recording still image data and moving image data. In the present embodiment, the flash memory is applied as the recording medium, but other non-volatile memory in which data can be written may be used. Further, it may be in the form of incorporating these memories.

The focal plane shutter 111 adjusts the exposure seconds when capturing a still image. In the present embodiment, the focal plane shutter 111 adjusts the exposure seconds of the image sensor 100, but the present embodiment is not limited to this, and the image sensor 100 has an electronic shutter function and the exposure seconds are controlled by a control pulse. It may be configured to adjust the time. The focus drive circuit 112 changes the focus position of the optical system, controls the drive of the focus actuator 114 based on the focus detection result of the AF calculation unit 109, and sets the third lens 119 as the focus lens in the optical axis direction. The focus is adjusted by driving forward and backward. The diaphragm drive circuit 113 controls the drive of the diaphragm actuator 115 to control the aperture diameter of the diaphragm 117. The first lens 116 is arranged at the tip of the imaging optical system (common optical system) and is held so as to be able to move forward and backward in the optical axis direction. The aperture 117 adjusts the amount of light at the time of imaging by adjusting the aperture diameter thereof. Then, the aperture 117 and the second lens 118 as a zoom lens move forward and backward in the optical axis direction as a unit, and a scaling action (zoom function) is realized by interlocking with the advance / backward operation of the first lens 116. The third lens 119 adjusts the focus of the imaging optical system by advancing and retreating in the optical axis direction. The correlation calculation unit 120 performs the correlation calculation using the pixel signal output from the image sensor 100.

FIG. 2 is a diagram for explaining the configuration of the image pickup device 100. FIG. 2A shows the configuration of the image sensor 100. In FIG. 2A, the image pickup element includes a pixel array 100a in which pixels are arranged two-dimensionally, a vertical scanning circuit 100d that selects a row of pixels in the pixel array 100a, and a horizontal that selects a row of pixels in the pixel array 100a. It has a scanning circuit 100c. Further, the image sensor 100 further includes a reading circuit 100b for reading the signal of the pixel selected by the vertical scanning circuit 100d and the horizontal scanning circuit 100c among the pixels of the pixel array 100a. The vertical scanning circuit 100d selects a row of the pixel array 100a,-out based on the horizontal synchronizing signal outputted from the CPU 103, a read pulse output from the TG 102, to enable the selected row. The read-out circuit 100b has an amplifier and a memory provided for each column, and stores the pixel signal of the scanning row in the memory via the amplifier. The pixel signals for one row stored in the memory are sequentially selected in the column direction by the horizontal scanning circuit 100c, and are output to the outside via the output circuit 100e. This operation is repeated to output the signals of all pixels to the outside.

The pixel array 100a of the image sensor 100 is shown in FIG. 2 (b). In FIG. 2B, the microlens array includes h microlenses 100f arranged in the horizontal direction and v microlenses arranged in the vertical direction. PDs (photodiodes) 100h and 100g are photoelectric conversion units that perform photoelectric conversion , and constitute a photoelectric conversion unit for A image and a photoelectric conversion unit for B image, which will be described later. Each pixel has a configuration in which one microlens 100f is arranged at the top for two PDs. That is, each pixel includes a plurality of photoelectric conversion units for one microlens. When one pixel is composed of two PDs sharing the microlens 100f, the pixel array 100a includes h pixels in the horizontal direction and v pixels in the vertical direction. The signals accumulated in the PD 100h and the PD 100g are simultaneously or independently converted into voltages by the pixel transfer operation described later, and output to the outside by the reading operation described above. The PD100h and PD100g have a pupil-divided configuration in which light from the pupil divided via a microlens is incident, and separate images having phase differences with each other are incident. Therefore, out read signals PD100h and PD100g, performs a correlation calculation processing in the correlation calculation unit 120 of the above, it is possible to calculate the driving amount of the focus lens by the AF calculation unit 109 described above with reference to the results. Here, PD100h is referred to as an A image photoelectric conversion unit, and PD100g is referred to as a B image photoelectric conversion unit. In FIG. 2B, two PDs are arranged for one microlens, but the present invention is not limited to this. The present invention can be applied even if a plurality of PDs are arranged vertically or horizontally with respect to one microlens, or the pupil can be divided into four or more. For example, FIG. 2C shows an example in which four PDs are arranged. Pixel 100i has PD100j, PD100k, PD100m and PD100n. Images of different phases from different pupils are incident on the PD100j, PD100k, PD100m and PD100n, and signals can be read out separately. Alternatively, depending on the situation, the signals of PD100j and PD100k are added and read, and the signals of PD100m and PD100n are added and read, so that the phase difference in the horizontal direction can be acquired through the calculation by the correlation calculation unit 120. .. Alternatively, the signals of PD100j and PD100m are added and read, and the signals of PD100k and PD100n are added and read, so that the phase difference in the vertical direction can be acquired through the calculation by the correlation calculation unit 120. In this way, the signal obtained from the PD can be used for phase difference detection. FIG. 2D is a diagram showing color filters arranged for each microlens. One square in FIG. 2D means a color filter corresponding to one microlens. The letters "R", "B" and "G" written on the microlens represent the hue of the color filter, and correspond to "red", "blue" and "green", respectively. In the present embodiment, as shown in FIG. 2D, the description will be based on the color filter of the Bayer arrangement, but the present embodiment is not limited to this, and for example, four hues of yellow, magenta, cyan, and green are reduced to 2 × 2. It can also be carried out by using an arranged color filter or the like.

In the following, an embodiment will be described using a configuration in which two PDs are arranged for one microlens. FIG. 3 shows pixels for two adjacent rows (j rows and (j + 1) rows) and two columns (i columns and (i + 1) columns) and two columns (of the plurality of pixels provided in the pixel array 100a). It is an equivalent circuit diagram which shows the structure of the read circuit 100b for i column and (i + 1) column).

The control signal ΦTXA (j) is input to the transfer switch 302a of the pixel 301 on the jth line, and the control signal ΦTXB (j) is input to the gate of the transfer switch 302b. The reset switch 304 is controlled by the reset signal ΦR (j), and the row selection switch 306 is controlled by the row selection signal ΦS (j). Similarly, the pixels on the (j + 1) th row are controlled by the control signals ΦTXA (j + 1) and ΦTXB (j + 1), the reset signal ΦR (j + 1), and the row selection signal ΦS (j + 1). The control signals ΦTXA (j) and ΦTXB (j), the reset signal ΦR (j), and the row selection signal ΦS (j) are controlled by the vertical scanning circuit 100d.

Further, vertical signal lines 308 are provided for each pixel row, and each vertical signal line 308 is connected to the current source 307 and transfer switches 310a and 310b of the read circuit 100b provided in each row.

The control signal ΦTN is input to the gate of the transfer switch 310a, and the control signal ΦTS is input to the gate of the transfer switch 310b. Further, the control signal ΦPH (i) output from the horizontal scanning circuit 100c is input to the gates of the transfer switch 312a and the transfer switch 312b. The storage capacity unit 311a stores the output of the vertical signal line 308 when the transfer switch 310a is on and the transfer switch 312a is off. Similarly, the storage capacity unit 311b stores the output of the vertical signal line 308 when the transfer switch 310b is on and the transfer switch 312b is off.

By turning on the transfer switch 312a and the transfer switch 312b in the i-th row by the row selection signal ΦPH (i) of the horizontal scanning circuit 100c, the outputs of the storage capacity unit 311a and the storage capacity unit 311b are separate horizontal outputs. It is transferred to the output circuit 100e via the line.

As a read operation for reading a signal from the image sensor 100 having the above configuration, it is possible to selectively perform an additive read operation (first read operation) and a split read operation (second read operation). Hereinafter, the additive read operation and the split read operation will be described. In the present embodiment, it is assumed that each switch is turned on when each control signal is in the H (high) state and turned off when each control signal is in the L (low) state.

<Additive read operation> (first read operation)
FIG. 4A shows the timing of the operation of reading a signal from the pixel of the j-th row of the image sensor 100 by the addition / reading operation. At time T1, the reset signal ΦR (j) becomes H. Next, when the control signals ΦTXA (j) and ΦTXB (j) become H at time T2, PD100h and 100g of the pixel 100f on the jth row are reset.

Next, when the control signals ΦTXA (j) and ΦTXB (j) become L at time T3, PD100h and 100g start charge accumulation. Subsequently, when the row selection signal ΦS (j) becomes H at time T4, the row selection switch 306 is turned on and connected to the vertical signal line 308, and the source follower amplifier 305 is in the operating state.

Next, when the reset signal ΦR (j) is set to L at time T5 and then the control signal ΦTN becomes H at time T6, the transfer switch 310a is turned on and the signal after reset release on the vertical signal line 308 ( The noise signal) is transferred to the storage capacitance unit 311a.

Next, the control signal ΦTN is set to L at time T7, and the noise signal is held in the storage capacitance unit 311a. After that, when the control signals ΦTXA (j) and ΦTXB (j) become H at time T8, the electric charge of PD100h and 100g is transferred to the FD region (floating diffusion region) 303. At this time, since the charges of the two PDs 100h and 100 g are transferred to the same FD region 303, the signal (optical signal for one pixel + noise signal) in which the charges of the two PDs 100h and 100 g are mixed is transferred to the vertical signal line 308. It is output.

Subsequently, at time T9, the control signals ΦTXA (j) and ΦTXB (j) are set to L. After that, when the control signal ΦTS becomes H at time T10, the transfer switch 310b is turned on, and the signal (optical signal for one pixel + noise signal for one pixel) on the vertical signal line 308 is transferred to the storage capacitance unit 311b. Next, the control signal ΦTS is set to L at time T11, the optical signal + noise signal for one pixel is held in the storage capacitance unit 311b, and then the row selection signal ΦS (j) is set to L at time T12.

After that, the transfer switches 312a and 312b are sequentially set to H by the row selection signal ΦPH of the horizontal scanning circuit 100c from the first pixel row to the last pixel row. As a result, the noise signal of the storage capacitance unit 311a and the optical signal + noise signal for one pixel of 311b are transferred to the output circuit 100e via different horizontal output lines. The output circuit 100e calculates the difference between these two horizontal output lines (optical signal for one pixel), and outputs a signal obtained by multiplying this by a predetermined gain. Hereinafter, the signal obtained by the above-mentioned addition / reading operation is referred to as a “first addition signal”.

<Split read operation> (second read operation)
Next, the split read operation will be described with reference to FIG. 4 (b). FIG. 4B shows the timing of the operation of reading a signal from the pixel on the jth row of the image sensor 100 by the split reading operation. At time T1, the reset signal ΦR (j) is set to H. Subsequently, when ΦTXA (j) and ΦTXB (j) become H at time T2, PD100h and 100g of the pixel 301 on the jth row are reset. Next, when the control signals ΦTXA (j) and ΦTXB (j) become L at time T3, PD100h and 100g start charge accumulation. Subsequently, when the row selection signal ΦS (j) becomes H at time T4, the row selection switch 306 is turned on and connected to the vertical signal line 308, and the source follower amplifier 305 is in the operating state.

When the reset signal ΦR (j) is set to L at time T5 and then the control signal ΦTN becomes H at time T6, the transfer switch 310a is turned on and the signal (noise signal) after the reset is released on the vertical signal line 308. Is transferred to the storage capacity unit 311a.

Next, when the control signal ΦTN is set to L at time T7, the noise signal is held in the storage capacitance unit 311a, and then ΦTXA (j) becomes H at time T8, the charge of PD100h is transferred to the FD region 303. .. At this time, since the charge of one of the two PD100h and 100g (PD100h in this case) is transferred to the FD region 303, only the signal corresponding to the charge of the PD100h is output to the vertical signal line 308.

Next, when the control signal ΦTXA (j) is set to L at time T9 and then the control signal ΦTS becomes H at time T10, the transfer switch 310b is turned on and the signal on the vertical signal line 308 (for 1PD). (Optical signal + noise signal) is transferred to the storage capacitance unit 311b. Next, at time T11, the control signal ΦTS is set to L.

After that, the transfer switches 312a and 312b are sequentially set to H by the row selection signal ΦPH of the horizontal scanning circuit 100c from the first pixel row to the last pixel row. As a result, the noise signal of the storage capacitance section 311a and the optical signal + noise signal for 1PD of 311b are transferred to the output circuit 100e by separate horizontal output lines. The output circuit 100e calculates the difference between these two horizontal output lines (optical signal for 1 PD), and outputs a signal obtained by multiplying this by a predetermined gain. Hereinafter, the signal obtained by the above-mentioned read operation is referred to as a “divided signal”.

After that, at time T12, ΦTXA (j) and ΦTXB (j) become H, and in addition to the previously transferred PD100h charge, the PD100g charge and the newly generated PD100h charge are further transferred to the FD region 303. .. At this time, since the charges of the two PDs 100h and 100 g are transferred to the same FD region 303, the signal (optical signal for one pixel + noise signal) in which the charges of the two PDs 100h and 100 g are mixed is transferred to the vertical signal line 308. It is output.

Subsequently, after the control signals ΦTXA (j) and ΦTXB (j) are set to L at time T13, when the control signal ΦTS becomes H at time T14, the transfer switch 310b is turned on. As a result, the signal (optical signal for one pixel + noise signal) on the vertical signal line 308 is transferred to the storage capacitance unit 311b.

Next, the control signal ΦTS is set to L at time T15, the optical signal + noise signal for one pixel is held in the storage capacitance unit 311b, and then the row selection signal ΦS (j) is set to L at time T16.

After that, the transfer switches 312a and 312b are sequentially set to H by the row selection signal ΦPH of the horizontal scanning circuit 100c from the first pixel row to the last pixel row. As a result, the noise signal of the storage capacitance units 311a and 311b and the optical signal + noise signal for one pixel are transferred to the output circuit 100e by different horizontal output lines. The output circuit 100e calculates the difference between these two horizontal output lines (optical signal for one pixel), and outputs a signal obtained by multiplying this by a predetermined gain. Hereinafter, the signal obtained by the read operation is referred to as a "second addition signal" in order to distinguish it from the first addition signal.

By subtracting the division signal corresponding to one PD100h from the second addition signal read in this way, the division signal corresponding to the other PD100g can be obtained. The pair of divided signals obtained in this way is called a "focus detection signal". Then, the phase difference between the signals can be calculated by performing a known correlation calculation on the obtained focus detection signal.

The above-mentioned divided read operation and additive read operation can be switched as necessary even for the same pixel. However, in the present embodiment, it is assumed that the division reading operation is performed between the pixel to be interpolated and the peripheral pixels described later.

After performing a series of operations of resetting, charge accumulation, and signal reading on the PD100h, the same operation is performed on the PD100g to perform two PD100h operations for one charge accumulation operation. , 100 g of signal may be read out independently. The PD100h, 100g signals read out in two steps in this way can be added to obtain a second addition signal. Further, as described above, the configuration is not limited to the configuration in which two PDs are arranged for one microlens, and even in the configuration having three or more PDs, the same idea is applied multiple times. It is possible to read the signal separately.

<Interpolation method of the signal read from the defective photoelectric conversion unit>
Next, a method of interpolating the signal read from the defective photoelectric conversion unit in the present embodiment will be described. FIG. 5 is a diagram showing a case where only one PD of one pixel has a defect in a configuration in which two PDs are arranged for one microlens. Here, since the PD 501 arranged to the left of the pixel 500 has a defect and a correct signal cannot be acquired, the signal of the PD 501 is the target of interpolation. On the other hand, it is assumed that the PD 502 arranged to the right of the same pixel 500 has no defect. The characters written on the upper part of each pixel represent the hue of the color filter.

The image processing unit 108 can interpolate the signal of PD501 by using the ratio of the weighted average result of the signals of the two photoelectric conversion units of each of the peripheral pixels by the following (Equation 1).

In the above (Equation 1), S501 means the signal value of PD501 obtained by interpolation, and w1, w2, w3 and w4 represent the weighting coefficient. The image processing unit 108 is a signal of two PD signal values S511 to S542 included in each of the peripheral pixels 510 to 540 arranged on the top, bottom, left, and right of the pixel 500 having the PD 501, and the other PD 502 included in the pixel 500. Interpolate using the value S502. Specifically, the signal values S511, S521, S531 and S541 of PD511, PD521, PD531 and PD541 on the left side of each of the peripheral pixels 510 to 540, which are the same as PD501, are multiplied by a weighting coefficient to obtain a value added. Further, the signal values S512, S522, S532 and S542 of PD512, PD522, PD532 and PD542 on the right side opposite to PD501 in each of the peripheral pixels 510 to 540 are multiplied by a weighting coefficient to obtain a value added. .. Then, the ratio of the value obtained by weighted averaging using the signal value of PD on the left side and the value obtained by weighted averaging using the signal value of PD on the right side is calculated. This ratio corresponds to the average ratio of the PD on the left side and the PD on the right side in the region including a plurality of pixels centered on the pixel 500. Then, the value obtained by multiplying the calculated ratio by the signal value of the PD502 on the right side of the pixel 500, which is considered to output a normal signal, is used as the interpolated value of the signal of the PD501 on the left side of the pixel 500.

Here, the weighting coefficients w1 to w4 have a role of improving the accuracy of interpolation by changing the values thereof according to the state of the pixels around the pixel 500 and the subject. For example, when a peripheral pixel contains a defective pixel, if the above ratio is obtained using the PD signal value obtained from this pixel, the value deviates from the average ratio of the peripheral pixels. May become. Therefore, the image processing unit 108 sets the weighting coefficient given to the signal value obtained from the peripheral pixels determined to have a defect to 0 based on the defect information stored in the ROM 107. For example, when at least one of PD511 and PD512 included in the pixel 500 is defective, the weighting coefficient w1 with respect to the signal values S511 and S512 obtained from PD511 and PD512 is set to 0.

Alternatively, when the signal value read from any of the PDs has reached the saturation level stored in advance in the ROM 107, the weighting coefficient corresponding to the signal value obtained from the pixels including the PD is set to 0. .. If the signal value read from at least one PD is saturated, the signal value read from that PD is smaller than the signal value corresponding to the original subject and is not a correct value, so the accuracy. This is because it cannot be used to calculate the high ratio of.

As described above, according to the above-mentioned method, first, the ratio of the signal values between a plurality of PDs corresponding to the same microlens is obtained by using the peripheral pixels of the pixel to be processed. Then, by multiplying this by a signal value obtained from another PD corresponding to the same microlens as the defective PD, the signal value of the defective PD is interpolated. Since the ratio of the signal values between PDs corresponding to the same microlens is due to the difference in the area on the subject incident on the PD, when obtaining this ratio, the hue is different from that of the pixel to be processed. Corresponding pixels can be used. Therefore, as shown in (Equation 1), it is possible to obtain the ratio of the signal values between the PDs by using the pixels adjacent to the pixels to be processed and corresponding to the hue different from that of the processing target.

Of course, as shown in (Equation 2) below, the image processing unit 108 was obtained by weighted averaging the signal values obtained from two PDs included in the pixels of the same color as the pixel to be processed among the peripheral pixels. The ratio of values can also be used to interpolate the signal values of defective PDs.

In the above (Equation 2), when the pixel 500 to be interpolated corresponds to the hue G, the interpolation is performed using the signal of the pixel having the same hue G color filter as the pixel 500. Since the pixels corresponding to hue G have a higher density than the pixels corresponding to hues R and B, even when the same color pixels are used, the distance between the pixels to be interpolated and the peripheral pixels is short, and the accuracy is high. High ratio can be obtained.

Next, FIG. 6 shows that the PD601 on the left of the pixel having the R hue color filter is defective. The image processing unit 108 can interpolate the signal of PD601 using the following (Equation 3) or (Equation 4).

In the above (Equation 3), S601 means the signal of PD601 obtained by interpolation, and w1 to w4 represent the weighting coefficient. For each of the peripheral pixels 610 to 640 arranged on the top, bottom, left, and right of the pixel 600 having the PD601, the signal values S611 to S644 of the PD611 to 641 on the left side like the PD601 are multiplied by a weighting coefficient to obtain a value added. .. Further, in each of the peripheral pixels 611 to 641, the signal values S612 to S642 of PD612 to 642 on the right side opposite to PD601 are multiplied by a weighting coefficient to obtain a value added. Then, the ratio of the value obtained by weighted averaging using the signal value of PD on the left side and the value obtained by weighted averaging using the signal value of PD on the right side is calculated. Then, the value obtained by multiplying the calculated ratio by the signal S602 of the PD602 on the right side of the pixel 600 is used as the interpolated value of the signal of the PD601 on the left side of the pixel 600.

Similarly, as described in (Equation 4) above, the image processing unit 108 receives signal values of signal values obtained from two PDs included in pixels 650, 660, 670 and 680 of the same color as the pixels to be processed among the peripheral pixels. The ratio of the values obtained by weighted averaging can also be used to interpolate the signal values of defective PDs.

Further, instead of performing interpolation using either one of the above (Equation 3) and (Equation 4), both may be used. That is, in each of the peripheral pixels 610 to 680, the value obtained by multiplying the signal value of PD611 to 681 on the left side like PD601 by a weighting coefficient is obtained. Further, in each of the peripheral pixels 610 to 680, the signal value of PD612 to 682 on the right side opposite to PD601 is multiplied by a weighting coefficient to obtain a value added. Then, the ratio of the value obtained by weighted averaging using the signal value of PD on the left side and the value obtained by weighted averaging using the signal value of PD on the right side is calculated. Then, the value obtained by multiplying the calculated ratio by the signal value of PD602 on the right side of the pixel 600 is used as the interpolated value of the signal of PD601 on the left side of the pixel 600.

The image processing unit 108 may be configured so that (Equation 3) and (Equation 4) can be selected according to the situation. For example, a region of 5 pixels × 5 pixels centered on the pixel to be interpolated is set, and each color filter uses the signal value read from the normal photoelectric conversion unit corresponding to each color filter. Find the average value of the signal values corresponding to. Then, when the average value of the signal values corresponding to any one of the color filters is larger than the average value of the signal values corresponding to the other color filters by exceeding a preset threshold value, (Equation 4) is performed. Use it, and use (Equation 3) in other cases.

This is because when only the signal value of a specific color filter is large, the accuracy is likely to decrease if interpolation is performed using the signal value of a pixel corresponding to a color filter different from the pixel to be corrected. This is to become. For example, consider a case where only the average value of the signal values corresponding to the hue R is large and the average value of the signal values corresponding to the hue G and the hue B is small. Since the S / N ratio of the hue R is large, the ratio of the signal values between the left and right PDs can be obtained with almost no influence of the noise component, but the S / N ratio of the hue G and the hue B is small. Therefore, the noise component has a great influence when obtaining the ratio of the signal values between PDs. In such a case, in order to interpolate the signal of the photoelectric conversion unit included in the pixel corresponding to the hue R, it is possible to use the ratio of the signal values between the PDs obtained from the peripheral pixels corresponding to the hue R. desirable. On the contrary, if the average values of the signal values corresponding to the respective hues are similar, the degree of influence of the noise component does not differ greatly depending on the hues, so that the peripheral pixels corresponding to the hues G and B are obtained. The ratio of the signal values between PDs may be used.

Further, the above-mentioned weighting coefficient may be changed not only by the presence or absence of defects or saturation but also by changes in the brightness and hue of the subject. Since the peripheral pixels referenced during interpolation output signal values corresponding to areas very close to the subject, the ratio of the left and right PD signal values in the peripheral pixels is similar if there is no change in the brightness or hue of the subject. Should do. On the contrary, if the ratio of the signal values of the left and right PDs of any of the peripheral pixels is significantly different from that of the other pixels, it is highly possible that the peripheral pixels are in the region where the brightness and color of the subject change.

Therefore, the image processing unit 108 first calculates the ratio of the signal values of the two PDs of the peripheral pixels 510, 520, 530, and 540 with respect to the image sensor shown in FIG. That is, the ratios R510, R520, R530 and R540 of the following (Equation 5) are calculated.

Then, the above ratios R510, R520, R530 and R540 are compared with each other, a small weighting coefficient is set for a pixel having a large deviation, and then the weighted average is calculated to calculate the signal value after interpolation.
That is, for example, S601 = S602x (w1xR510 + w2xR520 + w3xR530 + w4xR540) ... (Equation 5-2)
The interpolation value S601 may be obtained by the calculation.

The above setting method can be further modified as follows. The ratios R510 and R520 of the signal values of the two PDs of the symmetrical pixels 510 and 520 above and below the pixel 500 to be interpolated are calculated. The difference between these two ratios | R510-R520 | is calculated. If this difference is equal to or greater than a predetermined threshold value, it is determined that the change in the brightness and hue of the subject in the vertical direction is large, and the weighting coefficients w1 and w2 are set to 0.

Similarly, the ratios R530 and R540 of the signal values of the two PDs of the pixels 530 and the pixel 540 on the left and right of the pixel 500 to be interpolated are calculated, and the ratio difference | R530-R540 | is also calculated. .. If this difference is equal to or greater than a predetermined threshold value, it is set to 0 together with the weighting coefficients w3 and w4. If both | R510-R520 | and | R510-R520 | are equal to or greater than the threshold value, the weighting coefficients w1 to w4 are all set to 1. Alternatively, the weighting coefficients w1 to w4 are set so as to be inversely proportional to the ratio of the sizes of | R510-R520 | and | R510-R520 |. That is, if | R510-R520 | is smaller, the weighting coefficients w1 and w2 corresponding to the pixels R510 and R520 may be made larger than the weighting coefficients w3 and w4. On the contrary, if | R510-R520 | is larger, the weighting coefficients w1 and w2 may be made smaller than the weighting coefficients w3 and w4.

As described above, by using the weighting coefficient in the calculation of interpolation, a smaller weighting coefficient can be given to the data of the signal having low reliability, and a more accurate signal of defective pixels can be obtained by interpolation.

The above description is based on the configuration in which two PDs are arranged for one microlens, but the description is not limited to this. For example, the present embodiment can be implemented even in a configuration in which four PDs are arranged for one microlens. FIG. 7 shows a configuration in which four PDs are arranged for one microlens. Hereinafter, processing when the interpolation method in the present embodiment is used will be described using the configuration shown in FIG. 7. In FIG. 7, it is assumed that PD701 is defective.

In the configuration shown in FIG. 7, as described above, for example, when detecting the phase difference in the horizontal direction in the pixel 700, the signals of PD701 and PD702 are added and read, and the signals of PD703 and PD704 are added. Need to be read. Here, the region including PD701 and PD702 is referred to as a first photoelectric conversion region, and the region including PD703 and PD704 is referred to as a second photoelectric conversion region. Since the PD701 has a defect, when the signals of the PD701 and PD702 are added and read out, the correct signal as the first photoelectric conversion region cannot be obtained.

When detecting the horizontal phase difference also in the peripheral pixels 710, it is necessary to add and read the signals of PD711 and PD712, and to add and read the signals of PD713 and PD714. Here, PD711 and PD712, which are located at positions corresponding to the positions of the first photoelectric conversion region in the pixel 700, are referred to as a third photoelectric conversion region. Similarly, they are referred to as PD713, PD714, and a fourth photoelectric conversion region, which are located at positions corresponding to the positions of the second photoelectric conversion region in the pixel 700. Also in the pixels 720 to 740, the photoelectric conversion unit at the position corresponding to the position of the first photoelectric conversion region in the pixel 700 is the position corresponding to the position of the third photoelectric conversion region and the position of the second photoelectric conversion region in the pixel 700. The photoelectric conversion unit in the above is called a fourth photoelectric conversion region.

Following the element shown in FIG. 5, the image processing unit 108 can interpolate the signals of PD701 and PD702 using the following (Equation 6).

However, A1 is a value obtained by adding S711 and S712, A2 is a value obtained by adding S721 and S722, A3 is a value obtained by adding S731 and S732, and A4 is a value obtained by adding S741 and S742. Similarly, B1 is a value obtained by adding S713 and S714, B2 is a value obtained by adding S723 and S724, B3 is a value obtained by adding S733 and S734, and B4 is a value obtained by adding S743 and S744.

In the method shown in (Equation 6), in the pixel 700 having the defective PD701, the signals of PD701 and PD702 are added and read out. Similarly, in the other pixels, the signals of the two PDs on the left side are added and read, and the signals of the two PDs on the right side are added and read. In each of the peripheral pixels 710 to 740, the signals of the two PDs on the left side are multiplied by a weighting coefficient to obtain a weighted average value. Further, in each of the peripheral pixels 710 to 740, the signals of the two PDs on the right side are also multiplied by a weighting coefficient to obtain a weighted average value. Then, the ratio of the value obtained by weighted averaging using the signal values of the two PDs on the left side and the value obtained by weighted averaging using the signal values of the two PDs on the right side is calculated. Then, the calculated ratio is multiplied by the sum of the signal values of PD703 and PD704 on the right side of the pixel 700 to obtain the sum of the signal values of PD701 and PD702. This value is used to detect the phase difference in the horizontal direction.

Similarly, when detecting the phase difference of the vertical method, it is necessary to add and read the signals of PD701 and PD703 and add and read the signals of PD702 and PD704 in the pixel 700. The same applies to peripheral pixels. The image processing unit 108 can interpolate the signals of PD701 and PD703 using the following (Equation 7).

However, C1 is a value obtained by adding S711 and S713, C2 is a value obtained by adding S721 and S723, C3 is a value obtained by adding S731 and S733, and C4 is a value obtained by adding S741 and S743. Similarly, D1 is a value obtained by adding S712 and S714, D2 is a value obtained by adding S722 and S724, D3 is a value obtained by adding S732 and S734, and D4 is a value obtained by adding S742 and S744.

Here, if the signal values in the first to fourth photoelectric conversion regions are defined as SS1, SS2, SS3, and SS4, respectively, instead of the above (Equation 6) and (Equation 7), the following (Equation 8) ) Can also be used.

However, the above N is the number of peripheral pixels used in the calculation .

A similar method can be used when interpolating with signals from peripheral pixels 750, 760, 770 and 780 of the same color.

In the image sensor in which one pixel has two PDs, the defective PD can be regarded as the first photoelectric conversion region, and another PD in the same pixel can be regarded as the second photoelectric conversion region. For example, in the image pickup device shown in FIG. 5, PD501 can be regarded as a first photoelectric conversion region and PD502 can be regarded as a second photoelectric conversion region. Similarly, PD511, 521, 513 and 541 can be regarded as the third photoelectric conversion region, and PD512, 522, 532 and 542 can be regarded as the fourth photoelectric conversion region, respectively. Based on this interpretation, the above (Equation 8) can be applied even to an image sensor in which one pixel has two PDs.

Further, even in the case of an image sensor having a pixel structure having more than four PDs for one microlens, interpolation can be performed according to the above method.

Interpolation of the first photoelectric conversion region including the defective photoelectric conversion unit using the signal level read from the second photoelectric conversion region in the same pixel as the defective photoelectric conversion unit described so far. The method of performing the above is called the first interpolation.

This first interpolation cannot be used if the correct signal cannot be obtained from the second photoelectric conversion region. In such a case, it is necessary to use the second interpolation instead of the first interpolation.

Second interpolation is a technique similar to the interpolation described in the background art.

In step S808 of FIG. 8 described later , the image processing unit 108 is included in a plurality of peripheral pixels corresponding to the same color filter as the pixel including the defective photoelectric conversion unit, and the photoelectric is defective with respect to the microlens. Select the photoelectric conversion unit that exists at the same position as the signal of the conversion unit.

For example, when the image sensor shown in FIG. 6 is used, the signal value of PD601 can be obtained by the following (Equation 9). In (Equation 9), the interpolation value of the defective PD601 is obtained by averaging the signal values of the PD651 to PD681 on the left side of the four peripheral pixels 650 to 680 of the same color as the defective pixel 600, as with the PD601. It shows that it can be done.

A weighted average may be used instead of the simple average of the signal values of PD651 to PD681.

FIG. 8 is an example of a flowchart when the above interpolation method is used.

In step S801, the CPU 103 reads out the information of the defective photoelectric conversion unit stored in the ROM 107. In step S802, the CPU 103 compares the information of the defective photoelectric conversion unit read in step S801 with the information of the defective photoelectric conversion unit, and determines whether or not the pixel being processed includes the defective photoelectric conversion unit. If the pixel being processed is not a pixel including a defective photoelectric conversion unit, the process proceeds to step S810. If the pixel being processed is a pixel including a defective photoelectric conversion unit, the process proceeds to step S803.

In step S803, the CPU 103 further determines whether or not a normal photoelectric conversion region necessary for interpolating the signal of the defective photoelectric conversion unit exists in the same pixel. If a normal photoelectric conversion region required for interpolation exists, the process proceeds to step S804, and the CPU 103 determines to interpolate using the first interpolation. On the contrary, if there is no normal photoelectric conversion region required for interpolation, the process proceeds to step S807, and the CPU 103 determines to interpolate using the second interpolation.

In step S804, the image processing unit 108 determines peripheral pixels including the photoelectric conversion region used for interpolation. As an example thereof, as described above, the closest pixel may be a peripheral pixel, or the closest pixel of the same color may be a peripheral pixel. In step S806, the image processing unit 108 determines the weighting coefficient used for the settlement of interpolation. The method described above is used to determine the weighting factor.

Here, the process returns to step S807, and in the subsequent step S808, the image processing unit 108 selects a plurality of peripheral pixels corresponding to the same color filter as the pixel including the defective photoelectric conversion region.

In step S809, the image processing unit 108 calculates the interpolation using the signal of the photoelectric conversion region included in the selected pixel. In step S810, the CPU 103 determines whether or not the processing of all the pixels has been completed. If the processing of all the pixels is not completed, the process returns to step S802.

According to the present embodiment, when an image sensor having a plurality of pixels having a plurality of photoelectric conversion units is provided for one microlens and interpolation is performed for pixels having defects in some of the photoelectric conversion units. It is possible to provide a higher degree of freedom in determining the surrounding image used for interpolation.

(Other embodiments)
Although the above embodiments have been described based on the implementation in the image pickup apparatus, the present embodiment is not limited to the image pickup apparatus. For example, it may be carried out in a portable device having a built-in image sensor.

In the present invention, a program that realizes one or more functions of the above-described embodiment is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device program. It can also be realized by the process of reading and operating. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Image sensor 101 AFE
102 TG
103 CPU
104 Operation unit 105 Display unit 106 RAM
107 ROM
108 Image processing unit 109 AF calculation unit 110 Flash memory 111 Focal plane shutter 112 Focus drive circuit 113 Aperture drive circuit 114 Focus actuator 115 Aperture actuator 116 1st lens 117 Aperture 118 2nd lens 119 3rd lens 120 Correlation calculation unit

Claims (15)

An image pickup device having a plurality of pixels, each pixel having one microlens and at least a first photoelectric conversion region and a second photoelectric conversion region for receiving light through the microlens. A signal processing device that processes the signal obtained from
When there is an abnormality in the signal from the first photoelectric conversion region in the first pixel among the plurality of pixels, the first for interpolating the signal in the first photoelectric conversion region having the abnormality. The interpolation means has an interpolation means for performing interpolation, and the interpolation means is applied to a signal from the second photoelectric conversion region in the first pixel by a predetermined coefficient to obtain the abnormality from the first photoelectric conversion region. The coefficient is configured to interpolate the signals of the second photoelectric conversion region in at least one peripheral pixel around the first pixel and the first in the at least one peripheral pixel. It is a coefficient determined based on the ratio of signals in the photoelectric conversion region of.
When the interpolation means satisfies a predetermined condition, a pixel adjacent to the first pixel and having a color different from that of the first pixel is selected as the peripheral pixel, and the predetermined condition is not satisfied. In addition, a signal processing apparatus characterized in that a pixel that is not adjacent to the first pixel and has the same color as the first pixel is selected as the peripheral pixel. The signal processing apparatus according to claim 1, wherein the peripheral pixels used by the interpolation means to obtain the ratio are pixels having a color different from that of the pixels including the first photoelectric conversion region. Under the predetermined condition, in the region set based on the pixel to be processed, the signal of the pixel corresponding to any one color is larger than the signal of the pixel corresponding to the other color, and the difference sets a threshold value. The signal processing apparatus according to claim 1 or 2, wherein the signal processing apparatus exceeds the above. Any one of claims 1 to 3, wherein the interpolation means sets a coefficient for a photoelectric conversion region including a defective photoelectric conversion region among the photoelectric conversion regions of the plurality of peripheral pixels to 0. The signal processing device according to the section. Of the peripheral pixels, the ratio of the signal is calculated for each of the two pixels located symmetrically across the pixel.
The signal processing device according to claim 4, wherein the interpolation means sets the coefficient according to a difference in the ratio of the signals. The signal processing device according to claim 5, wherein the interpolation means sets a smaller coefficient as the difference in the ratio of the signals is larger. The signal processing device according to claim 6, wherein the interpolation means sets 0 to the coefficient when the difference in the ratio of the signals is larger than a predetermined threshold value. It has a detecting means for detecting that the output of each of the photoelectric conversion regions of the plurality of pixels has reached the saturation level.
The detection means is characterized in that the coefficient with respect to the photoelectric conversion region included in the pixel in which it is detected that the output of at least a part of the photoelectric conversion region has reached the saturation level is set to 0. The signal processing apparatus according to any one of 1 to 7. The interpolation means is a performs a second interpolation performing interpolation on the second photoelectric conversion region of the signal and the first signal of the photoelectric conversion region without using the ratios, the first The signal processing apparatus according to any one of claims 1 to 8, wherein one of the above-mentioned interpolation and the second interpolation is selected. When the second photoelectric conversion region includes a defective photoelectric conversion region, the interpolation means uses the signal of the photoelectric conversion region corresponding to the first photoelectric conversion region of the peripheral pixels to obtain the above-mentioned interpolation means. The signal processing apparatus according to claim 9, wherein the second interpolation is performed. In the second interpolation,
The signal processing apparatus according to claim 10, wherein the signal processing apparatus is performed by using the average of the signals of the photoelectric conversion region corresponding to the first photoelectric conversion region of the peripheral pixels. It has a storage means for storing information indicating a defective photoelectric conversion area, and has a storage means.
The interpolating means according to any one of claims 1 to 11, wherein the photoelectric conversion area including the photoelectric conversion area indicated by the information stored by the storage means is the target of the first interpolation. The signal processing device described. An image pickup device having a plurality of pixels, each pixel having one microlens and at least a first photoelectric conversion region and a second photoelectric conversion region for receiving light through the microlens. It is a signal processing method that processes the signal obtained from
When there is an abnormality in the signal from the first photoelectric conversion region in the first pixel among the plurality of pixels, an interpolation step for interpolating the signal in the first photoelectric conversion region having the abnormality is performed. In the interpolation step, the signal from the first photoelectric conversion region having the abnormality is interpolated by multiplying the signal from the second photoelectric conversion region in the first pixel by a predetermined coefficient. The coefficient is the signal of the second photoelectric conversion region in at least one peripheral pixel around the first pixel and the first photoelectric conversion region in the at least one peripheral pixel. It is a coefficient determined based on the ratio of signals,
In the interpolation step, when a predetermined condition is satisfied, a pixel adjacent to the first pixel and having a color different from that of the first pixel is selected as the peripheral pixel, and the predetermined condition is not satisfied. In this case, a signal processing method characterized in that a pixel that is not adjacent to the first pixel and has the same color as the first pixel is selected as the peripheral pixel. An image pickup device having a plurality of pixels, each pixel having one microlens and at least a first photoelectric conversion region and a second photoelectric conversion region for receiving light through the microlens. A computer program for signal processing that processes the signal obtained from
When there is an abnormality in the signal from the first photoelectric conversion region in the first pixel among the plurality of pixels, an interpolation step for interpolating the signal in the first photoelectric conversion region having the abnormality is performed. In the interpolation step, the signal from the first photoelectric conversion region having the abnormality is interpolated by multiplying the signal from the second photoelectric conversion region in the first pixel by a predetermined coefficient. The coefficient is the signal of the second photoelectric conversion region in at least one peripheral pixel around the first pixel and the first photoelectric conversion region in the at least one peripheral pixel. It is a coefficient determined based on the ratio of signals,
In the interpolation step, when a predetermined condition is satisfied, a pixel adjacent to the first pixel and having a color different from that of the first pixel is selected as the peripheral pixel, and the predetermined condition is not satisfied. In this case, a computer program for signal processing, characterized in that a pixel that is not adjacent to the first pixel and has the same color as the first pixel is selected as the peripheral pixel. A computer-readable storage medium that stores the computer program of claim 14.

Images