JP4128536B2 - Pixel signal processing apparatus and method - Google Patents

Description

The present invention relates to a pixel signal processing apparatus and method, and more particularly, to a pixel signal representing a value of each color component of a plurality of pixels arranged on a two-dimensional plane, each having one of a plurality of different color components. The present invention relates to a pixel signal processing apparatus and method for generating a pixel signal representing a value of another color component at a position of a target pixel having one color component by interpolation based on the set.
Such a pixel signal processing device is, for example, a pixel signal of any one of a plurality of color components (spectral sensitivity characteristics), for example, red (R), green (G), or blue (B). A plurality of types of photoelectric conversion elements that output an image sensor further include, for example, an image sensor that is arranged in a Bayer shape on a two-dimensional plane, and is used as a part of a color image pickup device to output pixel signals output from the image sensor. Of these, it is used to interpolate a pixel signal representing the value of a missing color component at each pixel position.

  In a conventional image pickup apparatus having an image pickup element in which color filters of three primary colors of red, green, and blue are arranged in a Bayer shape, for example, as shown in Patent Document 1 below, Based on the local output signal distribution for each pixel, the output signal of each pixel is replaced with an average value, and an interpolation method that assumes the linear similarity between the known color geometric figure and the insufficient color geometric figure is assumed. ing.

JP 2001-197512 A

  This conventional method assumes that there is a similar relationship in the state of change of each color component (for example, R, G, B components in a Bayer array) in order to enhance the resolution. For this reason, there is a problem that false colors are generated in the interpolation result in an area where there is no similarity in the state of change of each color component (for example, a boundary between one color and another color), and pixels around the target pixel There is a problem that the interpolation method may not be appropriate depending on how the color component value changes.

  The present invention can suppress the occurrence of false color in the interpolation result even in a region where the state of change of each color component is not similar, and how to change the color component value of pixels around the pixel of interest It is an object of the present invention to provide a pixel signal processing apparatus and method that can always perform interpolation with an optimal interpolation method regardless of the above.

The pixel signal processing apparatus of the present invention is
Based on a set of pixel signals representing the values of the respective color components of a plurality of pixels arranged on a two-dimensional plane, each having any of the first to Nth different color components, the J (J Is a pixel signal representing the value of the Kth color component (K is any one of 1 to N excluding J) at the position of the pixel of interest having one (1 to N) color component. In a pixel signal processing device that generates
First interpolation means for generating a first interpolation value by performing the above interpolation by a first interpolation method;
Second interpolation means for generating a second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
Synthesizing means for synthesizing the first interpolation value obtained by the first interpolation means and the second interpolation value obtained by the second interpolation means, and outputting a pixel signal representing the synthesized interpolation value; equipped with a,
The first interpolation means includes:
A pixel signal representing the value of the Jth color component of the pixel of interest and a pixel having the Jth color component in the vicinity of the pixel of interest, and the pixel of the pixel having the Kth color component in the vicinity of the pixel of interest. Regression equation calculation for calculating a constant of a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the pixel of interest based on the pixel signal representing the value of the K color component Means,
Interpolation calculation using a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component, including the constants calculated by the regression equation calculation means, or the color components of pixels around the pixel of interest An interpolation calculation circuit for performing an interpolation calculation to obtain an average of the values of
The second interpolation means includes:
In a three-dimensional orthogonal coordinate system in which an array of pixels on a two-dimensional plane is represented on a two-dimensional coordinate plane including an H axis and a V axis orthogonal to each other, and a color component value is represented on a Z axis perpendicular to the two-dimensional plane.
The average value of the values of the Jth color component of the pixels at the positions of the pixels having a plurality of Jth color components around the target pixel, and the color components of the target pixel at the position of the target pixel A first geometric figure created with the value of
The average value of the Kth color components of the pixels having a plurality of Kth color components around the target pixel and the target pixel obtained by interpolation by the second interpolation means The second geometric figure created by the value of the Kth color component at the position of is similar to each other,
Determining the value of the Kth color component at the position of the target pixel;
The regression equation calculating means also calculates a correlation coefficient k between the value of the Jth color component and the value of the Kth color component,
The synthesizing means determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k .

  According to the present invention, it is possible to suppress the occurrence of false colors, and it is possible to perform optimum interpolation according to how the color component values of pixels around the pixel of interest change.

  Embodiments of the present invention will be described below with reference to the drawings. The embodiment described below is suitable for use as a part of a full-color image capturing apparatus, for example, a digital camera, but the present invention is not limited to this.

Embodiment 1 FIG.
FIG. 1 is a block diagram showing the configuration of an imaging apparatus such as a digital camera provided with the pixel signal processing apparatus according to Embodiment 1 of the present invention.
Light incident from the lens 1 forms an image on the solid-state image sensor 2. Photoelectric conversion elements are two-dimensionally arranged in the solid-state imaging device 2, and each pixel is constituted by each photoelectric conversion element.

The solid-state imaging device 2 photoelectrically converts incident light and outputs an analog electric signal corresponding to the amount of incident light for each pixel. Each photoelectric conversion element has a red (R), green (G), or blue (B) color filter attached thereto.
The analog electric signal output from the solid-state imaging device 2 is converted into a pixel signal having a binary digital value by the A / D converter 3.
The interpolation processing circuit 4 performs an interpolation process on the pixel signal having a binary digital value sent from the A / D converter 3 and outputs an interpolation result.

  FIG. 2 is a diagram illustrating an arrangement of color filters attached to photoelectric conversion elements arranged two-dimensionally in the solid-state imaging device 2 and color components of pixel signals obtained by the solid-state imaging device 2.

  In FIG. 2, the vertical and horizontal directions correspond to the vertical and horizontal directions of the image, respectively. The arrangement of the color filters is a Bayer type array, and in the output of the image sensor 2, each pixel is a pixel given only an R component value (R pixel), a pixel given only a G component (G pixel), One of the pixels (B pixel) to which only the B component is given. In other words, only one color component value is obtained for each pixel position, and the component values of other colors are not obtained from the output of the image sensor 2.

  In order to obtain a full-color image from this signal obtained by the solid-state imaging device 2, as shown in FIG. 3, components of other colors (insufficient colors) exist at each position of the R pixel, the G pixel, and the B pixel. Then, the component value in the case of assuming that must be obtained by interpolation. In FIG. 3, lowercase letters represent component values obtained by interpolation, and uppercase letters represent component values obtained directly (without interpolation) from the output of the image sensor.

The interpolation processing circuit 4 in FIG. 1 is for interpolating the insufficient color component at each pixel position as described above based on a known color component.
FIG. 4 is a block diagram showing the internal configuration of the interpolation processing circuit 4.
The interpolation processing circuit 4 includes a frame memory 5, a read control circuit 6, a coordinate data calculation circuit 7, an interpolation calculation circuit 11, an interpolation circuit 12, a constant calculation circuit 9, a condition determination circuit 8, and a synthesis circuit. 13.

A pixel signal 5IN having a binary digital value sent from the A / D converter 3 to the interpolation processing circuit 4 is stored in the frame memory 5.
A part of the signal accumulated in the frame memory 5 is output from the frame memory 5 under the control of the read control circuit 6.
The output pixel signal is supplied to the coordinate data calculation circuit 7, the condition determination circuit 8, the interpolation calculation circuit 11, and the interpolation circuit 12.

  Based on the pixel signal input from the frame memory 5, the coordinate data calculation circuit 7 calculates either a color component value possessed by each pixel or a value calculated from a color component value possessed by a pixel near each pixel. Calculate and output two-dimensional coordinate data as an x-coordinate component and a y-coordinate component.

  The two-dimensional coordinate data output from the coordinate data calculation circuit 7 is supplied to the constant calculation circuit 9. The constant calculation circuit 9 performs a regression analysis using the two-dimensional coordinate data to obtain the slope a and y intercept b of the regression equation y = ax + b and the correlation coefficient k, and the interpolation calculation circuit 11 and the synthesis circuit 13 Output.

  Among the above, in the coordinate data calculation circuit 7 and the constant calculation circuit 9, as will be described later, the target pixel and the value of the first color component (R, G or B) in the vicinity of the target pixel and the vicinity of the target pixel Based on the value of the second color component (R, G, or B) (which differs from the first spectral sensitivity characteristic), the value of the first color component and the second value in the vicinity of the target pixel Regression formula calculating means for obtaining a regression formula representing the relationship between the values of the color components is configured.

The interpolation calculation circuit 11 performs an interpolation calculation of insufficient color components from the pixel signal sent from the frame memory 5 and the signals representing the constants a and b sent from the constant calculation circuit 9 and outputs the interpolation result 11OUT to the synthesis circuit 13. To do.
The interpolation calculation circuit 11 further includes a regression equation constant calculated by the regression equation determination means based on the pixel signal input from the frame memory 5 and the signal representing the result of the condition determination from the condition determination circuit 8. The interpolation value of the insufficient color component by performing an interpolation operation using a regression equation representing the relationship between the value of the first color component and the second color component, or by performing an interpolation operation to obtain the average of pixels around the target pixel. Ask for.
The condition determination circuit 8 performs condition determination described later.
The interpolation value 11OUT obtained by the interpolation calculation circuit 11 is supplied to the synthesis circuit 13 as the first interpolation value.

  The coordinate data calculation circuit 7, the condition determination circuit 8, the constant calculation circuit 9, and the interpolation calculation circuit 11 constitute a first interpolation circuit 31.

  On the other hand, the interpolation circuit 12 performs interpolation by a method different from that of the first interpolation circuit 31, and supplies the interpolation value 12OUT obtained as a result of the interpolation to the synthesis circuit 13 as a second interpolation value.

  The synthesizing circuit 13 weights the first interpolation value 11OUT supplied from the interpolation calculation circuit 11 and the second interpolation value 12OUT supplied from the interpolation circuit 12 based on the value sent from the constant calculation circuit 9. The result is added, and the result is output as a combined interpolation value 13OUT. The synthesized interpolation value is stored in the frame memory 5 and accumulated.

  The series of operations from reading out pixel signals accumulated in the frame memory 5 to writing the output 13OUT of the synthesis circuit 13 into the frame memory 5 as described above is completed for interpolation of all insufficient color components for all pixels. (That is, until a pixel signal corresponding to one still image is accumulated in the frame memory 5). When the processing is completed and a signal corresponding to one still image is accumulated in the frame memory 5, a signal 5 OUT corresponding to one still image is read from the frame memory 5 under the control of the read control circuit 6. And output to the outside of the interpolation processing circuit 4.

Next, an interpolation method for the insufficient color component performed in the interpolation processing circuit 4 will be described. For interpolation of missing color components,
(P1) Processing for obtaining G component values at the positions of the R pixel and the B pixel;
(P2) Processing for obtaining the R component value and the B component value at the position of the G pixel,
(P3) includes processing for obtaining the B component value at the position of the R pixel, and (P4) processing for obtaining the R component value at the position of the B pixel, and is performed, for example, in the order described above.

  First, a G component value interpolation method at the positions of the R pixel and the B pixel will be described. First, the interpolation of the G component value at the position of the R pixel will be described.

  The interpolation calculation circuit 11 performs interpolation in one of two modes as will be described in detail below. That is, in principle, interpolation is performed by regression analysis, but when a certain condition is satisfied as described below, that is, when any of [Condition 1] to [Condition 4] described later is satisfied, the target pixel Perform linear interpolation based on the surrounding pixels.

First, the operation when the interpolation calculation circuit 11 performs interpolation by regression analysis will be described.
In the case of interpolation by regression analysis, the R component value at the interpolation target pixel position (target pixel position) and its surrounding pixel positions and the G component value at the pixel positions around the target pixel position are received, and the R component Regression analysis is performed using the value as an explanatory variable and the G component value as an objective variable to obtain a regression equation (representing a regression line) representing the relationship between the R component value and the G component value, and the R component value at the target pixel position is calculated. On the other hand, the G component value at the target pixel position is obtained by using a conversion equation corresponding to the regression equation.

  In general, the slope a, y intercept b, and correlation coefficient of regression equation y = ax + b with respect to M two-dimensional coordinates (xi, yi), (i = 1 to M), where xi is an explanatory variable and yi is an objective variable k is obtained by the following formulas (1), (2), and (3).

The constant calculation circuit 9 calculates the slope a and y intercept b of the regression equation used in the interpolation calculation circuit 11 and the correlation coefficient k of the regression line.
The correlation coefficient k is the strength of the correlation between the X coordinate value and the Y coordinate value used as the explanatory variable and the objective variable, in other words, the two-dimensional coordinate used to calculate the regression equation of the calculated regression equation. It is a value that quantitatively indicates the validity of the data. The absolute value of the correlation coefficient k takes a value between 0 and 1, and when the two-dimensional coordinate data used to calculate the regression equation all fall on the regression line represented by the obtained regression equation, the correlation The absolute value of the number k is 1, and the closer the absolute value of the correlation coefficient k is to 0, the two-dimensional coordinate data used to obtain the regression equation deviates from the regression line represented by the obtained regression equation. Means that That is, the closer the absolute value of the correlation coefficient k is to 1, the higher the validity of the calculated regression equation.

  Two-dimensional coordinate data calculated by the coordinate data calculation circuit 7 is used as the explanatory variable xi and the objective variable yi necessary for the calculations of the expressions (1), (2), and (3).

Here, the expressions (1), (2), and (3) include division. Division is less accurate when the divisor takes a value close to zero. Therefore, when the divisor in the equations (1), (2), and (3) takes a value close to 0, an error increases if interpolation using a regression equation is performed.
The divisors of equations (1) and (2) take values close to 0 when all xi take values close to a certain value xo.
In addition, the divisor in equation (3) takes a value close to 0 when all xi take a value close to a certain value xo or when all yi take a value close to a certain value yo.
Accordingly, for the two-dimensional coordinate data (xi, yi) and (i = 1 to M) output from the coordinate data calculation circuit 7, all xi are close to a certain value xo or all yi are present. When the value is close to the value yo, an increase in error can be prevented by performing processing that does not require the operations of the equations (1), (2), and (3).

  Whether all xi are close to a certain value xo can be determined by whether or not the difference between the maximum value and the minimum value of xi is smaller than a certain threshold value. Similarly, whether all yi are close to a certain value yo can be determined by whether or not the difference between the maximum value and the minimum value of yi is smaller than a certain threshold value.

If all yi have the same value, the divisor of equation (3) becomes 0, so the calculation of equation (3) cannot be performed.
Further, when all xi have the same value, the divisors of the equations (1), (2), and (3) are 0, so that the equations (1), (2), and (3) can be calculated. Can not.
Therefore, when all xi or all yi have the same value, avoiding the situation where computation is impossible by performing processing that does not require the computations of formulas (1), (2), and (3). Can do.

  Therefore, the condition determination circuit 8 determines whether any of the following conditions 1 to 4 is satisfied for the two-dimensional coordinate data (xi, yi) and (i = 1 to M) output from the coordinate data calculation circuit 7. And the interpolation calculation circuit 11 and the constant calculation circuit 9 are controlled according to the determination result. That is, when any of the following conditions 1 to 4 is satisfied, the constant calculation circuit 9 is controlled so as not to calculate the expressions (1), (2), and (3), and the interpolation calculation circuit 11 is Control is performed so that linear interpolation based on pixels around the target pixel is performed based on the output from the frame memory 5 without using the output from the constant calculation circuit 9.

[Condition 1] All xi are the same value xo.
[Condition 2] All yi are the same value yo.
[Condition 3] The difference between the maximum value and the minimum value of xi is less than a certain threshold.
[Condition 4] The difference between the maximum value and the minimum value of yi is less than a certain threshold.

When [Condition 1] and [Condition 3] are satisfied, the two-dimensional coordinate data (xi, yi) and (i = 1 to M) can be approximated by a straight line x = xo.
When [Condition 2] and [Condition 4] are satisfied, the two-dimensional coordinate data (xi, yi) and (i = 1 to M) can be approximated by a straight line y = yo.

Therefore, in any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], the two-dimensional coordinate data (xi, yi), (i = 1 to M) can be approximated by a straight line with high accuracy. Therefore, the correlation coefficient may be set to 1.
Accordingly, the constant calculation circuit 9 outputs 1 as the value of the correlation coefficient in accordance with the output of the condition determination circuit 8 as described above, and does not calculate the slopes a and y intercepts b of the regression equation. The condition determination circuit 8 also instructs the interpolation calculation circuit 11 to perform linear interpolation instead of interpolation using a regression equation.

  Based on the above, the operations of the coordinate data calculation circuit 7, the condition determination circuit 8, the constant calculation circuit 9, and the interpolation calculation circuit 11 will be described in more detail.

  FIG. 5 schematically shows the pixel position of the pixel signal read from the frame memory 5 in order to interpolate the G component value at the position of the R pixel in the present embodiment.

As shown in the figure, the read control circuit 6 sends the R component value R44 of the target pixel R44 and the target pixel near the target pixel R44 to the coordinate data calculation circuit 7, the interpolation calculation circuit 11, and the interpolation circuit 12 from the frame memory 5. G pixel G14 of an insufficient color (color component to be obtained by interpolation of the target pixel position) in the vicinity of the R component values R24, R42, R46, R64 of the pixels R24, R42, R46, R64 of the same color and the target pixel R44. , G23, G25, G32, G34, G36, G41, G43, G45, G47, G52, G54, G56, G63, G65, G74 G component values G14, G23, G25, G32, G34, G36, G41, G43, G45, G47, G52, G54, G56, G63, G65, and G74 are output.
Note that, as described above, a pixel having a certain color component and the color component value of the pixel may be represented by the same symbol.

In the coordinate data calculation circuit 7 that has received these component values, R component values and G component values are obtained for each pixel position as data for obtaining a regression equation, and these are set as X coordinate values and Y coordinate values, respectively. These are paired to form two-dimensional coordinate data.
For example, as shown in the following equation, the R component value R24 of the R pixel R24 is set to the X coordinate value x1, and the average value of the G component values G14, G23, G25, and G34 of the G pixel located around the R pixel R24 is calculated as the R pixel. Two-dimensional coordinate data (x1, y1) having a Y coordinate value y1 corresponding to the position of R24 is configured.
Similarly, as shown by the following equation, the R component value R42 of the R pixel R42 is set as the X coordinate value x2, and the average value of the G component values G32, G41, G43, and G52 of the G pixel positioned around the R pixel R42 is set as the Y coordinate value. The two-dimensional coordinate data (x2, y2) to be performed and the R component value R44 of the R pixel R44 are set to the X coordinate value x3, and the average value of the G component values G34, G43, G45, and G54 of the G pixel located around the Y coordinate is Y The two-dimensional coordinate data (x3, y3) as the coordinate value y3 and the R component value R46 of the R pixel R46 are set as the X coordinate value x4, and the G component values G36, G45, G47, and G56 of the G pixel located in the vicinity thereof Two-dimensional coordinate data (x4, y4) having an average value as a Y-coordinate value y4, an R component value R64 of the R pixel R64 as an X coordinate value x5, and a G pixel and G component values G54, G63, Average of G65 and G74 A two-dimensional coordinate data and Y coordinate value y5 (x5, y5) is determined.

x1 = R24
y1 = (G14 + G23 + G25 + G34) / 4
x2 = R42
y2 = (G32 + G41 + G43 + G52) / 4
x3 = R44
y3 = (G34 + G43 + G45 + G54) / 4
x4 = R46
y4 = (G36 + G45 + G47 + G56) / 4
x5 = R64
y5 = (G54 + G63 + G65 + G74) / 4

  As can be seen from the above formula, in the above example, as the X coordinate value xi, the value (pixel value) of the R component (known color component) at each pixel position obtained from the image sensor 2 is used as it is. As the Y coordinate value yi, an average of G components (insufficient color components) of adjacent pixels located around each pixel position, for example, upper, left, right, and lower, is obtained. The process for obtaining the average can be regarded as a kind of low-pass filtering.

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 5) obtained as described above are output to the condition determination circuit 8 and the constant calculation circuit 9.

The condition determination circuit 8 performs the above [Condition 1], [Condition 2], [Condition 3] for the two-dimensional coordinate data (xi, yi) and (i = 1 to 5) output from the coordinate data calculation circuit 7. It is determined whether or not any of [Condition 4] is satisfied.
When any one of [Condition 1], [Condition 2], [Condition 3], and [Condition 4] is not satisfied, the constant calculation circuit 9 and the interpolation calculation circuit 11 perform the following processing.

  First, the constant calculation circuit 9 uses the two-dimensional coordinate data (xi, yi) and (i = 1 to 5) sent from the coordinate data calculation circuit 7, and the equations (1), (2), (3) , The slope a, y intercept b, and correlation coefficient k of the regression equation are calculated. Then, the values of the slopes a and y intercepts b of the regression line expressed by the regression equation are output to the interpolation calculation circuit 11, and the value of the correlation coefficient k is output to the synthesis circuit 13.

  The interpolation calculation circuit 11 uses the G component value 11OUT at the position of the target pixel R44, the R component value R44 of the target pixel R44 sent from the frame memory 5, and the slope a of the regression line sent from the constant calculation circuit 9. And the following equation using y-intercept b and output.

  11OUT = a × R44 + b

  Any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4] for the two-dimensional coordinate data (xi, yi) and (i = 1 to 5) output from the coordinate data calculation circuit 7 If the condition determination circuit 8 determines that the above is satisfied, the constant calculation circuit 9 and the interpolation calculation circuit 11 perform the following processing.

  That is, the constant calculation circuit 9 outputs “1” as the value of the correlation coefficient to the synthesis circuit 13. Further, it instructs the interpolation calculation circuit 11 to perform interpolation processing that does not use a regression equation, specifically, linear interpolation.

  In this case, the interpolation calculation circuit 11 uses the G component value 11OUT at the position of the target pixel R44 in the vicinity of the target pixel as described below among the signals sent from the frame memory 5, for example, upper, left, right The G component values G34, G43, G45, and G54 in the adjacent G pixels G34, G43, G45, and G54 located below are calculated and output by linear interpolation using the following equations.

  11OUT = (G34 + G43 + G45 + G54) / 4

  On the other hand, the interpolation circuit 12 sets the G component value 12OUT at the position of the target pixel R44 to positions around the target pixel R44 (for example, the upper, left, right, and lower positions) of the signal sent from the frame memory 5. And adjacent) G component values G34, G43, G45, G54 of the G pixels G34, G43, G45, G54, and the target pixel R44 and the surroundings, for example, above, to the left, to the right, and to the other Separated by only one pixel (in this way, “separated by only one other pixel” is sometimes referred to as “the distance between pixels is two pixels”, “ The same applies to the case of different pixel distances.) The lower values using R component values R44, R24, R42, R64, and R46 of R pixels R24, R42, R64, and R46. The calculated formula, and outputs.

FIG. 6 is a diagram for explaining how to determine the G component value 12OUT at the position of the target pixel R44 to be interpolated by the interpolation circuit 12. In the figure, the HV plane corresponds to the imaging surface of the solid-state imaging device 2, and the Z axis represents each component value of each pixel. R1, R2, R3, and R4 are points where the R component values of the pixels R42, R64, R46, and R24 are replaced with their average values, and R0 is a point that represents the R component value R44 of the pixel R44.
G1, G2, G3, and G4 are points where the respective G component values at the respective positions of the pixels G43, G54, G45, and G34 are replaced with their average values, and G0 is obtained by interpolation on the pixel R44. This is a point representing the G component value 12OUT.

  The above calculation is performed on the assumption that there is a similar relationship in the state of change of each color component. Specifically, the value of 12OUT is set so that the square weights R0R1R2R3R4 and the square weights G0G1G2G3G4 are similar. It stipulates. That is, with respect to the known color component R in the target pixel R44, a known color component polyhedron (geometrical figure) created by the average value of the R component value of the target pixel R44 and the R component value of the pixel having the R component in the vicinity of the target pixel R44; As for the insufficient color component G to be interpolated on the target pixel R44, the insufficient color component generated by the average value of the G component values of the pixel having the value 12OUT obtained by interpolation and the G component (insufficient color component) in the vicinity of the target pixel R44. 12OUT is determined so that the polyhedron (geometric figure) is similar.

  In other words, the difference between the average value of the R component values of the pixels (R42, R64, R46, R24) having a plurality of R components around the target pixel R44 and the R component value at the position of the target pixel R44; The ratio of the distance between the plurality of pixels (R42, R64, R46, R24) and the target pixel (R44) used for the calculation of the average value on the two-dimensional plane and the plurality of pixels around the target pixel R44. The difference between the average value of the G component values of the pixels having the G component (G43, G54, G45, G34) and the G component value G0 = 12OUT at the position of the target pixel R44 obtained by interpolation, and the average value The position of the target pixel is such that the ratio between the plurality of pixels (G43, G54, G45, G34) used for the calculation and the distance between the target pixel (R44) on the two-dimensional plane is the same. Value 12OUT of kicking the G component is determined.

  In the synthesis circuit 13, the interpolation value 11OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12OUT supplied from the interpolation circuit 12 are weighted and added according to the absolute value of the correlation coefficient k calculated by the constant calculation circuit 9. The synthesized interpolation value 13OUT is output. This weighted addition is performed so that the proportion occupied by 11OUT is larger than 12OUT if the absolute value of the correlation coefficient k is close to “1”, and 11OUT if the absolute value of the correlation coefficient k is close to “0”. Further, the ratio of 12OUT is increased.

  The interpolation value 13OUT calculated as described above is written in the frame memory 5 as a synthesized G component value at the position of the target pixel R44.

  The above interpolation processing is performed on the G component values at the positions of all R pixels.

  The above is a method of interpolating the G component value at the position of the R pixel, but the interpolation of the G component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  Next, an interpolation method for the R component value and the B component value at the position of the G pixel will be described. When the G pixel is centered, the arrangement of each of the R, G, and B pixels is different depending on whether the pixel of interest is in the odd row (a) or the even row (b), as shown in FIG. First, the interpolation of the R component value and the B component value at the position of the G pixel in the odd row will be described.

  FIG. 8 shows a signal read from the frame memory 5 in order to interpolate the R component value and the B component value at the position of the G pixel in the odd row.

  First, an interpolation method of R component values at the positions of odd-numbered G pixels will be described. As shown in FIG. 8, the pixel having the R component centered on the target pixel G pixel is located in the horizontal direction (left-right direction) and does not exist in the vertical direction (up-down direction).

  First, the G component value G44 of the target pixel G44 and the G component values G42 and G46 of the G pixel G42 and G46 which are located around the target pixel G44, for example, on the left and right sides and separated by two pixels, and the target pixel G44. Surrounding, for example, an R pixel R41 that is located on the left and separated by 3 pixels, an R pixel R43 that is located on the left and adjacent, an R pixel R45 that is located on the right and adjacent, and a pixel located on the right and separated by 3 pixels The R component values R41, R43, R45, and R47 of the R pixel R47 are read from the frame memory 5 by the read control circuit 6 and supplied to the coordinate data calculation circuit 7, the interpolation calculation circuit 11, and the interpolation circuit 12.

  In the coordinate data calculation circuit 7, two-dimensional coordinate data (x1, y1) created by the average value of the G component value G42 and the R component values R41 and R43, and the average value of the G component value G44 and the R component values R43 and R45 are created. Three of the two-dimensional coordinate data (x2, y2) and the two-dimensional coordinate data (x3, y3) created by the average value of the G component value G46 and the R component values R45, R47 are calculated by the following formula, and the constant calculation circuit 9 Is output.

x1 = G42
y1 = (R41 + R43) / 2
x2 = G44
y2 = (R43 + R45) / 2
x3 = G46
y3 = (R45 + R47) / 2

  As can be seen from the above formula, as the X coordinate value xi, the value of the G component (known color component) at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel is An average of R components (insufficient color components) of pixels located around the position, for example, on the left and right sides and adjacent to each pixel is obtained.

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 3) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that the above does not apply, the following processing is performed.

  First, the constant calculation circuit 9 uses the two-dimensional coordinate data (xi, yi), (i = 1 to 3) sent from the coordinate data calculation circuit 7, and uses the equations (1), (2), (3). The values of the slope a and y intercept b of the regression line obtained as a result are output to the interpolation calculation circuit 11 and the value of the correlation coefficient k is output to the synthesis circuit 13.

  In the interpolation calculation circuit 11, the R component value 11 OUT on the target pixel G 44 is converted into the G component value G 44 of the target pixel G 44 among the signals output from the frame memory 5 and the regression line sent from the constant calculation circuit 9. Calculate and output the following equation using the slope a and the y-intercept b.

  11OUT = a × G44 + b

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 3) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that this is the case, the following processing is performed.

  The constant calculation circuit 9 outputs “1” as the correlation coefficient k to the synthesis circuit 13 and instructs the interpolation calculation circuit 11 to perform linear interpolation without using the regression equation.

  In the interpolation calculation circuit 11, the R component value 11OUT at the position of the target pixel G44 is used as the R component values R43 and R45 of the R pixels R43 and R45 in the vicinity of the target pixel among the signals sent from the frame memory 5. Calculate and output by linear interpolation using the following formula.

  11OUT = (R43 + R45) / 2

  On the other hand, the interpolation circuit 12 uses the R component value 12OUT at the position of the target pixel G44 as the R component values R43, R45 of the pixels R43, R45 and the pixels G44, G42, G46 among the signals sent from the frame memory 5. Calculation is performed by the following equation using the G component values G44, G42, and G46, and the result is output.

FIG. 9 is a diagram for explaining how to determine the R component value 12OUT at the position of the target pixel G44 to be interpolated by the interpolation circuit 12. In the figure, the H axis corresponds to the horizontal axis on the imaging surface of the solid-state imaging device 2, and the Z axis represents the component value of each pixel. G1 and G2 are points where the G component values of the pixels G42 and G46 are replaced with average values of the G component values G42 and G46, and G0 is a point representing the G component value G44 of the pixel G44.
R1 and R2 are points where the R component values of the pixels R43 and R45 are replaced by their average values, and R0 is a point representing the R component value 12OUT to be interpolated on the pixel G44.

  The above calculation determines the value of R0 (= 12OUT) so that the triangle G0G1G2 and the triangle R0R1R2 are similar. That is, with respect to the known color component G in the target pixel G44, a known color component triangle (geometrical figure) formed by the average value of the G component value of the target pixel G44 and the G component value of the pixel having the G component in the vicinity of the target pixel G44; As for the insufficient color component R to be interpolated on the target pixel G44, the insufficient color component formed by the average value of the R component values of the pixel having the value 12OUT obtained by interpolation and the R component (insufficient color component) in the vicinity of the target pixel G44. 12OUT is determined so that triangles (geometric figures) are similar.

  In other words, the difference between the average value of the G component values of the pixels (G42, G46) having a plurality of G components around the target pixel G44 and the value of the G component at the position of the target pixel G44, and the average value The ratio of the distance between the plurality of pixels (G42, G46) and the target pixel (G44) used in the calculation of the distance on the two-dimensional plane to the pixel having the plurality of R components around the target pixel G44 (R43, The difference between the average value of the R component values of R45) and the G component value R0 = 12OUT on the target pixel R44 obtained by interpolation, and a plurality of pixels (R43, R45) used for the calculation of the average value The value 12OUT of the R component at the position of the target pixel is determined so that the ratio of the distance between the target pixel (G44) and the distance on the two-dimensional plane is the same.

  In the synthesis circuit 13, the interpolation value 11 OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12 OUT supplied from the interpolation circuit 12 are weighted according to the absolute value of the correlation coefficient k supplied from the constant calculation circuit 9. The result is synthesized by addition, and the result is output as a synthesized interpolation value 13OUT. This weighted addition is performed so that the proportion occupied by 11OUT is larger than 12OUT if the absolute value of the correlation coefficient k is close to “1”, and 11OUT if the absolute value of the correlation coefficient k is close to “0”. Further, the ratio of 12OUT is increased.

  The interpolation value 13OUT obtained as described above is written in the frame memory 5 as an interpolation value representing the R component value at the position of the target pixel.

  The above interpolation processing is performed on the R component values at the G pixel positions in all odd rows.

  Next, a method for interpolating the B component value at the position of the G pixel in the odd row will be described. As shown in FIG. 8, the pixel having the B component centered on the G pixel that is the target pixel is positioned in the vertical direction (vertical direction) and does not exist in the horizontal direction (horizontal direction).

  First, the read control circuit 6 causes the frame data 5 to the coordinate data calculation circuit 7, the interpolation calculation circuit 11, and the interpolation circuit 12, for example, above and below the G component value G44 of the pixel of interest G44 and the surroundings of the pixel of interest G44. G pixel values G24 and G64 of G pixels G24 and G64 separated by two pixels and surrounding the target pixel G44, for example, an upper B pixel B14 separated by three pixels, and an upper adjacent B pixel The B component values B14, B34, B54, and B74 of the B pixel B54, the B pixel B54 located below and adjacent to the B pixel B74, and the B pixel B74 located below and separated by three pixels are supplied.

  In the coordinate data calculation circuit 7, two-dimensional coordinate data (x1, y1) in which the G component value G24 is set as the X coordinate value x1 and the average value of the B component values B14 and B34 is set as the Y coordinate value y1, as shown by the following equation. ), Two-dimensional coordinate data (x2, y2) in which the G component value G44 is the X coordinate value x2, the average value of the B component values B34 and B54 is the Y coordinate value y2, and the G component value G46 is the X coordinate value x3. And two-dimensional coordinate data (x3, y3) in which the average value of the B component values B54 and B74 is set as the Y coordinate value is determined and output to the constant calculation circuit 9.

x1 = G24
y1 = (B14 + B34) / 2
x2 = G44
y2 = (B34 + B54) / 2
x3 = G64
y3 = (B54 + B74) / 2

  As can be seen from the above formula, as the X coordinate value xi, the value of the G component (known color component) at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel is The average of B components (insufficient color components) of pixels located around the position, for example, above and below and adjacent to each pixel is obtained.

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 3) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that the above does not apply, the following processing is performed.

  First, the constant calculation circuit 9 uses the two-dimensional coordinate data (xi, yi) and (i = 1 to 3) sent from the coordinate data calculation circuit 7, and uses equations (1), (2), and (3). The values of the slope a and y intercept b of the regression line obtained as a result are output to the interpolation calculation circuit 11 and the value of the correlation coefficient k is output to the synthesis circuit 13.

  In the interpolation calculation circuit 11, the B component value 11 OUT on the target pixel G 44 is converted into the G component value G 44 of the target pixel G 44 among the signals output from the frame memory 5 and the regression line sent from the constant calculation circuit 9. Calculate and output the following equation using the slope a and the y-intercept b.

  11OUT = a × G44 + b

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 3) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that this is the case, the following processing is performed.

The constant calculation circuit 9 outputs “1” as the value of the correlation coefficient k to the synthesis circuit 13 and instructs the interpolation calculation circuit 11 to perform linear interpolation without using a regression equation. put out.

  In the interpolation calculation circuit 11, the B component value 11OUT at the position of the target pixel G44 is used as the B component values B34 and B54 of the B pixels B34 and B54 in the vicinity of the target pixel among the signals sent from the frame memory 5. Calculate and output by linear interpolation using the following formula.

  11OUT = (B34 + B54) / 2

  On the other hand, in the interpolation circuit 12, the B component value 12OUT at the position of the pixel of interest G44 is converted into the B component values B34 and B54 of the pixels B34 and B54 and the pixels G44, G24 and G64 among the signals sent from the frame memory 5. Calculation is performed by the following equation using the G component values G44, G24, and G64, and output.

FIG. 10 is a diagram for explaining how to determine the B component value B0 (12OUT) at the position of the target pixel G44 to be interpolated by the interpolation circuit 12. In the figure, the V axis corresponds to the vertical axis of the imaging surface of the solid-state imaging device 2, and the Z axis represents the component value of each pixel. G1 and G2 are points where the G component values of the pixels G24 and G64 are replaced with average values of the G component values G24 and G64, and G0 is a point representing the G component value G44 of the pixel G44.
B1 and B2 are points where the R component values of the pixels B34 and B54 are replaced by their average values, and B0 is a point representing the R component value 12OUT to be interpolated on the pixel G44.

  The above calculation determines the value of 12OUT so that the triangle G0G1G2 and the triangle B0B1B2 are similar. That is, with respect to the known color component G in the target pixel G44, a known color component triangle (geometrical figure) formed by the average value of the G component value of the target pixel G44 and the G component value of the pixel having the G component in the vicinity of the target pixel G44; For the insufficient color component B to be interpolated on the target pixel G44, a short color component triangle (geometrical figure) formed by the average value of the value 12OUT obtained by interpolation and the R component value of the R component in the vicinity of the target pixel G44 ) Is determined so that 12OUT is similar.

  In other words, the difference between the average value of the G component values of the pixels (G24, G64) having a plurality of G components around the target pixel G44 and the value of the G component at the position of the target pixel G44, and the average value The ratio of the distance between the plurality of pixels (G24, G64) and the target pixel (G44) used in the calculation of the distance on the two-dimensional plane and the pixels (B34, B, The difference between the average value of the B component values of B54) and the G component value 12OUT on the target pixel B44 obtained by interpolation, the plurality of pixels (B34, B54) used for the calculation of the average value, and the target The B component value 12OUT at the position of the target pixel is determined so that the ratio of the distance between the pixels (G44) on the two-dimensional plane is the same.

  In the synthesis circuit 13, the interpolation value 11 OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12 OUT supplied from the interpolation circuit 12 are weighted according to the absolute value of the correlation coefficient k supplied from the constant calculation circuit 9. The result is synthesized by addition, and the result is output as a synthesized interpolation value 13OUT. This weighted addition is performed so that the proportion occupied by 11OUT is larger than 12OUT if the absolute value of the correlation coefficient k is close to “1”, and 11OUT if the absolute value of the correlation coefficient k is close to “0”. Further, the ratio of 12OUT is increased.

  The interpolation value 13OUT obtained as described above is written in the frame memory 5 as an interpolation value representing the B component value at the position of the target pixel.

  The above interpolation processing is performed on the B component values at the G pixel positions in all odd rows.

The explanation of the interpolation of the R component value at the G pixel position in the odd row and the explanation of the interpolation of the B component value at the G pixel position in the odd row are the vertical direction (up and down direction) and the horizontal direction (left and right direction). Is just replaced.
The above is the interpolation method of the R component value and the B component value at the position of the G pixel in the odd row.
Interpolation of the R component value and the B component value at the position of the G pixel in the even row is performed in the same manner as in the odd row. However, “vertical direction” and “horizontal direction” are interchanged. From another point of view, the interpolation of the R component value and the B component value at the position of the G pixel in the even row replaces the interpolation between the R component value and the B component value at the position of the G pixel in the odd row, and R and B. The other operations are the same.

  Next, an interpolation method for the B component value at the position of the R pixel will be described. The “R pixel” means a pixel having only an R component in the output of the image sensor 2. (The same applies to “G pixel” and “B pixel”.) However, this “R pixel” has a G component obtained by previous interpolation, and this G component value is an R value described below. This is used for interpolation of the B component value at the pixel position.

  FIG. 11 shows a signal read from the frame memory 5 in order to interpolate the B component value at the position of the R pixel having the G component. As indicated by reference numerals g44, g33, g35, g53, and g55 in FIG. 11, there are pixels having the G component not only at the position of the R pixel R44 that is the target pixel but also around it. The lowercase letter g indicates that the G component value was generated by previous interpolation. In the coordinate data calculation circuit 7, as shown by the following equation, two-dimensional coordinate data (x1, y1) in which the G component value g33 of the pixel B33 is set to the X coordinate value x1 and the B component value B33 is set to the Y coordinate value y1; Two-dimensional coordinate data (x2, y2) in which the G component value g35 of the pixel B35 is the X coordinate value x2, the B component value B35 is the Y coordinate value y2, and the G component value g53 of the pixel B53 is the X coordinate value x3. Two-dimensional coordinate data (x3, y3) in which the component value B53 is the Y coordinate value y3, two-dimensional coordinate data in which the G component value g55 of the pixel B55 is the X coordinate value x4, and the B component value B55 is the Y coordinate value y4 ( x4, y4) are determined and output.

x1 = g33
y1 = B33
x2 = g35
y2 = B35
x3 = g53
y3 = B53
x4 = g55
y4 = B55

  As can be seen from the above formula, as the X coordinate value xi (i = 1 to 4), the value of the G component (known color component) of each pixel position generated by the previous interpolation is used as it is, and the Y coordinate is used. As the value yi, the value of the B component (insufficient color component) at each pixel position obtained from the image sensor 2 is used as it is.

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 4) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that the above does not apply, the following processing is performed.

  First, the constant calculation circuit 9 uses the two-dimensional coordinate data (xi, yi), (i = 1 to 4) sent from the coordinate data calculation circuit 7, and uses the equations (1), (2), (3). And the values of the slope a and y intercept b of the regression line obtained as a result are output to the interpolation operation circuit 11 and the value of the correlation coefficient k is output to the synthesis circuit 13.

  In the interpolation calculation circuit 11, the B component value 11 OUT on the target pixel R 44, the G component value g 44 on the target pixel R 44 among the signals output from the frame memory 5, and the regression line sent from the constant calculation circuit 9. Is calculated by the following formula using the slope a and the y-intercept b and output.

  11OUT = a × g44 + b

  The two-dimensional coordinate data (xi, yi) and (i = 1 to 4) output from the coordinate data calculation circuit 7 are any of the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. If the condition determination circuit 8 determines that this is the case, the following processing is performed.

  The constant calculation circuit 9 outputs “1” as the correlation coefficient k to the synthesis circuit 13 and instructs the interpolation calculation circuit 11 to perform linear interpolation without using the regression equation.

  In the interpolation calculation circuit 11, the B component value 11OUT at the position of the target pixel R44 is used as the B component value B33 of the B pixels B33, B35, B53, and B55 in the vicinity of the target pixel among the signals sent from the frame memory 5. Calculation is performed by linear interpolation according to the following equation using B35, B53, and B55, and output.

  11OUT = (B33 + B35 + B53 + B55) / 4

  On the other hand, the interpolation circuit 12 uses the B component value 12OUT at the position of the target pixel R44 as the B component value B33, B35, B53, B55 in the pixels B33, B35, B53, B55 among the signals sent from the frame memory 5. And the G component values g44, g33, g35, g53, and g55 in the pixels R44, R33, R35, R53, and R55 are calculated by the following formula and output.

FIG. 12 is a diagram for explaining how to determine the B component value 12OUT at the position of the target pixel R44 to be interpolated by the interpolation circuit 12. In the figure, the HV plane corresponds to the imaging surface of the solid-state imaging device 2, and the Z axis represents the component value of each pixel. g1, g2, g3, and g4 are points where the G component values g53, g55, g35, and g33 of the pixels B53, B55, B35, and B33 are replaced with their average values, and g0 is the G component value g44 of the pixel R44. It is the point that expressed.
B1, B2, B3, and B4 are points where the B component values of the pixels B53, B55, B35, and B33 are replaced with their average values, and B0 is a point that represents the B component value 12OUT interpolated on the pixel R44. It is.

  In the above calculation, the value of 12OUT is determined so that the square weights g0g1g2g3g4 and the square weights B0B1B2B3B4 are similar. That is, with respect to the known color component G in the target pixel R44, a known color component polyhedron (geometrical figure) created by the average value of the G component value of the target pixel R44 and the G component value of the pixel having the G component in the vicinity of the target pixel R44; For the insufficient color component B to be interpolated on the target pixel R44, the insufficient color component polyhedron (geometrical figure) formed by the average value of the value 12OUT obtained by interpolation and the B component value of the B component in the vicinity of the target pixel R44 ) Is determined so that 12OUT is similar.

  In other words, the average value of the G component values (g53, g55, g35, g33) of the pixels (B53, B55, B35, B33) having a plurality of G components around the target pixel R44 and the position of the target pixel R44 And the distance between the plurality of pixels (B53, B55, B35, B33) and the target pixel (R44) used for calculating the average value on the two-dimensional plane, And the average value of the B component values (B53, B55, B35, B33) of the pixels (B53, B55, B35, B33) having a plurality of B components around the target pixel R44, and the attention obtained by interpolation The difference between the B component value 12OUT at the position of the pixel R44 and the plurality of pixels (B53, B55, B35, B33) used for the calculation of the average value and the target pixel (R44) As the ratio of the distance on the dimension plane is identical to each other, the value 12OUT of the B component at the position of the pixel of interest is determined.

  In the synthesis circuit 13, the interpolation value 11 OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12 OUT supplied from the interpolation circuit 12 are weighted according to the absolute value of the correlation coefficient k supplied from the constant calculation circuit 9. The result is synthesized by addition, and the result is output as a synthesized interpolation value 13OUT. This weighted addition is performed such that if the absolute value of the correlation coefficient k is close to “1”, the proportion occupied by 11OUT is larger than 12OUT. If the absolute value k of the correlation coefficient is close to “0”, 11OUT is used. Further, the ratio of 12OUT is increased.

  The interpolation value 13OUT obtained as described above is written in the frame memory 5 as an interpolation value representing the B component value at the position of the target pixel.

  The above interpolation processing is performed on the B component values at the positions of all R pixels.

  The above is a method of interpolating the B component value at the position of the R pixel, but the interpolation of the R component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  As described above, the interpolation processing is completed, all insufficient color component values are given to all pixels, the component values of R, G, and B colors are aligned, and a signal corresponding to one still image is accumulated in the frame memory 5. It will be done. When a signal corresponding to one still image is accumulated in the frame memory 5, the read control circuit 6 reads a signal 5 OUT corresponding to one still image from the frame memory 5 and outputs it to the outside of the interpolation processing circuit 4. .

  As described above, in the interpolation processing circuit 4, the interpolation calculation circuit 11 and the interpolation circuit 12 perform different interpolation processes, and the results are weighted and added.

The effects of the interpolation described above are as follows.
The process performed by the interpolation calculation circuit 11 is an interpolation process based on the property that the color signal locally changes from one color to another. Then, the change from one color to another color is obtained by a regression equation, and the color signal is locally changed from one color to another color by obtaining the missing color component value on the regression line represented by the obtained regression equation. Interpolation using the property of changing to.

  This method is excellent for interpolation in a region where the color smoothly changes from one color to another. Since such a color change occurs at a low frequency, the processing performed by the interpolation calculation circuit 11 is particularly excellent for color interpolation in a low frequency region.

  Further, the processing performed by the interpolation circuit 12 is, for example, in the case of interpolation of the G component value in the R pixel, the geometric figure formed by the average value of the R pixel in the local region and the value of the R pixel of interest, and the local region The geometric figure formed by the average value of the G pixels and the G component value generated as a result of the interpolation operation is made to be similar.

  Thus, by calculating | requiring an interpolation pixel value, the increase / decrease with respect to the average value of the local color of R signal and G signal corresponds. As a result, it can be said that the increase and decrease of the signal level of each color of R and G coincide.

  As an example of the description, the interpolation of the G component value at the position of the R pixel is taken up, but the same can be said for other interpolations.

  That is, interpolation processing is performed in the local region so that the increase and decrease of the signal level of each color of R, G, and B coincides, and the color interpolation is particularly excellent in the high frequency region.

  The correlation coefficient is a statistic that quantitatively indicates the validity of the obtained regression equation with respect to the two-dimensional coordinate data used for obtaining the regression equation. When the two-dimensional coordinate data used to obtain the regression equation is on the regression line represented by the obtained regression equation, the absolute value of the correlation coefficient is “1”, and the absolute value of the correlation coefficient is 0. The closer it is, the more the coordinate data used to obtain the regression equation is at a point that is more deviated from the regression line represented by the obtained regression equation.

  Therefore, the fact that the correlation coefficient is large means that the interpolation pixel value calculated by the interpolation calculation circuit 11 is appropriate, and that the correlation coefficient is small means that the interpolation pixel value calculated by the interpolation calculation circuit 11 is invalid. That is appropriate.

  Therefore, when the absolute value of the correlation coefficient is large, weighted addition is performed so that the interpolation pixel value calculated by the interpolation calculation circuit 11 is large. Conversely, when the absolute value of the correlation coefficient is small, the interpolation pixel calculated by the interpolation circuit 12 is used. If weighted addition is performed so as to increase the value, a more suitable interpolation pixel value can be calculated according to the situation.

  Hereinafter, the effect of this interpolation processing will be described in detail by taking, as an example, the case of interpolating the R component value at the G pixel position with reference to FIGS. 13, 14, 15, and 16.

  13 and 14, the horizontal axis represents the position of each pixel in the direction in which the pixels are aligned with GRGR... (For example, odd rows), the vertical axis represents the pixel value, ■ represents the pixel value of the R pixel, The number written on the left side represents the pixel value, the ◆ represents the pixel value of the G pixel, the number written on the side represents the pixel value, and the solid line represents the signal change curve of the R component and the G component. △ represents the interpolation result of the insufficient color component in the interpolation calculation circuit 11 and represents the value to which the number written horizontally is interpolated, ○ represents the interpolation result of the insufficient color component value in the interpolation circuit 12 and horizontal The number written in indicates the value to be interpolated.

  FIG. 15 shows the result of drawing a regression line with the data shown in FIG. FIG. 16 shows the result of drawing a regression line with the data shown in FIG. In FIGS. 15 and 16, the equation shown in the figure is a regression line, and k is a correlation coefficient.

  When the color gradually changes from one color to another as shown in FIG. 13, the absolute value of the correlation coefficient is close to “1”, but as shown in FIG. When the high frequency component is conspicuous, it can be seen that the absolute value of the correlation coefficient is a value far from “1” as compared with the case of FIG.

  Further, when the color changes smoothly as shown in FIG. 13, the insufficient color component value interpolated by the interpolation calculation circuit 11 takes a value close to a curve written by a solid line, but the high frequency component as shown in FIG. Is conspicuous, the deficient color component value interpolated by the interpolation circuit 12 takes a value close to a curve written by a solid line.

  Therefore, when the absolute value of the correlation coefficient is large, weighting addition is performed so that the insufficient color component value calculated by the interpolation calculation circuit 11 is large. Conversely, when the absolute value of the correlation coefficient is small, the shortage calculated by the interpolation circuit 12 is used. If weighted addition is performed so as to increase the color component value, a more suitable insufficient color component value can be calculated according to the situation.

  Note that the synthesis circuit 13 may select one of the interpolation value 11OUT from the interpolation calculation circuit 11 and the interpolation value 12OUT from the interpolation circuit 12. For example, when a threshold value is set and the absolute value of the correlation coefficient k calculated by the constant calculation circuit 13 is larger than this threshold value, the interpolation value 11OUT from the interpolation calculation circuit 11 is output and calculated by the constant calculation circuit 9 If the absolute value of the correlation coefficient k is less than or equal to this threshold value, a method of outputting the interpolation value 12OUT from the interpolation circuit 12 can be considered. Note that the operation of selecting one of them corresponds to switching the composition ratio to either 0 or 1.

  Various configurations of the full-color image capturing apparatus other than FIG. 1 are conceivable. For example, a storage medium for storing a captured image is mounted, or a circuit that performs image compression processing is combined.

Embodiment 2. FIG.
The overall configuration of an imaging apparatus such as a digital camera provided with the pixel signal processing apparatus according to the second embodiment of the present invention is as shown in FIG. 1 and is the same as that of the first embodiment. However, as described in detail below, the configuration of the interpolation processing circuit 4 is different.

  That is, the interpolation processing circuit 4 according to the second embodiment determines which direction is strong in the vertical and horizontal directions around the target pixel, and performs different interpolation processing depending on the direction with strong correlation. Do.

  As shown in FIG. 17, the interpolation processing circuit 4 of this embodiment includes a frame memory 5, a read control circuit 6, a correlation determination circuit 24, a coordinate data calculation circuit 7, a condition determination circuit 8, and a constant calculation. The circuit 9 includes an interpolation calculation circuit 11, an interpolation circuit 12, and a synthesis circuit 13.

The frame memory 5 stores the signal 5IN as in the first embodiment.
A part of the signal stored in the frame memory 5 is supplied to the coordinate data calculation circuit 7, the condition determination circuit 8, the correlation determination circuit 24, the interpolation calculation circuit 11, and the interpolation circuit 12 under the control of the read control circuit 6.

The correlation discriminating circuit 24 discriminates whether the image has a strong correlation in the vertical direction or the horizontal direction around the target pixel based on the signal supplied from the frame memory 5, and the discrimination result Is output to the coordinate data calculation circuit 7, the interpolation calculation circuit 11, and the interpolation circuit 12.
For example, when there is little change in the vertical direction and there are many changes in the horizontal direction like a fine vertical stripe image, it is determined that the correlation is strong in the vertical direction, and there is no change in the horizontal direction like a fine horizontal stripe image. If there are few and there are many changes in the vertical direction, it is determined that the correlation is strong in the horizontal direction.

The coordinate data calculation circuit 7 obtains two-dimensional coordinate data by performing later-described calculation based on the correlation determination result in the correlation determination circuit 24 and a signal input from the frame memory 5, and outputs it to the constant calculation circuit 9.
The constant calculation circuit 9 performs a predetermined calculation using the two-dimensional coordinate data sent from the coordinate data calculation circuit 7 and calculates the regression line slope a, y intercept b, and correlation coefficient k with the interpolation calculation circuit 11 and the synthesis circuit. 13 is output.
The interpolation calculation circuit 11 is based on the signal sent from the frame memory 5, the correlation direction judgment signal sent from the correlation judgment circuit 24, and the signals (constants a and b) sent from the constant calculation circuit 9 (insufficient color components). , And the interpolation result is output to the synthesis circuit 13 as the first interpolation value 11OUT.

The interpolation circuit 12 interpolates the insufficient color component based on the correlation direction determination signal from the correlation determination circuit 24 and the signal input from the frame memory 5, and outputs the interpolation result to the synthesis circuit 13 as the second interpolation value 12OUT. .
The synthesizing circuit 13 is based on the correlation coefficient k sent from the constant calculation circuit 9 and the first interpolation value 11OUT output from the interpolation calculation circuit 11 and the second interpolation value 12OUT output from the interpolation circuit 12. Are combined by weighted addition, and the combined interpolation value 13OUT is output. The frame memory 5 stores the combined interpolation value 13OUT output from the combining circuit 13.

  A series of operations from reading the signal accumulated in the frame memory 5 to writing the output 13OUT of the synthesis circuit 13 to the frame memory 5 as described above until the interpolation of all the insufficient color components for all the pixels is completed ( That is, it repeats until a signal corresponding to one still image is accumulated in the frame memory 5). When the processing is completed and a signal corresponding to one still image is accumulated in the frame memory 5, a signal 5 OUT corresponding to one still image is read from the frame memory 5 under the control of the read control circuit 6. And output to the outside of the interpolation processing circuit 4.

Next, an interpolation method for insufficient color components performed in the interpolation processing circuit 4 will be described. For interpolation of missing color components,
(P1) Processing for obtaining G component values at the positions of the R pixel and the B pixel;
(P2) Processing for obtaining the R component value and the B component value at the position of the G pixel,
(P3) includes processing for obtaining the B component value at the position of the R pixel, and (P4) processing for obtaining the R component value at the position of the B pixel, and is performed, for example, in the order described above.

  First, a G component value interpolation method at the positions of the R pixel and the B pixel will be described. First, the interpolation of the G component value at the position of the R pixel will be described.

  The interpolation calculation circuit 11 performs interpolation in one of two modes as will be described in detail below. That is, in principle, interpolation by regression analysis is performed, but when a certain condition is satisfied as described below, linear interpolation based on surrounding pixels is performed.

  In the case of interpolation by regression analysis, the R component value at the target pixel position and its surrounding pixel positions and the G component value at the pixel positions around the target pixel position are received, and the R component value is used as an explanatory variable, Regression analysis is performed using the G component value as an objective variable, a regression equation representing the relationship between the R component value and the G component value is calculated, and a conversion equation corresponding to the regression equation is calculated for the R component value at the target pixel position. By using this, the G component value at the target pixel position is obtained.

  FIG. 18 schematically shows the pixel position of the pixel signal read from the frame memory 5 and its component value in order to interpolate the G component value at the position of the R pixel in the present embodiment.

  As illustrated, the read control circuit 6 causes the R component value R44 of the target pixel R44 and the R of the pixels R24 and R64 of the same color as the target pixel to be in the vicinity of the target pixel R44, aligned in the vertical direction of the target pixel R44. G component values G14 of the pixels G14, G34, G54, and G74 that are in the vicinity of the component values R24 and R64 and the pixel of interest R44, are arranged in the vertical direction of the pixel of interest R44, and are insufficient colors (color components to be obtained by interpolation), G34, G54, G74, in the vicinity of the target pixel R44, aligned in the horizontal direction with the target pixel, in the vicinity of the R component values R42, R46 of the pixels R42, R46 of the same color as the target pixel, and the target pixel R44. The G component values G41, G43, G45, and G47 of the pixels G41, G43, G45, and G47 of the insufficiently colored pixels arranged in the horizontal direction are output from the frame memory 5, Discrimination circuit 24, the coordinate data calculating circuit 7, interpolation calculation circuit 11, is supplied to the interpolation circuit 12.

  Of the signals supplied from the frame memory 5, the correlation determination circuit 24 is a pixel in the vicinity of the target pixel, for example, an adjacent pixel and is aligned in the vertical direction with respect to the target pixel, and the horizontal direction to the target pixel. Using the G component values G34, G54, G43, and G45 of the pixels G43 and G45 arranged in the same manner, values dV and dH are calculated by the following equations. These values indicate differences in component values between neighboring pixels.

dV = | G34-G54 |
dH = | G43-G45 |

  If the vertical correlation is strong, dV <dH, and if the horizontal correlation is strong, dV> dH. If dV <dH, it is determined that the vertical correlation is strong. Otherwise, the horizontal correlation is determined. Is output to the coordinate data calculation circuit 7, the interpolation calculation circuit 11, and the interpolation circuit 12.

  The coordinate data calculation circuit 7 operates as follows according to the correlation direction determination signal supplied from the correlation determination circuit 24.

  When the correlation direction determination signal in the correlation determination circuit 24 indicates a determination result that the correlation in the vertical direction is strong, the coordinate data calculation circuit 7 has an R component value R44 of the target pixel R44 and a vicinity of the target pixel R44. The R component values R24 and R64 of the pixels R24 and R64 of the same color as the target pixel, which are aligned in the vertical direction with the target pixel R44, are also in the vicinity of the target pixel R44 and are aligned in the vertical direction with the target pixel R44. From the G component values G14, G34, G54, and G74 of the insufficiently colored pixels G14, G34, G54, and G74, two-dimensional coordinate data (xi, yi) and (i = 1 to 3) represented by the following formulas are obtained. It is determined and output to the constant calculation circuit 9.

x1 = R24
y1 = (G14 + G34) / 2
x2 = R44
y2 = (G34 + G54) / 2
x3 = R64
y3 = (G54 + G74) / 2

  As can be seen from the above equation, in the above example, as the X coordinate value xi, the value of the R component (known color component) at each pixel position obtained from the image sensor 2 is used as it is, and the Y coordinate value yi. For example, an average of G components (insufficient color components) of pixels located around each pixel position, for example, above and below is obtained.

  On the other hand, when the correlation direction determination signal from the correlation determination circuit 24 indicates the determination result that the correlation in the horizontal direction is strong, the coordinate data calculation circuit 7 determines the R component value R44 of the target pixel R44 and the target pixel R44. R component values R42 and R46 of pixels R42 and R46 of the same color as the target pixel, which are adjacent to the target pixel R44, and are adjacent to the target pixel R44 in the horizontal direction. The two-dimensional coordinate data (xi, yi), (i = 1 to 1) represented by the following equation from the G component values G41, G43, G45, G47 of the insufficiently colored pixels G41, G43, G45, G47 3) is determined and output to the constant calculation circuit 9.

x1 = R42
y1 = (G41 + G43) / 2
x2 = R44
y2 = (G43 + G45) / 2
x3 = R46
y3 = (G45 + G47) / 2

  As can be seen from the above equation, in the above example, as the X coordinate value xi, the value of the R component (known color component) at each pixel position obtained from the image sensor 2 is used as it is, and the Y coordinate value yi. For example, an average of G components (insufficient color components) of pixels located around each pixel position, for example, the left side and the right side is obtained.

  The constant calculation circuit 9 uses the supplied two-dimensional coordinate data (xi, yi), (i = 1 to 3) and uses the above equations (1), (2), (3) described in the first embodiment. The values of the slopes a and y intercepts b of the regression line obtained as a result are output to the interpolation operation circuit 11 and the correlation coefficient k is output to the synthesis circuit 13.

  The interpolation calculation circuit 11 uses the G component value 11OUT at the position of the target pixel R44 as the R component value R44 of the target pixel R44 among the signals sent from the frame memory 5 and the regression sent from the constant calculation circuit 9. The calculation is performed according to the following equation using the line inclination a and the y-intercept b and output.

  11OUT = a × R44 + b

  However, as in the first embodiment, the two-dimensional coordinate data (xi, yi), (i = 1 to 3) are converted into the above [Condition 1], [Condition 2], [Condition] described in the first embodiment. 3] and [Condition 4], when the condition determination circuit 8 determines that the condition is satisfied, the operations of the constant calculation circuit 9 and the interpolation calculation circuit 11 are as follows.

The constant calculation circuit 9 outputs “1” as the value of the correlation coefficient k to the synthesis circuit 13 and instructs the interpolation calculation circuit 11 to perform linear interpolation without using a regression equation. put out.

  In the interpolation calculation circuit 11, linear interpolation according to the following equation using the G component value 11 OUT at the position of the target pixel R 44 as a part of the signal sent from the frame memory 5 (the signal of the pixel in the vicinity of the target pixel R 44). To calculate and output.

11OUT = (G34 + G54) / 2 when the vertical correlation is strong
11OUT = (G43 + G45) / 2 when the horizontal correlation is strong

  On the other hand, the interpolation circuit 12 calculates and outputs the G component value 12OUT at the position of the target pixel R44 as follows according to the determination result supplied from the correlation determination circuit 24. First, when the correlation direction determination signal from the correlation determination circuit 24 indicates a determination result that the correlation in the vertical direction is strong, the interpolation circuit 12 determines that the R component value R44 of the target pixel R44 and the vicinity of the target pixel R44. The following equation using the R component values R24 and R64 of the pixels R24 and R64 aligned in the vertical direction and the G component values G34 and G54 of the pixels G34 and G54 aligned in the vertical direction in the vicinity of the target pixel R44 The G component value 12OUT at the position of the pixel R44 is calculated and output.

  On the other hand, when the correlation direction determination signal from the correlation determination circuit 24 indicates the determination result that the correlation in the horizontal direction is strong, the interpolation circuit 12 determines that the R component value R44 of the target pixel R44 and the vicinity of the target pixel R44. The target pixel R44 is expressed by the following equation using the R component values R42 and R46 of the pixels R42 and R46 arranged in the horizontal direction and the G component values G43 and G45 of the pixels G43 and G45 arranged in the horizontal direction near the target pixel R44. The G component value 12OUT at the position is calculated and output.

  The interpolation by the interpolation circuit 12 as described above is performed on the assumption that there is a similar relationship in the state of change of each color component, and the same idea as described with reference to FIGS. 9 and 10. The interpolation value is determined by.

  In the synthesis circuit 13, the interpolation value 11 OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12 OUT supplied from the interpolation circuit 12 are weighted according to the absolute value of the correlation coefficient k supplied from the constant calculation circuit 9. The result is synthesized by addition, and the result is output as a synthesized interpolation value 13OUT. This weighted addition is performed so that the proportion of 11OUT is larger than 12OUT if the absolute value of the correlation coefficient k is close to "1", and from 11OUT if the absolute value of the correlation coefficient k is close to "0". This is done so that the proportion occupied by 12OUT increases.

  The interpolation value 13OUT calculated as described above is written in the frame memory 5 as the G component value at the position of the target pixel R44.

  The above interpolation processing is performed on the G component values at the positions of all R pixels.

  The above is a method of interpolating the G component value at the position of the R pixel, but the interpolation of the G component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  Next, an interpolation method for the R component value and the B component value at the position of the G pixel will be described. When the G pixel is the center, the arrangement of the R, G, and B pixels is different between the case where the pixel of interest is in the odd row (a) and the case where the pixel of interest is in the even row (b), as shown in FIG. . First, the interpolation of the R component value and the B component value at the position of the G pixel in the odd row will be described.

  First, interpolation of the R component value and the B component value at the position of the G pixel in the odd-numbered row will be described with reference to FIG. As shown in FIG. 8, the pixel having the R component centered on the target pixel G pixel is located in the horizontal direction (left-right direction) and does not exist in the vertical direction (up-down direction).

  First, the G component value G44 of the target pixel G44 and the G component values G24 and G64 of the pixels G24 and G64 that are in the vicinity of the target pixel G44, are aligned in the vertical direction with the target pixel G44, and have the same color as the target pixel. Near the target pixel G44, aligned in the vertical direction, in the vicinity of the B pixel values B14, B34, B54, and B74 of the pixels B14, B34, B54, and B74 of insufficient colors and the target pixel G44. R of pixel colors R41, R43, R45, and R47 of pixels R41, R43, R45, and R47 that are in the vicinity of the G component values G42 and G46 of the pixels G42 and G46 of the same color as the pixel of interest G44 and in the vicinity of the pixel of interest G44. The component values R41, R43, R45, and R47 are output from the frame memory by the read control circuit 6, and the correlation determination circuit 24, the coordinate data calculation circuit 7, and the interpolation performance are output. Circuit 11, is supplied to the interpolation circuit 12.

  As can be seen from FIG. 8, there is no information regarding the R component value in the vertical direction with respect to the target pixel G44, and there is no information regarding the B component value in the horizontal direction. That is, when interpolating the R component value and the B component value at the G pixel position, the color signal in either the vertical direction or the horizontal direction is used regardless of whether the correlation is strong in the vertical direction or the horizontal direction. Only interpolation can be performed. Accordingly, when interpolating the R component value and the B component value at the position of the G pixel, the correlation determination circuit 24 does not perform the correlation direction determination.

The data output from the frame memory 5 is the same as the data output from the frame memory 5 when the R component value and the B component value at the G pixel position are interpolated in the first embodiment. The operations of the coordinate data calculation circuit 7, the constant calculation circuit 9, the interpolation calculation circuit 11, the interpolation circuit 12, and the synthesis circuit 13 in the interpolation of the R component value and the B component value at the position are the same as those in the first embodiment. . Therefore, detailed description thereof is omitted.
As in the first embodiment, when any one of the conditions 1 to 4 is satisfied, linear interpolation based on surrounding pixels is performed instead of interpolation based on regression analysis.

  In the above, the interpolation of the R component value and the B component value at the position of the G pixel in the odd-numbered row has been described, but the interpolation of the R component value and the B component value at the position of the G pixel in the even-numbered row is similarly performed. However, among the descriptions regarding the interpolation at the positions of the odd-numbered G pixels, it is necessary to exchange “R” and “B” or “vertical direction” and “horizontal direction”. As can be seen from FIG. 7, this is because there is an R component value in the vertical direction with respect to the target pixel G, no B component value, and there is a B component value and no R component value in the horizontal direction. It is.

  Next, interpolation of the B component value at the position of the R pixel will be described. The R pixel has a G component obtained by the previous interpolation, and this G component value is used for interpolation of the B component value at the position of the R pixel described below.

  FIG. 19 shows a signal read from the frame memory 5 in order to interpolate the B component value at the position of the R pixel having the G component. As shown in the figure, there are pixels having the G component not only at the position of the R pixel R44 which is the target pixel but also around it. In FIG. 19, the B component value at the position of the R pixel is read from the frame memory 5 at the time of interpolation, and is supplied to the coordinate data calculation circuit 7, the correlation determination circuit 24, the interpolation calculation circuit 11, and the interpolation circuit 12. The pixel position of the pixel signal is schematically shown.

  First, the readout control circuit 6 causes the G component value g44 at the position of the target pixel R44 (lower case g indicates that the G component value was generated by previous interpolation) and the vicinity of the target pixel R44. G components values G14, G34, G54, and G74 of pixels G14, G34, G54, and G74 that are aligned in the vertical direction with the pixel of interest R44 and have the same G component as the pixel of interest R44, and in the vicinity of the pixel of interest The B component values b14, b34, b54, and b74 of the insufficiently colored pixels aligned in the vertical direction with the pixel are in the vicinity of the target pixel R44, are aligned in the horizontal direction with the target pixel, and have the same G component as the target pixel R44. G component values G41, G43, G45, and G47 of the pixels G41, G43, G45, and G47, and pixels of insufficient color that are in the vicinity of the target pixel and are aligned with the target pixel in the horizontal direction B component values b41, b43, b45, b47 is output from the frame memory 5, the correlation determination circuit 24, the coordinate data calculating circuit 7, interpolation calculation circuit 11, is supplied to the interpolation circuit 12.

  The correlation determination circuit 24 uses the G component values G34, G43, G45, and G54 of the pixels G34, G43, G45, and G54 among the signals supplied from the frame memory 5, and uses the G component at the positions of the R pixel and the B pixel. Similar to the value interpolation, it is determined whether the image has a strong correlation in the vertical direction or the horizontal direction, and a correlation direction determination signal indicating the determination result is used as the coordinate data calculation circuit 7 and the interpolation calculation circuit 11. To the interpolation circuit 12.

  The coordinate data calculation circuit 7 operates as follows according to the correlation direction determination signal supplied from the correlation determination circuit 24.

  When the correlation direction determination signal from the correlation determination circuit 24 indicates a determination result that the correlation in the vertical direction is strong, the coordinate data calculation circuit 7 is in the vicinity of the target pixel R44 and in the vertical direction. G component value G14 and B component value b14 of pixel G14, G component value G34 and B component value b34 of pixel G34, G component value G54 and B component value b54 of pixel G54, G component value G74 of pixel G74 And the two-dimensional coordinate data (xi, yi) and (i = 1 to 4) represented by the following formula are determined from the B component value b74 and output to the constant calculation circuit 9.

x1 = G14
y1 = b14
x2 = G34
y2 = b34
x3 = G54
y3 = b54
x4 = G74
y4 = b74

  As can be seen from the above formula, as the X coordinate value xi, the G component at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel obtained by previous interpolation is used. The B component at the position is used as it is.

  On the other hand, when the correlation direction determination signal from the correlation determination circuit 24 indicates a determination result that the correlation in the horizontal direction is strong, the coordinate data calculation circuit 7 is in the vicinity of the target pixel R44, and the target pixel and the horizontal direction G component value G41 and B component value b41 of pixel G41, G component value G43 and B component value b43 of pixel G43, G component value G45 and B component value b45 of pixel G45, and G component value G47 of pixel G47. And the two-dimensional coordinate data (xi, yi) and (i = 1 to 4) represented by the following formula are determined from the B component value b47 and output to the constant calculation circuit 9.

x1 = G41
y1 = b41
x2 = G43
y2 = b43
x3 = G45
y3 = b45
x4 = G47
y4 = b47

  As can be seen from the above formula, as the X coordinate value xi, the G component at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel obtained by previous interpolation is used. The B component at the position is used as it is.

  The constant calculation circuit 9 uses the supplied two-dimensional coordinate data (xi, yi), (i = 1 to 4) to calculate the above equations (1), (2), (3), and the result The obtained values of the slope a and y intercept b of the regression line are output to the interpolation calculation circuit 11 and the correlation coefficient k is output to the synthesis circuit 13.

  The interpolation calculation circuit 11 uses the B component value 11OUT at the position of the pixel of interest R44 as the G component value g44 of the pixel of interest R44 among the signals sent from the frame memory 5 and the regression sent from the constant calculation circuit 9. The calculation is performed according to the following equation using the line inclination a and y intercept b, and output.

  11OUT = a × g44 + b

  However, as in the first embodiment, the two-dimensional coordinate data (xi, yi), (i = 1 to 4) are converted into the above [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. When the condition determination circuit 8 determines that any of the above conditions is satisfied, the operations of the interpolation calculation circuit 11 and the constant calculation circuit 9 are as follows.

The constant calculation circuit 9 outputs “1” as the correlation coefficient k to the synthesis circuit 13 and instructs the interpolation calculation circuit 11 to perform linear interpolation without using a regression equation.

  In the interpolation calculation circuit 11, linear interpolation according to the following equation using the B component value 11 OUT at the position of the target pixel R 44 as a part of the signal sent from the frame memory 5 (the signal of the pixel in the vicinity of the target pixel R 44). To calculate and output.

11OUT = (b34 + b54) / 2 when the correlation in the vertical direction is strong
11OUT = (b43 + b45) / 2 when the horizontal correlation is strong

  On the other hand, the interpolation circuit 12 calculates and outputs the G component value 12OUT at the position of the target pixel R44 as follows according to the determination result supplied from the correlation determination circuit 24. First, when the correlation direction determination signal from the correlation determination circuit 24 indicates a determination result that the correlation in the vertical direction is strong, the interpolation circuit 12 determines that the G component value g44 of the target pixel R44 and the vicinity of the target pixel R44. The following formula using the G component values G34 and G54 of the pixels G34 and G54 arranged in the vertical direction and the B component values b34 and b54 of the pixels B34 and B54 arranged in the vertical direction near the target pixel R44 The B component value 12OUT at the position of the pixel R44 is calculated and output.

  On the other hand, when the correlation direction determination signal from the correlation determination circuit 24 indicates the determination result that the correlation in the horizontal direction is strong, the interpolation circuit 12 determines that the G component value g44 of the target pixel R44 and the vicinity of the target pixel R44. The B component value 12OUT at the position of the target pixel R44 is calculated and output by the following equation using the G component values G43 and G45 and the B component values b43 and b45 of the pixels G43 and G45 arranged in the horizontal direction.

  In the synthesis circuit 13, the interpolation value 11 OUT supplied from the interpolation calculation circuit 11 and the interpolation value 12 OUT supplied from the interpolation circuit 12 are weighted according to the absolute value of the correlation coefficient k supplied from the constant calculation circuit 9. The result is synthesized by addition, and the result is output as a synthesized interpolation value 13OUT. This weighted addition is performed so that the proportion of 11OUT is larger than 12OUT if the absolute value of the correlation coefficient k is close to "1", and from 11OUT if the absolute value of the correlation coefficient k is close to "0". This is done so that the proportion occupied by 12OUT increases.

  The interpolation value 13OUT calculated as described above is written in the frame memory 5 as the B component value at the position of the target pixel R44.

  The above interpolation processing is performed on the B component values at the positions of all R pixels.

  The above is a method of interpolating the B component value at the position of the R pixel, but the interpolation of the R component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  As described above, the interpolation processing is completed, all insufficient color component values are given to all pixels, the component values of R, G, and B colors are aligned, and a signal corresponding to one still image is accumulated in the frame memory 5. It will be done. When a signal corresponding to one still image is accumulated in the frame memory 5, the read control circuit 6 reads a signal 5 OUT corresponding to one still image from the frame memory 5 and outputs it to the outside of the interpolation processing circuit 4. .

The effects of the interpolation described above are as follows.
First, the interpolation performed by the interpolation calculation circuit 11 is the same as the interpolation performed by the interpolation calculation circuit 11 in the first embodiment, and the interpolation performed by the interpolation circuit 12 is performed by the interpolation circuit 12 in the first embodiment. The value calculated by the constant calculation circuit 9 is the same as the value calculated by the constant calculation circuit 9 in the first embodiment, and the operation in the synthesis circuit 13 is the synthesis in the first embodiment. It is equivalent to the circuit 13. Therefore, this embodiment has the same effect as that of the first embodiment.

  In addition, in the second embodiment, the correlation determination circuit 24 performs correlation determination, and the signals used for the interpolation processing are limited to those having a strong correlation in the image. That is, since interpolation is performed using a signal having a strong correlation, interpolation processing with higher accuracy can be performed as compared with the first embodiment.

Note that the synthesis circuit 13 may select one of the interpolation value 11OUT from the interpolation calculation circuit 11 and the interpolation value 12OUT from the interpolation circuit 12. For example, if a threshold value is set and the absolute value of the correlation coefficient k calculated by the constant calculation circuit 9 is larger than this threshold value, the interpolation value 11OUT from the interpolation calculation circuit 11 is output and calculated by the constant calculation circuit 9 If the absolute value of the correlation coefficient k is less than or equal to this threshold value, a method of outputting the interpolation value 12OUT from the interpolation circuit 12 is conceivable.
Note that the operation of selecting one of them corresponds to setting the weighting coefficient to “0” or “1”.

  Various configurations of the full-color image capturing apparatus other than FIG. 1 are conceivable. For example, a storage medium for storing a captured image is mounted, or a circuit that performs image compression processing is combined.

Embodiment 3 FIG.
Next, an example in which a function corresponding to the interpolation processing described in the second embodiment is realized by software will be described.
The configuration of the interpolation processing circuit 34 used in the third embodiment is as shown in FIG. 20. Among the components of the interpolation processing circuit 4 in FIG. 17, the correlation determination circuit 24, the coordinate data calculation circuit 7, and the constant calculation circuit. 9. A processor 35 is provided instead of the interpolation calculation circuit 11, the interpolation circuit 12, and the synthesis circuit 13. The processor 35 operates as shown in the flowcharts of FIGS.

The target pixel group is arranged as shown in FIG. 2 as in the first embodiment. The interpolation performed by the interpolation processing circuit 34 includes:
(P1) Processing for obtaining G component values at the positions of the R pixel and the B pixel;
(P2) Processing for obtaining the R component value and the B component value at the position of the G pixel,
(P3) includes processing for obtaining the B component value at the position of the R pixel, and (P4) processing for obtaining the R component value at the position of the B pixel, and is performed, for example, in the order described above.

  First, a G component value interpolation method at the positions of the R pixel and the B pixel will be described. First, interpolation of the G component value at the position of the R pixel will be described with reference to FIGS. 21 and 18.

  FIG. 21 is a flowchart showing the operation of the processor 35 at the time of interpolation of the G component value at the position of the R pixel, and FIG. 18 is a pixel used at the time of interpolation of the G component value at the position of the R pixel and its color component value. Indicates. The R pixel R44 in FIG. 18 becomes the target pixel.

  First, in step ST21 in FIG. 21, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the target pixel. For this determination, using the G component values G34, G43, G45, and G54 of the G pixel in the vicinity of the target pixel R44, for example, the G pixel that is aligned with the target pixel in the horizontal and vertical directions and is adjacent to the target pixel, dV and dH are calculated.

dV = | G34-G54 |
dH = | G43-G45 |

  When the correlation in the vertical direction is strong, dV <dH and when the correlation in the horizontal direction is strong, dV> dH. Therefore, if dV <dH, it is determined that the correlation in the vertical direction is strong. Otherwise, the correlation in the horizontal direction is determined. Is determined to be strong, and a process to be performed next is selected according to the determination result.

  First, processing when the correlation in the vertical direction is strong will be described.

  In step ST22A, the R component value R44 of the target pixel R44 and the R component values R24 and R64 of the pixels R24 and R64 of the same color as the target pixel that are adjacent to the target pixel R44 and are aligned in the vertical direction are From the G component values G14, G34, G54, and G74 of the insufficiently colored pixels G14, G34, G54, and G74 that are also in the vicinity of the target pixel R44 and aligned in the vertical direction, Two-dimensional coordinate data (xi, yi), (i = 1 to 3) are determined.

x1 = R24
y1 = (G14 + G34) / 2
x2 = R44
y2 = (G34 + G54) / 2
x3 = R64
y3 = (G54 + G74) / 2

  As can be seen from the above formula, as the X coordinate value xi, the R component of each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, for example, an upper part around each pixel position. And the average of the G component of the lower pixel is obtained.

  Next, in step ST23A, the two-dimensional coordinate data (xi, yi), (i = 1 to 3) is set to any one of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. It is checked whether it is applicable. If the two-dimensional coordinate data (xi, yi), (i = 1 to 3) does not correspond to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], then step ST24A1 , ST25A1 and ST26A1 are performed, and if any of them is satisfied, the processes of steps ST25A2 and ST26A2 are performed.

  Next, processing in steps ST24A1, ST25A1, and ST26A1 will be described.

  In step ST24A1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above formulas (1) and (2) are calculated, and constants a and b (regression line slope a , The value of y intercept b).

  In step ST25A1, the G component value OUT1 at the position of the target pixel R44 is obtained by the following equation using the constants a and b and the R component value R44 of the target pixel R44.

  OUT1 = a × R44 + b

  In step ST26A1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above equation (3) is calculated, and the correlation coefficient k is obtained.

  Next, processing in steps ST25A2 and ST26A2 will be described.

  In step ST25A2, the G component value OUT1 at the position of the target pixel R44 is obtained by linear interpolation using the following formula using pixels in the vicinity of the target pixel.

  OUT1 = (G34 + G54) / 2

  Next, in step ST26A2, the value of the correlation coefficient is determined to be “1”.

  After the processing of step ST24A1, ST25A1, ST26A1, or step ST25A2, ST26A2 is performed according to the determination result in step ST23A, the G component value OUT2 at the position of the target pixel R44 is obtained by the following calculation formula in step ST27A. .

The calculation in step ST27A is interpolation assuming a similar relationship, and corresponds to the interpolation processing performed by the interpolation circuit 12 in the second embodiment.
On the other hand, the processing of steps ST24A1 to ST26A1, ST25A2, and ST25A2 corresponds to the interpolation processing performed by the interpolation arithmetic circuit 11 or the like in the second embodiment.

  The above is the processing when it is determined in step ST21 that the correlation in the vertical direction is strong. The processing when it is determined in step ST21 that the horizontal correlation is strong will be described.

  In step ST22B, the R component value R44 of the target pixel R44 and the R component values R42 and R46 of the pixels R42 and R46 of the same color as the target pixel that are in the vicinity of the target pixel and are aligned in the horizontal direction with the target pixel From the G component values G41, G43, G45, and G47 of the insufficiently colored pixels G41, G43, G45, and G47 that are also in the vicinity of the target pixel and are aligned in the horizontal direction, the following expression is used. Two-dimensional coordinate data (xi, yi), (i = 1 to 3) are determined.

x1 = R42
y1 = (G41 + G43) / 2
x2 = R44
y2 = (G43 + G45) / 2
x3 = R46
y3 = (G45 + G47) / 2

  As can be seen from the above formula, as the X coordinate value xi, the R component of each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, for example, an upper part around each pixel position. And the average of the G component of the lower pixel is obtained.

  Next, in step ST23B, the two-dimensional coordinate data (xi, yi), (i = 1 to 3) is set to any one of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. It is checked whether it is applicable. If the two-dimensional coordinate data (xi, yi), (i = 1 to 3) does not correspond to any of [Condition 1], [Condition 2], [Condition 3] and [Condition 4], then step ST24B1 , ST25B1 and ST26B1 are performed, and if any of them is satisfied, the processes of steps ST25B2 and ST26B2 are performed next.

  Next, processing in steps ST24B1, ST25B1, and ST26B1 will be described.

  In step ST24B1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above equations (1) and (2) are calculated, and constants a and b (regression line slope a, value of y intercept b) is obtained.

  In step ST25B1, the G component value OUT1 at the position of the target pixel R44 is obtained by the following equation using the constants a and b and the R pixel value R44 of the target pixel R44.

  OUT1 = a × R44 + b

  In step ST26B1, the above equation (3) is calculated using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), and the correlation coefficient k is obtained.

  Next, processing in steps ST25B2 and ST26B2 will be described.

  In step ST25B2, the G component value OUT1 at the position of the target pixel R44 is obtained by linear interpolation using the following formula using pixels near the target pixel.

  OUT1 = (G43 + G45) / 2

  Next, in step ST26B2, the value of the correlation coefficient is determined to be “1”.

  After the processing of step ST24B1, ST25B1, ST26B1 or step ST25B2, ST26B2 is performed according to the determination result in step ST23B, the G component value OUT2 at the position of the target pixel R44 is obtained by the following calculation formula in step ST27B. .

The calculation in step ST27B is interpolation assuming a similar relationship, and corresponds to the interpolation processing performed by the interpolation circuit 12 in the second embodiment.
On the other hand, the processing of steps ST24B1 to ST26B1, ST25B2, and ST26B2 corresponds to the interpolation processing performed by the interpolation arithmetic circuit 11 or the like in the second embodiment.

After the above processing, in step ST28, synthesis by weighted addition of OUT1 and OUT2 is performed according to the value of the correlation coefficient k.
The composition is such that if the absolute value of the correlation coefficient k is close to “1”, the proportion of the value of OUT1 increases. If the absolute value of the correlation coefficient k is close to “0”, the proportion of the value of OUT2 Is done to increase.

  In step ST29, the insufficient color component value (interpolation value) OUT interpolated as described above is written in the frame memory 5 as the synthesized G component value at the position of the target pixel R44.

  The above interpolation processing is performed on the G component values at the positions of all R pixels.

  The above is a method of interpolating the G component value at the position of the R pixel, but the interpolation of the G component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  Next, a method for interpolating the R component value and the B component value at the position of the G pixel will be described.

  When the G pixel is the center, the arrangement of the R, G, and B pixels is different between the case where the pixel of interest is in the odd row (a) and the case where the pixel of interest is in the even row (b), as shown in FIG. . First, the interpolation of the R component value and the B component value at the position of the G pixel in the odd row will be described with reference to FIGS.

  FIG. 22 is a flowchart showing the operation of the processor 35 at the time of interpolation of the R component value and the B component value at the position of the G pixel. The G pixel G44 in FIG.

  First, the interpolation of R component values at the positions of odd-numbered G pixels will be described.

  First, in step ST31, the G component value of the target pixel G44 and the G component values G42, G44, and G46 of the G pixels G42, G44, and G46 in the vicinity of the target pixel G44 and the R component values of the R pixels R41, R43, R45, and R47. Two-dimensional coordinate data (xi, yi) and (i = 1 to 3) are determined by the following formula using R41, R43, R45, and R47.

x1 = G42
y1 = (R41 + R43) / 2
x2 = G44
y2 = (R43 + R45) / 2
x3 = G46
y3 = (R45 + R47) / 2

  As can be seen from the above formula, as the X coordinate value xi, the G component of each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, for example, the left around each pixel position An average of the R components of the right and right pixels is obtained.

  Next, in step ST32, the two-dimensional coordinate data (xi, yi), (i = 1 to 3) corresponds to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. Is checked. If the two-dimensional coordinate data (xi, yi), (i = 1 to 3) does not correspond to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], steps ST33A and ST34A The process of ST35A is performed, and in the case of any of these, the processes of steps ST34B and ST35B are performed.

  Next, the processing of steps ST33A, ST34A, ST35A will be described.

  In step ST33A, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above formulas (1) and (2) are calculated, and constants a and b (regression line slope a , The value of y intercept b).

  In step ST34A, the R component value OUT1 at the position of the target pixel G44 is obtained by the following equation using the constants a and b and the G component value G44 of the target pixel G44.

  OUT1 = a × G44 + b

  In step ST35A, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above equation (3) is calculated to obtain the correlation coefficient k.

  Next, processing in steps ST34B and ST35B will be described.

  In step ST34B, the R component value OUT1 at the position of the target pixel G44 is obtained by linear interpolation according to the following equation using pixels in the vicinity of the target pixel.

OUT1 = (R43 + R45) / 2

  Next, in step ST35B, the value of correlation coefficient k is determined to be “1”.

  After the processing of steps ST33A, ST34A, ST35A or steps ST34B, ST35B is performed according to the determination result in step ST32, the R component value OUT2 in the target pixel G44 is obtained by the following calculation formula in step ST36.

The calculation in step ST36 is an interpolation assuming a similar relationship, and corresponds to the interpolation processing performed by the interpolation circuit 12 in the second embodiment.
On the other hand, the processing of steps ST33A to ST35A, ST34B, ST35B corresponds to the interpolation processing performed by the interpolation arithmetic circuit 11 or the like in the second embodiment.

  After the above processing, in step ST37, synthesis is performed by weighted addition of OUT1 and OUT2 according to the value of the correlation coefficient k. The composition is such that if the absolute value of the correlation coefficient k is close to “1”, the proportion of the value of OUT1 increases. If the absolute value of the correlation coefficient k is close to “0”, the proportion of the value of OUT2 Is done to increase.

  In step ST38, the insufficient color component value (interpolation value) OUT interpolated as described above is written in the frame memory 5 as the synthesized R component value at the position of the target pixel G44.

  The above interpolation processing is performed on the R component values at the positions of all the odd-numbered G pixels.

  Next, the interpolation of the B component value at the position of the G pixel in the odd row will be described.

  First, in step ST31, the G component value of the target pixel G44 and the G component values G24, G44, and G64 of the G pixels G24, G44, and G64 near the target pixel G44 and the B components of the B pixels B14, B34, B54, and B74 Using the values B14, B34, B54, and B74, two-dimensional coordinate data (xi, yi) and (i = 1 to 3) are determined by the following formula.

x1 = G24
y1 = (B14 + B34) / 2
x2 = G44
y2 = (B34 + B54) / 2
x3 = G64
y3 = (B54 + B74) / 2

  As can be seen from the above formula, as the X coordinate value xi, the G component at the pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, the pixels above and below each pixel position are used. The average of the B component is obtained.

  Next, in step ST32, the two-dimensional coordinate data (xi, yi), (i = 1 to 3) corresponds to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. Is checked. If the two-dimensional coordinate data (xi, yi), (i = 1 to 3) does not correspond to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], steps ST33A and ST34A The process of ST35A is performed, and in the case of any of these, the processes of steps ST34B and ST35B are performed.

  Next, the processing of steps ST33A, ST34A, ST35A will be described.

  In step ST33A, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above formulas (1) and (2) are calculated, and constants a and b (regression line slope a , The value of y intercept b).

  In step ST34A, the B component value OUT1 at the position of the target pixel G44 is obtained by the following equation using the constants a and b and the G component value G44 of the target pixel G44.

  OUT1 = a × G44 + b

  In step ST35A, using the two-dimensional coordinate data (xi, yi), (i = 1 to 3), the above equation (3) is calculated to obtain the correlation coefficient k.

  Next, processing in steps ST34B and ST35B will be described.

  In step ST34B, the B component value OUT1 at the position of the target pixel G44 is obtained by linear interpolation according to the following equation using pixels near the target pixel.

  OUT1 = (B34 + B54) / 2

  Next, in step ST35B, the value of correlation coefficient k is determined to be “1”.

  After the processing of steps ST33A, ST34A, ST35A or steps ST34B, ST35B is performed according to the determination result at step ST32, the B component value OUT2 at the target pixel G44 is obtained by the following calculation formula at step ST36.

The calculation in step ST36 is an interpolation assuming a similar relationship, and corresponds to the interpolation processing performed by the interpolation circuit 12 in the second embodiment.
On the other hand, the processing of steps ST33A to ST35A, ST34B, ST35B corresponds to the interpolation processing performed by the interpolation calculation circuit 11 or the like in the second embodiment.

  After the above processing, in step ST37, combining by weighted addition of OUT1 and OUT2 is performed according to the value of the correlation coefficient k. The composition is such that if the absolute value of the correlation coefficient k is close to “1”, the proportion of the value of OUT1 increases. If the absolute value of the correlation coefficient k is close to “0”, the proportion of the value of OUT2 Is done to increase.

  In step ST38, the insufficient color component value (interpolation value) OUT interpolated as described above is written in the frame memory 5 as the synthesized B component value at the position of the target pixel G44.

  The above interpolation processing is performed on the B component values at the positions of all odd-numbered G pixels.

  In the above, the interpolation of the R component value and the B component value at the position of the G pixel in the odd-numbered row has been described, but the interpolation of the R component value and the B component value at the position of the G pixel in the even-numbered row is similarly performed. However, among the descriptions regarding the interpolation at the positions of the odd-numbered G pixels, it is necessary to exchange “R” and “B” or “vertical direction” and “horizontal direction”. As can be seen from FIG. 7, this is because there is an R component value in the vertical direction with respect to the target pixel G, no B component value, and there is a B component value and no R component value in the horizontal direction. It is.

  Next, a method of interpolating the B component value at the R pixel position will be described with reference to FIGS.

  FIG. 23 is a flowchart showing the operation of the processor 35 at the time of interpolation of the B component value at the position of the R pixel. The R pixel R44 in FIG. 19 becomes the target pixel. The R pixel has a G component obtained by previous interpolation, and this G component value is used for interpolation of the B component value at the position of the R pixel described below.

  First, in step ST41 in FIG. 23, it is determined whether the correlation is strong in the horizontal direction or the vertical direction around the target pixel. For this determination, the following values dV and dH are calculated using G component values G34, G43, G45, and G54 of G pixels near the target pixel R44, for example, G pixels adjacent to the target pixel.

dV = | G34-G54 |
dH = | G43-G45 |

  When the correlation in the vertical direction is strong, dV <dH and when the correlation in the horizontal direction is strong, dV> dH. Therefore, if dV <dH, it is determined that the correlation in the vertical direction is strong. Otherwise, the correlation in the horizontal direction is determined. Is determined to be strong, and a process to be performed next is selected according to the determination result.

  First, processing when the correlation in the vertical direction is strong will be described.

  In step ST42A, the G component value G14 and the B component value b14 of the pixel G14 that are in the vicinity of the target pixel R44 and are arranged in the vertical direction of the target pixel R44 and have the same color component as the target pixel, and the G component value of the pixel G34 G34 and B component values b34, G54 G component values G54 and B component values b54, G74 G component values G74 and B component values b74, and two-dimensional coordinate data (xi, yi), (i = 1-4) are defined.

x1 = G14
y1 = b14
x2 = G34
y2 = b34
x3 = G54
y3 = b54
x4 = G74
y4 = b74

  As can be seen from the above formula, as the X coordinate value xi, the G component at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel obtained by previous interpolation is used. The B component at the position is used as it is.

  Next, in step ST43A, the two-dimensional coordinate data (xi, yi), (i = 1 to 4) corresponds to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. Is checked. If the two-dimensional coordinate data (xi, yi), (i = 1 to 4) does not correspond to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], steps ST44A1 and ST45A1 , ST46A1 is performed, and in any case, the processes of steps ST45A2 and ST46A2 are performed.

  Next, processing in steps ST44A1, ST45A1, and ST46A1 will be described.

  In step ST44A1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 4), the above formulas (1) and (2) are calculated, and constants a and b (regression line slope a , The value of y intercept b).

  In step ST45A1, the B component value OUT1 at the position of the target pixel R44 is obtained by the following formula using the constants a and b and the G component value g44 of the target pixel R44.

  OUT1 = a × g44 + b

  In step ST46A1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 4), the calculation of the above equation (3) is performed to obtain the correlation coefficient k.

  Next, processing in steps ST45A2 and ST46A2 will be described.

  In step ST45A2, the B component value OUT1 at the position of the target pixel R44 is obtained by linear interpolation using the following formula using pixels in the vicinity of the target pixel.

  OUT1 = (b34 + b54) / 2

  Next, in step ST46A2, the value of the correlation coefficient is determined to be “1”.

  After the processing of step ST44A1, ST45A1, ST46A1 or step ST45A2, ST46A2 is performed according to the determination result in step ST43A, the B component value OUT2 at the position of the target pixel R44 is calculated by the following calculation formula in step ST47A. The

  The calculation in step ST47A is an interpolation assuming a similar relationship, and corresponds to the interpolation process performed by the interpolation circuit 12 in the second embodiment. On the other hand, the processing of steps ST44A1 to ST46A1, ST45A2, and ST46A2 corresponds to the interpolation processing performed by the interpolation arithmetic circuit 11 or the like in the second embodiment.

  The above is the processing when it is determined in step ST41 that the correlation in the vertical direction is strong. The processing when it is determined in step ST41 that the horizontal correlation is strong will be described.

  In step ST42B, the G component value G41 and the B component value b41 of the pixel G41 that are in the vicinity of the target pixel and are arranged in the horizontal direction with the target pixel and that have the same color component as the target pixel, and the G component value G43 of the pixel G43. And B component value b43, G component value G45 and B component value b45 of pixel G45, G component value G47 and B component value b47 of pixel G47, two-dimensional coordinate data (xi, yi), ( i = 1 to 4) is determined.

x1 = G41
y1 = b41
x2 = G43
y2 = b43
x3 = G45
y3 = b45
x4 = G47
y4 = b47

  As can be seen from the above formula, as the X coordinate value xi, the G component at each pixel position obtained from the image sensor 2 is used as it is, and as the Y coordinate value yi, each pixel obtained by previous interpolation is used. The B component of the pixel at the position is used as it is.

  Next, in step ST43B, the two-dimensional coordinate data (xi, yi), (i = 1 to 4) is set to any one of [Condition 1], [Condition 2], [Condition 3], and [Condition 4]. It is checked whether it is applicable. If the two-dimensional coordinate data (xi, yi), (i = 1 to 4) does not correspond to any of [Condition 1], [Condition 2], [Condition 3], and [Condition 4], then step ST44B1 , ST45B1 and ST46B1 are performed, and if any of them is satisfied, the processes of steps ST45B2 and ST46B2 are performed.

  Next, processing in steps ST44B1, ST45B1, and ST46B1 will be described.

  In step ST44B1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 4), the above formulas (1) and (2) are calculated, and constants a and b (regression line slope a , The value of y intercept b).

  In step ST45B1, the B component value OUT1 at the position of the target pixel R44 is obtained by the following equation using the constants a and b and the G component value g44 of the target pixel R44.

  OUT1 = a × g44 + b

  In step ST46B1, using the two-dimensional coordinate data (xi, yi), (i = 1 to 4), the calculation of the above equation (3) is performed to obtain the correlation coefficient k.

  Next, processing in steps ST45B2 and ST46B2 will be described.

  In step ST45B2, the B component value OUT1 at the position of the target pixel R44 is obtained by linear interpolation using the following expression using pixels in the vicinity of the target pixel.

  OUT1 = (b43 + b45) / 2

  Next, in step ST46B2, the value of the correlation coefficient is determined to be “1”.

  After the processing of step ST44B1, ST45B1, ST46B1 or step ST45B2, ST46B2 is performed based on the determination result in step ST43B, in step ST47B, the B component value OUT2 at the position of the target pixel R44 is obtained by the following calculation formula.

  The calculation in step ST47B is interpolation assuming a similar relationship, and corresponds to the interpolation processing performed by the interpolation circuit 12 in the second embodiment. On the other hand, the processing of steps ST44B1 to ST46B1, ST45B2, and ST46B2 corresponds to the interpolation processing performed by the interpolation arithmetic circuit 11 or the like in the second embodiment.

  After the above processing, in step ST48, synthesis by weighted addition of OUT1 and OUT2 is performed according to the value of the correlation coefficient k. In the synthesis, if the absolute value of the correlation coefficient k is close to “1”, the proportion of the value of OUT1 increases, and if the absolute value of the correlation coefficient is close to 0, the proportion of the value of OUT2 increases. To be done.

  In step ST29, the insufficient color component value (interpolated value) OUT interpolated as described above is written in the frame memory 5 as the synthesized B component value at the position of the target pixel R44.

  The above interpolation processing is performed on the B component values at the positions of all R pixels.

  The above is a method of interpolating the B component value at the position of the R pixel, but the interpolation of the R component value at the position of the B pixel is similarly performed except that R and B are interchanged.

  As described above, the interpolation processing is completed, all insufficient color component values are given to all pixels, the component values of R, G, and B colors are aligned, and a signal corresponding to one full-color still image is received in the frame memory 5. Accumulated. When a signal corresponding to one still image is accumulated in the frame memory 5, the read control circuit 6 reads a signal 5 OUT corresponding to one still image from the frame memory 5 and outputs it to the outside of the interpolation processing circuit 34. .

  According to the interpolation method described above, the same processing as the interpolation processing circuit 4 in the pixel signal processing apparatus described in the second embodiment can be realized by software, and the same effect can be obtained.

  In steps ST28, ST37, and ST48, OUT1 and OUT2 are synthesized by weighted addition using the correlation coefficient. However, when a certain threshold is set and the absolute value of the correlation coefficient k is larger than this threshold, OUT1 is set. Otherwise, OUT2 may be selected. Note that the operation of selecting one of them corresponds to setting the weighting coefficient to 0 and 1.

  Although the case where the present invention is applied to interpolation of image data obtained from a Bayer array type image sensor has been described, the present invention is not limited to this.

  In the above-described embodiment, the two-dimensional image pickup device has three color filters of R, G, and B arranged in a Bayer shape. First to Nth types of photoelectric conversion elements having any one of the 1st to Nth (N = 3 in the above embodiment) spectral sensitivity characteristics are arranged on a two-dimensional plane. The present invention is applicable when an image sensor is provided. Further, the pixel signal (or color component value) used for interpolation is not limited to the signal obtained from the image sensor 2, and the pixel signal (color component value) obtained by interpolation is used for interpolation of other colors. May be. Therefore, in general terms, the pixel signal processing apparatus according to the present invention is arranged on a two-dimensional plane, and each color of a plurality of pixels each having any one of the first to Nth different color components. Based on the set of pixel signals representing the component values, the Kth (K is 1 to N excluding J) at the position of the pixel of interest having the Jth (J is one of 1 to N) color components. The pixel signal representing the value of any one of the color components is generated by interpolation. As the first interpolation means (31), the pixel signal indicating the value of the Jth color component of the pixel having the target pixel and the Jth color component in the vicinity of the target pixel, and the Kth in the vicinity of the target pixel. Based on the pixel signal representing the value of the Kth color component of the pixel having the color component, a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the target pixel Interpolation calculation using a regression equation that represents the relationship between the value of the Jth color component and the value of the Kth color component, including the regression equation calculation means (7 + 9) that calculates the constant and the constant calculated by the regression equation calculation means Alternatively, a circuit provided with an interpolation calculation circuit (11) for performing an interpolation calculation for obtaining an average of color component values of pixels around the target pixel can be used.

The regression equation calculation means (7 + 9) is configured to calculate, for each pixel at the position of the pixel of interest and M pixels in the vicinity thereof, based on the value of the Jth color component of each pixel or in the vicinity of each pixel. An X coordinate value xi is obtained based on the value of the Jth color component of the pixel having the Jth color component, and based on the value of the Kth color component of each pixel or in the vicinity of the Kth color of each pixel. Y coordinate value yi is obtained based on the value of the Kth color component of the pixel having the color components, and is composed of the X coordinate value (xi) and Y coordinate value yi obtained for each of the M pixels. , Coordinate data calculation means (7) for outputting two-dimensional coordinate data (xi, yi) (i = 1 to M), and M two-dimensional coordinate data (xi, yi) output from the coordinate data calculation means, Regression analysis is performed for (i = 1 to M) using the X coordinate value as the explanatory variable and the Y coordinate value as the objective variable. And a constant calculation circuit (9) for obtaining constants a and b of the regression equation y = ax + b, and the first interpolation means (31) uses the value Xj of the Jth color component of the pixel of interest and the constants a and b. And the following formula: Yk = a × Xj + b
Thus, an interpolation calculation circuit 11 for obtaining the value Yk of the Kth color component at the position of the target pixel is further provided.

  On the other hand, as the second interpolation means (12), the arrangement of the pixels on the two-dimensional plane is represented on a two-dimensional coordinate plane including the H axis and the V axis orthogonal to each other, and the color component value is perpendicular to the two-dimensional plane. In the three-dimensional orthogonal coordinate system represented on the Z axis, the average value of the values of the Jth color component of the pixels at the positions of the pixels having a plurality of Jth color components around the pixel of interest, and the pixel of interest A first geometrical figure formed by the value of the color component of the pixel of interest at the position of the pixel of interest, and the Kth of the pixels at the respective positions of the pixels having a plurality of Kth color components around the pixel of interest The second geometric figure formed by the average value of the color components and the value of the Kth color component at the position of the target pixel obtained by interpolation by the second interpolation means is similar to each other. What determines the value of the Kth color component at the position can be usedFrom another viewpoint, the second interpolation means calculates the average value of the values of the Jth color component of the pixels having a plurality of Jth color components around the pixel of interest and the Jth at the position of the pixel of interest. The ratio of the difference from the color component value to the distance between the plurality of pixels and the target pixel used for calculating the average value on the two-dimensional plane, and the plurality of Kth colors around the target pixel The difference between the average value of the Kth color component value of the pixel having the component and the value of the Kth color component at the position of the target pixel obtained by interpolation by the second interpolation means, and the calculation of the average value The value of the Kth color component at the position of the target pixel is determined so that the ratio between the plurality of used pixels and the distance between the target pixels on the two-dimensional plane is the same.

In the above-described embodiment, the first interpolation operation circuit performs interpolation by regression analysis or linear interpolation, and the second interpolation circuit performs interpolation assuming a similar relationship between color component values. The present invention is not limited to this, as long as the first interpolation circuit and the second interpolation circuit perform interpolation using different methods.
Furthermore, although the frame memory is used in the above embodiment, a line memory capable of storing the number of lines necessary for the interpolation process may be used instead.

1 is a block diagram illustrating a pixel signal processing device according to a first embodiment of the present invention. It is a figure which shows the Bayer type | mold arrangement | sequence of a pixel. It is a figure which shows the production | generation of the insufficient color component by an interpolation process. It is a block diagram which shows the structure of the interpolation process circuit 4 of the pixel signal processing apparatus of FIG. 6 is a diagram illustrating pixels used in an interpolation process for obtaining a value of a G pixel at a position of an R pixel by the interpolation calculation circuit 11 according to the first embodiment. FIG. 6 is a diagram illustrating a concept of an interpolation process for obtaining a value of a G component at a position of an R pixel by the interpolation circuit 12 according to the first embodiment. FIG. It is a figure which shows the pixel arrangement | positioning centering on G pixel in a Bayer type | mold arrangement | sequence, (a) shows the case centering on the odd-numbered G pixel, (b) centering on the even-numbered G pixel. Shows the case. 6 is a diagram illustrating pixels used in an interpolation process for obtaining values of R and B pixels at a G pixel position by the interpolation calculation circuit 11 according to the first embodiment. FIG. 6 is a diagram illustrating a concept of an interpolation process for obtaining an R component value at a position of a G pixel by the interpolation circuit 12 according to the first embodiment. FIG. 6 is a diagram illustrating a concept of an interpolation process for obtaining a value of a B component at a position of a G pixel by the interpolation circuit 12 according to the first embodiment. FIG. 4 is a diagram illustrating pixels used in an interpolation process for obtaining a value of a B pixel at a position of an R pixel by the interpolation calculation circuit 11 according to the first embodiment. FIG. 6 is a diagram illustrating a concept of an interpolation process for obtaining a value of a B component at a position of a pixel having a G component by the interpolation circuit 12 according to the first embodiment. FIG. FIG. 6 is a diagram illustrating an interpolation result when a color is gradually changed from one color to another in order to explain the effect of the first embodiment. In order to demonstrate the effect of Embodiment 1, it is a figure which shows the interpolation result when many high frequency components are contained. FIG. 14 is a diagram illustrating a relationship between a regression line and data in the case of FIG. 13 in order to explain the effect of the first embodiment. It is a figure which shows the relationship between the regression line in the case of FIG. 14, and data for demonstrating the effect of Embodiment 1. FIG. It is a block diagram which shows the structure of the interpolation processing circuit 4 of Embodiment 2 of this invention. It is a figure which shows the pixel used by the interpolation process for calculating | requiring the value of the G component of the position of R pixel by the interpolation calculation circuit 11 of Embodiment 2. It is a figure which shows the pixel used by the interpolation process for calculating | requiring the value of B component of the position of R pixel by the interpolation calculation circuit 11 of Embodiment 2. FIG. It is a block diagram which shows the structure of the interpolation processing circuit 4 of Embodiment 3 of this invention. 10 is a flowchart illustrating an interpolation process for obtaining a value of a G pixel at a position of an R pixel in the third embodiment. 10 is a flowchart illustrating an interpolation process for obtaining values of an R pixel and a B pixel at a G pixel position in the third embodiment. 14 is a flowchart illustrating an interpolation process for obtaining a value of a B pixel at a position of an R pixel in the third embodiment.

Explanation of symbols

  5 frame memory, 7 coordinate data calculation circuit, 9 constant calculation circuit, 11 interpolation circuit, 12 interpolation circuit, 13 synthesis circuit, 24 correlation determination circuit, ST25A1 step of performing interpolation based on regression line, ST25B1 interpolation based on regression line ST34A Step of performing interpolation based on regression line, ST45A1 Step of performing interpolation based on regression line, ST45B1 Step of performing interpolation based on regression line, ST27A Step of performing interpolation assuming similarity relationship with color components ST27B Performing interpolation assuming similarity relationship to color components, ST36 Performing interpolation assuming similarity relationship to color components, ST47A Performing interpolation assuming similarity relationship to color components, ST47B Perform hypothetical interpolation Step ST28 Combining the two interpolation results ST37 Combining the two interpolation results ST48 Combining the two interpolation results ST29 Step for storing the interpolation results ST38 Step for storing the interpolation results ST49 Interpolation ST22A Step of storing results, ST22A Step of calculating data for calculating regression line, ST22B Step of calculating data for calculating regression line, ST31 Step of calculating data for calculating regression line, ST42A regression Calculating data for calculating a line, ST42B calculating data for calculating a regression line, ST24A1 calculating a regression line, ST24B1 calculating a regression line, ST33A calculating a regression line ST44A1 calculating regression line, ST44B1 calculating regression line, ST21 correlation determining step, ST41 correlation determining step, ST26A1 calculating correlation coefficient of regression line, ST26B1 calculating correlation coefficient of regression line ST35A step of calculating the correlation coefficient of the regression line, ST46A1 step of calculating the correlation coefficient of the regression line, ST46B1 step of calculating the correlation coefficient of the regression line, ST23A two-dimensional coordinate data is any one of the conditions 1 to 4 ST23B Step of checking whether the two-dimensional coordinate data applies to any of conditions 1 to 4, ST32 Step of checking whether the two-dimensional coordinate data applies to any of conditions 1 to 4, ST43A Two-dimensional coordinate data ST43B is a step of checking whether the two-dimensional coordinate data is applied to any of the conditions 1 to 4.

Claims (24)

Based on a set of pixel signals representing the values of the respective color components of a plurality of pixels arranged on a two-dimensional plane, each having any of the first to Nth different color components, the J (J Is a pixel signal representing the value of the Kth color component (K is any one of 1 to N excluding J) at the position of the pixel of interest having one (1 to N) color component. In a pixel signal processing device that generates
First interpolation means for generating a first interpolation value by performing the above interpolation by a first interpolation method;
Second interpolation means for generating a second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
Synthesizing means for synthesizing the first interpolation value obtained by the first interpolation means and the second interpolation value obtained by the second interpolation means, and outputting a pixel signal representing the synthesized interpolation value; equipped with a,
The first interpolation means includes:
A pixel signal representing the value of the Jth color component of the pixel of interest and a pixel having the Jth color component in the vicinity of the pixel of interest, and the pixel of the pixel having the Kth color component in the vicinity of the pixel of interest. Regression equation calculation for calculating a constant of a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the pixel of interest based on the pixel signal representing the value of the K color component Means,
Interpolation calculation using a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component, including the constants calculated by the regression equation calculation means, or the color components of pixels around the pixel of interest An interpolation calculation circuit for performing an interpolation calculation to obtain an average of the values of
The second interpolation means includes:
In a three-dimensional orthogonal coordinate system in which an array of pixels on a two-dimensional plane is represented on a two-dimensional coordinate plane including an H axis and a V axis orthogonal to each other, and a color component value is represented on a Z axis perpendicular to the two-dimensional plane.
The average value of the values of the Jth color component of the pixels at the positions of the pixels having a plurality of Jth color components around the target pixel, and the color components of the target pixel at the position of the target pixel A first geometric figure created with the value of
The average value of the Kth color components of the pixels having a plurality of Kth color components around the target pixel and the target pixel obtained by interpolation by the second interpolation means The second geometric figure created by the value of the Kth color component at the position of is similar to each other,
Determining the value of the Kth color component at the position of the target pixel;
The regression equation calculating means also calculates a correlation coefficient k between the value of the Jth color component and the value of the Kth color component,
The pixel signal processing apparatus , wherein the synthesizing means determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k . Based on a set of pixel signals representing the values of the respective color components of a plurality of pixels arranged on a two-dimensional plane, each having any of the first to Nth different color components, the J (J Is a pixel signal representing the value of the Kth color component (K is any one of 1 to N excluding J) at the position of the pixel of interest having one (1 to N) color component. In a pixel signal processing device that generates
First interpolation means for generating a first interpolation value by performing the above interpolation by a first interpolation method;
Second interpolation means for generating a second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
Synthesizing means for synthesizing the first interpolation value obtained by the first interpolation means and the second interpolation value obtained by the second interpolation means, and outputting a pixel signal representing the synthesized interpolation value; With
The first interpolation means includes:
A pixel signal representing the value of the Jth color component of the pixel of interest and a pixel having the Jth color component in the vicinity of the pixel of interest, and the pixel of the pixel having the Kth color component in the vicinity of the pixel of interest. Regression equation calculation for calculating a constant of a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the pixel of interest based on the pixel signal representing the value of the K color component Means,
Interpolation calculation using a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component, including the constants calculated by the regression equation calculation means, or the color components of pixels around the pixel of interest An interpolation calculation circuit for performing an interpolation calculation to obtain an average of the values of
The second interpolation means includes:
The difference between the average value of the Jth color component value of a pixel having a plurality of Jth color components around the pixel of interest and the value of the Jth color component at the position of the pixel of interest, and the average value A ratio between the plurality of pixels used for the calculation of the pixel and the distance between the target pixels on the two-dimensional plane;
The average value of the Kth color component values of the pixels having a plurality of Kth color components around the pixel of interest and the Kth color at the position of the pixel of interest obtained by interpolation by the second interpolation means. The ratio between the difference between the component values and the distance between the plurality of pixels used in the calculation of the average value and the pixel of interest on the two-dimensional plane is the same.
Determining the value of the Kth color component at the position of the target pixel ;
The regression equation calculating means also calculates a correlation coefficient k between the value of the Jth color component and the value of the Kth color component,
The synthesis means determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k.
A pixel signal processing apparatus. Each of the photoelectric conversion elements further includes an imaging element in which first to Nth types of photoelectric conversion elements each having one of the first to Nth color components are arranged on a two-dimensional plane. 3. The pixel signal processing apparatus according to claim 1, wherein the obtained signal is used as at least part of the set of pixel signals. 2. The apparatus according to claim 1 , further comprising storage means for storing the pixel signal output by the combining means, wherein the pixel signal stored by the storage means is used as at least part of the set of pixel signals. 4. The pixel signal processing device according to any one of 3 . The regression equation calculation means is
A pixel having the Jth color component based on the value of the Jth color component of each pixel or the neighborhood of each pixel with respect to each pixel at the position of the target pixel and M pixels in the vicinity thereof X coordinate value xi is obtained based on the value of the Jth color component of the pixel, and based on the value of the Kth color component of each pixel or the number of pixels having the Kth color component in the vicinity of each pixel. Y coordinate value yi is obtained based on the value of the K color component, and two-dimensional coordinate data (xi) composed of the X coordinate value (xi) and Y coordinate value yi obtained for the M pixels. , Yi), coordinate data calculation means for outputting (i = 1 to M),
For M pieces of two-dimensional coordinate data (xi, yi) and (i = 1 to M) output from the coordinate data calculation means, regression analysis is performed using the X coordinate value as an explanatory variable and the Y coordinate value as an objective variable. And a constant calculation means for determining the constants a and b of the regression equation y = ax + b.
The interpolating circuit is
Using the value Xj of the J-th color component of the pixel of interest and the constants a and b, the following equation Yk = a × Xj + b
The pixel signal processing device according to claim 1, wherein the value Yk of the Kth color component at the position of the target pixel is obtained by the following. The constant calculation means is
The constants a and b are calculated by the following formulas (1) and (2) using the two-dimensional coordinate data (xi, yi) and (i = 1 to M).

The pixel signal processing device according to claim 5, wherein the pixel signal processing device is obtained by: The constant calculation means is
The correlation coefficient of the X and Y coordinate values were calculated as the correlation coefficient k value of the color component value and the first K color component of the first J,
The pixel signal processing apparatus according to claim 5 or 6, wherein the synthesizing unit determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k. The constant calculation means is
The correlation coefficient k is calculated by the following formula (3) using the two-dimensional coordinate data (xi, yi), (i = 1 to M).

The pixel signal processing device according to claim 7, wherein the pixel signal processing device is obtained by: The synthesis means is
When the absolute value of the correlation coefficient is close to 1, the ratio of the first interpolation value is increased.
When the absolute value of the correlation coefficient is close to 0, the ratio of the second interpolation value is increased.
The first interpolation value and the second interpolation value are synthesized.
The pixel signal processing device according to claim 7 or 8 . The first interpolation means includes:
For the two-dimensional coordinate data (xi, yi), (i = 1 to M), the following conditions 1 to 4, that is,
Condition 1: all xi are the same value,
Condition 2: all yi are the same value,
Condition 3: The difference between the maximum value and the minimum value of xi is below a certain threshold value,
Condition 4: The difference between the maximum value and the minimum value of yi is a certain threshold value or less.
Having a condition determination means for determining whether or not any of the above is satisfied,
The interpolation calculation circuit performs linear interpolation using pixels around the pixel of interest when the condition determination unit determines that any of the conditions 1 to 4 is satisfied, and otherwise performs as described above. Interpolate using a regression equation
When the condition determining unit determines that any one of the conditions 1 to 4 is satisfied, the synthesizing unit increases the proportion of the first interpolation value in the synthesized value. The pixel signal processing device according to claim 5 , wherein synthesis is performed. It further has a correlation determination means for detecting a direction having a strong correlation around the target pixel,
The first and second interpolation means are:
11. The pixel signal processing apparatus according to claim 1, wherein only the value of a color component of a pixel existing in a strongly correlated direction centering on the target pixel is used for the interpolation.   The regression equation calculation means uses the regression equation for each pixel having the Kth color component in the vicinity of the target pixel, using the result of low-pass filtering the value of the Kth color component of the surrounding pixels. The pixel signal processing apparatus according to claim 1, wherein a constant is calculated. Based on a set of pixel signals representing the values of the respective color components of a plurality of pixels arranged on a two-dimensional plane, each having any of the first to Nth different color components, the J (J Is a pixel signal representing the value of the Kth color component (K is any one of 1 to N excluding J) at the position of the pixel of interest having one (1 to N) color component. In the pixel signal processing method for generating by interpolation,
A first interpolation step for performing the interpolation by a first interpolation method to generate a first interpolation value;
A second interpolation step for generating a second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
A synthesis step of synthesizing the first interpolation value obtained in the first interpolation step and the second interpolation value obtained in the second interpolation step, and outputting a pixel signal representing the synthesized interpolation value; equipped with a,
The first interpolation step includes:
A pixel signal representing the value of the Jth color component of the pixel of interest and a pixel having the Jth color component in the vicinity of the pixel of interest, and the pixel of the pixel having the Kth color component in the vicinity of the pixel of interest. Regression equation calculation for calculating a constant of a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the pixel of interest based on the pixel signal representing the value of the K color component Steps,
Interpolation calculation using a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component, including the constants calculated in the regression equation calculation step, or color components of pixels around the pixel of interest An interpolation calculation step for performing an interpolation calculation to obtain an average of the values of
The second interpolation step includes
In a three-dimensional orthogonal coordinate system in which an array of pixels on a two-dimensional plane is represented on a two-dimensional coordinate plane including an H axis and a V axis orthogonal to each other, and a color component value is represented on a Z axis perpendicular to the two-dimensional plane.
The average value of the values of the Jth color component of the pixels at the positions of the pixels having a plurality of Jth color components around the target pixel, and the color components of the target pixel at the position of the target pixel A first geometric figure created with the value of
The average value of the K-th color component of each pixel having a plurality of K-th color components around the target pixel and the target pixel obtained by interpolation in the second interpolation step The second geometric figure created by the value of the Kth color component at the position of is similar to each other,
Determining the value of the Kth color component at the position of the target pixel;
The regression equation calculating step also calculates a correlation coefficient k between the value of the Jth color component and the value of the Kth color component,
The pixel signal processing method characterized in that the synthesis step determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k . Based on a set of pixel signals representing the values of the respective color components of a plurality of pixels arranged on a two-dimensional plane, each having any of the first to Nth different color components, the J (J Is a pixel signal representing the value of the Kth color component (K is any one of 1 to N excluding J) at the position of the pixel of interest having one (1 to N) color component. In the pixel signal processing method for generating by interpolation,
A first interpolation step for performing the interpolation by a first interpolation method to generate a first interpolation value;
A second interpolation step for generating a second interpolation value by performing the interpolation by a second interpolation method different from the first interpolation method;
A synthesis step of synthesizing the first interpolation value obtained in the first interpolation step and the second interpolation value obtained in the second interpolation step, and outputting a pixel signal representing the synthesized interpolation value; With
The first interpolation step includes:
A pixel signal representing the value of the Jth color component of the pixel of interest and a pixel having the Jth color component in the vicinity of the pixel of interest, and the pixel of the pixel having the Kth color component in the vicinity of the pixel of interest. Regression equation calculation for calculating a constant of a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component in the vicinity of the pixel of interest based on the pixel signal representing the value of the K color component Steps,
Interpolation operation using a regression equation representing the relationship between the value of the Jth color component and the value of the Kth color component, including the constants calculated in the regression equation calculation step, or color components of pixels around the pixel of interest An interpolation calculation step for performing an interpolation calculation to obtain an average of the values of
The second interpolation step includes
The difference between the average value of the Jth color component value of a pixel having a plurality of Jth color components around the pixel of interest and the value of the Jth color component at the position of the pixel of interest, and the average value A ratio between the plurality of pixels used for the calculation of the pixel and the distance between the target pixels on the two-dimensional plane;
The average value of the Kth color component values of pixels having a plurality of Kth color components around the target pixel and the Kth color at the position of the target pixel obtained by interpolation in the second interpolation step. The ratio between the difference between the component values and the distance between the plurality of pixels used in the calculation of the average value and the pixel of interest on the two-dimensional plane is the same.
Determining the value of the Kth color component at the position of the target pixel;
The regression equation calculating step also calculates a correlation coefficient k between the value of the Jth color component and the value of the Kth color component,
The synthesis step determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k.
And a pixel signal processing method. A signal obtained from each of the photoelectric conversion elements of the imaging element in which first to N-th types of photoelectric conversion elements each having one of the first to N-th color components are arranged on a two-dimensional plane. 15. The pixel signal processing method according to claim 13, wherein the pixel signal processing method is used as at least part of the set of pixel signals. 16. The method according to claim 13 , further comprising a storing step of storing the pixel signal output by the combining step, wherein the pixel signal stored by the storing step is used as at least part of the set of the pixel signals. The pixel signal processing method according to any one of the above. The regression equation calculation step is
A pixel having the Jth color component based on the value of the Jth color component of each pixel or the neighborhood of each pixel with respect to each pixel at the position of the target pixel and M pixels in the vicinity thereof X coordinate value xi is obtained based on the value of the Jth color component of the pixel, and based on the value of the Kth color component of each pixel or the number of pixels having the Kth color component in the vicinity of each pixel. Y coordinate value yi is obtained based on the value of the K color component, and two-dimensional coordinate data (xi) composed of the X coordinate value (xi) and Y coordinate value yi obtained for the M pixels. , Yi), coordinate data calculation step for outputting (i = 1 to M),
For the M pieces of two-dimensional coordinate data (xi, yi) and (i = 1 to M) output from the coordinate data calculation step, regression analysis is performed using the X coordinate value as an explanatory variable and the Y coordinate value as an objective variable. And a constant calculation step for determining the constants a and b of the regression equation y = ax + b.
The interpolation calculation step is
Using the value Xj of the J-th color component of the pixel of interest and the constants a and b, the following equation Yk = a × Xj + b
The pixel signal processing method according to claim 13 or 14, wherein the value Yk of the Kth color component at the position of the pixel of interest is obtained by: The above constant calculation step is
The constants a and b are calculated by the following formulas (1) and (2) using the two-dimensional coordinate data (xi, yi) and (i = 1 to M).

The pixel signal processing method according to claim 17, wherein the pixel signal processing method is obtained by: The above constant calculation step is
Calculating a correlation coefficient between the X coordinate value and the Y coordinate value as a correlation coefficient k between the value of the Jth color component and the value of the Kth color component ;
The pixel signal processing method according to claim 17 or 18, wherein the synthesis step determines a synthesis ratio of the first interpolation value and the second interpolation value based on the correlation coefficient k. The above constant calculation step is
The correlation coefficient k is calculated by the following formula (3) using the two-dimensional coordinate data (xi, yi), (i = 1 to M).

The pixel signal processing method according to claim 19, wherein the pixel signal processing method is obtained by: The synthesis step is
When the absolute value of the correlation coefficient is close to 1, the ratio of the first interpolation value is increased.
When the absolute value of the correlation coefficient is close to 0, the ratio of the second interpolation value is increased.
The first interpolation value and the second interpolation value are synthesized.
The pixel signal processing method according to claim 19 or 20 . The first interpolation step includes:
For the two-dimensional coordinate data (xi, yi), (i = 1 to M), the following conditions 1 to 4, that is,
Condition 1: all xi are the same value,
Condition 2: all yi are the same value,
Condition 3: The difference between the maximum value and the minimum value of xi is below a certain threshold value,
Condition 4: The difference between the maximum value and the minimum value of yi is a certain threshold value or less.
A condition determination step for determining whether or not any of the above is satisfied,
The interpolation calculation step performs linear interpolation using pixels around the pixel of interest when it is determined that the condition determination step applies to any of the conditions 1 to 4, and otherwise, as described above. Interpolate using a regression equation
In the synthesis step, when the condition determination step determines that any one of the conditions 1 to 4 is satisfied, the ratio of the first interpolation value in the synthesized value is increased. The pixel signal processing method according to any one of claims 17 to 21 , wherein synthesis is performed. A correlation determination step of detecting a direction having a strong correlation around the pixel of interest;
The first and second interpolation steps are:
23. The pixel signal processing method according to claim 13, wherein only the value of a color component of a pixel existing in a direction having a strong correlation around the target pixel is used for the interpolation.   The regression equation calculating step uses the result of low-pass filtering the value of the K-th color component of the surrounding pixels for each pixel having the K-th color component in the vicinity of the target pixel. The pixel signal processing method according to claim 13, wherein a constant is calculated.

Images