JP4893973B2 - Method and apparatus for providing color interpolation in a color filter array using edge detection and correction terms - Google Patents

Abstract

A method and apparatus for color plane interpolation are provided which interpolates the color values of pixels differently depending on an edge direction and whether a pixel is at an edge within an image. The use of the edge detection during the interpolation of each of the colors present in the color pattern helps reduce some of the disadvantages of the loss of image sharpness abundant in known demosaicing techniques.

Description

  The present invention relates to demosaicing for use with images captured by an image sensor.

  Image sensors, also known as imaging devices, were developed from the late 1960s to the early 1970s, primarily for the imaging, transmission and display of television images. The imaging device absorbs incident light having a specific wavelength (such as photons, x-rays, etc.) and generates an electrical signal corresponding to the absorbed radiation. There are several different types of semiconductor-based imagers, including charge coupled devices (CCD), photodiode arrays, charge injection devices (CID), hybrid focal plane arrays, and CMOS imagers.

  These imaging devices typically consist of an array of pixels including a photosensor, and when an image is imaged on the array, each pixel generates a signal corresponding to the intensity of light incident on that pixel. To do. These signals may then be stored, for example, for displaying the corresponding image on a monitor, or otherwise used to provide information about the optical image. The photosensor is typically a phototransistor, a photoconductor, or a photodiode. Therefore, the magnitude of the signal generated by each pixel is proportional to the amount of light incident on the photosensor.

  In order to allow the photosensor to capture a color image, the photosensor, for example, when using a Bayer pattern, a red (R) photon, a green (G) photon, and a blue (B ) It must be possible to detect photons separately. Thus, each pixel must sense only one color or one spectral band. For this purpose, a color filter array (CFA) is typically placed in front of the pixels so that each pixel measures the light of the color of the filter associated with it. In this manner, each pixel of the color image sensor is covered with either a red filter, a green filter, or a blue filter based on a specific pattern (for example, a Bayer pattern).

  In the case of the cheapest CMOS image sensor or CCD image sensor, the color filter is integrated with the sensor. A common example of a color filter pattern is the tiled color filter array described in US Pat. No. 3,971,065, which is hereby incorporated by reference. Is called a “Bayer pattern” color filter.

  As shown in FIG. 1, the Bayer pattern 100 is an array of repeatedly appearing red (R) filters, green (G) filters, and blue (B) filters. The underlying pixels are a red pixel, a blue pixel, and a green pixel, respectively. In the Bayer pattern 100, red pixels, green pixels, and blue pixels have alternating red and green pixels on the first row 105 of the array, and alternately appearing blue and green pixels. Arranged to exist on the next row 110. These rows of alternating pixels are arranged so that they appear repeatedly throughout the array. Therefore, when the image sensor is read out row by row, the pixel sequence for the first line is GRGRGR. . . And the alternate line sequence is BGBGBBG. . . Is read. This output is called sequential RGB or sRGB.

  In the Bayer pattern 100, the sampling rates for all three basic colors are adjusted based on the human vision system's sensing capabilities. That is, the green color that the human eye is most sensitive and responsive to is sensed by a greater number of sensors, and the blue and red colors where human vision has less resolution are sensed by a smaller number of sensors. This is why in the Bayer pattern, every other green sensing pixel is present in the array position and every fourth red sensing pixel and every blue sensing pixel is present in the array position.

  As shown in FIG. 2, in a solid-state image sensor, a Bayer patterned filter may be formed on an array 200 of pixel sensors 205. Specifically, the array 200 including the pixel sensors 205 is formed on the semiconductor substrate 210. Each pixel sensor 205 includes a photosensitive element 215, which may be any photoelectric conversion device such as a photogate, a photoconductor, or a photodiode. A color filter array 220 may be formed on the top metal layer 225 in the array 200, and various metallization and insulation layers such as an intermediate dielectric layer (ILD) 235 and a passivation layer 230 including an insulator metallization pattern. Is separated from the photosensor 215. The metal layer 225 may be opaque and may be used to shield pixel areas that are not light sensitive. A convex lens 240 is formed on the color filter 220. In operation, incident light is imaged by the lens 240 onto the photosensitive element 215 via the filter 220.

  For the Bayer pattern filter, red, green and blue values are required for each pixel. Since each pixel sensor cell senses only one color, the values of the remaining two colors are calculated by interpolation from neighboring pixels that sense those missing colors. This color plane interpolation is known as demosaic processing. For example, referring to FIG. 1, the pixel sensor cell 115 is combined with a green filter so that the pixel sensor cell 115 senses green light and generates a signal representing only the green light. Values may be interpolated from neighboring red pixel sensor cells 120 and 125 and neighboring blue pixel sensor cells 130 and 135, respectively, to obtain approximate amounts of red and blue light for pixel sensor cell 115. If demosaicing is not performed correctly, the resulting image may contain very noticeable color artifacts.

Bahadir K.K. Gunturk, Yucel Altunbasak and Russell M. et al. In March 2002 by Merseau, IEEE Transactions on Image Processing, Vol. II, no. A paper entitled “Color Plane Interpolating Using Alternative Projects” published in 9 (the content of this paper is incorporated herein by reference) compares several demosaicing techniques. As explained in this paper, each of these demosaicing techniques has its own advantages and disadvantages.

  As described above, for each pixel, the value used for the first color is based on the sensed color (ie, the light sensed by the pixel's sensor 205) and the remaining two colors. Is based on a value interpolated from the sensed value of the corresponding neighboring pixel. Each sensed value represents a color value at the center pixel. Each of the interpolated color values represents a value in the center pixel. The interpolated signal has essentially a lower quality than the originally sensed signal. For example, the red value interpolated at the center pixel will be different from the red value sensed for the same pixel at the center.

  Although the demosaicing method typically only reproduces missing color components, it may result in artifacts such as the so-called zipper effect and random color dots. In addition, there is a trade-off between filtering noise and blurring sharp edges. Interpolating a pixel using surrounding pixel values tends to blur the image when the pixel at the edge is interpolated. Furthermore, this will reduce the sharpness of the image at the edges. This is mainly due to averaging pixels straddling edges included in the image. The demosaicing method averages the pixels so that the pixels present at the edge do not have the sharpness required for the edge pixels. Thus, there is a need and need for color plane interpolation that provides an improved demosaicing technique that detects and takes into account image edges when calculating interpolation values.

  A method and apparatus for color plane interpolation is provided that interpolates the color values of a pixel separately depending on the edge direction and whether the pixel is at an edge in the image. The use of edge detection during the interpolation of each color present in the color pattern helps to reduce some of the image sharpness disadvantages commonly found in known demosaicing techniques.

  In the following detailed description, reference is made to the accompanying drawings, which form a part hereof, and in which are shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and other embodiments may be used and structures without departing from the spirit and scope of the invention. It should be understood that general, logical, and electrical changes may be made.

  The term “pixel” means an image element unit cell including a photosensor and a transistor for converting light radiation into an electrical signal. For purposes of explanation, representative pixels are illustrated in the drawings and descriptions herein, and typically, the fabrication of all pixels in an imaging device proceeds simultaneously in a similar manner. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

  It should be understood that reference to a CMOS imaging device according to the present invention is made for the sole purpose of describing only one example of the present invention. Of course, it can be readily understood that the present invention is not limited to CMOS imagers, but also applies to CCDs and other imagers that use color filters on pixels. Furthermore, the present invention is described using a standard three-color Bayer pattern. It should be understood that the present invention is not limited to that standard three-color Bayer pattern, but may be applied to a color space that uses different colors or uses more or less than three colors. It is.

3A and 3B are plan views of a partial pixel array 300 of an imaging device according to an embodiment of the present invention. Specifically, FIGS. 3A and 3B show a 5 × 5 kernel (described in more detail below) used in the present invention. As described above, each pixel, eg, pixel 23, senses a single color. Thus, each green (G) pixel provides a sensed value for green. Similarly, each blue (B) pixel provides a sensed value for blue and each red (R) pixel provides a sensed value for red. In order to reproduce a color image, the values for each remaining color are interpolated for each pixel of a particular color. Further, as described below, the sensed pixel values may also be replaced with interpolated values. The interpolation of the red, green and blue values uses the sensed value for one color and the interpolated values of the remaining two colors (typically performed in prior art systems). The imbalance that arises from the

  As shown in FIGS. 3A and 3B, when a particular pixel 45 is processed, a 5 × 5 kernel of surrounding pixels is used in the following interpolation calculations, but in many interpolation calculations 3 Only pixels present in the x3 kernel are required. However, other kernel sizes may be used in the present invention.

In general, for each selected pixel, a 3 × 3 interpolation kernel that exists within a 5 × 5 kernel is first defined and the color values for the pixels that exist within that kernel are calculated. . Next, the edge direction is determined. Depending on the pixel color and edge direction, some values of the 3 × 3 kernel may be recalculated. The 3x3 kernel is then used to determine if there is an edge at the selected pixel. Then, based on whether an edge is present, also, based on the pixel color, the complement between value is calculated. Different interpolation parameters are applied to the interpolation process based on whether the pixel being processed is present at the image edge and based on the perceived color of the pixel.

The interpolation processing of the present invention can be understood more clearly from the flowchart of FIG. 7 illustrating the entire interpolation processing. In step 701, a pixel is selected for processing. Throughout the description, the representative pixel 45 shown in FIGS. 3A and 3B and the pixel shown in FIG. 4 as the central pixel are used as the pixel to be processed as an example. Next, in step 703, a 3 × 3 kernel surrounding the selected pixel is defined (eg, pixels 34, 35, 36, 44, 45, 46, 54, 55, and 56 in FIGS. 3A and 3B). ). In step 705, it is determined whether pixel 45 is a sensed green pixel. If pixel 45 is green, then in step 707, the green value for the 3 × 3 kernel is obtained using the pixel values actually sensed from the pixels present in the 3 × 3 kernel, using the following set of equations (1 ). FIG. 4 shows the positions in the 3 × 3 kernel of the green values g11, g12, g13, g21, g22, g23, g31, g32, g33 calculated for the pixels present in the 3 × 3 kernel. The 3 × 3 kernel is calculated using the actual values of the green pixels 25, 34, 36, 45, 47, 54, 56, and 65 shown in FIG. 3A. The 3x3 kernel is calculated using the following formula:

[Formula (1)]

g11 _{= P 34
g12 = MEDIAN (P 25, P} 34, P 45, P 36)
g13 _{= P 36
g21 = MEDIAN (P 56, P} 47, P 45, P 36)
g22 _{= P 45
g23 = MEDIAN (P 56, P} 65, P 45, P 54)
g31 = P _{54
g32 = MEDIAN (P 43, P} 34, P 45, P 54)
g33 _{= P 56}

Here, “P” is a pixel value, and “xy” is a pixel number shown in FIGS. 3A to 3B. As shown in FIGS. 3A to 3B, “x” represents a row, and “y” represents a column. If pixel 45 is red or blue, as shown in FIG. 3B, then in step 709, the 3 × 3 kernel is the green pixel 24, 26, 33, 35, 37, 44, 46, 53, 55, The values 57, 64, and 65 are used to calculate based on the following set of equations (2): A 3 × 3 green kernel is then calculated using the following formula:

[Formula (2)]

_{g11 = MEDIAN (P 33, P} 24, P 35, P 44)
g12 _{= P 35
g13 = MEDIAN (P 26, P} 37, P 35, P 46)
g21 _{= P 44
g22 = MEDIAN (P 44, P} 35, P 55, P 46)
g23 = _{P 46
g31 = MEDIAN (P 53, P} 44, P 55, P 64)
g32 _{= P 55
g33 = MEDIAN (P 46, P} 57, P 55, P 66)

In this example embodiment, the same kernel value is interpolated using a median interpolation of the actually sensed values of the surrounding green pixels. The 3x3 interpolation kernel is green in this example embodiment. This is because, as described above, in the Bayer CFA, the green pixels are the most numerous pixels and the pixels that most affect the image. The pixel values present in the 3 × 3 kernels g11, g12, g13, g21, g22, g23, g31, g32, and g33 shown in FIG. 4 exist in the 3 × 3 kernels shown in FIGS. 3A and 3B. The green value of the pixel to be calculated is calculated. The green kernel is used during edge detection and when calculating correction terms. One skilled in the art will recognize that other pixel colors may be used to calculate the 3 × 3 interpolation kernel, or that the kernel may have a different size.

Next, in step 711 shown in FIG. 7, the calculated 3 × 3 green kernel is used to determine the edge direction. Four edge direction values are calculated in step 711. Dotted lines 2, 5, 8, and 11 shown in FIG. 4 are four directions in which the edge direction calculation is performed. The remaining dotted lines are merely examples used when calculating the four edge direction values. The sum of each dotted line is calculated in the following set of equations (3).

[Formula (3)]

S1 = g11 + g12 + g13
S2 = g21 + g22 + g23
S3 = g31 + g32 + g33
S4 = g11 + g21 + g31
S5 = g12 + g22 + g32
S6 = g13 + g23 + g33
S7 = g21 + g11 + g12
S8 = g13 + g22 + g31
S9 = g23 + g32 + g33
S10 = g21 + g31 + g32
S11 = g11 + g 22 + g33
S12 = g12 + g13 + g23

An edge direction value DIR _x is calculated for each direction line 2, 5, 8, and 11 where “DIR” is its value and “x” is the direction line number. The following set of equations (4) is used to calculate the four edge direction values.

[Formula (4)]

DIR ₂ = max (| S1-S2 |, | S2-S3 |)
DIR ₅ = max (| S4-S5 |, | S5-S6 |)
DIR ₈ = max (| S7-S8 |, | S8-S9 |)
DIR ₁₁ = max (| S10−S11 |, | S11−S12 |)

The direction with the largest DIR value is determined to be the edge direction, which is determined in step 711.

  Next, in step 713 shown in FIG. 7, it is determined whether the pixel 45 being processed is green. In step 713, if the pixel 45 is not green, it is determined in step 714 whether the edge direction is the direction of the dotted line 2 (horizontal direction). If the edge direction is horizontal, g22 is recalculated to take into account the edge direction, as shown in FIG. 3B. The value of g22 is recalculated using the formula: g22 = (g21 + g23) / 2. In step 714, if the edge direction is not horizontal, it is next determined in step 715 whether the edge direction is the direction of the dotted line 5 (vertical direction). If the direction is determined to be vertical, the pixel value of g22 is recalculated using the formula: g22 = (g12 + g32) / 2. However, if the edge direction is not the vertical direction in step 715, the value of g22 remains the same in step 719. If it is determined in step 713 that the pixel is green, the value of g22 remains unchanged.

Next, for all pixels that may be used when calculating the interpolated value to determine if there is an edge in the pixel 45 being processed, the difference value in the same color pixel pair. Is calculated. In step 723, if the pixel 45 to be processed is green, in step 725, a difference value is calculated based on the following set of equations (5).

[Formula (5)]

D ₁ = | P ₅₄ -P ₃₆ |
D ₂ = | P ₄₅ −P ₄₇ |
D ₃ = | P ₃₅ -P ₅₇ |
D ₄ = | P ₄₄ -P ₆₆ |
_{D 5 = | P 55 -P 37} |
D ₆ = | P ₆₄ -P ₄₆ |

If in step 723 it is determined that the pixel 45 to be processed is not green, then in step 727 the difference value is calculated based on the following set of equations (6):

[Formula (6)]

D ₁ = | P ₅₄ -P ₃₆ |
_{D 2 = | P 56 -P 34} |
D ₃ = | P ₄₆ -P ₄₄ |
D ₄ = | P ₅₅ -P ₃₅ |
D ₅ = | P ₄₅ −P ₄₃ |
D ₆ = | P ₄₅ -P ₄₇ |
D ₇ = | P ₄₅ -P ₆₅ |

In step 729, it is determined using equation (7) whether the maximum difference value among the calculated difference values is greater than a predetermined edge threshold value.

[Formula (7)]

MAX (D ₁ ... D ₇ )> edge threshold

If the maximum difference value is smaller than the threshold value, it is considered that the edge does not exist (step 731). If the maximum difference value is larger than the threshold value, the edge is considered to exist (step 733). ). The edge threshold may be set to be application specific or may be set based on user preferences.

If no edge is detected in step 729, it is again determined whether the pixel 45 being processed is green. If the pixel 45 being processed is green, an eighth set of equations (8) is used in step 739 to calculate the interpolated red, green, and blue values of the pixel 45. The The eighth set of interpolation equations (8) is

[Formula (8)]

R ₄₅ = (P ₃₅ + P ₅₅ + P ₃₇ + P ₅₇ ) / 4
_B45 = ( _P44 + _P46 + _P66 + _P64 ) / 4
G ₄₅ = (P ₄₅ + P ₄₇ + P ₅₄ + P ₃₆ ) / 4

Where R ₄₅ , G ₄₅ , and B ₄₅ are the red, green, and blue interpolation values for the pixel 45 being processed. Note that even if pixel 45 is a green pixel, its value is interpolated using the actual values of P ₄₅ together with the actual values of the other green pixels surrounding pixel 45.

If it is determined in step 735 that the pixel 45 to be processed is not green, it is determined in step 741 whether the pixel 45 to be processed is blue. If the pixel 45 being processed is blue, a ninth set of equations (9) is used in step 747 to calculate the red, green and blue values of the pixel 45. The ninth set of interpolation equations is

[Formula (9)]

R ₄₅ = (P ₅₆ + P ₅₄ + P ₃₆ + P ₃₄ ) / 4
B ₄₅ = (P ₄₅ + P ₄₃ + P ₄₇ + P ₆₅ ) / 4
_{G 45 = (P 44 + P} 35 + P 46 + P 55) / 4

Here, R ₄₅ , G ₄₅ , and B ₄₅ are a red interpolation value, a green interpolation value, and a blue interpolation value for the pixel 45. In step 741, if the pixel 45 being processed is not blue, the pixel 45 is red, and in step 749, a tenth set of equations (10) yields the red value, green value, and Used to calculate the blue value. The tenth set of interpolation equations is

[Formula (10)]

R ₄₅ = (P ₄₅ + P ₄₃ + P ₄₇ + P ₆₅ ) / 4
B ₄₅ = (P ₅₆ + P ₅₄ + P ₃₆ + P ₃₄ ) / 4
_{G 45 = (P 44 + P} 35 + P 46 + P 55) / 4

Here, R ₄₅ , G ₄₅ , and B ₄₅ are a red interpolation value, a green interpolation value, and a blue interpolation value for the pixel 45.

If an edge is detected in step 729 and the pixel 45 being processed is green in step 737, the red, green and blue interpolated values of the pixel 45 are the eleventh set of interpolation values. Calculated using equation (11). The eleventh set of interpolation equations is

[Formula (11)]

R ₄₅ = (P ₃₅ + P ₅₅ ) / 2 + DELTA _red
B ₄₅ = (P ₄₄ + P ₄₆ ) / 2 + DELTA _blue
G ₄₅ = g22

Where R ₄₅ , G ₄₅ , and B ₄₅ are a red interpolation value, a green interpolation value, and a blue interpolation value for the pixel 45, and DELTA _red and DELTA _blue are correction terms. The value of DELTA _blue and DELTA _red is calculated based on the edge direction determined in step 711. If the edge direction is a dotted line 2 in the horizontal direction, it is determined that DELTA _blue = g22− (g21 + g23) / 2, DELTA _red = 0, and the edge direction is a dotted line 5 in the vertical direction. If so, DELTA _blue = 0 and DELTA _red = g22- (g12 + g32) / 2. Finally, if it is determined that the edge direction is the diagonal direction 11 or 8, DELTA _blue = g22− (g21 + g23) / 2 and DELTA _red = g22− (g12 + g32) / 2.

If it is determined in step 737 that the pixel 45 to be processed is not green, it is determined in step 745 whether the pixel 45 to be processed is blue. If the pixel 45 being processed is blue, then in step 751, the twelfth set of equations (12) is used to calculate the red, green, and blue interpolation values. The twelfth set of interpolation equations is

[Formula (12)]

R ₄₅ = (P ₅₆ + P ₅₄ + P ₃₆ + P ₃₄ ) / 4 + DELTA _red
B ₄₅ = P ₄₅
G ₄₅ = g22

Where R ₄₅ , G ₄₅ , and B ₄₅ are a red interpolation value, a green interpolation value, and a blue interpolation value for the pixel 45, and the value of DELTA _red is expressed by the equation: DELTA _red = g22− Calculated using (g11 + g13 + g31 + g33) / 4. In step 745, if the pixel 45 to be processed is not blue, then in step 753, the thirteenth set of equations (13) is the red, green and blue interpolation values for the red pixel 45 to be processed. Used to calculate a value. The thirteenth set of interpolation equations is

[Formula (13)]

B ₄₅ = (P ₅₆ + P ₅₄ + P ₃₆ + P ₃₄ ) / 4 + DELTA _blue
R ₄₅ = P ₄₅
G ₄₅ = g22

Where R ₄₅ , G ₄₅ , and B ₄₅ are the red interpolation value, the green interpolation value, and the blue interpolation value for the pixel 45, and the value of DELTA _blue is expressed by the formula: DELTA _blue = g22− Calculated using (g11 + g13 + g31 + g33) / 4. The DELTA correction values for red and blue pixels are based on the edge direction and the green 3 × 3 interpolation kernel.

As mentioned above, the present invention provides a new and novel system and method for color plane interpolation, where the interpolated values for green, red, and blue are based solely on the sensed values obtained. Rather than being calculated, it is also based on whether the pixel being processed is at the edge of the image. According to the present invention, the imbalance between the sensed value and the interpolated value is eliminated by interpolating the value for each color. In addition, edge detection and correction terms are used to minimize image edge blurring.

  FIG. 5 shows a block diagram of a CMOS imager integrated circuit (IC) 500 having a pixel array 505 that includes a plurality of pixels arranged as a matrix, the pixel array 505 being, for example, two pixels arranged as a Bayer pattern A region 510 including a green pixel (G), one blue pixel (B), and one red pixel (R) is included. The pixels in each row in array 505 are all turned on simultaneously by row select line 515 and the pixels in each column are selectively output by respective column select line 520.

  Row line 515 is selectively driven by row driver 525 in response to row address decoder 530. The column selection line 520 is selectively driven by the column selector 535 according to the column address decoder 540. The pixel array 505 is operated by a timing & control circuit 545 that controls address decoders 530 and 540 to select appropriate row and column lines for pixel signal readout.

A pixel column signal, typically including a pixel reset signal (V _rst ) and a pixel image signal (V _sig ), is read by a sample and hold circuit 550 coupled to a column selector 535. A difference signal (V _rst −V _sig ) is generated for each pixel by the differential amplifier 555, and the difference signal is amplified and digitized by the analog-to-digital converter 570 (ADC). The analog-to-digital converter 570 supplies the digitized pixel signal to the image processor 575, which performs the processing method 700 of the present invention. Alternatively, the image processor 575 may output an image file having the raw format as received from the analog-to-digital converter 570, which is received by another processor in accordance with the present invention. And may be processed.

  A typical processor system that includes an imaging device 605 that includes a pixel array into which pixel signals are interpolated in accordance with the present invention is shown generally in FIG. The imaging device 605 generates an output image from signals supplied from the pixel array. The processor system is an example system that receives the output of the CMOS imager. Without limitation, such systems may include computer systems, camera systems, scanners, machine vision systems, medical sensor systems (such as medical drug sensors), automatic diagnostic systems, and other imaging systems. Well, all of them may use the present invention. Alternatively, the present invention may be implemented in the processing system 600 if the image processor 575 in the imaging device 605 provides an improved image signal.

  A processor system 600, such as a computer system, typically includes a central processing unit (CPU) 610, such as a microprocessor, which is connected to an input / output (I / O) via a bus 620. Communicate with device 615. The imaging device 605 also communicates with the system via the bus 620 or other communication link. The processor-based system 600 further includes a random access memory (RAM) 625 and may include a removable memory 630 such as a flash memory, and the memory 630 communicates with the CPU 610 via the bus 620. To do. The imaging device 600 may be combined with a processor such as a CPU, a digital signal processor, or a microprocessor that includes a storage device on a single integrated circuit or on a different chip than the processor. May or may not be provided.

  The present invention is not limited to the embodiment described above. For example, a program embodying the processing method according to the present invention may be stored on a carrier medium that may include RAM, floppy disk, data transmission, compact disk, etc., and an image file according to the present invention. The processing method according to the present invention may be embodied as a circuit or a combination thereof, which may be performed by an associated processor receiving and processing. Furthermore, the present invention may be implemented as a plug-in for an existing software application, or the plug-in may be used alone. The present invention is not limited to the transport medium specified here, and the present invention may be implemented using any transport medium known in the art.

  Other methods of interpolating values for green, red, and blue pixels are within the scope of the present invention, and the present invention is not limited to the use of the equations or correction terms described above. Should be understood. Specifically, for example, instead of or in addition to the pixels described herein, other surrounding pixels may be used when calculating the interpolation value. As such, the foregoing description and the accompanying drawings are to be considered merely illustrative of exemplary embodiments that implement the features and advantages of the present invention. Any modification of the present invention that comes within the spirit and scope of the appended claims should be considered part of the present invention. Accordingly, the invention is not to be seen as limited by the foregoing description and accompanying drawings, but only by the scope of the appended claims.

The foregoing and other advantages and features of the invention will become more apparent from the detailed description of example embodiments provided below with reference to the accompanying drawings.
It is a top view which shows a general color filter array. It is sectional drawing which shows a part of pixel array which has a color filter array. It is a top view as an example which shows the 5 * 5 pixel kernel used for the interpolation system as an example of this invention. It is a top view as an example which shows the 5 * 5 pixel kernel used for the interpolation system as an example of this invention. FIG. 3 is a block diagram illustrating a 3 × 3 kernel used in accordance with an example embodiment of the present invention. 1 is a block diagram illustrating a CMOS imager integrated circuit (IC) having a pixel array configured according to an example embodiment of the present invention. FIG. 1 is a diagram showing a processor system having a CMOS imaging device according to the present invention. FIG. It is a flowchart explaining the process of one Embodiment of this invention.

Claims (31)

A method of color plane interpolation for image pixels, performed by a processor using pixel values output by an image sensor ,
Selecting pixels for processing;
Forming a kernel comprising neighboring pixels associated with the selected pixel;
Calculating pixel values for a color of a color pattern for all pixels in the kernel;
Determining at least three edge direction values respectively representing a horizontal edge direction, a vertical edge direction and a diagonal edge direction;
Using the pixel values of the kernel of a certain color to the calculated, also based on the value of the edge direction, the selected pixel to determine whether there an edge in the image, and the edge direction Determining the direction of such an edge , determined as the value of the edge direction having the largest absolute value of the value ;
Recalculating at least one pixel value present in the kernel based on the determined edge direction only if the edge direction is determined to be a horizontal or vertical direction;
Generating said neighboring pixel, and whether said selected pixel is present in the edge, on the basis of said direction of said edge, the value for each of the selected pixels for each color of the color pattern When,
Including methods.   The method according to claim 1, wherein the color pattern is a Bayer pattern. Said one color, The method of claim 1 wherein the green. Wherein the determining whether the selected pixel is present or falling edge of di in the image,
Calculating a difference value in various pairs of pixels having the same color;
Comparing the largest difference value with a predetermined threshold;
The method of claim 1 comprising: The method of claim 4 , wherein the pair of pixels includes the selected pixel. A color imaging device including a sensing array of image pixel sensors,
A first plurality of image pixel sensors used to generate a sensed value of a first color;
A second plurality of image pixel sensors used to generate a sensed value of the second color;
A third plurality of image pixel sensors used to generate a sensed value of the third color;
The sensed of the first color to determine a first set of pixel values for a plurality of image pixels present in a pixel kernel for the first color including a selected pixel and a neighboring pixel. And using the first set of pixel values, determine at least three edge direction values representing a horizontal edge direction, a vertical edge direction, and a diagonal edge direction, respectively. based on, to determine the edge direction of said selected pixel, and determines whether the edge before Symbol selected pixel exists, the pixel values of the first set of the first color, the neighboring a pixel, on the basis of the edge direction, the first color of the selected pixel, the second color, and the interpolation means for calculating an interpolated value for the third color,
Means for generating a representative image using interpolated values of the first color, the second color, and the third color;
A color imaging device. The color imaging device according to claim 6 , wherein at least one value present in the pixel kernel is recalculated based on the edge direction before the interpolated value is calculated. The color imaging apparatus according to claim 6 , wherein the pixel kernel is a 3 × 3 kernel of pixels. The edge direction is
DIR _horizontal = max (| S1-S2 |, | S2-S3 |),
DIR _vertical = max (| S4-S5 |, | S5-S6 |),
DIR _diagonal1 = max (| S7-S8 |, | S8-S9 |),
DIR _diagonal2 = max (| S10-S11 |, | S11-S12 |),
Where S1, S2, S3, S4, S5, S6, S7, S8, S9, S10, S11, and S12 use the kernel,
S1 = g11 + g12 + g13,
S2 = g21 + g22 + g23,
S3 = g31 + g32 + g33,
S4 = g11 + g21 + g31,
S5 = g12 + g22 + g32,
S6 = g13 + g23 + g33,
S7 = g21 + g11 + g12,
S8 = g13 + g22 + g31,
S9 = g23 + g32 + g33,
S10 = g21 + g31 + g32,
S11 = g11 + g 22 + g33 ,
S12 = g12 + g13 + g23,
Where g11, g12, and g13 are the pixel values of the first row of the kernel, and g21, g22, and g23 are the pixels of the second row of the kernel. Values g31, g32, and g33 are pixel values of the third row of the kernel, and the edge direction is the direction having the largest value,
The color imaging device according to claim 6 . The color imaging apparatus according to claim 6 , wherein the edge is detected by calculating a difference value of the neighboring pixels and comparing the maximum difference value with an edge threshold value. If the selected pixel is green, the difference value is
D ₁ = | P _{x + 1, y-1} −P _{x-1, y + 1} |
D ₂ = | P _{x, y} −P _{x, y + 2} |,
_{D 3 = | P x-1} , y -P x + 1, y + 2 |,
D ₄ = | P _{x, y−1} −P _{x + 2, y + 1} |,
_{D 5 = | P x + 1} , y -P x-1, y + 2 |,
D ₆ = | P _{x + 2, y−1} −P _{x, y + 1} |,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 10 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . If the selected pixel is red or blue, the difference value is
D ₁ = | P _{x + 1, y-1} −P _{x-1, y + 1} |
D ₂ = | P _{x + 1, y + 1} -P _{x-1, y-1} |,
D ₃ = | P _{x, y + 1} −P _{x, y−1} |,
D ₄ = | P _{x + 1, y} −P _{x-1, y} |,
_{D 5 = | P x, y} -P x, y-2 |,
D ₆ = | P _{x, y} −P _{x, y + 2} |,
D ₇ = | P _{x, y} −P _{x + 2, y} |,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 10 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . When the pixel is green, the 3 × 3 kernel is
g11 = Px _{-1, y-1} ,
_{g12 = MEDIAN (P x-2} , y, P x-1, y-1, P x, y, P x, y + 1),
g13 = Px _{-1, y + 1} ,
_{g21 = MEDIAN (P x + 1} , y + 1, P x, y + 2, P x, y, P x-1, y + 1),
g22 = P _{x, y} ,
_{g23 = MEDIAN (P x + 1} , y + 1, P x + 2, y, P x, y, P x + 1, y-1),
g31 = P _{x + 1, y−1} ,
_{g32 = MEDIAN (P x, y} -2, P x-1, y-1, P x, y, P x + 1, y-1),
g33 = P _{x + 1, y + 1} ,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 8 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . If the pixel is red or blue, the 3 × 3 kernel is
_{g11 = MEDIAN (P x-1} , y _{over 2, P x-2, y} -1, P x-1, y, P x, y-1),
g12 = P _{x-1, y} ,
_{g13 = MEDIAN (P x-2} , y + 1, P x-1, y + 2, P x-1, y, P x, y + 1),
g21 = P _{x, y−1} ,
g22 = MEDIAN (Px _{, y-1} , Px _{-1, y} , Px _{+ 1, y} , Px _{, y + 1} ),
g23 = P _{x + 1, y + 1} ,
_{g31 = MEDIAN (P x + 1} , y-2, P x, y-1, P x, y + 1, P x + 2, y-1),
g32 = P _{x + 1, y} ,
_{g33 = MEDIAN (P x, y} + 1, P x + 1, y + 2, P x + 1, y, P x + 2, y + 1),
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 8 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . If the calculated interpolated values for the first color, the second color, and the third color of the selected pixel are green and the edge is absent,
R _{x, y} = (P _{x-1, y} + P _{x + 1, y} + P _{x-1, y + 2} + P _{x + 1, y + 2} ) / 4,
_{Bx, y} = ( _{Px, y-1} + _{Px, y + 1} + _{Px + 2, y + 1} + _{Px + 2, y-1} ) / 4,
G _{x, y} = (P _{x, y} + P _{x, y + 2} + P _{x + 1, y-1} + P _{x-1, y + 1} ) / 4,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 6 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . If the calculated interpolated values for the first color, the second color, and the third color of the selected pixel are blue and the edge is absent,
R _{x, y} = ( P _{x + 1, y + 1} + P _{x + 1, y-1} + P _{x-1, y + 1} + P _{x-1, y-1} ) / 4,
_{B x, y = (P x} , y + P x, y-2 + P x, y + 2 + P x + 2, y) / 4,
G _{x, y} = (P _{x, y-1} + P _{x-1, y} + P _{x, y + 1} + P _{x + 1, y} ) / 4,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 6 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . The calculated interpolation values for the first color, the second color, and the third color of the selected pixel are such that the selected pixel is red and there are no edges;
B _{x, y} = ( P _{x + 1, y + 1} + P _{x + 1, y-1} + P _{x-1, y + 1} + P _{x-1, y-1} ) / 4,
_{R x, y = (P x} , y + P x, y-2 + P x, y + 2 + P x + 2, y) / 4,
G _{x, y} = (P _{x, y-1} + P _{x-1, y} + P _{x, y + 1} + P _{x + 1, y} ) / 4,
Where P _{x, y} is the selected pixel and “x” is the row where the pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 6 , wherein “y” means a column in which pixels are relative to the selected pixel P _{x, y} . The calculated interpolation values for the first color, the second color, and the third color of the selected pixel are such that the selected pixel is green and there is an edge;
R _{x, y} = (P _{x-1, y} + P _{x + 1, y} ) / 2 + DELTA TA _red ,
_{Bx, y} = ( _{Px, y-1} + _{Px, y + 1} ) / 2 + DELTA _blue ,
G _{x, y} = g22,
Where g22 and correction terms DELTA _red and DELTA _blue are calculated based on the kernel value and the edge direction , and P _{x, y} is the selected pixel, "x", the pixel is the selected pixel P _x, means the line is relative to _y, "y" is the pixel P _x which pixels are the _selected, relative to _{the y} The color imaging device according to claim 6 , which means a column that is typical. The calculated interpolation values for the first color, the second color, and the third color of the selected pixel are such that the selected pixel is blue and there is an edge;
_{Rx, y} = ( _{Px + 1, y + 1} + _{Px + 1, y-1} + _{Px-1, y + 1} + _{Px-1, y-1} ) / 4 + DELTA _red ,
B _{x, y} = P _{x, y} ,
G _{x, y} = g22,
Where g22 and the correction term DELTA _red are calculated based on the kernel value and the edge direction , and P _{x, y} is the selected pixel and “x” the pixel means a line is relative to the selected pixel P _{x, y, "y"} is relative, for the pixel P _{x, y} of the pixel is the selected The color imaging device according to claim 6 , which means a column that is The calculated interpolation values for the first color, the second color, and the third color of the selected pixel are such that the selected pixel is red and there is an edge;
_{Bx, y} = ( _{Px + 1, y + 1} + _{Px + 1, y-1} + _{Px-1, y + 1} + _{Px-1, y-1} ) / 4 + DELTA _red ,
R _{x, y} = P _{x, y} ,
G _{x, y} = g22,
Where g22 and DELTA _red are calculated based on the kernel value and the edge direction , P _{x, y} is the selected pixel, and “x” is the pixel A row that is relative to the selected pixel P _{x, y} means “y” refers to a column in which a pixel is relative to the selected pixel P _{x, y} . The color imaging device according to claim 6, which means. A processing system,
Including an imaging device including an array of image pixel sensors;
The imaging device is
A first plurality of image pixel sensors used to generate a sensed value of a first color;
A second plurality of image pixel sensors used to generate a sensed value of the second color;
A third plurality of image pixel sensors used to generate a sensed value of the third color;
The sensed of the first color to determine a first set of pixel values for a plurality of image pixels present in a pixel kernel for the first color including a selected pixel and a neighboring pixel. And using the first set of pixel values, determine at least three edge direction values representing a horizontal edge direction, a vertical edge direction, and a diagonal edge direction, respectively. based on, to determine the edge direction of said selected pixel, and determines whether the edge before Symbol selected pixel exists, the pixel values of the first set, and the neighboring pixels, said selected was based on the the edge direction of the pixel, the first color of the selected pixel, the second color, and the interpolation circuit for calculating an interpolated value for the third color,
A processing system comprising: The processing system of claim 21 , wherein at least one value included in the pixel kernel is recalculated based on the edge direction. A method of processing a digital color signal , performed by an image processor using sensed values obtained from a sensing array of pixel sensors of a color imaging device ,
Red spectrum, blue spectrum, and obtaining a sensed value from each of a plurality of pixel sensor corresponding to the applied light having a wavelength which is one in the green spectrum,
Or falling edge of on the basis of out-di-detection, before for each of outs oxygen sensor, red values, blue values, and, and the step of interpolating the green values,
Only including, the red value, blue value, and the step of interpolating the green values,
For each of the plurality of pixel sensors, calculating a green kernel of interpolation values using pixel values of neighboring green pixel sensors;
Determining at least three edge direction values, each representing a horizontal edge direction, a vertical edge direction, and a diagonal edge direction, for each of the pixel sensors, using the green kernel of the interpolation values;
Determining the edge direction for each of the pixel sensors as an edge direction value having a maximum absolute value of the edge direction values;
Determining whether the selected pixel sensor is present at the edge;
The interpolated value for each of the pixel sensors based on the pixel value of the neighboring pixel sensor, the green kernel of the interpolated value, whether each of the pixel sensors is present at an edge, and the determined edge direction. A step of calculating
Including methods. Said step of determining whether said pixel is present at an edge;
Calculating difference values in various pairs of neighboring pixels having the same color;
Comparing the largest difference value with a predetermined threshold;
24. The method of claim 23 , comprising: A computer readable medium for storing computer instructions for enabling a computer to perform color plane interpolation for a color pattern comprising:
Obtaining a sensed value from each of a plurality of pixel sensors corresponding to applied light having a wavelength that is one of a red spectrum, a blue spectrum, and a green spectrum;
Based on or falling edge of output di detection, for each of said plurality of pixel sensor, a red value, a blue value, and a step of interpolating the green values,
Only including, the red value, blue value, and the step of interpolating the green values,
For each of the plurality of pixel sensors, calculating a green kernel of interpolation values using pixel values of neighboring green pixel sensors;
Determining at least three edge direction values, each representing a horizontal edge direction, a vertical edge direction, and a diagonal edge direction, for each of the pixel sensors, using the green kernel of the interpolation values;
Determining the edge direction for each of the pixel sensors as an edge direction value having a maximum absolute value of the edge direction values;
Determining whether the selected pixel sensor is present at the edge;
The interpolated value for each of the pixel sensors based on the pixel value of the neighboring pixel sensor, the green kernel of the interpolated value, whether each of the pixel sensors is present at an edge, and the determined edge direction. A step of calculating
The including computer-readable media. Determining whether each of the pixel sensors is present at an edge;
Calculating difference values in various pairs of neighboring pixels having the same color;
Comparing the largest difference value with a predetermined threshold;
26. The computer readable medium of claim 25 comprising: Processor executes using the pixel values output by the image sensor, a method of color plane interpolation against the selected pixels of the pixel array,
Determining whether the selected pixel is present at an image edge;
A first color to determine a first set of pixel values for a plurality of pixels present in a pixel kernel of the pixel array for the first color including the selected pixel and neighboring pixels; Use the sensed value,
If the selected pixel is present at the edge, using the first set of pixel values of the first color, at least three edges each representing a horizontal edge direction, a vertical edge direction, and a diagonal edge direction Determining a direction value, determining the edge direction based on the value of the edge direction, and a first set corresponding to the first set of pixel values, neighboring pixels and edge direction of the first color; At least when the selected pixel is associated with a sensed value for the second color or the third color of the color pattern. Calculating a calculated pixel value for one of the second and third colors based on the first set of pixel values and the edge direction; and Calculating a pixel value for each color of the color pattern based on the neighboring pixels using a second set of parameters if no pixel is present at the edge;
Including methods. 28. The method of claim 27 , wherein the first set of parameters and the second set of parameters are based on a color of the selected pixel. 28. The method of claim 27 , wherein the color pattern is a Bayer color pattern. 30. The method of claim 29 , wherein the first set of parameters comprises a third set of parameters for green pixels, a fourth set of parameters for blue pixels, and a fifth set of parameters for red pixels. . 30. The method of claim 29 , wherein the second set of parameters comprises a sixth set of parameters for green pixels, a seventh set of parameters for blue pixels, and an eighth set of parameters for red pixels. .