JP4892909B2 - Signal processing method, signal processing circuit, and camera system using the same - Google Patents

Description

  The present invention relates to a signal processing method, a signal processing circuit, and a camera system using a solid-state imaging device such as a CCD (Charge Coupled Device) and a CMOS (Complementary Metal Oxide Semiconductor), and particularly to a color signal at low illuminance. Improve color reproducibility by correcting contamination of infrared light.

As a technique for improving sensitivity at low illuminance in a color imaging device using a solid-state imaging device, it is generally known to remove an infrared cut filter arranged in front of the solid-state imaging device. There is no major problem when generating black and white video, but when color video is generated using this method, infrared light components that are not the original colors are taken in and processed, resulting in a significant reduction in color reproducibility. To do. Particularly in the evening, color reproducibility is significantly deteriorated because the color temperature decreases and the infrared light component increases.
In order to solve this, for example, in Patent Document 1 (Japanese Patent Laid-Open No. 2001-69519), CCD light receiving elements having sensitivity up to the near infrared light region are two-dimensionally arranged, and the light receiving elements are in a one-to-one relationship. A filter array is provided.
Specifically, a color filter for cutting near-infrared light is provided on R (red), G (green), and B (blue) of the imaging (light receiving) element, and on Y (luminance) of the light receiving element. In order to transmit near-infrared light, no near-infrared cut filter is provided.
In other words, by using an image sensor that includes a filter that transmits light in the near infrared region and a filter that does not transmit light in the near infrared region, the Y signal contains light in the near infrared region, so the sensitivity becomes high. The chroma (color) signal does not contain near-infrared light, and therefore has good color reproducibility.
Japanese Patent Laid-Open No. 2000-59798 discloses a configuration for switching the position of an infrared cut filter.
A visible light cut filter is not provided between the lens and the CCD image pickup device, but an IR (infrared light) cut filter and an optical low-pass filter are provided, and this IR cut filter is taken in and out of the image pickup optical path by a switching mechanism. (Positions P1 and P2) Both the sensitivity in the near-infrared light region and the sensitivity in the visible light region are effectively utilized to enable photographing with near-infrared light and visible light.
Further, Patent Document 3 (Japanese Patent Laid-Open No. 2003-70009) discloses a CCD or CMOS imaging device, imaging driving means, signal processing means, synchronization signal generating means, lens, and the like. A generation unit and a luminance signal generation unit are provided.
In order to eliminate the influence of infrared light, the color difference signal generation unit configured in the signal processing means performs correction to reduce the value of the color difference signal affected by the infrared light.
For example, the positive / negative of the RY signal of the color difference signal corresponding to Cr (= Ye + Mg− (Cy + G); Ye yellow, Mg magenta, Cy cyan, G green), which has a large influence of infrared light, is detected. When the signal is negative or zero, the correction is invalidated, and when the signal is positive, the correction for the RY signal is performed to improve the color reproducibility. The correction amount of the RY signal is multiplied by a parameter, and this value is subtracted from the luminance signal to vary the luminance signal. This is a method of processing the luminance signal with a color difference signal mixed with infrared light. It is.
JP 2001-69519 A JP 2000-59798 A Japanese Patent Laid-Open No. 2003-70009

However, Patent Documents 1 and 2 may improve the color reproducibility, but it seems difficult to reduce the cost. On the other hand, Patent Document 3 cannot suppress an infrared light component with respect to a component mixed in the BY signal.
As shown in the spectral characteristics of the CMOS image sensor in FIG. 1, the IR (Infrared) component is mixed not only in the R (red) filter but also in the G (green) and B (blue) filters. You can see that In particular, the mixing of infrared light is significant in the wavelength region of 780 to 900 nm.
Here, GR represents the spectral characteristic of the pixel G (green) arranged in the row of the pixel R (red), and GB represents the spectral characteristic of the pixel G arranged in the row of the pixel B (blue). This characteristic varies depending on the image sensor (sensor) and also varies depending on the spectral characteristic of the ambient light.
The method of Patent Document 3 seems to be less effective because it does not perform processing for each color filter but takes measures when signal processing is almost completed (after generating a color signal).

  The present invention has been made in view of the above problems, and the object of the present invention is not to complicate the manufacturing process, to eliminate the cost increase associated with a switching mechanism such as an IR cut filter, and to be effective in a circuit. Another object of the present invention is to provide a signal processing method, a signal processing circuit, and a camera system using the same.

Signal processing method of the present invention, a value obtained by multiplying a predetermined coefficient to the color data including luminance data or infrared component includes an infrared component, a plurality by performing an operation for subtracting the color data including the infrared light component for each color Generate corrected color data, generate a luminance signal from the generated plurality of corrected color data, and use the generated luminance signal and a color difference signal generated from a plurality of color data including the infrared components. Generate a signal.
The signal processing circuit of the present invention corrects the infrared light component mixed in the color data from the pixel by the estimation arithmetic processing circuit for each color filter of the pixel, and generates a luminance signal using the corrected color data. A signal processing circuit for generating an image signal using the luminance signal and a color difference signal generated from the color data from the pixel, wherein the estimation calculation processing circuit inputs the color data and the luminance data, and A selection circuit that selectively selects color data and luminance data by a control signal, a multiplier that multiplies the output signal output from the selection circuit by a predetermined coefficient, and the multiplier outputs the color data. A calculator that subtracts the signal and outputs the corrected color data.
The camera system of the present invention includes an imaging unit in which a plurality of light receiving elements having sensitivity up to the near-infrared region are arranged and an input image is captured by the plurality of light receiving elements through a plurality of color filters not including an infrared blocking filter And a signal processing unit that performs image processing on the imaging signal output from the imaging unit and outputs the image signal, and the signal processing unit is a value obtained by multiplying luminance data or color data obtained from the imaging signal by a predetermined coefficient. An estimation calculation process for generating a plurality of corrected color data by performing an operation for subtracting the color data from the color data for each of the color filters, generating a luminance signal from the generated plurality of corrected color data, and generating An image signal is generated using the luminance signal and the color difference signals generated from the plurality of color data.

The signal processing method, the signal processing circuit, and the camera system using the same according to the present invention are provided with an infrared light cut filter and an infrared light transmission filter on the image sensor because the infrared light cut filter is completely unnecessary. Compared with a camera system or a camera system that improves the color reproducibility when there is a lot of infrared light by replacing the infrared light cut filter from the imaging path, the cost can be reduced.
Further, the present invention improves color reproducibility only by adding an infrared light (IR) estimation processing circuit to the luminance (Y) signal processing circuit before the color signal processing, rather than correcting after the color signal processing. be able to.

FIG. 2 shows a block configuration of the camera system 10 according to the embodiment.
The camera system 10 includes, for example, a lens 11, an image sensor 12, a preprocessing unit 13 having a CDS (correlated double sampling), a preamplifier, and the like, an AD (analog / digital) converter 14, a correction processing unit (21), a luminance A signal processing unit (22), a signal processing unit 20 having a color signal processing unit (23), and the like are configured.

  The lens 11 projects an image of a subject (not shown) on the imaging surface of the imaging element 12. The imaging device 12 is made of, for example, a CCD or a CMOS, converts an image transmitted through the lens 11 into an electrical signal, and supplies the image signal to a preprocessing unit 13 having a CDS (correlated double sampling), a preamplifier, and the like.

  The preprocessing unit 13 samples and holds the pixel signal from the image sensor 12 to extract necessary data, performs gain control (AGC) to adjust to an appropriate level, and also performs black level adjustment. The output signal of the preprocessing unit 13 is output to the AD converter 14 at the subsequent stage.

  Since the AD converter 14 handles a pixel (image) signal, one having an accuracy of 10 to 12 bits is adopted, converts the output signal supplied from the preprocessing unit 13 from an analog signal to a digital signal, and converts the digital signal to a subsequent stage. Output to the connected signal processing unit 20.

The signal processing unit 20 includes a correction processing unit 21, a luminance signal processing unit 22, a color signal processing unit 23, and the like, and performs digital signal processing. Hereinafter, a digital signal handled by each processing unit is simply referred to as data.
In the correction processing unit 21, black detection, digital gain adjustment (digital gain control), shading correction generated in the lens 11, and a delay line are used to separate a luminance signal (data) and a color signal (data), and a delay line Is used to correct pixel defects.
The luminance signal processing unit 22 corrects Y (luminance) signal vertical / horizontal (direction) contour correction (VH aperture control), mixes (mixes; adds) Y luminance signal and vertical / horizontal contour correction signal, and γ (gamma) correction. Various image processing such as luminance (brightness) key processing, solarization for processing a part of the image to an arbitrary luminance, and negative processing for inverting the image are performed.
The color signal processing unit 23 performs color separation and clamping processing, removal of color signal noise and color false signals, RGB matrix (Matrix) processing, white balance (WB) adjustment for varying coefficients of R, G, and B colors, γ (gamma) correction, RG / BG conversion, color false signal suppression processing, color difference signal (Cr / Cb) generation, chroma suppression (suppression) processing, Hue / Gain adjustment, monotone effect processing, etc. .

FIG. 3 shows a more detailed configuration of the signal processing unit 50 (20), which is an embodiment of the present invention.
The signal processing unit (circuit) 50 is composed of three blocks: a preprocessing unit (PRE) 60, a Y-block (luminance signal processing unit) 70, and a C-block (color signal processing unit) 80.
The pre-processing unit 60 includes a black detection unit 61, an adder 62, a digital gain adjustment unit 63, a shading correction unit 64, a delay line / defect correction unit 65, and the like.
The black detector 61 calculates the black level of the input data (signal) using the data (ADin) output from the AD converter 14 and outputs data for calculating the clamp level.
The adder 62 subtracts the black level from the input data (ADin), and generates new pixel data based on the black level (clamped to the black level).
The digital gain adjustment unit 63 adjusts the brightness by digitally changing the gain of the data clamped at the black level output from the adder 62.
The shading correction unit 64 has a luminance difference (unevenness) in the peripheral portion with respect to the central portion of the image due to the aperture of the lens 11, and corrects this uneven luminance.
The delay line / defect correction unit 65 separates input data into a luminance signal (data) and a color signal (data) using a delay line and a calculator (adder / subtracter). Also, the input data is delayed by 1 pixel or 2 pixels, for example, by a delay element, and the delayed pixel values are added using an arithmetic unit to obtain a value obtained by averaging the data before and after the defective pixel, and replaced with the defective pixel. Thus, pixel defect correction is performed.

  The data corrected by the delay line / defect correction unit 65 and further separated into the luminance signal and the color signal are supplied to the luminance signal processing unit (Y-block) 70 and the color signal processing unit (C-block) 80, respectively. The

A luminance signal processing unit (Y-block) 70 includes a YLPF (luminance signal low pass filter) 71, a vertical / horizontal contour correction (VH aperture control; VH aperture control) unit 72, and a contour correction adder (contour correction). Mix; aperture control (Mix) 73, γ (gamma) correction unit 74, image effect processing unit 75, chroma suppress signal generation unit 76, and the like.
The YLPF 71 performs addition calculation processing on the data related to the luminance signal to equivalently remove noise.
In addition to the filter function, an estimation (arithmetic) processing unit related to IR (infrared light) is provided, and a function for generating a luminance signal by correcting a color signal (data) is provided.
Multiply R (red), GR (green data of R row), B (blue), and GB (green data of B row) constituting the luminance signal by a predetermined coefficient, and this multiplication is performed from each color data (signal). The corrected color data is obtained by subtracting the value, and a luminance signal is generated using the corrected color data.
In addition to this, for example, the luminance data (Y) is multiplied by a predetermined coefficient, and the respective color data R (red), GR (green data for R rows), B (blue), GB (green data for B rows). The luminance value can also be generated by subtracting the multiplication value from, obtaining corrected color data.
Furthermore, when obtaining a correction color signal, it is possible to arbitrarily select whether the multiplication value for the coefficient is the same color data as the color data (signal) or luminance (Y) data. It is also possible to generate a luminance signal.
A vertical / horizontal contour correction (VH aperture control) unit 72 generates a pulse for emphasizing a horizontal contour portion of an image by using a delay element and an arithmetic unit for addition / subtraction processing, and also generates a vertical contour of the image. A pulse that emphasizes the part is also generated.
The contour correction adder (contour correction mix) 73 is supplied with the horizontal contour correction pulse and the vertical contour correction pulse output from the vertical / horizontal contour correction unit 72 and the luminance data supplied from the YLPF 71, and adds them. Luminance data that has been processed and whose contour is emphasized is output.
For example, when the display device is a CRT (Cathode ray tube), the gamma (γ) correction unit 74 has a γ characteristic of 2.2 for this CRT, so that 1 / γ = 0.45 on the imaging side in advance. The curve of the input / output characteristic is corrected so that the gradation of the image is correctly reproduced on the CRT side so that a correct reproduced image can be obtained.
The image effect processing unit 75 performs luminance (brightness) key processing, solar processing for processing a partial area of the image to an arbitrary luminance, and negative processing for inverting the image.
The chroma suppress signal generation unit 76 generates a control signal for deleting the chroma (color) data from the pulse signal generated by the vertical / horizontal contour correction unit 72 using the pulse of the image contour.

Next, the color signal processing unit (C-block) 80 will be described.
The color signal processing unit 80 receives the color data processed by the delay line / defect correction unit 65, and receives a color separation / clamp processing unit 81, a color (chroma) signal low-pass filter (CLPF) 82, and RGB Matrix (RGB matrix). 83, white balance (WB) adjustment unit 84, color signal optical detector 85, γ correction unit 86, RG / BG conversion, color false suppression processing unit 87, Cr / Cb generation unit 88, chroma suppress, Hue ( Hue control) / Gain (gain) adjustment, monotone effect processing unit 89, and the like.
The image signal (data) output from the delay line / defect correction unit 65 is supplied to the color separation / clamp processing unit 81, where it is color-separated and clamped to a predetermined value.
The chroma signal low-pass filter (CLPF) 82 is constituted by a digital low-pass filter, for example, and removes false color signals and noise from the color-separated signals.
An RGB Matrix (RGB matrix) 83 computes output data from the CLPF 82 to obtain R, G, and B primary color data, and outputs them to the white balance (WB) adjustment unit 84.
The white balance (WB) adjustment unit 84 multiplies the color signals (data) of R (red), G (green), and B (blue) by a predetermined coefficient to set white for the target color temperature. Yes. The white balanced color data is output to a gamma correction unit 86 and a color signal optical detector (COPD) 85.
The color signal optical detector (COPD) 85 detects the color data of the three primary colors R, G, and B, and one field in the OPD section of the AE (automatic exposure) / AWB (auto white balance) / OPD (optical detector) section 95, for example. 1 frame period is integrated.
In the exposure / automatic white balance (AE / AWB S / W (software)) unit 96, the color temperature is obtained from the color data coordinates described above by software processing, and each color corresponding to the target color temperature is obtained. Find the correction factor for the data. This value is fed back to the white balance (WB) adjustment unit 84 to adjust the white balance.
The RG / BG conversion / color false suppression processing unit 87 corrects the color data output from the white balance adjustment unit 84, and then a color obtained by mixing other color signals between pixels of the image sensor. The false signal is suppressed and converted into R (red) -G (green) and B (blue) -G (green) color signals.
The Cr / Cb generator 88 generates RY and BY color difference signals. Here, Cr = R−Y, Cb = B−Y (Y is a luminance signal).
The chroma suppress, Hue / Gain adjustment, and monotone effect processing unit 89 prevents the output of Cr and Cb color difference signal data by the control signal output from the chroma suppress signal generation unit 76.
In addition, with respect to the data of the color difference signal, Hue (hue) adjustment and Gain (gain) adjustment are performed, and a monotone effect processing for removing a color to make a black and white image is also performed.
The processed (color) data is output to an encoder (not shown), encoded using a luminance (Y) signal output from a luminance signal processing unit (Y-block) 70, and a component or composite image signal is generated. Generated.

Next, the operation of the signal processing unit (circuit) 50 in FIG. 3 will be described.
The analog signal of the pixel output from the preprocessing unit 13 is converted into digital data (signal) with 10 to 12 bits precision by the AD converter 14, and the converted data is a preprocessing (PRE) unit of the signal processing circuit 50. 60. The input data (ADin) is input to one input terminal of the black detection unit 61 and the adder 62, the black level is calculated by the black detection unit 61, and the calculated data is transferred to the clamp software (CLAMP S / W) 97 and the clamp level is calculated by software processing. The clamp data output from the clamp software 97 is supplied to the black detection unit 61, and the black clamp level data is output from the black detection unit 61 to the adder 62.
The data (ADin) input to the adder 62 and the clamp data at the black level are subtracted and output to the digital gain adjustment unit 63, and the gain is varied to adjust the brightness.
The gain-adjusted pixel data is supplied to the shading correction unit 64, and after correcting luminance unevenness due to a lens provided in front of the image sensor of the camera system, the delay line / defect correction 67 corrects the pixel defect. In addition, color data and luminance data are separated.
Regarding the separated color data and luminance data, the luminance data is supplied to the luminance signal processing unit (Y-block (Block)) 70, and the color data is supplied to the color signal processing unit (C-block (Block)) 80, respectively.

Next, FIG. 4 shows a block circuit according to the present invention of the YLPF 71 as an embodiment.
The YLPF 100 (71) shown in FIG. 4 includes a filter (not shown) and an IR (infrared light) component estimation processing unit 101 and a luminance calculation processing unit 102 that constitute an estimation calculation processing circuit. The IR component estimation processing unit In 101, IR component estimation subtraction processing is performed. In addition, the luminance calculation processing unit 102 generates a luminance signal using the corrected value of each color obtained by the arithmetic processing by the IR component estimation processing unit 101.
Now, assuming that the color filter array is, for example, the primary color Bayer system, and color data is R (red), B (blue), GR (green in the R row) and GB (green in the B row), the estimation formula is as follows: It is expressed as
When only luminance (Y) data is used as correction data,
R ′ = R−K _R * Y
B ′ = B−K _B * Y
GR ′ = GR−K _GR * Y
GB ′ = GB−K _GB * Y (1)
It is expressed.
Here, K _R , K _B , K _GR , K _GB are correction coefficients relating to the respective color data R, B, GR, GB, * represents a multiplication symbol, and Y = (R + B + GR + GB) / 4.
In addition, when each color data is used for the correction data,
R ′ = R−K _R * R
B ′ = B−K _B * B
GR ′ = GR−K _GR * GR
GB ′ = GB−K _GB * GB (2)
It is expressed.
In addition, if you set the correction to match the ambient light,
R ′ = R−K _R * R
B ′ = B−K _B * Y
GR ′ = GR−K _GR * Y
GB ′ = GB−K _GB * Y (3)
It can be expressed as.
In this example shows an example of correction by multiplying the coefficient K _R only R 'color data into R data, also used appropriately color data for the other color data, it is also possible to combine these corrected color data.

The IR component estimation processing unit 101 corrects each color data, and the luminance calculation processing unit 102 performs arithmetic processing using the corrected data to generate a luminance (Y) signal.
Specifically, when the color data corrected for the color signals of R ′, B ′, GR ′, GB ′ of any one of the above formulas (1), (2), or (3) is used. ,
Y = (R ′ + B ′ + GR ′ + GB ′) / 4 (4)
And luminance data are calculated (generated), and this value is output as a luminance signal.

The generated luminance signal (data) and the contour emphasis data output from the vertical / horizontal contour correction (VH aperture control) unit 72 are supplied to the contour correction adder 73, both data are added, and then γ correction is performed.
Then, image effect processing such as luminance key processing and solari / negative processing is performed, and a signal subjected to the image effect processing is derived from an output terminal (YOUT).

On the other hand, the color data separated by the color separation / clamp processing unit (81) is subjected to removal of frequency components higher than the color signal by a CLPF (color (chroma) signal low-pass filter) 82, and noise and color false signals are obtained. Remove.
The color data from which the noise and the color false signal are removed is obtained as R, G, B three primary color signals by the RGB matrix 83, the white balance is adjusted by the white balance (WB) adjustment unit 84, and the color signal is obtained by the gamma correction unit 86. Perform gamma (γ) correction for. The data output from the white balance (WB) adjustment unit 84 is detected by the color signal optical detector 85 and the AE / AWB / OPD unit 95 using hardware.
The detected color data is supplied to the AE / AWB S / W unit 96, and the color temperature for the color signal (data) is obtained by software processing by a computer or arithmetic unit, and each color is corrected for the target (white) color temperature. The coefficient is calculated, and the data is fed back to the white balance (WB) adjustment unit 84 to automatically adjust the white balance.

The color data subjected to gamma correction is subjected to RG / BG conversion and color false suppression processing unit 87 for RG / BG conversion and color false signal suppression processing. Then, using the RG and BG data generated by the RG / BG conversion (87) in the Cr / Cb generation unit 88, the color difference signals of Cr = R−Y and Cb = BY are obtained. Generate color data.
Luminance Y data is
Y = 0.30R + 0.59G + 0.11B (5)
It is expressed as
Cr = R−Y = 0.70 (RG) −0.11 (B−G) (6)
Cb = B−Y = 0.89 (B−G) −0.30 (R−G) (7)
And can be transformed. That is, the color difference signals of Cr and Cb can be obtained from the equations (6) and (7) from the RG and BG color data.
The control signal output from the chroma suppress signal generator 76 and the color difference signals of Cr and Cb are supplied to the chroma suppress, Hue / Gain adjustment, monotone effect processor 89, and when the chroma suppress control signal is supplied. When the signal is turned off and Hue / Gain adjustment is performed, the hue and saturation are varied, and the color signal is adjusted. When the monotone effect is activated, the color is erased and a monochrome image is set.
The color data output from the chroma suppress, Hue / Gain adjustment and monotone effect processing unit 89 is output (COUT), and is supplied to the encoder (not shown) together with the luminance signal output (YOUT) from the luminance signal processing unit 70 described above. Then, a component or composite image signal with good color reproducibility in which the IR region is corrected is generated.
As a result, when the reproduced color signal is color-evaluated using, for example, Macbeth Chart (registered trademark), if there is no IR estimation calculation (subtraction) processing, the IR component causes white (red; color number 15) to be white, for example. Is mixed, and the difference from medium red (Moderate red; color number 9) and magenta (magenta; color number 17) is reduced, making it difficult to identify. In addition to this, each color displayed on the Macbeth chart changes to a whole white color, and the original color is not reproduced. On the other hand, when there is an IR estimation calculation process, although the brightness of the entire Macbeth chart becomes slightly dark, red (color number 17), medium red (color number 9), etc. are reproduced in the original color, These colors can be clearly identified and the color reproducibility is improved.

Next, FIG. 5 shows a circuit configuration of an IR estimation processing circuit 150 which is an estimation calculation processing circuit constituting a part of a YLPF (luminance signal low-pass filter) 71 which is another embodiment.
In the IR estimation processing circuit 150, R (red) color data is input to the input terminal T1. The input terminal T 1 is connected to one input of the adder 151, one terminal of a SEL (selector; switch) 153, and the + (plus) terminal of the adder 155. The other input of the adder 151 is connected to the terminal T 2, and the output of the adder 151 is connected to one input of the adder 152. The other input of the adder 152 is connected to the output of the adder 171, and the output is connected to the input of the ¼ arithmetic unit 156.
The output of the ¼ arithmetic unit 156 is connected to the other terminals of the SEL (selector) 153, 161, 172, 181 and supplies a luminance signal Y (data).
The control signal R-SEL is supplied to the SEL 153 to switch the input R color data or Y data. The output of SEL153 is supplied to one input of the multiplier 154 and the multiplier other coefficient K _R to the input of 154 is supplied, the output of the adder 155 - is connected to the (negative) terminal, wherein The R ′ color data in which the R signal (data) is corrected is derived from the output after the subtraction.

Similarly, the terminal T2 to which GR color data is input is connected to one input of the SEL 161 and the other terminal of the adder 151, and is also connected to the + (plus) terminal of the adder 163. The other terminal of the SEL 161 is connected to the output of the quarter calculator 156 as described above, and is switched by the GR-SEL control signal to select GR color data or Y (luminance) data. The output of the SEL 161 is connected to one input of the multiplier 162. A coefficient _KGR is input to the other input of the multiplier 162, and an output is connected to the − (minus) terminal of the adder 163.
The + (plus) terminal of the adder 163 is connected to the terminal T2, and the corrected GR ′ color data of the GR signal (data) is derived from the output.
A terminal T 3 to which B color data is input is connected to one input of the adder 171, one input of the SEL 172, and the + (plus) terminal of the adder 174. The other input of the SEL 172 is connected to the output of the quarter calculator 156 and is switched by the B-SEL control signal to select B color data or Y data. The output of SEL172 is connected to one terminal of the multiplier 173, the other input of the multiplier 173 is input the coefficient K _B, the output of the adder 174 - is connected to the (negative) terminal. Then, the corrected B ′ color data of the B signal (data) is derived from the output of the adder 174.
T4 to which GB color data is input is connected to the other input of the adder 171 and further connected to one input of the SEL 181 and the + (plus) terminal of the adder 183. A GB-SEL control signal is supplied to the SEL 181 to select either GB color data or Y data.
The output of the SEL 181 is connected to one terminal of the multiplier 182, the coefficient K _GB is supplied to the other terminal of the multiplier 182, and the output is connected to the − (minus) terminal of the adder 183. The output of the adder 183 is derived from GB ′ color data in which the GB color signal (data) is corrected.

Next, the operation of the IR estimation processing circuit 150 will be described.
When the R color data is input to the terminal T1 and the GR color data is input from the terminal T2, the R + GR color data is derived from the output of the adder 151. When B color data is input from the terminal T 3 and GB color data is input from the terminal T 4 to the adder 171, B + GB color data is derived from the output, and this B + GB color data is output to the adder 152.
Since the R + GR color data and the B + GB color data from the adder 171 are supplied to the input of the adder 152 from the output of the adder 151, R + GR + B + GB color data is output from the output. The R + GR + B + GB color data is processed to 1/4 by the 1/4 calculator 156 at the next stage, output as Y (luminance) data, and supplied to the SELs 153, 161, 172, and 181.

When one terminal of the input of SEL 153 is selected by the control signal of R-SEL, for example, input R color data is selected and output. When this R color data is input to multiplier 154 is multiplied by the coefficient K _R, as a result, K R _{* R} is the adder 155 - is input to the (negative) terminal. On the other hand, since the R color data is input to the + (plus) terminal of the adder 155, the subtraction process is performed and R−K _R * R is output as the R ′ color data.
When the other terminal of the input of SEL 153 is selected by the R-SEL control signal, the input Y data is selected. When this Y data is input to the multiplier 154, it is multiplied by the coefficient K _R, and as a result, K _R * Y is input to the − (minus) terminal of the adder 155. On the other hand, since the R color data is input to the + (plus) terminal of the adder 155, the subtraction process is performed and R−K _R * Y is output as the R ′ color data.

Similarly, when one terminal of the input of SEL161 is selected by the control signal of GR-SEL, the input GR color data is selected and output. When this GR color data is input to the multiplier 162, it is multiplied by the coefficient K _GR, and as a result, K _GR * GR is input to the − (minus) terminal of the adder 163. On the other hand, since the GR color data is input to the + (plus) terminal of the adder 163, the subtraction process is performed, and GR-K _GR * GR is output as the GR ′ color data.
When the other terminal of the input of SEL 161 is selected by the GR-SEL control signal, the input Y data is selected and output. When this Y data is input to the multiplier 162, it is multiplied by the coefficient K _GR, and as a result, K _GR * Y is input to the − (minus) terminal of the adder 163. On the other hand, since the GR color data is input to the + (plus) terminal of the adder 163, the subtraction process is performed, and GR-K _GR * Y is output as the GR ′ color data.

Next, when one terminal of the input signal of SEL 172 is selected by the control signal of B-SEL, the input B color data is selected and output. When this B color data is input to multiplier 172 is multiplied by the coefficient K _B, As a result, the K B _{* B} of the adder 174 - is input to the (negative) terminal. On the other hand, since the B color data is input to the + (plus) terminal of the adder 174, the subtraction process is performed and B−K _B * B is output as the B ′ color data.
Further, when the other terminal of the input of SEL 172 is selected by the B-SEL control signal, the input Y data is selected and output. When this Y data is inputted to the multiplier 173 is multiplied by the coefficient _{K B,} as a _{result, K} B * Y is the adder 174 - is input to the (negative) terminal. On the other hand, since the B color data is input to the + (plus) terminal of the adder 174, the subtraction process is performed and B−K _B * Y is output as the B ′ color data.

When one terminal of the input of SEL 181 is selected by the control signal of GB-SEL, the input GB color data is selected and output. When this GB color data is input to the multiplier 182, it is multiplied by the coefficient K _GB, and as a result, K _GB * GB is input to the − (minus) terminal of the adder 183. On the other hand, since the GB color data is input to the + (plus) terminal of the adder 183, subtraction processing is performed and GB-K _GB * GB is output as GB ′ color data.
When the other terminal of the input of SEL 181 is selected by the GB-SEL control signal, the input Y data is selected and output. When this Y data is input to the multiplier 182, it is multiplied by the coefficient K _GB, and as a result, K _GB * Y is input to the − (minus) terminal of the adder 183. On the other hand, since the GB color data is input to the + (plus) terminal of the adder 183, subtraction processing is performed, and GB-K _GB * Y is output as GB ′ color data.

In this way, R, GR, B, GB color data and Y data are selected by the SELs 153, 161, 172, 181 by the control signals of R-SEL, GR-SEL, B-SEL, GB-SEL, Color data is corrected.
With the control signal described above, color data or Y data can be arbitrarily selected as data to be multiplied by the coefficients K _R , K _GR , K _B , and K _GB , and various combinations can be made.
Luminance (Y) data is generated from the corrected color data, and signal processing of the contour correction adder (contour correction Mix) 73, γ correction unit 74, and image effect processing unit 75 is obtained, and IR (infrared light) correction is performed. The luminance (Y) data thus output is output (YOUT).
The IR (infrared light) -corrected luminance output (YOUT) and color output (COUT) are supplied to an encoder (not shown) and calculated to output, for example, a component or composite image signal.

  When the luminance signal is generated by correcting the color signal (IR correction) and the component or composite image signal is generated, the display screen becomes darker than the case where the IR estimation subtraction process is not performed because the luminance is reduced. Clearly, it is better to perform IR estimation processing. Furthermore, the IR correction can improve the color reproducibility of the image on the display screen by improving the luminance reproducibility.

Next, another embodiment will be described. In the IR component estimation processing unit 101 that is an estimation calculation processing circuit, color data or Y data is arbitrarily selected as data to be multiplied by the correction coefficients K _R , K _GR , K _B , and K _GB used for the IR estimation processing calculation. The color correction data corresponding to the light source can be generated.
For example, in Equation (3), R color data is selected as a multiplied value for the R color data, and is expressed as R ′ = R−K _R * R. Further, with respect to the other color data, luminance (Y) data is used as a multiplied value, and B ′ = B−K _B * Y, GR ′ = GR−K _GR * Y, and GB ′ = GB−K _GB * Y. Represents.
In this example, only R ′ is modified using R color data. However, for other color data, either color data or luminance (Y) data is appropriately selected as a multiplication value and combined. It can also be combined.
As described above, it is necessary to determine the (correction) coefficient, the multiplied value, and the subtraction processing method according to the ambient light, the spectral characteristics of the sensor, and the like. In order to realize this, a light source estimation function may be added to the camera system.
In the light source estimation function, the spectrum of the light source is measured, for example, the light intensity with respect to the measurement wavelength is measured, normalized with respect to the peak value, and the relative light intensity with respect to each wavelength is stored in a storage device such as a memory. These measurements are performed on various light sources such as halogen lamps, fluorescent lamps, artificial sun, sunlight (around 8:00, around 14:00, around 15:30), data is accumulated, and necessary Make sure that it can be read.
At the time of photographing, light incident on each image sensor such as R, G, B is measured, and the relative value of the measured optical data is compared with the stored spectral data (relative spectral data of light intensity with respect to wavelength). Thus, the light source is specified. A light receiving element may be newly provided for measurement.
Further, when specifying a spectrum, it is not necessary to verify all wavelengths, and several characteristic wavelengths may be measured and evaluated, and a light source can be specified in a short time.
When the light source is specified in this way, a color data correction coefficient or a subtraction method suitable for the light source can be adopted by software processing or the like, and the color reproducibility is further improved.

Next, an IR estimation processing circuit 200 which is an estimation calculation processing circuit of another embodiment will be described with reference to FIG.
The main reason for removing the infrared cut filter is to improve sensitivity. If the luminance is subtracted at low illuminance, the display screen becomes dark and the characteristics deteriorate. Sensitivity can be improved without a sense of incongruity by mixing the processed signal and an unprocessed signal and using the signal that has not been processed gradually as a main component.
FIG. 6 shows a block diagram of an IR estimation processing circuit 200 that improves color reproducibility at low illuminance.
In the IR estimation processing circuit 200, luminance (Y) data is supplied from the PRE (preprocessing unit) 60 to the terminal T10. The terminal T10 is connected to the input of the luminance calculation processing unit 201, and the output is connected to the gain control unit 202. The output of the gain control unit 202 is connected to one terminal of the adder 203 at a terminal.
The terminal T10 is connected to the IR component estimation processing unit 210, and this output is connected to the input of the luminance calculation processing unit 211. The output of the luminance calculation processing unit 211 is connected to the input of the gain control unit 212, and the output is connected to the other input of the adder 203.
The adder 203 adds the data supplied from the gain control units 202 and 212, and outputs the processing result from the terminal T11. Here, the gain controllers 202 and 212 determine the mixing ratio in the dark when the gain (K _Mix ) is varied.
As the estimation calculation formula for each color data of the IR component estimation processing unit 210, for example, a calculation formula expressed by Formula (3) may be used.

Next, the operation of the IR estimation processing circuit 200 will be described.
First, the operation when bright is described. Color data is input from the terminal T10, and the luminance calculation processing unit 201 calculates luminance (Y) data. For example, Y = (R + GR + B + GB) / 4 is calculated. This calculated luminance (Y) data is supplied to the gain control unit 202. Assuming that the gain control coefficient K _Mix at the bright time is 0.9, Y * (1−0.9) = 0.1 * Y is output to the output of the gain control unit 202.
On the other hand, the IR component estimation processing unit 210 performs an estimation process based on, for example, the equation (3), performs an arithmetic process with the corrected color data, and generates luminance data Y ′.
The generated luminance data Y ′ is supplied to the gain control unit 212, and a result of Y ′ * K _Mix = 0.9 * Y ′ is obtained and output to the adder 203.
Since the output data of the gain control units 202 and 212 is supplied to the adder 203, the output is 0.1 * Y + 0.9 * Y ′.
In this way, when the image is bright, for example, the correction value output from the IR component estimation processing circuit 210 is 90% of the entire correction value, and the correction value only through the through luminance calculation processing unit 201 without the IR component estimation processing circuit 210 is overall. IR correction is performed as 10% of the above. That is, when it is bright, the estimation process using the IR component estimation processing circuit 210 is mainly performed.
Since the brightness data is large when it is bright, a decrease in brightness due to IR correction has little effect on color reproducibility.

Next, the operation of the IR estimation processing circuit 200 in the dark will be described.
The color data input from the terminal T10 is arithmetically processed by the luminance calculation processing unit 201. For example, luminance (Y) data is calculated as Y = (R + GR + B + GB) / 4. The calculated luminance (Y) data is supplied to the gain control unit 202. When the coefficient K _Mix for gain control in the dark is 0.1, the output of the gain control unit 202 has Y * (1- 0.1) = 0.9 * Y is output.
On the other hand, the IR component estimation processing unit 210 performs an estimation process based on Expression (3), and the luminance calculation processing unit 211 performs an arithmetic process to generate luminance data Y ′.
The generated luminance data Y ′ is supplied to the gain control unit 212, and a result of Y ′ * K _Mix = 0.1 * Y ′ is obtained and output to the adder 203.
Since the output data of the gain control units 202 and 212 is supplied to the adder 203, the output is 0.9 * Y + 0.1 * Y ′.
Here, the case where K _Mix is 0.1 has been described, but this coefficient can be set to any value between 0.0 and 1.0.
In this way, when dark, the correction value output from the IR component estimation processing unit 210 is reduced in proportion to the whole, and the correction value is obtained only through the through luminance calculation processing unit 201 without the IR component estimation processing unit 210. The ratio is increased and IR correction is performed. That is, since the luminance data is small when dark, the estimation process is mainly performed without the IR component estimation processing unit 210 circuit (luminance (Y) data does not decrease).
Therefore, by changing the value of the coefficient K _Mix in accordance with the size of the luminance (Y) data, image quality deterioration due to a decrease in the luminance component is prevented.
Further, the coefficient K _Mix can be gradually changed according to the brightness so that the color reproducibility does not feel uncomfortable when the brightness changes.

Next, another embodiment will be described with reference to FIG.
In FIG. 2, a preprocessing unit 13 is connected between the image sensor 12 and the AD converter 14. The pre-processing unit 13 is composed of CDS, S / H, AGC (Automatic Gain Control), etc., samples and holds the sensor signal supplied from the CCD image pickup device, takes out necessary data, and gains by AGC Is set to an appropriate level.
In the control of the AGC provided in the preprocessing unit 13, a gain is added (increased) when dark and the brightness (Y) signal level is set to the same level as when the brightness is not subjected to the IR estimation process. .
That is, the pre-processing unit 13 adds gain to the sensor signal output, relatively increases the luminance signal, outputs it to the AD converter 14 at the next stage, converts it into digital data, and performs the IR estimation process described above. By performing the above, it is possible to prevent the deterioration of color reproducibility even in the dark.

As described above, the signal processing method, the signal processing circuit, and the camera system using the signal processing method according to the present invention do not require an infrared light cut filter. The cost can be reduced as compared with a camera system in which a camera device provided with a transmission filter and an infrared light cut filter are replaced from the imaging path to improve color reproducibility when there is a lot of infrared light.
Further, the color reproducibility is improved by adding an infrared light (IR) estimation processing circuit to the luminance (Y) signal processing circuit before the color signal processing, instead of performing correction after the color signal processing. be able to.

It is a figure which shows the spectral characteristic of a CMOS image sensor. It is the figure which showed the block configuration of the camera system of this invention. It is a figure which shows the block of the signal processing circuit of the camera system of this invention. It is the figure which showed the block of the luminance signal processing circuit. It is the figure which showed the circuit structure of the infrared-light estimation processing circuit. It is the figure which showed the circuit structure of the other infrared-light estimation processing circuit.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Camera system, 11 ... Lens, 12 ... Image pick-up element (light receiving element), 13, 60 ... Pre-processing part, 14 ... AD (analog-digital) converter, 20 ... Signal processing part, 21 ... Correction processing part, 22 , 70 ... Luminance signal processing section, 23, 80 ... Color signal processing section, 61 ... Black detection section, 62, 151, 152, 155, 163, 171, 174, 183, 203 ... Adder, 63 ... Digital gain adjustment section 64 ... Shading correction unit 65 ... Delay line / defect correction unit 71 ... Luminance signal low pass filter (YLPF) 72 ... Vertical / horizontal contour correction unit 73 ... Contour correction adder (Mix) 74,86 ... Gamma (Γ) correction unit, 75... Image effect processing unit, 76... Signal generation unit for chroma suppress, 81 .. color separation / clamp processing unit, 82 .. color signal low-pass filter (CLPF), 3 ... RGB matrix, 84 ... white balance (WB) adjustment unit, 85 ... color signal optical detector (COPD), 87 ... RG / BG conversion, color false suppression processing unit, 88 ... Cr / Cb (color difference signal) ) Generator, 89 ... chroma suppress, Hue / Gain adjustment, monotone effect processing unit, 101,210 ... IR (infrared light) component estimation processing unit (circuit), 102, 201, 211 ... luminance calculation processing unit, 150, DESCRIPTION OF SYMBOLS 200 ... IR estimation processing circuit, 153,161,172,181 ... Selector (SEL), 154,162,173,182 ... Multiplier, 156 ... 1/4 arithmetic unit, 202,212 ... Gain control part.

Claims (18)

A plurality of corrected color data are generated by performing, for each color , a value obtained by subtracting a value obtained by multiplying luminance data including an infrared component or color data including an infrared component by a predetermined coefficient from the color data including the infrared component , A signal processing method of generating a luminance signal from a plurality of generated corrected color data and generating an image signal using the generated luminance signal and a color difference signal generated from a plurality of color data including the infrared components . The signal processing method according to claim 1, wherein the calculation is performed for each color by selectively selecting color data including the infrared component and luminance data including the infrared component by a control signal. In the calculation for each color, for a plurality of predetermined coefficients determined for each color, arbitrarily select color data including the infrared component or luminance data including the infrared component as data to be multiplied to each predetermined coefficient, A method that uses only color data including the infrared component in all of a plurality of colors, a method that uses only luminance data that includes the infrared component in all of a plurality of colors , a color data that includes the infrared component in some colors, and the remaining colors The signal processing method according to claim 1, wherein the calculation method is switched to any one of methods using luminance data including the infrared component . Generating a first luminance data from the color data including a plurality of the infrared component of each color, the value obtained by multiplying a predetermined coefficient to the color data including the infrared component, a calculation for subtracting from the color data including the infrared light component Second luminance data is generated from a plurality of corrected color data generated by performing for each color, and the first luminance data is converted to the second luminance at a ratio according to the size of the first luminance data. The signal processing method according to claim 1, wherein the luminance signal is generated by mixing with data. 5. The ratio is variable so that the ratio of the first luminance data component in the luminance signal increases as the level of the first luminance data decreases during the mixing. Signal processing method. The infrared light component mixed in the color data from the pixel is corrected by the estimation arithmetic processing circuit for each color filter of the pixel, a luminance signal is generated using the corrected color data, the luminance signal, and the pixel A signal processing circuit for generating an image signal using a color difference signal generated from color data from
The estimation calculation processing circuit includes:
A selection circuit that inputs color data and luminance data, and selectively selects the color data and luminance data by a control signal;
A multiplier that multiplies the output signal output from the selection circuit by a predetermined coefficient;
A signal processing circuit comprising: an arithmetic unit that subtracts a signal output from the multiplier from the color data and outputs the corrected color data. The signal processing circuit according to claim 6, wherein the estimation calculation processing circuit uses the value obtained by multiplying the luminance data by a predetermined coefficient from the color data of each color as the color data after correction for each color. The signal processing circuit according to claim 6, wherein the estimation calculation processing circuit uses, as the corrected color data, a value obtained by subtracting a value obtained by multiplying color data of each color by a predetermined coefficient from color data corresponding to each color. The estimation calculation processing circuit performs the multiplication and subtraction for each color, and when executing the multiplication and subtraction, the color data as data to be multiplied by each predetermined coefficient for a plurality of predetermined coefficients determined for each color Alternatively, the luminance data can be arbitrarily selected, a method using only color data for all of a plurality of colors, a method using only luminance data for all of a plurality of colors, a method using color data for some colors and a luminance data for the remaining colors. The signal processing circuit according to claim 6, wherein the calculation method is switched to any one of the above. The estimation calculation processing circuit generates first luminance data from a plurality of color data input for each color filter, and generates a second luminance from the plurality of corrected color data obtained from the output of the calculator. The data is generated, and the first luminance data is mixed with the second luminance data at a ratio corresponding to the size of the first luminance data to generate the luminance signal. The signal processing circuit according to one item. The ratio is variable so that the ratio of the first luminance data component in the luminance signal increases as the level of the first luminance data decreases during the mixing. Signal processing circuit. A plurality of light receiving elements having sensitivity up to the near infrared region, and an imaging unit that captures an input image with the plurality of light receiving elements via a plurality of color filters not including an infrared blocking filter;
A signal processing unit that performs image processing on an imaging signal output from the imaging unit, and outputs the processed image signal;
Have
The signal processing unit generates a plurality of corrected color data by performing, for each color filter, an operation of subtracting a value obtained by multiplying luminance data or color data obtained from the imaging signal by a predetermined coefficient from the color data. A camera that executes an estimation calculation process, generates a luminance signal from a plurality of generated corrected color data, and generates an image signal using the generated luminance signal and a color difference signal generated from the plurality of color data system. The camera system according to claim 12, wherein in the estimation calculation process, the signal processing unit selectively selects the color data and the luminance data by a control signal and performs the calculation for each color. In the estimation calculation process, the signal processing unit is configured to calculate the color data or the luminance data as data to be multiplied by each predetermined coefficient with respect to a plurality of predetermined coefficients determined for each color filter in the calculation for each color filter. A method that uses only color data for all color filters, a method that uses only luminance data for all color filters, a method that uses color data for some color filters and uses luminance data for the remaining color filters The camera system according to claim 12, wherein the calculation method is switched to any one of the above. In the estimation calculation process, the signal processing unit generates first luminance data from the plurality of color data for each color filter, and subtracts a value obtained by multiplying the color data by a predetermined coefficient from the color data. The second luminance data is generated from a plurality of corrected color data generated by performing for each color, and the first luminance data is converted to the second luminance data at a ratio according to the size of the first luminance data. The camera system according to claim 12, wherein the luminance signal is generated by mixing with luminance data. The ratio is variable so that the ratio of the first luminance data component in the luminance signal increases as the level of the first luminance data decreases during the mixing. Camera system. The estimation calculation process has a light source estimation function, and the light source estimation function includes the predetermined coefficient and a calculation method according to a combination of the color data or the luminance data multiplied by the predetermined coefficient for each color filter. The camera system according to claim 12, wherein the selection is made according to the light source specified by. The camera system according to claim 12, wherein the camera system adjusts the analog AGC so that a signal input to a signal processing unit that performs the estimation calculation process is equal to or higher than a predetermined level in an unsaturated state when the illuminance is equal to or higher than a predetermined illuminance. .

Images