JP6536000B2 - Shooting device - Google Patents

Description

  The present invention relates to image processing in a photographing apparatus such as a digital camera, and more particularly to color correction processing of an image in which color saturation occurs.

  In a digital camera or the like, an imaging element provided with an R, G, B color filter array is provided, and color saturation may occur depending on the spectral sensitivity characteristic of the imaging element. For example, when photographing a subject of a specific color such as red or green with strong light, the pixel signal of that color is saturated, and when image processing such as white balance adjustment is performed, a region of a color saturation portion different from the original color Cause coloration.

  In order to prevent coloring, a method is known in which the dynamic range of the imaging device is expanded to obtain color information (see Patent Document 1). There, the actual output value of the saturated color signal is estimated from the non-saturated color signal. As a result, there is no coloring even after white balance adjustment, and the color balance of R, G, B in the color saturated portion does not break.

Patent No. 5091781 gazette

  When pixel signals of all colors are saturated in a partial region of a captured image, it is not possible to estimate the output value of the saturated pixel signal from pixel signals of other colors. At this time, if image processing such as multiple exposure or the like is performed to lower the entire luminance level of a captured image, color information is not interpolated and coloring occurs in a color saturated portion.

  Therefore, even if image processing for reducing the brightness of the subject image is performed, it is necessary to carry out color correction processing so as not to cause coloring.

  The imaging apparatus according to the present invention superposes an image composed of a plurality of R, G, B color signals for one frame based on pixel signals read out from an image sensor having a color filter array according to the multiple exposure method, and generates a composite image An image processing unit, and a color correction processing unit that suppresses output levels of color signals other than the reference color signal with respect to R, G, and B color signals for a predetermined one frame before overlapping images. .

  In the present invention, the color correction processing unit performs white balance gain values for R, G, B color signals of a color proper image area in which color saturation does not occur among predetermined one frame R, G, B color signals. Are set, and the other of the R, G, and B color signals of the appropriate image area is based on the reference value corresponding to the gain ratio between the reference color signal having the highest output level and the other color signal. Suppress the output level or value of the color signal.

  The color correction processing unit can remove a portion higher than a reference value determined according to the gain ratio with respect to other color signals. For example, clip processing can be performed based on a reference value. Further, the color correction processing unit can extract a color appropriate image area in which none of the R, G and B color signals are saturated out of R, G and B color signals for one frame. In addition, the image processing unit can perform WB adjustment with the white balance gain value set before the superposition after the superposition of the images. When two captured images are superimposed, the color correction processing unit can separately obtain gain ratios for the first and second one-frame R, G, and B color signals.

  An imaging apparatus according to another aspect of the present invention performs image processing for performing shading correction on an image formed of a plurality of R, G, and B color signals based on pixel signals read from an image sensor having a color filter array. And a color correction processing unit that suppresses output levels of color signals other than the reference color signal with respect to predetermined one frame R, G, and B color signals before shading correction, and a color correction processing unit Sets the white balance gain value for each of the R, G, B color signals in the color proper image area where color saturation does not occur among the predetermined one frame R, G, B color signals, and sets the appropriate image area The output level of another color signal is suppressed based on the reference value according to the value more than the gain ratio of the reference color signal having the highest output level among the R, G, and B color signals to the other color signal. .

  An image processing apparatus according to another aspect of the present invention is configured to receive other color signals relative to a reference color signal with respect to a plurality of R, G, B color signals corresponding to one frame based on pixel signals read out from an image sensor having a color filter array. And a color adjustment processing unit that suppresses the output levels of other color signals in accordance with the determined output levels for R, G, and B color signals for one predetermined frame. The setting unit sets white balance gain values for R, G, and B color signals of the color proper image area in which color saturation does not occur among the predetermined one frame of R, G, and B color signals, Upper limit output of the other color signal based on the reference value according to the value more than the gain ratio of the reference color signal with the highest output level among the R, G, B color signals of the proper image area and the other color signal Determine the level.

  The imaging method according to another aspect of the present invention superimposes an image composed of a plurality of R, G, B color signals for one frame based on pixel signals read out from an image sensor having a color filter array according to a multiple exposure method. A photographing method for generating a composite image and suppressing output levels of color signals other than the reference color signal with respect to R, G and B color signals for a predetermined one frame before overlapping the images, White balance gain values are set for R, G, and B color signals of the color proper image area where color saturation does not occur among R, G, and B color signals of one frame of R Among the G and B color signals, the output level of the other color signal is suppressed based on the reference value corresponding to the value greater than the gain ratio of the reference color signal having the highest output level to the other color signal.

  The imaging method according to another aspect of the present invention performs shading correction on an image composed of a plurality of R, G, B color signals based on pixel signals read from an image sensor having a color filter array, A photographing method for suppressing the output levels of color signals other than the reference color signal with respect to the predetermined one frame of R, G, B color signals before shading correction, which is for the predetermined one frame R, G, White balance gain values are set for R, G and B color signals of the color proper image area where color saturation does not occur among the B color signals, and among the R, G and B color signals of the proper image area The output level of the other color signal is suppressed based on the reference value according to the value greater than the gain ratio of the reference color signal having the highest output level to the other color signal.

  An image processing method according to another aspect of the present invention is a method for processing an image according to an embodiment of the present invention, in which a plurality of R, G, B color signals for one frame based on pixel signals read from an image sensor having a color filter array And an image processing method for suppressing output levels of other color signals in accordance with predetermined output levels for R, G and B color signals for one predetermined frame. White balance gain values are set for R, G, and B color signals in the color proper image area where color saturation does not occur among R, G, and B color signals for a predetermined one frame, and R for the appropriate image area The upper limit output level of another color signal is determined based on a reference value corresponding to a value greater than the gain ratio of the reference color signal having the highest output level among the G, B, and B color signals to the other color signal.

  According to the present invention, in a photographing device such as a camera or an image processing device, it is possible to obtain a photographed image in which coloration does not occur even if color saturation occurs.

FIG. 1 is a block diagram of a digital camera according to a first embodiment. FIG. 6 is a diagram showing an output ratio of R, G, B color signals in a multiple exposure shooting process. It is a figure showing a photography picture of superposition object. 5 is a flowchart of multiple exposure shooting including a color correction process performed by a controller. It is a flowchart of the shading correction process including the color correction process in 2nd Embodiment.

  Hereinafter, the digital camera according to the present embodiment will be described with reference to the drawings.

  FIG. 1 is a block diagram of a digital camera according to the first embodiment.

  Here, the digital camera 10 is configured as a compact camera, and according to the operation for the release button, the cross button, the mode dial (all not shown), etc., the photographing operation, the reproduction of the recorded image, the mode setting by the user, etc. To be done. The user can select the multiple exposure mode, the shading setting mode for image editing (processing), and the like by operating the mode dial. An LCD monitor 60 is provided on the back of the camera.

  The controller 30, which includes a CPU, outputs control signals to the exposure control unit 32, the image signal processing circuit 40, etc., and the release operation detected by the release switch 37, the mode dial operation detected by the mode dial switch 42, and the cross button switch Based on the operation of the cross button detected by the control unit 44, operation control of the entire camera such as exposure control and photographing / recording operation is performed. The program for camera operation control is stored in a memory such as a ROM (not shown).

  When the shooting mode is set with the power on, light from the subject passes through the shooting optical system 12, and a subject image is formed on the image sensor 14. The image sensor 14 is configured by a CCD, a CMOS or the like, and the color filter array 14A is disposed opposite to the light receiving surface of the image sensor 14. In the color filter array 14A, the color elements of R, G, and B are arranged in a checkered pattern or Bayer pattern.

In the photographing mode, in order to display a through image on the LCD monitor 60, pixel signals are read from the image sensor 14 at a predetermined frame rate. The read R, G, B pixel signals are sent to the image signal processing circuit 40 via the AFE circuit 16. One of R, G, and B color signals is read out from each pixel of the image sensor 14 in accordance with the color filter array 14A configured by R, G, and B color elements.

  The image signal processing circuit 40 configured by programmable circuits such as DSP and FPGA generates color signals of R, G and B for each pixel by color interpolation processing, and performs gain processing, color conversion processing, white balance adjustment, etc. Is applied to the R, G and B color signals. Thereby, R, G, B color image signals are generated. An LCD driver (not shown) drives the LCD monitor 60 based on the color image signal, whereby a real-time moving image is displayed on the LCD monitor 60 as a through image.

  When the release button is pressed halfway, the contrast AF process is performed based on the pixel signal read from the image sensor 14. The AF lens 34 drives the focusing lens in the photographing optical system 12. Along with the focusing operation by the contrast AF process, the brightness of the subject image is detected based on the pixel signal read from the image sensor 14, and the exposure value is calculated.

  When the release button (not shown) is fully pressed, the exposure control unit 32 performs exposure control by driving a shutter, a diaphragm, etc. not shown. Thus, pixel signals for one frame are read out from the image sensor 14. The image signal processing circuit 40 generates still image data based on the read pixel signal of one frame, and records the image data in the memory card 46 in a compressed / non-compressed state. In the image recording process, it is possible to record still image data as RAW data. When the reproduction mode is set, the image stored in the memory card 46 is displayed on the LCD monitor 60.

  When the multiple exposure mode is set by the mode dial, a photographing operation by multiple exposure is performed. The user shoots a different subject or shoots a moving object according to a predetermined number of times of exposure (here, twice). The image signal processing circuit 40 performs superposition processing on two image data to generate one photographed image. Here, the averaging method is applied, and in superposition of the images, the luminance levels are averaged using weighting coefficients.

  Then, white balance adjustment processing is performed on the composite image obtained by the superposition. That is, the color balance is adjusted so that the output ratio of R: G: B is 1: 1: 1 with respect to the white subject. On the other hand, in the image signal processing circuit 40, the first and second one frame R and G are provided in order to prevent coloring from occurring in the composite image finally generated before overlaying processing of two images. Color correction processing is performed on the B, and B color signals. Here, a composite image is generated and recorded by the RAW data.

  FIG. 2 is a diagram showing the output ratio of R, G and B color signals in the multiple exposure shooting process. FIG. 3 is a view showing a photographed image of a superposition target. The color correction processing for preventing coloring at the time of multiple exposure will be described with reference to FIGS.

  In FIG. 2, the output levels of R, G and B color signals at the first and second photographing are displayed graphically, and the output levels of R, G and B color signals at the second photographing are shown for each processing. It shows. Here, it is assumed that the image obtained by the first shooting has a relatively low brightness, and the image obtained by the second shooting has a relatively high brightness.

  FIG. 3 shows an image IM at the time of second shooting. The image IM is an image of the person taken indoors, and the window W and the indoor illumination light L are taken out together with the person. The image area (color correct image area) A1 showing the inside of the room is an image based on the proper exposure by the light of the indoor illumination light L, and all the R, G and B color signals are not saturated (see FIG. 2).

  On the other hand, due to the difference between indoor and outdoor lighting and the influence of strong natural light outside the room, the image of the outside scene projected through the window W turns white and the R, G and B color signals in the image area A2 where the window W is displayed Everything is saturated (see Figure 2).

  When such an image IM is used as the second image and is superimposed on the first image having low luminance, the brightness of the composite image obtained by superposition is 2 since the exposure amount is averaged. It is lower than the brightness of the first image.

  When white balance adjustment is performed on the entire combined image, white balance gain values are set for the R, G, and B color signals of the entire image in an image area not having color saturation. Therefore, in the area where color saturation occurs, the output level ratio of R: G: B does not become 1: 1: 1 even when multiplied by the white balance gain value. As a result, the area saturated in color in the second captured image is colored.

  Therefore, in the present embodiment, the color balance of R, G, B of the entire image is adjusted before the images are superimposed, and the output levels of the other color signals are suppressed based on the color signal which tends to be saturated.

  If it demonstrates concretely using FIG. 2, the output level of the color signal of G is the largest among R, G, B, for example in image area A1 which is not color-saturated. Therefore, considering the white balance gain value of this image area A1, the white balance gain value gg of G becomes the reference value (= 1), and the white balance gain values gr: gg, gb (here, 1.7: 1: 2.5) can be set.

  Here, considering the condition that the color saturation (white color) state of the image area A1 is maintained by white balance adjustment processing after superposition of images, that is, not coloring, the output level is highest in the image area A1 without color saturation. When the saturated R and B color signal levels are suppressed and limited to a low output level based on the saturation level of the G color signal, the R, G, B color signals are adjusted after white balance adjustment for the composite image. The upper limit can be reached so that is 1: 1: 1, and the white state is maintained also in the composite image.

  Since white balance gain values gr: gg and gb in the image area A1 which are not saturated become factors determining the color balance at the time of white balance adjustment after composite image generation, for R and B color signals Ratio of white balance gain values (here, the ratio of gg to gr and gb is called gain ratio) gg / gr and gg / gb multiplied by the color signal levels of R and B at minimum ensure white balance adjustment All the R, G and B color signals reach the saturation level. As a result, the color signals of R, G and B can be prevented from being colored even after white balance adjustment.

  Therefore, the gain ratio of the G color signal to the R and B color signals is gg / gr (= 1 / 1.7), gg / for the entire R, G, B color signals of one frame at the second shooting. According to gb (= 1 / 2.5), the output level (ratio) of R, B color signal to G color signal is determined as a reference value (threshold), and the output level of R, B color signal having a value higher than the reference value Is corrected to the reference value, that is, if the clipping process is performed so as to remove the portion above the reference value, the white state of the image area A2 in the second captured image is maintained after the white balance adjustment of the composite image, Do not color. The same effect can be obtained by setting a value (<1) larger than the gain ratio as a reference value.

  On the other hand, the same R and B color signals may be subjected to the clip processing even for an image area including no image area A1 and an image area where color saturation does not occur or an image area where only a part of color signals is not saturated. The color balance is properly maintained by white balance adjustment processing after composite image generation.

  In FIGS. 2 and 3, it is assumed that the image obtained by the first shooting has a low brightness level and the image obtained by the second shooting has a high brightness level, but the opposite may be true. In addition, even in the first shooting, color saturation may occur in some image areas. Therefore, in the present embodiment, the color correction process is performed on any of the images at the time of the first and second photographing.

  Here, it is considered that the G color signal has the highest sensitivity and the output level is relatively high and saturation is likely to occur, and the color correction coefficient is set based on the G color signal. When the output level of the color signal is relatively high, the color correction coefficient may be determined based on the color signal. Specifically, when the white balance gain value is obtained for an image area in which color saturation does not occur, the color signal having the white balance gain value of the minimum value (= 1) may be used as a reference.

  FIG. 4 is a flowchart of multiple exposure shooting including color correction processing performed by the controller 30.

  In step S101, an image area in which R, G, and B are not all saturated is extracted from the captured image, and provisional white balance gain values of R, G, and B in that area are calculated. Various methods can be applied to the selection and extraction method of the area size. For example, a threshold is provided for each of R, G, and B, and all R, G, and B pixels in a certain area are in a valid range. Only to extract the white balance gain as an element to calculate. Further, as a method of calculating the white balance gain value, for example, the white balance gain value can be obtained from the ratio of the integrated values of R: G: B.

  Once the white balance gain value is determined, the gain ratio is determined according to the gain ratio, which is the ratio of the gain value of the color signal (eg, G) with the highest output level to the gain value of the other color signal (eg, R, B). The reference value is determined as the upper limit value of another color signal (S102). Then, clipping processing based on the reference value is executed so that the output levels of the other color signals do not exceed the reference value (S103).

  After color correction processing is performed on each of the first photographed image and the second photographed image, the two images are superimposed by averaging (S105). Then, the composite image obtained by the superposition is subjected to white balance adjustment processing (S106). The image signal processing circuit 40 is provided with an image memory for temporarily storing a first shot image and a second shot image.

  As described above, according to the present embodiment, in the digital camera provided with the multiple exposure function, white in the captured image of one frame is targeted for the image area in which not all color signals of R, G, B are saturated. Determine the balance gain value. Then, the clip processing is performed based on the reference value corresponding to the gain ratio of the color signal having the highest output level (the smallest white balance gain value) and the other color signal. After the color correction process, the image superposition process is performed, and the white balance adjustment process is performed on the composite image.

  By such color correction processing, it is possible to generate a composite image without coloring even in the case of recording with RAW data. Also, by recording the composite image before white balance adjustment in the memory, it is possible to perform image editing and the like with RAW data without loss of color information.

  When three or more photographed images are superimposed, it is sufficient to superimpose a newly photographed image on the composite image using a weighting coefficient. After the color correction process is performed on the first, second, and third captured images, the color correction process may be performed only on the latest captured image.

  Next, a second embodiment will be described using FIG. In the second embodiment, color correction processing is performed when performing shading correction as image processing.

  In the shading correction, gradation processing is applied so as to shade the periphery of the photographed image. At this time, since the luminance levels of the R, G, and B color signals are partially lowered, if a color saturation portion occurs in the image area of the gradation target, coloring occurs after shading correction and white balance adjustment. Therefore, color correction processing is performed before shading correction is performed.

  FIG. 5 is a flowchart of shading correction processing including color correction processing.

  The execution of steps S201 to S203 corresponds to the execution of steps S101 to S103 in FIG. That is, an image area without color saturation is extracted, provisional white balance gain values of R, G, and B are determined, and clipping processing is performed based on a reference value corresponding to a gain ratio obtained from the white balance gain values. Apply. Then, after performing conventionally known shading correction processing, white balance adjustment processing is performed (S204, S205).

  Color correction processing may be performed on image processing other than the first and second embodiments (for example, interval photographing). In particular, when performing exposure control and image processing that lowers the luminance level of a captured image, color correction processing is effective. The color correction process may be performed in an image processing apparatus or the like having no photographing function other than a photographing apparatus such as a camera.

10 digital camera 14 image sensor 30 controller (color correction processing unit, setting unit, color adjustment processing unit)
40 Image signal processing circuit (color correction processing unit, image processing unit)

Claims (11)

An image processing unit that generates a composite image by superimposing images composed of a plurality of R, G, B color signals for one frame based on pixel signals read out from an image sensor having a color filter array according to a multiple exposure method;
A color correction processing unit that suppresses output levels of color signals other than the reference color signal with respect to predetermined one frame of R, G, and B color signals before overlapping the images;
The color correction processing unit generates white for the R, G, B color signals of the color proper image area where color saturation does not occur in the photographed image composed of R, G, B color signals for one predetermined frame. A balance gain value is set, and a reference value corresponding to a value that is equal to or greater than the gain ratio of the reference color signal having the highest output level among the R, G and B color signals of the appropriate color image area An image pickup apparatus characterized by suppressing the output level of another color signal of the photographed image on the basis of.   The photographing apparatus according to claim 1, wherein the color correction processing unit excludes a portion having a reference value or more determined according to a gain ratio with respect to another color signal.   The color correction processing unit extracts a color appropriate image area in which any one of R, G and B color signals is not saturated out of R, G and B color signals for one frame. The imaging device according to any one of to 2.   The photographing apparatus according to any one of claims 1 to 3, wherein the color correction processing unit separately obtains gain ratios for first and second R, G, B color signals for one frame. .   5. The apparatus according to any one of claims 1 to 4, wherein the image processing unit performs WB adjustment with a white balance gain value set before superposition after superposition of images.   The imaging apparatus according to any one of claims 1 to 5, wherein the image processing unit generates a composite image by using RAW data. An image processing unit that performs shading correction on an image composed of R, G, B color signals for one frame based on pixel signals read out from an image sensor having a color filter array;
Before shading correction, and a said one frame R, G, to B color signals, color suppressing output level of the other color signals other than the reference color signal correction processing unit,
The color correction processing unit, said one frame R, G, R color proper picture area color saturation does not occur in the B color signals, G, to B color signals, set the white balance gain value, respectively The other color based on the reference value according to the value more than the gain ratio of the reference color signal having the highest output level among the R, G, B color signals of the color proper image area to the other color signal An imaging device characterized by suppressing an output level of a signal. A setting unit configured to set output levels of other color signals relative to the reference color signal with respect to R, G, B color signals for one frame based on a pixel signal read out from an image sensor having a color filter array;
A color adjustment processing unit that suppresses output levels of other color signals according to a predetermined output level for the R, G, and B color signals for one frame;
A white balance gain value for R, G, B color signals of a color proper image area in which color saturation does not occur in the photographed image in which the setting unit comprises the one frame of R, G, B color signals Are set, and a reference value corresponding to a value greater than the gain ratio of the reference color signal having the highest output level among the R, G, and B color signals of the color proper image area to the other color signals is set. the upper limit output level of the other color signals of the photographed image constant because,
The image processing apparatus, wherein the color adjustment processing unit suppresses an output level of another color signal of the photographed image according to the upper limit output level . According to the multiple exposure method, based on pixel signals read out from an image sensor having a color filter array, images composed of a plurality of R, G, B color signals for one frame are superimposed to generate a composite image,
A photographing method for suppressing output levels of color signals other than a reference color signal with respect to R, G, B color signals for a predetermined one frame before superposing images.
White balance gain values are set for R, G, and B color signals in the color appropriate image area where color saturation does not occur in the captured image consisting of R, G, and B color signals for one predetermined frame. The photographed image is based on a reference value corresponding to a value greater than or equal to a gain ratio of the reference color signal having the highest output level among the R, G, B color signals of the color proper image area to the other color signal. A photographing method characterized by suppressing output levels of other color signals. Shading correction is performed on an image composed of R, G, B color signals for one frame based on pixel signals read out from an image sensor having a color filter array;
Before the shading correction, the one frame R, G, to B color signal, the output level of the other color signals other than the reference color signal to a suppressing imaging method,
Wherein one frame R, G, R color proper picture area color saturation does not occur in the B color signals, G, to B color signals, set white balance gain values, respectively, of the color proper image area The output level of the other color signal is suppressed based on the reference value according to the value of the reference color signal having the highest output level among the R, G, B color signals and the gain ratio of the other color signal or more. A photographing method characterized by Setting the output levels of the other color signals with respect to the reference color signal for R, G, B color signals for one frame based on the pixel signal read out from the image sensor having the color filter array ,
It is an image processing method for suppressing the output levels of other color signals according to a predetermined output level for the R, G, B color signals for one frame,
Wherein one frame R, G, R color proper picture area color saturation does not occur in the B color signals composed captured image, G, to B color signals, set white balance gain values, respectively, Of the R, G, B color signals of the color proper image area, the value of the photographed image is determined based on a reference value corresponding to a value greater than or equal to the gain ratio of the reference color signal to the other color signal . limit the output level of the other color signals constant because,
An image processing method comprising: suppressing output levels of other color signals of the photographed image according to the upper limit output level .

Images