JP6281682B2 - Image data generation apparatus, image data generation method and program - Google Patents

Description

  The present invention relates to a technique for obtaining a captured image using a plurality of image sensors.

  Conventionally, for example, in Patent Document 1 below, in a configuration including a single RGB imaging device and a monochrome imaging device as an imaging device, in order to enable imaging with a high dynamic range, an RGB signal of a single-plate RGB imaging device, A technique is described in which luminance information included in both the luminance signal of the monochrome image sensor is reflected as color information and luminance information of a final captured image through a plurality of stages of signal processing.

JP2011-223221A

  However, although the above-described technique enables imaging with a high dynamic range, different imaging signals obtained by the two imaging elements cannot be processed according to various imaging situations, and may not always be possible depending on the imaging conditions. There has been a problem that high-quality captured images are not always obtained.

The present invention has been made in view of such conventional problems, and an object of the present invention is to provide an image data generation device, an image data generation method, and a program capable of obtaining high-quality captured images in various shooting situations. And

In order to solve the above problems, in the present invention, an image generation unit that generates image data using two image pickup signals obtained by an image pickup device through a color filter and a colorless or single color filter; A luminance value of each pixel in the generation process of the luminance value of each pixel in the image data of the image generation unit according to the determination unit that determines the brightness of the subject and the brightness of the subject determined by the determination unit; Is replaced with a value obtained by averaging the luminance value of the imaging signal obtained by the imaging device via the color filter and the luminance value of the imaging signal obtained by the imaging device via the colorless or single color filter as a replacement value. And control means for controlling to generate image data.

  According to the present invention, it is possible to obtain high-quality captured images in various shooting situations.

It is a block diagram of a digital camera exemplified as an imaging apparatus of the present invention. 1A and 1B are diagrams illustrating a first embodiment, where FIG. 1A is a block diagram illustrating an imaging unit, FIG. 1B is a diagram illustrating a color filter of a first imaging element, and FIG. 2C is a color of a second imaging element. It is a figure which shows a filter. It is a flowchart which shows the imaging process in 1st Embodiment. It is a figure which shows the color filter of the 2nd image pick-up element in 2nd Embodiment. It is a flowchart which shows the imaging process in 2nd Embodiment. It is a figure which shows 3rd Embodiment, Comprising: (a) is a block diagram which shows a digital camera imaging part, (b) is a figure which shows a rotation filter board. It is a flowchart which shows the imaging process in 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described.

(Embodiment 1)
FIG. 1 is a block diagram showing an outline of an electrical configuration of a digital camera 1 exemplified as an imaging apparatus of the present invention.

  As shown in FIG. 1, the digital camera 1 includes a control unit 2, a lens unit 3, an imaging unit 4, an image storage unit 5, a program storage unit 6, a display unit 7, an operation unit 8, and a power supply unit 9. I have.

  The control unit 2 includes a CPU (Central Processing Unit) and its peripheral circuits, and a working internal memory such as a RAM (Random Access Memory), and the digital camera according to the control program stored in the program storage unit 6. 1 part is controlled.

  The program storage unit 6 is configured by a nonvolatile memory such as a flash memory in which stored data can be rewritten as needed. In the program storage unit 6, in addition to the above control program, a program for causing the control unit 2 to perform AE (Auto Exposure) control, AF (Auto Focus) control, AWB (Auto White Balance) control, and the like, and AE Various data such as data constituting a program diagram showing combinations of shutter speed, ISO sensitivity, and aperture value used in the control are also stored.

  The lens unit 3 includes a lens group including a focus adjustment lens and a zoom lens, a motor that drives the lens group, a diaphragm, and an actuator that opens and closes the diaphragm to adjust the opening.

  The imaging unit 4 images the subject via the lens group of the lens unit 3, generates image data of the subject, and supplies the image data to the control unit 2. FIG. 2A is a block diagram illustrating a configuration of the imaging unit 4.

  The imaging unit 4 mainly includes a half prism 41 that divides incident light from the lens unit 3 into two paths, a first imaging element 42 that receives one of the divided incident light, and a first prism that receives the other incident light. 2 image sensors 43, and an image processing engine 45 that processes image signals obtained by the first and second image sensors 42 and 43. Although not shown in FIG. 4A, the imaging unit 4 includes drive circuits for the first and second imaging elements 42 and 43.

  The first image sensor 42 and the second image sensor 43 are, for example, CMOS (Complementary Meta 10xide Semiconductor) type image sensors or CCDs (Charge Coupled Devices).

  The first image sensor 42 is provided with a color filter 42a in which the RGB colors shown in FIG. 2B are arranged in a Bayer array. An image signal (hereinafter referred to as an image signal) output from the first image sensor 42 is provided. , Referred to as a first image pickup signal) is a color image pickup signal including RGB color components.

  The second image sensor 43 is provided with a colorless color filter 43a in which the color of each pixel is indicated by N in FIG. 2C for convenience, and an image signal output from the second image sensor 43. (Hereinafter referred to as the second imaging signal) is a monochrome imaging signal having only a luminance component.

  The first image sensor 42 is always driven while the digital camera 1 is set to the recording mode for shooting, and images the subject. The second image sensor 43 is basically only for shooting. Driven to image the subject.

  In addition, the color filter 42a provided in the first image sensor 42 may have a color arrangement other than the Bayer arrangement as long as it is used in a normal single-plate imaging method in which imaging is performed using a single image sensor. Can be used. The half prism 41 can be replaced with a half mirror.

  The image processing engine 45 processes the first and second imaging signals obtained by the first and second imaging elements 42 and 43 in accordance with instructions from the control unit 2. The image processing engine 45 mainly includes first and second AFEs (Analog Front Ends) 46, which perform predetermined analog processing such as amplification of the first imaging signal and the second imaging signal, and conversion into digital signals individually. 47 and a DSP (Digital Signal Processor) 48.

  The DSP 48 includes a microcomputer having a function specialized for digital signal processing, a ROM (Read Only Memory) in which a program is stored, a memory, an arithmetic circuit, and the like. The DSP 48 performs a demosaic process for generating an RGB signal by interpolating the RGB value of each pixel from the surrounding pixels with respect to the first image signal (color image signal) converted into a digital signal, and a YUV conversion process for the RGB signal. The image data composed of the YUV signal values is generated and supplied to the control unit 2.

  The image data supplied to the control unit 2 is supplied to the display unit 7 and displayed on the display unit 7 as a live view image. The image data supplied to the control unit 2 at the time of shooting is compressed by the control unit 2 and then added with various attribute information and stored in the image storage unit 5 as a still image file.

  Still image data recorded as a still image file in the image storage unit 5 is read and expanded as necessary by the control unit 2 in the reproduction mode and then reproduced (displayed) as a still image on the display unit 7. The

  The image storage unit 5 includes, for example, a flash memory built in the digital camera 1, various memory cards that can be attached to and detached from the digital camera 1, and a card interface that enables data input and output to the memory card.

  The display unit 7 includes a color liquid crystal display panel and a display driving circuit that drives the color liquid crystal display panel in accordance with image data supplied from the control unit 2. The display unit 7 displays the subject image captured by the first image sensor 42 in live view in the recording mode, and displays a still image in the playback mode.

  The operation unit 8 includes a plurality of operation switches such as a power button, a shutter, a plurality of shooting mode selection switches prepared in advance, a zoom lever, and a control button used for selecting and setting various functions. The operation state of the operation switch in the operation unit 8 is monitored by the control unit 2 as needed.

  The power supply unit 9 includes a rechargeable battery and a DC / DC converter, and supplies power necessary for each operation to each unit of the digital camera 1.

  Next, the operation according to the present invention of the digital camera 1 having the above configuration will be described with reference to FIG. FIG. 3 is a flowchart showing the main part of the contents of the imaging process at the time of shooting by the digital camera 1.

  As shown in FIG. 3, at the time of shooting when the user presses the shutter, for example, the control unit 2 exposes the first image sensor 42 and the second image sensor 43, and the first and second image sensors. The imaging signals (charge signals) accumulated in 42 and 43 are transferred to the image processing engine 45 (step SA1).

  Here, the exposure time of the first image sensor 42 and the second image sensor 43 is an exposure time for obtaining an appropriate exposure in the AE control. Further, since this exposure time is obtained by splitting the incident light rays to the first and second image sensors 42 and 43 by the half prism 41, it is twice the exposure time when there is a single image sensor. This is a time obtained by adding a time corresponding to the attenuation of light accompanying the spectrum to the time.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted by the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaic processing on the first image signal after A / D conversion obtained by the first image sensor 42, interpolates the RGB value of each pixel from surrounding pixels, and sets the pixel value of each pixel. A pixel signal composed of R1, G1, and B1 is generated (step SA2).

  Next, the DSP 48 performs YUV conversion based on a predetermined conversion formula from the generated pixel signal values R1, G1, and B1, and obtains a luminance signal value Y1 and color difference signal values U1 and V1 of each pixel (step SA3). ). Further, the DSP 48 acquires the signal value of each pixel as the luminance signal value Y2 from the second image signal after A / D conversion obtained by the second image sensor 43 (step SA4).

  Next, the DSP 48 determines whether the current shooting situation is daytime or nighttime based on the luminance signal value Y2 obtained from the second imaging signal (step SA7). More specifically, the DSP 48 determines that the shooting state is daytime if the luminance signal value Y2 is equal to or greater than a predetermined value, and the luminance signal value Y2 is less than the predetermined value. If so, it is determined that the shooting situation is at night.

  When the DSP 48 determines that the shooting state is daytime (step SA5: daytime), the luminance signal value Y1 and the color difference signal values U1 and V1 of the pixels obtained from the first imaging signal are the luminance signal. Image data (YUV data) is generated by replacing only with the luminance signal value Y2 obtained from the second imaging signal (step SA6). Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SA8).

  Here, the reason why the luminance signal value is replaced as described above in the process of step SA6 is as follows.

  That is, the signal values Y1, U1, and V1 obtained from the first image pickup signal are generated based on the signal values R1, G1, and B1 including the signal values generated by interpolation of surrounding pixels in the demosaic process. Therefore, the resolution sensitivity is inevitably lowered.

  On the other hand, the luminance signal value Y2 obtained from the second imaging signal is the luminance signal value of the actual pixel obtained by the electric conversion element that actually exists in the second imaging element 43. Therefore, by adopting the luminance signal value Y2 obtained from the second imaging signal as the luminance signal value, the luminance component having the resolution of the actual pixel is secured in the final image data.

  Therefore, the image data (Y2, U1, V1) generated in step SA6 has higher color reproducibility and resolution sensitivity than the image data (Y1, U1, V1) obtained from only the first imaging signal. This is the image data. As a result, a high-quality captured image can be obtained.

  In contrast to the above, when the DSP 48 determines that the shooting situation is nighttime (step SA5: nighttime), the luminance signal value Y1 and the color difference signal values U1, V1 of each pixel obtained from the first imaging signal. Among them, only the luminance signal Y1 is replaced with a value ((Y1 + Y2) / 2) obtained by averaging the luminance signal value Y1 obtained from the first imaging signal and the luminance signal value Y2 obtained from the second imaging signal. Image data (YUV data) is generated (step SA7). Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SA8).

  Here, the reason why the luminance signal value is replaced as described above in the process of step SA7 is as follows.

  That is, in night shooting, if a process similar to that in the above-described day shooting is performed, a luminance component having a resolution of actual pixels can be secured in the final image data. However, when shooting at night, the subject is naturally dark. Furthermore, as described above, it is necessary to secure an exposure time twice as long as that when a single image sensor is used as described above. However, if the exposure time is increased, camera shake and subject blur may occur. There is a limit to the exposure time that can be achieved. Therefore, appropriate exposure is ensured by increasing the photographing sensitivity (gains of the first and second AFEs 46 and 47).

  For this reason, the second imaging signal includes noise corresponding to the increase in imaging sensitivity as compared with the imaging signal obtained by the single-plate imaging method. Therefore, if the luminance signal is simply replaced in the same manner as in the normal shooting situation, the resolution sensitivity can be increased, but only image data with noticeable noise can be obtained.

  Therefore, in night shooting, as described above, the luminance signal value of the final image data includes the luminance signal value Y1 obtained from the first imaging signal and the luminance signal obtained from the second imaging signal. The value Y2 is averaged ((Y1 + Y2) / 2). Thereby, it is possible to avoid the occurrence of noise.

  That is, both the first image pickup signal and the second image pickup signal include the noise described above, but the noise generated due to the increase in photographing sensitivity is a variation in sampling adjustment during A / D conversion. Random noise generated by the influence of the above.

  Therefore, there is no change in the number of pixels that cause noise in the final image data, but the degree of noise is much larger than when the luminance signal value Y2 obtained from the second imaging signal is used as it is. Can be reduced. As a result, it is possible to obtain a low-noise photographed image that is comparable to that obtained by a normal single-plate imaging method.

  As described above, in the present embodiment, in a configuration including two imaging elements, different imaging processes are performed at the time of shooting when the shooting state is determined to be daytime and when it is determined to be nighttime. Therefore, even when the brightness of the subject is greatly different, a higher-quality captured image can be obtained without being influenced by the difference in the shooting conditions.

  Moreover, the difference in the contents of the imaging process is the difference in the pixel signal value used for acquiring the component signal value of each pixel in the image data generation process. That is, the luminance signal value (Y) of the image data is set to the luminance signal value Y2 obtained from the second imaging signal, or the luminance signal value Y1 obtained from the first imaging signal and the second imaging signal. Only the difference between the luminance signal value Y2 obtained from (1) and the average value ((Y1 + Y2) / 2). Therefore, a high-quality photographed image can be obtained by extremely simple signal processing.

  In the present embodiment, a configuration has been described in which whether the shooting state is daytime or nighttime is automatically determined based on the luminance signal value Y2 obtained from the second imaging signal. However, the difference in the shooting situation may be determined as follows.

  For example, when the digital camera 1 is provided with shooting modes corresponding to a plurality of shooting scenes, and there are shooting modes used in night shooting scenes such as night views and fireworks as the shooting mode, the user at the time of shooting Alternatively, it may be configured to determine whether the shooting state is daytime or nighttime from the shooting mode set to. In that case, the process of automatically determining the shooting situation can be omitted.

  In that case, for example, information indicating the shooting mode set at the time of shooting may be appropriately supplied from the control unit 2 to the DSP 48, or the control unit 2 may determine whether the shooting state is daytime or nighttime, Only the information indicating the determination result may be appropriately supplied to the DSP 48.

  Furthermore, if the shooting mode is not limited to the shooting mode used in night shooting scenes, the high-sensitivity shooting mode is set if the configuration has a high-sensitivity shooting mode specialized for shooting dark subjects. For the convenience of shooting, it may be configured that the shooting situation is determined to be nighttime for convenience.

  In the present embodiment, the DSP 48 in the image processing engine 45 generates image data composed of YUV signal values based on the first and second imaging signals obtained by the first and second imaging elements 42 and 43. Explained what to do. However, the present invention can also be applied to a digital camera that does not have a signal processing circuit corresponding to the DSP 48. In that case, the processing by the DSP 48 may be configured to be performed by another microcomputer or the like that controls the digital camera like the control unit 2 described above.

(Embodiment 2)
Next, a second embodiment of the present invention will be described. In the present embodiment, in the digital camera 1 having the configuration shown in FIGS. 1 and 2, the second image sensor 43 has a large contribution to the luminance signal shown in FIG. 4 instead of the colorless color filter 43a. A single color filter 43b made of only G is provided. Also in the present embodiment, the DSP 48 performs signal processing (to be described later) according to the shooting situation automatically determined during shooting.

  Hereinafter, the operation of the digital camera of the present embodiment according to the present invention will be described with reference to the flowchart of FIG. 5 corresponding to FIG.

  As shown in FIG. 5, also in the present embodiment, for example, at the time of photographing when the shutter is pressed by the user, the control unit 2 exposes the first imaging element 42 and the second imaging element 43, and The imaging signals (charge signals) accumulated in the second imaging elements 42 and 43 are transferred to the image processing engine 45 (step SB1). The exposure time of the first image sensor 42 and the second image sensor 43 is as described above.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted by the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaic processing on the first image signal after A / D conversion obtained by the first image sensor 42, interpolates the RGB value of each pixel from surrounding pixels, and sets the pixel value of each pixel. A pixel signal composed of R1, G1, and B1 is generated (step SB2).

  Thereafter, in the present embodiment, the DSP 48 acquires the signal value of each pixel as the G component signal value G2 from the second image signal after A / D conversion obtained by the second image sensor 43 (step SB3). .

  Next, the DSP 48 determines whether the current shooting state is daytime or nighttime based on the signal value G2 obtained from the second imaging signal (step SB4).

  When the DSP 48 determines that the shooting state is daytime (step SB4: daytime), only the signal value of the G component among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal. Is replaced with the signal value G2 obtained from the second imaging signal, and YUV conversion is performed on the pixel signal values (R1, G2, B1) (step SB5).

  Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y and the color difference signal values U and V of each pixel. Then, the DSP 48 outputs the generated image data to the control unit 2 (step SB7).

  Here, the reason why the pixel value of the G component is replaced as described above in the process of step SB5 is as follows.

  That is, the pixel signal values R1, G1, and B1 obtained from the first imaging signal are generated by interpolation of surrounding pixels in the demosaic process, and the resolution sensitivity is inevitably lowered.

  On the other hand, the G component signal value G <b> 2 of each pixel obtained from the second imaging signal is a signal value of the actual pixel obtained by the electrical conversion element that actually exists in the second imaging element 43. Therefore, by adopting the signal value G2 obtained from the second imaging signal as the signal value of the G component, the luminance component having the resolution of the actual pixel and the G component are ensured in the final image data. is there.

  Accordingly, the image data (YUV data) generated in step SB5 is more color reproducible than the image data generated based on the pixel signal values R1, G1, and B1 of each pixel obtained only from the first imaging signal. In addition, the captured image data has high resolution sensitivity. As a result, a high-quality captured image can be obtained. In particular, with regard to color reproducibility, when a green region exists in the subject, the color reproducibility of that region can be reliably improved.

  In contrast to the above, when the DSP 48 determines that the shooting state is nighttime (step SB4: nighttime), the pixel signal value R1, G1, B1 of each pixel obtained from the first imaging signal is represented by G Only the signal value of the component is replaced with a value ((G1 + G2) / 2) obtained by averaging the signal value G1 obtained from the first imaging signal and the signal value G2 obtained from the second imaging signal, and the pixel YUV conversion is performed on the signal values (R1, ((G1 + G2) / 2), B1) (step SB6).

  Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y and the color difference signal values U and V of each pixel. Then, the DSP 48 outputs the generated image data to the control unit 2 (step SB7).

  Here, the reason why the luminance signal value is replaced as described above in the process of step SB6 is as follows.

  That is, as described in the first embodiment, the second imaging signal includes noise corresponding to the increase in imaging sensitivity as compared with an imaging signal obtained by a normal single-plate imaging method. Therefore, as in the case of the normal mode, if the signal value of the G component is simply replaced with the signal value G2 obtained from the second imaging signal, the resolution sensitivity can be increased, but only image data with noticeable noise is obtained. I can't.

  Therefore, in night shooting, as described above, the signal value of the G component before the YUV conversion, the signal value G1 obtained from the first imaging signal, and the signal obtained from the second imaging signal are used. The value G2 is averaged ((G1 + G2) / 2). Thereby, it is possible to avoid the occurrence of noise.

  Therefore, the number of pixels in which noise is generated in the final image data does not change, but the degree of noise is determined when the G component signal value G2 obtained from the second imaging signal is used as it is. Compared to this, it can be greatly reduced. As a result, it is possible to obtain a low-noise photographed image that is comparable to that obtained by a normal single-plate imaging method.

  As described above, also in the present embodiment, as in the first embodiment, in the configuration including the two imaging elements, when the shooting state is determined to be daytime and when it is determined to be nighttime, different imaging is performed at the time of shooting. It was made to process. Therefore, even when the brightness of the subject is greatly different, a higher-quality captured image can be obtained without being influenced by the difference in the shooting conditions.

  Moreover, also in the present embodiment, the difference in the contents of the imaging process is the difference in the pixel signal value used for acquiring the component signal value of each pixel in the image data generation process. That is, the signal value G2 obtained from the second imaging signal is used for obtaining the luminance signal value Y and the color difference signal values U and V of the image data, or the signal value G1 obtained from the first imaging signal is used. The only difference is whether the signal value G2 obtained from the second imaging signal is averaged ((G1 + G2) / 2). Therefore, also in this embodiment, a high-quality captured image can be obtained by extremely simple signal processing.

  Note that, in the present embodiment, as in the first embodiment, it is not always necessary to automatically determine whether the shooting state is daytime or nighttime, and it is determined from the shooting mode set by the user at the time of shooting. The specific determination method and the like may be arbitrary. Further, the signal processing by the DSP 48 described in the present embodiment can also be performed by the control unit 2.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. In this embodiment, in the digital camera 1 having the configuration shown in FIG. 1, the imaging unit 4 is different from that shown in FIG. 2A in the first embodiment, and the configuration shown in FIG. It is what you have.

  That is, in this embodiment, no color filter is provided on the photosensitive surface of the second image sensor 43, and the gap between the second image sensor 43 and the half prism 41 is shown in FIG. A rotary filter plate 51 is provided.

  The rotary filter plate 51 is provided with RGB color filters 51 a, 51 b, 51 c in a region divided into three in the circumferential direction, and is appropriately rotated by a motor 52. The operation of the motor 52 is controlled by the control unit 2, and when the motor 52 changes the rotational position of the rotary filter plate 51 in accordance with an instruction from the control unit 2, any one of the RGB color filters is the second color filter. This is applied to the image sensor 43.

  Since other configurations are the same as those shown in FIG. 2A, the same reference numerals are given to the same portions, and description thereof is omitted.

  Hereinafter, the operation of the digital camera according to the present embodiment according to the present invention will be described with reference to the flowchart of FIG. FIG. 7 is a flowchart showing a main part of the operation at the time of photographing according to the present embodiment.

  In the following description, in the present embodiment, for example, at the time of shooting when the user presses the shutter, the control unit 2 first determines the shooting status (step SC1). Specifically, the control unit 2 captures the color tone of the shooting target from the RGB color component information of the shot image captured by the first image sensor 42 and displayed as a live view image on the display unit 7 immediately before shooting. The shooting situation is determined.

  That is, if the proportion of the B component in the captured image is greater than or equal to the specified value, the control unit 2 determines that the color tone of the subject to be photographed is a bluish blue tone, and if the proportion of the G component is equal to or greater than the specified value, It is determined that the color tone of the shooting target is a greenish tone with a greenish color, and if the ratio of the R component is greater than or equal to the specified value, it is determined that the color tone of the shooting target is a reddish tone.

  Then, when the color tone to be photographed does not correspond to any of the above (“other” in FIG. 7), the control unit 2 immediately exposes only the first image sensor 42, and the first image sensor 42. Is transferred to the image processing engine 45 (step SC2).

  The image pickup signal transferred to the image processing engine 45 is amplified and A / D converted by the first AFEs 46 and 47, and then normal signal processing is performed by the DSP 48.

  That is, the DSP 48 performs demosaic processing on the first image signal after A / D conversion obtained by the first image sensor 42, interpolates the RGB value of each pixel from the surrounding pixels, and sets the pixel value of each pixel. A pixel signal composed of R1, G1, and B1 is generated (step SC3).

  Next, the DSP 48 performs YUV conversion based on a predetermined conversion formula from the generated pixel signal values R1, G1, and B1, and acquires the luminance signal value Y and the color difference signal values U and V of each pixel (step SC4). ). Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

  Therefore, when the control unit 2 determines that the color tone of the subject to be photographed does not correspond to any of blue, green, and red, the same as that obtained by a normal single-plate imaging method as a photographed image. A quality photographed image can be obtained.

  On the other hand, unlike the above, when the control unit 2 determines that the color tone is blue, for example, when the subject to be photographed is “blue sky” or “sea”, the control unit 2 sets the rotary filter plate. 51 is rotated to a predetermined position, and the B color filter is applied to the second image sensor 43 (step SC5).

  If the control unit 2 determines that the color tone is green, for example, when the shooting target is “meadow”, the G color filter is applied to the second image sensor 43 (step S <b> 3). SC6).

  Further, when it is determined that the color tone is red, for example, when the shooting target is “evening scene”, the control unit 2 applies the B color filter to the second image sensor 43 (step S2). SC7).

  Although not shown in FIG. 7, the control unit 2 supplies information indicating the filter color applied to the second image sensor 43 to the DSP 48 in each process of steps SC <b> 5 to SC <b> 7 described above.

  Subsequently, the control unit 2 exposes the first image sensor 42 and the second image sensor 43 to which any one of the RGB color filters is applied, to thereby expose the first and second image sensors 42, The imaging signal (charge signal) accumulated in 43 is transferred to the image processing engine 45 (step SC8). Note that the exposure times of the first image sensor 42 and the second image sensor 43 are as already described in the first embodiment.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted by the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaic processing on the first image signal after A / D conversion obtained by the first image sensor 42, interpolates the RGB value of each pixel from surrounding pixels, and sets the pixel value of each pixel. A pixel signal composed of R1, G1, and B1 is generated (step SC9).

  Next, the DSP 48 confirms the filter color applied to the second image sensor 43 from the filter color information supplied in advance from the control unit 2 (step SC10).

  When the filter color is B (step SC10: B), the DSP 48 converts the signal value of each pixel into a B component from the second image signal after A / D conversion obtained by the second image sensor 43. Is obtained as a signal value B2 (step SC11).

  Next, among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal, only the signal value of the B component is replaced with the signal value B2 obtained from the second imaging signal, and the pixel YUV conversion is performed on the signal values (R1, G1, B2) (step SC12). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y and the color difference signal values U and V of each pixel. Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

  Here, in the image data generated in step SC <b> 16, the signal value of the B component used for YUV conversion is the actual pixel signal value obtained by the actual electric conversion element of the second image sensor 43. Compared with the image data generated based on the pixel signal values R1, G1, and B1 of each pixel obtained from only the first imaging signal, the captured image data has high color reproducibility. In particular, the captured image data has a high color reproducibility ensured by the blue portion. As a result, for example, when the subject to be photographed is “blue sky” or “sea”, it is possible to obtain a high-quality photographed image in which the color of the blue sky portion or the sea portion is more faithfully reproduced.

  In addition, when the filter color is G (step SC10: G), the DSP 48 determines the signal of each pixel from the second image signal after A / D conversion obtained by the second image sensor 43 when the filter color is G (step SC10: G). The value is acquired as the G component signal value G2 (step SC13).

  Next, among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal, only the G component signal value is replaced with the signal value G2 obtained from the second imaging signal, and the pixel YUV conversion is performed on the signal values (R1, G2, B1) (step SC14). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y and the color difference signal values U and V of each pixel. Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

Here, in the image data generated in step SC14, the signal value of the G component used for YUV conversion is the actual pixel signal value obtained by the electrical conversion element of the second image sensor 43. Compared with the image data generated based on the pixel signal values R1, G1, and B1 of each pixel obtained from only the first imaging signal, the captured image data has high color reproducibility. In particular, high color reproducibility can be ensured by the green portion. In addition, since the pixel signal value of the G component has a large contribution ratio to the luminance signal, it becomes the data of the captured image with high resolution sensitivity. as a result,
For example, when the subject to be photographed is “meadow”, it is possible to obtain a high-quality photographed image in which the color of the grass portion is reproduced more faithfully and the resolution sensitivity is high.

  Further, when the filter color is R (step SC10: R) unlike the above, the DSP 48 determines the signal of each pixel from the second image pickup signal after A / D conversion obtained by the second image pickup element 43. The value is acquired as the R component signal value R2 (step SC15).

  Next, the DSP 48 replaces only the R component signal value of the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal with the signal value R2 obtained from the second imaging signal. Then, YUV conversion is performed on the pixel signal values (R2, G1, B1) (step SC16). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y and the color difference signal values U and V of each pixel. Thereafter, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

Here, in the image data generated in step SC <b> 16, the signal value of the R component used for YUV conversion is the actual pixel signal value obtained by the actual electrical conversion element of the second image sensor 43. Compared with the image data generated based on the pixel signal values R1, G1, and B1 of each pixel obtained from only the first imaging signal, the captured image data has high color reproducibility. In particular, the captured image data has high color reproducibility secured by the red portion. as a result,
For example, when the subject to be photographed is “evening scene”, it is possible to obtain a high quality photographed image in which the color of the reddish sky part is reproduced more faithfully.

  As described above, in the present embodiment, a plurality of color filters can be applied to the second image sensor 43, and the filter color of the color filter to be applied is changed according to the difference in shooting conditions such as the color tone of the shooting target. In addition, when the shooting conditions are different, different imaging processes are performed at the time of shooting. Therefore, even when the color tone of the shooting target is different, a higher-quality shot image can be obtained without being affected by the difference in shooting conditions.

  In the present embodiment, since the filter colors to be applied are R, G, and B colors, depending on the shooting time when the shooting target is, for example, “blue sky”, “sea”, “grass”, or “evening scene”. A high-quality captured image can be obtained.

  Moreover, also in the present embodiment, the difference in the contents of the imaging process is the difference in the pixel signal value used for acquiring the component signal value of each pixel in the image data generation process. In other words, when acquiring the luminance signal value Y and the color difference signal values U and V of the image data, the only difference is whether the pixel signal value obtained from the second imaging signal is used as the pixel signal value of RGB. . Therefore, also in this embodiment, a high-quality captured image can be obtained by extremely simple signal processing.

  In the present embodiment, the control unit 2 captures an image by the first imaging element 42 as to whether or not the color tone of the subject to be photographed corresponds to any one of blue, green, and red. A configuration for automatically determining from an image has been described.

  However, the determination of the shooting situation at the time of shooting may be performed by, for example, causing the digital camera to select an arbitrary shooting scene from a plurality of shooting scenes and shooting under shooting conditions corresponding to each selected shooting scene. If there is a configuration in which various shooting modes are provided, the shooting status (color tone of the shooting target) may be determined based on the shooting scene selected by the user. In that case, the process of automatically determining the shooting situation can be omitted.

  In that case, as shooting condition data indicating shooting conditions according to each shooting scene, whether or not the second image sensor 43 is used at the time of shooting and when the second image sensor 43 is used are applied. Information on the color filter color to be set may be set in advance.

  Also in the present embodiment, the signal processing by the DSP 48 described above can be configured to be performed by the control unit 2.

  Further, in place of the combination of the rotary filter plate 51 and the second image sensor, three independent image sensors having color filters for each color of RGB are used, and image signals from the three independent image sensors according to the shooting scene. May be used.

As mentioned above, although several embodiment of this invention and its modification were demonstrated, if these are in the range in which the effect of this invention is acquired, it can change suitably, and embodiment after change is also a claim. It is included in the scope of the invention described in the scope of the invention and the invention equivalent to the invention. The invention described in the scope of the claims of the present application will be appended below.
[Claim 1]
A plurality of image sensors with different color filters;
A judging means for judging the shooting situation;
Image generating means for generating image data based on different imaging signals obtained by the plurality of imaging elements;
Signal processing control means for performing imaging signal processing of different contents determined in advance according to the shooting situation determined by the determination means when the image data is generated by the image generation means. An imaging device.
[Claim 2]
The difference in the contents of the imaging signal processing is that the combination of the pixel signal values among the plurality of imaging elements used for acquiring the component signal value of each pixel in the image data generation process by the image generation means. The imaging apparatus according to claim 1, wherein:
[Claim 3]
The content of the imaging signal processing is processing content related to either one of generation processing of a luminance value of each pixel in the image data or generation processing of a luminance value and a color difference component value of each pixel. The imaging apparatus according to 1 or 2.
[Claim 4]
The plurality of image pickup devices are a first image pickup device provided with a color filter used in a single-plate type image pickup method in which RGB colors are arranged, and a second image pickup device provided with a single color filter. The imaging apparatus according to claim 1, wherein the imaging apparatus is provided.
[Claim 5]
The image pickup apparatus according to claim 4, wherein the color filter provided in the second image pickup element is colorless.
[Claim 6]
The signal processing control unit responds to the shooting situation determined by the determination unit when the image generation unit generates the image data according to the brightness of the subject as the shooting state determined by the determination unit. 6. The imaging apparatus according to claim 5, wherein imaging signal processing of different contents determined in advance is performed.
[Claim 7]
When the shooting situation determined by the determination means is a first shooting situation in which the brightness of the subject can be determined to be greater than or equal to a predetermined reference, the signal processing control means In the generation processing of the luminance value of each pixel in the above, the imaging signal processing is performed so that the luminance value in the imaging signal obtained by the second imaging element is adopted as the luminance value of each pixel. 6. The imaging device according to 6.
[Claim 8]
When the shooting situation determined by the determination means is a second shooting situation in which the brightness of the subject can be determined to be less than a predetermined reference, the signal processing control means sends the image data to the image generation means. In the generation processing of the luminance value of each pixel in FIG. 4, as the luminance value of each pixel, the luminance value based on the imaging signal obtained by the first imaging element and the luminance value in the imaging signal obtained from the second imaging element The image pickup apparatus according to claim 7, wherein the image pickup signal processing is performed so as to adopt a luminance value obtained by averaging
[Claim 9]
The imaging apparatus according to claim 4, wherein a color of a color filter provided in the second imaging element is G of the RGB.
[Claim 10]
The signal processing control unit responds to the shooting situation determined by the determination unit when the image generation unit generates the image data according to the brightness of the subject as the shooting state determined by the determination unit. The imaging apparatus according to claim 9, wherein imaging signal processing of different contents determined in advance is performed.
[Claim 11]
When the shooting situation determined by the determination means is a first shooting situation in which the brightness of the subject can be determined to be greater than or equal to a predetermined reference, the signal processing control means In the generation processing of the luminance value and the color difference component value of each pixel, the pixel value based on the imaging signal obtained by the first imaging element is adopted as the pixel value of the R component and B component of each pixel, and each pixel The image pickup apparatus according to claim 10, wherein the image pickup signal processing is performed so that the pixel value in the image pickup signal obtained by the second image pickup element is adopted as the pixel value of the G component.
[Claim 12]
When the shooting situation determined by the determination means is a second shooting situation in which the brightness of the subject can be determined to be less than a predetermined reference, the signal processing control means sends the image data to the image generation means. In the generation processing of the luminance value and the color difference component value of each pixel, the pixel value based on the imaging signal obtained by the first imaging element is adopted as the pixel value of the R component and B component of each pixel, and each pixel Content of adopting a luminance value obtained by averaging pixel values in an imaging signal obtained by the first imaging element and pixel values based on the imaging signal obtained by the second imaging element as the pixel value of the G component of The imaging apparatus according to claim 11, wherein the imaging signal processing is performed.
[Claim 13]
The plurality of image pickup elements are a first image pickup element provided with a color filter used in a single-plate type image pickup method in which RGB colors are arranged, and a second image pickup element to which a color filter of a different color can be applied. And
Color filter means including a plurality of color filters of different colors that are selectively applied to the second image sensor;
Filter control means for applying, to the second image sensor, a color filter of a predetermined color determined in advance according to the photographing situation determined by the determination means among the plurality of color filters in the color filter means; The imaging apparatus according to claim 2, further comprising:
[Claim 14]
14. The imaging apparatus according to claim 13, wherein each color of the plurality of color filters in the color filter means is one of RGB colors.
[Claim 15]
The signal processing control means is a first photographing situation in which the photographing situation determined by the judging means can determine that the use of the imaging signal obtained by the second imaging element is not necessary for generating the image data. In the case where the image signal is generated, the imaging signal obtained by the first image sensor as the pixel value of each color of RGB of each pixel in the generation processing of the luminance value and the color difference component value of each pixel in the image data. The image pickup apparatus according to claim 14, wherein the image pickup signal processing is performed so as to adopt only the pixel value based on the image.
[Claim 16]
The signal processing control means is a second shooting situation in which the shooting situation determined by the determining means can determine that the use of the imaging signal obtained by the second imaging element is effective for generating the image data. In the case where the image generation means performs the generation processing of the luminance value and the color difference component value of each pixel in the image data, the first image sensor as pixel values of two colors other than the predetermined color of each pixel. Imaging signal processing that adopts a pixel value based on the obtained imaging signal and adopts a pixel value based on the imaging signal obtained by the second imaging element as the pixel value of the predetermined color of each pixel 16. The imaging apparatus according to claim 14, wherein:
[Claim 17]
The imaging apparatus according to any one of claims 1 to 16, wherein the determination unit determines a shooting situation based on arbitrary setting information indicating a shooting situation set for a user at the time of shooting. .
[Claim 18]
A judgment process for judging the shooting situation;
An image generation step of generating image data based on different imaging signals obtained by a plurality of imaging elements having different color filters,
An imaging method characterized in that when the image data is generated by the image generation step, imaging signal processing of different contents determined in advance according to the shooting situation determined by the determination step is performed.
[Claim 19]
A computer having an image pickup apparatus having a plurality of image pickup elements having different color filters;
A judging means for judging the shooting situation;
Image generating means for generating image data based on different imaging signals obtained by the plurality of imaging elements;
When the image data is generated by the image generation unit, the image generation unit functions as a signal processing control unit that performs imaging signal processing of different contents determined in advance according to the shooting situation determined by the determination unit. Program.

DESCRIPTION OF SYMBOLS 1 Digital camera 2 Control part 3 Lens part 4 Image pick-up part 5 Image memory | storage part 6 Program memory | storage part 7 Display part 8 Operation part 9 Power supply part 41 Half prism 42 1st image pick-up element 42a Color filter 43 2nd image pick-up element 43a Color filter 43b Color filter 45 Image processing engine 46 First AFE
47 Second AFE
48 DSP
51 Rotating filter plate 51a Color filter 51b Color filter 51c Color filter 52 Motor

Claims (6)

Image generation means for generating image data using two image pickup signals obtained by an image pickup device through a color filter and a colorless or single color filter;
A judging means for judging the brightness of the subject;
Depending on the brightness of the subject determined by the determination means, the luminance value of each pixel in the image data generation process of the image data by the image generation means is passed through the color filter as a replacement value. Control means for controlling to generate image data by replacing the luminance value of the imaging signal obtained by the imaging device and the luminance value of the imaging signal obtained by the imaging device via the colorless or single color filter with an average. When,
An image data generation device comprising: The image data generation apparatus according to claim 1, wherein the determination unit determines whether the brightness of the subject is greater than or less than a predetermined reference . The image data generation device according to claim 1, wherein the two imaging signals are obtained at least at the time of imaging . 4. The image data generation apparatus according to claim 1, wherein the color of the single color filter is G of RGB . An image generation step of generating image data using two image pickup signals obtained by the image pickup device through a color filter and a colorless or single color filter;
A determination step for determining the brightness of the subject;
According to the brightness of the subject determined in the determination step, the color filter is used as a replacement value for the luminance value of each pixel in the generation processing of the luminance value of each pixel in the image data in the image generation step. And controlling to generate image data by replacing the luminance value of the imaging signal obtained by the imaging element with the average value of the luminance value of the imaging signal obtained by the imaging element via the colorless or single color filter. Control steps;
An image data generation method comprising: Computer
Image generation means for generating image data using two image pickup signals obtained by the image pickup device through a color filter and a colorless or single color filter,
A judging means for judging the brightness of the subject;
Depending on the brightness of the subject determined by the determination means, the luminance value of each pixel in the image data generation process of the image data by the image generation means is passed through the color filter as a replacement value. Control means for controlling to generate image data by replacing the luminance value of the imaging signal obtained by the imaging device and the luminance value of the imaging signal obtained by the imaging device via the colorless or single color filter with an average. ,
A program characterized by functioning as

Images