JP6547796B2 - Image data generating apparatus, image data generating method and program - Google Patents

Description

  The present invention relates to a technique for obtaining a photographed image using a plurality of imaging devices.

  Conventionally, for example, in Patent Document 1 below, RGB signals of a single-plate RGB imaging device are provided in order to enable imaging of a high dynamic range in a configuration provided with a single-plate RGB imaging device and a monochrome imaging device as imaging devices. A technique is disclosed that reflects luminance information included in both of the luminance signal of the monochrome imaging device through multiple steps of signal processing and as color information and luminance information of a final captured image.

JP 2011-223221

  However, although the above-described technique enables imaging with a high dynamic range, it is not possible to process different imaging signals obtained by two imaging devices in accordance with various imaging situations, and depending on the imaging situations, it is not always necessary. There is a problem that high-quality captured images can not always be obtained.

The present invention has been made in view of such conventional problems, and an object of the present invention is to provide an image data generation apparatus, an image data generation method, and a program capable of obtaining high-quality photographed images in various photographing situations. I assume.

In order to solve the above-mentioned problems, in the present invention, a first imaging element provided with a color filter used in a single-plate type imaging system in which each color of RGB is disposed and a color filter of different colors are applied. Image generation means for generating image data using two imaging signals obtained by the possible second imaging device, and a plurality of color filters different in color which are selectively applied to the second imaging device Of the plurality of color filters in the color filter means including the color filter means, the judgment means for judging the photographing condition, and the color filter means , a predetermined color determined in advance according to the photographing situation judged by the judgment means A filter control unit that applies a color filter to the second imaging element, and the two imaging signals according to the imaging situation determined by the determination unit And control means for controlling the image generation means to generate image data.

  According to the present invention, it is possible to obtain high quality photographed images in various photographing situations.

It is a block diagram of a digital camera illustrated as an imaging device of the present invention. It is a figure which shows 1st Embodiment, Comprising: (a) is a block diagram which shows an imaging part, (b) is a figure which shows the color filter of a 1st image sensor, (c) is a color of a 2nd image sensor. It is a figure showing a filter. It is a flowchart which shows the imaging process in 1st Embodiment. It is a figure which shows the color filter of the 2nd image sensor in 2nd Embodiment. It is a flowchart which shows the imaging process in 2nd Embodiment. It is a figure which shows 3rd Embodiment, Comprising: (a) is a block diagram which shows a digital camera imaging part, (b) is a figure which shows a rotation filter board. It is a flowchart which shows the imaging process in 3rd Embodiment.

  Hereinafter, embodiments of the present invention will be described.

(Embodiment 1)
FIG. 1 is a block diagram schematically showing an electrical configuration of a digital camera 1 exemplified as an image pickup apparatus of the present invention.

  As shown in FIG. 1, the digital camera 1 includes the control unit 2, the lens unit 3, the imaging unit 4, the image storage unit 5, the program storage unit 6, the display unit 7, the operation unit 8, and the power supply unit 9. Have.

  Control unit 2 includes a central processing unit (CPU), peripheral circuits thereof, and a working internal memory such as random access memory (RAM), and a digital camera according to a control program stored in program storage unit 6. Control each part of 1.

  The program storage unit 6 is configured of, for example, a non-volatile memory such as a flash memory in which stored data can be rewritten as needed. The program storage unit 6 includes a program for causing the control unit 2 to perform AE (Auto Exposure) control, AF (Auto Focus) control, AWB (Auto White Balance) control, etc. in addition to the above control program, and AE Various data such as data constituting a program chart showing combinations of shutter speed, ISO sensitivity, and aperture value used for control are also stored.

  The lens unit 3 includes a lens group including lenses for focus adjustment and zoom, a motor for driving the lens group, an aperture, and an actuator for opening and closing the aperture to adjust the opening degree.

  The imaging unit 4 images a subject via the lens group of the lens unit 3, generates image data of the subject, and supplies the image data to the control unit 2. FIG. 2A is a block diagram showing the configuration of the imaging unit 4.

  The imaging unit 4 mainly includes a half prism 41 that divides an incident light beam from the lens unit 3 into two paths, a first imaging element 42 that receives one of the divided incident light beams, and a second one that receives the other incident light beams. It comprises the image processing engine 45 which processes the imaging signal obtained by the second imaging element 43 and the first and second imaging elements 42 and 43. Although not shown in FIG. 4A, the imaging unit 4 also includes drive circuits for the first and second imaging elements 42 and 43.

  The first imaging device 42 and the second imaging device 43 are, for example, a CMOS (Complementary Meta10 Semiconductor) type imaging device or a CCD (Charge Coupled Device).

  The first imaging element 42 is provided with a color filter 42a in which each color of RGB shown in FIG. 2B is arranged in a Bayer array, and an imaging signal (hereinafter , And the first imaging signal) is a color imaging signal including each color component of RGB.

  The second imaging element 43 is provided with a colorless color filter 43a whose color of each pixel is indicated by N for convenience in FIG. 2C, and the imaging signal output from the second imaging element 43 (Hereafter, it is called 2nd imaging signal.) Is a monochrome imaging signal of only a luminance component.

  The first imaging device 42 is constantly driven to capture an object while the recording mode for imaging is set in the digital camera 1, and the second imaging device 43 basically basically performs imaging only. It is driven to image a subject.

  In addition, as the color filter 42a provided in the first imaging device 42, any color array other than the Bayer arrangement may be used as long as it is used in a normal single-plate imaging method in which imaging is performed using a single imaging device. It can be used. The half prism 41 can be replaced by a half mirror.

  The image processing engine 45 processes the first and second imaging signals obtained by the first and second imaging elements 42 and 43 according to an instruction of the control unit 2. The image processing engine 45 mainly performs predetermined analog processing such as amplification of the first imaging signal and the second imaging signal, and first and second AFEs (Analog Front End) 46, which individually convert into digital signals. 47 and a DSP (Digital Signal Processor) 48.

  The DSP 48 comprises a microcomputer having a function specialized for digital signal processing, a ROM (Read Only Memory) storing a program, a memory, an arithmetic circuit, and the like. The DSP 48 performs demosaicing processing to interpolate RGB values of each pixel from peripheral pixels to a first imaging signal (color imaging signal) converted to a digital signal to generate an RGB signal, and YUV conversion processing of the RGB signal. Then, image data composed of YUV signal values are generated and supplied to the control unit 2.

  The image data supplied to the control unit 2 is supplied to the display unit 7 and displayed on the display unit 7 as a live view image. Further, the image data supplied to the control unit 2 at the time of photographing is compressed by the control unit 2, and then, various attribute information is added and stored in the image storage unit 5 as a still image file.

  Still image data recorded as a still image file in the image storage unit 5 is read and expanded by the control unit 2 as necessary in the reproduction mode, and then reproduced (displayed) as a still image on the display unit 7 Ru.

  The image storage unit 5 is constituted by, for example, a flash memory built in the digital camera 1, various memory cards which are attachable to and detachable from the digital camera 1, and a card interface which enables input and output of data to the memory card.

  The display unit 7 includes a color liquid crystal display panel and a display drive circuit that drives the color liquid crystal display panel according to image data and the like supplied from the control unit 2. The display unit 7 performs live view display of a subject image captured by the first imaging device 42 in the recording mode, and displays a still image in the reproduction mode.

  The operation unit 8 includes a plurality of operation switches such as a power button, a shutter, a plurality of photographing mode changeover switches prepared in advance, a zoom lever, and control buttons used for selecting and setting various functions. The operation state of the operation switch in the operation unit 8 is monitored by the control unit 2 as needed.

  The power supply unit 9 is configured of a rechargeable battery, a DC / DC converter, and the like, and supplies power necessary for each operation to each unit of the digital camera 1.

  Next, the operation of the digital camera 1 having the above configuration according to the present invention will be described according to FIG. FIG. 3 is a flowchart showing the main part of the content of the imaging process at the time of shooting of the digital camera 1.

  As shown in FIG. 3, for example, at the time of shooting when the user presses the shutter, the control unit 2 exposes the first image sensor 42 and the second image sensor 43, and the first and second image sensors The image pickup signal (charge signal) accumulated in 42 and 43 is transferred to the image processing engine 45 (step SA1).

  Here, the exposure time of the first imaging device 42 and the second imaging device 43 is an exposure time at which a proper exposure can be obtained in the AE control. Furthermore, this exposure time is twice the exposure time when the single imaging device is used because the incident light beams to the first and second imaging devices 42 and 43 are split by the half prism 41. It is the time which added the time according to the decay of the light accompanying spectroscopy to time.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted in the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaicing processing on the first imaging signal after A / D conversion obtained by the first imaging element 42, interpolates RGB values of each pixel from peripheral pixels, and calculates pixel values of each pixel. A pixel signal consisting of R1, G1 and B1 is generated (step SA2).

  Next, the DSP 48 performs YUV conversion based on a predetermined conversion equation from the generated pixel signal values R1, G1 and B1 to obtain the luminance signal value Y1 and the color difference signal values U1 and V1 of each pixel (step SA3) ). Further, the DSP 48 acquires the signal value of each pixel as the luminance signal value Y2 from the second imaging signal after A / D conversion obtained by the second imaging device 43 (step SA4).

  Next, the DSP 48 determines, based on the luminance signal value Y2 obtained from the second imaging signal, whether the current imaging condition is daytime or nighttime (step SA7). More specifically, if the luminance signal value Y2 is equal to or greater than a predetermined value, the DSP 48 determines that the photographing condition is daytime, and the luminance signal value Y2 is less than a predetermined value. If it is, it is determined that the photographing situation is at night.

  Then, when the DSP 48 determines that the imaging condition is daytime (step SA5: daytime), the luminance signal value Y1 of each pixel obtained from the first imaging signal and the luminance signal among the color difference signal values U1 and V1 Image data (YUV data) in which only the luminance signal value Y2 obtained from the second imaging signal is replaced (step SA6). After that, the DSP 48 outputs the generated image data to the control unit 2 (step SA8).

  Here, in the process of step SA6, the reason for replacing the luminance signal value as described above is as follows.

  That is, in the signal values Y1, U1, and V1 obtained from the first imaging signal, it is generated based on the signal values R1, G1, and B1 including the signal values generated by the interpolation of the peripheral pixels in the demosaicing process. As a result, the resolution sensitivity is inevitably reduced.

  On the other hand, the luminance signal value Y2 obtained from the second imaging signal is a luminance signal value of an actual pixel obtained by the existing conversion element of the second imaging element 43. Accordingly, by adopting the luminance signal value Y2 obtained from the second imaging signal as the luminance signal value, a luminance component having the resolution of the actual pixel is secured in the final image data.

  Therefore, the image data (Y2, U1, V1) generated in step SA6 has higher color reproducibility and resolution sensitivity than the image data (Y1, U1, V1) obtained only from the first imaging signal. It becomes data of the photographed image. As a result, high quality photographed images can be obtained.

  Also, unlike the above, when the DSP 48 determines that the photographing condition is nighttime (step SA5: nighttime), the luminance signal value Y1 of each pixel obtained from the first imaging signal, and the color difference signal values U1, V1. Among them, only the luminance signal Y1 is replaced with a value ((Y1 + Y2) / 2) obtained by averaging the luminance signal value Y1 obtained from the first imaging signal and the luminance signal value Y2 obtained from the second imaging signal. Image data (YUV data) is generated (step SA7). After that, the DSP 48 outputs the generated image data to the control unit 2 (step SA8).

  Here, the reason for replacing the luminance signal value as described above in the process of step SA7 is as follows.

  That is, also in nighttime photographing, if the same processing as in daytime photographing described above is performed, it is possible to secure a luminance component having the resolution of an actual pixel in final image data. However, as a matter of course, when shooting at night, the subject is dark. Furthermore, at the time of shooting, as described above, it is necessary to secure an exposure time about twice that in the case of a single image sensor, but if the exposure time is prolonged, camera shake or subject shake may occur. There is a limit to the exposure time that can be done. Therefore, proper exposure is secured by increasing the imaging sensitivity (gain of the first and second AFEs 46 and 47).

  Therefore, noise corresponding to an increase in imaging sensitivity is present in the second imaging signal as compared with the imaging signal obtained by the single-panel imaging method. Therefore, if the luminance signal is simply replaced as in the normal case, the resolution sensitivity can be increased, but only image data with noticeable noise can be obtained.

  Because of this, in nighttime photography, as described above, the luminance signal value of the final image data, the luminance signal value Y1 obtained from the first imaging signal, and the luminance signal obtained from the second imaging signal A value ((Y1 + Y2) / 2) is obtained by averaging the value Y2. This can avoid the occurrence of noise.

  That is, although both the first imaging signal and the second imaging signal include the noise described above, the noise generated due to the increase in the imaging sensitivity is a variation in sampling adjustment at the time of A / D conversion It is a random noise generated by the influence of etc.

  Therefore, although the number of pixels in which noise occurs in the final image data does not change, the degree of the noise is significantly greater than when the luminance signal value Y2 obtained from the second imaging signal is adopted as it is. Can be reduced to As a result, it is possible to obtain a low noise photographed image which is not inferior to that obtained by the normal single-plate imaging method.

  As described above, in the present embodiment, in the configuration provided with two imaging elements, different imaging processing is performed at the time of imaging when it is determined that the imaging condition is daytime and when it is determined that it is nighttime. Therefore, even when the brightness of the subject is largely different, a higher quality photographed image can be obtained without being influenced by the difference in the photographing condition.

  Moreover, the difference in the content of the imaging process is the difference in pixel signal value used to acquire the component signal value of each pixel in the process of generating the image data. That is, the luminance signal value (Y) of the image data is set to the luminance signal value Y2 obtained from the second imaging signal, or the luminance signal value Y1 obtained from the first imaging signal, and the second imaging signal The luminance signal value Y2 obtained from the above is only the difference between the average value ((Y1 + Y2) / 2) and the average value. Therefore, a high quality photographed image can be obtained by extremely simple signal processing.

  In the present embodiment, the configuration has been described in which whether the photographing condition is daytime or nighttime is automatically determined based on the luminance signal value Y2 obtained from the second imaging signal. However, the difference in the photographing conditions may be determined as follows.

  For example, if the digital camera 1 is provided with a shooting mode corresponding to a plurality of shooting scenes, and there is a shooting mode used in a night-time shooting scene such as a night scene or fireworks as a shooting mode, the user It may be configured to determine whether the shooting condition is daytime or night based on the shooting mode set to. In that case, the process of automatically determining the imaging situation can be omitted.

  In that case, for example, information indicating the shooting mode set at the time of shooting may be appropriately supplied from the control unit 2 to the DSP 48, or the control unit 2 may determine whether the shooting situation is daytime or nighttime. Only the information indicating the determination result may be appropriately supplied to the DSP 48.

  Furthermore, if the high-sensitivity shooting mode specialized for shooting a dark subject is provided as well as the shooting mode used in the night-time shooting scene, the high-sensitivity shooting mode is set. At the time of shooting during shooting, it may be determined that the shooting situation is nighttime for convenience.

  Further, in the present embodiment, the DSP 48 in the image processing engine 45 generates image data composed of YUV signal values based on the first and second imaging signals obtained by the first and second imaging elements 42 and 43. Explained what to do. However, the present invention can also be applied to a digital camera that does not have a signal processing circuit equivalent to the DSP 48. In such a case, the processing by the DSP 48 may be performed by another microcomputer or the like that controls the digital camera as in the control unit 2 described above.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the present embodiment, in the digital camera 1 having the configuration shown in FIGS. 1 and 2, the second image sensor 43 is replaced by the colorless color filter 43a, and the ratio contributing to the luminance signal shown in FIG. 4 is large. A monochromatic color filter 43b consisting only of G is provided. Further, also in the present embodiment, at the time of shooting, the DSP 48 performs later-described signal processing of the contents according to the shooting situation which is automatically determined at the time of shooting.

  Hereinafter, the operation according to the present invention of the digital camera of the present embodiment will be described according to the flowchart of FIG. 5 corresponding to FIG.

  Also in the present embodiment as shown in FIG. 5, for example, at the time of shooting when the user presses the shutter, the control unit 2 exposes the first image sensor 42 and the second image sensor 43, and The imaging signal (charge signal) accumulated in the second imaging elements 42 and 43 is transferred to the image processing engine 45 (step SB1). The exposure time of the first imaging device 42 and the second imaging device 43 is as described above.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted in the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaicing processing on the first imaging signal after A / D conversion obtained by the first imaging element 42, interpolates RGB values of each pixel from peripheral pixels, and calculates pixel values of each pixel. A pixel signal consisting of R1, G1 and B1 is generated (step SB2).

  Thereafter, in the present embodiment, the DSP 48 acquires the signal value of each pixel as the signal value G2 of the G component from the second imaging signal after A / D conversion obtained by the second imaging device 43 (step SB3) .

  Next, the DSP 48 determines, based on the signal value G2 obtained from the second imaging signal, whether the current imaging situation is daytime or nighttime (step SB4).

  Then, when the DSP 48 determines that the imaging condition is daytime (step SB4: daytime), only the signal value of the G component among the pixel signal values R1, G1, B1 of each pixel obtained from the first imaging signal Are replaced with the signal value G2 obtained from the second imaging signal, and YUV conversion is performed on the pixel signal values (R1, G2, B1) (step SB5).

  Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y of each pixel and the color difference signal values U and V. Then, the DSP 48 outputs the generated image data to the control unit 2 (step SB7).

  Here, in the process of step SB5, the reason for replacing the pixel value of the G component as described above is as follows.

  That is, the pixel signal values R1, G1, and B1 obtained from the first imaging signal are generated by interpolation of the peripheral pixels in the demosaicing process, and the resolution sensitivity is inevitably reduced.

  On the other hand, the signal value G2 of the G component of each pixel obtained from the second image pickup signal is the signal value of the actual pixel obtained by the existing conversion element of the second image sensor 43. As such, by adopting the signal value G2 obtained from the second imaging signal as the signal value of the G component, in order to secure the luminance component having the resolution of the actual pixel and the G component in the final image data. is there.

  Therefore, the image data (YUV data) generated in step SB5 has color reproducibility compared to the image data generated based on the pixel signal values R1, G1, and B1 of each pixel obtained only from the first imaging signal. And the data of the captured image with high resolution sensitivity. As a result, high quality photographed images can be obtained. With regard to color reproducibility in particular, when a green area is present in the subject, the color reproducibility of the area can be reliably improved.

  Also, differently from the above, when the DSP 48 determines that the photographing condition is night (step SB4: night), G among the pixel signal values R1, G1, B1 of the respective pixels obtained from the first imaging signal Replace only the signal value of the component with a value ((G1 + G2) / 2) obtained by averaging the signal value G1 obtained from the first imaging signal and the signal value G2 obtained from the second imaging signal YUV conversion is performed on the signal values (R1, ((G1 + G2) / 2), B1) (step SB6).

  Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y of each pixel and the color difference signal values U and V. Then, the DSP 48 outputs the generated image data to the control unit 2 (step SB7).

  Here, in the process of step SB6, the reason for replacing the luminance signal value as described above is as follows.

  That is, as described in the first embodiment, the second imaging signal has noise corresponding to an increase in imaging sensitivity as compared with an imaging signal obtained by a normal single-panel imaging method. Therefore, as in the case of the normal mode, if the signal value of the G component is simply replaced with the signal value G2 obtained from the second imaging signal, the resolution sensitivity can be increased, but only image data with noticeable noise can be obtained. I can not do it.

  Since it concerns, in night time photography, as mentioned above, the signal value of G component of the step before YUV conversion, the signal value G1 obtained from the 1st image pick-up signal, and the signal obtained from the 2nd image pick-up signal It is set as the value ((G1 + G2) / 2) which averaged with the value G2. This can avoid the occurrence of noise.

  Therefore, although the number of pixels in which noise occurs in the final image data does not change, the degree of noise is determined by using the signal value G2 of the G component obtained from the second imaging signal as it is. It can be significantly reduced. As a result, it is possible to obtain a low noise photographed image which is not inferior to that obtained by the normal single-plate imaging method.

  As described above, also in the present embodiment, in the configuration including the two imaging elements as in the first embodiment, different imaging at the time of imaging in the case where the imaging condition is determined as daytime and in the case where it is determined as nighttime I made it to process. Therefore, even when the brightness of the subject is largely different, a higher quality photographed image can be obtained without being influenced by the difference in the photographing condition.

  Moreover, also in the present embodiment, the difference in the content of the imaging process is the difference in the pixel signal value used to acquire the component signal value of each pixel in the process of generating the image data. That is, the signal value G2 obtained from the second imaging signal is used to acquire the luminance signal value Y and the color difference signal values U and V of the image data, or the signal value G1 obtained from the first imaging signal The only difference is whether the signal value G2 obtained from the second imaging signal is averaged ((G1 + G2) / 2). Therefore, also in the present embodiment, a high quality photographed image can be obtained by extremely simple signal processing.

  In the present embodiment, as in the first embodiment, it is not necessary to automatically determine whether the shooting status is daytime or night, and it is determined from the shooting mode set by the user at the time of shooting. The specific judgment method etc. is optional. Further, the signal processing by the DSP 48 described in the present embodiment can also be performed by the control unit 2.

(Embodiment 3)
Next, a third embodiment of the present invention will be described. In this embodiment, in the digital camera 1 having the configuration shown in FIG. 1, the imaging unit 4 differs from that shown in FIG. 2A in the first embodiment, and the configuration shown in FIG. It is possessed.

  That is, in the present embodiment, no color filter is provided on the photosensitive surface of the second image sensor 43, and the space between the second image sensor 43 and the half prism 41 is shown in FIG. A rotating filter plate 51 is provided.

  The rotary filter plate 51 is provided with color filters 51a, 51b, 51c of respective colors of RGB in regions divided into three in the circumferential direction, and is appropriately rotated by a motor 52. The operation of the motor 52 is controlled by the control unit 2, and the motor 52 changes the rotational position of the rotary filter plate 51 according to the instruction of the control unit 2, whereby any one of the RGB color filters is Is applied to the imaging device 43 of FIG.

  In addition, about another structure, since it is the same as that of what was shown to Fig.2 (a), description is abbreviate | omitted by attaching | subjecting the same code | symbol to an identical part.

  Hereinafter, the operation according to the present invention of the digital camera of the present embodiment will be described according to the flowchart of FIG. FIG. 7 is a flowchart showing the main part of the operation at the time of shooting according to the present embodiment.

  Explained below, in the present embodiment, for example, at the time of shooting when the user presses the shutter, the control unit 2 first determines the shooting condition (step SC1). Specifically, the control unit 2 controls the color tone of the object to be photographed from the RGB color component information of the photographed image which is picked up by the first image pickup device 42 immediately before photographing and displayed as a live view image on the display unit 7. The shooting situation regarding.

  That is, the control unit 2 determines that the color tone of the object to be photographed is bluish-blue if the proportion occupied by the B component in the photographed image is equal to or higher than the specified, and if the proportion occupied by the G component is equal to or higher than the specified It is judged that the color tone of the object to be photographed is a greenish color tone, and if the ratio occupied by the R component is equal to or more than the specified, it is judged that the color tone of the object to be photographed is a reddish color.

  Then, when the color tone of the object to be photographed does not correspond to any of the above ("Other" in FIG. 7), the control unit 2 immediately exposes only the first imaging element 42, and the first imaging element 42 Are transferred to the image processing engine 45 (step SC2).

  The imaging signals transferred to the image processing engine 45 are amplified and A / D converted by the first AFEs 46 and 47, and then the DSP 48 performs normal signal processing.

  That is, the DSP 48 performs demosaicing processing on the first imaging signal after A / D conversion obtained by the first imaging device 42, interpolates RGB values of each pixel from peripheral pixels, and calculates pixel values of each pixel. A pixel signal consisting of R1, G1 and B1 is generated (step SC3).

  Next, the DSP 48 performs YUV conversion based on a predetermined conversion equation from the generated pixel signal values R1, G1 and B1 to acquire the luminance signal value Y and the color difference signal values U and V of each pixel (step SC4 ). After that, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

  Therefore, when the control unit 2 determines that the color tone of the subject to be photographed does not correspond to any of bluish, greenish, and redish, the same as that obtained by the normal single-chip imaging method as a photographed image A quality photographed image can be obtained.

  On the other hand, unlike the above, when the control unit 2 determines that the color tone is bluish, for example, when the object to be photographed is “blue sky” or “sea”, the control unit 2 may be a rotary filter plate. 51 is rotated to a predetermined position, and the B color filter is applied to the second imaging element 43 (step SC5).

  The control unit 2 applies a G color filter to the second imaging element 43 when it determines that the color tone is greenish, for example, as in the case where the object to be photographed is "grassland" SC6).

  Furthermore, the control unit 2 applies the color filter of B to the second imaging element 43 when it is determined that the color tone is reddish, for example, as in the case where the shooting target is “sunset” SC7).

  Although not shown in FIG. 7, the control unit 2 supplies information indicating the filter color applied to the second imaging device 43 to the DSP 48 in the processing of each of steps SC5 to SC7 described above.

  Subsequently, the control unit 2 exposes the first imaging element 42 and the second imaging element 43 in a state in which the color filter of any one of RGB is applied, and the first and second imaging elements 42, The imaging signal (charge signal) accumulated in 43 is transferred to the image processing engine 45 (step SC8). The exposure time of the first imaging device 42 and the second imaging device 43 is as described in the first embodiment.

  The imaging signal transferred to the image processing engine 45 is amplified and A / D converted in the first and second AFEs 46 and 47, and thereafter the following signal processing is performed by the DSP 48.

  First, the DSP 48 performs demosaicing processing on the first imaging signal after A / D conversion obtained by the first imaging element 42, interpolates RGB values of each pixel from peripheral pixels, and calculates pixel values of each pixel. A pixel signal consisting of R1, G1 and B1 is generated (step SC9).

  Next, the DSP 48 confirms the filter color applied to the second imaging element 43 from the information of the filter color supplied in advance from the control unit 2 (step SC10).

  Then, when the filter color is B (step SC10: B), the DSP 48 generates the B component of the signal value of each pixel from the second imaging signal after A / D conversion obtained by the second imaging device 43. Is acquired as the signal value B2 of (step SC11).

  Next, among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal, only the signal value of the B component is replaced with the signal value B2 obtained from the second imaging signal, and the pixel YUV conversion is performed on the signal values (R1, G1, B2) (step SC12). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y of each pixel and the color difference signal values U and V. After that, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

  Here, in the image data generated in step SC16, the signal value of the B component used for YUV conversion is the signal value of the actual pixel obtained by the existing conversion element of the second imaging element 43. The data of the photographed image is higher in color reproducibility than the image data generated based on the pixel signal values R1, G1, and B1 of the respective pixels obtained only from the first imaging signal. In particular, it becomes data of a photographed image in which high color reproducibility is ensured by the blue portion. As a result, for example, when the shooting target is "blue sky" or "sea", it is possible to obtain a high-quality shot image in which the color of the blue sky portion or the sea portion is more faithfully reproduced.

  Further, when the filter color is G unlike the above (step SC10: G), the DSP 48 generates a signal of each pixel from the second imaging signal after A / D conversion obtained by the second imaging device 43. The value is acquired as the signal value G2 of the G component (step SC13).

  Next, among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal, only the signal value of the G component is replaced with the signal value G2 obtained from the second imaging signal, and the pixel YUV conversion is performed on the signal values (R1, G2, B1) (step SC14). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y of each pixel and the color difference signal values U and V. After that, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

Here, in the image data generated in step SC14, the signal value of the G component used for YUV conversion is the signal value of the actual pixel obtained by the existing conversion element of the second imaging element 43. The data of the photographed image is higher in color reproducibility than the image data generated based on the pixel signal values R1, G1, and B1 of the respective pixels obtained only from the first imaging signal. In particular, high color reproducibility can be ensured by the green portion. In addition, since the pixel signal value of the G component contributes to the luminance signal at a high rate, it becomes data of a photographed image with high resolution sensitivity. as a result,
For example, when the object to be photographed is a "grassland", the color of the grass part can be reproduced more faithfully, and a high-quality photographed image with high resolution sensitivity can be obtained.

  Furthermore, unlike the above, when the filter color is R (step SC10: R), the DSP 48 generates a signal of each pixel from the second imaging signal after A / D conversion obtained by the second imaging device 43. The value is acquired as the signal value R2 of the R component (step SC15).

  Next, the DSP 48 replaces only the signal value of the R component among the pixel signal values R1, G1, and B1 of each pixel obtained from the first imaging signal with the signal value R2 obtained from the second imaging signal. , YUV conversion is performed on the pixel signal values (R2, G1, B1) (step SC16). Thereby, the DSP 48 generates image data (YUV data) including the luminance signal value Y of each pixel and the color difference signal values U and V. After that, the DSP 48 outputs the generated image data to the control unit 2 (step SC17).

Here, in the image data generated in step SC16, the signal value of the R component used for YUV conversion is the signal value of the actual pixel obtained by the existing conversion element of the second imaging element 43. The data of the photographed image is higher in color reproducibility than the image data generated based on the pixel signal values R1, G1, and B1 of the respective pixels obtained only from the first imaging signal. In particular, it becomes data of a photographed image in which high color reproducibility is ensured by the red part. as a result,
For example, when the shooting target is a “sunset”, it is possible to obtain a high-quality shot image in which the color of the reddish sky part is more faithfully reproduced.

  As described above, in the present embodiment, it is possible to apply a color filter of a plurality of colors to the second imaging element 43, and change the filter color of the applied color filter in accordance with the difference in shooting conditions such as the color tone of the shooting target. At the same time, when imaging conditions are different, different imaging processing is performed at the time of imaging. Therefore, even when the color tone of the object to be photographed is different, a higher quality photographed image can be obtained without being influenced by the difference in the photographing conditions.

  Further, in the present embodiment, since the filter colors to be applied are R, G, and B, for example, when the shooting target is "blue sky", "sea", "grassland", or "evening scene" It is possible to obtain a high quality photographed image.

  Moreover, also in the present embodiment, the difference in the content of the imaging process is the difference in the pixel signal value used to acquire the component signal value of each pixel in the process of generating the image data. That is, when acquiring the luminance signal value Y and the color difference signal values U and V of the image data, it is only the difference between using the pixel signal value obtained from the second imaging signal as the pixel signal value of any of RGB . Therefore, also in the present embodiment, a high quality photographed image can be obtained by extremely simple signal processing.

  In the present embodiment, the control unit 2 is imaged by the first imaging device 42 as to whether or not the color tone of the subject to be photographed corresponds to any one of bluish, greenish and reddish as the photographing condition. The configuration for automatically determining from the image has been described.

  However, in the determination of the shooting condition at the time of shooting, for example, the digital camera is made to allow the user to select an arbitrary shooting scene from among a plurality of shooting scenes, and shooting under shooting conditions respectively corresponding to the selected shooting scene If the configuration is such that a proper shooting mode is provided, the shooting situation (color tone of the shooting target) may be determined based on the shooting scene selected by the user. In that case, the process of automatically determining the imaging situation can be omitted.

  Also, in that case, as the shooting condition data indicating the shooting condition according to each shooting scene, the present invention is applied to the use of the second image sensor 43 at the time of shooting and the use of the second image sensor 43 Information on the color of the color filter to be set may be set in advance.

  Also in the present embodiment, the signal processing by the DSP 48 described above can be performed by the control unit 2.

  Further, instead of the combination of the rotary filter plate 51 and the second imaging device, three independent imaging devices provided with color filters of each color of RGB are used, and image signals from the three independent imaging devices according to the shooting scene May be used.

As mentioned above, although some embodiments of the present invention and its modification were explained, they can be suitably changed if within the limits which can acquire the operation effect of the present invention, and the embodiment after a change is also a claim. The invention described in the scope of the invention, and the invention equivalent to the invention are included in the scope of the invention. In the following, the invention described in the original claims of the present application is appended.
[Claim 1]
Multiple imaging elements with different color filters,
A judging means for judging the photographing condition;
An image generation unit configured to generate image data based on different imaging signals obtained by the plurality of imaging elements;
And signal processing control means for performing imaging signal processing of different contents predetermined according to the photographing condition determined by the determination means when the image data is generated by the image generation means. Imaging device.
[Claim 2]
The difference in the content of the imaging signal processing is the difference in the combination of pixel signal values among the plurality of imaging elements used to acquire the component signal value of each pixel in the process of generating image data by the image generation means. The imaging device according to claim 1, characterized in that:
[Claim 3]
The content of the image pickup signal processing is processing content relating to either the generation processing of the luminance value of each pixel in the image data or the generation processing of the luminance value of each pixel and the color difference component value. The imaging device according to 1 or 2.
[Claim 4]
The plurality of imaging elements are a first imaging element provided with a color filter used in a single-plate type imaging method in which each color of RGB is disposed, and a second imaging element provided with a monochromatic color filter The imaging device according to claim 1 or 2, characterized in that
[Claim 5]
5. The image pickup apparatus according to claim 4, wherein the color of the color filter provided in the second image pickup element is colorless.
[Claim 6]
The signal processing control means is responsive to the photographing condition judged by the judgment means at the time of generation of the image data by the image generation means according to the brightness of the subject as the photographing condition judged by the judgment means. 6. The image pickup apparatus according to claim 5, wherein image pickup signal processing of different contents determined in advance is performed.
[Claim 7]
The signal processing control means, when the photographing condition judged by the judgment means is a first photographing condition where it can be judged that the brightness of the subject is equal to or more than a predetermined reference, the image data is outputted to the image generation means In the generation processing of the luminance value of each pixel in the image processing, the imaging signal processing of the content adopting the luminance value in the imaging signal obtained by the second imaging element as the luminance value of each pixel is performed. 6. The imaging device according to 6.
[Claim 8]
When the image processing condition determined by the determining unit is a second image capturing condition in which the brightness of the subject is less than a predetermined reference, the signal processing control unit causes the image generating unit to generate the image data. In the process of generating the luminance value of each pixel in the above, as the luminance value of each pixel, the luminance value based on the imaging signal obtained by the first imaging device and the luminance value in the imaging signal obtained from the second imaging device 8. The image pickup apparatus according to claim 7, wherein the image pickup signal processing of the content adopting the luminance value obtained by averaging is performed.
[Claim 9]
5. The image pickup apparatus according to claim 4, wherein a color of a color filter provided in the second image pickup element is G among the RGB.
[Claim 10]
The signal processing control means is responsive to the photographing condition judged by the judgment means at the time of generation of the image data by the image generation means according to the brightness of the subject as the photographing condition judged by the judgment means. 10. The image pickup apparatus according to claim 9, wherein image pickup signal processing of different contents predetermined in advance is performed.
[Claim 11]
The signal processing control means, when the photographing condition judged by the judgment means is a first photographing condition where it can be judged that the brightness of the subject is equal to or more than a predetermined reference, the image data is outputted to the image generation means In the generation process of the luminance value and the color difference component value of each pixel in the pixel, the pixel value based on the imaging signal obtained by the first imaging device is adopted as the pixel value of R component and B component of each pixel, and each pixel 11. The image pickup apparatus according to claim 10, wherein image pickup signal processing of contents adopting the pixel value in the image pickup signal obtained by the second image pickup element as the pixel value of the G component is performed.
[Claim 12]
When the image processing condition determined by the determining unit is a second image capturing condition in which the brightness of the subject is less than a predetermined reference, the signal processing control unit causes the image generating unit to generate the image data. In the generation process of the luminance value and the color difference component value of each pixel in the pixel, the pixel value based on the imaging signal obtained by the first imaging device is adopted as the pixel value of R component and B component of each pixel, and each pixel Content that adopts the luminance value obtained by averaging the pixel value in the imaging signal obtained by the first imaging device and the pixel value based on the imaging signal obtained by the second imaging device as the pixel value of the G component of The image pickup apparatus according to claim 11, wherein the image pickup signal processing is performed.
[Claim 13]
The plurality of imaging elements are a first imaging element provided with a color filter used in a single-plate type imaging method in which each color of RGB is arranged, and a second imaging element to which color filters of different colors can be applied And
Color filter means including a plurality of color filters different in color which are selectively applied to the second imaging element;
Filter control means for applying, to the second image sensor, a color filter of a predetermined color determined in advance according to the photographing situation determined by the determination means among the plurality of color filters in the color filter means The imaging device according to any one of claims 2 to 12, further comprising:
[Claim 14]
14. The image pickup apparatus according to claim 13, wherein each color of the plurality of color filters in the color filter means is any one of RGB colors.
[Claim 15]
The signal processing control unit may determine that the photographing condition determined by the determination unit is not necessary to use the imaging signal obtained by the second imaging device to generate the image data. When the image generation unit generates the luminance value and the color difference component value of each pixel in the image data, the imaging signal obtained by the first imaging device as the pixel value of each color of RGB of each pixel The image pickup apparatus according to claim 14, characterized in that image pickup signal processing of contents adopting only pixel values based on is performed.
[Claim 16]
The signal processing control means may be a second imaging situation in which the imaging situation determined by the determination means may be effective to use the imaging signal obtained by the second imaging element to generate the image data. In the case where the image generation unit generates the luminance value and the color difference component value of each pixel in the image data, the first image pickup element is used as the pixel value of two colors other than the predetermined color of each pixel. Image pickup signal processing of a content that adopts a pixel value based on the obtained image pickup signal and that adopts a pixel value based on an image pickup signal obtained by the second image pickup element as the pixel value of the predetermined color of each pixel The imaging device according to claim 14 or 15, characterized in that:
[Claim 17]
The imaging device according to any one of claims 1 to 16, wherein the determination unit determines the imaging condition based on arbitrary setting information indicating an imaging condition set by the user at the time of imaging. .
[Claim 18]
A determination step of determining a photographing situation;
Generating an image data based on different imaging signals obtained by a plurality of imaging devices having different color filters,
At the time of generation of the image data in the image generation step, imaging signal processing of different contents predetermined according to the photographing situation determined in the determination step is performed.
[Claim 19]
A computer having an imaging device having a plurality of imaging elements provided with different color filters,
A judging means for judging the photographing condition;
An image generation unit configured to generate image data based on different imaging signals obtained by the plurality of imaging elements;
It is characterized in that it functions as signal processing control means for performing imaging signal processing of different contents predetermined according to the photographing situation judged by the judgment means at the time of generation of the image data by the image generation means. And the program to be.

Reference Signs List 1 digital camera 2 control unit 3 lens unit 4 imaging unit 5 image storage unit 6 program storage unit 7 display unit 8 operation unit 9 power supply unit 41 half prism 42 first image sensor 42 a color filter 43 second image sensor 43 a color filter 43b color filter 45 image processing engine 46 first AFE
47 Second AFE
48 DSP
51 rotating filter plate 51a color filter 51b color filter 51c color filter 52 motor

Claims (7)

Two images obtained by a first imaging device provided with a color filter used in a single-plate imaging method in which each color of RGB is arranged, and a second imaging device to which color filters of different colors can be applied An image generation unit that generates image data using an imaging signal;
Color filter means including a plurality of color filters different in color which are selectively applied to the second imaging element;
A judging means for judging the photographing condition;
Filter control means for applying, to the second image sensor, a color filter of a predetermined color determined in advance according to the photographing situation determined by the determination means among the plurality of color filters in the color filter means ,
A control unit configured to control the image generation unit to generate image data using the two imaging signals according to the imaging condition determined by the determination unit;
An image data generating apparatus comprising: 2. The image data generation apparatus according to claim 1 , wherein each color of the plurality of color filters in the color filter means is any one of RGB colors. The control unit is a first imaging condition in which the imaging condition determined by the determination unit can be determined to be unnecessary to use the imaging signal obtained by the second imaging device to generate the image data. In the case where the image generation unit generates the luminance value and the color difference component value of each pixel in the image data, the image generation unit is based on an imaging signal obtained by the first imaging device as a pixel value of each color of RGB of each pixel. 3. The image data generation apparatus according to claim 2 , wherein imaging signal processing of contents adopting only pixel values is performed. The control means is a second imaging situation in which the imaging situation determined by the determination means can be determined to be effective in using the imaging signal obtained by the second imaging device to generate the image data. In the case where the image generation unit generates the luminance value and the color difference component value of each pixel in the image data, it is obtained by the first imaging device as pixel values of two colors other than the predetermined color of each pixel. The image signal processing of the content which adopts the pixel value based on the image pickup signal, and adopts the pixel value based on the image pickup signal obtained by the second image pickup element as the pixel value of the predetermined color of each pixel is performed The image data generation apparatus according to claim 2 or 3 , characterized in that The image data according to any one of claims 1 to 4 , wherein the determination means determines the photographing condition based on arbitrary setting information indicating the photographing condition set by the user at the time of photographing. Generator. Two images obtained by a first imaging device provided with a color filter used in a single-plate imaging method in which each color of RGB is arranged, and a second imaging device to which color filters of different colors can be applied An image generation step of generating image data using an imaging signal;
A determination step of determining a photographing situation;
Among a plurality of color filters different from each other which are selectively applied to the second image sensor, a color filter of a predetermined color determined in advance according to the photographing situation determined in the determination step A filter control step applied to the second image sensor;
A control step of controlling to generate image data using the two imaging signals in the image generation step according to the imaging situation determined in the determination step;
A method of generating image data, comprising: Two images obtained by a first imaging device provided with a color filter used in a single-plate imaging method in which each color of RGB is arranged, and a second imaging device to which color filters of different colors can be applied A computer included in an image data generation apparatus that generates image data using an imaging signal;
Judgment means to judge the shooting situation,
Among a plurality of color filters different from each other which are selectively applied to the second image sensor, a color filter of a predetermined color determined in advance according to the photographing condition determined by the determination means Filter control means applied to the second imaging device,
According to the shooting situation is determined by the determination means, by the Hare system Gosuru control means generates the image data by using the two imaging signals,
A program characterized by acting as

Images