JP5062064B2 - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for compressing a dynamic range of an input image, and an imaging apparatus using the same.

  In recent years, in an imaging apparatus applied to a digital camera such as a digital still camera or a digital video camera, the luminance range of a subject (difference between the lowest luminance and the highest luminance, the lowest density in an image) One of the themes is to expand the dynamic range (DR), that is, the difference from the maximum density. Various techniques for expanding the dynamic range have been researched and developed.

  For example, Patent Document 1 discloses a technique for expanding a dynamic range by capturing a plurality of images with different exposure amounts, selecting an image portion of an appropriate level from the plurality of images, and combining the plurality of images. Has been. Further, for example, in Patent Document 2, signal charges of the photoelectric conversion unit obtained by one exposure are held by a plurality of capacitors, and a plurality of signal charges are read out by different capacitors among the plurality of capacitors. A technique for expanding a dynamic range by adding a plurality of signal charges is disclosed. Further, for example, Patent Document 3 discloses a technique for expanding a dynamic range by converting a signal charge transferred from a photoelectric conversion unit into a signal voltage by a charge-voltage conversion unit including a plurality of capacitors having different voltage dependencies. It is disclosed.

  Further, for example, in Patent Document 4, in order to expand the dynamic range, a photosensitive means capable of generating a photocurrent corresponding to the amount of incident light, a MOS transistor for inputting the photocurrent, and a subthreshold current for this MOS transistor. There is disclosed a solid-state imaging device that includes a biasing means for biasing in a state where it can flow and that outputs an electrical signal that has been subjected to photoelectric conversion in a natural logarithm with respect to an incident light amount. And in patent document 4, in the solid-state image sensor which converts a photocurrent into a logarithmic voltage using the subthreshold characteristic of a MOS transistor in order to expand a dynamic range, by giving a specific reset voltage to the said MOS transistor, solid state A solid-state image sensor that can automatically switch between the logarithmic operation state and the linear operation state that outputs an electrical signal linearly photoelectrically converted with respect to the incident light amount has been proposed. Yes.

  As described above, while the dynamic range of an imaging apparatus is being expanded, the dynamic range (the number of bits representing a pixel level) in an apparatus that transmits, stores, or displays an image is currently smaller than the dynamic range in the imaging apparatus. May be relatively narrow. In such a case, even if the dynamic range in the imaging apparatus is expanded, it is difficult to handle all the obtained information. For this reason, a dynamic range compression technique for converting a wide dynamic range image into a narrower dynamic range image has been researched and developed.

  In this dynamic range compression technique, many techniques based on Retinex theory (see Non-Patent Document 1) have been reported. According to this Retinex theory, the light entering the human eye is determined by the product of the illumination light and the reflectance of the object, but the human eye perception shows a strong correlation with the reflectance. For this reason, if only the dynamic range of the illumination light component is narrowed in an image with a wide dynamic range, the reflectance component having a strong correlation with the perception of the human eye is maintained, and an image with a high contrast and a compressed dynamic range is maintained. Can be obtained. That is, an image with a compressed dynamic range can be obtained by adjusting the density of the bright and dark portions while maintaining the gradation of the intermediate density portion of the image.

  For example, in Patent Document 5, a low-frequency component of an image signal S0, that is, a blur image S1 that fluctuates gently, is created by filtering the image signal S0 of the original image using a low-pass filter. The value of the blurred image signal S1 is inverted by conversion using the look-up table to create a processed image signal S4 with a compressed dynamic range, and this processed image signal S4 is added to the image signal S0 of the original image to generate an image. A technique for compressing the dynamic range of the original image by calculating the signal S5 is disclosed. Here, in this dynamic range compression technique, when the dynamic range is strongly compressed, for example, at the boundary where the luminance changes abruptly, such as the boundary between the subject and the background, the boundary (the contour of the subject in the above example) is used. A band-like pseudo contour with a certain width is generated along the halo (halo effect). For this reason, in Patent Document 5, a plurality of blurred image signals representing the blurred image of the original image are created from the image signal of the original image, and one synthesized blurred image signal is created by combining the plurality of blurred image signals. Then, an image processing method has been proposed that can suppress the occurrence of the pseudo contour by performing compression processing of the dynamic range of the original image on the image signal of the original image based on the synthesized blurred image signal. . Here, the compression function ftotal (α) used for the dynamic range compression processing in the image processing method proposed in Patent Document 5 calculates the overall compression rate α having the profile shown in FIG. It is generated by setting (α) and correcting it with the compression function flight (α) of the bright part and the compression function fdark (α) of the dark part. The overall compression rate α having the profile shown in FIG. 11 is the compression rate α when the dynamic range is smaller than the threshold value DRth, and when the dynamic range is larger than the threshold value DRmax (> DRth). Is fixed at the lower limit value αmax, and when the dynamic range is not less than the threshold value DRth and not more than the threshold value DRmax, the compression rate α is linear.

  For example, Patent Document 6 discloses an image processing method for converting an input image into an image having a relatively small dynamic range, and smoothing each of a plurality of divided input images. Performing a smoothing process for generating a plurality of smoothed images having different degrees of smoothing, an edge strength calculating process for calculating an edge strength based on the plurality of smoothed images, and based on the calculated edge strength, Coefficient calculation for calculating a coefficient for converting each pixel value of the input image based on a synthesis process for synthesizing the plurality of smooth images and a synthesized smooth image generated by synthesizing the plurality of smooth images. An image processing method including a process and a pixel value conversion process of converting each pixel value of the input image by the calculated coefficient is disclosed. Here, in the dynamic range compression processing in the image processing method disclosed in Patent Document 6, the coefficient calculation function having the profile shown in FIG. 12A from each pixel value R (x, y) of the synthesized smoothed image R. A coefficient C (x, y) (= F (R (x, y))) is calculated using F (l), and the pixel value I (x, y) of the input image I is calculated as the coefficient C (x, y). ) Is multiplied, the dynamic range is compressed. This coefficient calculation function F (l) is calculated from the level conversion function T (l) having the profile shown in FIG. 12B by F (l) = T (l) / l, and the minimum value of the coefficient C Cmin is given by a ratio Mmax / Lmax between the maximum value Lmax of the input level and the maximum value Mmax of the output level in the level conversion function T (l). The level conversion function T (l) includes a gamma function T (l) = (l / Lmax) g × Lmax, a LOG function T (l) = (log (l) / log (Lmax)) × Lmax, A histogram equalization method that adaptively changes the level conversion function according to the frequency distribution of the pixel level of the input image can also be used.

Patent Document 7 discloses an imaging apparatus that obtains the amplitude of a reflectance component in dynamic range compression processing, and sets the compression rate high when the amplitude is large.
Japanese Patent Laid-Open No. 63-306777 JP 2000-165754 A JP 2000-165755 A JP 2002-77733 A JP 2001-298619 A Japanese Patent No. 37509797 JP 2007-82181 A EHLand, JJMcCann, “Lightness and retinex theory”, Journalof the Optical Society of America, 61 (1), 1 (1971)

  However, in Patent Document 7, there is a problem that the scale of means for obtaining the amplitude of the reflectance component is increased and the cost is increased. In addition, the processing time becomes longer. Further, in the dynamic range compression disclosed in Patent Literature 5 and Patent Literature 6, since the above-described function is used, image quality degradation may occur depending on the input image. For example, in the case of Patent Document 6, since the minimum value Cmin of the coefficient C is determined by the maximum value (Mmax, Lmax) of the level conversion function T (l), there are one or several bright pixels that jump out in the input image. In some cases, most of the pixels in the input image become dark without being compressed, and image quality degradation occurs.

  The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing apparatus that can perform dynamic range compression more appropriately with a simpler configuration. It is to provide an image pickup apparatus.

  As a result of various studies, the present inventor has found that the above object is achieved by the present invention described below. That is, the image processing apparatus according to an aspect of the present invention includes a base component image generation unit that generates a base component image from an input image based on a difference in spatial frequency components, and the base based on the spatial frequency of the base component image. A compression characteristic generation unit that generates a compression characteristic for compressing a component image; and a compression base having a dynamic range smaller than that of the base component image by using the compression characteristic generated by the compression characteristic generation unit for the base component image. And a base component image compression unit that generates a component image. The image processing method according to another aspect of the present invention is generated in the base component image generation step for generating a base component image from the input image based on the difference in spatial frequency components, and in the base component image generation step. By using a compression characteristic generation step for generating a compression characteristic for compressing the base component image based on a spatial frequency of the base component image, and a compression characteristic generated in the compression characteristic generation step for the base component image And a base component image compression step for generating a compressed base component image having a dynamic range smaller than that of the base component image.

  In the image processing apparatus and the image processing method configured as described above, the base component image is generated from the input image based on the difference in the spatial frequency component, and the compression characteristics are generated based on the spatial frequency of the base component image. Compression characteristics are set. Also, since the spatial frequency of the base component image is determined at the device design stage, it does not fluctuate, and the compression characteristic generation unit (compression characteristic generation process) is simpler than generating compression characteristics based on the image. Since the image processing apparatus and the image processing method can be configured, the cost can be reduced. Further, since the compression characteristic can be easily obtained, the processing time can be shortened.

  The image processing apparatus further includes a compressed image generation unit that generates a compressed image in which a dynamic range of the input image is compressed based on the input image, the base component image, and the compressed base component image. Features.

  According to this configuration, since the compressed image generation unit is further provided, it is possible to provide an image processing apparatus that generates a compressed image.

  In the above-described image processing device, the base component image generation unit includes a low-pass filter unit, and the compression characteristic generation unit performs the compression based on at least one of a cutoff frequency and a window size of the low-pass filter unit. A characteristic is generated.

  According to this configuration, since the compression characteristic generation unit generates the compression characteristic based on at least one of the cutoff frequency and the window size of the low-pass filter unit, it is possible to easily generate a more appropriate compression characteristic.

  In the above-described image processing device, the base component image generation unit includes a nonlinear filter unit, and the compression characteristic generation unit performs the compression based on at least one of a cutoff frequency and a window size of the nonlinear filter unit. A characteristic is generated.

  According to this configuration, since the compression characteristic generation unit generates the compression characteristic based on at least one of the cutoff frequency and the window size of the nonlinear filter unit, a more appropriate compression characteristic can be easily generated.

  In the above-described image processing apparatus, the compression characteristic generation unit decreases the compression rate as the cutoff frequency increases, and increases the compression rate as the window size increases.

  According to this configuration, the compression characteristic generation unit can generate a more appropriate compression characteristic.

  Further, in the above-described image processing device, the base component image generation unit generates a plurality of smooth images having different spatial frequency components with different cutoff frequencies and different resolutions from the input image. The base component image having the same resolution as the input image is generated based on the smooth image.

  Here, the base component image generation unit generates, for example, a low-pass filter that generates a low spatial frequency image composed of a spatial frequency component lower than the cutoff frequency by performing low-pass filter processing on the input image at a predetermined cutoff frequency set in advance. A down-sampling unit that generates a low-resolution image having a resolution smaller than that of the input image, that is, a smooth image, by down-sampling the low-spatial-frequency image at a predetermined first rate, The up-sampling unit generates an up-resolution image by performing up-sampling processing on one of the plurality of blurred images at a second rate corresponding to the predetermined first rate. And obtained by upsampling processing in the upsampling unit. A blur image synthesizing unit that generates a synthesized blur image by synthesizing an up-resolution image and a blur image having a resolution equal to the resolution of the up-resolution image, and low-pass filter processing by the low-pass filter unit with respect to the input image And the down-sampling process by the down-sampling unit are generated a predetermined number of times to generate the plurality of smoothed images, and the up-sampling process by the up-sampling unit and the blurred image synthesis are performed on the generated plurality of blurred images The base component image may be generated by repeating the synthesis process by the unit a predetermined number of times. With this configuration, it is possible to use a low-pass filter having a smaller size than the case where the input image is subjected to low-pass filter processing in one stage by performing low-pass filter processing on the input image in multiple stages. The amount of calculation is reduced when using. Then, the blurred image synthesis unit detects, for example, the edge strength of the image obtained by the upsampling process by the upsampling unit, and the upresolution image obtained by the upsampling process by the upsampling unit and the upresolution image The up-resolution image obtained by the up-sampling process in the up-sampling unit and the out-of-resolution image equal in resolution to the up-resolution image are weighted according to the edge strength to each of the out-of-focus images having the same resolution. May be combined (added). By performing synthesis processing in this way, a base component image with a better edge preservation state can be generated.

  Further, in these image processing apparatuses described above, the base component image generation unit has a configuration in which a low-pass filter unit and a downsampling unit are connected in multiple stages, and the compression characteristic generation unit includes the number of connection stages of the low-pass filter unit and The compression characteristic is generated based on at least one of thinning rates of the downsampling unit. Here, the thinning rate (downsampling rate) is the ratio of the number of pixels reduced when the resolution is reduced by the downsampling unit. That is, the thinning rate increases as the resolution is reduced by the downsampling process.

  According to this configuration, the compression characteristic generation unit has a configuration in which the low-pass filter unit and the downsampling unit are connected in multiple stages, and the compression characteristic generation unit includes at least the number of connection stages of the low-pass filter unit and the thinning rate of the downsampling unit. Since the compression characteristic is generated based on one, a more appropriate compression characteristic can be easily generated.

  In the above-described image processing apparatuses, the base component image generation unit has a configuration in which a nonlinear filter and a downsampling unit are connected in multiple stages, and the compression characteristic generation unit includes the number of connection stages of the nonlinear filter and the downsampling unit. The compression characteristic is generated based on at least one thinning rate of the sampling unit.

  According to this configuration, the compression characteristic generation unit has a configuration in which the nonlinear filter and the downsampling unit are connected in multiple stages, and the compression characteristic generation unit includes at least one of the number of connection stages of the nonlinear filter and the thinning rate of the downsampling unit. Since the compression characteristic is generated based on the above, a more appropriate compression characteristic can be easily generated.

  In the above-described image processing apparatuses, the compression characteristic generation unit increases the compression rate as the number of connection stages increases, and increases the compression rate as the thinning rate increases.

  According to this configuration, the compression characteristic generation unit can generate a more appropriate compression characteristic.

  In another aspect of the present invention, an imaging unit that photoelectrically converts a subject light image to generate an image signal, and an image that generates an image by performing predetermined image processing on the image signal generated by the imaging unit In the imaging apparatus including the processing unit, the image processing unit includes any one of the above-described image processing devices.

  According to this configuration, it is possible to provide an imaging apparatus capable of appropriately compressing the dynamic range of a captured image with a simpler configuration and higher image quality than the background art.

  The image processing apparatus and the image processing method according to the present invention can set a more appropriate compression characteristic. Therefore, the dynamic range of an input image can be compressed with a simpler configuration and higher image quality than the background art. it can. The imaging apparatus according to the present invention can compress the dynamic range of a captured image with a simpler configuration and higher image quality than the background art.

DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, an embodiment of the invention will be described with reference to the drawings. In addition, the structure which attached | subjected the same code | symbol in each figure shows that it is the same structure, The description is abbreviate | omitted.
(Configuration of the embodiment)
First, the configuration of an imaging apparatus including the image processing apparatus according to the embodiment will be described. FIG. 1 is a diagram illustrating a configuration of an imaging apparatus including an image processing apparatus according to an embodiment. In FIG. 1, an imaging apparatus CA applied to a digital camera or the like includes a lens unit 1, an imaging sensor 2, an amplifier 3, an AD conversion unit 4, an image processing unit 5, a control unit 6, and an image memory 7. A monitor unit 8 and an operation unit 9.

  The lens unit 1 functions as a lens window for capturing subject light (light image), and the subject light is formed so as to be formed on the light receiving surface of the imaging sensor 2 disposed inside the imaging apparatus main body. An optical system for leading to the image sensor 2 is configured. The lens unit 1 includes, for example, a zoom lens, a focus lens, and other fixed lenses that are arranged in series along the optical axis AX of the subject light, and a diaphragm for adjusting the amount of light transmitted through the lens unit 1. (Not shown) and a shutter (not shown) are further provided, and the zoom unit, the focus lens, the diaphragm, and the shutter are driven and controlled by the control unit 6.

  The imaging sensor 2 includes a plurality of photoelectric conversion elements arranged so as to form a two-dimensional planar light receiving surface, and R (red) according to the amount of light of the subject light image formed on the light receiving surface by the lens unit 1. ), G (green), and B (blue) component signals that are photoelectrically converted and output to the subsequent amplifier 3. The imaging sensor 2 is, for example, a CCD (charge coupled device) type image sensor, a MOS (metal oxide semiconductor) type image sensor, a VMIS (threshold modulation type imaging device) type image sensor, or the like.

  The amplifier 3 is a circuit that amplifies the image signal output from the imaging sensor 2. The amplifier 3 includes, for example, an AGC (auto gain control) circuit, and adjusts the gain (amplification factor) of the output signal. The amplifier 3 may include a CDS (correlated double sampling) circuit that reduces sampling noise of the image signal as an analog value in addition to the AGC circuit. The AGC circuit also has a function of compensating for a lack of level in a captured image when a proper exposure is not obtained, for example, when a very low-luminance subject is captured. The gain value for the AGC circuit is set by the control unit 6.

  The AD conversion unit 4 is a circuit that performs AD conversion processing for converting an analog image signal (analog signal) amplified by the amplifier 3 into a digital image signal (digital signal). The AD conversion unit 4 converts each pixel signal obtained by receiving light at each pixel of the image sensor 2 into, for example, 12-bit pixel data.

  The image processing unit 5 is a circuit that performs various types of image processing on the image signal obtained by the AD conversion processing of the AD conversion unit 4. For example, the image processing unit 5 performs preprocessing such as black level correction and fixed pattern noise correction in the preprocessing unit 11, performs white balance correction processing in the white balance correction processing unit (WB processing unit) 12, and performs a color processing unit. 13, color processing such as color interpolation processing, color correction processing, and color space conversion is performed, dynamic range compression processing (DR compression unit) 15 performs dynamic range compression processing, and gamma correction unit (γ correction unit) 16 performs. Perform gamma correction. As described above, the dynamic range compression unit 15 is preferably arranged in the image processing unit 5 so as to perform dynamic range compression on an image after various adjustments of luminance and color are performed on the captured image. . These preprocessing, white balance correction processing, color processing, and gamma correction are processed by a known processing method.

The image memory 7 is a circuit that saves (stores) data such as RAW data before image processing by the image processing unit 5 and image data during or after various processes in the image processing unit 5 and the control unit 6. . The image memory 7 includes, for example, a memory such as an EEPROM (Electrically Erasable Programmable Read Only Memory) that is a rewritable nonvolatile storage element or a RAM (Random Access Memory) that is a volatile storage element. .

  The monitor unit 8 is a device that displays an image captured by the imaging sensor 2, that is, an image processed by the image processing unit 5, an image stored in the image memory 7, and the like. The monitor unit 8 includes, for example, a color liquid crystal display (LCD) disposed in the imaging apparatus main body.

  The operation unit 9 is an apparatus for performing an operation instruction (instruction input) by the user with respect to the imaging apparatus CA. The operation unit 9 includes, for example, various operation switch groups (operation button groups) such as a power switch, a release switch, a mode setting switch for setting various shooting modes, and a menu selection switch. For example, when the release switch is pressed (turned on), the subject light is imaged through the lens unit 1 by the imaging operation, that is, the imaging sensor 2, and necessary image processing is performed on the captured image obtained thereby. After that, a series of still image and moving image shooting operations such as recording in the image memory 7 or the like are executed.

  The control unit 6 is a circuit that controls operation of the entire imaging apparatus CA by controlling each unit according to the function. The control unit 6 calculates control parameters required by each unit, for example, an exposure amount control parameter for setting an optimal exposure amount at the time of shooting, based on various signals from the respective units such as the imaging sensor 2 and the operation unit 9. Then, the operation of each part is controlled by outputting the control parameter to each part. For example, the control unit 6 controls the imaging operation for the lens unit 1 and the imaging sensor 2 based on the control parameters, controls the image processing in the image processing unit 5, and stores image data stored in the image memory 7. The display on the monitor unit 8 is controlled. The control unit 6 is, for example, a ROM (Read Only Memory) which is a non-volatile storage element for storing each control program, a RAM for temporarily storing various data, a central operation for reading and executing the control program from the ROM A processing device (CPU) is provided.

  Next, the configuration of the dynamic range compression unit 15 will be described. FIG. 2 is a diagram illustrating a configuration of a dynamic range compression unit in the imaging apparatus according to the embodiment. FIG. 3 is a flowchart illustrating the operation of the dynamic range compression unit in the imaging apparatus of the embodiment.

  In FIG. 2, the dynamic range compression unit 15 compresses the dynamic range of the input image I based on the above Retinex theory, and includes a base component image generation unit 21, a compression characteristic generation unit 22, and a base component image compression. A unit 23, a division unit 24, and a synthesis unit 25 are provided.

  The base component image generation unit 21 generates, from the input image I, a plurality of smooth images having different spatial frequency components due to different cutoff frequencies, and having different resolutions (size, number of pixels). A base component image L equal to the resolution of the input image I is generated based on the image. The base component image L is a blurred image that is smoothed and gently changes, and corresponds to an image of the illumination light component in the Retinex theory. The base component image generation unit 21 may be configured to generate a base component image from the input image I based on a difference in spatial frequency components. That is, the base component image L may be generated by extracting a component having a predetermined spatial frequency or less from the input image I.

  The compression characteristic generation unit 22 generates the compression characteristic X (ib) based on the spatial frequency of the base component image L. More specifically, it relates to the spatial frequency of the base component image L, and a compression characteristic for compressing the base component image L based on a compression characteristic generation parameter S that is a parameter based on the configuration of the base component image generation unit 21. X (ib) is generated. The compression characteristic X (ib) represents the correspondence between the pixel value ib before compression and the pixel value ia after compression when the dynamic range of the input image I is compressed. The compression characteristic X (ib) is expressed by a functional expression as will be described later. Note that the compression characteristic X (ib) may be represented by a lookup table or the like. The spatial frequency of the base component image L is determined by the configuration of the base component image generation unit 21. That is, it is determined by the design items of the base component image generation unit 21. The compression characteristic generation parameter S corresponds to the design items of the base component image generation unit 21.

  The base component image compression unit 23 uses the compression characteristic X (ib) generated by the compression characteristic generation unit 22 for the base component image L generated by the base component image generation unit 21, so that the dynamic range is larger than that of the base component image L. The compression base component image Ls having a small size is generated.

  The division unit 24 generates the reflectance component image R by dividing the input image I by the base component image L generated by the base component image generation unit 21. The reflectance component image R is a high-frequency component image in the input image I obtained by removing the base component image from the input image I, and corresponds to an image of the reflectance component of the object in the Retinex theory.

  The combining unit 25 combines the compressed base component image Ls generated by the base component image compressing unit 23 and the reflectance component image R generated by the dividing unit 24.

  In the dynamic range compression unit 15 having such a configuration, the input image I is input to the base component image generation unit 21 and the division unit 24. The base component image generation unit 21 generates a plurality of smooth images from the input image I, generates a base component image L of the input image I using the plurality of smooth images, and the base component image L is compressed as a base component image. Input to the unit 23 and the division unit 24. Also, the compression characteristic generation parameter S extracted from the base component image generation unit 21 is input to the compression characteristic generation unit 22. The compression characteristic generation unit 22 generates a compression characteristic X (ib) based on the input compression characteristic generation parameter S, and the compression characteristic X (ib) is input to the base component image compression unit 23. In the base component image compression unit 23, the base component image L input from the base component image generation unit 21 is compressed by the compression characteristic X (ib) input from the compression characteristic generation unit 22, and the base component image L of the input image I is compressed. A compressed base component image Ls is generated. That is, in the base component image compression unit 23, the pixel value ib of the pixel of the base component image L is converted to X (ib) (= ia), and becomes the pixel value ia of the pixel of the compressed base component image Ls. The compressed base component image Ls is output from the base component image compression unit 23 to the synthesis unit 25. On the other hand, the division unit 24 divides the input image I by the base component image L input from the base component image generation unit 21 to generate a reflectance component image R. This reflectance component image R is output from the dividing unit 24 to the combining unit 25. The synthesizer 25 synthesizes (multiplies) the compressed base component image Ls input from the base component image compressor 23 and the reflectance component image R input from the divider 24 to compress the dynamic range of the input image I. The compressed image Is is generated.

  In this way, the dynamic range compression unit 15 first generates the base component image L from the input image I and the compression characteristic generation parameter S from the base component image generation unit 21 in the first step, as shown in FIG. Are extracted (S1). In the second step, the compression characteristic X (ib) is generated based on the compression characteristic generation parameter S (S2). In the third step, the compressed base component image Ls is generated by using the compression characteristic X (ib) generated in the second step as the base component image L (S3). On the other hand, in the fourth step, the reflectance component image R is generated by dividing the input image I by the base component image L (S4). In the fifth step, a compressed image Is of the input image I is generated by multiplying the compressed base component image Ls and the reflectance component image R (S5). Here, the first to third steps (S1 to S3) and the fourth step (S4) may be performed before the fifth step (S5) is performed. May be executed.

  Next, the configuration of the base component image generation unit 21 will be described. FIG. 4 is a diagram illustrating a configuration of a base component image generation unit having a three-stage configuration in the imaging apparatus according to the embodiment. FIG. 5 is a diagram illustrating a configuration of a blurred image synthesis unit in the base component image generation unit of the embodiment.

  The base component image generation unit 21A illustrated in FIG. 4 is an example of the base component image generation unit 21 in the dynamic range compression unit 15. The base component image generation unit 21A first performs a low-pass filter process with a preset cutoff frequency, thereby generating a low-spatial filter unit (LPF unit) 31 that generates a low spatial frequency image composed of a spatial frequency component lower than the cutoff frequency. A down-sampling unit (DS unit) 32 that generates a low-resolution image having a resolution smaller than that of the input image I by down-sampling processing (thinning-out processing) of the low spatial frequency image at a predetermined first rate set in advance; By repeating the low-pass filter processing by the LPF unit 31 and the downsampling processing by the DS unit 32 for the input image I a predetermined number of times n (three times in the example shown in FIG. 4), Generate multiple blurred images with different spatial frequency components and different resolutions Is constructed sea urchin. Then, the base component image generation unit 21A performs upsampling processing on one of the plurality of blurred images at a second rate corresponding to the first rate described above to generate an upresolution image ( (US unit) 33 and a blurred image composition unit that generates a synthesized blurred image by synthesizing an up-resolution image obtained by up-sampling processing in the US unit 33 and a blurred image having a resolution equal to the resolution of the up-resolution image. (Mix unit) 34, and the upsampling process by the US unit 33 and the synthesis process by the Mix unit 34 are repeated for the plurality of blurred images by the predetermined number of times n (three times in the example shown in FIG. 4). Thus, the base component image L is generated. The base component image L is a synthesized blurred image obtained by the final-stage synthesis process.

  Since the input image I is subjected to low-pass filter processing in multiple stages in this way, a low-pass filter having a smaller size than the case where the input image I is subjected to low-pass filter processing in one stage can be used. In addition, when a digital filter is adopted as the low-pass filter, the amount of calculation is reduced.

  For example, as shown in FIG. 4, the base component image generation unit 21A having a three-stage configuration includes three LPF units 31-1 to 31-3 and three DS units 32-1 to 32 according to the number of stages. -3, three US units 33-1 to 33-3, and three MiX units 34-1 to 34-3. The base component image generation unit 21A has a single-stage configuration including the LPF unit 31, the DS unit 32, the US unit 33, and the Mix unit 34, and outputs the output of the upper-stage DS unit 32 to the lower stage. It is assumed that the input image I of the configuration and the output of the Mix unit 34 of the lower configuration are regarded as the input of the US unit 33 of the upper configuration, so that the lower configuration is interposed between the DS unit 32 and the US unit 33 of the upper configuration. It is configured.

  In the present specification, when referring generically, it is indicated by a reference symbol without a suffix, and when referring to an individual configuration, it is indicated by a reference symbol with a suffix.

  The input image I (= I1) is input to the LPF unit 31-1 and the Mix unit 34-1. In the LPF unit 31-1, for example, a two-dimensional digital filter having a filter size of 7 taps is used, and a first low spatial frequency image is generated by low-pass filtering the input image I1. A spatial frequency image is input to the DS unit 32-1. In the DS unit 32-1, the first low spatial frequency image is down-sampled by, for example, pixel thinning to reduce the number of pixels of the first low spatial frequency image both vertically and horizontally by, for example, 1/2. That is, a first blurred image (first smoothed image) I2 is generated, and this first blurred image I2 is input to the LPF unit 31-2 and the Mix unit 34-2 as the lower input image I. In this example, the first rate is ½.

  In the LPF unit 31-2, the low-pass filter process is performed on the first blurred image I2 in the same manner as the LPF unit 31-1, and the LPF unit 31-2 includes lower spatial frequency components than the first low spatial frequency image (more smoothed, A second low spatial frequency image is generated, and the second low spatial frequency image is input to the DS unit 32-2. In the DS unit 32-2, the second low spatial frequency image is down-sampled in the same manner as the DS unit 32-1, so that the second resolution is smaller than that of the first low-resolution image (small size, small number of pixels). A low-resolution image, that is, a second blurred image (second smoothed image) I3 is generated, and this second blurred image I3 is input to the LPF unit 31-3 and the Mix unit 34-3 as the lower input image I.

  In the LPF unit 31-3, the third blurred image I3 is subjected to low-pass filter processing in the same manner as the LPF unit 31-1, and is composed of lower spatial frequency components than the second low spatial frequency image (more smoothed, more A third low spatial frequency image is generated, and the third low spatial frequency image is input to the DS unit 32-3. In the DS unit 32-3, the third low spatial frequency image is down-sampled in the same manner as the DS unit 32-1, so that the third resolution is smaller than the second low-resolution image (small in size and small in number of pixels). A low-resolution image, that is, a third blurred image (third smoothed image) I4 is generated. In the lowermost stage, since there is no lower-stage Mix section 34, the third blurred image I4 is input to the UP section 33-3 in the lowermost stage.

  In the US unit 33-2, as in the US unit 33-3, the third synthesized blurred image is up-sampled to generate a second up-resolution image J2, and the second up-resolution image J2 is sent to the Mix unit 34-2. Entered. In the Mix unit 34-2, similarly to the Mix unit 34-3, a second combined blurred image is generated by combining (adding) the second up-resolution image J2 and the first blurred image I2, and this second combined image is generated. The blurred image is input to the upper US section 33-1.

  In the US unit 33-1, similarly to the US unit 33-3, the second synthesized blurred image is up-sampled to generate a first up-resolution image J1, and the first up-resolution image J1 is sent to the Mix unit 34-1. Entered. In the Mix unit 34-1, similarly to the Mix unit 34-3, a first combined blurred image is generated by combining (adding) the first up-resolution image J <b> 1 and the input image I (= I1). This first synthesized blurred image is the base component image L.

  In this way, the base component image generation unit 21A is configured and operates to generate a base component image L after generating a plurality of blurred images (smooth images) from the input image I.

  Then, the blurred image synthesis unit (Mix unit) 34 in the base component image generation unit 21 of the present embodiment is configured and operates as follows.

  FIG. 5 is a diagram illustrating a configuration of a blurred image synthesis unit in the base component image generation unit of the embodiment. FIG. 5A shows the configuration of the blurred image composition unit, and FIG. 5B shows a weighting function representing the correspondence between the edge strength e and the weight w. The horizontal axis of FIG. 5B is the edge strength e, and the vertical axis is the weight w.

  5A, the Mix unit 34 includes a first multiplier 41, an adder 42, a second multiplier 43, an edge unit 44, and a lookup table unit (LUT unit) 45.

  The edge unit 44 receives the up-resolution image Jn from the US unit 33 in the stage configuration, obtains the edge strength e of the up-resolution image Jn, and outputs the obtained edge strength e to the LUT unit 45. The edge strength e is obtained by an edge extraction filter such as a Sobel filter (Sobel filter) or a pre-witt filter (Prewitt filter), for example.

  The LUT unit 45, based on the edge strength e input from the edge unit 44, the blurred image In-1 input from the DS unit 32 of the previous stage composition combined by the Mix unit 34 and the US unit 33 of the stage configuration. The weights w and 1-w weighted to each of the up-resolution images Jn input from are obtained, and the obtained weights w and 1-w are output to the first and second multipliers 41 and 43, respectively.

  This weight w is determined as follows. As shown in FIG. 5B, when the edge strength e is not less than 0 and not more than a predetermined first edge storage threshold th1, the weight w is set to 0 and the edge strength e is set in advance. The weight w is set to 1.0 when the predetermined second edge storage threshold th2 is equal to or higher than the predetermined second edge storage threshold th2, and the edge strength e is equal to or higher than the first edge storage threshold th1 and lower than the second edge storage threshold th2. In this case, the weight w is (e−th1) / (th2−th1).

  Returning to FIG. 5A, the first multiplier 41 multiplies the blurred image In-1 input from the DS unit 32 of the previous stage by the weight w input from the LUT unit 45, and weights with this weight w. The blurred image In-1 is output to the adder 42.

  The second multiplier 43 multiplies the up-resolution image Jn input from the US unit 33 of the stage configuration by the weight 1-w input from the LUT unit 45, and the up-resolution image weighted with the weight 1-w. In is output to the adder 42.

  The adder 42 adds the blurred image In-1 weighted with the weight w input from the first multiplier 41 and the up-resolution image Jn weighted with the weight 1-w input from the second multiplier 43. As a result, a synthesized blurred image is generated, and the generated synthesized blurred image is output to the US unit 33 having the previous configuration.

  In the Mix unit 34 having such a configuration, the edge strength e of the up-resolution image Jn generated by the US unit 33 having the stage configuration is obtained by the edge portion 44, and each weight is calculated by the LUT unit 45 based on the edge strength e. w and 1-w are required. The blurred image In-1 input from the DS unit 32 of the previous stage configuration is weighted with the weight w by the first multiplier 41, and the up-resolution image Jn input from the US unit 33 of the previous stage configuration is input by the second multiplier 43. Weighted with weight 1-w. Then, the blur image In-1 and the up-resolution image Jn weighted by the respective weights w and 1-w are added by the adder 42 to generate a synthesized blur image.

  The first and second edge storage thresholds th1 and th2 control the mixing ratio between the blurred image In-1 input from the DS unit 32 having the previous stage configuration and the up-resolution image Jn input from the US unit 33 having the previous stage configuration. The value to be The first edge storage threshold th1 may be the same value or a different value in each stage configuration, and the first edge storage threshold th2 may be the same value or a different value in each stage configuration.

  By configuring and operating the Mix unit 34 as described above, the edge portion of the base component image L includes more information on the edge portion of the input image I, and the edge is stored.

  4 shows the base component image generation unit 21A having a three-stage configuration, the base component image generation unit 21A may have a one-stage configuration, a two-stage configuration, or a four-stage configuration or more. Any number of stages may be used. In the above description, the first rate is set to 1/2, but may be an arbitrary rate such as 1/3 or 1/4. The second rate is set according to the first rate as described above. For example, the second rate is 3 when the first rate is 1/3, and is 4 when the first rate is 1/4.

  Next, the compression characteristic generation unit 22 will be described. FIG. 6 is a diagram illustrating compression characteristics in the dynamic range compression unit of the embodiment. The horizontal axis in FIG. 6 is the pixel value ib before compression (the pixel value ib of the base component image L), and the vertical axis is the pixel value ia after compression (the pixel value ia of the compressed base component image Ls). .

  The compression characteristic X (ib) of the present embodiment is represented by two continuous first and second straight lines SL1 and SL2, and is uniquely determined by the coordinate values of both end points O and Q and the coordinate value of the continuous point (connection point) P. To be determined. If the slopes and intercepts of the first and second straight lines SL1, SL2 are a1, a2, and b1, b2, respectively, the first straight line SL1 is represented by ia = a1 × ib + b1, and the second straight line SL2 is represented by ia = It is represented by a2 × ib + b2.

  Since the compression characteristic X (ib) is uniquely determined by the coordinate values of the end points O and Q and the coordinate value of the continuous point P, the compression characteristic generation unit 22 sets the coordinate values of these points O, P and Q. By doing so, the compression characteristic X (ib) is generated. In setting the coordinate points of these points O, P, and Q, the compression characteristic generation unit 22 receives the coordinate value of at least one of these points O, P, and Q from the base component image generation unit 21. It is set based on the extracted compression characteristic generation parameter S. Here, the compression characteristic generation parameter S is a parameter related to the spatial frequency of the base component image L and depends on the configuration of the base component image generation unit 21. More specifically, the compression characteristic generation parameter S is the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, the thinning rate at the time of downsampling processing in the DS unit 32, and the like. . In the case of a single-stage configuration, since there is only one LPF unit 31, the value of the LPF unit 31 may be used. However, in the case of a multi-stage configuration, the average of the LPF units 31 in all stages is used. A typical value may be used. Similarly, in the case of multiple stages, the thinning rate may be an average thinning rate for the DS units 32 in all stages. It should be noted that the thinning rate of the DS unit 32 is preferably set to the same value in all stages in consideration of ease of design.

  The coordinate value of one end point O of the compression characteristic X (ib) is set to a coordinate origin (0, 0) as shown in FIG. 6, for example, and is fixed (b1 = 0). The end point O may be set to a point (0, b1) (b1 ≠ 0) on the ia axis and may be fixed.

  In the coordinate value (qi, qo) of the other end point Q of the compression characteristic X (ib), for example, the value qi may be set to a value obtained based on the compression characteristic generation parameter S. Specifically, it may be set based on the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, the thinning rate in the DS unit 32, and the like. The value of qi is monotonously decreased with respect to the cutoff frequency of the LPF unit 31, and is monotonously increased with respect to the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, and the thinning rate in the DS unit 32. And it is sufficient. That is, if the cutoff frequency of the LPF unit 31 is increased, the value of qi may be decreased. Further, if the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, and the thinning rate in the DS unit 32 are increased, the value of qi may be increased. Note that the compression ratio increases as qi increases, and the compression ratio decreases as qi decreases. The value qo is set to the maximum pixel value in the dynamic range after compression. The value qi is the maximum output value of the predetermined image output system (may be the maximum output value or the maximum pixel value of the imaging sensor 2), and for example, a gradation value of “255” in an 8-bit image. Take.

  In the coordinate value (pi, po) of the continuous point (connection point) P of the compression characteristic X (ib), each value pi, po is set to a value such that, for example, the dark part before compression has appropriate brightness after compression. And fixed.

  The slope a1 and the intercept b1 of the first straight line SL1 are obtained by substituting the coordinate value of the point O and the coordinate value of the point P into ia = a1 × ib + b1 and solving the simultaneous equations of a1 and b1. The slope a2 and the intercept b2 of the second straight line SL2 are obtained by substituting the coordinate value of the point P and the coordinate value of the point Q into ia = a2 × ib + b2 and solving the simultaneous equations of a2 and b2.

  As described above, the compression characteristic generation unit 22 uses the compression characteristic generation parameter S, such as the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, or the thinning rate in the DS unit 32. To generate the compression characteristic X (ib).

  As described above, the compression characteristic X (ib) is compressed based on the compression characteristic generation parameter S which is a parameter related to the configuration of the base component image generation unit 21 and is related to the spatial frequency of the base component image L. Since it is generated by the characteristic generation unit 22, the compression characteristic generation unit 22 can be simply configured, and an appropriate compression characteristic X (ib) can be generated.

  Further, the compression characteristic generation unit 22 may generate the compression characteristic X (ib) as follows. FIG. 7 is a diagram illustrating another compression characteristic in the dynamic range compression unit of the embodiment. The horizontal axis in FIG. 7 is the pixel value ib before compression (the pixel value ib of the base component image L), and the vertical axis is the pixel value ia after compression (the pixel value ia of the compressed base component image Ls). .

  Another compression characteristic X (ib) of the present embodiment is represented by a curve, for example, a logarithmic curve CL, and is uniquely determined by the coordinate values of the end points O and Q and the coordinate value of the intermediate point P therebetween. The logarithmic curve CL is represented by, for example, ia = exp (ln (ib) × c) × d (c and d are constants).

  Since the compression characteristic X (ib) is uniquely determined by the coordinate values of the end points O and Q and the coordinate value of the intermediate point P, the compression characteristic generation unit 22 sets the coordinate values of these points O, P and Q. By doing so, the compression characteristic X (ib) is generated. In setting the coordinate points of these points O, P, and Q, the compression characteristic generation unit 22 uses the coordinate value of at least one of these points O, P, and Q as the compression characteristic generation parameter S. Set based on.

  The coordinate value of one end point O of the compression characteristic X (ib) is set to the coordinate origin (0, 0) as shown in FIG. 7, for example. The end point O may be set to a point on the ia axis and fixed.

  In the coordinate value (qi, qo) of the other end point Q of the compression characteristic X (ib), the value qi may be set based on the compression characteristic generation parameter S. Specifically, it may be set based on the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, the thinning rate in the DS unit 32, and the like. The value of qi is monotonously decreased with respect to the cutoff frequency of the LPF unit 31, and is monotonously increased with respect to the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, and the thinning rate in the DS unit 32. And it is sufficient. That is, if the cutoff frequency of the LPF unit 31 is increased, the value of qi may be decreased. Further, if the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, and the thinning rate in the DS unit 32 are increased, the value of qi may be increased. Note that the compression ratio increases as qi increases, and the compression ratio decreases as qi decreases. The value qo is set to the maximum pixel value in the dynamic range after compression. The value qi is the maximum output value of the predetermined image output system (may be the maximum output value or the maximum pixel value of the imaging sensor 2), and for example, a gradation value of “255” in an 8-bit image. Take.

  In the coordinate value (pi, po) of the continuous point (connection point) P of the compression characteristic X (ib), each value pi, po is set to a value such that, for example, the dark part before compression has appropriate brightness after compression. And fixed.

  The constants c and d of the logarithmic curve CL are obtained by substituting the coordinate value of the point P and the coordinate value of the point Q into ia = exp (ln (ib) × c) × d and solving the simultaneous equations of c and d. It is done.

As described above, the compression characteristic generation unit 22 uses the compression characteristic generation parameter S, such as the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, or the thinning rate in the DS unit 32. To generate the compression characteristic X (ib).
Thereby, since the compression characteristic X (ib) is generated by the compression characteristic generation unit 22 based on the spatial frequency of the base component image L, the compression characteristic generation unit 22 can be simply configured, and appropriate compression is performed. It is possible to generate the characteristic X (ib).

  Here, the dynamic range of the input image I input to the dynamic range compression unit 15 may be arbitrary, but the dynamic range compression unit 15 preferably functions when an input image I having a wide dynamic range is input.

  The wide dynamic range input image I is generated as follows, for example. First, the control unit 6 controls the exposure of the lens unit 1 and the image sensor 2 so that a plurality of images (multiple exposure images) having different exposure amounts (exposure amounts) are taken. Then, a wide dynamic range image generation unit (wide DR image generation unit) 14A adds a plurality of images to the plurality of images by applying a gain that is inversely proportional to the exposure of the image, thereby generating a wide dynamic range image. For example, three images of exposure 1, 1/2 and 1/4 are taken, gain 1 is applied to the image of exposure 1, gain 2 is applied to the image of exposure 1/2, and exposure 1/4 is applied. The image is multiplied by a gain of 4, and the images of the exposures multiplied by the gains are added to generate an image with a wide dynamic range. The wide DR image generation unit 14A is arranged downstream of the AD conversion unit 4 and upstream of the DR compression unit 15 of the image processing unit 5. FIG. 1 shows an example in which the wide DR image generation unit 14 </ b> A is arranged in front of the DR compression unit 15.

  Further, for example, the input image I having a wide dynamic range is generated as follows. FIG. 8 is a diagram illustrating photoelectric conversion characteristics and expansion of photoelectric conversion characteristics in an image sensor having knee characteristics. The horizontal axis in FIG. 8 is the subject brightness, and the vertical axis is the pixel value. The solid line in FIG. 8 indicates the photoelectric conversion characteristic of the image sensor 2 itself, and the alternate long and short dash line indicates the photoelectric conversion characteristic converted so that the photoelectric conversion characteristic in the high luminance range is on the extension line of the photoelectric conversion characteristic in the low luminance range. Indicates. First, an imaging sensor having a knee characteristic of photoelectric conversion characteristics is used for the imaging sensor 2. In the knee characteristic imaging sensor, the photoelectric conversion characteristic indicating the correspondence between the subject brightness D and the pixel value i of the imaging sensor 2 has a predetermined threshold value Dth and the first photoelectric conversion characteristic in the low brightness range and the high brightness range. The second photoelectric conversion characteristic is divided into two. When the subject brightness D is within a low brightness range from 0 to a predetermined threshold value Dth, the subject brightness D is photoelectrically converted to a pixel value i with the first inclination, and the subject brightness D is changed from the predetermined threshold value Dth to the image sensor. When the pixel value i is within the high luminance range up to the maximum value Dmax that can be photoelectrically converted, the pixel value i is such that the pixel value i is continuous at the subject luminance Dth and the second gradient is smaller than the first gradient. Photoelectrically converted to i. That is, the photoelectric conversion characteristic in the high luminance range is compressed more than the photoelectric conversion characteristic in the low luminance range. In the example shown in FIG. 8, the first inclination is 1, and the second inclination is a3 (0 <a3 <1). An image photographed by the imaging sensor 2 having such knee characteristics is on the extended line of the photoelectric conversion characteristics in the high luminance range and the photoelectric conversion characteristics in the low luminance range by the wide dynamic range image generation unit (wide DR image generation unit) 14B. The image is converted so as to be ridden and expanded to generate a wide dynamic range image. More specifically, when the photoelectric conversion characteristic in the wide luminance range is represented by i = a3 × D + b3 (b3 = ith−a3 × Dth), the pixel value corresponding to a predetermined threshold value Dth from 0. Up to ith, the pixel value i of the image sensor 2 is adopted as it is, and the pixel value i of the image sensor 2 is (i−b3) / from the pixel value ith to the pixel value imax corresponding to the maximum value Dmax. A wide dynamic range image is generated by converting to a3. The wide DR image generation unit 14B is arranged after the AD conversion unit 4 and before the DR compression unit 15 of the image processing unit 5. FIG. 1 shows an example in which the wide DR image generation unit 14 </ b> B is arranged in the preceding stage of the DR compression unit 15.

  Further, for example, the input image I having a wide dynamic range is generated as follows. FIG. 9 is a diagram illustrating photoelectric conversion characteristics and expansion of photoelectric conversion characteristics in an image sensor having linear log characteristics. The horizontal axis in FIG. 9 is the subject brightness, and the vertical axis is the pixel value. The solid line in FIG. 9 indicates the photoelectric conversion characteristic of the image sensor 2 itself, and the alternate long and short dash line indicates the photoelectric conversion characteristic converted so that the photoelectric conversion characteristic in the high luminance range is on the extended line of the photoelectric conversion characteristic in the low luminance range. Indicates. First, an imaging sensor having a linear log characteristic as a photoelectric conversion characteristic is used as the imaging sensor 2. In the linear log characteristic imaging sensor, the photoelectric conversion characteristic indicating the correspondence between the subject luminance D and the pixel value i of the imaging sensor 2 has a predetermined threshold value Dth and the first photoelectric conversion characteristic in the low luminance range and the high luminance range. The second photoelectric conversion characteristic is divided into two. When the subject brightness D is within a low brightness range from 0 to a predetermined threshold value Dth, the subject brightness D is linearly photoelectrically converted to the pixel value i, and the subject brightness D is photoelectrically converted from the predetermined threshold value Dth to the imaging sensor. When it is within the high luminance range up to the maximum possible value Dmax, the subject luminance D is photoelectrically converted logarithmically into the pixel value i so that the pixel value i is continuous with the subject luminance Dth. That is, the photoelectric conversion characteristic in the high luminance range is compressed more than the photoelectric conversion characteristic in the low luminance range. In the example shown in FIG. 9, the linear first photoelectric conversion characteristic in the low luminance range is i = D, and the logarithmic second photoelectric conversion characteristic in the high luminance range is i = α × ln (D) + β (β = ith). −α × ln (Dth)). The image captured by the imaging sensor 2 having such a linear log characteristic is on the extension line of the photoelectric conversion characteristic in the high luminance range by the wide dynamic range image generation unit (wide DR image generation unit) 14C. The image is converted so as to be ridden and expanded to generate a wide dynamic range image. More specifically, in the case where the first and second photoelectric conversion characteristics are expressed as described above, the pixel value of the image sensor 2 is from 0 to the pixel value ith corresponding to the predetermined threshold value Dth. i is employed as it is, and from pixel value ith to pixel value imax corresponding to maximum value Dmax, pixel value i of imaging sensor 2 is converted to exp ((i−β) / α), and wide dynamic A range image is generated. The wide DR image generation unit 14 </ b> C is arranged downstream of the AD conversion unit 4 and upstream of the DR compression unit 15 of the image processing unit 5. FIG. 1 shows an example in which the wide DR image generation unit 14 </ b> C is arranged in front of the DR compression unit 15.

  In addition to this, the input image I having a wide dynamic range may be obtained by a known method, for example, the method disclosed in any one of Patent Documents 1 to 4.

  As described above, in the imaging apparatus CA including the image processing apparatus according to the present embodiment, the cutoff frequency of the LPF unit 31, the window size of the LPF unit 31, the number of connection stages of the LPF unit 31, which are compression characteristic generation parameters S, The compression characteristic X (ib) is generated based on at least one of the thinning rates and the like at the time of downsampling processing in the DS unit 32. That is, since the compression characteristic X (ib) is generated based on the spatial frequency L of the base component image, a more appropriate compression characteristic X (ib) is set. As a result, for example, the dynamic range of the input image I can be compressed with higher image quality than the background technology without so-called whiteout or blackout. Since the compression characteristic X (ib) is determined by these parameters, it is easier than the case where the compression characteristic X (ib) is obtained based on the image, and the processing time can be shortened.

  In the above-described embodiment, the base component image generation unit 21 is not limited to the configuration shown in FIG. For example, the image processing device disclosed in Patent Document 6 can be used. The image processing apparatus disclosed in Patent Document 6 is described in detail in Patent Document 6, but is generally configured and operates as follows.

  FIG. 10 is a diagram illustrating another configuration of the base component image generation unit in the dynamic range compression of the embodiment. In FIG. 10, the base component image generation unit 21B includes a plurality of epsilon filters 120, 120A, 120B, a plurality of linear low-pass filters (linear LPF) 121A, 121B, a plurality of down-samplers 122A, 122B, and a plurality of up-samplers. Sampling devices 123A and 123B, a plurality of interpolators 124A and 124B, and a plurality of combiners 125A and 125B are provided.

  The input image I1 is divided and input to the first epsilon filter 120 and the first linear LPF 121A. The first epsilon filter 120 performs a predetermined nonlinear filtering process on the input image I1 to generate the first smoothed image RH0 having the lowest degree of smoothing and the highest resolution, and the first smoothed image RH0 is the first smoothed image RH0. 2 is output to the combiner 125B. The first linear LPF 121A is for avoiding the occurrence of aliasing in the downsampling process in the first downsampler 122A in the subsequent stage, and performs an appropriate filter process according to the downsampling rate. The output of the first linear LPF 121A is input to the first downsampler 122A and subjected to downsampling processing at a predetermined rate preset by the first downsampler 122A. The image I2 reduced by the first linear LPF 121A and the first downsampler 122A is input to the second epsilon filter 120A and the second linear LPF 121B.

  The second epsilon filter 120A and the second linear LPF 121B are different from each other in that the size of the input image I is different, that is, the image I2 from the first downsampler 122A is input as the input image I. 120 and substantially the same function as the first linear LPF 121A. That is, the second epsilon filter 120A performs a predetermined nonlinear filtering process on the input image I2 from the first downsampler 122A to generate a second smoothed image RH1, and this second smoothed image RH1 is first synthesized. Output to the device 125A. The second linear LPF 121B performs an appropriate filter process according to the downsampling rate of the second downsampler 122B at the subsequent stage. The output of the second linear LPF 121B is input to the second downsampler 122B, and downsampling is performed at a rate set in advance by the second downsampler 122B. The reduced image I3 is input to the third epsilon filter 120B.

  The third epsilon filter 120B performs substantially the same function as the first epsilon filter 120. That is, in the third epsilon filter 120B, a predetermined nonlinear filtering process is performed on the input image I3 from the second downsampler 122B to generate a third smoothed image RH2, and this third smoothed image RH2 is first increased. Output to the sampler 123A.

  The first upsampler 123A performs the upsampling process on the third smoothed image RH2 at the same rate as the second downsampler 122B. That is, when the downsampling rate in the second downsampler 122B is 1 / N, the first upsampler 123A performs a process of inserting N−1 pixels having a value of 0 between adjacent pixels. Do. The first interpolator 124A generates an intermediate composite image by performing an appropriate interpolation process on the image subjected to the upsampling process, and outputs the intermediate composite image to the first combiner 125A. In the first synthesizer 125A, the edge strength G is calculated, and based on this, the second smoothed image RH1 from the second epsilon filter 120A and the intermediate synthesized image from the first interpolator 124A are synthesized to obtain the first synthesis. A smooth image is generated, and the first combined smooth image is output to the second upsampler 123B.

  The second upsampler 123B and the second interpolator 124B function substantially the same as the first upsampler 123A and the first interpolator 124A, respectively. The intermediate composite image generated by these is output to the second synthesizer 125B. The second synthesizer 125B performs substantially the same function as the first synthesizer 125A. In other words, the second synthesizer 125A synthesizes the first smoothed image RH0 output from the first epsilon filter 120 and the intermediate synthesized image output from the second interpolator 124B to obtain the final base representing the base component. A second synthesized smooth image is generated as the component image L and output.

  In this way, the base component image generation unit 21B divides and inputs the input image I1, performs smoothing on each of the divided input images I, and outputs a plurality of smoothed images RH0 having different degrees of smoothing. RH1 and RH2 are generated. Then, the base component image generation unit 21B calculates the edge strength G based on the plurality of smoothed images RH0, RH1, and RH2, and further, based on the calculated edge strength G, the plurality of smoothed images RH0, RH1, By combining RH2, a final base component image L representing the base component is generated.

  Then, a parameter relating to the spatial frequency of the base component image L, which is based on the configuration of the base component image generation unit 21B, may be used as the compression characteristic generation parameter S. For example, the cutoff frequency of the epsilon filters 120, 120A, 120B, the window size of the epsilon filters 120, 120A, 120B, the number of connected stages of the epsilon filters 120, 120A, 120B, and the thinning rate during the downsampling process in the downsamplers 122A, 122B Etc. Note that the base component image generation unit 21B may have a single-stage or multi-stage configuration. When the base component image generation unit 21B has a single-stage configuration, since there is one epsilon filter, the value may be used. However, in the case of a multi-stage configuration, the epsilon filters 120, 120A, and 120B in all stages are used. An average value may be used. Similarly, in the case of multiple stages, the thinning rate may be an average thinning rate with respect to the thinning rates in all stages.

  Similarly to the compression characteristic generation parameter S of the base component image generation unit 21A described above, the compression characteristic generation unit 22 may set the compression characteristic based on this. This makes it possible to easily generate more appropriate compression characteristics.

  In the above-described embodiment, as an example of a configuration in which the compressed image Is is generated based on the input image I, the base component image L, and the compressed base component image Ls, the DR compression unit 15 uses the input image I and the base component image. Although the reflectance component image R is generated based on L and the compressed image Is is generated based on the compressed base component image Ls and the reflectance component image R, the configuration is not limited to this. Other configurations may be used. For example, since the input image I is I = L × R and the compressed image Is is Is = Ls × R, Is = (Ls / L) × I. The component image Ls may be divided by the base component image L, and the division result may be multiplied by the input image I. In this case, the DR compression unit 15 receives the base component image generation unit 21, the compression characteristic generation unit 22, the base component image compression unit 23, the input image I, the base component image L, and the compression base component image Ls. A division unit that divides the compressed base component image Ls by the base component image L and then multiplies the division result by the input image I.

  Moreover, although the said description was about a still image, the same effect can be acquired also in a moving image by performing the process similar to the above for every frame of a moving image.

  In order to express the present invention, the present invention has been properly and fully described through the embodiments with reference to the drawings. However, those skilled in the art can easily change and / or improve the above-described embodiments. It should be recognized that this is possible. Accordingly, unless the modifications or improvements implemented by those skilled in the art are at a level that departs from the scope of the claims recited in the claims, the modifications or improvements are not limited to the scope of the claims. To be construed as inclusive.

It is a figure which shows the structure of the imaging device provided with the image processing apparatus in embodiment. It is a figure which shows the structure of the dynamic range compression part in the imaging device of embodiment. It is a flowchart which shows operation | movement of the dynamic range compression part in the imaging device of embodiment. It is a figure which shows the structure of the base component image generation part of 3 steps | paragraphs structure in the imaging device of embodiment. It is a figure which shows the structure of the blur image synthetic | combination part in the base component image generation part of embodiment. It is a figure which shows the compression characteristic in the dynamic range compression part of embodiment. It is a figure which shows the other compression characteristic in the dynamic range compression part of embodiment. It is a figure which shows the expansion | extension of the photoelectric conversion characteristic and photoelectric conversion characteristic in an imaging sensor of knee characteristics. It is a figure which shows the expansion | extension of the photoelectric conversion characteristic in the imaging sensor of a linear log characteristic, and a photoelectric conversion characteristic. It is a figure which shows the other structure of the base component image generation part in the dynamic range compression of embodiment. 10 is a diagram for explaining a compression ratio used for dynamic range compression processing in the image processing method disclosed in Patent Document 5. FIG. 10 is a diagram for explaining coefficients used for dynamic range compression processing in the image processing method disclosed in Patent Document 6. FIG.

Explanation of symbols

CA imaging device L base component image Ls compressed base component image
R Reflectance component image S Compression characteristic generation image CL Logarithmic curve SL1, SL2 Straight line 1 Lens unit 2 Imaging sensor 5 Image processing unit 6 Control unit 7 Image memories 14, 14A, 14B, 14C Wide dynamic range image generation unit (wide DR) Image generation unit) 15 Dynamic range compression unit (DR compression unit)
21, 21A, 21B Base component image generation unit 22 Compression characteristic generation unit 23 Base component image compression unit 24 Division unit 25 Composition unit 31 (31-1 to 31-3) Low-pass filter unit 32 (32-1 to 32-3) Downsampling unit 33 (33-1 to 33-3) Upsampling unit 34 (34-1 to 34-3) Blurred image composition unit (Mix unit)
41 First multiplier 42 Adder 43 Second multiplier 44 Edge section 45 Look-up table section (LUT section)
120, 120A, 120B Epsilon filters 122A, 122B Downsamplers 123A, 123B Upsamplers 124A, 124B Interpolators
125A, 125B combiner

Claims (11)

A base component image generation unit that generates a base component image based on a difference in spatial frequency components from an input image;
A compression characteristic generation unit that generates a compression characteristic for compressing the base component image based on a spatial frequency of the base component image;
A base component image compression unit that generates a compressed base component image having a dynamic range smaller than that of the base component image by using the compression characteristic generated by the compression characteristic generation unit for the base component image. Image processing device. The compressed image generation unit that generates a compressed image obtained by compressing a dynamic range of the input image based on the input image, the base component image, and the compressed base component image. Image processing device. The base component image generation unit includes a low pass filter unit,
The image processing apparatus according to claim 1, wherein the compression characteristic generation unit generates the compression characteristic based on at least one of a cutoff frequency and a window size of the low-pass filter unit. The base component image generation unit includes a nonlinear filter unit,
The image processing apparatus according to claim 1, wherein the compression characteristic generation unit generates the compression characteristic based on at least one of a cutoff frequency and a window size of the nonlinear filter unit. The image processing apparatus according to claim 3, wherein the compression characteristic generation unit decreases the compression rate as the cutoff frequency increases, and increases the compression rate as the window size increases. The base component image generation unit generates, from the input image, a plurality of smooth images having different spatial frequency components with different cutoff frequencies and having different resolutions, and the input image is based on the plurality of smooth images. The image processing apparatus according to claim 1, wherein the base component image having a resolution equal to that of the base image is generated. The base component image generation unit has a configuration in which a low-pass filter unit and a downsampling unit are connected in multiple stages,
The image processing apparatus according to claim 6, wherein the compression characteristic generation unit generates the compression characteristic based on at least one of a connection stage number of the low-pass filter unit and a thinning rate of the downsampling unit. The base component image generation unit has a configuration in which a nonlinear filter unit and a downsampling unit are connected in multiple stages,
The image processing apparatus according to claim 6, wherein the compression characteristic generation unit generates the compression characteristic based on at least one of a connection stage number of the nonlinear filter and a thinning rate of the downsampling unit. The image processing apparatus according to claim 7, wherein the compression characteristic generation unit increases the compression rate as the number of connection stages increases, and increases the compression rate as the thinning rate increases. . A base component image generating step for generating a base component image from the input image due to a difference in spatial frequency components;
A compression characteristic generation step for generating a compression characteristic for compressing the base component image based on a spatial frequency of the base component image generated in the base component image generation step;
A base component image compression step for generating a compressed base component image having a dynamic range smaller than that of the base component image by using the compression characteristic generated in the compression property generation step for the base component image. Image processing method. An imaging apparatus comprising: an imaging unit that photoelectrically converts a subject light image to generate an image signal; and an image processing unit that performs predetermined image processing on the image signal generated by the imaging unit to generate an image.
The image processing unit includes the image processing apparatus according to any one of claims 1 to 9.
An imaging apparatus characterized by the above.

Images