JP7618989B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method , and a program.

For example, Patent Document 1 describes an image processing device that performs tone correction and noise reduction processing on an image.

JP 2016-127388 A

However, conventional image processing devices, including those described in Patent Document 1, do not sufficiently reduce the circuit size, and image quality deteriorates when the desired noise reduction processing cannot be performed.

In view of the above-described circumstances, an object of one aspect of the present invention is to provide an image processing device, an image processing method , and a program that can realize good noise reduction processing while suppressing the circuit scale.

An image processing device of a first aspect according to the present invention has a noise reduction intensity data generation unit that outputs first noise reduction intensity data generated based on specified image data to a filter processing unit, and generates second noise reduction intensity data that takes into account processed data generated by the filter processing unit performing a specified filter processing on the first noise reduction intensity data, and a noise reduction processing unit that performs noise reduction processing on the image data by applying the second noise reduction intensity data , and is characterized in that the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter or a NonLocalMeans filter .
Also, an image processing device according to a second aspect of the present invention includes a gradation correction unit that generates gradation corrected image data by performing gradation correction on predetermined RGB image data, a YUV conversion unit that generates YUV image data by YUV converting the gradation corrected image data, and outputs first noise reduction intensity data to a filter processing unit, the first noise reduction intensity data being generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, The noise reduction intensity data generating unit receives processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, and generates second noise reduction intensity data taking into account the processed data, and a noise reduction processing unit performs noise reduction processing on the YUV image data by applying the second noise reduction intensity data, wherein the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter .
Furthermore, a first aspect of the image processing method according to the present invention is an image processing method executed by an image processing device, and includes a noise reduction intensity data generation step of outputting first noise reduction intensity data generated based on specified image data to a filter processing unit, and receiving processed data generated by the filter processing unit performing a specified filter processing on the first noise reduction intensity data, thereby generating second noise reduction intensity data that takes into account the processed data, and a noise reduction processing step of performing noise reduction processing on the image data by applying the second noise reduction intensity data, wherein the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter .
Further, an image processing method according to a second aspect of the present invention is an image processing method executed by an image processing device, comprising: a gradation correction step of generating gradation corrected image data by performing gradation correction on predetermined RGB image data; a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data; and a filter processing unit that outputs first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when generating the gradation corrected image data in the gradation correction step, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color. the noise reduction intensity data generating step of outputting the first noise reduction intensity data to the filter processing unit and generating second noise reduction intensity data taking into account the processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, and a noise reduction processing step of performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data, wherein the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter or a NonLocalMeans filter .
Furthermore, a first aspect of the program according to the present invention causes a computer to function as: noise reduction intensity data generating means that outputs first noise reduction intensity data generated based on specified image data to a filter processing unit, and that generates second noise reduction intensity data taking into account processed data generated by the filter processing unit performing a specified filter processing on the first noise reduction intensity data, and noise reduction processing means that performs noise reduction processing on the image data by applying the second noise reduction intensity data, and the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter .
A program according to a second aspect of the present invention includes a computer, a gradation correction means for generating gradation corrected image data by performing gradation correction on predetermined RGB image data, a YUV conversion means for generating YUV image data by YUV converting the gradation corrected image data, gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and outputs first noise reduction intensity data to a filter processing unit, and a noise reduction intensity data generating means for generating second noise reduction intensity data taking into account processed data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data, and a noise reduction processing means for performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data, wherein the filter processing unit performs the filter processing using a low-pass filter, an epsilon filter, a Bilateral filter or a NonLocalMeans filter .

According to the present invention , it is possible to provide an image processing device, an image processing method , and a program that can realize good noise reduction processing while suppressing the circuit scale.

FIG. 1 is an external view showing an example of a digital camera 1X. 1 is a first block diagram showing a configuration of an image processing device 1. FIG. FIG. 2 is a second block diagram showing the configuration of the image processing device 1. FIG. 11 is a conceptual diagram showing an example of noise reduction intensity data D9. FIG. 2 is a diagram showing a detailed configuration of a noise reduction intensity data generating unit 50. 10 is a conceptual diagram showing the flow of noise reduction processing using first and second noise reduction intensity data. FIG. 11A and 11B are diagrams showing an example of noise reduction intensity data before and after map processing (filter processing) is performed by a filter processing unit 80. FIG. 4 is a flowchart showing an example of an image processing method by the image processing device 1.

The image processing device 1 of this embodiment will be described in detail with reference to Figures 1 to 8. Note that the image processing method and image processing program of this embodiment are realized by causing the computer (circuit components) of the image processing device 1 to execute various processing steps. The image processing device 1 may also be called, for example, a Color Process Pipeline. The image processing device 1 may also be implemented in a large scale integrated circuit (LSI: Large Scale Integration) or as a part thereof.

The image processing device 1 may be mounted on a fixed imaging device such as a security camera or surveillance camera, or on a portable imaging device such as a digital camera or various terminals. The image processing device 1 may also be specialized for performing image processing on images transmitted by wire or wirelessly from an imaging device or other device, or images stored in various storage media. In other words, there is a degree of freedom in the configuration of the image processing device 1, and various design changes are possible.

Figure 1 is an external view of an example of a digital camera (photographing device) 1X. In digital camera 1X, light from a subject is guided into camera body 11X via lens 10X, reflected by mirror 12X, and forms a real image of the subject on focusing glass 13X. This real image of the subject is guided to viewfinder 15X by pentaprism 14X and can be observed. A display device 16X is disposed between focusing glass 13X and pentaprism 14X, and displays various information superimposed on the subject image reflected on focusing glass 13X. The broken lines in Figure 1 indicate the optical path of light from the subject guided to viewfinder 15X. In digital camera 1X, mirror 12X is raised, shutter 17X is opened, and subject light is guided to image sensor 18X to perform photography, so the structure is such that the subject image does not reach display device 16X in the mirror-up state.

The digital camera 1X may incorporate an image processing device 1 according to an embodiment of the present invention that applies image processing to an image captured by the image sensor 18X, or may have a communication unit (such as a connection terminal or an antenna) that transmits, by wire or wirelessly, an image captured by the image sensor 18X to an external image processing device 1 according to an embodiment of the present invention. In other words, the present invention is not limited to a digital camera 1X that has a photographing function, but may also be applied to any device that can acquire an image captured by an external device and apply noise reduction processing to it. In other words, the present invention may also be applied to a system made up of multiple devices.

2 and 3 are first and second block diagrams showing the configuration of the image processing device 1. The image processing device 1 has a demosaic processing unit 10, a tone correction unit 20, a YUV conversion unit 30, a color analysis unit 40, a noise reduction intensity data generation unit (NR intensity data generation unit) 50, a noise reduction processing unit (NR processing unit) 60, and a post-processing unit 70. As shown in FIG. 2, the image processing device 1 is connected to a filter processing unit 80 as an external circuit and a DRAM (Dynamic Random Access Memory) 90 as an external memory.

In FIG. 2, the demosaic processing unit 10, the tone correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50 are enclosed by solid lines as the common circuit portion 100 for generating noise reduction intensity data (first and second noise reduction intensity data) described later (the demosaic processing unit 10, the tone correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50 constitute the common circuit portion 100). Furthermore, in FIG. 2, the filter processing unit 80 and the DRAM 90 are enclosed by dashed lines as the device external (external to the common circuit) 200 that contributes to the generation of the noise reduction intensity data (first and second noise reduction intensity data) (the filter processing unit 80 and the DRAM 90 are included in the device external (external to the common circuit) 200).

The demosaic processing unit 10 performs demosaic processing on the input image (RAW image, Bayer image) D1, which is an image before gradation correction, and converts it into an RGB image D2. Although not shown here, a block that performs isolated point removal processing and chromatic aberration correction on the input image D1 may be provided before the demosaic processing unit 10.

The tone correction unit 20 performs tone correction on the RGB image D2 before tone correction from the demosaic processing unit 10. The tone correction is adaptively performed on a pixel-by-pixel basis based on the image analysis results, for example, when shooting a subject with a large difference in brightness, such as backlight, or when it is desired to brighten a specific area such as a face. In addition, in processes such as backlight correction and multiple exposure synthesis (HDR: High Dynamic Range), an adaptive gain-up amount is calculated for each pixel or pixel block, and tone correction is performed for the entire image. Furthermore, in gamma correction, a different gain-up amount may be applied to each pixel or pixel block. This gain-up amount may be uniform for RGB. There is a degree of freedom in the manner of tone correction by the tone correction unit 20, and various design changes are possible.

The tone correction unit 20 performs tone correction to output a tone-corrected image (RGB image after tone correction) D3 and gain map data (Gain Map) D4. The gain map data D4 is data based on the tone correction amount of the tone corrected image D3, and corresponds to the logarithm of the tone correction amount (gain) of the tone corrected image D3. The gain map data D4 divides the tone corrected image D3 into regions for each pixel or pixel block, and has feature values for each divided region. The tone corrected image D3 is input to the YUV conversion unit 30 and the color analysis unit 40, and the gain map data D4 is input to the noise reduction intensity data generation unit 50.

The YUV conversion unit 30 obtains luminance data (luminance data of the gradation corrected image) D5 by YUV converting the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20. The luminance data D5 is generated by dividing the gradation corrected image D3 into regions for each pixel or pixel block, and has feature values for each divided region. The luminance data D5 is input to the noise reduction intensity data generation unit 50. The YUV conversion unit 30 also outputs the YUV converted gradation corrected image D6 to the noise reduction processing unit 60.

The color analysis unit 40 obtains color likelihood data (color likelihood data of the gradation corrected image) D7 by performing color analysis (RGB analysis) on the gradation corrected image (RGB image after gradation correction) D3 from the gradation correction unit 20. The color likelihood data D7 indicates the likelihood of a specific color, such as the color of human skin or flesh color. The color likelihood data D7 divides the gradation corrected image D3 into regions for each pixel or pixel block, and has feature values for each divided region. The color likelihood data D7 is input to the noise reduction strength data generation unit 50.

The noise reduction intensity data generation unit 50 receives gain map data D4 from the gradation correction unit 20, luminance data D5 from the YUV conversion unit 30, and color likelihood data D7 from the color analysis unit 40.

In addition, the noise reduction intensity data generating unit 50 receives an Additional Map and/or an Extra Map as external map data D8. The Additional Map is used to convert the noise reduction intensity data (NR intensity data) D9 once generated as the NR level output D10 into "first noise reduction intensity data (first NR intensity data) D12", filter the "first noise reduction intensity data (first NR intensity data) D12" in the filter processing unit 80, and store the data once in the DRAM 90 as "processed data D12X", which is then looped (input) into the noise reduction intensity data generating unit 50. That is, the processed data D12X generated based on the first noise reduction intensity data D12 is included in the external map data D8 (Additional Map) and looped (input) to the noise reduction intensity data generating unit 50. In addition, the "processed data D12X generated based on the first noise reduction intensity data (first NR intensity data) D12" looped (input) to the noise reduction intensity data generating unit 50 is used to improve the accuracy (reliability) of the "second noise reduction intensity data (second NR intensity data) D11" to be applied to the actual noise reduction processing (NR processing). This point will be described in detail later. The Extra Map can be the gain amount of the HDR processing for the image before the tone correction, for example, the logarithm of the gain amount during the HDR processing. In addition, in FIG. 2, the Additional Map and the Extra Map as the external map data D8 are depicted together as a single Extra Map, whereas in FIG. 3, the Additional Map and the Extra Map as the external map data D8 are depicted separately.

The noise reduction intensity data generating unit 50 generates noise reduction intensity data (NR intensity data) D9 based on any combination of multiple data generated according to the image after gradation correction. More specifically, the noise reduction intensity data generating unit 50 generates noise reduction intensity data D9 based on any combination of the luminance data D5 from the YUV conversion unit 30, the color likelihood data D7 from the color analysis unit 40, the gain map data D4 from the gradation correction unit 20, and the external map data D8. This makes it possible to provide a high-quality image in which noise reduction processing has been performed satisfactorily for captured images captured under various conditions. The noise reduction intensity data generating unit 50 generates noise reduction intensity data D9 based on any one of the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8, and any combination of these data. In particular, the noise reduction intensity data generating unit 50 generates second noise reduction intensity data D11 to be applied to the actual noise reduction process based on at least the external map data D8 among the above data, i.e., processed data D12X obtained by processing the first noise reduction intensity data D12 looped (input) from an input circuit for external map data (filter processing unit 80 and DRAM 90 outside the device (outside the common circuit) 200).

In the image processing device 1, the demosaic processing unit 10, the tone correction unit 20, the YUV conversion unit 30, the color analysis unit 40, and the noise reduction intensity data generation unit 50 share the circuit (common circuit portion 100) (reducing the circuit size), and generate the second noise reduction intensity data D11 to be applied to the actual noise reduction process, and the processed data D12X obtained by processing the first noise reduction intensity data D12 used to improve the accuracy (reliability) of the second noise reduction intensity data. That is, the noise reduction intensity data generation unit 50 generates the first noise reduction intensity data D12 for the image by the common circuit (common circuit portion 100), and generates the second noise reduction intensity data D11 for the image in response to the input of the processed data D12X generated based on the first noise reduction intensity data D12.

In the initial operation (first round operation) of the image processing device 1, the noise reduction intensity data D9 generated by the noise reduction intensity data generating unit 50 is output to the outside of the device (outside of the common circuit) 200 as the first noise reduction intensity data D12 as the noise reduction level output (NR level output) D10. The first noise reduction intensity data D12 output to the outside of the device 200 is filtered by the filter processing unit (external circuit) 80, temporarily stored in the DRAM (external memory) 90, and then included in the external map data D8 as "processed data D12X of the first noise reduction intensity data D12" and looped (input) to the noise reduction intensity data generating unit 50.

The filter processing unit 80 executes various types of filter processing, such as a low-pass filter, an epsilon filter, a bilateral filter, and a nonlocal means filter. The filter processing unit 80 may be realized by a circuit equipped with a general-purpose image processing group, and may use, for example, a CPU (Central Processing Unit) or a DRP (Dynamic Reconfigurable Processor).

In the above description, a case has been exemplified in which the noise reduction intensity data D9 that has been subjected to filtering is stored and then looped (input) to the noise reduction intensity data generation unit 50 as processed data D12X of the first noise reduction intensity data D12. However, the noise reduction intensity data D9 may be stored as is, filtered after being read out, and looped (input) to the noise reduction intensity data generation unit 50 as processed data D12X of the first noise reduction intensity data D12.

Even in the second or subsequent rounds of operation of the image processing device 1, the loop (input) of the "first noise reduction strength data D12 and its processed data D12X" via the noise reduction strength data generation unit 50, the filter processing unit 80, and the DRAM 90 is repeated until the actual noise reduction process is performed using the "second noise reduction strength data D11". That is, the noise reduction strength data D9 generated by the noise reduction strength data generation unit 50 becomes the "first noise reduction strength data D12 and its processed data D12X" when looped (input) to the noise reduction strength data generation unit 50, and becomes the "second noise reduction strength data D11" when input to the noise reduction processing unit 60.

When the predetermined condition is satisfied, the loop (input) of the first noise reduction strength data D12 and its processed data D12X is terminated, and the noise reduction strength data D9 generated by the noise reduction strength data generation unit 50 is input to the noise reduction processing unit 60 as "second noise reduction strength data D11". There is a degree of freedom in how the predetermined condition is set, and for example, the predetermined condition can be set as follows: the loop (input) of the first noise reduction strength data D12 and its processed data D12X is repeated a predetermined number of times, a predetermined time has passed, or the quality of the first noise reduction strength data D12 and its processed data D12X exceeds a predetermined level.

In this way, by efficiently using calculations outside the device (outside the common circuit) 200 while minimizing the circuit usage portion (common circuit portion 100) of the image processing device 1, it is possible to improve the accuracy (reliability) of the first noise reduction strength data D12 and therefore the second noise reduction strength data D11 while suppressing the circuit scale, thereby achieving excellent image quality based on flexible and appropriate noise reduction processing.

In addition, in this embodiment, a mechanism for memory output of NR intensity is provided before NR processing in the image processing pipeline. Also, a mechanism for input from memory is provided for NR intensity calculation before NR processing in the image processing pipeline. Also, the external map (including the first noise reduction intensity data), luminance, color likelihood, and gradation correction gain input from the memory are recalculated to generate NR intensity (second noise reduction intensity data). Also, by applying filter processing (epsilon filter, bilateral filter, etc.) to the NR intensity input and output to the memory to mitigate the influence of noise on the NR intensity map, it becomes possible to set an appropriate NR intensity with more reduced noise influence. In this way, noise reduction is applied to values such as luminance, color likelihood, and gradation correction gain before noise reduction and treated as NR intensity.

The noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) is for applying an optimized noise reduction process to the gradation corrected image D6, taking into consideration the extent to which gradation correction has been performed in which part of the gradation corrected image (level distribution of gradation correction gain). In other words, the noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) can be in a map format in which the gradation corrected image D3 is divided into regions for each pixel or pixel block, and the level of noise reduction processing to be applied to each divided region is associated with each divided region.

Figures 4A and 4B are conceptual diagrams showing an example of noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11). For ease of explanation, in Figures 4A and 4B, the gradation correction image is simplified and drawn into a divided area of 25 squares in total, 5 squares vertically and 5 squares horizontally. In Figure 4A, the noise reduction intensity data of each divided area is drawn in three stages: H (High), M (Middle), and L (Low). In Figure 4B, the noise reduction intensity data of each divided area is drawn in five stages: 1, 2, 3, 4, and 5 (the higher the number, the higher the intensity, and the lower the number, the lower the intensity). In Figures 4A and 4B, the greater the degree of tone correction (tone correction gain level) of a divided region, the greater the intensity of the noise reduction intensity data is set to, and the smaller the degree of tone correction (tone correction gain level) of a divided region, the smaller the intensity of the noise reduction intensity data is set to. Note that there is a degree of freedom in how each divided region of the tone correction image is set and how many stages the intensity of the noise reduction intensity data is divided into, and various design changes are possible.

How the above noise reduction intensity data D9 (first noise reduction intensity data D12, second noise reduction intensity data D11) is generated will be explained in detail later.

The noise reduction processing unit 60 performs noise reduction processing (NR processing) on the gradation corrected image D6 that has been gradation corrected by the gradation correction unit 20 and YUV converted by the YUV conversion unit 30 based on the noise reduction intensity data D9 (second noise reduction intensity data D11) (see, for example, Figures 4A and 4B) generated by the noise reduction intensity data generation unit 50. By performing noise reduction processing using noise reduction intensity data D9 (second noise reduction intensity data D11) optimized in consideration of the extent to which gradation correction has been performed on which part of the gradation corrected image D3 (level distribution of gradation correction gain), it is possible to achieve a good balance between gradation correction and noise reduction processing and to achieve excellent image quality based on flexible and appropriate noise reduction processing.

The post-processing unit 70 performs post-processing such as gamma correction, edge enhancement, and super-resolution processing on the noise-reduction-processed YUV image (NR YUV) D13 to generate an output image (YUV) D14. Note that there is a degree of freedom in the post-processing performed by the post-processing unit 70, and various design changes are possible.

Figure 5 is a diagram showing a detailed configuration of the noise reduction intensity data generation unit 50. As described above, the noise reduction intensity data generation unit 50 receives luminance data D5, color likelihood data D7, gain map data D4, and external map data D8. The external map data D8 includes processed data D12X of the first noise reduction intensity data D12 that is looped (input) to the noise reduction intensity data generation unit 50.

The noise reduction intensity data generating unit 50 has a selection input unit 51 that selectively passes some or all of the luminance data D5, color likelihood data D7, gain map data D4, and external map data D8. In the following explanation, it is assumed that all of the luminance data D5, color likelihood data D7, gain map data D4, and external map data D8 have passed through the selection input unit 51. In particular, it is assumed that the external map data D8 including the first noise reduction intensity data D12 has passed through the selection input unit 51.

It is also possible to omit the selection input unit 51 from the noise reduction intensity data generation unit 50. In this case, only the data to be used among the luminance data D5, color likelihood data D7, gain map data D4, and external map data D8 may be input to the noise reduction intensity data generation unit 50, and unused data may not be input to the noise reduction intensity data generation unit 50.

The noise reduction intensity data generating unit 50 has a multiplication unit 52 that multiplies the gain map data D4 passed through the selection input unit 51 by the external map data D8. The noise reduction intensity data generating unit 50 has a selection input unit 53 that selectively passes either the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52. In the following description, the gain map data D4 or the multiplication result of the gain map data D4 and the external map data D8 by the multiplication unit 52 will be collectively referred to as "gain map data D4" (the output of the selection input unit 53 is "gain map data D4").

In this embodiment, when two pieces of data are multiplied, if there is no input of one piece of data (for example, if it does not pass through the selection input unit 51), one piece of data may be set to 1, and the other piece of data may be output as the multiplication result as is.

The noise reduction intensity data generating unit 50 has a front-stage lookup table 54 for application to each (at least any one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8. More specifically, the front-stage lookup table 54 has a luminance data lookup table (LUT(Y)) 54A applied to the luminance data D5, a color likelihood data lookup table (LUT(Color)) 54B applied to the color likelihood data D7, a gain map data lookup table (LUT(Gain)) 54C applied to the gain map data D4, and an external map data lookup table (LUT(EXT)) 54D applied to the external map data D8.

The noise reduction intensity data generating unit 50 has a multiplication unit 55A that multiplies the luminance data D5 that has passed through the luminance data lookup table 54A by the color likelihood data D7 that has passed through the color likelihood data lookup table 54B, a multiplication unit 55B that multiplies the multiplication result by the multiplication unit 55A by the gain map data D4 that has passed through the gain map data lookup table 54C, and a multiplication unit 55C that multiplies the multiplication result by the multiplication unit 55B by the external map data D8 that has passed through the external map data lookup table 54D. The multiplication result by the multiplication unit 55C is output to a selection input unit (Sel) 57.

The noise reduction intensity data generating unit 50 has an addition value calculation unit (Sum) 56A that calculates the addition value of luminance data D5 that has passed through a luminance data lookup table 54A, color likelihood data D7 that has passed through a color likelihood data lookup table 54B, gain map data D4 that has passed through a gain map data lookup table 54C, and external map data D8 that has passed through an external map data lookup table 54D. The addition calculation result by the addition value calculation unit 56A is output to a selection input unit (Sel) 57.

The noise reduction intensity data generating unit 50 has a maximum value selection calculation unit (Max) 56B that selectively passes the maximum value of the luminance data D5 that has passed through the luminance data lookup table 54A, the color likelihood data D7 that has passed through the color likelihood data lookup table 54B, the gain map data D4 that has passed through the gain map data lookup table 54C, and the external map data D8 that has passed through the external map data lookup table 54D. The result of the maximum value selection calculation by the maximum value selection calculation unit 56B is output to the selection input unit (Sel) 57.

The noise reduction intensity data generating unit 50 has a minimum value selection calculation unit (Min) 56C that selectively passes the minimum value of the luminance data D5 that has passed through the luminance data lookup table 54A, the color likelihood data D7 that has passed through the color likelihood data lookup table 54B, the gain map data D4 that has passed through the gain map data lookup table 54C, and the external map data D8 that has passed through the external map data lookup table 54D. The result of the minimum value selection calculation by the minimum value selection calculation unit 56C is output to the selection input unit (Sel) 57.

The noise reduction intensity data generating unit 50 has a selection input unit (Sel) 57. The selection input unit 57 selectively passes one of the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, and the minimum value selection calculation result by the minimum value selection calculation unit 56C. The selection input unit 57 may select, for example, the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C, randomly, or the calculator to be used may be preset. Alternatively, the selection input unit 57 may determine whether to select the multiplication result by the multiplication unit 55C, the addition calculation result by the addition value calculation unit 56A, the maximum value selection calculation result by the maximum value selection calculation unit 56B, or the minimum value selection calculation result by the minimum value selection calculation unit 56C, based on some criteria for generating more appropriate noise reduction intensity data.

Here, multiplication unit 55A, multiplication unit 55B, multiplication unit 55C, addition value calculation unit 56A, maximum value selection calculation unit 56B, minimum value selection calculation unit 56C, and selection input unit 57 function as a "calculation unit" that applies at least one of addition processing, multiplication processing, maximum value selection processing, minimum value selection processing, and random selection processing to each output of front-stage lookup table 54 (54A, 54B, 54C, 54D) to calculate a single output. Note that the configuration of the "calculation unit" described with reference to FIG. 5 is merely an example, and it is sufficient if a single output can be calculated from each output of front-stage lookup table 54 (54A, 54B, 54C, 54D).

The noise reduction intensity data generating unit 50 has a subsequent lookup table 58 for application to the single output calculated by the above-mentioned "calculation unit". The subsequent lookup table 58 is composed of a single lookup table (LUT) 58A corresponding to the single output calculated by the above-mentioned "calculation unit". The output that passes through the subsequent lookup table 58 (single lookup table 58A) becomes noise reduction intensity data that is optimized taking into account the extent to which tone correction has been performed on which part of the tone correction image (level distribution of tone correction gain) (for example, Figures 4A and 4B).

In this way, the front-stage lookup table 54 (54A, 54B, 54C, 54D) for application to each (at least one of) the luminance data D5, the color likelihood data D7, the gain map data D4, and the external map data D8, and the rear-stage lookup table 58 (58A) for application to a single output calculated from each output of the front-stage lookup table 54 (54A, 54B, 54C, 54D) are arranged in the front and rear stages sandwiching the "calculation unit". This enables flexible and appropriate characteristic conversion. For example, in the front-stage lookup table, a wide range is handled by applying logarithmic processing, and in the rear-stage lookup table applied to the single output calculated from each output of the front-stage lookup table, characteristic conversion such as returning the logarithm to an antilogarithm is possible. Alternatively, conversely, antilogarithms may be calculated in the front-stage lookup table and logarithms may be calculated in the rear-stage lookup table.

In this way, in the image processing device, image processing method, and image processing program of this embodiment, the input image is subjected to gradation correction, noise reduction intensity data generation, and noise reduction processing in that order. The gradation correction block outputs the gradation correction gain (gain map) along with the gradation correction image. The noise reduction intensity data generation block receives the luminance and gain map from the gradation correction image, and further performs color analysis to generate color likelihood. Furthermore, a map for each pixel is input from the outside, and the front-stage lookup table (LUT) is applied to each of the gain amount, luminance, color likelihood, and external map (including processed data of the first noise reduction intensity data), and the result of adding, multiplying, the maximum value, the minimum value, etc. of each is selected, and the rear-stage lookup table (LUT) is applied to the result as the noise reduction intensity data. In addition, the external map can also input the gain amount when the gain is multiplied by HDR processing or the like before the gradation correction input, and in that case, it can also be added directly to the gain map. By treating the HDR gain and tone correction gain logarithmically, simple addition becomes possible. The noise reduction strength data is realized by contributing as a coefficient to the correlation between the subject pixel and surrounding pixels. For example, in the case of a Bilateral filter, it is contributed as a coefficient to the difference between the subject pixel and surrounding pixels, and in the case of a NonLocalMeans filter, it is contributed as a coefficient to the patch correlation, so that the strength of the noise reduction effect can be controlled according to the noise reduction strength.

In processes such as backlight correction and multiple exposure synthesis (HDR), an adaptive gain-up amount is calculated for each pixel or pixel block, and tone correction of the entire image may be performed. Also, in gamma correction and the like, a different gain-up amount is applied to each pixel or pixel block. If the intensity of noise reduction can be changed according to each of these conditions, tone correction and noise reduction can be achieved in a well-balanced manner. Therefore, by enabling processing that takes into account not only the gain amount and brightness but also color likelihood and an external map (including processed data of the first noise reduction intensity data), it is possible to control the smoothness of skin-colored parts and the smoothness of a specific area that is freely set. In addition, by enabling free map input from the outside, the intensity of noise reduction can be changed according to elements other than gain, brightness, and color likelihood, and it can also be applied to applications such as blurring specific areas such as backgrounds, faces, and vehicle numbers.

Figure 6 is a conceptual diagram showing the flow of noise reduction processing using the first and second noise reduction strength data. Figure 6 illustrates an example in which the processed data D12X of the first noise reduction strength data D12 is looped (input) only once to the noise reduction strength data generating unit 50. That is, the noise reduction strength data in the first round becomes the "first noise reduction strength data D12 and its processed data D12X," and the noise reduction strength data in the second round becomes the "second noise reduction strength data D11."

In step ST100, the common circuit part 100 (demosaic processing part 10, gradation correction part 20, YUV conversion part 30, color analysis part 40 and noise reduction intensity data generation part 50) of the image processing device 1 executes a first round of processing by the input image (RAW image, Bayer image) and outputs the first round of noise reduction intensity data D9 (first noise reduction intensity data D12) (NR Level output D10). In step ST200, the first round of noise reduction intensity data D9 (first noise reduction intensity data D12) is subjected to map processing (filter processing) by the filter processing part 80, and memory storage is performed in the DRAM 90. Then, the noise reduction intensity data read from the DRAM 90 is looped (input) to the noise reduction intensity data generation part 50 as the "processed data D12X of the first noise reduction intensity data D12" included in the external map data D8. As is clear from the images before and after filtering, the processed data D12X of the first noise reduction strength data D12 is of higher quality and has less adverse effects from noise than the first noise reduction strength data D12 before processing.

In step ST300, the common circuit portion 100 (demosaic processing unit 10, gradation correction unit 20, YUV conversion unit 30, color analysis unit 40, and noise reduction intensity data generation unit 50) of the image processing device 1 performs a second round of processing on the input image (RAW image, Bayer image) and outputs second round noise reduction intensity data D9 (NR Level output D10). This second round of noise reduction intensity data is "second noise reduction intensity data D11" based on the "processed data D12X of the first noise reduction intensity data D12" included in the external map data D8 input in the first round of processing.

Then, a noise reduction processing unit 60 separate from the common circuit of the image processing device 1 performs noise reduction processing using the second noise reduction strength data D11, and a post-processing unit 70 performs post-processing to produce the developed output (final image) after NR processing. Note that in the second round of processing, the noise reduction strength data (NR Level) is processed internally and is not output to memory. What is output to memory is the YUV image after noise reduction processing (NR YUV).

Figures 7A and 7B are diagrams showing an example of noise reduction strength data before and after map processing (filter processing) by the filter processing unit 80. As shown in Figure 7A, the noise reduction strength data before filter processing has a rough appearance (rough grain and resolution) of the wall and doll's face, and is greatly affected by noise, whereas as shown in Figure 7B, the noise reduction strength data after filter processing has a smooth appearance (fine grain and resolution) of the wall and doll's face, and the effects of noise are suppressed. Furthermore, contours with large differences in light and dark are emphasized more in Figure 7B than in Figure 7A, and the effects of noise are suppressed.

Fig. 8 is a flowchart showing an example of an image processing method by the image processing device 1. Fig. 8 illustrates an example in which the processed data D12X of the first noise reduction strength data D12 is looped (input) only once to the noise reduction strength data generating unit 50. That is, the noise reduction strength data in the first round becomes "first noise reduction strength data D12 and its processed data D12X", and the noise reduction strength data in the second round becomes "second noise reduction strength data D11".

In the first round of processing, first, a RAW image is input to the image processing device 1 (step ST1). Next, the demosaic processing unit 10 performs demosaic processing on the RAW image to convert it into an RGB image (step ST2). Next, the tone correction unit 20 performs tone correction processing on the RGB image to output a tone-corrected RGB image and gain map data D4 (step ST3). Next, the YUV conversion unit 30 YUV converts the tone-corrected RGB image to output luminance data D5 (step ST4). Next, the color analysis unit 40 performs color analysis on the tone-corrected RGB image to output color likelihood data (e.g., skin map) D7 (step ST5). Next, the noise reduction intensity data generation unit 50 generates noise reduction intensity data (NR level) D9 based on the input luminance data D5, color likelihood data D7, and gain map data D4 (step ST6). The generated noise reduction intensity data (NR level) D9 is output to the outside of the image processing device 1 (outside the common circuit) 200 (step ST7). Note that in the first round of processing, since the "first noise reduction intensity data D12" does not exist, the "processed data D12X of the first noise reduction intensity data D12" is not looped (input) to the noise reduction intensity data generating unit 50 as the external map data D8 (see dashed line).

The processing of steps ST1 to ST7 in FIG. 8 corresponds to the processing of step ST100 in FIG. 6.

The noise reduction intensity data (NR level) D9 obtained in the first round of processing is input to the outside of the device (outside the common circuit) 200 (step ST8). Then, in the outside of the device (outside the common circuit) 200, the noise reduction intensity data (NR level) D9 is filtered by the filter processing unit (external circuit) 80 and stored in the DRAM (external memory) 90 (step ST9). After that, the stored noise reduction intensity data (NR level) is read out and output as an image (step ST10). This noise reduction intensity data (NR level) is looped (input) to the image processing device 1 (noise reduction intensity data generation unit 50) as "processed data D12X of the first noise reduction intensity data D12" as the processed NR level.

The processing of steps ST8 to ST10 in FIG. 8 corresponds to the processing of step ST200 in FIG. 6.

In the second round of processing, the processed data D12X (processed NR level) of the first noise reduction intensity data D12 is output to the noise reduction intensity data generation unit 50. Also, a RAW image is input to the image processing device 1 (step ST11). Next, the demosaic processing unit 10 performs demosaic processing on the RAW image to convert it into an RGB image (step ST12). Next, the tone correction unit 20 performs tone correction processing on the RGB image to output a tone corrected RGB image (step ST13). Next, the YUV conversion unit 30 outputs a tone corrected YUV image obtained by YUV converting the tone corrected RGB image (step ST14). Note that in the second round of processing, since the luminance data D5, color likelihood data D7, and gain map data D4 have already been acquired in the first round of processing, the luminance data D5, color likelihood data D7, and gain map data D4 are not reacquired or reused (see dashed lines). However, if necessary, all or part of the luminance data D5, color likelihood data D7, and gain map data D4 may be reacquired or reused.

Next, the noise reduction strength data generating unit 50 generates "second noise reduction strength data D11" as a more accurate (highly reliable) NR level based on the "processed data D12X (processed NR level) of the first noise reduction strength data D12" that has been looped (input) from the first round of processing (step ST15). Next, the noise reduction processing unit 60 performs noise reduction processing (NR processing) on the tone-corrected YUV image from the YUV conversion unit 30 based on the "second noise reduction strength data (NR level) D11" (step ST16). Next, the post-processing unit 70 performs post-processing (YUV processing) such as gamma correction, edge enhancement, and super-resolution processing on the YUV image after the noise reduction processing (step ST17). Finally, the post-processed YUV image is output as the final image (step ST18).

The processing of steps ST11 to ST18 in FIG. 8 corresponds to the processing of step ST300 in FIG. 6.

In addition, the present invention is not limited to the above-described embodiment, and various modifications can be made in the implementation stage without departing from the gist of the invention. Furthermore, the functions performed in the above-described embodiment may be implemented in appropriate combinations as much as possible. The above-described embodiment includes various steps, and various inventions can be extracted by appropriate combinations of the multiple components disclosed. For example, if the effect can be obtained even if some components are deleted from all the components shown in the embodiment, then the configuration from which these components are deleted can be extracted as an invention.

The invention as described in the claims of the original application is set forth below.
[Appendix 1]
a noise reduction intensity data generating unit that generates first noise reduction intensity data for an image, and generates second noise reduction intensity data for the image in response to input of processing data generated based on the first noise reduction intensity data;
a noise reduction processing unit that performs a noise reduction process on the image based on the second noise reduction intensity data;
13. An image processing device comprising:
[Appendix 2]
After the first noise reduction intensity data is stored in an external memory, the processed data is input to the noise reduction intensity data generating unit.
2. The image processing device according to claim 1 .
[Appendix 3]
the first noise reduction intensity data is subjected to a filtering process by an external circuit, and then the processed data is input to the noise reduction intensity data generating unit;
3. The image processing device according to claim 1 or 2.
[Appendix 4]
A tone correction unit that performs tone correction on the image,
the noise reduction intensity data generating unit generates the first noise reduction intensity data based on any combination data of a plurality of data generated according to the image;
4. The image processing device according to claim 1,
[Appendix 5]
the noise reduction intensity data generating unit generates the first noise reduction intensity data based on any combination of luminance data of the tone corrected image, color likelihood data of the tone corrected image, gain map data based on a tone correction amount of the tone corrected image, and external map data.
5. The image processing device according to claim 1,
[Appendix 6]
The processing data is input from an input circuit for the external map data.
6. The image processing device of claim 5.
[Appendix 7]
A method for executing each process of the image processing device according to any one of claims 1 to 6 by a computer.
[Appendix 8]
A program for causing a computer to execute each process of the image processing device according to any one of claims 1 to 6.

1: Image processing device 1X: Digital camera (photographing device)
10: Demosaic processing unit 20: Tone correction unit 30: YUV conversion unit 40: Color analysis unit 50: Noise reduction intensity data generation unit (NR intensity data generation unit)
51: Selection input unit 52: Multiplication unit 53: Selection input unit 54: Pre-stage lookup table 54A: Luminance data lookup table (LUT(Y))
54B: Color likelihood data lookup table (LUT(color))
54C...Lookup table for gain map data (LUT(Gain))
54D: Lookup table for external map data (LUT(EXT))
55A: Multiplication unit (arithmetic unit)
55B: Multiplication unit (arithmetic unit)
55C: Multiplication unit (calculation unit)
56A: Summation value calculation unit (Sum) (calculation unit)
56B: Maximum value selection calculation unit (Max) (calculation unit)
56C: Minimum value selection calculation unit (Min) (calculation unit)
57: Selection input unit (Sel) (calculation unit)
58: Rear-stage lookup table 58A: Single lookup table (LUT)
60: Noise reduction processing section (NR processing section)
70: Post-processing section 80: Filter processing section (external circuit)
90...DRAM (Dynamic Random Access Memory) (external memory)
100...Common circuit part 200...External to the device (external to the common circuit)

Claims (6)

a noise reduction intensity data generating unit that outputs first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receives processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, and generates second noise reduction intensity data taking into account the processed data;
a noise reduction processing unit that performs noise reduction processing on the image data by applying the second noise reduction intensity data;
having
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
13. An image processing device comprising: a gradation correction unit that performs gradation correction on predetermined RGB image data to generate gradation-corrected image data;
a YUV conversion unit that generates YUV image data by YUV converting the gradation corrected image data;
a noise reduction intensity data generating unit which outputs to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction unit generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and which receives processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data;
a noise reduction processing unit that performs noise reduction processing on the YUV image data by applying the second noise reduction intensity data;
having
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
13. An image processing device comprising: An image processing method executed by an image processing device, comprising:
a noise reduction intensity data generating step of outputting first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data;
a noise reduction processing step of performing noise reduction processing on the image data by applying the second noise reduction intensity data;
having
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
13. An image processing method comprising: An image processing method executed by an image processing device, comprising:
a gradation correction step for generating gradation-corrected image data by performing gradation correction on predetermined RGB image data;
a YUV conversion step of generating YUV image data by YUV converting the gradation corrected image data;
a noise reduction intensity data generating step of outputting to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when generating the gradation corrected image data in the gradation correction step, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color, and generating second noise reduction intensity data taking into account processing data generated by the filter processing unit performing a predetermined filter processing on the first noise reduction intensity data;
a noise reduction processing step of performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data;
having
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
13. An image processing method comprising: Computer,
a noise reduction intensity data generating means for outputting first noise reduction intensity data generated based on predetermined image data to a filter processing unit, and for receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data;
a noise reduction processing means for performing noise reduction processing on the image data by applying the second noise reduction intensity data;
Function as a
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
A program characterized by: Computer,
A tone correction means for generating tone-corrected image data by performing tone correction on predetermined RGB image data;
YUV conversion means for generating YUV image data by YUV conversion of the gradation corrected image data;
a noise reduction intensity data generating means for outputting to a filter processing unit first noise reduction intensity data generated based on gain map data corresponding to an amount of gradation correction for the RGB image data when the gradation correction means generates the gradation corrected image data, luminance data in the YUV image data, or likelihood data indicating a likelihood between a color indicated by the YUV image data and a predetermined color , and for receiving processed data generated by the filter processing unit performing a predetermined filter process on the first noise reduction intensity data, thereby generating second noise reduction intensity data taking into account the processed data;
a noise reduction processing means for performing noise reduction processing on the YUV image data by applying the second noise reduction intensity data;
Function as a
The filtering process is performed by a low-pass filter, an epsilon filter, a Bilateral filter, or a NonLocalMeans filter.
A program characterized by:

Images