JP7814728B2 - Image signal processing method - Google Patents

Description

The present invention relates to an image signal processing method for reducing color noise.

In recent years, image signal processing methods have been proposed to reduce color noise, which is noise related to color in image data.

Color noise is noise that occurs when color information that should not actually be present in image data is included for some reason.

As a result, when image data containing this color noise is output to the rear LCD screen of a digital camera, a monitor screen, or paper media, color information that does not actually exist is displayed as an image signal, causing the photographer to feel uncomfortable.

Patent document 1 discloses that for image data that has undergone white balance correction, color difference data is calculated for each pixel, and for image data of pixels whose color difference data falls within a first color difference range and whose brightness falls within a specified brightness range, color noise reduction processing is performed to reduce the color difference data of those pixels.

Patent No. 5460202

However, the image processing device disclosed in Patent Document 1 has the problem that color differences are eliminated by performing color noise reduction processing to reduce color difference data, as described above.

In other words, the loss of color differences in an image that has undergone color noise reduction processing poses the problem of the image becoming achromatic.

Furthermore, each color of the image signal contains high-frequency noise components, and high-frequency noise poses the problem of generating isolated, monochromatic color noise.

As a result, applying a low-pass filter or similar to suppress isolated color noise can result in the color noise spreading to the surrounding area, which can be problematic.

The present invention was made in consideration of these circumstances, and aims to provide an image signal processing method that reduces noise by removing high-frequency components that cause color noise.

A first invention as a means for solving the above problem is an image signal processing method for processing three-color image signals, comprising an image signal determination unit for determining an image signal of one color to be corrected, a linear sum calculation unit for calculating a linear sum of image signals of two colors other than the image signal to be corrected, a first subtraction unit for subtracting the linear sum calculated by the linear sum calculation unit from the image signal to be corrected, a low-frequency extraction unit for extracting low-frequency components from the calculation result of the first subtraction unit, a second subtraction unit for subtracting an image signal that has passed through the low-frequency extraction unit from the calculation result of the first subtraction unit, and a color difference function unit that calculates a color difference from an image signal and determines a first coefficient from the color difference based on a function; a multiplication unit that multiplies a calculation result of the second subtraction unit by the first coefficient determined by the color difference function unit; a threshold comparison and selection unit that compares the calculation result of the multiplication unit with an arbitrary parameter and selects either the calculation result or the parameter; and a noise removal unit that subtracts the selection result of the threshold comparison and selection unit from the image signal to be corrected, and the calculation result of the noise removal unit is used as a corrected image signal for the image signal to be corrected.

The second invention, which is a means for solving the above-mentioned problem, is an image signal processing method according to the first invention, characterized in that the color difference function unit calculates multiple color differences using image signals of two colors from the image signals of the three colors, determines each coefficient based on a function from the multiple color differences, and determines the first coefficient using one of the linear sum, maximum value, minimum value, or median value of the each coefficient.

The present invention provides an image signal processing method that reduces noise by removing high-frequency components that cause color noise.

FIG. 1 is a block diagram showing a configuration of a solid-state imaging device according to an embodiment of the present invention. Flowchart for noise reduction according to an embodiment of the present invention

The best mode for carrying out the present invention will now be described with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment.

In this embodiment, 100 is a lens barrel, 101 is a solid-state image sensor, 1011 is a microlens, 1012 is a photoelectric conversion unit, 1013 is a wiring unit, 102 is a CPU, 1021 is an image signal determination unit, 1022 is a linear sum calculation unit, 1023 is a first subtraction unit, 1024 is a low-frequency extraction unit, 1025 is a second subtraction unit, 1026 is a color difference function unit, 1027 is a multiplication unit, and 1028 is a threshold comparison and selection unit.

Figure 1 is a diagram showing the configuration of the imaging device of this embodiment.

The imaging device 10 consists of a lens barrel 100 and a camera body (not shown).

The lens barrel 100 is detachable from the camera body, and the mount of the lens barrel 100 is connected to the mount of the camera body via a bayonet mechanism or the like. Note that the lens barrel 100 may also be fixed to the camera body.

The solid-state imaging element 101 includes a microlens 1011, a photoelectric conversion unit 1012, a wiring unit 1013, etc.

The microlens 1011 is a lens that focuses the light beam incident on the lens barrel 100 so that it is suitable for reception by each photoelectric conversion unit 1012.

The photoelectric conversion unit 1012 converts the light beam that passes through the microlens 1011 into an image signal.

The image signal converted by the photoelectric conversion unit 1012 will be explained. The photoelectric conversion unit 1012, which converts light beams into image signals, is arranged in each pixel. When the solid-state imaging element 101 is a Bayer type, the photoelectric conversion unit 1012 arranged in each pixel converts the signal into an image signal of the same color as the color filter arranged in the microlens 1011 and photoelectric conversion unit 1012. Color filters of three colors, red (R), green (G), and blue (B), are generally used, and in this case, three types of image signals are converted.

On the other hand, if the solid-state imaging device 101 is a stacked type, image signals corresponding to the number of stacked photoelectric conversion units are converted from one pixel. In this embodiment, the photoelectric conversion units 1012 are stacked in three layers to obtain image signals of three colors: red (R), green (G), and blue (B) from one pixel.

The wiring unit 1013 transfers the image signals converted by the photoelectric conversion units 1012 from each photoelectric conversion unit 1012 to the CPU 102.

The CPU 102 performs various processes on the image signals converted by the photoelectric conversion unit 1012 and transferred via the wiring unit 1013.

The image signal determination unit 1021 determines the image signal to be corrected from the image signals converted by the photoelectric conversion unit 1012 and transferred via the wiring unit 1031.

The linear sum calculation unit 1022 calculates a linear sum from image signals other than the image signal to be corrected determined by the image signal determination unit 1021. The specific calculation method will be described later.

The first subtraction unit 1023 subtracts the linear sum of image signals other than those to be corrected, calculated by the linear sum calculation unit 1022, from the image signal to be corrected, determined by the image signal determination unit 1021.

The low-frequency extraction unit 1024 extracts low-frequency components from the results calculated by the first subtraction unit 1023.

The second subtraction unit 1025 subtracts the image signal extracted by the low-frequency extraction means 1024 from the calculation result of the first subtraction unit 1023.

The color difference function 1026 calculates the color difference between the image signals converted by the photoelectric conversion unit 1012 of the solid-state imaging element 101, and returns a coefficient based on the calculated color difference function.

The multiplication unit 1027 multiplies the result calculated by the second subtraction means 1025 by the coefficient returned by the color difference function.

The threshold comparison and selection unit 1028 compares the result calculated by the multiplication unit 1027 with an arbitrary parameter and selects an appropriate value.

The noise removal unit 1029 subtracts the appropriate value selected by the threshold comparison and selection unit 1028 from the image signal to be corrected determined by the image signal determination unit 1021. The calculation result is the corrected image signal.

Next, an example of the image signal processing method of the present invention will be explained using the flowchart in Figure 2.

As shown in Figure 1, a light beam passing through the lens barrel 100 reaches the solid-state image sensor 101, passes through the microlens 1011, and is guided to the photoelectric conversion unit 1012, where it is converted into an image signal. The solid-state image sensor 101 used in this embodiment has three layers of photoelectric conversion units 1012 stacked per pixel. Therefore, three types of image signals are converted from one pixel.

In step #1, the image signal determination unit 1021 determines the image signal to be corrected from the three types of image signals converted by the photoelectric conversion unit 1012 of the solid-state imaging element 101.

In step #2, the linear sum calculation unit 1022 calculates a linear sum using two types of image signals other than the image signal to be corrected determined by the image signal determination unit 1021 in step #1.

In step #3, the first subtraction unit 1023 subtracts the linear sum calculated by the linear sum calculation unit 1022 in step #2 from the image signal to be corrected determined by the image signal determination unit 1021 in step #1. The result of this subtraction is defined as the first subtraction result.

In step #4, the low-frequency extraction unit 1024 extracts low-frequency components by resizing the result calculated by the first subtraction unit 1023 in step #3. Note that as long as low-frequency components can be extracted, a low-pass filter can also be used instead of resizing.

In step #5, the second subtraction unit 1025 subtracts the low-frequency components extracted in step #4 from the first subtraction result, which is the calculation result of step #3. As a result, it becomes possible to extract high-frequency components.

In step #6, the color difference function unit 1026 calculates the color differences between the three types of image signals converted by the photoelectric conversion unit 1012. Furthermore, it returns coefficients based on the function from the calculated color differences.

In step #7, the multiplication unit 1027 multiplies the result calculated in step #5 by the coefficient determined in step #6.

In step #8, the threshold comparison and selection unit 1028 compares the result calculated in step #7 with the parameters.

In step #9, the noise removal unit 1029 subtracts the comparison result selected in step #8 from the image signal to be corrected determined in step #1.

The result of the calculation in step #9 is the corrected image signal after noise removal from the image signal to be corrected in step #1.

The calculations performed in each of the above steps are explained in detail below.

In this example, the correction method will be specifically explained using green (G) as the image signal to be corrected from the image signals converted by the photoelectric conversion unit 1012.

Since the image signal to be corrected is green (G), in step #1 the image signal determination unit 1021 selects green (G) as the image signal to be corrected.

Next, in step #2, the linear sum calculation unit 1022 calculates a linear sum using the red (R) and blue (B) image signals other than the green (G) image signal to be corrected.

The specific calculation method performed by the linear sum calculation unit 1022 in this embodiment is shown below.

When the red (R) and blue (B) image signals are R and B, respectively, and the coefficient is α, the linear sum can be expressed as follows:
Linear sum: α*R+(1-α)B (0<α<1)

Next, the calculation of the first subtractor 1023 in step #3 will be described. If the green (G) image signal converted by the photoelectric converter 1012 is G and the green (G) after calculation is G1, the following equation is obtained.
G1=G-(α*R+(1-α)B) (0<α<1)

In step #4, low-frequency components are extracted from G1 calculated in step #3. Specifically, a reduction process is applied to G1, which is the calculated green (G), and then it is enlarged to its original size (resized) to create a low-frequency image signal G2. Alternatively, the image signal G2 may be created by applying a low-pass filter.

In step #5, the low-frequency image signal G2 extracted in step #4 is subtracted from G1 calculated in step #4. By subtracting the low-frequency image signal G2 from G1, it is possible to extract only the high-frequency components. The image signal from which only the high-frequency components have been extracted is designated image signal G3. As a result, magenta noise is reduced.

In step #6, the color differences of (R-G), (G-B), and (B-R) are calculated. Next, for the color difference (R-G), the coefficient f1 is calculated using the Gaussian function exp(-a*(R-G)^2). Similarly, coefficients f2 and f3 are calculated from the Gaussian function for the other color differences (G-B) and (B-R). The coefficient f is determined using the coefficients f1 to f3 calculated for each color difference.

There are various methods for determining coefficient f from coefficients f1 to f3. For example, it may be a linear sum of f1 to f3, or it may be the maximum, minimum, or median value. Furthermore, these methods may be used depending on the state of noise contained in the image.

In addition, in step #6, by using all three types of color difference, it is possible to correct the image signal with high precision. On the other hand, coefficients f1 and f2 can be calculated using a Gaussian function from the color differences (R-G) and (G-B), which correspond to the CbCr components of YCbCr. These can also be used to determine the coefficient f, as described above. In this case, the amount of calculation can be reduced.

In step #7, the image signal G3 extracted in step #5 is multiplied by the coefficient f calculated in step #6. The multiplication result is G4.

In step #8, the multiplication result G4 is compared with the parameters. Specifically, the multiplication result G4 is clipped so that it falls within the range of -ThMin to +ThMax. If the clipped value is G5, then G5 = Min(+ThMax, Max(-ThMin, G4)).

In step #9, by using the clipped G5 calculated in step #8 to set G = G - G5, the image signal corrected in step #1 can be converted into an image signal with color noise components removed.

In addition, in step #6 of this embodiment, the coefficients were calculated by applying a Gaussian function to each color difference. However, any function other than a Gaussian function that attenuates as the color difference (image signal difference) increases can be applied. For example, exp(-a * |R-G|^b) or a / (|R-G|^b+1) can be used. (However, |X| is the absolute value of X, and b > 0.)

Of the high-frequency color noise, green/magenta noise is particularly noticeable at high ISO settings. A major cause of this green/magenta noise is the fact that the high frequencies of green (G) behave independently compared to the other colors, red (R) and blue (B). Therefore, by assuming that the independent component of G is G1 = G - (α * R + (1 - α) B) (0 < α < 1) and extracting only the high-frequency components, it is possible to reduce green/magenta noise. Furthermore, because this alone cannot avoid the destruction of the color patterns in the original image (color bleeding, loss of color detail), noise reduction was performed after adjusting the intensity based on color differences in step #6 and setting a threshold in step #7.

Furthermore, in the embodiments described above, the solid-state imaging element 101 was assumed to be a stacked type, but a Bayer type solid-state imaging element having one type of image signal per pixel may also be used.

When using a stacked solid-state imaging element, signals can be acquired from the same position, making it easier to prevent the loss of high-frequency information and achieve color noise suppression. Furthermore, in the case of a sensor with mixed colors, the high-frequency information of the target signal is also contained in other color signals, making it easier to prevent the loss of high-frequency information and achieve color noise suppression.

Furthermore, with a Bayer-type solid-state imaging device, only one type of image signal can be obtained from one pixel. Therefore, the other two types of image signals can be obtained by interpolating from surrounding pixels. When a Bayer-type solid-state imaging device is used in the present invention, the image signal for one pixel is converted into three types through interpolation processing, and then the image signal to be corrected in step #1 is determined, allowing subsequent processing to be the same as when a stacked-type solid-state imaging device is used.

10 Imaging device 100 Lens barrel 101 Solid-state imaging element 1011 Microlens 1012 Photoelectric conversion unit 1013 Wiring layer 102 CPU
1021 Image signal determination unit 1022 Linear sum calculation unit 1023 First subtraction unit 1024 Low frequency extraction unit 1025 Second subtraction unit 1026 Color difference function unit 1027 Multiplication unit 1028 Threshold comparison and selection unit 1029 Noise removal unit

Claims (2)

1. An image signal processing method for processing three-color image signals,
an image signal determination unit that determines an image signal of one color to be corrected;
a linear sum calculation unit that calculates a linear sum of image signals of two colors other than the correction target;
a first subtraction unit that subtracts the linear sum calculated by the linear sum calculation unit from the image signal to be corrected;
a low-frequency extraction unit that extracts low-frequency components from the calculation result of the first subtraction unit;
a second subtraction unit that subtracts the image signal that has passed through the low-frequency extraction unit from the calculation result of the first subtraction unit;
a color difference function unit that calculates color differences from the three-color image signals and determines a first coefficient based on a function from the color differences;
a multiplication unit that multiplies the calculation result of the second subtraction unit by the first coefficient determined by the color difference function unit;
a threshold comparison and selection unit that compares the calculation result of the multiplication unit with an arbitrary parameter and selects either the calculation result or the parameter;
a noise removal unit that subtracts a selection result from the threshold comparison and selection unit from the image signal to be corrected;
The image signal processing method is characterized in that the calculation result of the noise removal unit is used as a corrected image signal for the image signal to be corrected. The image signal processing method described in claim 1, characterized in that the color difference function unit calculates multiple color differences using image signals of two colors from the three-color image signals, determines each coefficient based on a function from the multiple color differences, and determines the first coefficient using one of the linear sum, maximum value, minimum value, or median value of the each coefficient.