JP7409607B2 - Image processing device, image processing method and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program for reducing noise in an image.

BACKGROUND ART Conventionally, in order to suppress noise contained in images taken with a camera, a technique is known in which noise is reduced on a patch basis by focusing on patches made up of multiple pixels in an image. The process described in Non-Patent Document 1 selects a patch set consisting of a plurality of patches in a photographed image, generates a probability model of the patch represented by the mean and covariance matrix, and calculates each patch constituting the patch set. Perform noise reduction processing based on the maximum a posteriori probability method. Then, an output image is generated using the noise-reduced patch. At this time, an inverse covariance matrix, which is an inverse matrix of the covariance matrix, is calculated, and based on this, a pixel value after the noise reduction process is determined.

Implementation of the “Non-Local Bayes” (NL-Bayes) Image Denoising Algorithm. Image Processing On Line, 3 (2013), pp. 1-42.

However, in the conventional technology, the value of a pixel within a patch after noise reduction processing may become an abnormal value, and as a result, there is a possibility that the output image may be corrupted.

The present invention has been made in view of the above problems, and an object of the present invention is to suppress failure of an output image after noise reduction processing.

The present invention provides an image processing device for reducing noise in an image, comprising: a setting means for setting a plurality of patches in the image; a calculation means for calculating a covariance matrix based on the plurality of patches; a first correction means for performing a first correction on pixels included in the plurality of patches using a dispersion matrix; and a first correction means for performing a first correction on pixels included in the plurality of patches using a dispersion matrix; a second correction means for performing a second correction different from the first correction; a determination means for determining whether or not each of the plurality of patches has an abnormality after the first correction; a correction means for correcting the first corrected pixels included in the patch determined to be abnormal with the result of the second correction; and an output image using the first corrected pixels or the corrected pixels. and generating means for generating.

According to the present invention, it is possible to suppress failure of an output image after noise reduction processing.

Hardware configuration of embodiment 1 Block diagram of image processing in Embodiment 1 Flowchart of image processing in Embodiment 1 Flowchart of the first correction process of the first embodiment Flowchart of abnormality determination processing in Embodiment 1 Block diagram of image processing in Embodiment 2 Flowchart of image processing in Embodiment 2 Flowchart of first abnormality determination in embodiment 2 Flowchart of second abnormality determination in embodiment 2

(Embodiment 1)
In this embodiment, a method will be described in which an abnormality in correction using a covariance matrix is determined in patch-based noise reduction processing, and in the case of an abnormality, a failure in the output image is avoided.

In this embodiment, an example of noise reduction processing that obtains suitable image quality by adaptively determining the number of similar patches based on the amount of noise in a patch of interest and a small area around it will be described.

FIG. 1 is a block diagram showing the hardware configuration according to this embodiment. In FIG. 1, the image processing apparatus includes a CPU 101, a RAM 102, an HDD 103, a general-purpose interface (I/F) 104, a monitor 108, and a main bus 109. The general-purpose I/F connects an imaging device 105 such as a camera, an input device 106 such as a mouse and a keyboard, and an external memory 107 such as a memory card to the main bus 109.

Below, various processes realized by the CPU 101 operating various software (computer programs) stored in the HDD 103 will be described.

First, the CPU 101 starts an image processing application stored in the HDD 103, loads it in the RAM 102, and displays a user interface (UI) on the monitor 108. Subsequently, various data stored in the HDD 103 and external memory 107, image data captured by the imaging device 105, instructions from the input device 106, and the like are transferred to the RAM 102. Further, according to processing within the image processing application, various calculations are performed on the data stored in the RAM 102 based on instructions from the CPU 101. The calculation results are displayed on the monitor 108 or stored in the HDD 103 or external memory 107. Note that image data stored in the HDD 103 or external memory 107 may be transferred to the RAM 102. Further, image data transmitted from a server via a network (not shown) may be transferred to the RAM 102.

In the above configuration, details of a process of inputting image data to an image processing application, generating noise-reduced image data, and outputting the image data based on a command from the CPU 101 will be described.

Here, an overview of patch-based noise reduction processing will be explained. A partial image obtained by cutting out a part of input image data is called a patch. A plurality of patches existing in a predetermined reference area near the patch of interest are used as reference patches, and are referred to when performing noise reduction processing corresponding to the patch of interest. In patch-based noise reduction processing, patches similar to the pixel value distribution of the patch of interest are detected from among the reference patches as similar patches, and these are set as a patch set. Note that the patch of interest itself may be included among the plurality of reference patches. Noise reduction processing is performed on each similar patch based on the patch set. By aggregating similar patches with reduced noise, an image in which input image data is subjected to noise reduction processing is generated.

FIG. 2 is a block diagram of image processing according to this embodiment.

The image input unit 201 acquires input image data.

A patch of interest setting unit 202 cuts out a partial image of a predetermined shape from the input image data and sets it as a patch of interest. In this embodiment, the predetermined shape is a square of 6 pixels x 6 pixels.

The similar patch detection unit 203 sets a plurality of patches existing in a predetermined reference area near the patch of interest as reference patches, and detects a patch similar to the patch of interest among the reference patches as a similar patch. Each reference patch has the same shape as the patch of interest. Further, in this embodiment, it is assumed that the patch of interest is included in the similar patches.

The covariance matrix calculation unit 204 calculates a covariance matrix based on a plurality of similar patches.

The first correction unit 205 corrects the plurality of similar patches using the covariance matrix.

The abnormality determining unit 206 determines whether each of the corrected plurality of similar patches is abnormal (normal).

When the abnormality determination unit 206 determines that the patch is abnormal, the second correction unit 207 performs a correction different from the first correction on the patch before the first correction. The patch modification unit 208 modifies the plurality of similar patches using the results of the second correction. The patch aggregation unit 209 aggregates a plurality of corrected similar patches to generate an output image with reduced noise.

In patch-based noise reduction processing, it has an abnormality determination section that determines abnormalities in the patch after correction based on changes in multiple similar patches before and after correction, and if it is abnormal, performs another correction to correct the patch. This is a feature of the present invention. In this embodiment, a configuration will be described in which there are two different correction processing units, and correction results from the two correction processing units are selected and used according to the determination result of the abnormality determination unit. In this embodiment, only when it is determined that the first correction result is abnormal for the patch to be processed, the second correction unit 207 corrects the patch, and the patch correction unit 208 corrects the corrected result. The configuration for modifying the patch using the following will be explained.

FIG. 3 is a flowchart of image processing according to this embodiment.

In step S301, the image input unit 201 acquires input image data. In addition, the image data for aggregation is initialized to zero. The image data for aggregation consists of denominator image data and molecule image data, both of which have the same number of pixels as the input image data.

In step S302, the patch of interest setting unit 202 sets one patch of interest. Generally, a plurality of patches can be set as patches of interest for one piece of input image data. The processing after this step is repeatedly executed by changing the position of the patch of interest to be set. At this time, control may be performed so that all patches are set as the patch of interest once, or only some patches may be set as the patch of interest. In this embodiment, all patches are set as patches of interest in raster order from the upper left. As described above, in this embodiment, a 6 pixel x 6 pixel square partial image is cut out from the input image data and used as a patch of interest. In other words, the patch of interest consists of 36 pixels. In the following, the patch of interest is represented by a column vector T in which 36 pixels are arranged vertically. Generally, when a patch of interest consists of M pixels, the patch of interest is represented by a column vector T in which M pixels are arranged vertically.

In step S303, the similar patch detection unit 203 detects similar patches. First, a plurality of patches existing in a predetermined reference area near the patch of interest T are set as reference patches. The shape of each reference patch matches the shape of the patch T of interest. In the following, the i-th reference patch is represented by a column vector R_i in which M pixels are arranged vertically. Next, the degree of similarity with the target patch T is calculated for each of the plurality of reference patches. The degree of similarity is calculated by the sum of squared differences (SSD) of each pixel between the reference patch and the patch of interest T. Specifically, the similarity SSD_i of the i-th reference patch R_i is calculated using the following equation (1).

Here, the j-th component (pixel value) of the vector T is represented by T(j). As is clear from this calculation method, the smaller the value of the similarity SSD_i, the more similar the reference patch R_i is to the patch T of interest. Furthermore, some patches are detected as similar patches from among the plurality of reference patches based on the degree of similarity. In this embodiment, patches whose degree of similarity is smaller than a predetermined threshold are detected as similar patches. Let N be the number of similar patches detected here, and represent the i-th similar patch as P_i.

In step S304, the covariance matrix calculation unit 204 calculates a covariance matrix based on a plurality of similar patches. First, a reference patch Q is calculated using the following equation (2) based on a plurality of similar patches.

Next, a noise amount patch V is calculated using the reference patch Q using the following equation (3).

Here, k is a constant, and V_0 is a vector having the same shape as the reference patch Q, both of which are determined and stored in advance.

Then, a covariance matrix C is calculated using the following equation (4) using a reference patch Q based on a plurality of similar patches.

Here, a vector representing a patch is regarded as a matrix, and calculations are performed as a matrix. t is the transpose of the matrix. In step S305, the first correction unit 205 performs a first correction process on a plurality of similar patches. FIG. 4 is a flowchart of the first correction process. All steps are performed by the first correction unit 205. In the following, a similar patch before performing the first correction will be referred to as a patch before first correction, and a similar patch after performing the first correction will be referred to as a patch after first correction.

In step S401, an inverse covariance matrix C^-1, which is an inverse matrix of the covariance matrix C calculated in step S304, is calculated.

In step S402, a correction matrix H is calculated using the following equation (5).

Here, diag(V) represents a square diagonal matrix having elements of vector V on the main diagonal.

In step S403, a corrected patch O_i corresponding to each of the plurality of similar patches P_i is calculated using the following equation (6).

In step S306, the abnormality determination unit 206 performs abnormality determination on the first corrected patch. FIG. 5 is a flowchart of abnormality determination. All steps are performed by the abnormality determination unit 206.

In step S501, determination data that stores abnormality determination results is initialized. The determination data stores the results of determining whether all the processing target patches are normal (="0") or abnormal (="1") and the pixel positions of all the processing target patches.

In step S502, one patch to be processed is selected from the first patch before correction and the first patch after correction. The patch selected from the first pre-correction patch is designated as a processing target patch I, and the patch selected from the first post-correction patch is designated as a processing target patch I'. Furthermore, it is assumed that both patches to be selected have the same pixel position.

In step S503, standard deviations σ_I _and σ_I' are calculated for each of the processing target patch I and the processing target patch I'.

In step S504, the ratio of the standard deviations σ_I _and σ_I' is compared with the threshold value th based on the following equation (7).

If the ratio is greater than or equal to the threshold th, it is determined that the change is large and the process proceeds to S505; otherwise, the process proceeds to S506. Here, the threshold th may be a preset value, or may be calculated based on the statistics (average value, variance) of each pixel included in the patch.

In step S505, it is determined that the patch I' to be processed is abnormal, and the pixel position of the patch I' to be processed and the determination result "1" are output as determination data.

In step S506, it is determined that the patch I' to be processed is normal, and the pixel position of the patch I' to be processed and the determination result "0" are output as determination data.

In step S507, it is determined whether all the processing target patches have been processed. If all the target patches have been processed, the abnormality determination process ends; otherwise, the process advances to S502.

In step S307, the abnormality determining unit 206 determines whether or not the correction results are abnormal for all the first corrected patches based on the determination data. The first post-correction patch that is determined to be abnormal proceeds to step S308, and the first post-correction patch that is determined to be normal proceeds to step S309.

In step S308, the second correction unit 207 performs the second correction on the first pre-correction patch whose pixel position corresponds to the first post-correction patch determined to be abnormal. The second correction is a process different from the first correction, and in this embodiment, an average value filter is applied.

In step S309, the patch correction unit 208 uses the second correction result to correct the first correction patch. Specifically, the first corrected patch is replaced with the second correction result.

In step S310, it is determined whether all patches of interest have been processed. If all the patches of interest have been processed, the process advances to S311; otherwise, the process advances to S302.

In step S311, the patch aggregation unit 209 edits the aggregation image data based on the plurality of corrected patches that have undergone the first or second correction. As mentioned above, the image data for aggregation consists of denominator image data and molecular image data. For each of the plurality of corrected patches O_i, the values of all pixels are added to the corresponding position of the molecular image data. This position is the position where each pixel existed in the input image data. Furthermore, 1 is added to the same position in the denominator image.

The above is the flow of image processing in this embodiment.

Through the above processing, even if an abnormality occurs after noise reduction in patch-based noise reduction processing using a covariance matrix, the abnormality can be corrected and a suitable output image can be obtained.

Note that the processing of this embodiment can be applied to image data in various formats. For example, it can be applied to monochrome images, color images consisting of a plurality of color components, and Bayer array images.

Further, although the similarity between the reference patch and the patch T of interest is calculated using the sum of squared differences (SSD) of each pixel, it may be calculated using other methods. For example, it may be calculated using the sum of absolute differences (SAD) of each pixel, or the maximum value, minimum value, average value, median value, etc. of the absolute value of the difference between each pixel may be used.

In addition, for abnormality determination, we selected one patch to be processed from the first patch before correction and the first patch after correction, and compared the ratio of the standard deviation of the patch with the threshold th. may be determined. For example, a ratio or difference of other statistical quantities such as patch variance, average, minimum value, and maximum value may be used. Further, it is sufficient to know the amount of change other than the ratio or difference.

Further, as the second correction, an example has been described in which an average value filter is applied to the patch before the first correction, but correction may be performed using other methods. For example, a low-pass filter capable of reducing noise, such as a Gaussian filter, may be applied to the first patch before correction. In addition, for the first pre-correction patch, only the pixels determined to be abnormal may be output without correction, and the value before correction may be output as is, or the average pixel value calculated from several pixels in the patch may be output. You may.

Further, although the reference patch is calculated as the average of a plurality of similar patches, the present invention is not limited to this. For example, the value of each pixel of the reference patch may be determined by the median value, minimum value, etc. of corresponding pixels of a plurality of similar patches. It has an anomaly determination unit that determines whether the covariance matrix is abnormal, and performs normal "patch-based noise reduction processing" when the covariance matrix is determined to be normal. On the other hand, the main focus of this embodiment is to perform a correction process different from the normal "patch-based noise reduction process" when it is determined that the covariance matrix is abnormal. This correction processing is not limited to that described in this embodiment. For example, there is a method of calculating the corrected patch as a similar patch to be corrected, a reference patch, or a linear sum thereof.

(Embodiment 2)
In the first embodiment, an example has been described in which it is determined whether or not a corrected patch is abnormal on a patch-by-patch basis, and correction processing is performed on a patch-by-patch basis. In this embodiment, a method will be described in which the process of determining whether or not a corrected patch is abnormal is performed in two stages.

Specifically, in addition to the process of determining abnormality for each pixel in the patch to be processed (first stage), the patch to be processed is determined to be abnormal using the number of pixels determined to be abnormal in the first stage abnormality determination result. A method for determining whether or not this is the case (second stage) will be explained. Description of parts common to Embodiment 1 will be omitted.

FIG. 6 is a block diagram of image processing according to this embodiment.

The first abnormality determination unit 601 determines whether each pixel of a similar patch (=patch after first correction) that has undergone the first correction is abnormal (normal).

The second abnormality determination unit 602 performs a second abnormality determination using the first abnormality determination result in order to determine the abnormality with higher accuracy.

The second correction unit 603 performs a correction different from the first correction on the patch before the first correction when the second abnormality determination unit 602 determines that the patch is abnormal.

FIG. 7 is a flowchart of image processing according to this embodiment.

In step S701, the first abnormality determination unit 601 performs abnormality determination on the first corrected patch.

FIG. 8 is a flowchart of the first abnormality determination. All steps are performed by the first abnormality determination unit 601.

In step S801, first determination data that stores the first abnormality determination result is initialized. The first judgment data includes the results of determining whether each pixel is normal (= "0") or abnormal (= "1") for all patches to be processed, and a count of pixels determined to be abnormal for each patch to be processed. The value CNT and the pixel position of the patch are stored. CNT is initialized with zero.

Step S802 is the same process as step S502.

In step S803, one pixel at the same position is selected from each of the processing target patch I and the processing target patch I'. The selected pixels are designated as I_j and I'_j in order.

In step S804, the absolute difference value between pixels I_j and I'_j is compared with threshold value th1. Here, the threshold th1 is calculated by the following equation (8) using the noise amount patch V and a predetermined coefficient m.

If the absolute difference value is greater than or equal to the threshold th1, the process advances to S805; otherwise, the process advances to S806.

In step S805, CNT of the first determination data is incremented by one, and the pixel position is also stored.

In step S806, it is determined whether all pixels in the patch to be processed have been selected. If all pixels in the patch to be processed have been selected, the first abnormality determination process ends; otherwise, the process advances to S803.

In step S807, it is determined whether all the processing target patches have been processed. If all the target patches have been processed, the first abnormality determination process ends; otherwise, the process advances to S802.

In step S702, the second abnormality determination unit 602 performs a second abnormality determination on the patch for which the first abnormality determination was performed based on the first determination data.

FIG. 9 is a flowchart of the second abnormality determination. All steps are performed by the second abnormality determination unit 602.

In step S901, first determination data is read and second determination data is initialized. The second determination data stores the results of determining whether all the processing target patches are normal (="0") or abnormal (="1") and the pixel positions of all the processing target patches.

Step S902 is the same process as step S502.

In step S903, CNT is compared with a preset threshold th2. The threshold value th2 is set as x% of the number of pixels in the patch to be processed. For example, when x=10, th2=3 (rounding down the decimal point). If CNT is equal to or greater than the threshold value 2, the process advances to S904; otherwise, the process advances to S905.

In step S904, it is determined that the patch I' to be processed is abnormal, and the pixel position of the patch I' to be processed and the determination result "1" are output as determination data.

In step S905, the patch to be processed I' is determined to be normal, and the pixel position of the patch to be processed I' and the determination result "0" are output as determination data.

In step S906, it is determined whether all the processing target patches have been processed. If all the target patches have been processed, the second abnormality determination process ends; otherwise, the process advances to S902.

In step S703, the second correction unit 603 performs a second correction on the patch before the first correction. The second correction is a process different from the first correction, and in this embodiment, an average value filter is applied to the patch before the first correction using only the pixels counted up in the first abnormality determination. Apply.

As described above, by performing the process of determining whether or not a corrected patch is abnormal in two stages, it is possible to more accurately correct the abnormal result of the first correction process and obtain a suitable output image.

In addition, for abnormality determination, one patch to be processed is selected from the first patch before correction and the first patch after correction, and the absolute difference value of the pixel value and the threshold value th are calculated for each pixel at the same position in each patch. However, other methods may be used to determine abnormality. For example, the difference between the pixel value of each pixel in the patch to be processed and the square root of the noise amount patch V_i may be compared. Alternatively, the difference between the average value of all pixels in the patch to be processed and each pixel in the patch may be compared.

(Other embodiments)
The present invention provides a system or device with a program that implements one or more functions of each of the above-described embodiments via a network or a storage medium, and one or more processors in the computer of the system or device reads the program. This can also be achieved by executing a process.

It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

The above-described embodiments are merely examples of implementation of the present invention, and the technical scope of the present invention should not be construed as limited by these embodiments.

That is, the present invention can be implemented in various forms without departing from its technical idea or its main features. For example, a part of the quadratic function coefficient calculation section is operated by a calculation start signal given from a higher-level control means, holds the calculation result, and refers to the data held here while an image is being input. It is possible to reduce power consumption by only using

Claims (9)

An image processing device that reduces image noise,
a setting means for setting a plurality of patches in the image; a calculation means for calculating a covariance matrix based on the plurality of patches; and a calculation means for calculating a covariance matrix based on the plurality of patches; a first correction means for correcting;
a second correction unit that performs a second correction different from the first correction on the pixels included in the plurality of patches before the first correction;
determining means for determining whether or not each of the plurality of patches after the first correction has an abnormality;
correction means for correcting the pixels after the first correction, which are included in the patch determined to be abnormal by the determination means, with the result of the second correction;
An image processing apparatus comprising: a generating unit that generates an output image using the pixels after the first correction or the corrected pixels. The determining means calculates the amount of change between the statistic of each of the plurality of patches before the first correction and the statistic of each of the plurality of patches after the first correction, and if the amount of change is less than a predetermined value. 2. The image processing apparatus according to claim 1, wherein a patch after the first correction having a large value is determined to be abnormal. The statistic is at least one of the standard deviation, variance, maximum value pixel, and minimum value pixel of pixels included in the patch,
The image processing apparatus according to claim 2, wherein the amount of change is calculated based on at least one of a difference and a ratio. The determining means calculates a difference between a pixel of each of the plurality of patches before the first correction and a pixel of each of the plurality of patches after the first correction, and if the difference is larger than a predetermined value, A claim characterized in that the pixel has a first determining means for determining that the pixel is abnormal, and based on the determination by the determining means, it is determined whether the patch after the first correction is abnormal. Item 1. The image processing device according to item 1. 5. The image processing apparatus according to claim 2, wherein the predetermined value is a statistic calculated from pixels included in the plurality of patches. 4. The determining means determines that the patch after the first correction has a larger number of pixels determined to be abnormal by the first determining means than a predetermined value is abnormal. The image processing device described in . The second correction means applies a low-pass filter to the pixels included in the plurality of patches before the first correction, or applies a low-pass filter to the pixels included in the plurality of patches before the first correction. The image processing device according to any one of claims 1 to 6, wherein the image processing device outputs a value or an average value of pixels included in the plurality of patches before the first correction. An image processing method for reducing image noise, the method comprising:
a setting step in which the setting means sets a plurality of patches in the image; a calculation step in which the calculation means calculates a covariance matrix based on the plurality of patches; and a first correction means using the covariance matrix. a first correction step of performing a first correction on pixels included in the plurality of patches;
a second correction step in which a second correction means performs a second correction different from the first correction on the pixels included in the plurality of patches before the first correction;
a determining step in which the determining means determines whether or not each of the plurality of patches after the first correction has an abnormality;
a correction step in which the correction means corrects the pixels after the first correction included in the patch determined to be abnormal in the determination step, using the result of the second correction;
An image processing method characterized in that the generating means has a generating step of generating an output image using the pixels after the first correction or the corrected pixels. computer,
An image processing device that reduces image noise,
a setting means for setting a plurality of patches in the image; a calculation means for calculating a covariance matrix based on the plurality of patches; and a calculation means for calculating a covariance matrix based on the plurality of patches; a first correction means for correcting;
a second correction unit that performs a second correction different from the first correction on the pixels included in the plurality of patches before the first correction;
determining means for determining whether or not each of the plurality of patches after the first correction has an abnormality;
correction means for correcting the pixels after the first correction, which are included in the patch determined to be abnormal by the determination means, with the result of the second correction;
A program for functioning as an image processing apparatus, comprising: generating means for generating an output image using the pixels after the first correction or the corrected pixels.

Images