JP7437249B2 - Noise removal device and its program - Google Patents

Description

The present invention relates to a noise removal device for removing noise from moving images and a program thereof.

Conventionally, as a noise removal method for removing noise from moving images, BM3D (Block Matching and A method called 3D Collaborative Filtering is known (for example, see Non-Patent Document 1).

In BM3D, a space defined by a discrete cosine transform base or a discrete wave red transform base for a three-dimensional array of overlapping blocks is used as a noise removal space, and noise reduction processing is performed using a fixed threshold value.

K. Dabov, A, Foi, and V. Katkovnik, “Image denoising by sparse 3-D transform-domain collaborative filtering,” IEEE Transactions on Image processing, vol.16, no.8, pp.2080-2095, 2007

In general, the power of signal components of moving images (particularly natural images) is biased towards the spatial low frequency band side. On the other hand, thermal noise and shot noise in images with a sufficiently large number of pixels have whiteness, so the power of the noise component is almost constant in all spatial frequency bands.

Therefore, the signal-to-noise ratio is high in the spatial low frequency band of moving images, and the signal-to-noise ratio is low in the spatial high frequency band. In conventional methods such as BM3D that do not take into account the signal-to-noise ratio for each spatial frequency band, sufficient noise removal performance may not be obtained for elements in the spatial high-frequency band where the signal-to-noise ratio is low.

Therefore, it is desired to implement noise removal processing that takes into account the difference in signal-to-noise ratio for each spatial frequency band in a moving image.

SUMMARY OF THE INVENTION Therefore, an object of the present invention is to provide a noise removal device and a program therefor that removes noise from a moving image by taking into consideration the difference in signal-to-noise ratio for each spatial frequency band in the moving image.

The noise removal device of the present invention is a noise removal device that removes noise from a moving image, and decomposes a frame image in a moving image into a spatial frequency band of a predetermined decomposition rank, and generates a decomposed element image of each band consisting of band decomposed elements. A spatial frequency band decomposition unit generates a decomposed element image of each decomposed element image of the target frame, using a frame image from which noise is to be removed from a moving image as a target frame, and a frame image to be referred to for searching for similar elements as a reference frame. is divided into blocks, and for each block area of the band decomposition element of the target frame, the band of the reference frame in the same band is calculated based on the similarity specified by a predetermined noise removal mode and controlled by the similarity control amount. The degree of similarity is determined based on a similar element search unit that searches for block areas having similar pixel values from within the decomposition elements, and the search results for similar block areas for all block areas in each band in the target frame. a summing processing unit that processes to add the block area in the band decomposition element of the reference frame that exceeds a threshold value to the block area of the band decomposition element of the target frame; Reconstruction in which a noise-removed frame image is generated by performing discrete wavelet reconstruction of the predetermined decomposition rank on the target frame after addition processing for similar block areas to the block area, and the noise-removed moving image is output to the outside. It is characterized by comprising: and.

Further, in the noise removal device of the present invention, the similar element search unit is characterized in that, as the noise removal mode, a different similarity control amount is set for each band having a different signal-to-noise ratio.

Further, in the noise removal device of the present invention, the similar element search unit sets a different similarity control amount for each spatial frequency band of the predetermined decomposition rank as the noise removal mode.

Further, in the noise removal device of the present invention, the noise removal mode relatively increases the degree of similarity to the predetermined threshold as the spatial frequency becomes higher, and the size of the block region increases regardless of whether the spatial frequency is high or not. and a second mode in which the higher the spatial frequency is, the higher the degree of similarity to the predetermined threshold value is, and the higher the spatial frequency is, the larger the size of the block region is. It is characterized by

Further, the program according to the present invention is configured as a program for causing a computer to function as the noise removal device according to the present invention.

According to the present invention, similar elements in n-th order spatial frequency band decomposition elements are searched within the same band, taking into account differences in signal-to-noise ratio for each spatial frequency band in a moving image, and the searched elements are added together. Since n-th order discrete wavelet reconstruction is performed, the noise removal performance of moving images can be improved compared to conventional BM3D.

FIG. 1 is a block diagram showing a schematic configuration of a noise removal device according to an embodiment of the present invention. 3 is a flowchart showing noise removal processing in a noise removal device according to an embodiment of the present invention. FIG. 3 is an explanatory diagram of spatial frequency band decomposition processing in the noise removal device of one embodiment according to the present invention. FIG. 3 is a diagram showing an example of searching for similar elements in a noise removal device according to an embodiment of the present invention.

Hereinafter, a noise removal device 1 according to an embodiment of the present invention will be described in detail.

(Device configuration)
FIG. 1 is a block diagram showing a schematic configuration of a noise removal device 1 according to an embodiment of the present invention. The noise removal device 1 of this embodiment is a device that performs noise removal by taking into consideration the difference in signal-to-noise ratio for each spatial frequency band in a moving image, and includes a spatial frequency band decomposition unit 11, a similar element search unit 12, It includes a combination processing section 13 and a reconstruction section 14.

The spatial frequency band decomposition unit 11 inputs a frame image I(t) of a moving image at a time position t and a spatial frequency band decomposition rank n, decomposes the input frame image I(t) into an n-th spatial frequency band, and performs spatial frequency band decomposition. It is decomposed into each band of low frequency band L, horizontal high frequency area H, vertical high frequency area V, and diagonal high frequency area D, and respectively spatial low frequency I _L ⁿ (t), horizontal high frequency I _H ⁿ (t), and vertical high frequency A decomposed element image of each band consisting of the band decomposed elements of I _V ⁿ (t) and diagonal high frequency I _D ⁿ (t) is generated and output to the similar element search unit 12 . Note that the spatial frequency band decomposition rank n can be arbitrarily specified from the outside.

First, the similar element search unit 12 uses a frame image from which noise is to be removed as a target frame, and refers to a frame image that is the same as the target frame or a predetermined different frame image (preferably an adjacent frame image) in order to search for similar elements. A frame image to be used is predetermined as a reference frame. Then, the similar element search unit 12 extracts the band decomposition elements of the spatial low frequency I _L ⁿ (t), the horizontal high frequency I _H ⁿ (t), the vertical high frequency I _V ⁿ (t), and the diagonal high frequency I _D ⁿ (t). The decomposed element images of each band consisting of are input, and a search for similar elements (more precisely, a search for similar block regions) is performed.

More specifically, the similar element search unit 12 divides into blocks the band decomposition element C _m ⁿ of each decomposition element image of the target frame, which is made up of any band decomposition element that has been subjected to band decomposition. Then _, the similar element search unit 12 calculates the degree of ^similarity s(i, j, t), a block region having similar pixel values is searched from within the band decomposition element R _m ⁿ of the reference frame in the same band.

Then, the similar element search unit 12 searches for similar block areas in all block areas of each band, and outputs the search results to the addition processing unit 13. Here, although the details will be described later, different similarity control amounts α, b _x , b _y are set in the similar element search unit 12 for each spatial frequency band of the spatial frequency band decomposition rank n by the noise removal mode d. Furthermore, it is preferable that at least two types of noise removal modes d=1 and 2 can be selected.

Based on the search results for similar block regions for all block regions of each band in the target frame, the addition processing unit 13 selects the relevant reference whose similarity s(i, j, t) exceeds a predetermined threshold _th . A process (addition process) is performed to add the block area in the band decomposition element R _m ⁿ of the frame to the block area of the band decomposition element C _m ⁿ of the target frame, and the result is output to the reconstruction unit 14 .

The reconstruction unit 14 generates a frame image RI(t) from which noise has been removed by performing n-th order discrete wavelet reconstruction on the target frame after addition processing of similar block regions for all block regions of each band in the target frame. is generated, and the moving image after noise removal is output to the outside. Note that the spatial frequency band decomposition rank n specified externally for the reconstruction unit 14 is the same value as that for the spatial frequency band decomposition unit 11.

(Noise removal processing)
With reference to FIG. 2, the noise removal process in the noise removal device 1 according to one embodiment of the present invention will be described in more detail. FIG. 2 is a flowchart showing the noise removal process in the noise removal device 1 of this embodiment.

First, the noise removal device 1 inputs a frame image I(t) of a moving image at a time position t and a spatial frequency band decomposition rank n through the spatial frequency band decomposition unit 11, and inputs the n of the input frame image I(t). The spatial frequency band is decomposed into spatial low frequency band L, horizontal high frequency area H, vertical high frequency area V, and diagonal high frequency area D. Spatial low frequency I _L ⁿ (t) and horizontal high frequency I are respectively A decomposed element image of each band consisting of band decomposed elements of _H ⁿ (t), vertical high frequency I _V ⁿ (t), and diagonal high frequency I _D ⁿ (t) is generated (step S1).

FIG. 3 is an explanatory diagram of spatial frequency band decomposition processing in the noise removal device 1 of this embodiment. By performing n-th order spatial frequency band decomposition of frame image I(t), decomposed element images of each band of spatial low frequency band L, horizontal high frequency area H, vertical high frequency area V, and diagonal high frequency area D are obtained. , the band decomposition elements of spatial low frequency I _L ⁿ (t), horizontal high frequency I _H ⁿ (t), vertical high frequency I _V ⁿ (t), and diagonal high frequency I _D ⁿ (t) are expressed as pixel values of each band. expressed.

Next, the noise removal device 1 takes the frame image to be noise removed as the target frame, and uses the similar element search unit 12 to search for similar elements that are the same as the target frame or a predetermined different frame image (preferably an adjacent frame image). A frame image to be referred to for the search is predetermined as a reference frame. Then, the decomposition elements of each band are composed of the spatial low frequency I _L ⁿ (t), the horizontal high frequency I _H ⁿ (t), the vertical high frequency I _V ⁿ (t), and the diagonal high frequency I _D ⁿ (t). Input an image and search for similar block areas.

More specifically, the similar element search unit 12 divides the decomposed element image of the band decomposed elements C _m ⁿ of the target frame consisting of any of the band decomposed elements into blocks, and performs the band decomposition of the target frame. For each block region of the element C _m ⁿ , the reference frame in the same band is calculated based on the similarity s(i, j, t) specified by the predetermined noise removal mode d and controlled by the similarity control amount. A block region having similar pixel values is searched within the band decomposition element R _m ⁿ . Then, the similar element search unit 12 searches for similar block areas in all block areas of each band, and outputs the search results to the addition processing unit 13 (step S2).

The degree of similarity s(i, j, t) used in the search for similar block regions in the similar element search unit 12 is defined as in equation (1).

In equation (1), the band decomposition element C _m ⁿ in the target frame and the band decomposition element R _m ⁿ in the reference frame are spatial low frequency I _L ⁿ (t) and horizontal high frequency in the target frame and reference frame, respectively. I _H ⁿ (t), vertical high frequency I _V ⁿ (t), and diagonal high frequency I _D ⁿ (t), where t may be the time position of the same frame image or a predetermined time position. The time positions of different frame images (preferably adjacent frame images) may be used. m represents any one of the spatial low frequency band L, the horizontal high frequency area H, the vertical high frequency area V, and the diagonal high frequency area D, and m and n indicated by C _m ⁿ and R _m ⁿ are m and n shown are the same value. In other words, similar block regions may be searched within the same frame image or between different frame images, but are performed at the same spatial frequency band decomposition rank and in the same spatial frequency band direction.

i is a pixel position in C _m ⁿ , j is a pixel position in R _m ⁿ , and the similarity control amount to be set is a positive coefficient α that controls the similarity weight in equation (1) and “i and "b _x , b _y " in "Calculation term for the sum of squared errors of all pixel values in a block area of horizontal ± b _x and vertical ± b _y pixels from the pixel position of j". Note that the smaller the value of the coefficient α, the higher the degree of similarity relative to the predetermined threshold _th .

FIG. 4 is a diagram showing an example of searching for similar block regions in the noise removal device 1 of this embodiment. FIG. 4 shows the horizontal high frequency band of the target frame and the horizontal high frequency band of the reference frame during second-order spatial frequency band decomposition. The block area of the target frame and the block area of the reference frame are expressed as areas of ±b _x and ±b _y pixels from the pixel positions i and j. Then, the similar element search unit 12 shifts the block area of the horizontal high frequency band R _H ² (t) of the reference frame centered on j within that R _H ² (t), and searches for a certain block area of the target frame. Search and select the one most similar to.

Further, as explained below, different similarity control amounts α, b _x , b _y are set in the similar element search unit 12 for each spatial frequency band of the spatial frequency band decomposition rank n by the noise removal mode d. Ru. Note that it is preferable that the similar element search unit 12 be able to select at least two types of noise removal modes d=1 and 2.

[Noise removal mode d=1 (first mode)]
The noise removal mode d=1 (first mode) is a mode in which the noise component is attenuated while maintaining the signal component as much as possible. The higher the spatial frequency, the lower the signal-to-noise ratio, so by decreasing α as the spatial frequency increases, the signal-to-noise ratio is low and the similarity is maintained high even if the error amount when searching for similar block regions is large. Do it like this. However, in order to retain signal components with high spatial frequencies, the sizes b _x and b _y of the block regions are kept constant even if the spatial frequencies are high. Table 1 shows an example of α, b _x , and b _y in (first mode).

[Noise removal mode d=2 (second mode)]
The noise removal mode d=2 (second mode) is a mode in which the signal component is degraded but the noise component is greatly attenuated. The higher the spatial frequency, the lower the signal-to-noise ratio, so by decreasing α as the spatial frequency increases, the signal-to-noise ratio is low and the degree of similarity is maintained high even if the error amount when searching for similar block regions is large. However, in order to improve the noise removal performance even if the signal-to-noise ratio is low, the higher the spatial frequency is, the larger the block area sizes b _x and b _y are. do. Table 2 shows an example of α, b _x , and b _y in (second mode).

In this way, the noise removal device 1 takes into account the difference in the signal-to-noise ratio for each spatial frequency band in a moving image, and instructs the similar element search unit 12 to , different similarity control amounts α, b _x , b _y can be specified from the outside, and it is preferable that at least two types of noise removal modes d=1 and 2 can be selected.

Next, the noise removal device 1 uses the addition processing unit 13 to calculate the similarity degree s(i, j, t) based on the search results for similar block regions for all block regions in each band in the target frame. The block area in the band decomposition element R _m ⁿ of the reference frame that exceeds a predetermined threshold th _h is added to the block area of the band decomposition element C _m ⁿ of the target frame (addition process) (step S3 ). Note that in this addition process, when the similarity s (i, j, t) does not exceed the predetermined threshold _th , the block in the band decomposition element R _m ⁿ of the reference frame searched as the most similar is Even if it is a region, it is processed so as not to be added to the block region in the target frame.

Finally, the noise removal device 1 causes the reconstruction unit 14 to perform n-th order discrete wavelet reconstruction on the target frame after the addition processing of similar block regions to all block regions of each band in the target frame. , generates a noise-removed frame image RI(t), and outputs the noise-removed moving image to the outside (step S4).

As described above, the noise removal device 1 according to the present invention performs spatial frequency band decomposition on a frame image of a moving image, and sets a different similarity control amount for each spatial frequency band of the spatial frequency band decomposition rank n. Noise is removed by searching for similar elements, adding up the searched elements, and then performing reconstruction. The similarity control amount is set in consideration of the signal-to-noise ratio for each spatial frequency band. As a result, noise removal performance for moving images can be improved compared to conventional BM3D.

The noise removal device 1 in the above embodiment can be configured by a computer, and a program for making each processing section of the noise removal device 1 function can be suitably used. Specifically, the control unit for controlling each processing unit of the noise removal device 1 can be configured with a central processing unit (CPU) in a computer, and the programs necessary to operate each processing unit can be configured. The storage section for storing appropriate information can be configured with at least one memory. That is, by causing such a computer to execute the program using the CPU, the functions of each processing section of the noise removal device 1 can be realized. Furthermore, a program for realizing the functions of each processing section of the noise removal device 1 can be stored in a predetermined area of the storage section (memory) described above. Such a storage unit can be configured with a RAM or ROM inside the device, or can also be configured with an external storage device (for example, a hard disk). Moreover, such a program can be configured as a part of software (stored in ROM or external storage device) on the OS used in the computer. Further, a program for causing such a computer to function as each processing section of the noise removal device 1 can be recorded on a computer-readable recording medium. Further, each processing unit of the noise removal device 1 can be configured as a part of hardware or software, and each can be realized by combining them.

Although the present invention has been described above with reference to specific embodiments, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the technical idea thereof. For example, in the above-described embodiment, an example was mainly explained in which different similarity control amounts are set for each spatial frequency band of spatial frequency band decomposition rank n in Tables 1 and 2. Different similarity control amounts are set for each band of spatial low frequency I _L ⁿ (t), horizontal high frequency I _H ⁿ (t), vertical high frequency I _V ⁿ (t), and diagonal high frequency I _D ⁿ (t). In other words, it is effective if the similarity control amount is set to be different for each band with a different signal-to-noise ratio. Therefore, the noise removal device 1 according to the present invention is not limited to the above-described embodiments, but is limited only by the claims.

According to the present invention, noise removal performance for moving images can be improved compared to conventional BM3D, and therefore it is useful for applications that transmit moving images.

1 Noise removal device 11 Spatial frequency band decomposition unit 12 Similar element search unit 13 Addition processing unit 14 Reconstruction unit

Claims (5)

A noise removal device that removes noise from moving images,
a spatial frequency band decomposition unit that decomposes a frame image in a moving image into a spatial frequency band of a predetermined decomposition rank and generates a decomposed element image of each band consisting of band decomposed elements;
The decomposed element images of each decomposed element image of the target frame are divided into blocks, and the decomposed element images of the decomposed element images of the target frame are divided into blocks, with the frame image from which noise is to be removed in the moving image as the target frame, and the frame image to be referred to for searching for similar elements as the reference frame. For each block area of the band decomposition element, similar pixel values are extracted from within the band decomposition element of the reference frame in the same band based on the similarity specified by a predetermined noise removal mode and controlled by the similarity control amount. a similar element search unit that searches for a block area having
Based on the search results for similar block regions for all block regions of each band in the target frame, block regions in the band decomposition elements of the reference frame for which the degree of similarity exceeds a predetermined threshold are selected from the target frame. an addition processing unit that performs processing to add the block area of the band decomposition element;
Generate a noise-removed frame image by performing discrete wavelet reconstruction of the predetermined decomposition rank on the object frame after addition processing of similar block regions for all block regions of each band in the object frame, and perform noise removal. a reconstruction unit that outputs the subsequent moving image to the outside;
A noise removal device characterized by comprising: 2. The noise removal apparatus according to claim 1, wherein the similar element search unit sets a different similarity control amount for each band having a different signal-to-noise ratio as the noise removal mode. 3. The noise removal device according to claim 1, wherein the similar element search unit sets a different similarity control amount for each spatial frequency band of the predetermined decomposition rank as the noise removal mode. The noise removal mode is
a first mode in which the higher the spatial frequency, the higher the degree of similarity to the predetermined threshold value, and the size of the block region is constant regardless of whether the spatial frequency is high;
a second mode in which the higher the spatial frequency, the higher the similarity to the predetermined threshold value, and the higher the spatial frequency, the larger the size of the block area;
The noise removal device according to any one of claims 1 to 3, characterized in that it includes: A program for causing a computer to function as the noise removal device according to any one of claims 1 to 4.