JP7710482B2 - Image processing device and image processing method - Google Patents

Description

Embodiments of the present invention relate to an image processing device and an image processing method.

Conventionally, there is known technology that uses noise reduction processing to remove noise contained in video input, making it possible to view noise-free images.

JP 2008-35439 A

However, in the case of film content, producers may intentionally add film grain (the texture of old film), a type of noise, to the image, and for such images it is desirable not to apply too much noise reduction.

The problem that this invention aims to solve is to provide an image processing device and an image processing method that can output an image that preserves the original character of a video containing film grain without degrading the characteristics of the film grain.

An image processing device according to an embodiment performs image correction on an input image, and includes a noise amount detection unit, a film grain detection unit, a control unit, and an image correction unit. The noise amount detection unit determines the amount of noise in the image. The film grain detection unit determines whether the image contains film grain. The control unit determines an amount of correction for image processing on the image based on the determination results of the noise amount detection unit and the film grain detection unit. The image correction unit performs image processing based on the amount of correction determined by the control unit. The film grain detection unit detects the size and amount of film grain particles. The control unit changes the amount of correction for the image processing depending on the degree of size and amount of film grain particles.

FIG. 1 is a diagram illustrating an example of a hardware configuration of a television device according to an embodiment. FIG. 2 is a block diagram showing an example of a circuit configuration of the image processing circuit. FIG. 3 is a diagram showing how the film grain detection unit determines the presence or absence and the strength of film grain. FIG. 4 is a diagram showing an example of determining the strength of film grain. FIG. 5 is a flowchart showing the flow of processing for determining the presence or absence, and the strength, of film grain. FIG. 6 is a diagram showing the process in the noise reduction processing section. FIG. 7 is a diagram showing the process in the sharpening processing section. FIG. 8 is a diagram showing the relationship between the offset value and the strength of the film grain.

FIG. 1 is a diagram showing an example of the hardware configuration of a television device 10 according to an embodiment. The television device 10 is an image processing device that performs image processing on an image input by broadcasting or the like. The television device 10 then displays the image that has undergone the image processing.

As shown in FIG. 1, the television device 10 includes an antenna 101, input terminals 102a-102c, a tuner 103, a demodulator 104, a demultiplexer 105, an A/D (analog/digital) converter 106, a selector 107, a signal processing unit 108, a speaker 109, a display panel 110, an operation unit 111, a light receiving unit 112, an IP communication unit 113, a CPU (Central Processing Unit) 114, a memory 115, storage 116, a microphone 117, and an audio I/F (interface) 118.

The antenna 101 receives a digital broadcast signal and supplies the received broadcast signal to the tuner 103 via the input terminal 102a.

The tuner 103 selects a broadcast signal of a desired channel from the broadcast signals supplied from the antenna 101, and supplies the selected broadcast signal to the demodulator 104.

The demodulator 104 demodulates the broadcast signal supplied from the tuner 103 and supplies the demodulated broadcast signal to the demultiplexer 105.

The demultiplexer 105 separates the broadcast signal supplied from the demodulator 104 to generate a video signal and an audio signal, and supplies the generated video signal and audio signal to the selector 107.

The selector 107 selects one of the multiple signals supplied from the demultiplexer 105, the A/D converter 106, and the input terminal 102c, and supplies the selected signal to the signal processing unit 108.

The signal processing unit 108 performs a predetermined signal processing on the video signal supplied from the selector 107, and supplies the processed video signal to the display panel 110. The signal processing unit 108 also performs a predetermined signal processing on the audio signal supplied from the selector 107, and supplies the processed audio signal to the speaker 109. The signal processing unit 108 has an image processing circuit 1000 shown in FIG. 2.

The speaker 109 outputs voice or various sounds based on the audio signal supplied from the signal processing unit 108. The speaker 109 also changes the volume of the voice or various sounds it outputs based on the control of the CPU 114.

The display panel 110, which serves as a display unit, displays still and moving images, other images, text information, etc., based on the video signal supplied from the signal processing unit 108 or control by the CPU 114.

The input terminal 102b accepts analog signals such as video and audio signals input from the outside. The input terminal 102c accepts digital signals such as video and audio signals input from the outside. For example, the input terminal 102c can input digital signals from a recorder or the like that is equipped with a drive device that drives a recording and playback recording medium such as a Blu-ray (registered trademark) Disc (BD) to record and play back.

The A/D converter 106 performs A/D conversion on the analog signal supplied from the input terminal 102b to generate a digital signal, which is then supplied to the selector 107.

The operation unit 111 accepts operation input from the user.

The light receiving unit 112 receives infrared light from the remote controller 119.

The IP communication unit 113 is a communication interface for performing IP (Internet Protocol) communication via the network 40.

The CPU 114 controls the entire television device 10.

Memory 115 includes a ROM that stores various computer programs executed by CPU 114, and a RAM that provides a working area for CPU 114.

The storage 116 is a hard disk drive (HDD) or a solid state drive (SSD), etc. The storage 116 records, for example, the signal selected by the selector 107 as recording data.

The microphone 117, which serves as an audio input unit, captures the voice spoken by the user and sends it to the audio I/F 118.

The audio I/F 118 converts the audio picked up by the microphone 117 from analog to digital and sends it to the CPU 114 as an audio signal.

Next, we will explain the image processing circuit 1000 that the signal processing unit 108 has.

Figure 2 is a block diagram showing an example of the circuit configuration of the image processing circuit 1000. As shown in Figure 2, the image processing circuit 1000 includes a noise amount detection unit 1001, a film grain detection unit 1002, a control unit 1003, and a high image quality processing unit 1004 that functions as an image correction unit.

The noise amount detection unit 1001 determines the degree of noise in the input image input to the image processing circuit 1000.

The film grain detection unit 1002 determines the presence and strength of film grain, a type of noise, in the input image input to the image processing circuit 1000.

The high image quality processing unit 1004 performs high image quality processing on the input image input to the image processing circuit 1000. The high image quality processing unit 1004 includes a noise reduction processing unit 1005 that functions as a noise removal unit, and a sharpening processing unit 1006 that functions as a sharpening unit.

When a large amount of noise is detected by the noise amount detection unit 1001, the control unit 1003 instructs the noise reduction processing unit 1005 of the high image quality processing unit 1004 to increase noise reduction.

Here, we explain how the film grain detection unit 1002 determines the presence or absence and strength of film grain.

Figure 3 is a diagram showing how the film grain detection unit 1002 determines the presence or absence and strength of film grain. As shown in Figure 3, the film grain detection unit 1002 analyzes the input image input to the image processing circuit 1000 and determines the degree of film grain contained therein. The film grain detection unit 1002 uses machine learning to classify images into those that contain film grain and those that do not. More specifically, the film grain detection unit 1002 uses a CNN (convolutional neural network) to extract features such as film grain from the image, and then fully connects them to create an activation function.

The film grain detection unit 1002 outputs the strength of the film grain F(x, y, z) according to the output value as shown in FIG. 3 detected based on the result of judging the presence or absence and the strength of the film grain, for example, as expressed by the following formula.
F (x, y, z) = αx + βy + γz

The film grain may be measured by any of the following methods:
- Signal to noise ratio - Film grain size (average, maximum)
・Difference from surrounding pixels ・Difference between frames ・Frame rate of input video

In this embodiment, the film grain detection unit 1002 determines the strength of the film grain based on the size and amount of film grain particles. FIG. 4 is a diagram showing an example of determining the strength of the film grain. As shown in FIGS. 4(a) and 4(b), the film grain detection unit 1002 determines that the strength of the film grain is strong when the proportion of the film grain per unit area is large (when the amount of film grain per unit area is large, when the amount of film grain per unit area is not so large but the grain size is large, etc.). On the other hand, as shown in FIG. 4(c), the film grain detection unit 1002 determines that the strength of the film grain is weak when the proportion of the film grain per unit area is small (when the amount of film grain per unit area is small, when the amount of film grain per unit area is not so large and the grain size is small, etc.).

Figure 5 is a flowchart showing the process flow for determining the presence or absence and strength of film grain in the film grain detection unit 1002. As shown in Figure 5, the film grain detection unit 1002 determines the presence or absence of film grain (step S1).

If it is determined that film grain is present (Yes in step S1), the film grain detection unit 1002 measures the amount of film grain noise (step S2) and determines the strength of the film grain (step S3).

If it is determined that there is no film grain (No in step S1), the film grain detection unit 1002 determines that there is no film grain (step S4).

Next, we will explain the processing performed by the noise reduction processing unit 1005 when detecting film grain.

Here, FIG. 6 is a diagram showing the processing in the noise reduction processing unit 1005. As described above, generally, when a large amount of noise is detected by the noise amount detection unit 1001, the control unit 1003 controls the noise reduction processing unit 1005 to strengthen noise reduction. However, when film grain is detected, the control unit 1003 sets an offset value as shown in FIG. 6, and controls the noise reduction processing unit 1005 to reduce the amount of noise reduction correction. This is done in order to output an image that retains the flavor of film grain.

Next, we will explain the processing performed by the sharpening processing unit 1006 when detecting film grain.

Here, FIG. 7 is a diagram showing the processing in the sharpening processing unit 1006. When film grain is detected by the film grain detection unit 1002, the control unit 1003 sets an offset value to change the degree of processing in the sharpening processing unit 1006. The control unit 1003 also determines the amount of change (offset value) in the processing in the sharpening processing unit 1006 depending on the strength of the film grain detected by the film grain detection unit 1002. That is, as shown in FIG. 7, when film grain is detected, the control unit 1003 controls the sharpening processing unit 1006 to reduce the amount of sharpening correction. This is done in order to avoid emphasizing the remaining noise by the amount of weakened noise reduction processing.

Here, FIG. 8 is a diagram showing the relationship between the offset value and the strength of the film grain. As shown in FIG. 8, the control unit 1003 controls the offset value to be larger as the strength of the film grain increases, and the amount of noise reduction correction and the amount of sharpening correction to be smaller.

In this way, according to the image processing device and image processing method of this embodiment, by performing image processing based on the amount of image processing correction for the image determined based on the results of judging the presence or absence and strength of film grain, it is possible to output an image that preserves the original flavor of video that contains film grain without degrading the characteristics of the film grain. Furthermore, for video that does not contain film grain, it is possible to output an image with sufficient noise removal and sharpened details.

Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the gist of the invention. This embodiment and its modifications are included in the scope and gist of the invention, and are included in the scope of the invention and its equivalents described in the claims.

REFERENCE SIGNS LIST 10 Image processing device 1001 Noise amount detection unit 1002 Film grain detection unit 1003 Control unit 1004 Image correction unit 1005 Noise removal unit 1006 Sharpening unit

Claims (3)

An image processing device that corrects an input image,
a noise amount detection unit for determining the amount of noise in the image;
a film grain detection unit that determines whether the image contains film grain;
a control unit that determines an amount of correction for image processing for the image based on the determination results of the noise amount detection unit and the film grain detection unit;
an image correction unit that performs image processing based on the correction amount determined by the control unit;
Equipped with
the film grain detection unit detects the size and amount of film grain particles;
the control unit varies an amount of correction of the image processing in accordance with a degree of size and amount of the film grain particles.
13. An image processing device comprising: The image correction unit includes:
a noise removal unit that removes noise from the image;
a sharpening unit for enhancing contours or textures of the image;
Equipped with
changing the amount of the noise to be removed or the degree of the emphasis in accordance with the correction amount determined by the control unit;
2. The image processing device according to claim 1 , 1. An image processing method in an image processing device that corrects an input image, comprising:
a noise amount detection step of calculating the amount of noise in the image;
a film grain detection step for determining whether the image contains film grain;
a control step of determining an amount of correction of image processing for the image based on the results of the noise amount detection step and the film grain detection step;
an image correction step of performing image processing based on the correction amount determined in the control step;
Including,
The film grain detection step detects the size and amount of film grain particles;
the control step varies an amount of correction of the image processing in accordance with a degree of size and amount of the film grain particles;
13. An image processing method comprising: