JP7748503B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program.

In the surveillance market, where extremely low-light environments require improved visibility of target subjects, very high gain is applied when capturing images with cameras. While applying high gain brightens the captured image, it also increases noise. However, for surveillance applications, subject visibility is important, even at the expense of image quality for viewing purposes.

Noise reduction (hereinafter referred to as NR) has been commonly used to reduce noise contained in captured images. NR can be performed inside the camera, or in some cases, images from the camera are input and processed by an external image processing device. In recent years, noise reduction processing based on deep learning artificial intelligence technology (hereinafter referred to as DLNR) has also come into use. DLNR has been shown to be more effective than conventional NR.

By the way, various image processing functions for captured images are made up of a combination of various image processing methods, and the parameters used for each image processing method vary. In order to obtain the optimal image processing effect for capturing various scenes, we have responded by storing image processing functions and parameter combinations for each scene.

In Patent Document 1, a predicted image is generated from two interpolated images using bidirectional gradient change processing based on gradient change. This makes it possible for the viewer to see the image transition from a first state to a second state.

The technology in Patent Document 2 gradually changes parameters from group A to group B in order to adjust image quality so that there are fewer sudden fluctuations in image quality and less sense of discomfort when switching scenes. Patent Document 2 lists brightness, sharpness, hue, saturation, etc. as examples of image processing functions and parameters.

Japanese Patent Application Laid-Open No. 2020-109919 Japanese Patent Application Laid-Open No. 2007-288587

However, when switching parameters in DLNR, shocks can occur in the image. In Patent Document 1, when generating an interpolated image from two images, an image that differs from the actual image is predicted and generated, and shock can occur in the image if the prediction differs from the actual image. Furthermore, in Patent Document 2, simply applying interpolation from pre-transition A to post-transition B can result in the image transitioning via parameters that make the image look unnatural. There was no disclosure of parameters that are difficult to change in stages, such as those in trained models used in DLNR.

The present invention aims to provide high-quality images with a smooth NR effect by reducing the shock that occurs when switching parameters of a trained model such as DLNR or intensity parameters for noise reduction.

To solve this problem, for example, an image processing device of the present invention comprises the following configuration: image processing means for performing image processing on an acquired input image using a trained model having first parameters, which are network parameters of a neural network obtained by training noise characteristics of the image;
a noise reduction means for reducing the noise in the image-processed input image using a second parameter, which is an intensity parameter for emphasizing and reducing the noise ;
a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter;
a setting means for changing and setting the parameter to be changed across a plurality of input images acquired sequentially, when changing any of the first parameter, the second parameter, and the third parameter, based on a control amount for bringing the parameter closer to a target value selected by a user or preset reference data;
Equipped with.

This invention reduces the shock to the image when switching parameters of a trained model that performs image processing such as DLNR or parameters such as intensity parameters for noise reduction, and applies a smooth NR effect to provide high-quality images.

FIG. 1 is a block diagram showing the configuration of an image processing apparatus. 4 is a flowchart of image processing executed by the image processing device. 10 is a flowchart of a subroutine of the initialization process of S100. An explanatory diagram of a neural network that processes an input image. 10 is a flowchart of a subroutine for setting image processing in step S120. 10 is a flowchart of a subroutine of noise estimation processing in S130. FIG. 10 is a diagram showing an example of flat area extraction. Examples of noise dispersion characteristics for different camera settings. 10 is a flowchart of a subroutine for image processing change control in S140. 10 is a flowchart of a subroutine of the NR process of S150. 1 is a diagram showing images to be subjected to noise reduction processing arranged in the order of processing. 10 is a flowchart of a subroutine of image output processing in S160. 10 is a flowchart of a subroutine including NN redundancy switching processing in the image processing setting of S120. 10 is a flowchart executed by the image processing apparatus according to the second embodiment. 10 is a flowchart of a subroutine for image processing change control in S240. 10 shows an example of a reference table of setting values to be referred to as the next setting values in S2417. FIG. 10 is a block diagram showing the configuration of an image processing apparatus according to a third embodiment. 10 is a flowchart executed by the image processing apparatus according to the third embodiment. 10 is a flowchart of a subroutine of a camera parameter acquisition process in S320. 10 is a flowchart of a subroutine for setting image processing in step S330.

The following describes the embodiments in detail with reference to the attached drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the attached drawings, the same reference numbers are used to designate identical or similar components, and redundant explanations will be omitted.

This embodiment will be described below. In this embodiment, an image processing device is used as an example, which estimates noise from an input image from a connected camera, infers a desired output image using a neural network (NN), and outputs the inferred image. Images include video, still images, videos, and images of one frame of a video. In NN training, multiple student images and multiple corresponding teacher images are prepared. Next, training is performed to bring the feature distribution of the student images closer to the feature distribution of the teacher images, and network parameters such as weights and biases are optimized. This enables accurate inference even for untrained input images. By retaining trained parameters obtained by performing multiple training sessions according to the camera's characteristics, it is possible to perform inference on the input image and obtain an inferred image with reduced noise.

First Embodiment
1 is a block diagram showing the configuration of an image processing device 100. Note that, although an image processing device will be described in this embodiment, the processing described below can also be performed in a camera with interchangeable lenses, such as a video camera with interchangeable lenses, a single-lens reflex camera, or a mirrorless single-lens camera. For example, the processing described below can also be performed in a control device within the camera 200 shown in FIG. 1.

In FIG. 1, camera 200 captures a light beam incident from the subject field to generate an image. Camera 200 is equipped with or has built-in lenses (not shown), and is equipped with a group of zoom lenses, a group of focus lenses, an iris mechanism, etc. Camera 200 can change the exposure accumulation time, and an auto-exposure function can apply gain to the captured image when capturing in a dark place. Camera 200 sends the captured image to image processing device 100. Image processing device 100 acquires the image via image input unit 110.

The image processing device 100 comprises an image input unit 110, a bus 120, a CPU 130, a memory 140, an operation input unit 150, an image processing unit 160, and an image output unit 170. The image input unit 110, the CPU 130, the memory 140, the operation input unit 150, the image processing unit 160, and the image output unit 170 are connected via the bus 120 so that they can send and receive data with each other. The CPU 130 or the image processing unit 160 executes programs to perform various processes, such as the image processing described below, executed by the image processing device 100, and realize various functions. For example, the CPU 130 and the image processing unit 160 execute programs to realize functions such as a trained model, noise reduction means, synthesis means, setting means, and image synthesis means.

The image input unit 110 stores the input image acquired from the camera 200 in the memory 140 or the storage unit 180 via the bus 120.

The operation input unit 150 acquires operation signals input from a controller 300 external to the image processing device 100. The controller 300 includes switches, a keyboard, a mouse, a touch panel, etc. The user inputs operation signals to the image processing device 100 via the controller 300.

The CPU 130 executes various processes, such as image processing, based on the programs and parameters stored in the storage unit 180. The CPU 130 executes various processes based on operation signals acquired by the operation input unit 150. For example, the CPU 130 executes setting processes required for image processing executed by the image processing unit 160. In addition to the CPU 130, the image processing device 100 may have other processors, such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or QPU (Quantum Processing Unit).

Memory 140 functions as a work area when CPU 130 executes a program. Memory 140 also temporarily stores parameters and other data required when CPU 130 executes a program. Memory 140 may be RAM (Random Access Memory).

The image processing unit 160 reads images stored in the memory 140 or the storage unit 180 and performs various image-related processes, such as noise estimation processing, image signal synthesis processing, NR processing, and UI image generation processing for display on a user interface (hereinafter referred to as UI), which will be described later. The image processing unit 160 stores the images that have undergone various processes in the memory 140 or the storage unit 180. The image processing unit 160 may include a processor such as a GPU (Graphics Processing Unit). The image processing unit 160 may also be implemented as part of the functions of the CPU 130 that executes programs.

The image output unit 170 outputs images processed by the image processing unit 160 and stored in the memory 140 or the storage unit 180 to the outside of the image processing device 100. The image output unit 170 outputs an image signal for output from an HDMI (registered trademark) (High Definition Multimedia Interface) terminal or an SDI (Serial Digital Interface) terminal provided on the image processing device 100.

The storage unit 180 stores programs executed by the CPU 130, parameters required for the programs, and the like. The storage unit 180 stores images acquired from the camera 200 and images processed by the image processing unit 160. The storage unit 180 has non-volatile storage devices such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), and a ROM (Read Only Memory).

The monitor 400 receives and displays the image output by the image processing device 100. Users such as the photographer and viewers can view the image captured by the camera 200, the menu screen of the image processing device 100, and the image after image processing through the monitor 400. The monitor 400 may be, for example, a liquid crystal display device or an organic EL (Electro Luminescence) display device.

Note that while Figure 1 shows each function as a separate component, these may be implemented using hardware such as one or more ASICs (Application Specific Integrated Circuits), programmable logic arrays (PLAs), or FPGAs (Field Programmable Gate Arrays). Furthermore, each of the above functions may be implemented by a programmable processor such as a CPU or MPU executing software.

Next, the image processing performed by the CPU 130 in the image processing device 100 will be described using Figure 2. Figure 2 is a flowchart of the image processing performed by the image processing device 100. When the image processing device 100 is powered on, the CPU 130 loads a computer program stored in the storage unit 180 and executes the processing in order from S100 in Figure 2. Note that once the CPU 130 has executed the processing up to S160 in Figure 2, it will again execute the loop processing from S110. The CPU 130 and image processing unit 160 execute the processing from S110 to S160 for each frame or every few frames of the input image sequentially acquired from the camera 200. Note that either the CPU 130 or the image processing unit 160 executes each process, but the entity that executes each process may be changed as appropriate.

In S100 of Figure 2, the CPU 130 performs initialization processing of the image processing device 100. Figure 3 shows a flowchart of the subroutine for the initialization processing of S100.

In S101 of Figure 3, the CPU 130 executes image input/output initialization processing. The CPU 130 initializes the image input unit 110 and the image output unit 170 so that the image processing device 100 can execute image input/output processing.

In S102, the CPU 130 performs initialization processing of the image processing function. The CPU 130 initializes the image processing unit 160 so that the image processing unit 160 can execute image processing. For example, the CPU 130 performs initial settings to perform luminance signal synthesis processing and color signal synthesis processing of the image after NR processing and the input image in the image synthesis processing described below. For example, when the image processing program is started, the CPU 130 performs initialization processing with the input image at an addition ratio of 100%. The addition ratio is the ratio used when synthesizing the luminance signal or color signal with the inference image.

In S103, the CPU 130 performs NR initialization processing. The CPU 130 loads trained NN parameters that have been pre-trained in advance for the NR processing performed by the image processing unit 160, and applies them to the trained model of the image processing unit 160. This embodiment assumes various models of cameras 200 connected to the image processing device 100 and setting states including the image quality of the cameras 200. Information about the various models of cameras 200 and the image quality settings of the cameras 200 is an example of information about the camera 200. The memory 140 or storage unit 180 holds trained NN parameters optimized for each model of camera 200 and each image quality setting.

Here, the neural network (NN) loaded into the image processing unit 160 will be described using Figure 4. Figure 4 is an explanatory diagram of a neural network (NN) that processes an input image. As an example, a convolutional neural network (hereafter referred to as a CNN) is used, but the NN in this embodiment is not limited to a CNN. The NN may be, for example, a generative adversarial network (GAN) or may have skip connections. The NN may also be a recurrent type such as a recurrent neural network (RNN).

In Figure 4(a), input image 501 represents the image input to the NN or the feature map described below. Convolution matrix 503 in Figure 4(a) is a filter that performs convolution operation 502 on input image 501. Bias 504 is a value added to the result output by convolution operation 502 of input image 501 and convolution matrix 503. Feature map 505 is the result of the convolution operation after bias 504 is added to the result output by convolution operation 502. For simplicity, Figure 4 depicts a small number of neurons, hidden layers, and channels, but the number of neurons and layers, as well as the number and weights of connections between neurons, are not limited to this example. Furthermore, when the NN shown in Figure 4 is implemented in an FPGA, ASIC, etc., the connections and weights between neurons may be reduced.

A CNN obtains a feature map of an input image by performing a convolution operation on the input image using a filter. The size of the filter can be arbitrary. In the next layer, a different feature map is obtained by performing a convolution operation on the feature map of the previous layer using a different filter. In each layer, an input signal is multiplied by a filter and summed with a bias. An activation function is then applied to the result to obtain an output signal for each neuron. The weights and biases in each layer are called NN parameters, and the NN parameters are updated during training. Examples of activation functions include the sigmoid function and the ReLU function. In this embodiment, the Leaky ReLU function shown in the following equation (1) is used, but this is not limited to this. In equation (1), max represents a function that outputs the maximum value among its arguments.
f(x)=max(x, x×0.2) Formula (1)
As described above, in the pre-learning to obtain the NN parameters, an image having the noise characteristics of the camera 200 is used as a student image, and a corresponding noise-free image of the student image is used as a teacher image. Learning is performed using a pair of the student image and the teacher image, thereby realizing NR.

The aforementioned feature map 505 focuses on noise in the input image, and the CNN can learn a region of interest that emphasizes noise components by applying a different parameter to the feature map 505. As shown in Figure 4(b), the CNN generates an intermediate layer 602 by averaging the channel-wise average of input image 601, which is obtained by dividing the input image 501 into channels, for example, by color. The CNN performs multiple convolutions on intermediate layer 602 using equation (1) to obtain intermediate layer 603. The CNN further performs convolutions so that the output channel is 1 to obtain attention layer 604. Attention layer 604 is an intermediate layer in which features appear in noise regions excluding high-frequency components of the subject. The CNN multiplies attention layer 604 by a noise intensity parameter 605 to obtain attention layer 606, in which noise is emphasized. By convolving the attention layer 606 with the above-mentioned input image 501 together, the degree of attention paid to noise regions contained in the input images 501 and 601 is adjusted, i.e., emphasized, resulting in a neural network capable of noise reduction processing.

Returning to the explanation of FIG. 3, in S104, the CPU 130 performs operation input initialization. The CPU 130 executes initialization processing to input operation signals from the controller 300 external to the image processing device 100. The image processing device 100 also has a function capable of displaying a menu for the NR function. The CPU 130 can switch the NR function between enabled and disabled based on user input from the controller 300. The CPU 130 can also switch the NR mode setting between auto mode and manual mode, and in manual mode, can display an NR intensity menu setting screen for setting the NR intensity. As an example of this initialization processing, the CPU 130 sets NR enabled/disabled to enabled and sets the NR mode setting to auto mode. After the CPU 130 executes S104, it ends the subroutine S100 and proceeds to S110 in FIG. 2.

In S110 of Figure 2, the CPU 130 stores the input image received by the image input unit 110 from the camera 200 in the memory 140.

In S120, the CPU 130 performs image processing settings. Figure 5 shows a flowchart of the image processing setting subroutine in S120.

In S121, the CPU 130 selects and sets the model of the camera 200 connected to the image processing device 100 based on information input via the controller 300 and the operation input unit 150. Note that the CPU 130 may select the model of the camera 200 based on information selected by the user from a setting menu of the image processing device 100, or may select the model of the camera 200 based on information automatically detected by the image input unit 110 from information embedded in the image signal.

In S122, similar to S121, the CPU 130 selects the gamma applied by the camera 200 connected to the image processing device 100. Note that the CPU 130 may select the gamma based on information selected by the user from a setting menu of the image processing device 100, or may select the gamma based on information automatically detected by the image input unit 110 from information embedded in the image signal.

In S123, the CPU 130 determines whether the camera model was changed in S121 or whether the gamma selection was changed in S122. If the gamma selection was changed, the CPU 130 proceeds to S124; if the gamma selection was not changed, the CPU 130 proceeds to S125.

In S124, the CPU 130 loads and sets the trained NN parameters corresponding to the model and gamma of the camera 200 selected in S121 and S122 into the image processing unit 160. As a result, the NN loaded into the image processing unit 160 becomes an NN with appropriate NN parameters trained according to the connected camera 200 and its gamma setting. In S124, the NN performs inference, i.e., the image processing unit 160 loads the image for NR processing from memory 140 and is ready for processing.

In S125 of FIG. 5, the CPU 130 selects the NR mode. The CPU 130 selects the NR mode based on an operation signal input from the controller 300 via the operation input unit 150 by the user via a selection menu or the like. For example, the CPU 130 selects and sets either the auto mode or the manual mode as the NR mode based on the operation signal input by the user. Details of the NR mode processing will be described later in S140 and S150. After executing S125, the CPU 130 ends the subroutine S120 and proceeds to step S130 of FIG. 2.

In S130 of Figure 2, the image processing unit 160 performs noise estimation processing on the input image acquired in S110. Figure 6 shows a flowchart of the noise estimation processing subroutine of S130.

In S131 of FIG. 6, the CPU 130 sets the input image as a noise detection target region and divides the noise detection target region into blocks. Note that in this embodiment, the entire input image is described as the noise detection target region, but the CPU 130 may set a partial region of the input image as the noise detection target region. For example, the CPU 130 may set 3 x 3 pixels of the input image as one block.

Next, in S132, the CPU 130 calculates the luminance and noise variance of each block in each divided region.

Next, in S133, the CPU 130 extracts flat areas of noise variance in the noise detection area. Figure 7 is a diagram illustrating an example of flat area extraction. Specifically, the CPU 130 extracts flat areas based on whether the noise variance value of each block is below a threshold. For example, as shown in Figure 7, if the noise variance of a block is below the threshold, the CPU 130 determines that the block is a flat area and sets the block to 1. On the other hand, if the noise variance of a block is above the threshold, the CPU 130 determines that the block is not a flat area and sets the block to 0. If the above-mentioned blocks contain brightness differences, such as texture information of the subject, the brightness differences will be calculated as the noise variance value, making it difficult for the CPU 130 to accurately estimate noise.

In S134, the CPU 130 collects statistical values of luminance versus noise variance limited to blocks determined to be flat areas, and plots the luminance versus noise variance characteristics. As a result, even if luminance differences such as texture information of the subject are included in the above-mentioned block units, the CPU 130 can prevent the luminance differences from being calculated as noise variance values, thereby enabling accurate noise estimation.

In S135, the CPU 130 loads noise variance characteristic data corresponding to the currently set gamma characteristics. This noise variance characteristic data is stored in the memory unit 180 by measuring the luminance-to-noise variance characteristic for each gain in advance for the model of the connected camera 200 and the set gamma. Figure 8 is a diagram showing examples of noise variance characteristics for different camera 200 settings. For example, the memory unit 180 stores table data as noise variance characteristics, including noise variance characteristics associated with luminance and gain, as shown in Figures 8(a) and 8(b).

In S136 of Figure 6, the CPU 130 selects and references the noise dispersion characteristics for each gain contained in the noise dispersion characteristics table loaded in S135.

In S137, the CPU 130 measures the distance of the noise variance characteristic data corresponding to the gain selected in S136 relative to the variance value at each luminance plotted in S134, and performs a characteristic match determination. At this time, a match determination may be made if the distance is below an arbitrary threshold. If the noise variance characteristics do not match in S137, i.e., if the result is false, the CPU 130 returns to step 136 and references the noise variance characteristic data for the next gain to perform a noise variance characteristic match determination. If the CPU 130 determines a match in S137, i.e., if the result is true, the process proceeds to S138.

In S138, the CPU 130 estimates the gain for which it was determined in S136 and S137 that the noise variance characteristics match as the gain input to the image processing device 100.

Various scenes are captured by the camera 200 and input to the image processing device 100. When estimating the noise variance characteristics shown in Figure 6, it is desirable to ensure that the detection target area is as large as possible. The larger the detection target area, the higher the probability of detecting the number of flat blocks described above. As a result, more variance characteristic information can be collected in S134 of Figure 6.

Returning to the explanation of Figure 6, after executing S138, the CPU 130 ends the subroutine S130 in Figure 6 and proceeds to S140 in Figure 2. In S140 in Figure 2, the CPU 130 executes image processing change control. Figure 9 shows a flowchart of the subroutine for image processing change control in S140.

In S1401 of FIG. 9, the CPU 130 selects whether NR is enabled or disabled. The CPU 130 may select whether NR is enabled or disabled based on an operation signal input by a user or the like from the controller 300 via the operation input unit 150. The CPU 130 may also select whether NR is enabled or disabled based on a signal or command input from outside via a network or the like. If S1401 is true, i.e., NR is enabled, the CPU 130 proceeds to S1402, and if it is false, i.e., NR is disabled, the CPU 130 proceeds to S1409.

In S1402, the CPU 130 determines the NR mode selected in S125 of Figure 5. If the determination in S1402 is true, i.e., the NR mode is auto mode, the CPU 130 proceeds to S1403; if the determination in S1402 is false, i.e., the NR mode is manual mode, the CPU 130 proceeds to S1405.

In S1403, i.e., when the NR mode is auto mode, the CPU 130 obtains the gain of the input image estimated in S138 of Figure 6.

In S1404, the CPU 130 sets a target value A when the NR mode is auto mode. The target value A here refers to the target values for the enhancement parameter, luminance signal addition ratio, and color signal addition ratio assigned to the noise intensity parameter 605 in FIG. 4. Since the higher the gain, the more noise is included, the CPU 130 sets a high value for the enhancement parameter target value A when the estimated gain of the camera 200 acquired in S1403 is high. On the other hand, the CPU 130 sets a low value for the enhancement parameter target value A when the estimated gain is low. As a result, the luminance signal addition ratio and color signal addition ratio are set to low values when the estimated gain is high, and high values when the estimated gain is low. Color noise may affect image quality, particularly when the gain of the camera 200 is high. When the estimated gain is high, the CPU 130 may not use a color signal addition ratio.

The emphasis parameter will now be explained. The amount of noise input by the camera 200 and the gamma setting value of the camera 200 can fluctuate not only depending on the gain but also on the set gamma. If a high gain is estimated, the emphasis parameter is set to a high value, and the CPU 130 can increase the sensitivity to noise so as to obtain a higher NR effect. On the other hand, if a low gain is estimated, the emphasis parameter is set to a low value, preventing NR from being applied more than necessary. As a result, the CPU 130 can prevent adverse effects of NR, such as loss of resolution and texture loss.

In S1405, i.e., when the NR mode is manual mode, the CPU 130 sets the NR intensity in the same manner as in S125 of FIG. 5. For example, the CPU 130 obtains information on High, Middle, or Low selected by the user in the NR intensity menu and sets the NR intensity. In other words, the NR intensity is different from internal settings like the aforementioned emphasis parameters, and is selected externally by the user of the image processing device 100 via a UI, etc. If the CPU 130 sets High, the process proceeds to S1406 and sets target value B. If the CPU 130 sets Middle, the process proceeds to S1407 and sets target value C. If the CPU 130 sets Low, the process proceeds to S1408 and sets target value D. As with target value A, the CPU 130 sets target value B, target value C, and target value D for the emphasis parameter, luminance signal addition ratio, and color signal addition ratio, respectively. These processes are performed when the NR mode is manual, so the image processing device 100 stores predetermined parameters for each of the target values B, C, and D in the storage unit 180 or the like in advance. The target values of the enhancement parameters are stored in the relationship B>C>D. The target values of the luminance signal addition ratio and the color signal addition ratio are stored in the relationship B<C<D. As with S1404 described above, if the estimated gain is high for target value B, the CPU 130 may not use the color signal addition ratio.

In S1409, the CPU 130 sets the target value E when NR is disabled. For example, the CPU 130 sets the target value E to disable the emphasis parameter, and to 100% for the luminance signal addition ratio and the color signal addition ratio.

When the CPU 130 executes one of S1404, S1406, S1407, S1408, and S1409 to set target values A through E for each NR operating state, it proceeds to S1410.

In S1410, the CPU 130 calculates the deviation of the current enhancement parameters, luminance signal addition ratio, and color signal addition ratio from the aforementioned target values.

In S1411, CPU 130 determines whether the deviation from the target value calculated in S1410 is greater than threshold value α. As described above, CPU 130 calculates the deviation from the target value for each of the emphasis parameter, luminance signal addition ratio, and color signal addition ratio, so threshold value α may be set separately for each of the emphasis parameter deviation, luminance signal addition ratio deviation, and color signal addition ratio deviation. If S1411 is true, i.e., the deviation of the target value is greater than threshold value α, CPU 130 determines that each of the emphasis parameter, luminance signal addition ratio, and color signal addition ratio has not reached the target value and proceeds to S1412; if it is false, it determines that they have come sufficiently close to the target value and proceeds to S1416.

In S1412, the CPU 130 sets the current control amount for each target value, i.e., the emphasis parameter, luminance signal addition ratio, and color signal addition ratio. In S1412, the control amount may be set using a method recognized in general control engineering. The control amount is a value added to the parameter to be changed, among the emphasis parameter, luminance signal addition ratio, and color signal addition ratio, in order to bring it closer to the target value.

In S1413, the CPU 130 determines whether the control amount calculated in S1412 exceeds the threshold value β. The value of the threshold value β may be any value. The threshold value β may be a constant value for the various NR states described above, or may be increased or decreased in conjunction with the threshold value α. The control amount calculated by the CPU 130 in S1412 may be applied as is, but in S1413, the CPU 130 limits the control amount for one input image to a constant amount, i.e., the threshold value β, with the aim of mitigating the amount of image change toward the set target value. If S1413 is true, i.e., the control amount exceeds the threshold value β, the CPU 130 proceeds to S1414; if false, i.e., the control amount is equal to or less than the threshold value β, the CPU 130 proceeds to S1415.

In S1414, the CPU 130 sets the threshold value β to the control amount. In S1416, the CPU 130 sets the control amount to 0.

In S1415, the CPU 130 sets the next setting values to the current settings, i.e., the current emphasis parameters, luminance signal addition ratio, and color signal addition ratio, plus the control amounts, then ends the subroutine in FIG. 9 and proceeds to S150 in FIG. 2. The CPU 130 then repeats step S1415 multiple times for each input image acquired sequentially, thereby adding the control amounts multiple times to the emphasis parameters, luminance signal addition ratio, and color signal addition ratio. In this way, the CPU 130 brings the emphasis parameters, luminance signal addition ratio, and color signal addition ratio closer to the target values. FIG. 10 shows a flowchart of the NR processing subroutine in S150.

In S151 of Figure 10, the CPU 130 sets the emphasis parameter set in S1415 of Figure 9. The emphasis parameter is applied to the noise intensity parameter 605 described in Figure 4, and acts as an emphasis parameter applied to an attention map focusing on noise.

In S152, the CPU 130 loads and acquires the image acquired in S110 of Figure 2 from the memory 140, and inputs it to the image processing unit 160 for noise reduction inference. Figure 11 is a diagram showing images to be subjected to noise reduction in the order of processing. Figure 11(a) is an example of an image of a dark area captured by the camera 200. The image in Figure 11(a) includes an image of a person, although it is difficult to see. Because the camera 200 applies gain to the image in Figure 11(a), the image processing device 100 acquires an image containing noise, as shown in Figure 11(b). Therefore, the CPU 130 loads and acquires an image such as that shown in Figure 11(b) from the memory 140 as an image for noise reduction inference.

In S153, the image processing unit 160 executes NR inference processing.

In S154, the CPU 130 stores the image resulting from the inference performed by the image processing unit 160 in S153 (hereinafter referred to as the inference image) in memory 140 or the like. The inference image obtained in S153 has had noise reduced by applying NR inference processing, as shown in Figure 11(c). When the CPU 130 executes S154, it ends the subroutine in Figure 10 and proceeds to S160 in Figure 2.

In S160 of Figure 2, the CPU 130 executes image output processing to output the inference image that has undergone NR processing up to S150. Figure 12 shows a flowchart of the image output processing subroutine of S160.

In S161 of Figure 12, the CPU 130 acquires the inference image saved in S154 of Figure 11.

In S162, the image processing unit 160 performs a luminance signal synthesis process on the inference image saved in S154. The image processing unit 160 synthesizes the luminance signal with the inference image based on the luminance signal addition ratio set as the next setting value in S1415 of FIG. 9.

In S163, the image processing unit 160 performs a color signal synthesis process on the inference image with the synthesized luminance signal. The image processing unit 160 synthesizes the color signal with the inference image based on the color signal addition ratio set as the next setting value.

In S164, the CPU 130 outputs the image obtained by the synthesis processing up to S163 by the image processing unit 160 from the image processing device 100 via the image output unit 170 and displays it on the monitor 400. For example, a portion of the image in Figure 11(b) in NR mode and strength menu status is synthesized with the inference image shown in Figure 11(c) to produce the image shown in Figure 11(d).

After executing S164, the CPU 130 ends the image output processing subroutine in FIG. 12 and returns to the processing in FIG. 2. After executing S160 in FIG. 2, the CPU 130 returns to S110 and repeats image processing again.

By repeatedly executing the image processing shown in Figure 2 by the CPU 130 when switching NR between enabled and disabled, when switching NR mode between auto and manual modes, and when changing the intensity setting from the menu, S1415 in Figure 9 is updated, and the image output in S164 in Figure 12 changes smoothly.

In the above-described embodiment, an example was given in which target values were set for the emphasis parameters, luminance signal addition ratio, and color signal addition ratio, and control amounts were added to sequentially set the next setting values, but the targets to which control amounts are added and sequentially set are not limited to the above example. For example, in this embodiment, target values may be set for NN parameters, and control amounts may be added to the NN parameters to sequentially set the NN parameters.

As described above, in this embodiment, target values for the emphasis parameters, luminance signal addition ratio, and color signal addition ratio are set by switching NR between enabled and disabled, switching the NR mode between auto mode and manual mode, and changing the intensity setting from a menu. This embodiment shows a method in which the next setting values for the emphasis parameters, luminance signal addition ratio, and color signal addition ratio are sequentially updated for each frame of image based on the control amount relative to the set target values. As a result, this embodiment suppresses sudden changes in NR parameters, including the emphasis parameters, luminance signal addition ratio, and color signal addition ratio, reduces image shock when parameters are changed, and provides high-quality images with a smooth NR effect applied.

In this embodiment, the control amount is compared with a threshold value β, and if the control amount is greater than the threshold value β, the threshold value β is set as the control amount. This prevents the control amount from exceeding the threshold value β, and can mitigate changes in the emphasis parameters, luminance signal addition ratio, and color signal addition ratio to which the control amount is added.

In this embodiment, the target value is set based on the NR strength selected by the user. This allows the user to set a target value that meets their needs, and the output image, which has been noise-removed using the enhancement parameters, luminance signal addition ratio, and color signal addition ratio set based on the target value, can be tailored to the user's needs.

The present invention has been described in detail above based on preferred embodiments, but the present invention is not limited to these specific embodiments, and various forms within the scope of the invention are also included. Parts of the above-described embodiments may be combined as appropriate.

In the above embodiment, as shown in FIG. 3, if there is a change in the model selection and gamma selection of the camera 200, the CPU 130 performs NR initialization in S124. At this time, it may take time for the CPU 130 to load the DLNR trained model, making it difficult to smoothly infer the NR effect. Therefore, the image processing device 100 may implement the image processing setting of S120 in FIG. 2 using the subroutine shown in FIG. 13 by making the NN processing redundant. FIG. 13 is a flowchart of a subroutine that includes NN redundancy switching processing in the image processing setting of S120. Note that in the description of FIG. 13, explanations of steps similar to those in FIG. 5 will be omitted or simplified.

For example, the image processing device 100 may prepare two NN systems, NN-A and NN-B, as NN redundant processing systems, and perform NN processing using either NN-A or NN-B, while performing NR initialization using the other NN to change the NN parameters based on the control amount.

In S123, if the CPU 130 determines that the model or gamma of the camera 200 has been changed, it determines whether the NN in use is NN-A or NN-B, as shown in S221. If NN-A is in use, the process proceeds to S222, and if NN-B is in use, the process proceeds to S224.

In S222, the CPU 130 initializes NN-B, which is exclusive to NN-A, and continues NR processing by NN-A.

At S223, the CPU 130 sets the NN-B switching flag.

In S224 and S225, the CPU 130 executes processing exclusive of S222 and S226. Specifically, in S224, the CPU 130 initializes NN-A, which is exclusive to NN-B, and continues NN processing by NN-B. In S225, the CPU 130 sets the switching in progress flag for NN-A.

From the next time onwards, processing will begin at S110 in Figure 2, and S123 in Figure 13 will be determined to be false, and processing will proceed to S226.

In S226, the CPU 130 references the switching flag set in S223 or S225, and if S226 is determined to be true, i.e., NN switching is in progress, proceeds to S227; if S226 is determined to be false, i.e., NN switching has been completed, proceeds to S125.

In S227, the CPU 130 updates the composition ratio, which is the ratio for combining an image from NN-A with an image from NN-B, or combining an image from NN-B with an image from NN-A. Here, the CPU 130 changes the NN parameters of the NN system to be switched to target values by transitioning them across multiple images using a predetermined control amount, as shown in FIG. 9. Therefore, the CPU 130 can simply transition and update the composition ratio in accordance with the progress of the transition of the NN parameters being changed. The image processing unit 160 composites the image output from NN-A and the image output from NN-B based on the composition ratio.

In S228, the CPU 130 determines whether the NN switching has been completed. For example, when the composition ratio, which indicates the composition ratio of the image from either NN-A or NN-B to the other, transitions to 100%, the CPU 130 determines that this is true, i.e., that the NN switching has been completed, and proceeds to S229. If this is false, i.e., if the NN switching is in progress, the CPU 130 proceeds to S125.

In S229, the CPU 130 clears the NN-A switching flag and the NN-B switching flag set in S223 or S225, and then proceeds to S125.

The redundancy of the NN process and the subroutine shown in Figure 13 enable smoother DLNR switching processing. Furthermore, this may be implemented in combination with the switching processing shown in Figures 9, 11, and 12.

Second Embodiment
In the second embodiment, a method is shown in which the next set values of the enhancement parameters, luminance signal addition ratio, and color signal addition ratio for the set target values are stored in advance as a reference table, and are updated sequentially for each frame of image.

The configuration of the image processing device 100 in the second embodiment is the same as that in the first embodiment (i.e., Figure 1), so a description will be omitted.

Next, the image processing performed by the CPU 130 in the image processing device 100 will be described using Figure 14. Steps S100 to S130 in Figure 14 are the same as those in the first embodiment, and then the process proceeds to S240. Image processing change control is performed in S240 in Figure 14. The subroutine for S240 is shown in Figure 15.

In S1401 of FIG. 15, the CPU 130 selects whether NR is enabled or disabled, as in FIG. 9. If S1401 is true, the CPU 130 proceeds to S1402, and if it is false, the CPU 130 proceeds to S2409. In S1402, as in FIG. 9, the CPU 130 determines the NR mode. If S1402 is true, i.e., the NR mode is auto mode, the CPU 130 proceeds to S2403, and if it is false, i.e., the NR mode is manual mode, the CPU 130 proceeds to S1405. In S2403, the CPU 130 acquires an estimated gain value A for the input image, as in S138 and S1403 of the first embodiment, and proceeds to S2404.

In S2404, the CPU 130 stores the estimated gain value A when the NR mode is auto mode in the variable TempGain, and proceeds to S2410.

In S1405, i.e., when the NR mode is manual mode, the user of the image processing device 100 operates an external NR intensity menu such as a UI to switch the gain value. If the NR intensity setting is set to High in accordance with the user's selection, the CPU 130 proceeds to S2406 and stores gain value B in the variable TempGain. If the NR intensity setting is set to Middle, the CPU 130 proceeds to S2407 and stores gain value C in the variable TempGain. If the NR intensity setting is set to Low, the CPU 130 proceeds to S2408 and stores gain value D in the variable TempGain.

As with the first embodiment, gain values B to D operate when the NR mode is manual. The magnitude relationship between the gain values is B>C>D.

In S2409, the CPU 130 stores the gain value E in the case where NR is disabled in the variable TempGain. For example, the CPU 130 sets the gain value E to, for example, 1x or 0 dB. Note that the gain value E may generally be set to a low gain, i.e., a gain value where NR is not required. Note that the relationship with the gain value D is D>E.

After S2404, S2406, S2407, S2408, and S2409 are executed, proceed to S2410.

In this embodiment, the gain values stored in the variable TempGain are stored in the array Arr to smoothly transition the enhancement parameters, luminance signal addition ratio, and color signal addition ratio. To retain the gain values for the most recent frames, the gain values are stored in a so-called FIFO (First In First Out) format. Detailed processing will be described later from S2410 onward. For example, if the array length of array Arr is 10, the TempGain values updated every frame will be retained for the most recent 10 frames. By calculating the average of the gain values stored in array Arr, the gain values for the most recent 10 frames to which a time filter has been applied are obtained. The gain values stored in TempGain are subject to a time filter to accommodate changes in settings or, if the NR mode is auto, changes in camera noise, resulting in a gradually changing average gain value. The array length of array Arr is not limited and can be any length. Additionally, the array length of array Arr is indicated by the variable length.

At S2410, the CPU 130 initializes the reference index variable i of the array Arr to 0. Furthermore, it initializes the variable Sum, which is used to calculate the sum of the gain values stored in the array Arr, to the value stored in TempGain, i.e., the most recent gain value.

At S2411, the CPU 130 stores Arr[i+1], the i+1th element of the array Arr, in Arr[i], the i-th element of the array Arr. At S2412, the CPU 130 increments the index variable i by 1. At S2413, the CPU 130 adds the gain value stored in the i-th element of the array Arr to the variable Sum. At S2414, the CPU 130 determines whether the index variable i is smaller than the array length variable length-1 of the array Arr. If S2414 is true, that is, if the CPU 130 determines that the index variable i is smaller than the array length variable length-1, the process returns to S2411 and repeats the process. If S2414 is false, that is, if the CPU 130 determines that the index variable i is not less than the array length variable length-1, and once FIFO processing of the array Arr and the sum Sum stored in the array Arr have been calculated, the process proceeds to S2415.

In S2415, the CPU 130 stores the value obtained by dividing the aforementioned total value Sum by the array length length of the array Arr, i.e., the average value of the gain values, in the variable AveGain. In S2416, the CPU 130 converts the variable AveGain into a gain index value for referencing the tables for the emphasis parameters, luminance signal addition ratio, and color signal addition ratio. For example, if the array length length of the array Arr is 10, the CPU 130 calculates the average value of the gain values for the most recent 10 frames and derives the corresponding gain index value.

FIG. 16 shows examples of reference tables for the setting values to be referenced as the next setting values for the enhancement parameters, luminance signal addition ratio, and color signal addition ratio. A reference table is an example of reference data. FIG. 16(a) is an enhancement parameter table. FIG. 16(b) is a luminance signal addition ratio table. FIG. 16(c) is a color signal addition ratio table. Each row stores parameters for each gain index value for a camera and its gamma. For example, in the enhancement parameter table of FIG. 16(a), the enhancement multiplier is set to 1.0 when the gain value is low, and the higher the gain, the higher the value. In the luminance signal addition ratio table of FIG. 16(b) and the color signal addition ratio table of FIG. 16(c), the ratio is increased when the gain value is low and decreased when the gain value is low. Therefore, each table can be based on the same concept as in the first embodiment. It is recommended that each parameter be determined in advance to achieve optimal NR image quality based on the gain characteristics of the camera and gamma. The resolution of the gain index value can be set to, for example, 1 dB. A finer resolution will result in a smoother NR change. However, since a finer resolution may increase the memory size of each table, it is fine to use 3 dB or 6 dB increments. In S2417, you can implement a formula for converting gain index values according to the table resolution.

Returning to the explanation of Figure 15, in S2417, the CPU 130 references the reference tables described in Figure 16 based on the table reference index converted in S2416, obtains the enhancement parameters, luminance signal addition ratio, and color signal addition ratio, and sets them in the image processing unit 160.

By repeating step S2417 multiple times for each input image acquired sequentially, the CPU 130 can gradually change the average values described above, gradually bringing the referenced enhancement parameters, luminance signal addition ratio, and color signal addition ratio closer to their target values.

After the CPU 130 executes S2417, the process proceeds to S2418. In S2418, the CPU 130 stores the TempGain value in Arr[i+1], which is the i+1th element of the array Arr. If the array length is 10, the latest gain value will be stored at i+1=9, i.e., at the end of the array Arr.

When the CPU 130 executes S2418, it ends the subroutine in FIG. 15 and proceeds to S150 in FIG. 14. S150 and S160 in FIG. 14 are similar to S150 and S160 in FIG. 2, so a description thereof will be omitted.

In the second embodiment, the next setting values for the enhancement parameters, luminance signal addition ratio, and color signal addition ratio are stored in advance as a reference table. Furthermore, the second embodiment averages the gain values obtained for each frame over multiple frames, i.e., applies a time filter, making it possible to smoothly change the next setting values to be referenced.

Third Embodiment
In the first and second embodiments, an example has been shown in which the gain value set in the camera 200 is estimated from the amount of noise contained in the video input from the camera 200 to the image processing device 100 by the noise estimation process shown in S130 in Figures 2 and 14. However, if the camera 200 is equipped with a communication means, the image processing device 500 may be able to acquire the camera name, gamma setting, and gain setting of the camera 200. The image processing device 500, which can acquire the setting values themselves, can accurately acquire gain value information of the camera 200, making it possible to perform more optimal noise reduction processing.

Figure 17 is a diagram showing the configuration of an image processing device 500 in the third embodiment. The image processing device 500 has a configuration in which a communication unit 510 is added to the image processing device 100 shown in Figure 1. The communication unit 510 is connected to the bus 120. The CPU 130 sends commands corresponding to the protocol of the camera 200 via the communication unit 510 to control the camera 200 and obtain the setting values of the camera 200. The physical communication format between the camera 200 and the communication unit 510 can be Ethernet or serial communication, but the communication format is not particularly limited. In the case of Ethernet, a protocol based on socket communication or a Web API using the http protocol is common. In the case of serial communication, a protocol based on asynchronous communication is used. There are also no particular limitations on these protocols.

Next, the image processing performed by the CPU 130 in the image processing device 500 will be described using Figures 18 to 20. From S100 to S110 in Figure 18, the CPU 130 performs the same processing as in the first embodiment, and then proceeds to S320. In S320 in Figure 18, the CPU 130 performs camera parameter acquisition processing to acquire camera information. As described above, the CPU 130 acquires the setting values and other information of the camera 200 via the communication unit 510 in accordance with the camera 200's communication protocol.

Figure 19 is a diagram of the subroutine for the camera parameter acquisition process of S320. In S321 of Figure 19, the CPU 130 acquires model information for the camera 200. In S322, the CPU 130 acquires gamma information currently set in the camera 200. In S323, the CPU 130 acquires the gain value currently set in the camera 200. The CPU 130 also sets the acquired gain value as an estimated gain value, which will be described later. In S324, the CPU 130 acquires the shutter value, i.e., the exposure time setting value, currently set in the camera 200. When the CPU 130 has executed up to S324, it ends the subroutine of Figure 19 and proceeds to S330 of Figure 18.

In S330 of Figure 18, the CPU 130 performs image processing settings.

Figure 20 is a diagram of the image processing setting subroutine of S330 in Figure 18. Of the steps in Figure 20 for image processing setting, steps similar to those in S120 in Figure 2 will be explained briefly. In S331, the CPU 130 selects the camera model information acquired in S321 in Figure 19. In S332, the CPU 130 selects the gamma value information selected for the camera 200 acquired in S322 in Figure 19. In the first and second embodiments, examples were shown in which the user selected the camera and gamma value information for the camera via a menu on the image processing device 100, but in this embodiment, the CPU 130 selects the camera and gamma value information obtained via communication from the camera 200. Therefore, the NR setting value is automatically selected according to the gamma value setting value of the connected camera 200. After the CPU 130 executes S332, the process proceeds to S123. The same processing as in the first embodiment is performed from S123 to S125, so a description thereof will be omitted. When the connected camera 200 is changed or the gamma value information of the camera 200 is changed, the CPU 130 automatically performs NR initialization in this subroutine. When the CPU 130 executes the NR mode selection processing of S125, the subroutine in FIG. 20 ends and the process proceeds to S140 in FIG. 18.

In S140 of Figure 18, the CPU 130 executes the same processing as in the first embodiment, so the explanation will be omitted. Note that even if S140 in this embodiment is replaced with S240 described in Figure 14 of the second embodiment, the subsequent processing will be executed in the same way.

The processing of S150 and S160 in Figure 18 is the same as that in the first embodiment, so a description thereof will be omitted.

As described above, the third embodiment has shown that it is possible to perform processing similar to that of the above-described embodiments by obtaining setting value information directly from the camera 200.

The present invention has been described in detail above based on preferred embodiments, but the present invention is not limited to these specific embodiments, and various forms within the scope of the invention are also included. Parts of the above-described embodiments may be combined as appropriate.

Note that while this embodiment shows an example in which an image is input to the image processing device from a camera, the present invention can also be implemented in a configuration in which input is from an image output device other than a camera.

The present invention also includes cases where a software program that realizes the functions of the above-described embodiments is supplied to a system or device having a computer capable of executing the program, either directly from a recording medium or via wired or wireless communication, and the program is then executed.

Therefore, the program code supplied to and installed on a computer to realize the functional processing of the present invention also realizes the present invention. In other words, the computer program itself for realizing the functional processing of the present invention is also included in the present invention.

In this case, as long as it has the functionality of a program, the form of the program is irrelevant, such as object code, a program executed by an interpreter, or script data supplied to the OS.

The recording medium for supplying the program may be, for example, a hard disk, a magnetic recording medium such as magnetic tape, an optical/magneto-optical storage medium, or a non-volatile semiconductor memory.

Another possible method of providing the program is to store the computer program that forms the present invention on a server on a computer network, and have connected client computers download and program the computer program.

(Other Examples)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

The disclosure of this specification includes the following image processing device, image processing method, and program.
(Item 1)
an image processing means for performing image processing on an acquired input image using a trained model having first parameters obtained by training noise characteristics of the image;
a noise reduction means for reducing noise in the image-processed input image using a second parameter;
a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter;
a setting means for changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount or preset reference data when changing any of the first parameter, the second parameter, and the third parameter;
An image processing device comprising:
(Item 2)
2. The image processing device according to item 1, wherein the first parameter is a parameter of a neural network resulting from learning noise characteristics in information relating to a camera that captured the input image.
(Item 3)
3. The image processing device according to claim 1, wherein the second parameter is a parameter indicating a strength of the noise reduction.
(Item 4)
4. The image processing device according to claim 1, wherein the third parameter is a ratio for combining the input image in which the noise has been reduced with the input image.
(Item 5)
5. The image processing device according to claim 1, wherein the third parameter is a ratio for combining the luminance signal of the input image from which the noise has been reduced with the luminance signal of the input image.
(Item 6)
6. The image processing device according to claim 1, wherein the third parameter is a ratio for combining the color signal of the input image from which the noise has been reduced with the color signal of the input image.
(Item 7)
7. The image processing device according to item 6, wherein the setting means lowers the ratio for combining the color signals as the input image has more noise.
(Item 8)
the image processing means has two trained models based on the first parameters, and when one trained model is in use, the first parameters of the other trained model are changed across a plurality of input images using the control amount;
8. The image processing device according to any one of items 1 to 7, further comprising: an image synthesis unit that synthesizes, based on a synthesis ratio, an input image that is processed by the one trained model and output after noise reduction by the noise reduction unit, and an input image that is processed by the other trained model and output after noise reduction by the noise reduction unit.
(Item 9)
9. The image processing device according to item 8, wherein the combining means combines the images based on the combining ratio that changes in accordance with the progress of the change of the first parameter that is being changed.
(Item 10)
10. The image processing device according to any one of items 1 to 9, wherein the setting unit changes the second parameter and the third parameter by setting the control amount so that the second parameter and the third parameter approach target values of the parameters to be changed.
(Item 11)
Item 11. The image processing device according to item 10, wherein the setting means changes the first parameter, the second parameter, and the third parameter based on the control amount when either the first parameter or the target value is changed.
(Item 12)
Item 11. The image processing device according to item 10, wherein the setting means compares the control amount with a threshold value, and if the control amount is greater than the threshold value, sets the threshold value as the control amount.
(Item 13)
13. The image processing device according to item 12, wherein the setting means sets the target value in accordance with a strength for reducing the noise selected by a user.
(Item 14)
14. The image processing device according to any one of items 1 to 13, wherein the setting unit acquires at least one of the second parameter and the third parameter from reference data stored in advance.
(Item 15)
Item 15. The image processing device according to item 14, wherein the setting means acquires at least one of the second parameter and the third parameter from the reference data based on an average value of gain values of images of a plurality of frames.
(Item 16)
16. The image processing device according to any one of items 1 to 15, wherein the setting means sets at least one of the second parameter and the third parameter based on information about a camera acquired from the camera that captured the input image.
(Item 17)
an image processing step of processing an acquired input image using a trained model having first parameters obtained by training noise characteristics of the image;
a noise reduction step of reducing noise in the image-processed input image using a second parameter;
a combining step of combining the input image and the noise-reduced input image based on a third parameter;
a setting step of changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount or preset reference data when changing any of the first parameter, the second parameter, and the third parameter;
An image processing method comprising:
(Item 18)
A program for causing a computer to function as each means of the image processing device according to any one of items 1 to 16.

The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to clarify the scope of the invention.

100: Image processing device, 200: Camera, 130: CPU, 160: Image processing unit, 170: Image output unit.

Claims (18)

an image processing means for performing image processing on an acquired input image using a trained model having first parameters which are network parameters of a neural network obtained by training noise characteristics of the image;
a noise reduction means for reducing the noise in the image-processed input image using a second parameter, which is an intensity parameter for emphasizing and reducing the noise ;
a synthesis means for synthesizing the input image and the noise-reduced input image based on a third parameter;
a setting means for changing and setting the parameter to be changed across a plurality of input images acquired sequentially, when changing any of the first parameter, the second parameter, and the third parameter, based on a control amount for bringing the parameter closer to a target value selected by a user or preset reference data;
An image processing device comprising: The image processing device according to claim 1, wherein the first parameter is a parameter of a neural network resulting from learning noise characteristics from information about the camera , which is information about the model or image quality settings of the camera that captured the input image. The image processing device according to claim 1 , wherein the second parameter is a parameter indicating a strength of the noise reduction. The image processing device according to claim 1 , wherein the third parameter is a ratio for combining the input image with the noise-reduced input image. The image processing device according to claim 1 , wherein the third parameter is a ratio for combining the luminance signal of the input image from which the noise has been reduced with the luminance signal of the input image. The image processing device according to claim 1 , wherein the third parameter is a ratio for combining the color signal of the input image from which the noise has been reduced with the color signal of the input image. 7. The image processing device according to claim 6, wherein the setting means lowers the ratio for combining the color signals as the input image has more noise. the image processing means has two trained models based on the first parameters, and when one trained model is in use, the first parameters of the other trained model are changed across a plurality of input images using the control amount;
2. The image processing device according to claim 1, further comprising: an image synthesis means for synthesizing, based on a synthesis ratio, an input image that has been image processed by the one trained model and output after noise reduction by the noise reduction means, and an input image that has been image processed by the other trained model and output after noise reduction by the noise reduction means. The image processing device according to claim 8 , wherein the combining means combines the images based on the combining ratio that changes in accordance with the progress of the change of the first parameter that is being changed. 2. The image processing apparatus according to claim 1, wherein the setting means changes the second parameter and the third parameter by setting the control amount so that the second parameter and the third parameter approach target values of the parameters to be changed. 11. The image processing device according to claim 10, wherein the setting means changes the first parameter, the second parameter, and the third parameter based on the control amount when either the first parameter or the target value is changed. 11. The image processing apparatus according to claim 10, wherein the setting means compares the control amount with a threshold value, and if the control amount is greater than the threshold value, sets the threshold value as the control amount. 13. The image processing apparatus according to claim 12, wherein the setting means sets the target value in accordance with a noise reduction intensity selected by a user. 2. The image processing apparatus according to claim 1, wherein the setting means acquires and sets at least one of the second parameter and the third parameter from reference data stored in advance. 15. The image processing device according to claim 14, wherein the setting means acquires at least one of the second parameter and the third parameter from the reference data based on an average value of gain values of images of a plurality of frames. The image processing device according to claim 1 , wherein the setting means sets at least one of the second parameter and the third parameter based on information about a camera acquired from the camera that captured the input image. an image processing step of performing image processing on the acquired input image using a trained model having first parameters which are network parameters of a neural network obtained by training noise characteristics of the image;
a noise reduction step of reducing the noise in the image-processed input image using a second parameter, which is an intensity parameter for emphasizing and reducing the noise ;
a combining step of combining the input image and the noise-reduced input image based on a third parameter;
a setting step of changing and setting the parameter to be changed across a plurality of input images acquired sequentially based on a control amount for bringing the parameter closer to a target value selected by a user or based on preset reference data , when changing any of the first parameter, the second parameter, and the third parameter;
An image processing method comprising: A program for causing a computer to function as each means of the image processing device described in any one of claims 1 to 16.