JP7686428B2 - IMAGE PROCESSING METHOD, MACHINE LEARNING MODEL GENERATION METHOD, IMAGE PROCESSING DEVICE, IMAGE PROCESSING SYSTEM, AND PROGRAM - Google Patents

Description

The present invention relates to an image processing method that uses machine learning to correct multiple components.

When correcting multiple components, such as aberration components and noise components, contained in a captured image, it is desirable to perform correction for each component in order to achieve an appropriate correction strength for each component. Conventionally, when the components are correlated, it has been difficult to correct one component from the captured image without changing the other component, but in recent years, it has become possible to correct only one component with high accuracy using deep learning. Patent Document 1 discloses a method of using a neural network to correct blurring due to aberration and diffraction from a captured image without changing the noise component, and then performing noise reduction processing.

JP 2020-144489 A

In the method of Patent Document 1, noise reduction processing is performed on a captured image in which the intensity of blur correction has been adjusted. Therefore, if the intensity of blur correction is weakened to adjust sharpness or undershoot, the aberration components will change if noise reduction processing is performed without using deep learning. Generally, noise reduction processing is a blurring process, so the blur components spread and chromatic aberration also spreads and becomes noticeable. In addition, since the blur components are not acquired as a residual map, the intensity of blur correction cannot be adjusted after noise reduction processing, and noise reduction processing must be performed every time the intensity of blur correction is adjusted. Since the estimation processing of deep learning requires a large calculation load, the calculation load will be significantly increased if noise reduction processing is performed using deep learning every time the intensity of blur correction is adjusted. In addition, since the correlation between the blur components and the noise components is not taken into consideration, the noise components do not follow the change in luminance value in the area where the blur components have changed, resulting in unnaturalness. In addition, since the noise components are not acquired as a residual map, the noise components cannot be corrected taking the blur components into consideration.

The present invention aims to provide an image processing method that can perform image correction corresponding to multiple correlated components with appropriate correction strength to obtain a natural corrected image.

An image processing method according to one aspect of the present invention is characterized by comprising the steps of: acquiring a first residual map corresponding to a first component and a second residual map corresponding to a second component by inputting an input image to at least one machine learning model ; generating a third residual map based on the first residual map and the second residual map ; and acquiring an output image based on the input image, the first residual map, and the third residual map.

The present invention provides an image processing method that can perform image correction corresponding to multiple correlated components with appropriate correction strength to obtain a natural corrected image.

FIG. 1 is a block diagram of an image processing system according to a first embodiment. 1 is an external view of an image processing system according to a first embodiment. 11 is a flowchart relating to weight learning in the first embodiment. FIG. 1 is a diagram showing a flow of learning weights of a neural network in the first embodiment. 4 is a flowchart relating to generation of a corrected image in the first embodiment. FIG. 2 is a block diagram of an image processing flow according to the first embodiment. FIG. 11 is a block diagram of an image processing flow according to a modified example of the first embodiment. FIG. 11 is a block diagram of an image processing system according to a second embodiment. FIG. 11 is an external view of an image processing system according to a second embodiment. FIG. 11 is a block diagram of an image processing system according to a third embodiment. 13 is a flowchart relating to image processing in the third embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. In each drawing, the same components are given the same reference symbols, and duplicate descriptions are omitted.

First, before describing the specific embodiments, the gist of the present invention will be described. The present invention relates to image processing that performs image correction corresponding to multiple correlated components with appropriate correction strength to obtain a natural corrected image.

Each component can be expressed as a two-dimensional map that corresponds to the amount of correction at each pixel position of the image. The components are, for example, noise components, blur components caused by the imaging system, blur components caused by defocus, camera shake, and subject motion, scattering components caused by fog, relighting components, and background blur components that correct background blur. Noise components are random noise that occurs randomly with a predetermined variance in the captured image and occurs with a different distribution each time imaging is performed. Noise components include dark current noise, shot noise, and readout noise. Blur components caused by the imaging system include aberration, diffraction, low-pass filters, and aperture effects. Relighting is a virtual change to the light source environment of an image after imaging, and includes adding a light source and changing the light source. The luminance change components caused by relighting are referred to as relighting components below.

Image correction corresponding to each component refers to modifying each component, such as reducing or adding noise, and correcting or adding blur, to create a preferable image or an image that reflects the user's editing intent. In this embodiment, a residual map corresponding to each component is obtained. A residual map is a two-dimensional map that extracts the amount of correction (or the component value itself) of a specific component contained in an image, and correction corresponding to each component is performed by adding or subtracting.

Multiple components may inherently have a correlation. For example, noise components have different variances depending on shooting conditions such as ISO sensitivity and differences in image sensors, but when shot noise is included, they have different variances depending on the luminance value. When the luminance value of a pixel changes due to correction of blur components or scattering components due to fog, the variance of noise that would normally occur in that pixel if there was no blur or scattering will be different. If these corrections are made without considering the correlation, the image will have unnatural noise. For example, noise with a large variance corresponding to the luminance value before the blur was removed is superimposed on pixels or image areas that have become low luminance after the diffraction blur is removed. In addition, unnaturally small noise corresponding to the dark luminance value before the relighting is superimposed on pixels or image areas that have become brighter due to relighting. Furthermore, blur caused by the optical characteristics of the imaging system, defocus, camera shake, and subject motion causes blur that spreads to surrounding pixels depending on the brightness of the subject. Therefore, when the brightness of the subject changes due to lighting, the blur also changes at the same time. Therefore, even if relighting is done using image processing, correcting the blur without considering the correlation between components can result in an image with unnatural blur or with the blur corrected excessively.

Conventionally, when multiple components are correlated, it has been difficult to correct one component from an image without changing the other. For example, when correcting blur components caused by the imaging system, sharpening processing applies a gain to the frequency characteristics, and noise components are emphasized depending on the frequency characteristics of the blur components or the sharpening processing. Conversely, when background blur is increased, blurring processing is applied, and noise components are reduced. Even when noise components are corrected first, the noise reduction processing is a blurring process that uses information from the surrounding image area, so the blur components become larger. In addition, when the luminance value is brightened or darkened by performing relighting processing, the blur components, noise components, and scattering components due to fog become larger or smaller. To separate and obtain these correlated components, it is necessary to capture the changes in each component according to the scene and perform processing according to the subject in the image.

In recent years, machine learning, especially deep learning, has made it possible to understand scenes and separate components, and to correct only one component with high accuracy. This makes it possible to accurately correct each component for multiple components that are correlated. For example, a case will be described in which noise reduction processing and sharpening processing by blur correction are applied to an image containing noise components and blur components. By first correcting only the blur components with high accuracy using machine learning, an image containing only noise components and no blur components can be obtained. The noise components can be reduced by performing conventional noise reduction processing on the obtained image. However, even if the image does not contain blur components, blurring occurs due to noise reduction processing, so it is preferable to perform noise reduction processing using machine learning. Note that the appropriate degree of correction for sharpness and noise level changes depending on the imaging device manufacturer's drawing intention and the user's editing intention, so it is preferable to be able to adjust the correction strength. It is also preferable to be able to adjust the correction strength because artifacts may occur due to the correction processing depending on the scene. Artifacts include blurring caused by noise reduction processing, undershooting caused by sharpening processing, ringing, etc., and may occur depending on the scene even in high-precision correction processing using machine learning. For example, artifacts are likely to occur in scenes where a light source is reflected or the subject is high contrast. When adjusting the correction intensity in this way, the blur correction intensity is adjusted so that some of the blur components remain. Therefore, the noise reduction processing executed subsequently can be performed with high precision without being affected by the blur components by using machine learning.

As described above, each component can be corrected by performing correction of multiple components one by one. When the correction intensity of the previous stage is changed, the correction process of the subsequent stage needs to be executed again, but machine learning generally has a large calculation load and requires re-execution, making it difficult to adjust the correction intensity. Therefore, in the present invention, each component is acquired as a corresponding residual map. By adjusting the correction intensity based on the residual map, a corrected image in which each component is corrected with an appropriate intensity can be acquired. Since it is necessary to acquire each component separately, it is necessary to use a machine learning model for each. When correcting multiple components that are correlated, even if one component is corrected without changing the other component, the original size of the other component changes. Therefore, in the present invention, one residual map is corrected based on the other residual map. As a result, a natural corrected image can be acquired while adjusting the correction intensity for multiple components that are correlated. The residual maps before and after correction have different distributions. Here, the fact that the two residual maps have different distributions means that they are not in a relationship in which a uniform value is multiplied or an offset is added to each pixel. This is because the correction is not a uniform process for the entire screen, but a process for each pixel or image region, and the value of one component for each pixel or image region corrects the other component. This is because the correlation between components in the present invention is local, and one component is affected by the local value of the other component. In addition, by being able to separate and acquire each component as a residual map, it is possible to perform physically appropriate correction rather than empirical. There may be a physical order to multiple correlated components, such as when a subject whose brightness has been changed by a light source becomes blurred due to the optical characteristics of the imaging system, and noise is generated by sensing the blurred light. Since a component that occurs later physically is affected by a component that occurs earlier, it is preferable to correct the residual map corresponding to the component that occurs later based on the residual map corresponding to the component that occurs earlier.

Note that with machine learning, it is possible to learn the latter correction assuming the former processing, rather than learning to correct only one component in the former correction. In this case, since learning is based on the change in the component to be corrected in the latter, there is no need to modify the residual map. However, learning in response to the presence or absence of former processing or changes in correction strength increases the learning conditions, leading to more learning work, an increase in data volume, and deterioration of correction performance. Also, if there are changes to the former processing or if new processing is to be added, the latter processing must be reviewed.

In this embodiment, an image processing system is described that uses a multi-layer neural network (machine learning model) to learn and execute correction of a blur component (first component) derived from the optical characteristics of an imaging system and correction of a noise component (second component). The blur component and the noise component correspond to different tasks of image correction. Note that the present invention is not limited to image processing that combines correction of a blur component and correction of a noise component, and can be applied to other types of image processing.

Figure 1 is a block diagram of an image processing system 100 according to this embodiment. Figure 2 is an external view of the image processing system 100. The image processing system 100 includes a learning device 101, an imaging device 102, an image estimation device (image processing device) 103, a display device 104, a recording medium 105, an output device 106, and a network 107.

The learning device 101 has a memory unit 101a, an acquisition unit 101b, a generation unit 101c, and an update unit 101d.

The imaging device 102 has an optical system 102a and an imaging element 102b. The optical system 102a collects light incident on the imaging device 102 from the subject space. The imaging element 102b receives (photoelectrically converts) the optical image (subject image) formed via the optical system 102a to obtain an image. The imaging element 102b is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The image obtained by the imaging device 102 includes blur due to aberration and diffraction of the optical system 102a and noise due to the imaging element 102b.

The image estimation device 103 has a storage unit 103a, an acquisition unit (first acquisition unit) 103b, and a correction unit (modification unit, second acquisition unit) 103c. The image estimation device 103 acquires a captured image from the imaging device 102, and acquires a blur residual map (first residual map) corresponding to the blur component and a noise residual map (second residual map) corresponding to the noise component using at least one machine learning model. Then, the noise residual map is corrected based on the captured image (input image) and the blur residual map to acquire a corrected noise residual map (third residual map). Then, an estimated image (corrected image, output image) in which blur and noise are corrected with an appropriate strength is generated based on the captured image, the blur residual map, and the corrected noise residual map. The image estimation device 103 has a function of performing development processing and other image processing as necessary.

A multi-layer neural network is used to obtain the residual map, and weight information (machine learning model weights) is read from the memory unit 103a. The weight information is learned by the learning device 101. The image estimation device 103 reads the weight information from the memory unit 101a via the network 107 in advance and stores it in the memory unit 103a. The weight information may be the weight's numerical value itself, or may be information in an encoded format. Details regarding learning the weights and obtaining each residual map using the weights will be described later.

The corrected image is output to at least one of the display device 104, the recording medium 105, and the output device 106. The display device 104 is, for example, a liquid crystal display or a projector. The user can perform editing work while checking the image being processed via the display device 104. The recording medium 105 is, for example, a semiconductor memory, a hard disk, or a server on a network. The output device 106 is, for example, a printer.

The weight (weight information) learning method (method of manufacturing a trained model) executed by the learning device 101 of this embodiment will be described below with reference to Figs. 3 and 4. Fig. 3 is a flowchart related to weight learning. The processing of each step in Fig. 3 is mainly executed by the acquisition unit 101b, the generation unit 101c, or the update unit 101d of the learning device 101. Fig. 4 is a diagram showing the flow of learning the weights of a neural network.

In step S101, the acquisition unit 101b acquires an original image (subject image). In this embodiment, the original image is a high-resolution (high-quality) image with little blurring due to aberration or diffraction of the optical system 102a. A plurality of original images are acquired, and are images of various subjects, that is, images having edges, textures, gradations, flat parts, etc. of various strengths and directions. The original image may be a real-life image or an image generated by CG (Computer Graphics). In particular, when a real-life image is used as the original image, since blurring has already occurred due to aberration or diffraction, the effect of blurring can be reduced by shrinking the image, and a high-resolution (high-quality) image can be obtained. Note that if the original image contains sufficient high-frequency components, shrinking may not be necessary.

It is preferable that the original image has a signal value higher than the brightness saturation value of the image sensor 102b. This is because there are actual subjects that do not fall within the brightness saturation value when photographed by the image capture device 102 under specific exposure conditions. When using a real image as the original image, it can be obtained by HDR photography or by photographing with an image capture device that has a higher dynamic range than the image capture device 102. When using an image photographed with an image capture device that has the same dynamic range as the image capture device 102 as the original image, it is also possible to obtain a high signal value by proportionally multiplying the signal value. However, it is preferable to do this within a range where the reduction in gradation due to proportional multiplication does not affect the learning results. In addition, the original image may have noise components. In this case, the noise contained in the original image can be considered to be the subject, so the noise in the original image is not a particular problem.

In step S102, the acquisition unit 101b acquires the blur used to perform the imaging simulation described later. First, the acquisition unit 101b acquires the imaging conditions corresponding to the lens state (zoom, aperture, and focal distance state) of the optical system 102a. Then, the acquisition unit 101b acquires the blur determined by the imaging conditions and the screen position. Here, the blur is the PSF (point spread function) or OTF (optical transfer function) of the optical system 102a. The blur can be acquired by optical simulation or measurement in the optical system 102a. Note that the lens state, image height, azimuth aberration, and diffraction blur that differ for each original image are acquired. This makes it possible to perform imaging simulations corresponding to multiple imaging conditions, image heights, and azimuths. In addition, if necessary, a component such as an optical low-pass filter included in the imaging device 102 may be added to the blur to be applied.

In step S103, the generating unit 101c generates learning data that combines correct answer data consisting of correct answer patches (correct answer images) and training data consisting of training patches (training images). The correct answer patches and training patches are changed depending on the function or effect to be learned, and the corresponding images can be used as the correct answer patches and training patches. Images with different first and second components are generated as the images used for the correct answer patches and training patches. In this embodiment, a component-free patch generated from the original image and a blurred patch are generated by adding the blur component added in step S102 to the component-free patch. In addition, a noise patch and a noise-blurred patch are generated by adding a noise component to the component-free patch and the blurred patch. A plurality of component-free patches and blurred patches are generated, and one or more patches are generated corresponding to one original image. In this embodiment, the component-free patch and the blurred patch are images in which the same subject is captured. In this embodiment, the component-free patch and the blurred patch are used as the correct answer patch and the training patch, respectively. Then, a collection of multiple combinations of these is used as training data to train a machine learning model that acquires a blur residual map for correcting the aberration and diffraction of the optical system 102a. Therefore, the component-free patch needs to be an image with less blur compared to the blur patch. However, as described below, there may be cases where correction is not required depending on the conditions of the original image, so the training data may include cases where the component-free patch and the blur patch are the same image.

Note that a patch refers to an image having a predetermined number of pixels (e.g., 64 x 64 pixels). The number of pixels of the correct patch and the training patch does not necessarily have to be the same. In this embodiment, mini-batch learning is used to learn the weights of the multi-layered neural network. For this reason, in step S103, multiple pairs of correct patches and training patches are generated. However, the present invention is not limited to this, and online learning or batch learning may also be used.

In this embodiment, a plurality of pairs of component-free images and blurred images with relatively different effects of blurring due to aberration and diffraction are generated by performing an imaging simulation using a plurality of original images stored in the storage unit 101a as subjects. At this point, the component-free images and blurred images have the same or larger number of pixels as the number of pixels of the patch used as learning data. Then, a plurality of component-free patches and blurred patches are obtained by extracting partial areas of a specified pixel size at the same position from the plurality of pairs of component-free images and blurred images. In this embodiment, the original image is an undeveloped RAW image, and the component-free patches and blurred patches are also RAW images. However, the present invention is not limited to this, and may be images after development. In addition, the position of the partial area refers to the center of the partial area. Note that in this embodiment, the component-free patches and blurred patches are obtained by the above method, but the present invention is not limited to this.

In addition, since noise occurs in the image sensor 102b, a random number corresponding to the noise characteristics of the image sensor 102b that adds noise to the learning data may be generated and added, and shooting conditions such as ISO sensitivity may be taken into consideration. By adding the same noise to the blurred image and the component-free image, a noise blurred image (noise blurred patch) and a noise image (noise patch) are obtained. The noise to be added is represented by σ(x, y)·r(x, y) and satisfies the following formula (1).
σ ² (x,y)=[k1(S1(x,y)-OB)+k0)]×ISO/100 (1)
(x, y) is a two-dimensional spatial coordinate, and S1(x, y) is a signal value of a pixel at the coordinates (x, y) of the blurred patch before noise is added. r(x, y) is a numerical value at the coordinates (x, y) of a random number map with a standard deviation of 1, and σ(x, y) is the standard deviation of noise (σ ² (x, y) is the variance). OB is a signal value of optical black (black level image), ISO is ISO sensitivity, and k1 and k0 are a proportionality coefficient and a constant for the signal value at ISO sensitivity 100. The proportionality coefficient k1 represents the influence of shot noise, and the constant k0 represents the influence of dark current and readout noise. The values of k1 and k0 are determined by the noise characteristics of the image sensor 102b. As a result, a common noise is added to corresponding pixels (corresponding pixels) of the component-free patch and the blurred patch. A corresponding pixel is a pixel that captures the same position in the subject space, or a pixel that is at the same position in the component-free patch and the blurred patch. When various ISO sensitivities of the image sensor 102b are supported, noises of different ISO sensitivities are added to the multiple component-free patches 201 and blur patches 202.

When reducing an actual image to create the original image, the order of reduction and blurring may be reversed. If blurring is performed first, the sampling rate of the blur must be finer to take into account the reduction. In the case of a PSF (point spread function), the spatial sampling points should be finer, and in the case of an OTF (optical transfer function), the maximum frequency should be increased.

It is preferable that the blur to be applied does not include distortion. This is because if distortion is large during sharpening processing (aberration correction processing, blur component correction), the position of the subject will change, and the subject may be different between the component-free patch and the blur patch. For this reason, the blur residual map used in this embodiment does not include distortion. Distortion is corrected separately after blur correction using bilinear interpolation, bicubic interpolation, etc.

In step S104, the generation unit 101c inputs the noise blur patch as input data 212 (training patch, training image) in FIG. 4 to a multi-layer neural network, and generates an estimated patch (estimated image) 213. For mini-batch learning, the generation unit 101c generates estimated patches 213 corresponding to multiple pieces of input data 212. FIG. 4 shows the flow from step S104 to step S105.

In this embodiment, a noise patch is used as the correct answer data 211. The estimated patch 213 is a noise blur patch with blur corrected, and ideally matches the correct answer data (correct answer patch, correct answer image) 211. The neural network outputs an estimated residual map 214 that corresponds to the difference between the input data 212 (noise blur patch in this embodiment) and the correct answer data 211 (noise patch in this embodiment). The estimated residual map 214 is an estimated blur residual map. Note that in this embodiment, the neural network configuration shown in FIG. 3 is used, but the present invention is not limited to this.

CN in FIG. 3 represents a convolutional layer, and DC represents a deconvolutional layer. In both CN and DC, the sum of the input, the convolution of the filter, and the bias is calculated, and the result is nonlinearly transformed by the activation function. The initial values of each component of the filter and the bias are arbitrary, and in this embodiment, they are determined by random numbers. For example, ReLU (Rectified Linear Unit) or a sigmoid function can be used as the activation function. The output of each layer except the final layer is called a feature map. The skip connections 222 and 223 combine the feature maps output from discontinuous layers. The combination of the feature maps may be performed by taking the sum of each element, or may be concatenated in the channel direction. In this embodiment, the sum of each element is taken. The skip connection 221 takes the sum of the estimated residual map 214 estimated from the input data 212 and the input data 212 to generate an estimated patch 213. An estimated patch 213 is generated for each of the multiple input data 212.

In step S105, the update unit 101d updates the weights of the neural network based on the error between the estimated patch 213 and the ground truth data 211. The weights include the filter components and biases of each layer. Backpropagation is used to update the weights, but the present invention is not limited to this. For mini-batch learning, the error between multiple noise patches input as the ground truth data 211 and the corresponding estimated patches 213 is calculated, and the weights are updated. For example, the L2 norm or the L1 norm may be used as the loss function.

In step S106, the update unit 101d determines whether the weight learning is complete. Completion can be determined by whether the number of iterations of learning (weight update) has reached a specified value, or whether the amount of change in weight at the time of update is smaller than a specified value. If it is determined to be incomplete, the process returns to step S104, and multiple new noise patches and noise blur patches are obtained. On the other hand, if it is determined to be complete, the learning device 101 (update unit 101d) ends the learning and stores the weight information in the storage unit 101a. By learning the noise patch and noise blur patch to which the same random noise has been added as the correct answer data 211 and the input data 212, the neural network can learn by separating the subject, the blur component, and the noise component. Therefore, it is possible to obtain a blur residual map corresponding to the blur component of the subject only while suppressing noise fluctuation.

In this embodiment, the estimated error is obtained from the noise patch and the estimated patch 213 used in the correct answer data 211, but the estimated error may be obtained from the estimated residual map 214 and the correct answer of the blur residual map without outputting the estimated patch 213. The correct answer of the blur residual map is the difference between the noise patch and the noise blur patch, and is equal to the difference between the component-free patch and the blur patch. In this case, it is possible to train a machine learning model that directly outputs the blur residual map estimated as the estimated residual map 214.

In addition, although the present embodiment describes the learning of a machine learning model that acquires a blur residual map, a machine learning model that acquires a noise residual map can also be trained in a similar manner. Specifically, in FIG. 4, the input data 212 remains a noise blur patch, and a blur patch is used instead of a noise patch as the correct answer data 211. In this case, by learning using a patch in which the blur component is unchanged and only the noise component is different, the neural network can learn by separating the subject, the blur component, and the noise component. Therefore, it is possible to acquire a noise residual map that corresponds to the noise component while suppressing changes in the blur component.

In this case, too, an estimated error may be obtained from the estimated residual map 214 and the correct solution of the noise residual map without outputting the estimated patch 213. The correct solution of the noise residual map is the difference between the noise blur patch and the blur patch, which is equal to the difference between the noise patch and the component-free patch. In this case, a machine learning model can be trained that directly outputs the noise residual map estimated as the estimated residual map 214.

If the correct answer for the blur residual map is obtained from the difference between the component-free patch and the blur patch, and the correct answer for the noise residual map is obtained from the difference between the noise patch and the component-free patch, the noise blur patch is not used. Therefore, in this case, it is not necessary to obtain the noise blur patch.

In addition, in this embodiment, the machine learning model that obtains the blur residual map and the machine learning model that obtains the noise residual map are trained separately, so each may have a different network configuration.

In addition, since neural networks can learn by separating the subject, blur components, and noise components, a common network (one network) may output both the blur residual map and the noise residual map. In this case, since learning and estimation in the machine learning model can be done only once, the computational load can be reduced.

The generation of a corrected image (estimated image, output image) performed by the image estimation device 103 will be described below with reference to Figs. 5 and 6. Fig. 5 is a flowchart related to the generation of a corrected image. Fig. 6 is a block diagram of an image processing flow. The processing of each step in Fig. 5 is mainly performed by the acquisition unit 103b and the correction unit 103c of the image estimation device 103.

In step S201, the acquisition unit 103b acquires the input image 401 and weight information. The input image 401 is an undeveloped RAW image, and in this embodiment, is transmitted from the imaging device 102. The weight information is the weights of the machine learning model that acquires the blur residual map and the machine learning model that acquires the noise residual map, which are transmitted from the learning device 101 and stored in the storage unit 103a.

In step S202, the correction unit 103c inputs the input image 401 to a machine learning model that acquires the blur residual map acquired in step S201 to acquire a blur residual map (first residual map) 402. The correction unit 103c also inputs the input image 401 to a machine learning model that acquires a noise residual map to acquire a noise residual map (second residual map) 403.

In step S203, the correction unit 103c obtains a correction intensity using the blur residual map and a correction intensity using the noise residual map. The correction intensity can be determined by using the correction ratio of each component. In this embodiment, a predefined value is used, but a value specified by the user may also be obtained.

In step S204, the correction unit 103c corrects the noise residual map 403 based on the input image 401 and the blur residual map 402 to obtain a modified noise residual map (third residual map) 404. First, the correction unit 103c adds the blur residual map 402 to the input image 401 by multiplying it by the correction ratio of the blur component obtained in step S203. For example, if the correction ratio is 0.5, an intermediate corrected image 410 in which the blur component is corrected by 50% is obtained. Next, the standard deviations σ1(x,y) and σ2(x,y) of noise are obtained when the signal value S1(x,y) of formula (1) is the input image 401 and when the intermediate corrected image 410 is used. The noise residual map 403 corresponds to the noise component contained in the input image 401. The modified noise residual map 404 is obtained by multiplying the noise residual map 403 by the ratio σ2(x,y)/σ1(x,y) for each pixel. The modified noise residual map 404 corresponds to the noise components contained in the intermediate corrected image 410. Since the luminance values of the input image 401 and the values of the blur residual map usually differ for each screen position, the multiplication ratio differs for each pixel. Therefore, the modified noise residual map 404 is multiplied by a different coefficient for each pixel of the noise residual map 403, resulting in a different distribution. This makes it possible to obtain a residual map that enables natural correction that reflects the correlation of each component for each pixel.

In step S205, the correction unit 103c multiplies the modified noise residual map 404 by the correction ratio of the noise component obtained in step S203 and adds the result to the intermediate corrected image 410 to obtain an estimated image 405. Since the obtained image is a RAW image, development processing is performed as necessary.

In this embodiment, the correction strength of each component may be adjustable while viewing the estimated image.

Furthermore, the configuration of this embodiment is not limited to the above. For example, the correction intensity may be obtained at any time before the correction intensity of each component is used.

In addition, in this embodiment, the estimated image 405 is obtained by adding the intermediate corrected image 410 to the corrected noise residual map 404 multiplied by the correction ratio, but the present invention is not limited to this. The estimated image 405 may be obtained by adding the blurred residual map 402 and the corrected noise residual map 404 to the input image 401 according to the correction ratio of each component.

In addition, both the blur residual map 402 and the noise residual map 403 may be output using a common machine learning model (one machine learning model).

In addition, the signal value S1(x,y) when obtaining the standard deviation σ1(x,y) is the input image 401, but it may be an image after noise components have been corrected. In other words, an image obtained by adding a noise residual map to the input image 401 with a correction ratio of 1 may be used. This method can improve accuracy, especially when the noise components are large.

In addition, in this embodiment, each component is corrected (removed) by adding a residual map, but it is also possible to define a code that can remove each component by subtracting the residual map.

Processing may also be performed according to the image processing flow of FIG. 7. FIG. 7 is a block diagram of the image processing flow of the modified example. First, the input image 401 is input to a machine learning model (second learning model) that has learned about noise components to obtain a noise-corrected image 411 and a noise residual map 403. The noise-corrected image 411 is an image obtained by subtracting 100% of the acquired noise components from the input image 401. Next, the noise-corrected image 411 is input to a machine learning model (first learning model) that has learned about blur components to obtain a noise-blur-corrected image 412 and a blur residual map 402. The noise-blur-corrected image 412 is an image obtained by subtracting 100% of the acquired blur components from the noise-corrected image 411. Next, the blur residual map 402 is multiplied by (1-blur component correction ratio) and subtracted from the noise-blur-corrected image 412 to weaken the correction strength of the blur components, and an intermediate corrected image 413 is obtained. As a result, the intermediate corrected image 413 is a corrected image that reflects the correction ratio of the blur component. The noise residual map 403 corresponds to the noise component contained in the input image 401. As in this embodiment, the modified noise residual map 404 is obtained by multiplying the ratio of each standard deviation of noise to the luminance value of the noise corrected image 411 and the intermediate corrected image 413. The correction strength of the noise component is weakened by multiplying the modified noise residual map 404 by (1 - blur component correction ratio) and subtracting it from the intermediate corrected image 413, and the process is completed by obtaining an estimated image 405.

According to this embodiment and the modified example, even if the correction strength of each component is adjusted, there is no need to perform the calculation of the machine learning model again. When adjusting the intensity of blur correction, it is sufficient to execute the process again from the step of acquiring the corrected noise residual map based on the luminance values before and after blur correction. Therefore, the calculation load can be significantly reduced, and it is also possible to adjust the correction strength while viewing the image.

In this embodiment, we dealt with the correction of blur components resulting from the optical characteristics of the imaging system, but the present invention is also effective for blur caused by other factors (defocus, blur, etc.). By changing the blur added to the blur patch during learning to defocus, blur, etc., it is possible to separate blur caused by these factors from noise components and obtain a residual map.

The present invention can also be applied to the conversion of defocus blur (background blur) instead of the correction of the blur component. The conversion of defocus blur is a process of converting the defocus blur in a captured image into a shape and distribution desired by the user. The defocus blur in a captured image includes chipping due to vignetting, two-line blur, annular patterns due to cutting marks on an aspheric lens, and central occlusion due to a catadioptric optical system. These defocus blurs are converted into a shape and distribution desired by the user (for example, a flat circle or a normal distribution function) by a neural network. A neural network that realizes the conversion of defocus blur can be trained by the following method. For the same original image, a captured equivalent image with the defocus blur generated in the captured image and an ideal equivalent image with the defocus blur desired by the user are generated for multiple defocus amounts. However, since it is desired that the conversion of defocus blur does not cause any change to the subject at the focal distance, a captured equivalent image and an ideal equivalent image with a defocus amount of zero are also generated. From the generated multiple captured equivalent images and ideal equivalent images, multiple first defocus patches and second defocus patches are extracted, and noise is added to these to obtain noisy first defocus patches and noisy second defocus patches. By performing learning similar to that in this embodiment, the correction components of the defocus blur conversion can be separated from the noise components and obtained as a residual map.

The present invention can also be applied to lighting conversion (relighting) instead of blur component correction. A neural network that realizes lighting conversion can be trained in the following manner. A captured equivalent image is generated by rendering an original image, which is the same normal map, in a light source environment assumed for the captured image. Similarly, an ideal equivalent image is generated by rendering the normal map in a light source environment desired by the user. A plurality of first lighting patches and second lighting patches are extracted from the captured equivalent image and the ideal equivalent image, respectively, and noise is added to these to obtain a noisy first lighting patch and a noisy second lighting patch. The first lighting corresponds to the lighting before relighting, and the second lighting corresponds to the lighting after relighting. By performing learning similar to that of this embodiment, the relighting component can be separated from the noise component and obtained as a residual map.

The present invention can also be applied as relighting instead of correcting blur components, and as correcting blur components instead of correcting noise components. Blurring caused by the optical characteristics of the imaging system, defocus, and shaking causes blurring that spreads to surrounding pixels depending on the brightness of the subject. Therefore, when the brightness of the subject changes due to lighting, the original blur also changes at the same time. In a RAW image, the brightness of the subject and the blur components change in a proportional relationship. In this embodiment, the standard deviation change of noise is obtained based on the brightness change before and after shake correction, and a modified noise residual map is obtained by multiplying the ratio of the standard deviation change. Similarly, the blur residual map is modified for each pixel based on the ratio of the brightness values before and after light source correction by the relighting component. This results in a modified blur residual map. A neural network that separates and obtains blur components and relighting components can be trained in the following manner. A captured equivalent image is generated by rendering the original image, which is the same normal map, in a light source environment assumed in the captured image. Similarly, an ideal equivalent image is generated by rendering the normal map in a light source environment desired by the user. A plurality of first lighting patches and second lighting patches are extracted from the captured equivalent image and the ideal equivalent image, respectively, and blurred first lighting patches and blurred second lighting patches are obtained by adding blur to these. Here, since the second component is the blur component, the blurred first lighting patch and blurred second lighting patch correspond to the noise blur patch and noise patch in this embodiment. By performing learning similar to that in this embodiment, the relighting component can be separated from the blur component and obtained as a residual map.

In addition, when acquiring the blur component and the relighting component to perform image correction, the residual map of the relighting component may be corrected based on the ratio of brightness values before and after the blur component correction.

The present invention can also be applied to three or more components. Even when there are three or more components, each component can be separated and obtained as a residual map, so the residual map of each component can be modified appropriately while taking into account the correlation between the components, just as in the case of two components. For example, when performing blur component correction, relighting, and noise component correction, first the blur residual map is modified for each pixel based on the ratio of luminance values before and after light source correction by the relighting component. After that, the noise component map is modified for each pixel based on the ratio of luminance values when neither light source correction by the relighting component nor blur component correction is applied to the luminance values when both are applied.

In this embodiment, the case where the learning device 101 and the image estimation device 103 are separate has been described, but the present invention is not limited to this. The learning device 101 and the image estimation device 103 may be integrated. In other words, learning (the process shown in FIG. 3) and estimation (the process shown in FIG. 5) may be performed within a single device.

With the above configuration, it is possible to obtain a natural corrected image by performing image correction while adjusting the correction strength corresponding to multiple correlated components.

The image processing system of this embodiment differs from the image processing system of the first embodiment in that the generation of the corrected image is performed by an image estimation unit in the imaging device.

Figure 8 is a block diagram of the image processing system 300 of this embodiment. Figure 9 is an external view of the image processing system 300. The image processing system 300 has a learning device 301 and an imaging device 302. The learning device 301 and the imaging device 302 are connected via a network 303. The learning device 301 has a memory unit 311, an acquisition unit 312, a generation unit 313, and an update unit 314, and learns weights (weight information) for acquiring a residual map in a neural network. The imaging device 302 has an optical system 321, an imaging element 322, an image estimation unit (image processing device) 323, a memory unit 324, a recording medium 325, a display unit 326, and a system controller 327. The imaging device 302 captures an image of a subject space to acquire an image, and acquires a blur residual map and a noise residual map from the image using the read weight information. The imaging device 302 generates a corrected image using a modified noise residual map obtained by modifying the noise residual map. The image estimation unit 323 has an acquisition unit (first acquisition unit) 323a and a correction unit (modification unit, second acquisition unit) 323b, and acquires each residual map using the weight information stored in the memory unit 324, and performs correction based on each residual map.

The weight information is learned in advance by the learning device 301 and stored in the memory unit 311. The imaging device 302 reads out the weight information from the memory unit 311 via the network 303 and stores it in the memory unit 324. The corrected image is stored in the recording medium 325. When an instruction to display the corrected image is issued by the user, the stored corrected image is read out and displayed on the display unit 326. Note that an image already stored in the recording medium 325 may be read out and performance deviation correction may be performed by the image estimation unit 323. The above series of controls are performed by the system controller 327.

The learning of the machine learning model performed by the learning device 301 is equivalent to the learning of the machine learning model described in Example 1.

The acquisition of the corrected image performed by the image estimation unit 323 is performed in the same manner as in the flowchart of FIG. 5. Instead of being performed by the acquisition unit 103b and the correction unit 103c in the first embodiment, it is mainly performed by the acquisition unit 323a and the correction unit 323b of the image estimation unit 323.

With the above configuration, it is possible to obtain a natural corrected image by performing image correction while adjusting the correction strength corresponding to multiple correlated components.

The image processing system of this embodiment differs from the image processing systems of embodiments 1 and 2 in that it has a processing device (computer) that transmits a captured image that is the subject of image processing to the image estimation device and receives a processed output image (corrected image, estimated image) from the image estimation device.

Figure 10 is a block diagram of an image processing system 600 of this embodiment. The image processing system 600 has a learning device 601, an imaging device 602, an image estimation device (image processing device) 603, and a processing device (computer) 604. The learning device 601 and the image estimation device 603 are, for example, servers. The processing device 604 is, for example, a user terminal (personal computer or smartphone), and is connected to the image estimation device 603 via a network 605. That is, the image estimation device 603 and the processing device 604 are configured to be able to communicate with each other. The image estimation device 603 is connected to the learning device 601 via a network 606. That is, the learning device 601 and the image estimation device 603 are configured to be able to communicate with each other.

The configuration of the learning device 601 is similar to that of the learning device 101 in the first embodiment, and therefore a description thereof will be omitted. Also, the configuration of the imaging device 602 is similar to that of the imaging device 102 in the first embodiment, and therefore a description thereof will be omitted.

The image estimation device 603 has a memory unit 603a, an acquisition unit (first acquisition unit) 603b, a correction unit (modification unit, second acquisition unit) 603c, and a communication unit (reception unit) 603d. The memory unit 603a, the acquisition unit 603b, and the correction unit 603c are similar to the memory unit 103a, the acquisition unit 103b, and the correction unit 103c of the image estimation device 103 in Example 1, respectively. The communication unit 603d has a function of receiving a request transmitted from the processing device 604, and a function of transmitting an output image generated by the image estimation device 603 to the processing device 604.

The processing device 604 has a communication unit (transmission unit) 604a, a display unit 604b, an image processing unit 604c, and a recording unit 604d. The communication unit 604a has a function of transmitting a request to the image estimation device 603 to cause the image estimation device 603 to execute processing on the captured image, and a function of receiving an output image processed by the image estimation device 603. The display unit 604b has a function of displaying various information. The information displayed by the display unit 604b includes, for example, the captured image to be transmitted to the image estimation device 603 and the output image received from the image estimation device 603. The image processing unit 604c has a function of performing image processing on the output image received from the image estimation device 603. The recording unit 604d records the captured image acquired from the imaging device 602, the output image received from the image estimation device 603, etc.

The image processing of this embodiment will be described below with reference to FIG. 11. FIG. 11 is a flowchart of the image processing.

The image processing in FIG. 11 is started when a command to start image processing is given by the user via the processing device 604. First, the operation of the processing device 604 will be described.

In step S701, the processing device 604 transmits a request for processing the captured image to the image estimation device 603. The method of transmitting the captured image to be processed to the image estimation device 603 is not important. For example, the captured image may be uploaded to the image estimation device 603 simultaneously with the processing of step S701, or may be uploaded to the image estimation device 603 before the processing of step S701. The captured image may also be an image stored on a server different from the image estimation device 603. In step S701, the processing device 604 may transmit ID information for authenticating a user together with the request for processing the captured image.

In step S702, the processing device 604 receives the output image generated in the image estimation device 603. The output image is an image in which each component of the captured image has been corrected using the residual map, as in Example 1.

Next, the operation of the image estimation device 603 will be described.

In step S801, the image estimation device 603 receives a request for processing the captured image sent from the processing device 604. The image estimation device 603 determines that processing for the captured image has been instructed, and executes the processing from step S802 onwards.

In step S802, the image estimation device 603 acquires weight information. The weight information is information (trained model) learned in the same manner as in Example 1. The image estimation device 603 may acquire the weight information from the learning device 601, or may acquire weight information previously acquired from the learning device 601 and stored in the memory unit 603a.

The processes from step S803 to step S806 are similar to the processes from step S202 to step S205 in FIG. 5 described in Example 1, and therefore will not be described.

In step S807, the image estimation device 603 transmits the output image to the processing device 604.

When the correction process is performed within the image estimation device 603 as in this embodiment, the processing load due to the correction process can be borne within the image estimation device 603, thereby reducing the processing capacity required of the processing device 604.

As described above, the image estimation device 603 may be configured to be controlled using a processing device 604 communicatively connected to the image estimation device 603 .
[Other Examples]
The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

103 Image estimation device 103b Acquisition unit 103c Correction unit

Claims (14)

obtaining a first residual map corresponding to the first component and a second residual map corresponding to the second component by inputting an input image to at least one machine learning model;
generating a third residual map based on the first residual map and the second residual map;
obtaining an output image based on the input image, the first residual map, and the third residual map. The image processing method according to claim 1, characterized in that the first component and the second component correspond to different tasks of image correction. the first residual map corresponds to a difference between the input image corrected for the first component and the input image;
3. The image processing method of claim 1, wherein the second residual map corresponds to a difference between the input image corrected for the second component and the input image. The image processing method according to any one of claims 1 to 3, characterized in that the at least one machine learning model is trained to output the first residual map that changes the first component without changing the second component, and is trained to output the second residual map that changes the second component without changing the first component. The image processing method according to claim 4, characterized in that the at least one machine learning model is composed of one machine learning model, and the one machine learning model is trained to output the first residual map that changes the first component without changing the second component, and is trained to output the second residual map that changes the second component without changing the first component. The image processing method according to claim 4, characterized in that the at least one machine learning model includes a first learning model trained to output the first residual map that changes the first component without changing the second component, and a second learning model trained to output the second residual map that changes the second component without changing the first component. The image processing method according to any one of claims 1 to 6, characterized in that the first component is at least one of a blur component, a scattering component due to fog, a relighting component, and a background blur component that corrects background blur. The image processing method according to any one of claims 1 to 7, characterized in that the second component is a noise component. The image processing method according to any one of claims 1 to 8, characterized in that the third residual map is obtained using luminance values of an image before and after correction using the first residual map. The image processing method according to claim 9, characterized in that the third residual map is obtained by multiplying a coefficient obtained for each pixel based on the luminance value by a value of a corresponding pixel in the second residual map. The image processing method according to any one of claims 1 to 10, characterized in that the output image is obtained by weighting the first residual map and the third residual map and adding them to an image based on the input image. a first acquisition unit that acquires a first residual map corresponding to a first component and a second residual map corresponding to a second component by inputting an input image to at least one machine learning model;
a correction unit that generates a third residual map based on the first residual map and the second residual map;
and a second acquisition unit that acquires an output image based on the input image, the first residual map, and the third residual map. An image processing system comprising the image processing device according to claim 12 and a control device capable of communicating with the image processing device,
the control device has a transmission means for transmitting a request for execution of processing on a captured image to the image processing device,
The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and generates the output image in response to the request. A program for causing a computer to execute the image processing method according to any one of claims 1 to 11.