JP7818966B2 - Image processing method, image processing device, image processing system, and program - Google Patents

Description

The present invention relates to an image processing method, an image processing device, an image processing system, and a program.

In image recognition or regression tasks, methods using machine learning models can achieve higher accuracy than theory-based methods that use assumptions and approximations. With theory-based methods, accuracy decreases due to elements ignored by assumptions and approximations. However, with methods using machine learning models, by training the machine learning model using training data that includes these elements, it is possible to achieve estimations that are consistent with the training data without assumptions or approximations, thereby improving the accuracy of the task.

Patent Document 1 discloses a method for sharpening blur in a captured image using a convolutional neural network (CNN), a type of machine learning model. Patent Document 1 also discloses a method for adjusting the strength of sharpening by performing a weighted average of the captured image and an estimated image (blur-sharpened image) based on the brightness saturation region.

Japanese Patent Application Laid-Open No. 2020-166628

The method disclosed in Patent Document 1 does not address the problems caused by the diffraction of light. When capturing an image of a bright subject with a large aperture value (F-number) of the optical system, airy disks and light beams occur due to the diffraction of light. When correcting for airy disks and light beams using a machine learning model, problems such as unnatural emphasis can occur due to insufficient accuracy in the training image or insufficient parameters in the machine learning model. Furthermore, the degree to which this problem is noticeable varies depending on the peripheral brightness value of the correction target in the captured image. For example, unnatural emphasis is noticeable in dark images such as night scenes, but is not noticeable in bright images captured outdoors during the day.

Furthermore, because the occurrence of airy disks and streaks of light depends on the aperture value of the optical system, the correction effect is reduced even for images captured at aperture values where no adverse effects occur. Furthermore, the effects are averaged according to a set weight without taking into account the differing conspicuousness of the adverse effects depending on the brightness value of the captured image. In other words, in dark images, the adverse effects are reduced around saturated areas, but in bright images, the correction effect around saturated areas is reduced more than necessary.

The present invention therefore aims to provide an image processing method that can maintain blur correction effects while suppressing the adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

An image processing method according to one aspect of the present invention includes the steps of : generating information regarding correction of a captured image based on the captured image obtained by imaging using an optical system ; generating a weight map based on information regarding the aperture value of the optical system and information regarding the brightness value of the captured image; and generating a first image based on the captured image, the information regarding the correction , and the weight map.

Other objects and features of the present invention are described in the following examples.

This invention provides an image processing method that can maintain blur correction effects while suppressing adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

FIG. 2 is an explanatory diagram of a machine learning model in the first embodiment. FIG. 1 is a block diagram of an image processing system according to a first embodiment. 1 is an external view of an image processing system according to a first embodiment. FIG. 10 is an explanatory diagram of adverse effects caused by sharpening in Examples 1 to 3. FIG. 1 is an explanatory diagram of an airy disk in Examples 1 to 3. FIG. 10 is an explanatory diagram of a beam of light in Examples 1 to 3. 1 is a flowchart of learning of a machine learning model in Examples 1 to 3. 10 is a flowchart of generating a model output in the first and second embodiments. 10 is a flowchart of adjusting the strength of sharpening in the first embodiment. 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. 4A and 4B are explanatory diagrams of a captured image and a saturation influence map in the first embodiment. FIG. 4 is an explanatory diagram of a weight map in the first embodiment. FIG. 4 is an explanatory diagram of a weight map in the first embodiment. FIG. 10 is a block diagram of an image processing system according to a second embodiment. FIG. 10 is an external view of an image processing system according to a second embodiment. 10 is a flowchart of adjusting the sharpening strength in the second embodiment. FIG. 10 is an explanatory diagram of a weight map in the second embodiment. FIG. 10 is a block diagram of an image processing system according to a third embodiment. FIG. 11 is an external view of an image processing system according to a third embodiment. 13 is a flowchart of model output and sharpening strength adjustment in the third embodiment.

Embodiments of the present invention will now be described in detail with reference to the drawings. In each drawing, identical components will be designated by the same reference numerals, and duplicate descriptions will be omitted.

Before describing the specific embodiments, the gist of the present invention will be explained. The present invention generates an estimated image from a captured image captured using an optical system, using a machine learning model to sharpen the blur caused by the optical system. A weight map is then generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and the captured image and estimated image are weighted-averaged. The weight map is used to determine the proportion of each image when weighting-averaging the captured image and the estimated image, and has continuous signal values. For example, if the value of the weight map determines the proportion of the captured image, a value of 0.5 results in an intensity-adjusted image that is a weighted average of the proportion of the captured image and the blur-sharpened image at 50%. Furthermore, if the value of the weight map is 1, the intensity-adjusted image is the captured image.

Blur caused by the optical system includes blur due to aberration, diffraction, defocus, the effects of optical low-pass filters, and degradation of the pixel aperture of the image sensor. Machine learning models include, for example, neural networks, genetic programming, and Bayesian networks. Neural networks include CNN (Convolutional Neural Network), GAN (Generative Adversarial Network), and RNN (Recurrent Neural Network).

Blur sharpening refers to the process of restoring frequency components of a subject that have been reduced or lost due to blur. When sharpening blur, if the aperture value of the optical system is increased to capture a high-brightness subject, airy disks and light beams occur due to the diffraction of light. If airy disks and light beams are corrected using a machine learning model, problems such as unnatural emphasis can occur due to insufficient precision in the training images or insufficient parameters in the machine learning model. More details on airy disks and light beams, insufficient precision in the training images, and insufficient parameters in the machine learning model will be provided later.

Furthermore, the degree to which this drawback is noticeable varies depending on the brightness of the image. For example, unnatural emphasis is noticeable in dark images such as night scenes, but is not noticeable in bright images taken outdoors during the day. Therefore, the present invention generates a weight map based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and performs a weighted average of the captured image and the estimated image. This makes it possible to maintain the blur correction effect while suppressing the drawbacks caused by the aperture value of the optical system and the luminance value of the captured image. Note that below, the stage in which the weights of the machine learning model are learned will be referred to as the learning phase, and the stage in which blur sharpening is performed using the machine learning model using the learned weights will be referred to as the estimation phase.

First, an image processing system according to a first embodiment of the present invention will be described. In this embodiment, the task performed by the machine learning model is to sharpen blur in a captured image containing luminance saturation (increasing the resolution of the captured image). The blur to be sharpened is caused by aberrations, diffraction, and an optical low-pass filter that occur in the optical system. However, the effects of the present invention can be similarly achieved when sharpening blur caused by pixel aperture, defocus, or shaking. This embodiment can also be applied to tasks other than blur sharpening, achieving similar benefits. Specifically, this includes tasks such as upsampling to increase the number of pixels in a captured image and transforming (shape conversion) defocus blur in a captured image. Examples of defocus blur transformation include converting bilinear blur to Gaussian blur or spherical blur. Bilinear blur has a point spread function (PSF) with separate peaks. This causes a subject that is actually a single line to appear doubly blurred when defocused. Sphere blur has a PSF with flat intensity. Gaussian blur has a PSF with a Gaussian distribution. Other defocus blurs that can be transformed include, for example, defocus blur caused by vignetting and ring-shaped defocus blur caused by pupil obstruction by a catadioptric lens or the like.

Figure 2 is a block diagram of the image processing system 100 in this embodiment. Figure 3 is an external view of the image processing system 100. The image processing system 100 has a learning device 101 and an image processing device 103, which are connected via a wired or wireless network. The image processing device 103 is connected to an imaging device 102, a display device 104, a recording medium 105, and an output device 106, either wired or wirelessly. A captured image of the subject space captured using the imaging device 102 is input to the image processing device 103. The captured image is blurred due to aberration and diffraction caused by the optical system 102a in the imaging device 102 and the optical low-pass filter of the image sensor 102b, and information about the subject is attenuated.

The image processing device 103 uses a machine learning model to perform blur sharpening on the captured image and generate a saturation influence map and a blur-sharpened image (model output). Details of the saturation influence map will be described later. The machine learning model is trained by the learning device 101, and the image processing device 103 acquires information about the machine learning model from the learning device 101 in advance and stores it in the storage unit 103a. The image processing device 103 also has the function of adjusting the intensity of blur sharpening by performing a weighted addition of the captured image and the blur-sharpened image. Details of the learning and estimation of the machine learning model and the adjustment of the intensity of blur sharpening will be described later. The user can adjust the intensity of blur sharpening while checking the image displayed on the display device 104. The intensity-adjusted blur-sharpened image is stored in the storage unit 103a or the recording medium 105 and, as necessary, output to an output device 106 such as a printer. The captured image may be grayscale or have multiple color components. It may also be an undeveloped RAW image or a developed image.

Next, with reference to Figures 4(A)-(C) and Figure 5, we will explain the adverse effects of optical diffraction that occur when blur sharpening is performed using a machine learning model. Figures 4(A) and 4(B) are cross-sectional views of the PSF (vertical axis represents signal value, horizontal axis represents spatial coordinates) when the aperture value of the optical system is increased, with different scales on the vertical axis. Figure 5 is a plan view of the PSF. Due to the optical diffraction phenomenon, a diffraction pattern is generated with a bright central area surrounded by dark concentric rings. This is called an Airy disk. While this diffraction pattern has a weak signal value, it can be visually confirmed in the captured image when a point light source or strong light is captured. In Figure 4(C), the dashed-dotted line 141 represents the ideal signal value obtained by sharpening the PSF and reducing the diffraction pattern, while the dotted line 142 represents the signal value obtained when the diffraction pattern is enhanced by sharpening. The dashed-dotted line 141 is preferable for blur sharpening. However, due to insufficient precision in the training images or insufficient parameters in the machine learning model, the diffraction pattern may be emphasized, as shown by the dotted line 142. This is because the edges of the diffraction pattern are mistakenly recognized as the subject and sharpened. Insufficient precision in the training images and insufficient parameters in the machine learning model will be discussed later.

Next, referring to Figure 6, we will explain light beams caused by the diffraction of light. Light beams refer to long, thin streaks of light that appear when capturing an image of a point light source or intense light. Figure 6 shows an example of light beams. The number of streaks depends on the number of aperture blades 143 in the optical system 102a. If the number of aperture blades 143 is odd, twice as many streaks will appear. On the other hand, if the number of aperture blades 143 is even, the number of streaks will be equal to the number of aperture blades 143. Figure 6 shows five aperture blades 143, resulting in ten light streaks 144. The sharpness of the light beams depends on the aperture value of the optical system. As the aperture value increases, the sharpness increases. Furthermore, even with the same aperture value, the appearance of the light beams varies depending on the type of optical system 102a. As with airy disks, insufficient precision in the training images or insufficient parameters in the machine learning model can cause unnatural enhancement of the light beams.

Next, we will explain in detail the lack of accuracy of the training images. When sharpening blur using a machine learning model, the accuracy of the sharpening depends on the accuracy of the training images. In other words, training images that accurately reproduce the blur of the optical system 102a that is the target of sharpening are required. However, if we attempt to comprehensively learn the blur at zoom positions, aperture values, and subject distances that the optical system 102a can assume during image capture, the number of training images becomes enormous. As a result, problems arise such as a decrease in the accuracy of individual blur sharpening and failure of learning to converge. For this reason, it is necessary to discretely learn the blur at zoom positions, aperture values, and subject distances that the optical system 102a can assume, and have the machine learning model predict the intermediate regions. However, in this case, sharpening the blur in the intermediate regions that are not included in the training images may result in the edges of the diffraction pattern being mistakenly recognized as the subject and sharpened.

Next, we will explain in detail the parameter deficiency of machine learning models. Machine learning models have multiple layers, and at each layer, a linear sum of the layer's input and weight is taken. If the machine learning model is a CNN, which uses the convolution of the input and filter as a linear sum (the value of each filter element corresponds to the weight, and may also include a sum with a bias), the number of filter layers corresponds to the number of parameters. In other words, parameter deficiency means an insufficient number of filter layers. Because there is a trade-off between the number of filter layers, training time, and processing speed, parameter deficiency can occur. Using a machine learning model trained using a method that uses a captured image and a corresponding brightness saturation map as input data for the machine learning model, and a method that generates a saturation influence map, it is possible to reduce the adverse effects, but it is difficult to completely eliminate them. The method using the brightness saturation map and the method generating a saturation influence map are each explained in detail.

Next, we will explain the brightness saturation map. A brightness saturation map is a map that represents brightness saturated areas in a captured image. In areas where brightness saturation occurs (brightness saturated areas), information about the structure of the subject space is lost, and false edges may appear at the boundaries between areas, making it impossible to extract correct features of the subject. Therefore, by inputting a brightness saturation map, the neural network can identify problematic areas such as those mentioned above, thereby preventing a decrease in estimation accuracy.

Next, we will explain the saturation influence map. Even when a brightness saturation map is used, the machine learning model may not make an accurate judgment. For example, if the region of interest is near a brightness saturated region, the machine learning model can determine that the region of interest is affected by brightness saturation because there is a brightness saturated region near the region of interest. However, if the region of interest is located far from the brightness saturated region, it is not easy to determine whether the region is affected by brightness saturation, and ambiguity increases. As a result, the machine learning model may make an incorrect judgment at a position far from the brightness saturated region. For this reason, when the task is to sharpen the blur, sharpening processing corresponding to a saturated blur image is performed on a non-saturated blur image. In this case, artifacts occur in the sharpened blur image, reducing the accuracy of the task. For this reason, it is preferable to use a machine learning model to generate a saturation influence map from a captured image in which blur has occurred.

A saturation influence map is a map (spatially arranged signal sequence) that represents the magnitude and range of signal values that have spread due to blurring of subjects in brightness-saturated areas of a captured image. By having a machine learning model generate a saturation influence map, the machine learning model can accurately estimate the presence and magnitude of the influence of brightness saturation in a captured image. By generating a saturation influence map, the machine learning model can execute the processing that should be performed on areas affected by brightness saturation and the processing that should be performed on other areas, respectively, in the appropriate areas. Therefore, by having a machine learning model generate a saturation influence map, task accuracy is improved compared to when saturation influence map generation is not required (only recognition labels and blur-sharpened images are generated directly from the captured image).

While the above two methods are effective, it is difficult to completely eliminate the adverse effects. Therefore, in this embodiment, a weight map is generated based on information about the aperture value of the optical system used to capture the captured image and information about the luminance value of the captured image, and a weighted average is calculated between the captured image and the estimated image. This makes it possible to maintain the blur correction effect while suppressing the adverse effects caused by the aperture value of the optical system and the luminance value of the captured image.

Next, with reference to Figure 7, we will explain the learning of the machine learning model executed by the learning device 101. Figure 7 is a flowchart of learning of the machine learning model. The learning device 101 has a memory unit 101a, an acquisition unit 101b, a calculation unit 101c, and an update unit 101d, and each step in Figure 7 is mainly executed by each unit of the learning device 101.

First, in step S101, the acquisition unit 101b acquires one or more original images from the storage unit 101a. The original image is an image having a signal value higher than the second signal value. The second signal value is a signal value corresponding to the luminance saturation value of the captured image. However, since the signal value may be normalized when input into the machine learning model, the second signal value does not necessarily have to match the luminance saturation value of the captured image. Since the machine learning model is trained based on the original image, it is desirable that the original image be an image having various frequency components (edges of different directions and intensities, gradations, flat areas, etc.). The original image may be a real-life image or CG (Computer Graphics).

Next, in step S102, the calculation unit 101c blurs the original image to generate a blurred image. The blurred image is an image input to the machine learning model during training and corresponds to the captured image during estimation. The blur to be added is the blur to be sharpened. In this embodiment, blur generated by the aberration and diffraction of the optical system 102a and the optical low-pass filter of the image sensor 102b is added. The shape of the blur caused by the aberration and diffraction of the optical system 102a varies depending on the image plane coordinates (image height and azimuth). It also varies depending on the magnification, aperture, and focus state of the optical system 102a. If you want to simultaneously train a machine learning model that sharpens all of these blurs, it is recommended to generate multiple blurred images using multiple blurs generated by the optical system 102a. Furthermore, in the blurred image, signal values exceeding a second signal value are clipped. This is done to reproduce the brightness saturation that occurs during the capture process of the captured image. If necessary, noise generated by the image sensor 102b may be added to the blurred image.

Next, in step S103, the calculation unit 101c sets a first region based on an image based on the original image and a signal value threshold. In this embodiment, a blurred image is used as the image based on the original image, but the original image itself may also be used. The first region is set by comparing the signal value of the blurred image with the signal value threshold. More specifically, the first region is defined as the region where the signal value of the blurred image is equal to or greater than the signal value threshold. In this embodiment, the signal value threshold is the second signal value. Therefore, the first region represents a brightness-saturated region of the blurred image (saturated region). However, the signal value threshold and the second signal value do not have to match. The signal value threshold may be set to a value slightly smaller than the second signal value (for example, 0.9 times).

Next, in step S104, the calculation unit 101c generates a first region image having the signal value of the original image in the first region. The first region image has a signal value different from that of the original image in regions other than the first region. More preferably, the first region image has a first signal value in regions other than the first region. In this embodiment, the first signal value is 0, but the invention is not limited to this. In this embodiment, the first region image has the signal value of the original image only in regions where the blurred image is saturated in brightness, and the signal value in other regions is 0.

Next, in step S105, the calculation unit 101c blurs the first region image and generates a saturation influence correct map. The blur applied is the same as the blur applied to the blurred image. As a result, a saturation influence correct map is generated, which is a map (spatially arranged signal sequence) that represents the magnitude and range of signal values that have expanded due to blur (deterioration) during imaging from the subject in the brightness-saturated region of the blurred image. In this embodiment, as with the blurred image, the saturation influence correct map is clipped with the second signal value, but clipping is not necessarily required.

Next, in step S106, the acquisition unit 101b acquires the correct model output. Since the task in this embodiment is blur sharpening, the correct model output is an image with less blur than the blurred image. In this embodiment, the correct model output is generated by clipping the original image with the second signal value. If the original image lacks high-frequency components, an image obtained by reducing the original image may be used as the correct model output. In this case, reduction is also performed in the same way when generating the blurred image in step S102. Furthermore, step S106 may be executed at any time after step S101 and before step S107.

Next, in step S107, the calculation unit 101c uses a machine learning model to generate a saturation influence map and model output based on the blurred image. In this embodiment, the machine learning model shown in FIG. 1 is used, but the invention is not limited to this. A blurred image 201 and a brightness saturation map 202 are input to the machine learning model. The brightness saturation map 202 is a map that indicates areas of the blurred image 201 where brightness is saturated (where the signal value is equal to or greater than a second signal value). For example, the brightness saturation map 202 can be generated by binarizing the blurred image 201 using the second signal value. However, the brightness saturation map 202 is not necessarily required. The blurred image 201 and the brightness saturation map 202 are linked in the channel direction and input to the machine learning model. However, the invention is not limited to this. For example, the blurred image 201 and the brightness saturation map 202 may each be converted into a feature map, and these feature maps may be linked in the channel direction. Furthermore, information other than the brightness saturation map 202 may be added to the input.

The machine learning model has multiple layers, and at each layer, a linear sum of the layer's input and weights is calculated. The initial values of the weights can be determined using random numbers or the like. In this embodiment, the machine learning model is a CNN that uses the convolution of the input and filter as the linear sum (the value of each filter element corresponds to the weight, and may also include a sum with a bias), but the invention is not limited to this. Furthermore, at each layer, nonlinear transformations are performed using activation functions such as ReLU (Rectified Linear Unit) and sigmoid functions as necessary. Furthermore, the machine learning model may have residual blocks or skip connections (also known as shortcut connections) as necessary. A saturation influence map 203 is generated as a result of passing through multiple layers (16 convolutional layers in this embodiment).

In this embodiment, the saturation influence map 203 is generated by taking the element-by-element sum of the output of layer 211 and the intensity saturation map 202, but the configuration is not limited to this. The saturation influence map may be generated directly as the output of layer 211. Alternatively, the saturation influence map 203 may be generated by performing any processing on the output of layer 211. Next, the saturation influence map 203 and the blurred image 201 are concatenated in the channel direction and input to a subsequent layer, and model output 204 is generated after passing through multiple layers (16 convolutional layers in this embodiment). The model output 204 is also generated by taking the element-by-element sum of the output of layer 212 and the blurred image 201, but the configuration is not limited to this. Note that in this embodiment, each layer performs convolution with 64 types of 3x3 filters (however, layers 211 and 212 perform convolution with the same number of filter types as the number of channels of the blurred image 201), but the configuration is not limited to this.

Next, in step S108, the update unit 101d updates the weights of the machine learning model based on the error function. In this embodiment, the error function is a weighted sum of the error between the saturation influence map 203 and the saturation influence correct map and the error between the model output 204 and the correct model output. MSE (Mean Squared Error) is used to calculate the error. Both weights are set to 1. However, the error function and weights are not limited to this. Backpropagation or the like may be used to update the weights. The error may also be calculated for the residual component. For the residual component, the error between the difference component between the saturation influence map 203 and the brightness saturation map 202 and the difference component between the saturation influence correct map and the brightness saturation map 202 is used. Similarly, the error between the difference component between the model output 204 and the blurred image 201 and the difference component between the correct model output and the blurred image 201 is used.

Next, in step S109, the update unit 101d determines whether learning of the machine learning model is complete. Completion of learning can be determined by, for example, whether the number of iterations of weight update has reached a predetermined number, or whether the amount of change in weight during update is smaller than a predetermined value. If it is determined in step S109 that learning is not complete, the process returns to step S101, and the acquisition unit 101b acquires one or more new original images. On the other hand, if it is determined that learning is complete, the update unit 101d ends learning and stores the configuration of the machine learning model and weight information in the storage unit 101a.

The above learning method allows the machine learning model to estimate a saturation influence map that represents the magnitude and range of signal values that have spread due to blurring of subjects in brightness-saturated areas of a blurred image (or captured image when estimated). Explicitly estimating the saturation influence map allows the machine learning model to sharpen the blur for both saturated and non-saturated blurred images in the appropriate areas, thereby suppressing the occurrence of artifacts.

Next, referring to Figure 8, we will explain the blur sharpening of a captured image using a trained machine learning model, which is executed by the image processing device 103. Figure 8 is a flowchart for generating model output. The image processing device 103 has a storage unit 103a, an acquisition unit 103b, and a sharpening unit 103c, and each step in Figure 8 is mainly executed by each unit of the image processing device 103.

First, in step S201, the acquisition unit 103b acquires a captured image and a machine learning model. Information about the configuration and weights of the machine learning model is acquired from the storage unit 103a.

Next, in step S202, the sharpening unit (first generation means) 103c generates information related to the correction using a machine learning model. In this embodiment, the information related to the correction is a blur-sharpened image (model output) in which the blur of the captured image has been sharpened from the captured image. Note that instead of a blur-sharpened image (an image obtained by correcting the captured image), a blur-sharpening correction component may be used. The machine learning model has the same configuration as during training as shown in FIG. 1. As with training, a brightness saturation map representing brightness-saturated areas of the captured image is generated and input, and a saturation influence map and model output are generated. As examples, FIGS. 10(A) and 11(A) show a blur-sharpened image (captured image), and FIGS. 10(B) and 11(B) show saturation influence maps. FIGS. 10(A) and (B) show a scene in which the average brightness value of the captured image is bright. FIGS. 11(A) and (B) show a scene in which the average brightness value of the captured image is dark. Note that the method of calculating the average brightness value will be described later.

Next, referring to Figure 9, we will explain the synthesis of a captured image and model output, which is performed by the image processing device 103. Figure 9 is a flowchart for adjusting the sharpening strength. Each step in Figure 9 is mainly performed by each unit of the image processing device 103.

First, in step S211, the acquisition unit 103b acquires the imaging state from the captured image. The imaging state is, for example, the aperture value (F-number) of the optical system 102a and the pixel pitch of the image sensor 102b. The appearance of the rays of light and airy disks in the captured image depends on the aperture value of the optical system 102a and the pixel pitch of the image sensor 102b. Specifically, the larger the aperture value of the optical system 102a and the smaller the pixel pitch, the more noticeable the rays of light and airy disks become. Therefore, a weight map is generated according to the aperture value and pixel pitch acquired in step S211.

Next, in step S212, the acquisition unit 103b acquires information related to the luminance values of the captured image. Here, the information related to the luminance values of the captured image is a statistical quantity related to the luminance values of the captured image, and is at least one of the mean, median, variance, or histogram of the luminance values of the captured image. Furthermore, the statistical quantity related to the luminance values of the captured image, which is information related to the luminance values of the captured image, may be related to the entire captured image, or statistics for each region obtained by dividing the captured image may be used.

In this embodiment, the average luminance value of the captured image is acquired as information regarding the luminance value of the captured image. However, when acquiring the average luminance value of the captured image, if the captured image contains many saturated areas, a large average luminance value may be acquired even for a dark image such as a night scene, and the image may be determined to be bright. Therefore, it is preferable to acquire the average luminance value of a second image obtained by excluding the saturation influence map from the captured image. In this case, information regarding the luminance value of the captured image is generated based on statistics regarding the luminance values of the second image obtained by excluding the saturation influence map from the captured image. By acquiring the average luminance value of the non-saturated areas in the second image, it is possible to appropriately determine whether the captured image represents a bright scene or a dark scene. Note that the use of a saturation influence map is not required; information regarding saturated areas will suffice. For example, a second image may be generated by excluding the luminance saturation map from the captured image regarding the luminance saturation of the captured image.

Next, in step S213, the sharpening unit 103c generates a weight map based on information related to the aperture value of the optical system 102a and information related to the luminance values of the captured image. In this embodiment, if the aperture value of the optical system 102a is F22 (predetermined aperture value) or greater and the pixel pitch of the image sensor 102b is less than 6 μm (predetermined pixel pitch), the sharpening unit 103c performs intensity adjustment. In other cases, the sharpening unit 103c generates a weight map in which all weights of the captured image are 0. Alternatively, the processing of steps S212 to S215 may be omitted without generating a weight map. The aperture value (predetermined aperture value) and pixel pitch threshold (predetermined pixel pitch) used to perform intensity adjustment can be set to any numerical value. For example, intensity adjustment may be performed if the aperture value of the optical system 102a is F16 or greater and the pixel pitch of the image sensor 102b is less than 4 μm. In this embodiment, the weight map may be generated based on the aperture value without considering the pixel pitch.

Furthermore, in this embodiment, the intensity adjustment is not limited to whether it is performed based on a predetermined aperture value (e.g., F22). For example, a weight map in which the weight of the captured image changes continuously (in stages) based on the aperture value may be used. In other words, if the aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value, the weight map contains data in which the weight of the captured image is greater for the second aperture value than for the first aperture value.

Next, we will explain the weight map based on brightness values. In this embodiment, the weight map is data in which the smaller the brightness value, the greater the weight of the captured image. For example, if the average brightness value of the captured image is used as information related to the brightness value of the captured image, the smaller the average brightness value, the greater the weight of the captured image. Specifically, the relationship between the weight of the weight map and the average brightness value is stored as a linear function, and a weight corresponding to the average brightness value of the captured image is obtained to generate the weight map.

For example, the weight of the captured image corresponding to the average luminance value obtained from the relational expression shown in FIG. 12 is used. FIG. 12 is an explanatory diagram of a weight map, with the horizontal axis representing the average luminance value and the vertical axis representing the weight of the captured image. However, the relationship between the average luminance value and the weight map adjustment value is not limited to this. While FIG. 12 shows the average luminance value of an undeveloped RAW image, the average luminance value after development may also be used. When calculating the average luminance value of a RAW image, it is preferable to subtract the signal value in the optical black region of the image sensor 102b before calculating the average luminance value. This makes it possible to calculate an average luminance value that is independent of the ISO sensitivity or the image sensor 102b. Furthermore, if the average luminance value is obtained for each region into which the captured image is divided, a weight may be obtained from the average luminance value of each region to generate a weight map. Details of this will be described later in Example 2.

In addition, even when the average, median, variance, or histogram of the brightness values of the captured image is obtained as a statistical quantity related to the signal values of the captured image, a weight map is generated so that the smaller the statistical quantity, the greater the weight of the captured image. For example, when a histogram of the brightness values of the captured image is used, a weight map is generated so that the smaller the center of gravity or peak of the histogram, the greater the weight of the captured image.

Next, in step S214 of FIG. 9 , the sharpening unit 103c adjusts the weight map based on information about saturated regions in the captured image. In this embodiment, the information about saturated regions is a saturation influence map, and not all RGB colors need to be saturated. The saturation influence map is normalized from 0 to 1 using a set signal value and used to adjust the weight map. Specifically, the normalized weight map is applied (multiplied) to the weight map generated in step S213. This causes the weights of the weight map generated in step S213 to affect the saturated region influence, making it possible to adjust the intensity only in saturated affected regions where rays of light and airy disks are more noticeable. Note that applying the saturation influence map is not required; the entire captured image may be subject to intensity adjustment. Using the saturation influence map makes it possible to adjust the correction intensity up to the saturated affected region. Note that instead of the saturation influence map, a brightness saturation map may be used, or a brightness saturation map blurred for each image height may be used.

As an example, the final weight maps are shown in Figures 13(A) and (B). In this embodiment, the weight map indicates that the larger the pixel value, the greater the weight of the captured image, and the smaller the pixel value, the smaller the weight of the captured image. Figure 13(A) shows a scene with a bright average luminance value, and Figure 13(B) shows a scene with a dark average luminance value. The weight of the captured image is large in the saturation affected area in Figure 13(B), which is an area where adverse effects are easily noticeable.

It is preferable that the weight map is further generated based on information regarding the optical performance of the optical system used to capture the captured image. Specifically, if the optical performance is low, the weight of the captured image is increased, and if the optical performance is high, the weight of the captured image is decreased. Information regarding optical performance can be calculated based on at least one of the zoom position, aperture diameter, or subject distance when the captured image was captured, and the magnitude and range of the PSF signal value for each image height of the optical system. Note that it is not necessary to use a PSF; any information regarding optical performance will suffice. For example, an optical transfer function may be used.

Next, in step S215 of FIG. 9, the sharpening unit (second generation means) 103c uses the weight map generated in step S214 to perform a weighted average (combine) of the captured image and the blur-sharpened image to generate the intensity-adjusted image 205.

With the above configuration, this embodiment can provide an image processing system that can maintain blur correction effects while suppressing adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

Next, an image processing system according to a second embodiment of the present invention will be described. In this embodiment, an average brightness value is obtained for each region into which a captured image is divided, and a weight corresponding to the average brightness value is obtained for each region to generate a weight map. Figure 14 is a block diagram of an image processing system 300 according to this embodiment. Figure 15 is an external view of the image processing system 300. The image processing system 300 has a learning device 301, an image capturing device 302, and an image processing device 303. The learning device 301 and the image processing device 303, and the image processing device 303 and the image capturing device 302 are each connected via a wired or wireless network.

The imaging device 302 has an optical system 321, an image sensor 322, a memory unit 323, a communication unit 324, and a display unit 325. The captured image is transmitted to the image processing device 303 via the communication unit 324. The image processing device 303 receives the captured image via the communication unit 332 and performs blur sharpening using information about the machine learning model configuration and weights stored in the memory unit 331. The information about the machine learning model configuration and weights is learned by the learning device 301, acquired in advance from the learning device 301, and stored in the memory unit 331. The image processing device 303 also has a function for adjusting the intensity of blur sharpening. A blur-sharpened image (model output) in which the blur of the captured image has been sharpened, and an intensity-adjusted image in which the intensity has been adjusted, are transmitted to the imaging device 302, stored in the memory unit 323, and displayed on the display unit 325.

The generation of training data and weight training (training phase) performed by the training device 301 and the sharpening of blur in captured images using a trained machine learning model (estimation phase) performed by the image processing device 303 are the same as in Example 1, and therefore will not be repeated.

Next, referring to Figure 16, we will explain the synthesis of a captured image and model output, which is performed by the image processing device 303. Figure 16 is a flowchart for adjusting the sharpening strength. Each step in Figure 16 is mainly performed by each unit of the image processing device 303.

First, in step S311, the acquisition unit 333 acquires the imaging state from the captured image. The imaging state includes, but is not limited to, the aperture value of the optical system 321 in the imaging device 302 and the pixel pitch state of the image sensor 322. In this embodiment, intensity adjustment is performed when the aperture value of the optical system 321 is F16 or greater and the pixel pitch of the image sensor 322 is 4 μm or greater. In other cases, when generating a weight map (described below), a weight map is generated in which all weights of the captured image are 0, and intensity adjustment is not performed. Alternatively, the processing of steps S312 to S314 may be omitted without generating a weight map.

Next, in step S312, the acquisition unit 333 acquires information related to the brightness values of the captured image. In this embodiment, the average brightness value is acquired for each region into which the captured image is divided, and a weight corresponding to the average brightness value is acquired for each region to generate a weight map. Figures 17(A) and (B) are explanatory diagrams of the weight map, with Figure 17(A) showing the captured image and Figure 17(B) showing the average brightness value for each region. In Figure 17(B), 2001 is a region with a low average brightness value, 2002 is a region with a high average brightness value, and 2003 is a region with an intermediate average brightness value.

Next, in step S313 of FIG. 16, the sharpening unit 334 generates a weight map based on information relating to the aperture value of the optical system 321 and the luminance value of the captured image. Note that the method for generating the weight map is the same as in Example 1, and therefore its description will be omitted. Next, in step S314, the sharpening unit 334 uses the weight map generated in step S312 to perform a weighted average (combination) of the captured image and the blur-sharpened image (model output) to generate the intensity-adjusted image 205.

With the above configuration, this embodiment can provide an image processing system that can maintain blur correction effects while suppressing adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

Next, an image processing system according to a third embodiment of the present invention will be described. Figure 18 is a block diagram of an image processing system 400 according to this embodiment. Figure 19 is an external view of the image processing system 400. The image processing system 400 includes a learning device 401, a lens device 402, an imaging device 403, a control device (first device) 404, an image estimation device (second device) 405, and networks 406 and 407.

The learning device 401 and the image estimation device 405 are each, for example, a server. The control device 404 is a device operated by a user, such as a personal computer or mobile terminal. The learning device 401 has a memory unit 401a, an acquisition unit 401b, a calculation unit 401c, and an update unit 401d, and learns the weights of a machine learning model that sharpens blur from an image captured using the lens device 402 and the imaging device 403. Note that the learning method, i.e., the generation of learning data and weight learning (learning phase) performed by the learning device 401, is the same as in Example 1, and therefore will not be described here.

The imaging device 403 has an imaging element 403a, which photoelectrically converts the optical image formed by the lens device 402 to obtain a captured image. The lens device 402 and imaging device 403 are detachable and can be combined with multiple types of each other. The control device 404 has a communication unit 404a, a display unit 404b, a memory unit 404c, and an acquisition unit 404d, and controls the processing to be performed on the captured image obtained from the imaging device 403 connected via a wired or wireless connection in accordance with user operations. Alternatively, the captured image taken by the imaging device 403 may be stored in advance in the memory unit 404c and then read out.

The image estimation device 405 has a communication unit 405a, an acquisition unit 405b, a storage unit 405c, and a sharpening unit 405d, and is configured to be able to communicate with the control device 404. The image estimation device 405 performs blur sharpening processing on a captured image in response to a request from the control device 404, which is connected via the network 406. The image estimation device 405 acquires learned weight information from the learning device 401, which is connected via the network 406, when estimating blur sharpening or in advance, and uses this information to estimate blur sharpening on a captured image. The estimated image after blur sharpening estimation is subjected to sharpening strength adjustment, and then transmitted again to the control device 404, where it is stored in the storage unit 404c and displayed on the display unit 404b.

Next, referring to Figure 20, we will explain the blur sharpening of captured images performed by the control device 404 and image estimation device 405. Figure 20 is a flowchart of model output and sharpening intensity adjustment. Each step in Figure 20 is mainly performed by each section of the control device 404 or image estimation device 405.

First, in step S401, the acquisition unit 404d of the control device 404 acquires the captured image and the sharpening strength specified by the user. Next, in step S402, the communication unit (transmission means) 404a transmits the captured image and a request to execute blur sharpening estimation processing to the image estimation device 405.

Next, in step S403, the communication unit (receiving means) 405a of the image estimation device 405 receives and acquires the captured image and processing request transmitted from the control device 404. Next, in step S404, the acquisition unit 405b acquires learned weight information corresponding to the captured image from the storage unit 405c. The weight information is read out in advance from the storage unit 401a and stored in the storage unit 405c. Next, in step S405, the sharpening unit 405d uses the machine learning model to generate a blur-sharpened image (model output) from the captured image, in which the blur of the captured image has been sharpened. The machine learning model has the same configuration as during learning, as shown in Figure 1. As with learning, a brightness saturation map representing brightness-saturated regions of the captured image is generated and input, and a saturation influence map and model output are generated.

Next, in step S406, the sharpening unit 405d generates a weight map. The method for generating the weight map is the same as in Example 1. The default weight map is adjusted according to the sharpening strength specified by the user. Note that a pre-adjusted weight map may be stored within an adjustable strength range. Next, in step S407, the sharpening unit 405d combines the captured image with the blur-sharpened image (model output) based on the weight map. Next, in step S408, the communication unit 405a transmits the combined image to the control device 404. Next, in step S409, the communication unit 404a of the control device 404 acquires the estimated image transmitted from the image estimation device 405.

With the above configuration, this embodiment can provide an image processing system that can maintain the blur correction effect while suppressing adverse effects caused by the aperture value of the optical system and the luminance value of the captured image.
(Other Examples)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that realizes one or more functions. The image processing device can be any device that has the image processing function of the present invention, and can be realized in the form of an imaging device or a PC.

Each embodiment provides an image processing method, image processing device, image processing program, and storage medium that can maintain blur correction effects while suppressing adverse effects caused by the aperture value of the optical system and the brightness value of the captured image.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments and various modifications and variations are possible within the scope of the invention.

103 Image processing device 103c Sharpening unit (first generating means, second generating means)

Claims (16)

a step in which a computer generates information related to correction of a captured image based on the captured image obtained by imaging using an optical system;
generating a weight map by a computer based on information about the aperture value of the optical system and information about the luminance value of the captured image;
and generating a first image by a computer based on the captured image, the information related to the correction, and the weight map. The image processing method described in claim 1, characterized in that, in the step of generating the weight map, the weight map is further generated based on information regarding saturated regions in the captured image. The image processing method described in claim 2, characterized in that the information regarding the saturated region is information representing an area in which a subject in the saturated region of the captured image has expanded due to blurring that occurred during the capture. The image processing method described in claim 3, characterized in that information regarding the luminance values of the captured image is generated based on statistics regarding the luminance values of a second image based on the difference between the luminance values of the captured image and the luminance values in the region. The image processing method of claim 4, wherein the statistics include at least one of the mean, median, variance, or histogram. An image processing method described in any one of claims 2 to 5, characterized in that the information regarding the saturated region is generated using a machine learning model. The image processing method described in any one of claims 1 to 6, characterized in that in the step of generating the weight map, the weight map is further generated based on information regarding the optical performance of the optical system. The image processing method described in claim 7, characterized in that the information regarding the optical performance is calculated based on at least one of the zoom position, aperture value, and subject distance at the time of capturing the image, and the point spread function for each image height of the optical system. An image processing method according to any one of claims 1 to 8, characterized in that the information related to the correction is a corrected image obtained by performing the correction on the captured image, or a correction component which is the difference between the corrected image and the captured image. An image processing method according to any one of claims 1 to 8, characterized in that the correction includes at least one of increasing the resolution of the captured image and converting the shape of defocus blur in the captured image. An image processing method according to any one of claims 1 to 10, characterized in that the weight map weights the captured image more heavily as the brightness value decreases. the statistic regarding the luminance values of the second image is an average value of the luminance values in the second image;
6. The image processing method according to claim 5, wherein the weight map weights the captured image more heavily as the average value decreases. the aperture value can be set to a first aperture value or a second aperture value greater than the first aperture value;
13. The image processing method according to claim 1, wherein a weight in the weight map for the second aperture value is greater than a weight in the weight map for the first aperture value. a first generating means for generating information related to correction of a captured image based on the captured image obtained by imaging using an optical system;
a second generating means for generating a weight map based on information about the aperture value of the optical system and information about the luminance value of the captured image;
and third generating means for generating a first image based on the captured image, the information related to the correction, and the weight map. An image processing system comprising the image processing device according to claim 14 and a control device capable of communicating with the image processing device,
the control device has a transmission means for transmitting a request to cause the image processing device to execute processing on the captured image,
The image processing system is characterized in that the image processing device has a receiving means for receiving the request, and executes processing on the captured image in response to the request. A program causing a computer to execute the image processing method described in any one of claims 1 to 13.