JP7207893B2 - Imaging device, lens device - Google Patents

Description

The present invention relates to an imaging apparatus capable of improving the image quality of captured images.

2. Description of the Related Art In imaging using an optical system, light generated from one point on an object is affected by diffraction, aberration, and the like occurring in the optical system and reaches the image plane with a slight spread. Therefore, the captured image is blurred due to the diffraction and aberration of the optical system.

It is known that such image blur can be sharpened by image processing using a point spread function (PSF), an optical transfer function (OTF), or the like. However, when trying to sharpen the image, there is a problem that the noise of the image is amplified.

Japanese Patent Application Laid-Open No. 2002-200003 describes adjusting correction effects using information on noise characteristics (ISO sensitivity) in order to reduce amplification of noise when sharpening an image.

JP 2017-41763 A

In the method of Patent Document 1, the image correction effect is adjusted by controlling the gain value of the two-dimensional image restoration filter in the frequency space. Therefore, it is necessary to perform a two-dimensional inverse Fourier transform during image processing. Therefore, the method of Patent Document 1 has a problem that the load required for image processing increases.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus capable of reducing amplification of noise associated with sharpening while reducing processing load.

An image pickup apparatus of the present invention is an image pickup apparatus to which a lens device having an optical system is detachably attached and capable of communicating with the lens device, wherein the image pickup device photoelectrically converts an image formed by the optical system. An imaging device, a first acquisition means for acquiring noise information relating to noise characteristics in an input image generated by imaging using the optical system, and acquiring a sharpening filter in real space based on the optical characteristics of the optical system. and third obtaining means for obtaining gain information relating to the gain of the sharpening filter based on the sum of squares of the components of the sharpening filter, wherein the lens device comprises: a storage unit for storing information about a sharpening filter, wherein the second obtaining means obtains information about the sharpening filter from the lens device, and obtains the sharpening filter based on the information about the sharpening filter; and sharpening processing is performed on the input image using the sharpening filter based on the noise information and the gain information.

Further, another imaging device of the present invention is an imaging device to which a lens device having an optical system is detachably mounted and which can communicate with the lens device, wherein the imaging device comprises an image formed by the optical system. a first acquisition means for acquiring noise information relating to noise characteristics in an input image generated by imaging using the optical system; sharpness in real space based on the optical characteristics of the optical system a second acquiring means for acquiring a sharpening filter; and a third acquiring means for acquiring gain information regarding a gain of the sharpening filter at a specific frequency based on the component of the sharpening filter, The lens device includes a storage unit that stores information about the sharpening filter, and the second obtaining means obtains the information about the sharpening filter from the lens device, and based on the information about the sharpening filter, the A sharpening filter is acquired, and sharpening processing using the sharpening filter is performed on the input image based on the noise information and the gain information .

According to the present invention, it is possible to realize an imaging apparatus that can reduce the amplification of noise that accompanies sharpening while reducing the processing load.

FIG. 5 is a diagram for explaining sharpening by unsharp mask processing; FIG. 4 is a schematic diagram showing the PSF of the optical system; FIG. 10 is a diagram illustrating sharpening processing using a rotationally symmetric unsharp mask; FIG. 10 is a diagram illustrating sharpening processing using a rotationally asymmetric unsharp mask; FIG. 4 is a schematic diagram showing gains of a sharpening filter; 4A and 4B are schematic diagrams of MTFs before and after sharpening processing; FIG. 1 is a block diagram of an imaging device; FIG. 4 is a flowchart of image processing in Example 1. FIG. It is a figure which shows the relationship between gain information and a weighted addition rate. FIG. 10 is a diagram illustrating generation of a sharpening filter in Example 2; 10 is a flowchart of image processing in Example 2. FIG. 1 is a schematic diagram of a camera system; FIG.

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of an imaging device of the present invention having an image processing section as an image processing device will be described below with reference to the accompanying drawings.

The image processing unit of this embodiment performs unsharp mask processing as sharpening processing using a sharpening filter based on the optical characteristics of the optical system on an input image generated by photographing using the optical system. First, before describing the configuration of the imaging apparatus of this embodiment, the sharpening process of this embodiment will be described.

FIG. 1 is a schematic diagram of sharpening by unsharp mask processing. In FIG. 1A, the solid line represents the input image, the dashed line represents the image obtained by blurring the input image with an unsharp mask, and the dotted line represents the image after sharpening. Further, the solid line in FIG. 1(b) represents the correction component. The horizontal axis of each drawing in FIG. 1 is the coordinate, and the vertical axis is the pixel value or luminance value. FIG. 1 corresponds to a cross-section in a predetermined direction (eg, x-direction) in FIG. 2, which will be described later.

Assuming that the input image is f(x, y) and the correction component is h(x, y), the sharpened image g(x, y) can be expressed by the following equation.
g(x, y)=f(x, y)+m×h(x, y) (1)

In equation (1), m is an adjustment coefficient for changing the strength of correction, and the amount of correction can be adjusted by changing the value of the adjustment coefficient m. It should be noted that m may be a fixed constant regardless of the position of the input image, or may be changed according to the position of the input image to adjust the correction amount according to the position of the input image. Also, the adjustment coefficient m(x, y) can be varied according to shooting conditions such as the focal length of the optical system, the aperture value, and the subject distance (shooting distance).

Assuming that the unsharp mask is USM(x, y), the correction component h(x, y) can be expressed as follows. USM(x,y) is, for example, the tap value at coordinates (x,y).
h(x,y)=f(x,y)−f(x,y)*USM(x,y) (2)

The right side of equation (2) can be transformed into the following equation.
h (x, y) = f (x, y) * (δ (x, y) - USM (x, y)) (3)

Here, "*" is a convolution (convolution integral, sum of products), and "δ(x, y)" is a delta function. The delta function is data that has the same number of taps as USM(x, y), has a value of 1 at the center, and is filled with 0s elsewhere.

Equation (3) can be expressed by transforming Equation (2), so Equations (2) and (3) are equivalent. Therefore, generation of the correction component will be described below using equation (2).

In equation (2), the difference between the input image f(x, y) and the image obtained by blurring the input image f(x, y) with an unsharp mask is taken, and based on this difference information, the correction component h(x, y) is generating In general unsharp mask processing, smoothing filters such as Gaussian filters, median filters, and moving average filters are used for unsharp masks.

For example, when a Gaussian filter is used as an unsharp mask for the input image f(x, y) indicated by the solid line in FIG. a) as indicated by the dashed line. The correction component h(x, y) is the component indicated by the solid line in FIG. 1(b) obtained by subtracting the broken line in FIG. 1(a) from the solid line in FIG. 1(a). Using the correction components calculated in this way, the input image f(x, y) indicated by the solid line in FIG. can be sharpened to

Next, a case of sharpening an image by applying unsharp mask processing to an image that has been deteriorated due to the effects of diffraction, aberration, etc., generated in the optical system will be described. An input image f(x, y) generated by photographing using an optical system is an image (hereinafter referred to as an original image) when there is no deterioration due to the optical system. If the PSF of is psf(x, y), it can be represented by the following equation.
f(x,y)=I(x,y)*psf(x,y) (4)

First, the case where the input image f(x, y) is rotationally symmetrically blurred with respect to the original image I(x, y) will be described. If the optical system is a rotationally symmetrical coaxial optical system, the PSF corresponding to the central portion of the image will be rotationally symmetrical. Therefore, by applying a rotationally symmetric unsharp mask to the central portion of the image, it is possible to sharpen the input image f(x, y) closer to the original image I(x, y). Since the correction component is the difference value between the image obtained by blurring the input image with the unsharp mask and the input image, in order to perform accurate correction, the unsharp mask is not a simple smoothing filter, but more psf (x, y) It is preferable to use a shape close to .

For example, consider the case where the input image is degraded by the influence of spherical aberration. Spherical aberration has a rotationally symmetrical effect with respect to the center of the screen, but a smoothing filter such as a Gaussian filter has a distribution shape different from the PSF reflecting the influence of spherical aberration. Therefore, by using the PSF of the optical system for the unsharp mask, the input image can be accurately corrected (sharpened).

Therefore, in this embodiment, the PSF of the optical system is used as the unsharp mask USM(x, y). The input image f(x, y) shown in FIG. 1A has a line-symmetrical shape for simplification, but the image does not have to have an axis of symmetry. Even if the shape of the original image I(x, y) is asymmetric, if the degradation function (corresponding to psf(x, y)) convoluted with the original image I(x, y) is rotationally symmetric, , can be sharpened using a rotationally symmetric unsharp mask.

Next, the case where the input image f(x, y) is rotationally asymmetrically blurred with respect to the original image I(x, y) will be described. Even if the optical system is a rotationally symmetric coaxial optical system, the PSF at positions other than the center of the image usually has an asymmetric shape. FIG. 2 is a schematic diagram of the PSF of the optical system in the xy plane, FIG. 2(a) representing the on-axis PSF and FIG. 2(b) representing the off-axis PSF.

For example, assuming that the original image consists of a set of ideal point images, the input image f(x, y) is the PSF of the optical system from equation (4). If there is an ideal point image at the angle of view corresponding to FIG. image. A description will be given of a case where sharpening is performed by unsharp mask processing for such an asymmetrically blurred image.

3 and 4 are diagrams for explaining the case where unsharp mask processing is performed on an asymmetrically blurred image. FIG. 3 shows the case of processing using a rotationally symmetric unsharp mask, and FIG. 4 shows the case of processing using a rotationally asymmetric unsharp mask. The vertical axis in each of FIGS. 3 and 4 is the pixel value or luminance value. 3 and 4 correspond to the y direction in FIG. 2(b).

The solid lines in FIGS. 3A and 4A represent the cross section in the y-axis direction of FIG. 2B, and the dotted lines represent the input image blurred by the unsharp mask. A Gaussian filter is applied to the rotationally symmetric unsharp mask of FIG. 3, and the PSF of the optical system is applied to the rotationally asymmetric unsharp mask of FIG.

3(b) and 4(b) plot the difference values between the input image blurred by each unsharp mask and the input image, and represent the correction component. In FIGS. 3(a) and 4(a), the side where the base of the input image is wider is the positive side of the Y-axis.

In FIG. 3A, the difference value between the input image and the original image is small on the plus side of the peak position of the solid line, and the difference value between the input image and the original image is large on the minus side. Therefore, the extreme value on the minus side is smaller than the extreme value on the plus side with respect to the central peak position of the correction component in FIG. 3(b).

As can be seen by comparing the curves of FIGS. 3A and 3B, the correction amount on the plus side is small and the correction amount on the minus side is large. Therefore, the asymmetric blur cannot be corrected even if sharpening is performed according to Equation (4). FIG. 3(c) shows the result after sharpening when m=1. Although the solid line in FIG. 3(a) is sharpened, the minus side is larger than the plus side. It can be seen that the image is sunken and the asymmetric blur cannot be corrected.

Here, consider a case where the correction amount is adjusted by changing the adjustment coefficient m in Equation (4) without changing the unsharp mask. If the value of the adjustment coefficient m is increased to sufficiently correct the plus side of the image, the minus side of the image will be overcorrected (undershoot). If you set a value of , the plus side of the image will be undercorrected.

Thus, even if unsharp mask processing is performed on an asymmetrically blurred image using a rotationally symmetric unsharp mask, it is difficult to sharpen an asymmetrically degraded input image. Such a problem also occurs even if a rotationally symmetrical filter other than the Gaussian filter is used as the rotationally symmetrical unsharp mask.

On the other hand, in FIG. 4A, the difference value between the input image and the original image is large on the positive side with respect to the peak position of the solid line, and the difference value between the input image and the original image is small on the negative side. That is, a tendency opposite to that of FIG. 3(a) appears. Therefore, the correction component in FIG. 4B also has a smaller extreme value on the positive side than on the negative side with respect to the central peak position.

By applying the correction component shown in FIG. 4B to the input image represented by the solid line in FIG. On the large and negative side (smaller blur), the correction amount can be reduced.

In the case of such an asymmetric unsharp mask, the tendency of the balance of the blurring of the input image and the amount of correction of the correction component match, so there is a problem of excessive or insufficient correction when applying a rotationally symmetric unsharp mask. It becomes difficult to get up. FIG. 4(c) shows the result after sharpening when m=1. The sinking on the negative side has been improved.

Furthermore, as compared with the case of a rotationally symmetrical unsharp mask, excessive correction is less likely to occur, so the value of the adjustment coefficient m in equation (4) can be set relatively large, and sharpening can be achieved while reducing asymmetry. can do.

Further, in order to further improve the correction accuracy, it is necessary that the portion that is greatly blurred by the PSF of the optical system is also greatly blurred by the unsharp mask. Therefore, even when the input image f(x, y) is rotationally asymmetrically blurred with respect to the original image I(x, y), it is preferable to use the PSF of the optical system as the unsharp mask. In this case, the sharpening filter is a two-dimensional filter whose components are rotationally asymmetrically distributed. Note that the sharpening filter is a coefficient matrix that is convoluted on the input image in the sharpening process, and the components of the sharpening filter are the components (tap values) of the coefficient matrix.

Note that the PSF of the optical system differs for each shooting condition including the focal length of the optical system, the F-number of the optical system, and the subject distance (shooting distance). The PSF also differs depending on the position within the input image. Therefore, it is preferable to change the sharpening filter for each position in the input image for each shooting condition.

Next, amplification of noise components contained in an input image will be described.

Normally, an image generated by imaging using an optical system contains components deteriorated by noise in addition to the components deteriorated by the PSF of the optical system.

Since the sharpening process brings the MTF (Modulation Transfer Function), which is the amplitude component of the optical transfer function of the optical system, closer to 1, if the input image contains noise components, the noise components are also amplified. There is a risk that it will be lost. That is, when the sharpening process as described above is performed on the input image, the input image is sharpened and noise components contained in the input image become more noticeable.

This can be explained as follows using equations (1) and (3).

From the expressions (1) and (3), the image g(x, y) after the sharpening process can be expressed as the following expression.
g(x, y)=f(x, y)+m×f(x, y)*{δ(x, y)−USM(x, y)} (5)

Formula (6) is obtained by summarizing the right side of formula (5) for the input image f(x, y).
g(x, y)=f(x, y)*{δ(x, y)+m×(δ(x, y)-USM(x, y))} (6)

In Expression (6), the term in the braces on the right side corresponds to the sharpening filter. By Fourier transforming the equation (6), the following equation (7) can be obtained.
G(u,v)=F(u,v)×{1+m×(1−U(u,v))} (7)

G(u, v) in equation (7) is the Fourier transform of g(x, y), F(u, v) is the Fourier transform of f(x, y), U(u, v) is USM(x, y) is the Fourier transform.

FIG. 5 shows absolute values in brackets { } of equation (7). The horizontal axis of FIG. 5 is the spatial frequency, and the vertical axis is the gain. If the real part of the Fourier transform U(u, v) of the unsharp mask USM(x, y) is Re(U(u, v)) and the imaginary part is Im(U(u, v)), the unsharp mask A gain Ga in the processing is represented by the following equation (8).

If the unsharp mask USM(x,y) has a rotationally symmetric Gaussian distribution, the function U(u,v) also has a Gaussian distribution. Therefore, the imaginary part Im(U(u,v)) is 0, the real part Re(U(u,v)) is 0≦Re(U(u,v))≦1, and the gain Ga is 1≦Ga≦ (1+m). Moreover, since the Gaussian distribution approaches zero as it moves away from the center of the distribution, the real part Re(U(u,v)) also approaches zero. Therefore, the higher the frequency, the closer the gain Ga to (1+m), and the curve shown in FIG. 5 is obtained. Note that FIG. 5 shows the case where the adjustment coefficient m is 1, and the gain Ga is asymptotically approaching 2 on the high frequency side.

When using the PSF of the optical system as the unsharp mask USM(x, y), the Fourier transform U(u, v) of the unsharp mask USM(x, y) is the optical transfer function (OTF ). The absolute value of OTF is the amplitude component MTF.

FIG. 6 is a diagram showing changes in the amplitude component MTF before and after sharpening by unsharp mask processing. The solid line in FIG. 6 represents the amplitude component MTF before sharpening, and the dashed line represents the amplitude component MTF after sharpening.

FIG. 6 shows the frequency characteristics of the amplitude component of image degradation due to aberration and the like. As shown in FIG. 6, the low frequency side is high and the high frequency side is low. When the amplitude component MTF approaches zero, the real and imaginary parts of the OTF approach zero, and above the frequency resolvable by the optical system, the real and imaginary parts of the OTF become zero. At this time, the gain Ga(u, v) is (1+m) from equation (8). Therefore, even when the PSF of the optical system is used as the unsharp mask USM(x, y), the gain Ga is the dotted line in FIG. It becomes a curve that increases on the high frequency side indicated by the dashed line. The gain Ga is constant in any direction (cross section) when the PSF is rotationally symmetrical, but differs depending on the direction when the PSF is rotationally asymmetrical. Therefore, the gain Ga is also rotationally asymmetric off-axis where the PSF is rotationally asymmetrical. Also, since the real and imaginary parts of the OTF may be negative, the gain Ga may be Ga>(1+m) at a predetermined frequency.

As described above, the gain Ga is It becomes a curve that increases from the low frequency side to the high frequency side.

Comparing the amplitude component MTF before and after the sharpening process, the difference is maximized at the frequency fr. This is because the amplitude component MTF before the sharpening process indicated by the solid line in FIG. 6 is multiplied by the gain Ga shown in FIG. 5 to obtain the amplitude component MTF after the sharpening process. When the amplitude component MTF before the sharpening process is low, even if the gain Ga is large, the change in the amplitude component MTF due to sharpening is small.

Here, consider the case where the input image contains noise components. Assuming that the noise component contained in the input image is n(x, y), Equation (6) can be expressed as Equation (9) below.
g (x, y) = (f (x, y) + n (x, y)) * {δ (x, y) + m x (δ (x, y) - USM (x, y))} (9)

Similarly, if the Fourier transform of n(x, y) is N(u, v), Equation (7) can be expressed as Equation (10) below.
G(u,v)=(F(u,v)+N(u,v))×{1+m×(1−U(u,v))} (10)

When unshark mask processing is performed on an input image containing noise components according to equation (10), Fourier transform F(u, v) of f(x, y), which is a component other than noise in the input image, and noise Both components N(u, v) are covered by brackets { }. Also, F(u, v) is obtained by Fourier transforming equation (4), so it is the product of the Fourier transform of the original image I(x, y) and the OTF of the optical system. On the high frequency side where the amplitude component MTF approaches zero, the real and imaginary parts of the OTF approach zero as described above, so the Fourier transform F(u, v) of the input image f(x, y) is also represented by the solid line in FIG. similarly approaches zero as shown by . On the other hand, if a noise component such as white noise is considered as the noise component N(u, v), it will have a uniform distribution without depending on the frequency. Therefore, since the ratio of N(u, v) to F(u, v) increases on the high frequency side, if a large gain is applied to the input image containing noise components on the high frequency side, the noise components will be reduced from the sharpening effect. is amplified.

Therefore, in order to correct the input image satisfactorily, it is necessary to estimate the gain in the sharpening process and adjust the degree of sharpening by the sharpening process based on the gain and the noise characteristics of the input image.

The gain of the sharpening filter can be obtained using equation (8), which requires a two-dimensional Fourier transform of the real-space sharpening filter. Therefore, when calculating the gain using the equation (8), the amount of calculation increases, and the processing load required for the sharpening process increases.

On the other hand, when pre-calculated gains are stored, although the amount of calculation required to obtain the gains is reduced, it is necessary to allocate storage capacity for storing the gains. In order to properly correct the input image, it is necessary to change the sharpening filter for each shooting condition. It becomes necessary to allocate for the memory of.

Therefore, in the present embodiment, as will be described later, the gain is estimated from a real-space sharpening filter, and the degree of sharpening by the sharpening process is adjusted based on the gain and the noise characteristics of the input image. This makes it possible to suppress the amplification of noise accompanying sharpening while reducing the processing load required for the sharpening process.

The sharpening process performed in the image processing section of the present embodiment has been described above.

Next, the configuration of the imaging apparatus of this embodiment will be described.

An imaging device 100 according to the present embodiment will be described with reference to FIG. FIG. 7 is a block diagram showing the configuration of the imaging device 100. As shown in FIG. An image processing program for performing the sharpening process of the present embodiment is installed in the imaging apparatus 100, and this sharpening process is executed by an image processing unit (image processing apparatus) 104 inside the imaging apparatus 100. . Note that the imaging device 100 in FIG. 7 illustrates one form of the imaging device of the present invention. The imaging apparatus of the present invention may have a form as shown in FIG. 12, which will be described later.

The imaging device 100 includes an optical system 101 and an imaging device body (camera body).

The optical system 101 includes an aperture 101a and a focus lens 101b. The optical system 101 is, for example, an optical system provided in an interchangeable lens, and is detachably attached to the camera body. Further, the optical system 101 may include an optical element having a refracting surface such as a lens, an optical element having a diffractive surface, and an optical element having a reflecting surface.

The imaging device 102 is composed of a CCD sensor or a CMOS sensor, and photoelectrically converts a subject image obtained by the optical system 101 (an optical image formed by the optical system 101) to generate a captured image. That is, the subject image is converted into an analog signal (electrical signal) by photoelectric conversion by the imaging device 102 . This analog signal is then converted into a digital signal by the A/D converter 103 . This digital signal is input to the image processing unit 104 .

The image processing unit 104 performs predetermined processing on the digital signal and sharpening processing of the present embodiment. As shown in FIG. 7, the image processing unit 104 includes an imaging condition acquisition unit (first acquisition unit) 104a, a sharpening filter acquisition unit (second acquisition unit) 104b, a gain information acquisition unit (third acquisition unit) 104b. means) 104c and a processing unit (processing means) 104d. The imaging condition acquisition unit 104 a acquires the imaging conditions of the imaging device 100 from the state detection unit 107 . The shooting conditions include the aperture value, shooting distance, and focal length of the optical system 101 . The state detection unit 107 may acquire the imaging conditions directly from the system controller 110 or from the optical system control unit 106 .

The PSF or data necessary for generating the PSF are held in a storage unit (storage means) 108 . As will be described later, this data is obtained by communication from the interchangeable lens and stored. The storage unit 108 is composed of, for example, a ROM. An output image processed by the image processing unit 104 is stored in an image recording medium 109 in a predetermined format. A display unit 105 configured by a liquid crystal monitor or an organic EL display displays an image that has undergone a predetermined display process on the image that has undergone the sharpening process. However, the image displayed on the display unit 105 is not limited to this, and an image subjected to simple processing for high-speed display may be displayed on the display unit 105 .

A system controller 110 controls the imaging apparatus 100 . The optical system 101 is mechanically driven by an optical system controller 106 based on instructions from the system controller 110 . The optical system control unit 106 controls the aperture diameter of the diaphragm 101a so as to obtain a predetermined F-number. Further, the optical system control unit 106 controls the position of the focus lens 101b by an autofocus (AF) mechanism (not shown) or a manual focus mechanism to adjust the focus according to the object distance. Note that functions such as aperture diameter control of the diaphragm 101a and manual focus may not be executed according to the specifications of the imaging device 100. FIG.

Note that the image processing unit 104 is configured by an ASIC, and the optical system control unit 106, state detection unit 107, and system controller 110 are each configured by a CPU or MPU. Also, one or more of the image processing unit 104, the optical system control unit 106, the state detection unit 107, and the system controller 110 may be configured to share the same CPU or MPU.

Next, an example of sharpening processing performed in the image processing unit 104 will be described.

[Example 1]
FIG. 8 is a flowchart of sharpening processing of the first embodiment. The flowchart in FIG. 8 is executed based on instructions from the image processing unit 104 . In FIG. 8, "S" represents a step. The flowchart of FIG. 8 can also be embodied as a program for causing a computer to execute each step and exhibit functions similar to those of the image processing unit 104 . These also apply to the flow chart shown in FIG.

In S101, the image processing unit 104 acquires an image captured by the imaging device 100 as an input image. The input image is stored in storage unit 108 . Alternatively, an image stored in the image recording medium 109 may be acquired as the input image.

In S102, the photographing condition acquisition unit 104a acquires the photographing conditions at the time of photographing the input image. The shooting conditions are the focal length, aperture value, shooting distance, etc. of the optical system 101 . In the case of an imaging apparatus in which the optical system 101 is replaceably attached to the camera body, the shooting conditions further include the lens ID and the camera ID. The shooting conditions may be obtained directly from the imaging device, or may be obtained from information attached to the input image (for example, EXIF information).

In S102, the imaging condition acquisition unit 104a also acquires noise information regarding the noise characteristics of the input image. The noise information may be any information that can be associated with the magnitude of noise contained in the input image. Noise information includes, for example, the sensitivity and temperature of the image sensor 102 . Also, if image processing has been performed on the input image prior to sharpening processing, noise characteristics of the input image may be affected. Therefore, the noise information may include information regarding image processing performed on the input image prior to the sharpening processing. In addition, in a general imaging apparatus, ISO sensitivity is recorded together with the photographed image as the sensitivity of the imaging element together with the above-described imaging conditions, so it is preferable to use the ISO sensitivity as the noise information. An example in which the ISO sensitivity is used as noise information will be described below.

In S<b>103 , the sharpening filter acquisition unit 104 b acquires the optical characteristics of the optical system 101 from the storage unit 108 . Here, PSF is acquired as an optical characteristic. However, as optical characteristics, such as OTF, which corresponds to PSF, or coefficient data of a function that approximates PSF or OTF, or the like may be acquired.

In S104, the sharpening filter acquisition unit 104b acquires the sharpening filter corresponding to the shooting conditions acquired in S102 based on the optical characteristics acquired in S103. In this embodiment, in the sharpening filter in equation (6), a sharpening filter is acquired using the PSF acquired in S103 for USM(x, y).

If the PSF corresponding to the shooting conditions acquired in S103 is not saved, the PSF corresponding to the shooting conditions closest to the shooting conditions acquired in S103 among the PSFs stored in the storage unit 108 is used. A sharpening filter may be obtained. Also, a sharpening filter based on a PSF obtained by interpolating a PSF under shooting conditions close to those acquired in S103 is acquired, or a sharpening filter based on a PSF under shooting conditions close to those acquired in S103 is interpolated. You may acquire the sharpening filter obtained by carrying out.

Moreover, although the sharpening filter is acquired based on the optical characteristics in this embodiment, the present invention is not limited to this. A sharpening filter generated based on the optical characteristics of the optical system 101 may be stored in advance in the storage unit 108 , and the sharpening filter acquisition unit 104 b may directly acquire the sharpening filter from the storage unit 108 . In this case, S103 is skipped.

In S104, the sharpening filter acquisition unit 104b acquires a plurality of sharpening filters to be applied to each of a plurality of positions of the input image. Since the optical properties (PSF) at each position of the input image are different, each sharpening filter applied to each position of the input image is different.

Also, the sharpening filter acquired in S104 is a filter in real space, not in frequency space. When obtaining a filter in the frequency space as a sharpening filter, it is necessary to perform an inverse Fourier transform when sharpening the input image, but if a filter in the real space is obtained, it can be applied directly to the image. . Therefore, the processing load required for sharpening processing can be reduced. This makes it possible to improve the computation speed of the sharpening process. Further, since the processing load required for sharpening processing is reduced, sharpening processing can be realized with a relatively inexpensive chip. As a result, the product cost can be reduced.

In S105, the gain information acquisition unit 104c acquires the gain information of the sharpening filter acquired in S104. Gain information is information about the gain of a sharpening filter or data that can be treated in the same way as the gain. The gain information may be information about the gain of the sharpening filter at a specific frequency, or may be information about a value obtained by averaging the gain of the sharpening filter with respect to frequency.

As described above, in this embodiment, gain information is acquired from the components of the sharpening filter in real space. One method for obtaining gain information from a real-space sharpening filter is to use Parseval's equation.

k(x, y) is the sharpening filter in real space, K(fx, fy) is the Fourier transform of k(x, y), Nx is the number of taps in the x direction of k(x, y), taps in the y direction Letting the number be Ny, the following Parseval equation holds.

That is, by calculating the sum of squares of the components of the sharpening filter in the real space, it is possible to calculate the average of the sum of squares for the frequency components of the gain without Fourier transform. That is, the average value of the gain of the sharpening filter with respect to frequency can be calculated.

Further, as another method for obtaining gain information from the components of the sharpening filter in real space, as shown in the following equation (12), the product of the component of the sharpening filter in real space and a specific frequency component There is a way to take the sum of In this case, only gain values at specific frequencies of the sharpening filter can be calculated.

The gain obtained by Equation (12) is the gain for a specific frequency in the gain characteristics shown in FIG. By calculating only the gain for a specific frequency, it is possible to acquire gain information with a lower processing load than performing a two-dimensional Fourier transform. The value of "exp" in the expression (12) corresponding to the specific frequency component may be stored in the storage unit 108 in advance.

By doing so, it is possible to acquire information about the gain in the frequency space without performing a two-dimensional Fourier transform. It should be noted that the gain may be obtained by calculation using other formulas that can obtain results equivalent to formulas (11) and (12).

As the gain information in the shooting conditions of the input image, if a value under shooting conditions different from the shooting conditions of the input image is used, or interpolation is performed from values under shooting conditions that differ from the shooting conditions of the input image, sufficient accuracy can be obtained. Gain information may not be obtained. On the other hand, in this embodiment, gain information is obtained directly from the sharpening filter, so there is no need to interpolate gain information with respect to imaging conditions. Therefore, gain information can be obtained with high accuracy.

In this embodiment, the gain information acquisition unit 104c acquires the gain information of each sharpening filter acquired in S104 based on the component of each sharpening filter, but the present invention is not limited to this. That is, it is not necessary to acquire gain information from the components of each sharpening filter for all sharpening filters applied to each screen position of the input image. For example, gain information (first gain information) acquired using the components of a sharpening filter (first sharpening filter) for a specific position (first position) is used to obtain another position (second position) may be subjected to sharpening processing.

Specifically, gain information is acquired based on the components of the sharpening filter only for the sharpening filter applied to some screen positions of the input image, and the gain information at other screen positions is You may acquire by interpolating from gain information. This can reduce the processing load when acquiring gain information. In this case, the screen position in the input image is interpolated, but the imaging conditions are not interpolated. Therefore, it is possible to obtain gain information with sufficient accuracy while reducing the processing load.

In S106, the weighted addition rate (proportion of the input image in the output image) in sharpening processing is determined from the noise information acquired in S102 and the gain information acquired in S105. An example of a method for determining the weighted addition rate will be described with reference to FIG.

FIG. 9 is a diagram showing a weighted addition rate for gain information at a given ISO sensitivity. In FIG. 9, the weighted addition rate is set to 0% when the gain information is smaller than a. That is, when the gain information is smaller than a, the image after correction using the sharpening filter (g(x, y) in Equation (6)) is directly output.

Also, in FIG. 9, when the gain information is greater than b, the weighted addition rate is set to 80%. That is, the output image is an image obtained by weighting and averaging 80% of the input image and 20% of the corrected image. This is equivalent to adding 20% of the difference between the post-correction image and the pre-correction image to the pre-correction image, and such calculation may be performed. That is, the value of the adjustment coefficient m in Equation (6) may be adjusted.

By determining the weighted addition rate based on the gain information and the noise information and performing the sharpening process in this way, it is possible to increase the signal of the input image before correction when the gain becomes large. In other words, based on the gain information and the noise information, the degree of sharpening by the sharpening process can be weakened when the influence of noise amplification is considered to be large. As a result, amplification of noise due to sharpening processing can be reduced by simple processing.

Note that the relationship between the gain information and the weighted addition rate as shown in FIG. 9 may be stored in the storage unit 108 in association with the noise information. For example, if the noise information is the ISO sensitivity, the relationship between the gain information and the weighted addition rate may be stored as table data for each possible value of the ISO sensitivity. Also, the value of a and the gradient between a and b for representative ISO sensitivity values are stored, and the relationship between the gain information and the weighted addition rate for ISO sensitivity that is not stored is calculated from the stored relationship. You can do it.

In S<b>107 , the image processing unit 104 uses the sharpening filter acquired in step 104 to sharpen the input image. That is, the input image is sharpened by convolving the sharpening filter with the input image.

In this embodiment, the weighted addition rate (or the adjustment coefficient m) is determined based on the gain information and the noise information to output the output image, but the present invention is not limited to this. The processing in S106 and S107 may be processing that reduces amplification of noise due to sharpening processing based on gain information and noise information.

[Example 2]
Next, the sharpening process of Example 2 will be described.

In Example 1, the gain information of the acquired sharpening filter was acquired, and the sharpening process was executed using the same sharpening filter. Accordingly, it is possible to accurately acquire gain information for a filter that performs sharpening processing. On the other hand, in the present embodiment, gain information (first gain information) acquired using a component of a sharpening filter (first sharpening filter) for a specific position (first position) is used to obtain other Obtain a sharpening filter (second sharpening filter) for the position of (second position). Then, sharpening processing is performed on the second position using the second sharpening filter. Thereby, the gain information can be estimated more easily, and the processing load of the sharpening process can be further reduced.

Before explaining the gain information acquisition method in this embodiment, the acquisition of the sharpening filter in this embodiment will be described with reference to FIG.

FIG. 10(a) shows the positions on the input image where the sharpening filter is generated. The white circles in FIG. 10(a) represent the positions on the input image where the sharpening filter is generated. In this embodiment, a sharpening filter is generated based on the optical characteristics of the optical system 101 for 81 discrete positions on the input image. By performing linear interpolation or the like on these generated sharpening filters, it is possible to generate a sharpening filter at an arbitrary position on the input image. This makes it possible to apply an appropriate sharpening filter to an arbitrary position on the input image while reducing the processing load for generating the sharpening filter.

In FIG. 10A, there are 9×9=81 positions for generating the sharpening filter. may be incremented to .

Further, although it is possible to directly acquire the optical characteristics for each point of the white circles in FIG. 10A and generate a sharpening filter, these optical characteristics and the sharpening filter may also be generated by interpolation. FIG. 10(b) shows an example of this, showing a case where the optical characteristics at each position are generated by interpolation.

First, the optical characteristics at the position of the black circle in FIG. 10(b) are acquired. Since the PSF of an optical system generally has rotational symmetry with respect to the optical axis, the optical characteristics also have rotational symmetry with respect to the optical axis. Using this feature, in FIG. 10B, the optical characteristics are obtained at a plurality of positions (10 positions indicated by black circles) with different image heights (distances from the center of the input image), and these are obtained from the input image. The optical characteristics of the position corresponding to each white circle are generated by interpolation while rotating about the center. Then, a sharpening filter at each position is acquired based on the optical characteristics at each position. This eliminates the need to obtain the optical characteristics for each position indicated by the white circle, thereby reducing the processing load and the amount of data.

Although the optical characteristics of 81 points are obtained by interpolation from the optical characteristics of 10 points here, the sharpening filter is first obtained from the optical characteristics of 10 points and the sharpening filter is interpolated to obtain 81 points. A pointwise sharpening filter may be obtained.

Next, a method of acquiring gain information in this embodiment will be described. In FIG. 10B, in order to obtain accurate gain information for each position, it is necessary to calculate gain information based on the sharpening filter at each position. On the other hand, gain information at each image height can be obtained by obtaining a sharpening filter corresponding to the optical characteristics of the black circles in FIG. 10B and calculating gain information from the sharpening filter. That is, it is possible to acquire the change in gain information with respect to the image height.

By using the gain information of each information calculated in this way, the gain information of the sharpening filter generated at the position corresponding to each white circle in FIG. 10B can be estimated. In this embodiment, the gain information of each image height is used to adjust the gain of each sharpening filter generated at the position corresponding to each white circle in FIG. The sharpening process is performed at an appropriate degree. As a result, the amplification of noise associated with sharpening is suppressed while reducing the processing load.

That is, in Embodiment 1, gain information was calculated for each sharpening filter generated, but in this embodiment, gain information for each image height is obtained, and the input image is This differs from the first embodiment in that a sharpening filter is acquired for each position of .

When acquiring gain information for each sharpening filter generated at a position corresponding to each white circle in FIG. should be calculated based on On the other hand, according to this embodiment, the gain information can be calculated based on the equations (11), (12), etc. for only the 10 locations indicated by the black circles in FIG. The processing load for calculation can be further reduced.

FIG. 11 is a flowchart of sharpening processing in the second embodiment. The flowchart in FIG. 11 is executed based on instructions from the image processing unit 104 .

Since S201 to S203 are the same as S101 to S103, description thereof is omitted.

In S204, the sharpening filter acquisition unit 104b acquires a sharpening filter (first sharpening filter) at each black circle position based on the optical characteristics (PSF) at the black circle positions in FIG. do.

In S205, as in the first embodiment, gain information for each image height is obtained by obtaining gain information from each first sharpening filter.

In S206, the sharpening filter acquisition unit 104b acquires a sharpening filter (second sharpening filter). At this time, the sharpening filter acquisition unit 104b generates the second sharpening filter based on the gain information and noise information of each image height so that the gain of the second sharpening filter becomes a desired value. Specifically, the gain of each second sharpening filter is adjusted by adjusting the adjustment coefficient m in equation (6).

Note that the gain of the second sharpening filter acquired in S206 may be adjusted based on the gain information and noise information at each image height. Therefore, the second sharpening filter can be obtained in various ways. For example, the PSF at the positions indicated by the black circles may be acquired from the PSF at the positions indicated by the white circles, and the second sharpening filter may be generated based on the PSF at the positions indicated by the white circles and the gain information and noise information for each image height. . In this case, an adjustment factor m is determined for each of the secondary sharpening filters generated for each open circle position.

Further, based on the PSF at the position indicated by the black circle and the gain information and noise information for each image height, the adjustment coefficient m is determined for each position indicated by the black circle, and the gain produces the adjusted sharpening filter. This gain-adjusted sharpening filter may be rotated about the center of the input image and interpolated to generate a second sharpening filter at each white circle location. In this case, it is only necessary to determine the adjustment coefficient m for each black dot position.

Also, based on the PSF at the position indicated by the black circle and the gain information and noise information at each image height, the adjustment coefficient m is determined for each position indicated by the black circle. Using the m of each image height determined in this way and the PSF at each position indicated by the black circle, m and the PSF at each position indicated by the black circle (corresponding to USM (x, y) in equation (6) ). The product of m and PSF at each position indicated by a black circle is rotated about the center of the input image and interpolated to obtain m and PSF at each white circle position. The product of m and the PSF thus obtained may be used to generate a second sharpening filter. Also in this case, the adjustment coefficient m only needs to be determined for each black dot position.

In S207, the processing unit 104d executes sharpening processing. The sharpening process in S207 is performed by convoluting the input image with the second filter acquired in S206.

As described above, sufficiently accurate gain information can be obtained by simpler processing than in the first embodiment. This makes it possible to suppress the amplification of noise accompanying sharpening while further reducing the processing load.

In each of the above embodiments, unsharp mask processing using PSF is performed as sharpening processing, but the present invention is not limited to this. The sharpening process in this embodiment may be any process using a sharpening filter based on the optical characteristics of the optical system 101 . For example, unsharp mask processing or edge enhancement processing that does not use PSF may be used. Alternatively, image restoration processing typified by a Wiener filter or iterative processing type image restoration processing typified by the RL method may be used.

Note that each processing in the sharpening processing in the first and second embodiments does not need to be performed by a single image processing apparatus. For example, a program for implementing part or all of the processing in each embodiment may be supplied to a system consisting of one or more devices via a network or a recording medium, and the system or device may execute the program. good.

Next, a case where the image processing methods described in the first and second embodiments are implemented in a lens-interchangeable camera system 200 shown in FIG. 12 will be described.

The camera system 200 has a camera body (imaging device body) 216 and an interchangeable lens (lens device) 202 detachable from the camera body 216 . A storage unit 209 is configured by a ROM (memory), a hard disk drive, or the like. A program for executing the image processing method described in the first and second embodiments is installed in the storage unit 209 . A program for executing the image processing methods described in the first and second embodiments is executed by the image processing unit (image processing device) 205 of the camera body 216 . The configuration of the image processing unit 205 is the same as that of the image processing unit 104 in FIG. 7, and can execute the image processing of the first or second embodiment. Note that the programs of the image processing methods of the first and second embodiments may be installed in a storage unit (not shown) provided inside the image processing unit 205 instead of the storage unit 209 . Alternatively, the image processing may be executed by designing a circuit corresponding to the program and operating the circuit.

The image processing unit 205 performs predetermined processing on the digital signal and sharpening processing described in the first or second embodiment. The image processing unit 205 acquires imaging conditions (imaging condition information) from the state detection unit 208 . The imaging condition information is information regarding image height, aperture value (F number), shooting distance, focal length of a zoom lens, and the like. The state detection unit 208 can acquire imaging condition information directly from the camera CPU 211, but is not limited to this. For example, imaging condition information regarding the imaging optical system 201 of the interchangeable lens 202 can be obtained from the optical system control unit 207 via the lens CPU 213 .

An output image processed by the image processing unit 205 is saved in a predetermined format in the storage unit 209 .

The display unit 206 can display an image obtained by performing predetermined processing for display after the sharpening processing. The display unit 206 may display an image obtained by performing simple processing for high-speed display.

A camera body 216 also includes an optical low-pass filter 203 , an image sensor 204 , and an image recording medium 210 . The imaging element 204 is a two-dimensional imaging element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide Semiconductor) sensor. The imaging device 204 photoelectrically converts a subject image (optical image, imaging light) obtained via the imaging optical system 201 and the optical low-pass filter 203 to generate a captured image. A subject image is subjected to photoelectric conversion by the imaging device 204 and converted into an analog signal (electrical signal). The analog signal is converted into a digital signal by an A/D converter (not shown), and the digital signal is input to the image processing unit 205 .

When using optical elements that affect imaging performance, such as low-pass filters and infrared cut filters, highly accurate sharpening processing is possible by considering the effects of these optical elements at the time of creating the sharpening filter.

The interchangeable lens 202 includes an imaging optical system 201 , an optical system control section 207 , a storage section 212 , a lens CPU 213 and a contact unit 214 . The imaging optical system 201 is configured with a diaphragm 201a and a focus lens 201b. The lens CPU 213 includes a communication circuit (communication means) for communicating between the interchangeable lens 202 and the camera body 216, reset exception processing, A/D, timer, input/output port, built-in ROM, built-in RAM, and the like. have a function. The communication circuit performs communication between the interchangeable lens 202 and the camera body 216 using a communication method including control information corresponding to the shooting mode (moving image shooting mode, still image shooting mode). The optical system control unit 207 is lens control means for controlling each component inside the interchangeable lens 202, and based on instructions from the lens CPU 213, controls the lens, aperture, etc. using control information obtained via a communication circuit. drive control of the optical element. The contact unit 214 has a plurality of metal contacts for communication between the interchangeable lens 202 and the camera body 216, and is connection means for electrically connecting the lens CPU 213 and the camera CPU 211 together.

The storage unit 212 is a rewritable nonvolatile memory. The data stored in the storage unit 212 includes information indicating optical characteristics unique to the interchangeable lens 202 .

Information stored in the storage unit 212 will be specifically described. The storage unit 212 stores information about the PSF, such as coefficient data of a function used to reconstruct (approximate) the PSF of the imaging optical system 201 and the adjustment coefficient m. These pieces of information are transmitted from the interchangeable lens 102 to the camera body 216 via the contact units 214 and 215 at predetermined timings. The predetermined timing is when the interchangeable lens 202 is attached to the camera body 216, during initial communication when the camera body 216 is activated after the interchangeable lens 202 is attached, or when the image processing unit 205 sharpens the captured image. It may be time to do some processing.

The camera body 216 performs the image processing described in the first or second embodiment based on the PSF-related information transmitted from the interchangeable lens 202 . That is, the sharpening filter acquisition unit of the image processing unit 205 generates a sharpening filter based on information regarding the PSF transmitted from the interchangeable lens 202 . The gain information acquisition unit and processing unit perform the above-described processing based on the generated sharpening filter. Note that the camera body 216 may store the information about the PSF transmitted from the interchangeable lens 202 in the storage unit 209 and perform the image processing described in the first or second embodiment at the necessary timing. The necessary timing may be when an image is captured or when instructed by a user's operation. In this way, the information about the PSF stored in the storage unit 212 is information used to generate the sharpening filter, and is also information about the sharpening filter (information for generating the sharpening filter). I can say

As described above, in the case of a lens-interchangeable camera system, the information on the sharpening filter is stored in the interchangeable lens, and the camera body acquires the information on the sharpening filter from the interchangeable lens and performs the sharpening process. is preferably performed. In the case of an interchangeable-lens camera system, a plurality of interchangeable lenses can be attached to one camera body. If the camera body acquires the information on the sharpening filter from the interchangeable lens in this way, it is possible to reduce the capacity of the storage unit on the camera body side. This is because there is no need to store information on sharpening filters for multiple types of interchangeable lenses in the camera body.

Although preferred embodiments and examples of the present invention have been described above, the present invention is not limited to these embodiments and examples, and various combinations, modifications, and changes are possible within the scope of the gist.

104 image processing unit (image processing device)
104a imaging condition acquisition unit (first acquisition unit)
104b sharpening filter acquisition unit (second acquisition unit)
104c gain information acquisition unit (third acquisition means)
202 lens device 212 storage unit 216 imaging device

Claims (15)

An image pickup device detachably mounted with a lens device having an optical system and capable of communicating with the lens device,
The imaging device is
an imaging device that photoelectrically converts an image formed by the optical system;
a first acquisition means for acquiring noise information relating to noise characteristics in an input image generated by imaging using the optical system;
a second acquiring means for acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
a third obtaining means for obtaining gain information about the gain of the sharpening filter based on the sum of squares of the components of the sharpening filter;
The lens device includes a storage unit that stores information about the sharpening filter,
the second obtaining means obtains information about the sharpening filter from the lens device, obtains the sharpening filter based on the information about the sharpening filter,
An imaging apparatus, wherein sharpening processing using the sharpening filter is performed on the input image based on the noise information and the gain information. An image pickup device detachably mounted with a lens device having an optical system and capable of communicating with the lens device,
The imaging device is
an imaging device that photoelectrically converts an image formed by the optical system;
a first acquisition means for acquiring noise information relating to noise characteristics in an input image generated by imaging using the optical system;
a second acquiring means for acquiring a sharpening filter in real space based on the optical characteristics of the optical system;
a third obtaining means for obtaining gain information about the gain of the sharpening filter at a specific frequency based on the components of the sharpening filter;
The lens device includes a storage unit that stores information about the sharpening filter,
the second obtaining means obtains information about the sharpening filter from the lens device, obtains the sharpening filter based on the information about the sharpening filter,
An imaging apparatus, wherein sharpening processing using the sharpening filter is performed on the input image based on the noise information and the gain information. 3. The imaging apparatus according to claim 1, wherein the optical characteristics are optical characteristics under imaging conditions when the input image is captured. 4. The imaging apparatus according to claim 1, wherein said optical characteristic is a point image intensity distribution of said optical system. 5. The imaging apparatus according to claim 1, wherein the noise information is ISO sensitivity. 6. The imaging apparatus according to claim 1, wherein the sharpening process is unsharp masking. the sharpening filter is a two-dimensional filter;
7. The imaging apparatus according to claim 1, wherein the components of the sharpening filter are distributed rotationally asymmetrically. 8. The imaging apparatus according to any one of claims 1 to 7, wherein said second obtaining means obtains a different sharpening filter for each of a plurality of positions in said input image. 3. The third acquisition means acquires gain information for each of the sharpening filters acquired by the second acquisition means based on the components of the respective sharpening filters. 9. The imaging device according to 8. The sharpening process is performed by adding the input image and an image obtained by correcting the input image using the sharpening filter at a ratio determined based on the noise information and the gain information. The imaging device according to claim 9 . the third acquisition means acquires first gain information using a sharpening filter component for a first position in the input image;
9. The imaging apparatus according to claim 8, wherein sharpening processing is performed for a second position different from the first position using the first gain information and the noise information. the second obtaining means obtains a sharpening filter for the second position based on the first gain information and the noise characteristic;
12. The imaging apparatus according to claim 11, wherein the sharpening process for the second position is performed using a sharpening filter for the second position. the third acquisition means acquires gain information at each image height of the input image using a first sharpening filter component for each of a plurality of positions with different image heights in the input image;
9. The imaging apparatus according to claim 8, wherein sharpening processing is performed on the input image using the gain information and the noise information at each image height. the second acquisition means acquires a second sharpening filter for each of the plurality of positions in the input image based on the gain information and the noise characteristics at each image height;
14. The imaging apparatus according to claim 13, wherein the sharpening process is performed using the second sharpening filter. A lens device detachably attached to the imaging device according to any one of claims 1 to 14 and capable of communicating with the imaging device,
an optical system;
a storage unit that stores information about the sharpening filter of the optical system;
communication means for transmitting information about the sharpening filter to the imaging device;
A lens device comprising:

Images