JP7672838B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

Conventionally, a technology has been proposed that extends the depth of field by combining an optical system including a wavefront modulation element that modulates the phase of transmitted light with image processing. Specifically, the wavefront modulation element distributes the light beam regularly, making the optical transfer function (OTF) of the optical system essentially unchanged with respect to changes in the subject distance. Then, an image with a deep depth of field is generated by performing restoration processing on the image obtained via an image sensor or the like. In addition, since the image sensor discretely samples the image of the subject field, the relationship between the rotational positions of the wavefront modulation element and the image sensor is arbitrary.

Patent document 1 discloses an imaging device equipped with an optical system that forms a rotationally asymmetric point spread function (PSF) with an apex. In order to realize a depth-extending optical system with loose tolerances, this imaging device sets the relative mounting positions of the optical system and the imaging element so that the tip of the diagonal side from the apex is aligned diagonally with the pixel array.

JP 2008-136144 A

By the way, pixel-enhancing processing is a process for increasing the number of pixels in an image beyond the number of pixels in the imaging element. A known method of pixel-enhancing processing is to use pixel-shifted images to achieve high pixel counts. When pixel-shifting is performed to achieve high pixel counts, the two-dimensional sampling pattern is determined based not only on the pixel arrangement of the imaging element, but also on the method of shifting. However, the imaging device disclosed in Patent Document 1 does not take into account the relationship between the pixel-shifting method and the installation of the imaging element, and therefore is unable to obtain an image of good quality when pixel-enhancing processing is performed using pixel-shifting.

The present invention aims to provide an imaging device that can obtain high-quality images using a depth-expanding optical system.

An imaging device according to one aspect of the present invention includes an imaging optical system including a wavefront modulation element that is rotationally asymmetric with respect to an optical axis, and an imaging unit that photoelectrically converts an optical image formed by the imaging optical system, the imaging unit acquiring one or more shifted image data including first image data corresponding to a first color and second image data corresponding to a second color, the one or more shifted image data being acquired by imaging in which the optical image is shifted by a shift amount of a non-integer pixel of the imaging unit relative to the first image data, a direction in which a modulation transfer function of the imaging optical system is largest is defined as a first direction, a shift direction corresponding to the smallest shift amount among the shift amounts of the one or more shifted image data is defined as a second direction, and an angle between the first direction and the second direction is defined as θ [°],
30≦θ≦60
The conditional expression is satisfied .

Other objects and features of the present invention are described in the following examples.

The present invention provides an imaging device that can obtain high-quality images using a depth-expanding optical system.

5 is an explanatory diagram of an image restoration filter in each embodiment. FIG. 4A to 4C are explanatory diagrams (sectional views) of an image restoration filter in each embodiment. FIG. 4 is an explanatory diagram of a point spread function in each embodiment. 5A and 5B are explanatory diagrams of the amplitude component MTF and the phase component PTF of the optical transfer function in each embodiment. 5A to 5C are diagrams illustrating the MTF of an optical system including a wavefront modulation element in each embodiment. 11A to 11C are diagrams illustrating an example of imaging using pixel shifting and high pixel processing in each embodiment. 1A to 1C are diagrams illustrating different examples of imaging using pixel shifting and pixel-enhancing processing in each embodiment. 5A to 5C are explanatory diagrams of pixel-enhancing processing in each embodiment. 10A to 10C are explanatory diagrams of Nyquist frequencies obtained by imaging using pixel shifting and high pixel processing in each embodiment. FIG. 2 is a block diagram of an imaging device in each embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings.

First, before describing the specific examples, the gist of the present invention will be described. The imaging device of this embodiment uses a depth-expanding optical system that uses a wavefront modulation element that is rotationally asymmetric with respect to the optical axis as the imaging optical system. A plurality of captured images corresponding to a plurality of colors are acquired, and pixel-high resolution processing is performed by pixel shifting. In addition, sharpening processing (image restoration processing) is performed based on the optical characteristics of the optical system. Below, before describing the configuration of the imaging device of this embodiment, various technologies used in this embodiment will be described.

[Image restoration processing]
First, an overview of the image restoration process will be described. When a captured image (degraded image) is g(x, y), an original image is f(x, y), and a point spread function PSF, which is a Fourier pair of an optical transfer function OTF, is h(x, y), the following formula (1) is established.

g(x,y)=h(x,y)*f(x,y)...(1)
Here, * denotes convolution (convolution integral, product sum), and (x, y) are coordinates on the captured image.

Furthermore, if equation (1) is Fourier transformed and converted to a frequency domain display format, equation (2), which is expressed as a product for each frequency, is obtained.

G(u,v)=H(u,v)・F(u,v)...(2)
Here, H is the optical transfer function OTF obtained by Fourier transforming the point spread function PSF(h), G and F are functions obtained by Fourier transforming the degraded image g and the original image f, respectively, and (u, v) are coordinates on the two-dimensional frequency plane, i.e., frequency.

To obtain the original image f from the captured degraded image g, we simply divide both sides of the following equation (3) by the optical transfer function H.

G(u,v)/H(u,v)=F(u,v)...(3)
Then, F(u,v), i.e., G(u,v)/H(u,v), is subjected to an inverse Fourier transform to return it to the real plane, thereby obtaining the original image f(x,y) as a restored image.

If the inverse Fourier transform of H ⁻¹ is taken as R, then the original image f(x, y) can be obtained by performing convolution processing on the image on the real surface as shown in the following equation (4).

g(x,y)*R(x,y)=f(x,y)...(4)
Here, R(x, y) is called an image restoration filter. When the image is a two-dimensional image, the image restoration filter R is generally a two-dimensional filter having taps (cells) corresponding to each pixel of the image. In addition, the more the number of taps (cells) of the image restoration filter R, the higher the restoration accuracy. For this reason, a feasible number of taps is set according to the required image quality, image processing capability, aberration characteristics, and the like. The image restoration filter R must at least reflect the characteristics of aberration, and therefore is different from a conventional edge enhancement filter having about three taps each in the horizontal and vertical directions. The image restoration filter R is set based on the optical transfer function OTF, and therefore both the deterioration of the amplitude component and the phase component can be corrected with high accuracy.

Furthermore, because actual images contain noise components, if an image restoration filter R created by taking the inverse of the optical transfer function OTF as described above is used, the noise components will be significantly amplified while restoring the degraded image. This is because the MTF (amplitude component) of the optical system is raised to return it to 1 across all frequencies in a state where the amplitude of noise is added to the amplitude component of the image. The MTF, which is the amplitude degradation caused by the optical system, returns to 1, but at the same time the power spectrum of the noise is also raised, resulting in amplified noise according to the degree to which the MTF is raised (restoration gain).

Therefore, when noise is included, an image of good quality for viewing cannot be obtained. This is expressed by the following equations (5-1) and (5-2).
G(u,v)=H(u,v)・F(u,v)+N(u,v)...(5-1)
G(u,v)/H(u,v)=F(u,v)+N(u,v)/H(u,v)...(5-2)
Here, N is the noise component.

For images that contain noise components, there is a method for controlling the degree of restoration depending on the intensity ratio SNR of the image signal to the noise signal, such as the Wiener filter expressed by the following equation (6).

Here, M(u,v) is the frequency characteristic of the Wiener filter, and |H(u,v)| is the absolute value of the optical transfer function OTF (modulation transfer function MTF). In this method, for each frequency, the smaller the MTF is, the smaller the restoration gain (degree of restoration) is, and the larger the MTF is, the larger the restoration gain is. Generally, the MTF of an imaging optical system is high on the low frequency side and low on the high frequency side, so this method essentially reduces the restoration gain on the high frequency side of the image.

Next, the image restoration filter will be described with reference to Figs. 1 and 2. Figs. 1 and 2 are explanatory diagrams of the image restoration filter. The number of taps of the image restoration filter is determined according to the aberration characteristics of the imaging optical system and the required restoration accuracy. The image restoration filter of Fig. 1 is, as an example, a two-dimensional filter with 11 x 11 taps. In Fig. 1, the values (coefficients) in each tap are omitted, but a cross section of this image restoration filter is shown in Fig. 2. The distribution of the values (coefficient values) of each tap of the image restoration filter has the function of ideally returning the signal value (PSF) that has been spatially spread due to aberration to the original single point.

Each tap of the image restoration filter is subjected to convolution processing (convolution integral, product-sum) in the image restoration process corresponding to each pixel of the image. In the convolution process, in order to improve the signal value of a specific pixel, the pixel is aligned with the center of the image restoration filter. Then, for each corresponding pixel in the image and image restoration filter, the product of the image's signal value and the filter's coefficient value is taken, and the sum of these products is replaced as the signal value of the central pixel.

Next, the characteristics of image restoration in real space and frequency space will be described with reference to Fig. 3 and Fig. 4. Fig. 3 is an explanatory diagram of the point spread function PSF, Fig. 3(a) shows the point spread function PSF before image restoration, and Fig. 3(b) shows the point spread function PSF after image restoration. Fig. 4 is an explanatory diagram of the amplitude component MTF (Fig. 4(M)) and phase component PTF (Fig. 4(P)) of the optical transfer function OTF. The dashed line (a) in Fig. 4(M) shows the MTF before image restoration, and the dashed line (b) shows the MTF after image restoration. The dashed line (a) in Fig. 4(P) shows the PTF before image restoration, and the dashed line (b) shows the PTF after image restoration. As shown in Fig. 3(a), the point spread function PSF before image restoration has an asymmetric spread, and due to this asymmetry, the phase component PTF has a nonlinear value with respect to frequency. In the image restoration process, the amplitude component MTF is amplified and the phase component PTF is corrected to zero, so that the point spread function PSF after image restoration has a symmetrical and sharp shape.
In this way, the image restoration filter can be obtained by performing an inverse Fourier transform on a function designed based on the inverse function of the optical transfer function OTF of the imaging optical system. The image restoration filter that can be used as the sharpening filter in each embodiment can be changed as appropriate, and for example, the Wiener filter described above can be used. When using a Wiener filter, it is possible to create a real-space image restoration filter that is actually convolved with the image by performing an inverse Fourier transform on Equation (6).

[Extended depth of field]
In each embodiment, a wavefront modulation element (phase mask) that reduces the performance variation in the depth direction is inserted into the optical system, and a method (WFC: WaveFront Coding) is performed to expand the depth. The wavefront modulation element is one that gives a phase with a two-dimensional distribution in a plane perpendicular to the optical axis of the optical system, and is, for example, an optical element whose thickness or refractive index changes in the plane or a liquid crystal spatial phase modulation element, and may be given to an optical element in the optical system as an aspheric shape. In each embodiment, a cubic phase mask whose surface shape is expressed by a cubic function is used. If the direction perpendicular to the imaging element (the optical axis direction of the optical system) is the Z axis, and two axes perpendicular to the Z axis and perpendicular to each other are the X axis and the Y axis, the phase distribution φ of the wavefront modulation element in each embodiment can be expressed by an odd function shape such as the following formula (7).

φ=α( ^X3 + ^Y3 )...(7)
In addition, X and Y here do not necessarily coincide with x and y of the captured image or the imaging element, and hereinafter, the axial direction of the wavefront modulation element (the first axis and the second axis perpendicular to each other) is represented by X and Y, respectively. By providing this phase distribution, it is possible to slow down the performance fluctuation in the depth direction. In this embodiment, an optical system in which the depth is expanded using the wavefront modulation element is called a depth expansion optical system. In addition, the phase distribution used for depth expansion is not limited to the above. Since the robustness against the change in defocus and the balance between the high frequency side and the low frequency side of the MTF change, it is sufficient to select appropriately according to the required depth of field and frequency characteristics. By reducing the fluctuation of the optical performance in the depth direction by the wavefront modulation element, it is possible to apply the image restoration process using the same image restoration filter regardless of the depth. In addition, if a phase mask is inserted into the optical system to slow down the performance fluctuation in the depth direction, the performance fluctuation also tends to be slowed down with respect to the shooting conditions such as the image height and the object distance. In other words, if the performance variation with respect to image height in a depth-of-field optical system is sufficiently small, it can be treated as a uniform OTF for shooting conditions such as image height and object distance, and a uniform (fixed) image restoration filter can be used.

Figure 5 is a cross section of the MTF of an optical system including a wavefront modulation element represented by formula (7). Here, the aberration of the optical system is assumed to be small compared to the aberration imparted by the wavefront modulation element. In Figure 5, the horizontal axis represents the spatial frequency, F represents the F-number (aperture value) of the optical system, the solid line represents the MTF in the X-axis direction, and the dashed line represents the MTF in the diagonal (45 degree) direction. When the wavefront modulation element represented by formula (7) is used, the X-axis direction and the Y-axis direction are symmetrical, so the MTFs for the spatial frequencies in the X-axis direction and the Y-axis direction match. However, when comparing the MTF of the solid line with the MTF of the dashed line, the MTF of the dashed line is low from the low frequency side to the high frequency side, and there is a large difference from the MTF of the solid line. In this way, it is generally believed that the depth can be expanded by using a Cubic Phase Mask, but the MTF characteristics differ greatly depending on the direction.

[Pixel shift imaging and high pixel processing]
Next, an overview of the pixel-enhancing process will be described. In the pixel-enhancing process, a pixel-enhanced image is obtained from a plurality of captured images taken at different positions. In each embodiment, image data including images corresponding to a plurality of colors is obtained using a monochromatic image sensor in which all pixels receive light corresponding to the same wavelength range. In addition, at least one image included in the image data is captured by shifting it by a non-integer pixel with respect to a reference image included in the image data. Here, the "reference image" is an arbitrary image included in the image data that can be selected as a reference, and is a reference for defining the amount and direction of the shift. In addition, "imaging with a shift" refers to capturing an image by translating the position where the image of the subject formed by the imaging optical system is sampled in a plane perpendicular to the optical axis.

For example, by separating light passing through the imaging optical system into colors using a prism and arranging multiple imaging elements corresponding to each separated light beam, images corresponding to multiple colors can be simultaneously obtained. Since each imaging element corresponds to a different color, it may have sensitivity to a different wavelength. For example, it can be separated into three light beams of RGB and obtained. The number of separations is not limited to three, and may be two or four or more. In addition, in the case of three or more, multiple images corresponding to the same color may be obtained. Then, by arranging one imaging element by shifting it by a non-integer pixel relative to another imaging element, it is possible to sample the image of the subject at a position shifted by a non-integer pixel. Also, "arranging the imaging element by shifting" refers to arranging the imaging element so that the position where the image of the subject formed by the imaging optical system is sampled is moved in parallel in a plane perpendicular to the optical axis and imaged. Therefore, when the light beam is separated and bent, it is sufficient to shift it in a plane perpendicular to the optical axis.

Also, imaging may be performed multiple times with a single image sensor. Assuming there is no external light, the color of the illumination light illuminating the subject is changed in synchronization with the imaging, so that the images acquired by imaging multiple times each correspond to a different color based on the illumination light. By imaging while driving the image sensor in a direction perpendicular to the optical axis, the images corresponding to the multiple colors are shifted relatively. For example, such a method may be used in an endoscope device, etc., which does not have external light. Instead of driving the image sensor, this may be done by moving the optical system, some of the lenses included in the optical system, or the entire imaging device.

A more preferable method is to separate the light that has passed through the imaging optical system by color and arrange multiple imaging elements corresponding to each separated light beam. This eliminates the need for driving, and multiple images that are shifted by non-integer pixels can be obtained in one imaging session. This makes it possible to suppress the effects of subject blurring and also allows the images to be captured as a video. The high pixel processing described below can be performed for each frame of the video in the same way as for still images. By performing high pixel processing on images captured using this imaging method, it is possible to obtain a high pixel image with a resolution that exceeds that which can be obtained with a single imaging element.

Next, referring to FIG. 6, an example of imaging and high-pixel processing by pixel shifting is shown. FIG. 6(A) is an example of an imaging pattern shifted by a non-integer pixel. Here, three images, a reference image 51, an image 52, and an image 53, are acquired. The reference image 51 and the image 52 are captured at the same position without shift. On the other hand, the image 53 is captured with a half-pixel shift in the horizontal and vertical directions with respect to the reference image 51 and the image 52. For example, when capturing images using different image sensors for RGB, the shooting position of G can be shifted by a half pixel with respect to the shooting positions of R and B. That is, the reference image 51, the image 52, and the image 53 correspond to R, B, and G, respectively. Also, as shown in FIG. 6(B), an image of G that is not shifted relatively to the reference image 51 may be acquired as an image 54. Based on the multiple images included in the acquired image data, a high-pixel processing to be described later is performed, and a high-pixel image 55 having a larger number of pixels than each captured image can be acquired.

Next, referring to FIG. 7, a different example of imaging by pixel shifting and high pixel processing will be shown. FIG. 7 is an example of imaging in which the imaging pattern is different from that of FIG. 6 and the imaging is relatively shifted by a non-integer pixel. Here, four captured images to be used for high pixel processing are obtained by performing imaging with the shooting positions shifted from each other by half a pixel. The four captured images are a reference image 61 (no shift), an image 62 shifted by half a pixel in the horizontal direction, an image 63 shifted by half a pixel in the diagonal direction, and an image 64 shifted in the vertical direction. Image 63 is equivalent to a position shifted by half a pixel in the horizontal and vertical directions. By acquiring four captured images, a high pixel image 65 with four times the number of pixels due to high pixel processing can be acquired. For example, when capturing images using different imaging elements corresponding to four colors, the shooting positions can be shifted to acquire the images, and the reference image 61, image 62, image 63, and image 64 are made to correspond to R, G, B, and Y (yellow), respectively. Also, the reference images 61 and 63 may be G, image 62 may be R, and image 64 may be B.

In the high pixel processing shown in FIG. 7, the high pixel processing can be performed simply by arranging four images according to the sampling positions of the image of the subject. The high pixel processing in FIG. 6(A) is performed based on image 51 (or image 52) and image 53. FIG. 8 is an explanatory diagram of the high pixel processing. The white circles shown by solid lines and the white circles shown by dashed lines in FIG. 8 are the sampling positions of image 51 and image 53, respectively. The sampling positions may be, for example, the center positions of the pixels. The luminance value of the image corresponding to the position of the black circle can be obtained by interpolating the white circles shown by solid lines and the white circles shown by dashed lines with the nearest neighbors. The obtained black circle is twice as high pixelated in the horizontal and vertical directions as image 51 and image 53, and the luminance values corresponding to the positions of the black circles are arranged to form a high pixel image 55. As described above, the high pixel processing in each embodiment is performed based on a plurality of relatively shifted images, so that the effective sampling of the image of the subject is increased. In other words, unlike pixel-enhancing processing that uses interpolation from a single image, this method generates a pixel-enhancing image that has resolution up to a higher frequency than the Nyquist frequency corresponding to the single image or the Nyquist frequency of the image sensor. The Nyquist frequency will be described in more detail later.

However, in each embodiment, a pixel-enhancing process is performed to obtain a highly pixelated image of a specified color based on images of different colors shifted by a non-integer pixel. Since the pixel-enhancing process is based on images corresponding to different colors, it differs from the case of capturing images with a smaller pixel pitch (sampling period) and a high-pixel image sensor. Therefore, by reducing the difference between images corresponding to different colors and based on an image of a specified color and an image of a color different from the specified color, a highly pixelated image of the specified color is generated based on the pixel-enhancing process described above. When the subject has low saturation and is close to black and white, the difference between images corresponding to different colors can be reduced by adjusting the white balance.

On the other hand, in a normal color subject, the ratio of colors is different, so by acquiring only the change in luminance value in the image of each color, the difference between images corresponding to different colors can be reduced. This allows only the change in luminance value in the image of each color to be acquired, and by interpolating between different colors, it is possible to sample luminance changes up to high frequencies corresponding to a highly pixelated image. Since the highly pixelated process can be performed on colors that are relatively shifted by a non-integer pixel, it is possible to perform the highly pixelated process corresponding to all colors. For example, by performing the highly pixelated process on R and G, highly pixelated images of R and G can be obtained, and further, by performing the highly pixelated process on B and G, a highly pixelated image of B can be obtained. In addition, the RGB image may be converted into luminance and color difference signals, and the highly pixelated process may be performed on the luminance signal. Note that the highly pixelated process and the shift pattern of the captured image used therefor are not limited to those described above, and other methods may be used.

[Nyquist frequency]
Next, the relationship between image sampling and the Nyquist frequency will be described with reference to Fig. 9. Fig. 9 is an explanatory diagram of the Nyquist frequency resulting from imaging by pixel shifting and high pixel processing. When an image of a subject is sampled at a pixel pitch p, i.e., a sampling period p, the sampling frequency is the reciprocal of the sampling period, 1/p. The Nyquist frequency is half the sampling frequency, i.e., 1/2p. According to the sampling theorem, when an object is sampled, it can be accurately reproduced as long as the object has a frequency equal to or lower than the Nyquist frequency. Since an image is a two-dimensional sampling of an image of a subject, the Nyquist frequency is also given in two dimensions.

The Nyquist frequency when sampling with the white circles shown by the solid lines in FIG. 9(A) is the dashed line 81 in FIG. 9(B). Even when sampling with the white circles shown by the dashed lines in FIG. 9(A), the sampling position shifts, but the period of the two-dimensional arrangement and the arrangement direction do not change. Therefore, the Nyquist frequency when sampling with the white circles shown by the dashed lines is similarly the dashed line 81 in FIG. 9(B). However, the Nyquist frequency when sampling with both the white circles shown by the solid lines in FIG. 9(A) and the white circles shown by the dashed lines in FIG. 9(A) is the solid line 82 in FIG. 9(B). This corresponds to the Nyquist frequency in a highly pixelated image when high-pixel processing equivalent to FIG. 6 is performed. On the other hand, the white circles shown by the solid lines in FIG. 9(C) are the same as those in FIG. 9(A), but the white circles shown by the dashed lines correspond to FIG. 7 and are more numerous than those in FIG. 9(A). The Nyquist frequency when sampling is performed on all the white circles indicated by the solid line in Figure 9 (C) and the white circles indicated by the dashed line is the solid line 83 in Figure 9 (D).

Therefore, if the pixel pitch of the imaging element used in each embodiment is p, the Nyquist frequency of each captured image is the dashed line 81 in FIG. 9(B). However, as described above, when a high-pixel image of a specific color is generated based on images of different colors, the Nyquist frequency changes. Therefore, the Nyquist frequency of each image included in the image data is not necessarily the same as that of the high-pixel image. For example, in the examples of FIGS. 9(A) and (B), the direction in which the Nyquist frequency is highest also changes from the diagonal direction to the x direction and y direction. The above describes the image restoration process, depth of field extension, pixel shift imaging and high-pixel processing, and two-dimensional Nyquist frequency in this embodiment.

Next, the configuration of the imaging device in each embodiment will be described with reference to FIG. 10. FIG. 10 is a block diagram of an imaging device 100. An image processing program for performing the sharpening process (image restoration process) of this embodiment is installed in the imaging device 100, and this sharpening process is executed by an image processing unit (image processing device) 104 inside the imaging device 100.

The imaging device 100 comprises an optical system (imaging optical system) 101 and an imaging device body (camera body). The optical system 101 comprises an aperture 101a, a focus lens 101b, and a wavefront modulation element 101c, and is configured integrally with the camera body. However, each embodiment is not limited to this, and can also be applied to an imaging device in which the optical system 101 is detachably attached to the camera body. Furthermore, the optical system 101 may be configured to include optical elements having refractive surfaces such as lenses, as well as optical elements having diffractive surfaces and optical elements having reflective surfaces.

The wavefront modulation element 101c is non-rotationally symmetric with respect to the optical axis. Note that the wavefront modulation element 101c is arranged adjacent to the image side of the aperture 101a, but is not limited to this and may be arranged at a position different from the position shown in FIG. 10. The imaging optical system is composed of each optical element arranged on the object side of the imaging unit 102.

The imaging unit 102 has one or more imaging elements such as a CCD or CMOS sensor, and generates a captured image by photoelectrically converting the subject image (optical image formed by the optical system 101) formed by the optical system 101. That is, the subject image is converted into an analog signal (electrical signal) by photoelectric conversion by the imaging unit 102. This analog signal is then converted into a digital signal by the A/D converter 103. This digital signal is input to the image processing unit 104.

The image processing unit 104 performs a predetermined process on the digital signal, and also performs the sharpening process of this embodiment. In addition to the predetermined process, the image processing unit 104 also performs high pixel processing and image restoration processing.

Optical characteristic data such as OTF and PSF required for sharpening processing, recovery filters, or data required for generating them are stored in a memory unit (storage means) 108. The memory unit 108 is composed of, for example, a ROM. The output image processed by the image processing unit 104 is saved in a predetermined format in an image recording medium 109. An image obtained by performing predetermined processing for display on an image that has been subjected to sharpening processing is displayed on a display unit 105 composed of an LCD monitor or an organic EL display. However, the images displayed on the display unit 105 are not limited to this, and an image that has been subjected to simple processing for high-speed display may be displayed on the display unit 105.

The system controller 110 controls the imaging device 100. The optical system 101 is mechanically driven by the optical system control unit 106 based on instructions from the system controller 110. The optical system control unit 106 controls the aperture diameter of the diaphragm 101a to obtain a predetermined F-number. The optical system control unit 106 also controls the position of the focus lens 101b using an autofocus (AF) mechanism or a manual focus mechanism (not shown) to adjust the focus according to the subject distance. Note that functions such as the aperture diameter control of the diaphragm 101a and manual focus do not have to be performed depending on the specifications of the imaging device 100.

Note that optical elements such as a low-pass filter or an infrared cut filter may be placed between the optical system 101 and the imaging unit 102, but when using an element that affects the optical characteristics such as a low-pass filter, consideration may be required when creating the sharpening filter. The infrared cut filter also affects each PSF of the RGB channels, which is the integral value of the point spread function (PSF) of the spectral wavelength, especially the PSF of the R channel, so consideration may be required when creating the sharpening filter. Therefore, the sharpening filter may be changed depending on the presence or absence of a low-pass filter or an infrared cut filter.

The image processing unit 104 is configured as an ASIC, and the optical system control unit 106, the state detection unit 107, and the system controller 110 are each configured as a CPU or an MPU. In addition, one or more of the image processing unit 104, the optical system control unit 106, the state detection unit 107, and the system controller 110 may be configured to share the same CPU or MPU.

[Example 1]
Next, a first embodiment of the present invention will be described. In this embodiment, the light passing through the imaging optical system is separated into three light beams corresponding to RGB by a prism, and imaging elements corresponding to each of them are arranged. All of the imaging elements have a square lattice pixel arrangement with the same pixel pitch, and the rotation directions around the optical axis are the same, that is, the pixel arrangement directions of the three imaging elements are the same. Here, the "direction of pixel arrangement" refers to the direction in which the interval between pixel centers is narrowest, and in the case of a square lattice pixel arrangement, it is the horizontal direction and the vertical direction. Therefore, in this embodiment, the two-dimensional Nyquist frequency is equal for all of RGB. Then, only the imaging element corresponding to G is arranged by shifting it by a non-integer pixel relative to the imaging elements corresponding to R and B. Here, it is shifted in the diagonal direction by shifting it by half a pixel in the horizontal direction and the vertical direction, respectively. At this time, the shift pattern during imaging corresponds to FIG. 6(A). Image data consisting of three images corresponding to RGB, respectively, is obtained by these three imaging elements. In addition, the wavefront modulation element has a phase distribution according to formula (7). The X and Y directions of the wavefront modulation element coincide with the pixel array directions (x and y directions).

Here, the Nyquist frequency of each of RGB corresponds to the dashed line 81 in FIG. 9B. In this case, the direction with the highest Nyquist frequency corresponding to each color of RGB is the diagonal direction in the lattice of the pixel array of the image sensor. On the other hand, the direction with the largest MTF of the image sensor is determined by the direction around the optical axis of the wavefront modulation element, and is the X direction and Y direction in formula (7). In order to make the most of the performance of the image sensor, it is preferable that the direction with the largest MTF of the image sensor and the direction with the highest Nyquist frequency of the image coincide with each other. However, in this embodiment, high pixel processing is performed on all colors of RGB, and the Nyquist frequency of each of RGB of the high pixel image corresponds to the solid line 82 in FIG. 9B. That is, the Nyquist frequency as an image (the Nyquist frequency in each image) is the largest in the diagonal direction, but by performing high pixel processing using images of different colors that are shifted in the diagonal direction, the direction with the highest Nyquist frequency changes before and after image processing. Therefore, in the arrangement of the image sensor and the high pixel processing of this embodiment, by arranging the pixel array direction (x direction and y direction) and the X direction and Y direction of the wavefront modulation element to coincide, the performance of the imaging optical system can be maximized by the image sensor, resulting in a high pixel image with a higher resolution.

In this embodiment, the RGB images are captured one by one, but as described in FIG. 6B, the G image that is not shifted relative to the R and B images may be captured in an arrangement. This can be achieved by further separating the light beam corresponding to G and arranging an image pickup element for the two separated light beams. Therefore, the image data consists of four images of RGGB. In this case, the Nyquist frequency of each image does not change, but sampling is performed for G at the positions of the white circle shown by the solid line in FIG. 9A and the white circle shown by the dashed line, and the Nyquist frequency corresponding to G is the solid line 82 in FIG. 9B. In this case, the directions in which the Nyquist frequency is highest differ between the R, B images and the G image contained in the image data, and the optimal direction of the wavefront modulation element is not uniquely determined. However, by performing the above-mentioned high pixel processing, the direction in which the Nyquist frequency is highest coincides with the horizontal direction, so it is sufficient to arrange the pixel array direction and the X and Y directions of the wavefront modulation element to coincide with each other. If high pixel count processing is not performed by performing sampling shifted diagonally by G, the direction of the highest Nyquist frequency corresponding to each color will be different, and the orientation of the wavefront modulation element cannot be optimized for all colors in the image data. However, by combining different colors and performing high pixel count processing, the direction of the highest Nyquist frequency of the high pixel count image can be aligned for all colors. Therefore, the optimal arrangement is one in which the pixel array direction (x direction, y direction) and the X direction and Y direction of the wavefront modulation element are aligned.

Here, "arrangement in which the pixel array direction (x direction, y direction) and the X direction and Y direction of the wavefront modulation element coincide" corresponds to an arrangement in which the wavefront modulation element is arranged so that the direction in which the MTF of the imaging optical system is the largest forms an angle of 45 degrees with respect to the shift direction, which is the diagonal direction of the imaging element. Note that "diagonal direction of the imaging element" means the diagonal direction in the lattice of the pixel array of the imaging element. In other words, the angle between the direction in which the MTF of the imaging optical system is the largest (first direction) and the shortest shift direction (second direction) is approximately 45 degrees. By bringing the angle between the first direction and the second direction closer to 45 degrees, the performance of the imaging optical system can be utilized, but it is not strictly limited to 45 degrees, and it is sufficient that it is approximately 45 degrees. For example, this angle is set to 30 degrees or more and 60 degrees or less. More preferably, this angle is set to 40 degrees or more and 50 degrees or less. The direction (first direction) in which the MTF of the imaging optical system is greatest is the two directions of the X and Y directions of the wavefront modulation element (directions indicated by arrows a and b in Figures 9(A) and (C)), each of which is at 45 degrees to the shift direction (second direction (direction indicated by arrow c)).

According to this embodiment, by taking into account the shift caused by pixel shifting during image capture using a depth-expanding optical system, it is possible to realize an image capture device capable of capturing images with more preferable image quality.

[Example 2]
Next, a second embodiment of the present invention will be described. In this embodiment, the pattern of shifting the imaging element is different from that of the first embodiment. The light passing through the imaging optical system is separated into four light beams corresponding to RGBY (Y is yellow) by a prism, and the corresponding imaging element is arranged for each. All four imaging elements have the same pixel pitch and a square lattice pixel arrangement, and the rotation direction around the optical axis is the same, that is, the direction of the pixel arrangement is the same. Therefore, in this embodiment, the two-dimensional Nyquist frequency is the same for all RGBY. Then, the imaging elements are arranged so that the image obtained by the R imaging element is used as a reference image, the image corresponding to G is shifted by half a pixel in the horizontal direction, the image corresponding to Y is shifted by half a pixel in the vertical direction, and the image corresponding to B is shifted by half a pixel in the horizontal and vertical directions. At this time, the shift pattern during imaging corresponds to FIG. 7. These four imaging elements obtain image data consisting of four images corresponding to RGBY. In addition, the wavefront modulation element has a phase distribution according to formula (7). The X and Y directions of the wavefront modulation element are approximately 45 degrees to the pixel arrangement direction (x and y directions).

Here, the Nyquist frequency of each of RGBY corresponds to the dashed line 81 in FIG. 9C. In this case, the direction with the highest Nyquist frequency corresponding to each color of RGBY is the diagonal direction in the lattice of the pixel array of the image sensor. On the other hand, the direction with the highest MTF of the image sensor is determined by the direction around the optical axis of the wavefront modulation element, and is the diagonal (45 degree diagonal) direction expressed by Y=X using X and Y in equation (7). In order to make the most of the performance of the image sensor, it is preferable that the direction with the highest MTF of the image sensor and the direction with the highest Nyquist frequency of the image coincide with each other. In this embodiment, high pixel processing is performed for all colors of RGBY. The Nyquist frequency of each of RGBY corresponds to the solid line 83 in FIG. 9D. That is, the Nyquist frequency as an image (the Nyquist frequency in each image included in the image data) is the highest in the diagonal direction. In addition, by performing high pixel processing using images of different colors that are shifted in the diagonal direction, the direction with the highest Nyquist frequency before and after the image processing also coincides. Therefore, in the arrangement of the image sensor and the high pixel processing of this embodiment, by arranging the pixel array direction (x direction, y direction) and the Y=X (45 degree diagonal) direction of the wavefront modulation element to coincide, the performance of the imaging optical system can be maximized by the image sensor, resulting in a high pixel image with a higher resolution.

Compared with Example 1, it can be seen that even if the orientation of the imaging element is the same, the direction in which the Nyquist frequency of the highly pixelated image is greatest changes depending on the shift direction when acquiring the captured image used in the high pixelization process. In this example, there are three types of shift with respect to the reference image, and the orientation of the wavefront modulation element can be set with respect to the shortest shift direction, that is, the horizontal or vertical direction. For the smallest shift amount, i.e., the shortest shift direction, the diagonal sampling is the finest. Therefore, by setting the direction in which the MTF of the imaging optical system is greatest with respect to the shortest shift direction to approximately 45 degrees, the imaging element can make the most of the performance of the imaging optical system to obtain a highly pixelated image with a higher sense of resolution.

The direction (first direction) in which the MTF of the imaging optical system is greatest is the two directions of the X and Y directions of the wavefront modulation element (directions indicated by arrows a and b in FIG. 9C), each of which is at 45 degrees to the shift direction (second direction (directions indicated by arrows c and d)).

It is also preferable that the sampling periods of all images contained in the captured image data are equal. If the sampling periods are different, the shift direction in each local region of the image will change, and the direction in which the Nyquist frequency of the highly pixelated image is greatest will also change. By making the sampling periods equal, the shift direction can be uniquely determined, and by arranging the direction in which the MTF of the imaging optical system is greatest and the direction of the shortest shift among the shifts at approximately 45 degrees, an optimal sense of resolution can be obtained across the entire image.

It is also preferable that the directions of the highest Nyquist frequencies corresponding to the colors included in the image data are all the same. By making the directions of the highest Nyquist frequencies the same for each color, the directions of the highest Nyquist frequencies of the highly pixelated image become the same for all colors. Therefore, an arrangement in which the direction of the largest MTF of the imaging optical system and the direction of the shortest shift among the shifts are at approximately 45 degrees can be optimal for all colors.

The system may also have a mode in which an output image is obtained using an image captured with a relative shift from among the acquired image data, and a mode in which the output image is obtained without using an image captured with a shift. In this case, high pixel processing is performed in the former mode, but not in the latter mode. In imaging using the shift pattern of Example 1, the direction in which the MTF of the imaging optical system is largest differs between the former and latter modes. Therefore, it is preferable to determine the direction of the wavefront modulation element depending on the mode. In this case, the wavefront modulation element may be driven, or an instruction may be given to the user to change the direction of the wavefront modulation element.

The wavefront modulation element and the imaging unit may be detachable. When an imaging unit in which multiple imaging elements are shifted and arranged is used, the shift pattern differs depending on the imaging unit. Therefore, by changing the direction of the wavefront modulation element depending on the imaging unit, each imaging unit can be arranged to make the most of the optical performance of the imaging optical system. In particular, an image captured by an imaging unit that does not shift the imaging element and a highly pixelated image captured by an imaging unit in which the imaging element is shifted only in the diagonal direction of the pixel array lattice have different directions in which the Nyquist frequency is greatest. Therefore, in an imaging optical system in which both of these can be attached, it is preferable to determine the direction of the wavefront modulation element by the imaging unit. In this case, the wavefront modulation element may be driven, or an instruction may be given to the user to change the direction of the wavefront modulation element.

In each embodiment, an input image may be output from the imaging device to an image processing device provided separately from the imaging device, and image processing may be performed by the image processing device. Furthermore, the image restoration processing and high pixel processing in each embodiment do not have to be performed by a single image processing device. For example, a program for realizing part or all of the processing in each embodiment may be supplied to one or more systems or devices via a network or recording medium, and the program may be executed by the system or device.

As described above, in each embodiment, the imaging device 100 has an imaging optical system (optical system 101) including a wavefront modulation element 101c that is rotationally asymmetric with respect to the optical axis, and an imaging unit 102 that photoelectrically converts an optical image formed by the imaging optical system. The imaging unit acquires first image data corresponding to a first color in the first imaging, and acquires second image data corresponding to a second color in the second imaging (image data corresponding to multiple colors is acquired by acquiring multiple images corresponding to a single color acquired for each imaging). The second image data is image data that is imaged by shifting the first image data by a non-integer pixel of the imaging unit (at least one image of the image data is imaged by shifting the image by a non-integer pixel with respect to a reference image that serves as a reference among the image data). In addition, the angle between the first direction in which the modulation transfer function MTF of the imaging optical system is largest and the second direction, which is the shortest among the shift directions, is 30 degrees to 60 degrees (approximately 45 degrees).

Preferably, the second direction is different from the pixel array direction (x direction and y direction) of the imaging unit. Also preferably, the sampling periods (pixel pitch) of the first image data and the second image data are equal to each other. Also preferably, the directions of the highest Nyquist frequencies in the first image data and the second image data are equal to each other. Also preferably, the first direction and the direction of the highest Nyquist frequency corresponding to at least one image sensor of the imaging unit are different from each other. Also preferably, the first direction and the direction of the highest Nyquist frequency corresponding to at least one of the first color or the second color are different from each other. Also preferably, the first color is R or B, and the second color is G.

Preferably, the imaging device has an image processing unit 104 that generates a high-pixel image using the first image data and the second image data. More preferably, the image processing unit performs image restoration processing using the first image data and the second image data. Also preferably, the image processing unit has a mode for generating an output image. The modes include a first mode for generating an output image using both the first image data and the second image data, and a second mode for generating an output image using either the first image data or the second image data. More preferably, the direction of the wavefront modulation element is determined based on the mode.

Preferably, the wavefront modulation element modulates the wavefront equally with respect to a first axis and a second axis that are orthogonal to each other, and has an odd function shape along each of the first axis and the second axis. The odd function shape is, for example, a shape expressed by Equation (7), but is not limited thereto. Preferably, the imaging optical system and the imaging unit are detachable. At this time, for example, when the imaging optical system is attached to an imaging unit that does not acquire image data including a relatively shifted image, the wavefront modulation element is arranged in a different direction from when it is attached to an imaging unit that acquires image data including a relatively shifted image. Also preferably, the imaging unit has a first imaging element that receives light corresponding to a first color and a second imaging element that receives light corresponding to a second color. At this time, for example, the second imaging element is arranged so as to acquire an image of the subject at a position that differs by a non-integer pixel of the pixel array from the first imaging element. Alternatively, preferably, the imaging unit has one imaging element that receives both light corresponding to the first color and light corresponding to the second color.

According to each embodiment, it is possible to provide an imaging device capable of obtaining high-quality images using a depth-expanding optical system.

The above describes preferred embodiments of the present invention, but the present invention is not limited to these embodiments, and various modifications and variations are possible within the scope of the gist of the invention.

100 Imaging device 101 Imaging optical system 101c Wavefront modulation element 102 Imaging section

Claims (15)

an imaging optical system including a wavefront modulation element that is rotationally asymmetric with respect to an optical axis;
an imaging unit that photoelectrically converts an optical image formed by the imaging optical system,
The imaging unit acquires first image data corresponding to a first color and one or more shifted image data including second image data corresponding to a second color;
the one or more shifted image data are each acquired by imaging in which the optical image is shifted by a shift amount of a non-integer pixel of the imaging unit with respect to the first image data,
A direction in which the modulation transfer function of the imaging optical system is the largest is defined as a first direction, a shift direction corresponding to the smallest shift amount among the shift amounts of each of the one or more shift image data is defined as a second direction, and an angle between the first direction and the second direction is defined as θ [°].
30≦θ≦60
An imaging apparatus characterized in that the following condition is satisfied . The imaging device according to claim 1, characterized in that the second direction is different from the pixel array direction of the imaging unit. 3. The imaging apparatus according to claim 1, wherein the first image data and the second image data are sampled at the same period. 4. The imaging device according to claim 1, wherein the first image data and the second image data have the same direction in which the Nyquist frequency is highest. 5. The imaging device according to claim 1, wherein the first direction and a direction in which a Nyquist frequency corresponding to at least one image sensor of the imaging section is highest are different from each other. 6. The imaging device according to claim 1, wherein the first direction and a direction in which a Nyquist frequency corresponding to at least one of the first color and the second color is highest are different from each other. the first color is R or B;
7. The imaging device according to claim 1, wherein the second color is G. The imaging device according to any one of claims 1 to 7, further comprising an image processing unit that generates a high-pixel image using the first image data and the second image data. The imaging device according to claim 8, characterized in that the image processing unit performs image restoration processing using the first image data and the second image data. the image processor has a mode for generating an output image;
The mode is
a first mode for generating the output image using the first image data and the second image data ;
10. The imaging apparatus according to claim 8, further comprising a second mode in which the output image is generated using one of the first image data and the second image data. 11. The imaging apparatus according to claim 10, wherein a rotation direction of the wavefront modulation element with respect to the optical axis is determined based on the mode. The imaging device according to any one of claims 1 to 11, characterized in that the wavefront modulation element modulates the wavefront equally along a first axis and a second axis that are orthogonal to each other, and has an odd function shape along each of the first axis and the second axis. The imaging device according to any one of claims 1 to 12, characterized in that the imaging optical system and the imaging unit are detachable. The imaging unit includes:
a first image sensor that receives light corresponding to the first color;
14. The imaging device according to claim 1, further comprising a second imaging element that receives light corresponding to the second color. 14. The imaging device according to claim 1, wherein the imaging section includes one imaging element that receives light corresponding to the first color and light corresponding to the second color.