JP7620801B2 - Imaging device and imaging method - Google Patents

Description

The present invention relates to an imaging device and an imaging method.

Recently, research into plenoptic cameras has progressed with the development of content that includes three-dimensional (hereinafter referred to as "3D") images. Plenoptic cameras differ from conventional cameras that record two-dimensional light intensity distributions in that they directly obtain information about the light intensity distribution in three-dimensional space.

In a plenoptic camera, a microlens array is placed between the main lens and the image sensor to capture the aerial image as a group of light rays from a large number of point light sources. Depending on the optical arrangement of the plenoptic camera, there are configurations that have good angular resolution (a characteristic related to perspective) and configurations that have good spatial resolution (a characteristic related to resolution).

As a configuration for increasing spatial resolution, a plenoptic camera using a microlens array with different focal lengths has been proposed (see, for example, Patent Document 1). This plenoptic camera uses two or more microlens arrays with fixed shapes to capture microimages from image planes at different distances, increasing spatial resolution in the depth direction.

On the other hand, there are known variable-focus zoom lenses that combine a liquid concave lens, in which a liquid is sealed in a transparent film, with a convex lens (see, for example, Patent Document 2), and microlens arrays that adjust the optical properties of optical scatterers formed in openings in electrodes by adjusting the applied voltage (see, for example, Patent Document 3).

U.S. Patent No. 8,400,555 JP 2011-13578 A (Sony) JP 2019-120947 A

It is generally said that in a plenoptic camera, there is a trade-off between angular resolution and spatial resolution, and it is difficult to achieve both angular resolution and spatial resolution with a simple configuration. In Patent Document 1, spatial resolution is increased by using multiple microlens arrays with fixed shapes, but there is no mention of achieving both spatial resolution and angular resolution or switching between them.

The present invention aims to provide an imaging device that has a simple configuration and can achieve both angular resolution and spatial resolution, or can switch between them.

In one aspect of the present invention, an imaging device includes a light sensor having a plurality of sensor elements, a main lens that guides light from an object to the light sensor, and a microlens array disposed between the light sensor and the main lens,
The microlens array has a three-layer structure including a first electrode that is transparent to visible light, a second electrode that has a plurality of openings and is transparent to visible light and faces the first electrode, and a dielectric polymer that is disposed between the first electrode and the second electrode. Lenses are formed in the openings by application of a voltage.
The imaging device selectively operates in a first mode with good angular resolution and a second mode with good spatial resolution depending on the level of the applied voltage.

An imaging device that can achieve both angular resolution and spatial resolution or switch between them with a simple configuration is realized.

FIG. 1 is a diagram illustrating an optical system with good angular resolution. FIG. 1 is a diagram illustrating an optical system with good spatial resolution. 5A to 5C are schematic diagrams illustrating the operation of the optical system when a first mode with good angular resolution is selected in the imaging device of the embodiment. FIG. 13 is a schematic diagram for explaining the operation of the optical system when a second mode having good spatial resolution is selected in the imaging device of the embodiment; 13A and 13B are diagrams illustrating adjustment of the focal length in the second mode. FIG. 1 is a diagram illustrating an example in which the imaging device according to the embodiment is used as a normal camera. 1A to 1C are diagrams illustrating a setup for stereoscopic imaging using an imaging apparatus according to an embodiment. 4 shows an image captured by controlling on/off of a voltage applied to a microlens array according to an embodiment. 1 is a block diagram of an information processing apparatus incorporating an imaging apparatus according to an embodiment of the present invention. 4 is a flowchart of an imaging method according to an embodiment.

Figure 1A is a schematic diagram of a plenoptic optical system that focuses on angular resolution, and Figure 1B is a schematic diagram of a plenoptic optical system that focuses on spatial resolution. The configuration in Figure 1A is called "Plenoptic 1.0," and the configuration in Figure 1B is called "Plenoptic 2.0." As described below, the imaging device of the embodiment achieves both angular resolution and spatial resolution in a single imaging device, or makes them switchable, thereby realizing both "Plenoptic 1.0" and "Plenoptic 2.0."

In FIG. 1A, a light ray emitted from a point on an object OB is guided through a main lens Lmain to a microlens ML1 on a microlens array MLA. The light ray incident on the microlens ML1 is split into multiple light rays depending on the direction of the light contained in the light ray, and exposes pixels P1, P2, and P3 of an image sensor IS.

In FIG. 1A, only three pixels are shown, but if, for example, the diameter of the microlens ML1 is 100 μm and one side of the pixel of the image sensor IS is 10 μm, then 10 x 10 pieces of information can be obtained in the angular direction. The distance between the microlens array MLA and the image sensor IS is the focal length f of the microlens array. The focal length f refers to the distance between the principal point pp, defined by the refractive index and radius of curvature of the microlens array, and the focal point of the microlens. Note that, although the convex side of the lens faces away from the image sensor IS in FIG. 1A, it does not matter if it faces the image sensor IS side.

In the configuration of FIG. 1B, light rays emitted from a point on an object OB are focused by multiple microlenses as a virtual image on the image plane IP of the main lens Lmain. This configuration has a lower angular resolution than FIG. 1A, but because the light focused by each microlens is detected, it has a high ability to distinguish between multiple objects at different spatial positions.

The distance a from the microlens array MLA to the image plane IP of the main lens 30 and the distance b from the microlens array MLA to the image sensor IS are expressed as follows, where f is the focal length of the microlens: 1/f=1/a+1/b.
In the configuration of FIG. 1B, f≠b.

In this embodiment, a variable-focus microlens array is used, in which the curvature of the microlenses can be controlled by controlling the applied voltage. The position of the microlens array is fixed, and the array is switched between a mode with good angular resolution and a mode with good spatial resolution by controlling the applied voltage and driving at least one of the main lens and the optical sensor.

(Switching between first and second modes)
Fig. 2 is a schematic diagram of the imaging device 10 according to the embodiment. In Fig. 2, the imaging device 10 operates in a first mode (see Fig. 1A) having good angular resolution. The imaging device 10 includes a main lens 30, an optical sensor 11, and a microlens array 20 disposed between the main lens 30 and the optical sensor 11.

The microlens array 20 has a three-layer structure consisting of a first electrode 21 that is transparent to visible light, a second electrode 23 with a plurality of openings 24, and a dielectric polymer 22 disposed between the first electrode 21 and the second electrode 23. The first electrode 21 and the second electrode 23 are connected to a variable voltage source 26.

The dielectric polymer 22 is polyvinyl chloride (PVC), polymethyl methacrylate (PMMA), polyurethane (PU), polystyrene (PSt), polyvinyl acetate (PVAc), polyvinyl alcohol (PVA), polycarbonate (PC), polyethylene terephthalate (PET), polyacrylonitrile (PAN), silicone rubber (SR), etc. As the dielectric polymer 22, a material that is transparent to visible light and has good voltage response can be appropriately selected. In the embodiment, PVC is used, which is easily handled and deforms greatly under the action of an electric field.

At least one of a plasticizer, an ionic liquid, and an ionic surfactant may be added to the dielectric polymer 22. The plasticizer provides flexibility to the dielectric polymer 22. The ionic liquid and the ionic surfactant can promote deformation of the dielectric polymer 22 and reduce the applied voltage. The dielectric polymer 22 may be dissolved in a suitable solvent.

When a first voltage is applied to the microlens array 20, a predetermined amount of the dielectric polymer protrudes from the opening 24 of the second electrode 23. The protruding polymer functions as a lens 25. The shape of the lens 25 is determined by the applied voltage, the type of dielectric polymer, the surface tension acting on the dielectric polymer, etc.

The openings 24 have a size and shape that allow lenses 25 to be formed in the openings 24 by deformation of the dielectric polymer 22 when a voltage is applied to the microlens array 20. The planar shape of the openings 24 can be designed according to the application of the lenses, such as a circle, an ellipse, an oval, a rectangle, etc. Furthermore, the planar arrangement of the openings 24 can be freely set, such as a lattice arrangement or a honeycomb arrangement. The image quality of the displayed image depends on the amount of light entering through the lenses 25 formed in the openings 24, and high image quality can be obtained by arranging the openings so that the aperture ratio, which represents the area occupied by the openings, is high. The aperture ratio is preferably 50% or more.

The range of voltage applied to the microlens array 20 is 800 V or less, and more preferably 600 V or less. The voltage actually applied differs depending on whether the operation of the imaging device 10 prioritizes angular resolution or spatial resolution. Even in the same operating mode, the voltage is also affected by the size of the opening 24 of the electrode 23, the thickness of the second electrode 23, etc.

The focal length of the lenses 25 of the microlens array 20 depends on the protruding displacement of the polymer (lens thickness) and can be controlled by the applied voltage. In this sense, the microlens array 20 may be called an "active lens." The focal length of the lenses 25 is 0.1 mm to 10 mm, depending on the diameter of the opening 24. The amount of light entering the lens also depends on the brightness of the image, so it is desirable for it to be as high as possible. As a guideline, the F-number (= focal length/lens diameter) is preferably 8.0 or less.

The main lens 30 is a fixed focal length lens. The main lens 30 is placed between the optical sensor 11 and the object to be photographed, and focuses light rays from a point on the object onto the lens 25 of the microlens array 20.

The optical sensor 11 is a CCD sensor, a CMOS sensor, or the like, having a plurality of sensor elements 111. The CCD sensor, the CMOS sensor, or the like digitally detects the amount of incident light and outputs an electrical signal according to the intensity of the incident light. Each sensor element 111 may be provided with a color filter of red (R), blue (B), green (G), or the like.

When lenses 25 are formed in the microlens array 20 by applying a voltage, light rays from an object are focused by the main lens 30 and enter the lens 25. The light rays incident on the lens 25 are split into multiple sensor elements 111 covered by the lens 25. A three-dimensional image with good angular resolution is formed by computing and processing the information obtained by each sensor element 111.

In FIG. 3, the imaging device 10 operates in a second mode with better spatial resolution. The components of the imaging device 10 are the same as those in FIG. 2, including a main lens 30, a light sensor 11, and a microlens array 20 disposed between the main lens 30 and the light sensor 11.

When a voltage V2 different from that shown in FIG. 2 is applied to the microlens array 20, the protrusion amount or curvature of the lens 25 formed in the opening 24 changes, and the focal length of the microlens array 20 changes.

The main lens 30 is a single-focus lens, and the position of the microlens array 20 is fixed. The focal length of the microlens array 20 is shortened by voltage control, and the position of the main lens 30 and the position of the optical sensor 11 are adjusted. The virtual image on the image plane IP is focused by multiple lenses 25, and images at different spatial positions are captured by the sensor elements 111 in the areas covered by each lens 25.

When the position of the microlens array 20 is fixed, the focal length f in FIG. 1A is not equal to the distance "b" in FIG. 1B (b ≠ f). However, by changing the focal length of the microlens array 20, the distance a to the image plane IP can be adjusted to capture a three-dimensional image with good spatial resolution. The distance b from the lens 25 to the incident surface of the optical sensor 11 may also be adjusted.

(Change focal length)
The microlens array 20 is capable of not only switching between the first mode and the second mode, but also adjusting the focal length in the first mode or the second mode.

Figure 4 shows the adjustment of the focal length in the second mode with good spatial resolution. For example, only the microlens array 20 may be fixed, and the positions of the main lens 30 and the optical sensor 11 may be variable. In (A) of Figure 4, the voltage applied to the microlens array 20 is set to V1, and a lens 25 with a small curvature is formed in the aperture 24. The virtual image on the image plane IP of the main lens 30 is focused by each lens 25 and captured by the sensor element 111 covered by this lens 25.

In FIG. 4B, the voltage applied to the microlens array 20 is set to V2 (V2>V1), which is greater than V1, and lenses 25 with a greater curvature than those in FIG. 4A are formed in the aperture 24. The virtual image on the image plane IP of the main lens 30 is focused by each lens 25 and captured by the sensor element 111 of the corresponding optical sensor 11. According to the change in the focal length of the lens 25, the position of the optical sensor 11 is adjusted to adjust the distance b from the lens 25 to the sensor plane, and also to adjust the distance from the lens 25 to the image plane IP.

By using one microlens array 20 and controlling the voltage applied to the microlens array 20, it is possible to obtain closer or more distant spatial information.

This change in focal length by controlling the applied voltage can also be applied to adjusting the focal length in the first mode of FIG. 2. The position of the microlens array 20 is fixed, and the positions of the optical sensor 11 and main lens 30 are made variable to change the voltage applied to the microlens array 20. If the focal length of the lens 25 formed in the opening 24 when voltage V1 is applied is "f", then applying a voltage V2 larger than V1 can increase the curvature of the lens 25 formed in the opening 24 and shorten f.

Even if the position of the microlens array 20 is fixed, by adjusting the positions of the optical sensor 11 and the main lens 30, angle information can be obtained from a closer or more distant space.

(Switching to normal camera)
5 shows an example in which the imaging device 10 of the embodiment is used as a normal camera. When a voltage is applied to the microlens array 20, a lens 25 is formed in the opening 24 of the second electrode 23, and the imaging device 10 functions as a plenoptic camera. By turning off the voltage applied to the microlens array 20, the imaging device 10 can be used as a normal camera.

When no voltage is applied to the microlens array 20, no lens 25 is formed in the opening 24. The light emitted from the object and collected by the main lens 30 passes through the microlens array 20 as is and is collected on the optical sensor 11.

The focal point F of the main lens 30 is located at the incident surface of the light sensor 11. If the position of the microlens array 20 is fixed, the position of at least one of the main lens 30 and the light sensor 11 is adjusted so that the light beam collected by the main lens 30 is focused at the incident surface of the light sensor 11. In this arrangement, two-dimensional light intensity information is captured by the light sensor 11.

By controlling the on/off of the voltage applied to the microlens array 20, it is possible to switch between the plenoptic camera function and the normal camera function.

(Experimental Example)
Fig. 6 shows a setup for stereoscopic photography using the imaging device 10 of the embodiment. As shown in Fig. 6A, an object 50 is photographed by the imaging device 10. Fig. 6B shows the object 50 actually used in the photography.

When a voltage is applied to the microlens array 20, multiple lenses 25 are formed. For ease of illustration, the first electrode 21, the second electrode 23, the dielectric polymer 22 sandwiched between the electrodes, and the openings 24 are omitted, and only the formed lenses 25 are shown.

The distance b from the microlens array 20 to the incident surface of the optical sensor 11 is adjusted between 0.1 and 0.5 mm. The distance L from the microlens array 20 to the main lens 30 is adjusted between 1 mm and 10 mm.
The specifications of the embodiment are as follows.

(1) Imaging system specifications
(1-1) Optical sensor 11 (image sensor)
・Number of pixels: 35 million pixels ・Pixel size: 4.8 μm
(1-2) Main lens 35
・Size: 60mmΦ
・Focal length: 80mm
(2) Active lens specifications
(2-1) Second electrode 23 (anode)
-Material: Stainless steel -Opening diameter/pitch: 50μm/60μm
・Arrangement: Grid pattern
(2-2) Dielectric polymer 22
Composition: PVC with dibutyl adipate plasticizer (addition amount 83 wt%)
Thickness: 0.2 mm
(2-3) First electrode 21 (cathode)
-Material: ITO film -Thickness: 0.1mm
・Surface resistance: 30Ω/sq. or less ・Light transmittance (wavelength 550nm): 85% or more
(2-4) Lens characteristics Applied voltage: 600V
・Focal length: 0.4mm
F value: 8.0

Figure 7 shows images captured depending on whether the voltage applied to the microlens array 20 is on or off. In Figure 7 (A), when the voltage applied to the microlens array 20 is off, the light collected by the main lens 30 passes through the microlens array 20 without being focused by the microlens array 20. No image is formed on each of the lenses 25 that make up the microlens array 20. The light that passes through the microlens array 20 forms a two-dimensional image on the optical sensor 11.

In FIG. 7B, when the voltage applied to the microlens array 20 is 400 V, a light image from the object 50 is formed on each of the lenses 25 that make up the microlens array 20.

Image 7-1 is an image obtained by combining the outputs from multiple sensor elements 111 covered by four lenses in area 7a of the microlens array 20.

Image 7-2 is an image obtained by combining the outputs from multiple sensor elements 111 covered by four lenses in area 7b of the microlens array 20.

Image 7-3 is an image obtained by combining the outputs from multiple sensor elements 111 covered by four lenses in area 7c of the microlens array 20.

By changing the positions of areas 7a, 7b, and 7c in the microlens array 20, the viewpoint of the resulting image shifts from bottom to top. Similarly, in the areas of the microlens array 20 that are aligned from right to left on the page, the viewpoint of the resulting image shifts from left to right. By combining the outputs from all the sensor elements 111 through arithmetic processing, a three-dimensional image with depth is obtained.

(Application to information processing devices)
The imaging device 10 of the embodiment is applicable to information processing devices such as smartphones and tablet terminals.

Figure 8 is a block diagram of an information processing device 100 incorporating an image capture device 10 according to an embodiment. The information processing device 100 includes the image capture device 10, an integrated circuit board 51 such as a SoC (System-on-Chip), a display device 52, a recording device 53, an input/output device 54, a power supply 55, and a wireless communication device 56. These devices are interconnected by a signal bus.

As described above, the imaging device 10 can switch between a plenoptic camera and a normal camera by controlling the voltage, switch between angular resolution and spatial resolution modes with the plenoptic camera, and adjust the focal length in each mode.

The integrated circuit board 51 is equipped with an image processor such as a GPU (Graphical Processing Unit) 511 and a control processor such as a CPU (Central Processing Unit) 512. The CPU 512 controls the on/off of the voltage applied to the microlens array 20 of the imaging device 10 and adjusts the applied voltage level. The GPU 511 performs digital arithmetic processing on the electrical signal output from the optical sensor 11 of the imaging device 10 to generate image data. When the imaging device 10 is in a plenoptic camera mode, the digital arithmetic processing by the GPU 511 may be performed based on a three-dimensional image measurement method. The digital arithmetic processing may include correction of the image data, noise removal, etc.

The recording device 53 stores information acquired or generated by the information processing device 100, commands, parameters, programs, etc. required for the operation of the information processing device 100. The recording device 53 includes both a memory built into the information processing device 100 and an external memory that is detachably connected.

The input/output device 54 includes a user interface such as a touch panel, microphone, and speaker.

The power supply 55 supplies the power necessary for the operation of each part of the information processing device 100. The wireless communication device 56 performs short-range wireless communication such as infrared communication, in addition to communication with a wireless base station via an antenna.

The display device 52 displays image data generated by the GPU 511, image data stored in the recording device 53, image data corresponding to commands input by the input/output device 54, etc.

The imaging device 10 of the embodiment uses a thin, three-layer microlens array 20, and the focal length of the microlens array 20 can be changed simply by controlling the voltage application. Therefore, it is suitable for application to small information processing terminals and mobile terminals.

Figure 9 is a flowchart of an imaging method according to an embodiment. When the camera is turned on (S11), it is determined whether the camera is in normal camera mode or plenoptic camera mode (S12). If the camera is in normal camera mode (YES in S12), the voltage applied to the microlens array (abbreviated as "MLA" in the figure) is turned off (S7). At least one of the main lens 30 and the optical sensor 11 may be moved along the optical axis so that light passing through the main lens 30 is focused on the incident surface of the optical sensor 11.

If the normal camera mode is not selected (NO in S12), it is determined whether a mode with good spatial resolution has been selected (S13). If a mode with good spatial resolution has been selected (YES in S13), a voltage V1 is applied to the microlens array (S15). The position of at least one of the main lens 30 and the optical sensor 11 may be adjusted along the optical axis so that the image plane of the main lens 30 is on the incident side of the microlens array and the light focused by the microlens array 20 is imaged on the incident surface of the optical sensor 11.

If a mode with good spatial resolution has not been selected (NO in S13), it is determined whether a mode with good angular resolution has been selected (S14). If a mode with good angular resolution has been selected (YES in S14), a voltage V2 smaller than V1 is applied to the microlens array to adjust the focal length (S16). At least one of the main lens 30 and the optical sensor 11 may be moved along the optical axis so that the light beam transmitted through each point of the main lens 30 is split into multiple light beams by each lens 25 of the microlens array 20 and enters the sensor element 111 of the optical sensor 11.

The control flow in Figure 9 makes it possible to use a single microlens array 20 to switch between a normal camera and a plenoptic camera, and to select a mode with good spatial resolution and a mode with good angular resolution for the plenoptic camera. In addition, regardless of whether a normal camera or a plenoptic camera is used, the focal length of the microlens array 20 can be adjusted simply by controlling the applied voltage.

10 Imaging device 11 Optical sensor 20 Microlens array 21 First electrode 22 Dielectric polymer 23 Second electrode 24 Opening 25 Lens 30 Main lens 100 Information processing device 51 Integrated circuit board 511 GPU
512 CPU
IS Image Sensor

Claims (9)

An imaging device having a light sensor having a plurality of sensor elements, a main lens that guides light from an object to the light sensor, and a microlens array disposed between the light sensor and the main lens,
The microlens array has a three-layer structure including a first electrode that is transparent to visible light, a second electrode that has a plurality of openings and is transparent to visible light and faces the first electrode, and a dielectric polymer that is disposed between the first electrode and the second electrode. Lenses are formed in the openings by application of a voltage.
the imaging device selectively operates in a first mode having good angular resolution and a second mode having good spatial resolution according to a level of the applied voltage;
A focal length of the microlens array in the second mode is set to be shorter than a focal length of the microlens array in the first mode in response to a level of the voltage;
an image plane of the main lens in the second mode is located between the main lens and the microlens array, and an image plane of the main lens in the first mode is located at the microlens array.
Imaging device. In the first mode, a light beam incident on one lens of the microlens array is split to at least two sensor elements of the plurality of sensor elements;
2. The imaging device according to claim 1, wherein in the second mode, the light beam incident on the one lens is irradiated onto one of the plurality of sensor elements. 3. The imaging device according to claim 1, wherein the imaging device functions as a normal camera by turning off the voltage applied to the microlens array and using the image plane as the plurality of sensor elements . 4. The imaging device according to claim 1, wherein the focal length of the microlens array is made variable by changing the level of the voltage applied to the microlens array in the first mode and/or the second mode. 1. An imaging method for an imaging device including an optical sensor having a plurality of sensor elements, a main lens that guides light from an object to the optical sensor, and a microlens array disposed between the optical sensor and the main lens, comprising:
The microlens array is formed of a first electrode transparent to visible light, a second electrode having a plurality of openings and facing the first electrode, and a dielectric polymer disposed between the first electrode and the second electrode;
applying a voltage to the microlens array to form lenses in the apertures;
selectively operating the imaging device between a first mode having good angular resolution and a second mode having good spatial resolution according to a level of the voltage applied to the microlens array ;
A focal length of the microlens array in the second mode is set to be shorter than a focal length of the microlens array in the first mode in response to a level of the voltage;
an image plane of the main lens in the second mode is located between the main lens and the microlens array, and an image plane of the main lens in the first mode is located at the microlens array.
Imaging method. In the first mode, a light beam incident on one lens of the microlens array is split to at least two sensor elements of the plurality of sensor elements;
6. The imaging method according to claim 5, wherein in the second mode, the light beam incident on the one lens is irradiated onto one of the plurality of sensor elements. 7. The imaging method according to claim 5, wherein the image pickup device functions as a normal camera by turning off the voltage applied to the microlens array and using the image plane as the plurality of sensor elements . The imaging method according to any one of claims 5 to 7, characterized in that the focal length of the microlens array is made variable by changing the level of the voltage applied to the microlens array in the first mode and/or the second mode. applying a second voltage to the microlens array to form the lenses in the apertures when the second mode is selected;
directing light from an image plane formed by the main lens to a plurality of lenses of the microlens array;
The light incident on each of the plurality of lenses is condensed onto N sensor elements.
The imaging method according to any one of claims 5 to 8 .

Images