JP7838577B2 - Imaging device and method of operating the imaging device - Google Patents

Description

This disclosure relates to an imaging device and a method of operating an imaging device, and more particularly to an imaging device and a method of operating an imaging device that can appropriately limit the information that may be included when capturing an image.

A technique has been proposed to limit the information captured as an image by placing a cover or filter in front of the imaging surface of an image sensor or similar device, thereby preventing the capture of a portion of the scene (see Patent Document 1). Furthermore, a technique has been disclosed to similarly limit the captured information by using a polarizing filter (see Patent Document 2).

U.S. Patent Application Publication No. 2013/0050549 U.S. Patent Application Publication No. 2018/0348536

Incidentally, a first example of an application implemented using a filter like the one described in Patent Document 1 is one that, for example, physically blocks light from the entire scene or a part of the scene to remove specific information, thereby achieving, for example, privacy protection.

However, this application may not function properly with scenes that have movement.

In other words, for this application to function in scenes with movement, it is necessary to incorporate processes that internally block incident light, or change the entire scene or a portion of the scene.

However, in this case, in order to identify the area where privacy should be protected, it is necessary to first image the entire scene and then identify the area that needs to be protected from the entire imaged scene, which makes it impossible to guarantee strict privacy protection.

Another example involves using a polarizing filter that blocks information from a display emitting polarizing components, thereby physically shielding the entire scene or a portion of the scene from light, and removing specific information to protect privacy.

However, in this second example, the filter blocks the unpolarized component and transmits the polarized component, making it the opposite of the filter in the first example.

Furthermore, in prior art 2, since the unpolarized component contains all the polarized components, the polarizing filter does not function as a filter for some of the unpolarized components.

This disclosure is made in view of the above circumstances, and in particular, aims to realize an imaging device that can appropriately limit the information included in the resulting image when imaging using an image sensor or the like.

An imaging device according to one aspect of this disclosure comprises a mask having a plurality of transmission regions formed in a predetermined pattern, which transmit incident light for each of the plurality of components, and which transmits the incident light with modulation applied to each of the plurality of components; an image sensor that captures a transmitted image consisting of the light transmitted through the mask; and a reconstruction unit that reconstructs an image corresponding to the incident light based on the transmitted image.

One aspect of the present disclosure is a method for operating an imaging device, comprising a mask having a plurality of transmission regions formed in a predetermined pattern that transmit incident light component by component, an image sensor, and a reconstruction unit, wherein the method includes the steps of: the mask modulating each component of the incident light and transmitting it as modulated light; the image sensor capturing a transmitted image consisting of the light transmitted through the mask; and the reconstruction unit reconstructing an image corresponding to the incident light based on the transmitted image.

In one aspect of this disclosure, a mask is formed in a predetermined pattern in which multiple transmission regions are used to transmit incident light for each of the multiple components, thereby modulating each of the multiple components of the incident light and transmitting it. An image sensor captures a transmitted image consisting of the light that has passed through the mask, and an image corresponding to the incident light is reconstructed based on the transmitted image.

This is a diagram illustrating the basics of a lensless camera. This is a diagram explaining the imaging principle of a lensless camera. This is a flowchart explaining the image processing of a lensless camera. This figure illustrates an example configuration of a first embodiment of the imaging device of the present disclosure. This diagram illustrates the structure of the mask shown in Figure 4. This figure illustrates the imaging principle of the imaging device shown in Figure 4. This figure illustrates the final image generated by the imaging device shown in Figure 4. Figure 4 is a flowchart illustrating the imaging process of the imaging device. This figure illustrates an example configuration of a second embodiment of the imaging device of the present disclosure. This figure illustrates the imaging principle of the imaging device shown in Figure 9. This figure illustrates the final image generated by the imaging device shown in Figure 9. Figure 9 is a flowchart illustrating the imaging process of the imaging device. This figure illustrates the configuration of a mask that is an application example of a second embodiment of the imaging device of the present disclosure. This figure illustrates an example configuration of a third embodiment of the imaging device of the present disclosure. Figure 14 is a diagram illustrating the imaging principle of the imaging device. This figure illustrates the final image generated by the imaging device shown in Figure 14. Figure 14 is a flowchart illustrating the imaging process of the imaging device. This figure illustrates the configuration of a mask that is an application example of a third embodiment of the imaging device of the present disclosure. This diagram illustrates an example configuration of a general-purpose personal computer.

Preferred embodiments of this disclosure will be described in detail below with reference to the attached drawings. In this specification and the drawings, components having substantially the same functional configuration are denoted by the same reference numerals, thus omitting redundant descriptions.

The following describes the configurations for implementing this technology. The explanation will proceed in the following order.
1. Overview of the lensless imaging device 2. First embodiment 3. Second embodiment 4. Application example of the second embodiment 5. Third embodiment 6. Application example of the third embodiment 7. Example of execution by software

<<1. Overview of Lensless Imaging Devices>>
The overview of the lensless imaging device will be described with reference to Figure 1. Figure 1 is a side cross-sectional view of the lensless imaging device 11.

The imaging device 11 in Figure 1 is a so-called lensless camera and includes a mask 31, an image sensor 32, a reconstruction unit 33, and an output unit 34.

The mask 31 is a plate-shaped structure made of light-shielding material and is provided in front of the image sensor 32. For example, as shown in the left part of Figure 2, it consists of a transparent region 41 with a hole-shaped opening that transmits incident light, and a light-shielding region 42 that is otherwise shielded from light.

When the mask 31 receives light from the subject surface G1 (actually the surface from which radiation is emitted from a three-dimensional subject) indicated by the optical axis AX, it transmits the incident light through the transmission region 41, thereby modulating the incident light from the subject surface G1 as a whole, converting it into modulated light, and the converted modulated light is received by the image sensor 32 to capture an image.

The image sensor 32 consists of a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor. It captures modulated light, which is the incident light from the subject surface G1 modulated by the mask 31, and outputs it to the reconstruction unit 33 as an image G2 consisting of pixel-level signals.

Furthermore, the mask 31 is large enough to encompass at least the entire surface of the image sensor 32, and is configured so that only modulated light, which has been modulated by passing through the mask 31, is received by the image sensor 32.

Furthermore, the transparent region 41 formed in the mask 31 is at least larger than the pixel size of the image sensor 32. Also, a small gap of distance d is provided between the image sensor 32 and the mask 31.

For example, as shown in the upper left of Figure 2, incident light from point light sources PA, PB, and PC on the subject surface G1 is transmitted through the transmission region 41 of the mask 31 and received as light rays with light intensities a, b, and c at positions Pa, Pb, and Pc on the image sensor 32, respectively.

As shown in the upper left of Figure 2, the detection sensitivity of each pixel has directivity according to the angle of incidence because the incident light is modulated by the transmission region 41 set in the mask 31. Here, giving each pixel detection sensitivity directivity according to the angle of incidence means that the light-receiving sensitivity characteristics differ depending on the region on the image sensor 32, according to the angle of incidence of the incident light.

In other words, assuming that the light source constituting the subject surface G1 is a point light source, light rays of the same intensity emitted from the same point light source will be incident on the image sensor 32. However, due to modulation by the mask 31, the incident angle changes for each region on the imaging surface of the image sensor 32. As the incident angle of the incident light changes according to the region on the image sensor 32 due to the mask 31, it has a light-receiving sensitivity characteristic, i.e., incident angle directivity. Therefore, even if the light rays have the same intensity, they will be detected with different sensitivities for each region on the image sensor 32 by the mask 31 provided in front of the imaging surface of the image sensor 32, and detection signals with different detection signal levels will be detected for each region.

More specifically, as shown in the upper right of Figure 2, the detection signal levels DA, DB, and DC of pixels at positions Pa, Pb, and Pc on the image sensor 32 are expressed by the following equations (1) to (3), respectively. Note that the vertical relationship between equations (1) to (3) in Figure 2 and the positions Pa, Pb, and Pc on the image sensor 32 in Figure 2 is inverted.

DA=α1×a+β1×b+γ1×c
... (1)
DB=α2×a+β2×b+γ2×c
... (2)
DC=α3×a+β3×b+γ3×c
... (3)

Here, α1 is a coefficient for the detection signal level a, which is set according to the angle of incidence of the light rays from the point light source PA on the subject surface G1 to be restored at position Pa on the image sensor 32.

Furthermore, β1 is a coefficient for the detection signal level b, which is set according to the angle of incidence of light rays from the point light source PB on the subject surface G1 to be restored at position Pa on the image sensor 32.

Furthermore, γ1 is a coefficient for the detection signal level c, which is set according to the angle of incidence of the light rays from the point light source PC on the subject surface G1 to be restored at position Pa on the image sensor 32.

Therefore, (α1 × a) in the detection signal level DA represents the detection signal level caused by the light ray from the point light source PA at position Pa.

Furthermore, (β1 × b) of the detection signal level DA represents the detection signal level caused by the light ray from the point light source PB at position Pa.

Furthermore, (γ1 × c) of the detection signal level DA represents the detection signal level caused by the light ray from the point light source PC at position Pa.

Therefore, the detected signal level DA is expressed as a composite value obtained by multiplying each component of the point light source PA, PB, and PC at position Pa by their respective coefficients α1, β1, and γ1. Hereafter, the coefficients α1, β1, and γ1 will be collectively referred to as the coefficient set.

Similarly, for the detection signal level DB at the point light source Pb, the coefficient sets α2, β2, and γ2 correspond to the coefficient sets α1, β1, and γ1, respectively, for the detection signal level DA at the point light source PA. Furthermore, for the detection signal level DC at the point light source Pc, the coefficient sets α3, β3, and γ3 correspond to the coefficient sets α1, β1, and γ1, respectively, for the detection signal level DA at the point light source Pa.

However, the detection signal levels of the pixels at positions Pa, Pb, and Pc are values expressed by the sum of the products of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, respectively, and a coefficient. Therefore, these detection signal levels are a mixture of the light intensities a, b, and c of the light rays emitted from the point light sources PA, PB, and PC, respectively, and are different from the image formed from the image of the subject. The image consisting of the detection signal levels DA, DB, and DC of the pixels at positions Pa, Pb, and Pc corresponds to image G2 in Figure 1.

Specifically, a system of equations is constructed using the coefficient sets α1, β1, γ1, α2, β2, γ2, α3, β3, γ3, and detection signal levels DA, DB, DC. By solving for light intensities a, b, and c, the pixel values at each position Pa, Pb, and Pc are obtained, as shown in the lower right of Figure 2. This reconstructs and restores the reconstructed image (final image), which is a set of pixel values. This reconstructed image corresponds to image G3 in Figure 1.

Furthermore, when the distance between the image sensor 32 and the subject surface G1, as shown in the upper left of Figure 2, changes, the coefficient sets α1, β1, γ1, α2, β2, γ2, and α3, β3, γ3 will change accordingly. By changing these coefficient sets, it is possible to reconstruct restored images (final images) of subject surfaces at various distances.

Therefore, by changing the coefficient set to correspond to various distances with a single image acquisition, it is possible to reconstruct images of the subject plane at various distances from the acquisition position.

As a result, in imaging using the imaging device 11 in Figure 1, there is no need to be aware of the phenomenon known as "out-of-focus," which occurs when imaging with an imaging device using a lens, as long as the subject to be imaged is included within the field of view. By changing the coefficient set according to the distance, images of subject planes at various distances can be reconstructed after imaging.

Furthermore, the detection signal level shown in the upper right of Figure 3 is not the detection signal level corresponding to the image in which the subject's image is formed; therefore, it is merely an observed value, not a pixel value, and the image consisting of these observed values corresponds to image G2. Also, the detection signal level shown in the lower right of Figure 3 is the signal value for each pixel corresponding to the image in which the subject's image is formed; that is, it is the value of each pixel in the reconstructed image (final image) reconstructed based on image G2, and therefore it is a pixel value. In other words, this reconstructed image (final image) of the subject plane G1 corresponds to image G3.

This configuration allows the imaging device 11 to function as a so-called lensless camera. As a result, since an imaging lens is not an essential component, it becomes possible to reduce the height of the imaging device, that is, to reduce the thickness in the direction of light incidence in the configuration that realizes the imaging function. Furthermore, by changing the coefficient set in various ways, it becomes possible to reconstruct and restore the final image (reconstructed image) at various distances on the subject plane.

Hereafter, the image corresponding to the image G2 captured by the image sensor 32 before reconstruction will simply be referred to as the "captured image," and the image corresponding to the image G3 that is reconstructed and restored by signal processing of the captured image will be referred to as the "final image (reconstructed image)." Therefore, from a single captured image, by varying the coefficient set described above, images on the subject plane G1 at various distances can be reconstructed as the final image.

The reconstruction unit 33 is equipped with the coefficient set described above, and uses the coefficient set corresponding to the distance from the imaging position of the imaging device 11 to the subject surface G1 to reconstruct the final image (reconstructed image) (image G3 in Figure 1) based on the image captured by the image sensor 32 (image G2 in Figure 1), and outputs it to the output unit 34.

The output unit 34 applies signal processing to the final image supplied by the reconstruction unit 33 and outputs it as an image signal.

<Imaging process using the imaging device shown in Figure 1>
Next, the imaging process performed by the imaging device 11 in Figure 1 will be explained with reference to the flowchart in Figure 3.

In step S11, the mask 31 modulates the light from the subject surface G1 and causes it to be incident on the image sensor 32.

In step S12, the image sensor 32 captures an image consisting of light from the subject surface G1 that has been modulated by the mask 31, and outputs the captured image (corresponding to image G2) to the reconstruction unit 33.

In step S13, the reconstruction unit 33 reconstructs the image based on the captured image (corresponding to image G2), which is an image consisting of modulated light output from the image sensor 32, using a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device 11 to the subject surface G1, and outputs the final image (reconstructed image) (corresponding to image G3) to the output unit 34. In other words, the final image (reconstructed image) is obtained by constructing and solving a system of equations using the coefficient sets described above with reference to equations (1) to (3) for the captured image.

In step S14, the output unit 34 performs signal processing and outputs it as an image signal.

In other words, by performing the above series of processes, modulation is applied using a mask without using a lens, and then the final image (reconstructed image) is reconstructed using a coefficient set. This achieves a low profile and makes it possible to reconstruct images at various distances in a single imaging.

The imaging device of this disclosure applies the principle of the lensless imaging device described above and modulates and reconstructs specific light components by using filters that transmit at least two or more specific light components as a transmission region 41 and a light-shielding region 42 for each light component.

This enables the realization of an imaging device that can appropriately limit the information included during image acquisition by reconstructing a final image consisting of specific light components.

<<2. First Embodiment>>
<Example of configuration of the imaging device in this disclosure>
Next, with reference to Figure 4, an example configuration of the first embodiment of the imaging device of this disclosure will be described.

The imaging device 101 in Figure 4 reconstructs only the image consisting of polarized components and does not reconstruct the image consisting of unpolarized components, thereby limiting the information contained in the image consisting of unpolarized components by making the final image an image consisting only of polarized components.

More specifically, the imaging device 101 consists of a mask 111, an image sensor 112, a reconstruction unit 113, and an output unit 114. The mask 111, image sensor 112, reconstruction unit 113, and output unit 114 correspond to the mask 31, image sensor 32, reconstruction unit 33, and output unit 34 in Figure 1, respectively.

The mask 111 has a configuration corresponding to the mask 31 in Figure 1, and applies a predetermined modulation to the incident light before it is incident on the image sensor 112.

As described above, the mask 31 is a plate-shaped structure made of a light-shielding material, and is configured to modulate incident light by forming a transmission region 41 and a light-shielding region 42.

In contrast, the mask 111 has a configuration in which a vertical polarizing filter 111v that transmits the vertical polarization component of the light incident from the subject surface 121 on the left side of the figure, and a horizontal polarizing filter 111h that transmits the horizontal polarization component, are arranged in a pseudo-random pattern.

The arrangement pattern of the vertical polarizing filter 111v and the horizontal polarizing filter 111h may be a pattern other than a pseudo-random pattern, for example, a random pattern, a URA (Uniformly Redundant Arrays) pattern, or a MURA (Modified Uniformly Redundant Arrays) pattern. The light transmittance of the vertical polarizing filter 111v and the horizontal polarizing filter 111h are equal.

Furthermore, in Figure 4, the area on the mask 111 where the vertical polarizing filter 111v is placed shows a vertical striped pattern, and the area where the horizontal polarizing filter 111h is placed shows a horizontal striped pattern.

Here, we consider an example where the subject surface 121 contains a smartphone 131 emitting a vertical polarization component Lv consisting of the polarization component of light in the vertical direction, a person 132 emitting a non-polarizing component Lu consisting of unpolarized light, and a display 133 emitting a horizontal polarization component Lh consisting of the polarization component of light in the horizontal direction.

In this case, the vertical polarization component Lv emitted by the smartphone 131 is modulated by passing through, for example, the region of the vertical polarization filter 111v of the mask 111, shown as the white region in the center of Figure 5, and then incident on the image sensor 112. The image sensor 112 captures an image consisting of the modulated vertical polarization component Lv. The reconstruction unit 113 reconstructs an image of the smartphone 131 consisting of the vertical polarization component Lv based on the image consisting of the modulated vertical polarization component Lv.

Furthermore, the unpolarized component Lu emitted by person 132 passes through both the horizontal polarizing filter 111h and the vertical polarizing filter 111v, as shown in the left part of Figure 5. Therefore, light transmitted from the entire surface of the mask 111 is incident on the image sensor 112, and the image of person 132 cannot be reconstructed by the image reconstruction process described above. In other words, because the unpolarized component Lu passes through both the horizontal polarizing filter 111h and the vertical polarizing filter 111v, it is as if it were passing through a transparent space where the mask 111 does not exist, and is incident on the image sensor 112 in an unmodulated state and is captured. To put it another way, when the unpolarized component Lu passes through the mask 111, there is only a region corresponding to the transmission region 41 on the mask 111, and there is no region corresponding to the light-shielding region 42, so the unpolarized component Lu is incident on the image sensor 112 in an unmodulated state and is captured. Therefore, the reconstruction unit 113 cannot reconstruct an image consisting of the unpolarized component Lu, which is the unmodulated incident light, and thus cannot reconstruct the person 132.

Furthermore, the horizontal polarization component Lh emitted by the display 133 is modulated by passing through the region of the horizontal polarization filter 111h of the mask 111, shown as the white region in the right side of Figure 5, and then incident on the image sensor 112. The image sensor 112 captures an image consisting of the modulated horizontal polarization component Lh. The reconstruction unit 113 reconstructs the image of the display 133 consisting of the horizontal polarization component Lh based on the image consisting of the modulated horizontal polarization component Lh.

Furthermore, the white and black areas in the central and right parts of Figure 5 correspond to each other, indicating that each region of the mask 111 is at least one of the regions of the horizontal polarizing filter 111h and the vertical polarizing filter 111v.

In other words, as shown in Figure 6, the mask 111 is transmitted from the subject surface 121, which includes the images of the smartphone 131, the person 132, and the display 133, in a combined form of an image 121u consisting of a non-polarized component Lu, an image 121v consisting of a vertically polarized component Lv, and an image 121h consisting of a horizontally polarized component Lh.

Here, the image 121v in Figure 6, which consists only of the vertical polarization component Lv, is observed as only the balloon displayed on the smartphone 131 that emits the vertical polarization component Lv, with everything else appearing as a black image.

Furthermore, the image 121h in Figure 6, which consists only of the horizontal polarization component Lh, is observed as showing only the car, which is a display within the display 133 that emits the horizontal polarization component Lh, while everything else appears as a black image.

Furthermore, the image 121u, consisting only of the non-polarized component Lu in Figure 6, is observed as an image in which the display image in the smartphone 131 in image 121v and the display image in the display 133 in image 121h are black, when viewed directly from the subject surface 121.

As a result, when a person directly views the subject surface 121, the subject surface 121 itself is viewed as image 121u in Figure 6. However, in the mask 111, the vertical polarization component Lv and the horizontal polarization component Lh are modulated, while the unpolarized component Lu is not modulated, and these are incident on the image sensor 112. As a result, the image 121v consisting of the vertical polarization component Lv and the image 121h consisting of the horizontal polarization component Lh are reconstructed by the reconstruction unit 113 based on the image captured by the image sensor 112. In contrast, the image 121u consisting of the unmodulated unpolarized component Lu is not reconstructed by the reconstruction unit 113.

To summarize, the following can be said: As shown in Figure 6, light from the subject surface 121 passes through the mask 111 and is modulated, so that the image sensor 112 captures an image such as image 112i.

Image 112i is an image in which the images of the smartphone 131 on the subject surface 121, which consists of a modulated vertical polarization component Lv, the person 132, which consists of an unmodulated unpolarized component Lu, and the display 133, which consists of a modulated horizontal polarization component Lh, have not been imaged. As a result, the overall image is blurry, as shown in Figure 6.

The reconstruction unit 113 reconstructs images 121v' and 121h' corresponding to image 121v consisting of a modulated vertical polarization component Lv and image 121h consisting of a modulated horizontal polarization component Lh, based on the image 112i supplied from the image sensor 112, and outputs them to the output unit 114. At this time, the reconstruction unit 113 cannot reconstruct the unmodulated non-polarized component Lu, so the image corresponding to image 121u consisting of the unmodulated non-polarized component Lu cannot be reconstructed.

As explained with reference to Figure 5, the mask 111 is configured such that the horizontal polarizing filter 111h and the vertical polarizing filter 111v are arranged in a pseudo-random pattern.

Therefore, when the horizontal polarization component Lh passes through the mask 111, the horizontal polarization filter 111h functions as the transmission region 41 of the mask 31 in Figure 1, and the vertical polarization filter 111v functions as the light-shielding region 42 in Figure 1, thereby modulating the horizontal polarization component Lh.

On the other hand, when the vertical polarization component Lv passes through the mask 111, the vertical polarization filter 111v functions as the transmission region 41 of the mask 31 in Figure 1, and the horizontal polarization filter 111h functions as the light-shielding region 42 in Figure 1, thereby modulating the vertical polarization component Lv.

In other words, the image sensor 112 captures the results of observing the horizontal polarization component Lh and the vertical polarization component Lv as an image 112i, comparing the light with the modulation applied by the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111, with the unpolarized component light, and outputs it to the reconstruction unit 113.

As shown in Figure 7, the reconstruction unit 113 uses a set of coefficients corresponding to the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111 to reconstruct image 121v', which corresponds to image 121v consisting of the vertical polarization component Lv, and image 121h', which corresponds to image 121h consisting of the horizontal polarization component Lh, from image 112i. However, at this time, the reconstruction unit 113 cannot reconstruct image 121u, which consists of the unpolarized component Lu, because the processing is based on an image captured without modulation.

The output unit 114 combines the image 121v' consisting of the vertical polarization component Lv and the image 121h' consisting of the horizontal polarization component Lh supplied from the reconstruction unit 113 to generate the image 122 shown in the right part of Figure 4, and outputs it as the final image.

As a result, an image 122 is output consisting of an image 121v' made up of the vertical polarization component Lv displayed by the smartphone 131 and an image 121h' made up of the horizontal polarization component Lh displayed by the display 133. Therefore, images consisting only of unpolarized components, such as human faces, are not included.

As a result, it becomes possible to capture only the image displayed on the smartphone 131 or display 133, while restricting the capture of information related to the privacy of person 132, especially their face.

For example, when it is necessary to capture (re-capture) a QR code (registered trademark) displayed on a smartphone 131 or display 133, if the face of the person holding the smartphone 131 or display 133 is within the field of view, conventional imaging technology would capture the face of the person holding the smartphone 131 or display 133 along with the QR code, making it difficult to respect privacy.

Even in such cases, by using the imaging device 101 of this disclosure, even if the face of a person holding a smartphone 131 or display 133 displaying a QR code is within the imaging field of view, the person's face, consisting of non-polarized components, will not be captured. As a result, only the QR code will be captured, making it possible to achieve appropriate restrictions on imaging that take privacy into consideration.

<About reconstruction by the reconstruction unit>
Next, we will explain the specific calculations related to reconstruction performed by the reconstruction unit 113.

The basic reconstruction calculations are the same as those performed using the coefficient set in the lensless camera shown in Figure 1.

Here, we consider an example where the subject surface 121 is a rectangular image with M pixels × N pixels, and the light from each pixel as a light source consists of a vertically polarized component, a horizontally polarized component, and an unpolarized component.

Furthermore, the image sensor 112 is composed of P pixels × Q pixels, and in the mask 111, the vertical polarizing filter 111v and the horizontal polarizing filter 111h are formed in a pseudo-random pattern so as to be distributed in a (M+P-1) × (N+Q-1) grid, and are substantially the same in shape and size as the pixels of the image sensor 112. Light from the subject surface 121 passes through either the vertical polarizing filter 111v or the horizontal polarizing filter 111h formed in the mask 111, is modulated, and then incident on the image sensor 112.

Based on this relationship, by using the coefficient set described above, a system of linear equations like the one shown in equation (4) below can be constructed. Note that equation (4) is the system of linear equations expressed in terms of a determinant.

Here, p _i is the pixel value of the image sensor 112, and a _ij is a coefficient set according to the design of the mask 111, where 0 represents the horizontal polarizing filter 111h and 1 represents the vertical polarizing filter 111v. _s ^i∥ is the light intensity of the incident light at the position where the vertical polarizing filter 111v is provided, ^{and s i⊥} _represents the light intensity of the incident light at the position where the horizontal polarizing filter 111h is provided.

The image will be reconstructed by solving the system of equations represented by equation (4). To solve the system of equations in equation (4), we transform it as shown in equation (5) below.

Here, A is a matrix represented by the following equation (6).

Furthermore, s _i is expressed by the following equation (7).

Furthermore, s _i can be estimated by solving equation (5) above. There are various methods for solving equation (5), but for example, it can be solved using a pseudo-inverse matrix obtained by normalization.

In particular, when the rank of matrix A is MN+1, equation (5) can be expressed as an exact solution as shown in equation (8) below.

However, in reality, the pixel values captured by the image sensor 112 will contain observation errors, so the regularized solution shown in equation (9) below can be considered a more appropriate solution.

Here, λ ² B ^T B are normalization parameters.

The value of s _i , expressed by equation (8) or equation (9), can be determined by giving an approximate value to the difference ( _{s i ||} ^- _{s i} ^⊥ ) between the light intensity of the incident light at the position where the _vertical polarizing filter ^111v is installed and the light intensity of the incident light at the position where the horizontal polarizing filter ^111h is installed.

In other words, based on the assumption that the light from each pixel constituting the subject surface 121 is either vertically polarized, horizontally polarized, or unpolarized, the result s _i can be divided into vertically polarized, horizontally polarized, and unpolarized components by using the following equation (10).

Here, δ is a threshold parameter set to reduce artifacts that occur in the reconstructed image. For example, in the ideal case where there is no noise, δ = 0.

Furthermore, since the vertical polarization filter 111v and the horizontal polarization filter 111h of the mask 111 act similarly on the unpolarized component (the component with 50% transmittance), the image information of the unpolarized component is recorded by the image sensor 112 as an offset for estimating the intensity of all unpolarized components.

Therefore, information from images consisting of unpolarized components cannot be reconstructed. Furthermore, not only can pixels at locations consisting of unpolarized components not be reconstructed, but unpolarized components at locations of vertically polarized and horizontally polarized components where the degree of linear polarization (DoLP) is less than 1 (DoLP < 1) also cannot be reconstructed.

Furthermore, equations (8) and (9), which are solutions as described above, represent the difference between the vertical and horizontal polarization components at each pixel position constituting the subject surface 121. When the intensity in the vertical polarization direction at pixel position s _i is I _i , the polarization angle with respect to the vertical polarization direction is φ _i , and the linear polarization degree (DoLP) is ρ _i (0 ≤ ρ _i ≤ 1), then, based on Malus's law, the following equation (11) holds.

Therefore, the imaging device 101 in Figure 4 can be used to reconstruct pixels at any position on the subject surface 121 that have a polarization component, and it is possible to determine which pixels are polarized and whether the polarization direction is close to vertical or close to horizontal (unless their polarization angle is _φi = π/4 or _φi = 3π/4).

Furthermore, in equation (11), the polarization angle _φi cannot be determined unless the product of the intensity _Ii in the perpendicular polarization direction and the linear polarization degree (DoLP) _ρi is known. However, if the polarization angle _φi is restricted to two angles _φa and _φb ( _φi ∈{ _φa , _φb }) such that _{cos2φa cos2φb} < 0, then the exact solution between the polarization angle and light intensity at the pixel position _si on the subject surface 121 can be determined based on the sign and value of _si obtained in equation (7).

<Image processing by the imaging device shown in Figure 4>
Next, the imaging process performed by the imaging device 101 shown in Figure 4 will be explained with reference to the flowchart in Figure 8.

In step S31, the horizontal polarization filter 111h and vertical polarization filter 111v of the mask 111 modulate the horizontal and vertical polarization components of the light from the subject surface 121, respectively, and direct them onto the image sensor 112. The unpolarized component is not modulated by the mask 111 and therefore passes through the mask 111 and is incident onto the image sensor 112.

In step S32, the image sensor 112 captures the light from the subject surface 121 that has passed through the mask 111 and outputs it as an image to the reconstruction unit 113.

In step S33, the reconstruction unit 113, based on the captured image obtained by light transmitted through the mask 111 output from the image sensor 112, uses the above-described formula (4), which consists of a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device 101 to the subject surface 121, to reconstruct a final image (reconstructed image) consisting of horizontal polarization components and a final image (reconstructed image) consisting of vertical polarization components, and outputs them to the output unit 34.

In step S34, the output unit 114 combines the reconstructed final image consisting of the horizontal polarization component with the final image consisting of the vertical polarization component.

In step S35, the output unit 114 performs signal processing on the image obtained by combining the reconstructed image consisting of the horizontal polarization component and the reconstructed image consisting of the vertical polarization component, and outputs it as an image signal.

Through the above series of processes, modulation is applied using a mask without the use of a lens, and then the final image (reconstructed image) is reconstructed using a coefficient set. This achieves a low profile and makes it possible to reconstruct images at various distances in a single imaging scan.

Furthermore, the imaging device 101 of this disclosure applies the principle of the lensless imaging device described above and modulates and reconstructs the horizontally polarized and vertically polarized components using a mask 111 consisting of filters that transmit at least two specific light components. At this time, the unpolarized component is not modulated and therefore not reconstructed.

This allows for the reconstruction of the final image consisting of horizontal and vertical polarization components, thereby preventing the reconstruction of the image consisting of unpolarized components.

As a result, it becomes possible to capture only images consisting of polarized components while restricting the capture of images consisting of unpolarized components. For example, it becomes possible to appropriately restrict the capture of privacy-related information, such as facial images of people, which consist of unpolarized components, and capture only images composed solely of polarized light from smartphones, displays, etc.

<<3. Second Embodiment>>
In the above method, the incident light was modulated using a mask 111 consisting of two types of polarizing filters, a horizontal polarizing filter 111h and a vertical polarizing filter 111v, and an image was captured, and an image consisting of polarized components was reconstructed.

This results in an image where the unpolarized component is not reconstructed. Therefore, we have explained an example where it is possible to limit the information in the captured image by using the unpolarized component for the information that you want to restrict.

However, the types of polarizing filters used in the mask 111 are not limited to two types; more than two types of polarizing filters may be included.

For example, the mask may be formed from three types of polarizing filters, including a horizontal polarizing filter, a vertical polarizing filter, and a polarizing filter that transmits polarizing components in the diagonal direction (45-degree direction).

Figure 9 shows an example of an imaging device configuration in which a mask is formed from three types of polarizing filters, including a horizontal polarizing filter, a vertical polarizing filter, and a polarizing filter that transmits polarizing components in the diagonal direction (45-degree direction).

The imaging device 201 in Figure 9 consists of a mask 211, an image sensor 212, a reconstruction unit 213, and an output unit 214.

The mask 211, image sensor 212, reconstruction unit 213, and output unit 214 each have a configuration that basically corresponds to the functions of the mask 111, image sensor 112, reconstruction unit 113, and output unit 114 of the imaging device 101 in Figure 4.

However, while mask 111 consists of a horizontal polarizing filter 111h and a vertical polarizing filter 111v, mask 211 includes, in addition to the horizontal polarizing filter 211h and the vertical polarizing filter 211v, a filter that transmits polarization components in the diagonal direction (45-degree direction) (hereinafter referred to as the 45-degree polarizing filter 211s).

In other words, the mask 111 in Figure 4 was configured such that a vertical polarizing filter 111v, which transmits the vertical polarization component of the light incident from the subject surface 121 on the left side of the figure, and a horizontal polarizing filter 111h, which transmits the horizontal polarization component, were arranged in a pseudo-random pattern.

In contrast, the mask 211 in Figure 9 is configured such that, in addition to a vertical polarizing filter 211v that transmits the vertical polarizing component of the light incident from the subject surface 221 on the left side of the figure, and a horizontal polarizing filter 211h that transmits the horizontal polarizing component, a 45-degree polarizing filter 211s that transmits the diagonal (45-degree) polarizing component is also present, all arranged in a pseudo-random pattern. Here as well, the unpolarized component is transmitted through all three—the vertical polarizing filter 211v, the horizontal polarizing filter 211h, and the 45-degree polarizing filter 211s—so it is transmitted without modulation. For this reason, even if it is captured by the image sensor 212, the reconstruction unit 213 cannot reconstruct an image of the unpolarized component.

The arrangement patterns of the vertical polarizing filter 211v, the horizontal polarizing filter 211h, and the 45-degree polarizing filter 211s may be patterns other than a pseudo-random pattern, such as a random pattern, a URA (Uniformly Redundant Arrays) pattern, or a MURA (Modified Uniformly Redundant Arrays) pattern. Furthermore, the light transmittance of the vertical polarizing filter 211v, the horizontal polarizing filter 211h, and the 45-degree polarizing filter 211s are all equal.

Furthermore, in Figure 9, the area on the mask 211 where the vertical polarizing filter 211v is placed shows a vertical striped pattern, the area where the horizontal polarizing filter 211h is placed shows a horizontal striped pattern, and the area where the 45-degree polarizing filter 211s is placed shows a diagonal striped pattern.

Here, we consider the case where a subject surface 221 is imaged, as shown in Figure 10, on which there are people 231 and 232 emitting unpolarized light, and smartphones 233 to 235 emitting polarized light in a predetermined direction to display an image. In this example, the smartphones 233 to 235 are displaying an image when a QR code is displayed on their display surface.

In this case, the polarized light emitted by smartphones 233 to 235 is modulated by passing through the regions of the vertical polarization filter 211v, horizontal polarization filter 211h, and 45-degree polarization filter 211s of the mask 211, and then incident on the image sensor 212.

In Figure 10, the mask 211 shows the black area as the region of the vertical polarizing filter 211v, the white area as the region of the horizontal polarizing filter 211h, and the gray area as the region of the 45-degree polarizing filter 211s.

As a result, as shown in the lower left of Figure 10, the mask 211 is transmitted from the subject surface 221 in a combined state: an image 221u consisting of the intensity of all light components, both polarized and unpolarized; an image 221d consisting of the degree of linear polarization (DoLP); and an image 221a consisting of the polarization angle.

In image 221d, each pixel is assigned a pixel value corresponding to a value between 0 and 1 that corresponds to the degree of linear polarization (DoLP). In image 221a, each pixel is assigned a pixel value corresponding to a value between 0 and π (Rad) that corresponds to the polarization angle.

In other words, in image 221d, pixel values corresponding to the degree of linear polarization (DoLP) are set for the areas where the displays of smartphones 233 to 235 are located, while the pixel values for the other areas are set to 0, resulting in a black color.

Furthermore, in image 221a, pixel values corresponding to the polarization angle are set for the areas where the displays of smartphones 233 to 235 are located, while the pixel values for the other areas are set to 0, resulting in a black color.

As a result, when a person directly views the incident light from the subject surface 221, the subject surface 221 itself in Figure 10 is viewed as image 221u. However, in the mask 211, the polarized component is modulated by passing through the vertical polarizing filter 211v, horizontal polarizing filter 211h, and 45-degree polarizing filter 211s of the mask 211, while the unpolarized component remains unmodulated. Consequently, the light constituting the images corresponding to images 221d and 221a, which consist of polarized components, is incident on the image sensor in a modulated state, while the unpolarized component of image 221u is incident on the image sensor 212 in an unmodulated state.

As shown in Figure 10, light from the subject surface 221 passes through the mask 211, and modulated polarized components and unmodulated unpolarized components are incident on the image sensor 212, and an image like image 212i is captured on the image sensor 212.

Image 212i is an image consisting of a modulated polarization component and an unmodulated unpolarized component, where the images of the people 231, 232 and the smartphones 233 to 235 within the subject surface 221 are not yet formed. As a result, the overall image is blurry, as shown in Figure 10.

The reconstruction unit 213 reconstructs images 221d and 221a, which consist of modulated polarization components, based on the image 212i supplied from the image sensor 212, and outputs them to the output unit 224.

Therefore, the reconstruction unit 213 uses a set of coefficients corresponding to the vertical polarization filter 211v, horizontal polarization filter 211h, and 45-degree polarization filter 211s of the mask 211 to reconstruct image 221d', which consists of pixel values obtained by multiplying the linear polarization degree by the light intensity, and image 221a, which consists of polarization angles, respectively, from image 212i.

As shown in the right side of Figure 11, the output unit 214 synthesizes the images 221d' and 221a supplied from the reconstruction unit 213 to generate image 222, which is then output as the final image.

As a result, as shown in the right side of Figure 9, an image 222 consisting of modulated components displayed by smartphones 233 to 235 will be output, and therefore, images consisting only of unpolarized components such as human faces will not be included.

As a result, it becomes possible to capture only the images displayed on smartphones 233 to 235, while restricting the capture of information related to the privacy of individuals 231 and 232, particularly their faces.

<Regarding reconstruction by the reconstruction unit in Figure 9>
Next, we will explain a specific example of the reconstruction calculation performed by the reconstruction unit 213 in Figure 9.

The basic reconstruction calculations are the same as those performed using the coefficient set in the lensless camera shown in Figure 1.

Here, the subject surface 221 is assumed to be a rectangular image with M pixels × N pixels, and the light from each pixel as a light source is assumed to consist of a vertically polarized component, a horizontally polarized component, a 45-degree polarized component, and an unpolarized component.

The image sensor 212 is composed of P pixels × Q pixels, and in the mask 211, a vertical polarizing filter 211v, a horizontal polarizing filter 211h, and a 45-degree polarizing filter 211s are formed in a pseudo-random pattern so as to be distributed in a (M+P-1) × (N+Q-1) grid, and the shape, size, and transmittance of the pixels of the image sensor 212 are assumed to be substantially the same. Light from the subject surface 221 passes through one of the vertical polarizing filters 211v, horizontal polarizing filters 211h, and 45-degree polarizing filters 211s formed in the mask 211, is modulated, and then incident on the image sensor 212.

Based on this relationship, by using the coefficient set described above, a system of linear equations like the one shown in equation (12) below can be constructed. Note that equation (12) is the system of linear equations expressed in terms of a determinant.

Here, _A0 is a matrix represented by the following equation (13).

Here, p _i is the pixel value of the image sensor 212, a _ij and b _ij are coefficients set according to the design of the mask 211. Here, when a _ij is 1, it represents a vertical polarization filter 211v, when b _ij is 1, it represents a horizontal polarization filter 211h, and when both a _ij and b _ij are 0, it represents a 45-degree polarization filter 211s.

s _i ^|| represents the light intensity of the incident light at the position where the vertical polarizing filter 211v is installed, s _i ^⊥ represents the light intensity of the incident light at the position where the horizontal polarizing filter 211h is installed, and s _i ⁴⁵ represents the light intensity of the incident light at the position where the 45-degree polarizing filter 211s is installed.

The image will be reconstructed by solving the system of equations represented by equation (12). However, equation (12) can be transformed into equation (14) below in order to solve the system of equations.

Here, A and S are matrices represented by the following equation (15).

Here, S is estimated by solving equation (14) above. While various methods exist for solving equation (14), one could, for example, use a pseudo-inverse matrix obtained through normalization.

In particular, when the rank of matrix A is 2MN+1, equation (14) can be expressed as an exact solution as shown in equation (16) below.

However, in reality, the pixel values captured by the image sensor 212 will contain observation errors, so the regularized solution shown in equation (17) below can be considered a more appropriate solution.

Here, λ ² B ^T B are normalization parameters.

The value S, represented by equation (16) or equation (17), can be determined by assigning approximate ^{values to the difference between the light intensity s i∥} _of the incident light at the position where the vertical polarizing filter 111v is installed and the light intensity s _i ⁴⁵ of the 45-degree polarizing filter 211s ( ^{s i∥} _{- s i} ⁴⁵ ), and the difference between the light intensity s _i⊥ of the incident light at the position where the horizontal polarizing filter 111h is installed and the light intensity s _i ⁴⁵ of the 45-degree polarizing filter 211s ( ^{s i⊥} _- ^s _i ⁴⁵ ).

For example, if the light intensity at position s _i on the subject surface 221 is I _i , the linear polarization degree DoLP is ρ _i , and the polarization angle is φ _i , then using Malus's law, the parameters a _i and b _i can be defined as shown in equations (18) and (19) below.

Furthermore, by calculating the sum and difference of the parameters a _i and _bi , we can obtain the following relationships: equations (20) and (21).

Therefore, as shown in equations (22) and (23) below, the light intensity I _i ρ _i and polarization angle φ _i corresponding to the linear polarization degree ρ _i can be determined for each pixel position.

In other words, by solving the system of equations (14) and (15), all polarization components can be obtained at all points on the subject surface 221. Furthermore, by using equations (22) and (23), the light intensity I _i ρ _i and polarization angle φ _i corresponding to the linear polarization degree ρ _i can be solved at all points, while images consisting of unpolarized components cannot be reconstructed.

As a result, an image 221d' obtained by multiplying the light intensity _Ii in Figure 11 by the degree of linear polarization _ρi , and an image 221a consisting of the polarization angle _φi are reconstructed. Furthermore, the reconstructed image 221d' and image 221a are combined to generate image 222.

<Image processing by the imaging device shown in Figure 9>
Next, the imaging process performed by the imaging device 201 in Figure 9 will be explained with reference to the flowchart in Figure 12.

In step S51, the horizontal polarization filter 211h, vertical polarization filter 211v, and 45-degree polarization filter 211s of the mask 211 modulate the horizontal polarization component, vertical polarization component, and 45-degree polarization component of the light from the subject surface 221, respectively, and direct them onto the image sensor 212. The unpolarized component is not modulated by the mask 211 and therefore passes through the mask 211 and is incident onto the image sensor 212.

In step S52, the image sensor 212 captures the light from the subject surface 221 that has passed through the mask 211 and outputs it as an image to the reconstruction unit 213.

In step S53, the reconstruction unit 213 outputs to the output unit 214 a final image (reconstructed image) consisting of light intensity I i ρ i corresponding to linear polarization degree and a final image (reconstructed image) consisting of polarization angle φ _i , based on the captured image obtained by light transmitted through the mask 211 output from the image sensor 212, using the above-described equations ( ₁₄ ) to (23), which consist of a predetermined set of coefficients corresponding to the distance from the imaging position of the imaging device ₂₀₁ to the subject surface 221.

In step S54, the output unit 214 synthesizes a final image (reconstructed image) consisting of linear polarization degree and a final image (reconstructed image) consisting of polarization angle.

In step S55, the output unit 214 performs signal processing on the image obtained by combining the final image (reconstructed image) consisting of light intensities I _i ρ _i corresponding to the linear polarization degree and the final image (reconstructed image) consisting of the polarization angle φ _i , and outputs it as an image signal.

Through the above series of processes, applying the principle of a lensless imaging device, a mask 211 consisting of filters that transmit three or more specific light components modulates and reconstructs the horizontal polarization component, vertical polarization component, and 45-degree polarization component, while not modulating and thus not reconstructing the unpolarized component.

This allows for the reconstruction of a final image consisting of light intensity corresponding to the linear polarization degree of the polarization component, and a final image consisting of the polarization angle, while the image consisting of the unpolarized component is not reconstructed. Therefore, it is possible to appropriately limit privacy-related information, such as a person's face image, which is included in the unpolarized component during imaging, and to capture only images consisting of the polarization component, such as those from smartphones and displays.

<<4. Application Examples of the Second Embodiment>>
In the above, we have described an example in which a mask 211 equipped with three types of filters—a horizontal polarizing filter 211h, a vertical polarizing filter 211v, and a 45-degree polarizing filter 211s—is placed in front of the image sensor 212.

However, the 45-degree polarizing filter 211s may be a polarizing filter with an angle other than 45 degrees. For example, a 30-degree polarizing filter that transmits only the polarization component with a polarization angle of 30 degrees, or a 60-degree polarizing filter that transmits only the polarization component with a polarization angle of 60 degrees, may also be used.

Furthermore, in addition to a filter that transmits only polarization components at a predetermined polarization angle, a filter such as an ND (Neutral Density) filter may also be used.

Figure 13 shows an example configuration of mask 251 in which an ND filter is provided instead of the 45-degree polarizing filter 211s in mask 211 of Figure 9.

In Figure 13, the area on the mask 251 where the vertical polarizing filter 251v is placed is depicted with vertical stripes, the area where the horizontal polarizing filter 251h is placed is depicted with horizontal stripes, and the area where the ND filter 251n is placed is depicted in gray. Furthermore, in the mask 251, the vertical polarizing filter 251v, the horizontal polarizing filter 251h, and the ND filter 251n each have equal transmittance.

<<5. Third Embodiment>>
In the above, we have described an example in which information in the captured image is limited by placing a mask in front of the image sensor, which has filters randomly arranged to transmit polarization components corresponding to multiple polarization directions, modulating the incident light with the mask and taking an image, and then reconstructing the final image so that only specific components of the incident light are reconstructed as an image.

However, the filter formed in the mask may be configured to filter not only components corresponding to the polarization direction, but also components corresponding to other optical properties.

For example, the mask may be constructed by arranging filters in a pseudo-random pattern that transmit specific color components from the RGB (red, green, and blue) components that make up the incident light.

Figure 14 shows an example of an imaging device configuration in which filters that transmit blue and red components of incident light are arranged in a pseudo-random pattern, thereby reconstructing only images consisting of blue and red components, and preventing the reconstruction of images consisting of components that equally contain blue and red components. For example, this limits the information of an image composed of a gray component that equally contains blue, red, and green components.

More specifically, the imaging device 301 in Figure 14 consists of a mask 311, an image sensor 312, a reconstruction unit 313, and an output unit 314. The mask 311, image sensor 312, reconstruction unit 313, and output unit 314 correspond to the mask 31, image sensor 32, reconstruction unit 33, and output unit 34 in Figure 1, respectively.

The mask 311 has the same configuration as the mask 31 in Figure 1, and applies a predetermined polarization to the incident light before directing it onto the image sensor 312.

As described above, the mask 31 is a plate-shaped structure made of a light-shielding material, and is configured to modulate incident light by forming a transmission region 41 and a light-shielding region 42.

In contrast, the mask 311 has a configuration in which a blue filter 311b that transmits the blue component of the light incident from the subject surface 321 on the left side of the figure and a red filter 311r that transmits the red component are arranged in a pseudo-random pattern.

In Figure 14, the area on the mask 311 where the blue filter 311b is placed is depicted as a light gray area, and the area where the red filter 311r is placed is depicted as a dark gray area. Furthermore, in the mask 311, the blue filter 311b and the red filter 311r have equal transmittance.

Here, we consider an example where a smartphone 331 and a display 333, which emit light containing red and blue components on the subject surface 321, and a document 332 on which a confidential document is written in black (or gray) ink on a white surface are present. It is assumed that, for example, white light is projected onto the document 332 from approximately the front, and the confidential document itself written on the paper is essentially colored with gray components.

In this case, the blue component of the light emitted by the smartphone 331, the document 332, and the display 333 is modulated by passing through the blue filter 311b region of the mask 311, which is shown as a light gray area in the figure, and then incident on the image sensor 312.

Furthermore, the red component of the light emitted by the smartphone 331, the document 332, and the display 333 is modulated by passing through the red filter 311r region of the mask 311, which is shown as a dark gray area in the figure, before being incident on the image sensor 312.

In other words, as shown in Figure 15, the mask 311 is transmitted from the subject surface 321, which includes the images of the smartphone 331, the document 332, and the display 333, in a combined form of an image 321r consisting of a red component, an image 321p consisting of a gray component, and an image 321b consisting of a blue component.

As shown in Figure 15, when light from the subject surface 321 passes through the mask 311, modulation is applied to the blue and red components, while the gray component, which contains both blue and red components equally, is not modulated. As a result, an image like image 312i is captured. That is, for the gray component, which contains both blue and red components equally, it passes through both the blue filter 311b and the red filter 311r, so the entire surface of the mask 311 becomes a transmission area, and there is no light-blocking area, and therefore modulation is not applied.

Image 312i is an image consisting of modulated light and unmodulated light in which the images of the smartphone 331, document 332, and display 333 within the subject surface 321 are not yet formed. As a result, as shown in Figure 15, the overall image is blurry.

The reconstruction unit 313 reconstructs images 321b and 321r based on the image 312i supplied from the image sensor 312, and outputs them to the output unit 314.

As explained with reference to Figure 15, the mask 311 is configured such that the blue filter 311b and the red filter 311r are arranged in a pseudo-random pattern.

Therefore, when the blue component passes through the mask 311, the blue filter 311b functions as the transmission region 41 of the mask 31 in Figure 1, and the red filter 311r functions as the light-shielding region 42 in Figure 1, thereby modulating the blue component.

On the other hand, when the red component passes through the mask 311, the red filter 311r functions as the transmission region 41 of the mask 31 in Figure 1, and the blue filter 311b functions as the light-shielding region 42 in Figure 1, thereby modulating the red component.

In other words, the image sensor 312 captures an image 312i of light consisting of the blue component and the red component, which are modulated by the blue filter 311b and the red filter 311r of the mask 311, and the gray component, which is unmodulated and contains equal amounts of the blue and red components, as the observation result of light, and outputs it to the reconstruction unit 313.

Therefore, as shown in Figure 16, the reconstruction unit 313 uses the respective coefficient sets of the blue filter 311b and red filter 311r of the mask 311 to reconstruct image 321b consisting of the blue component and image 321r consisting of the red component from image 312i. At this time, the reconstruction unit 313 cannot reconstruct the unmodulated gray component, which contains equal amounts of blue and red components, and therefore cannot reconstruct the image corresponding to image 321p.

The output unit 314 combines the image 321b, consisting of the blue component, and the image 321r, consisting of the red component, supplied by the reconstruction unit 313, to generate the image 322 shown in the right side of Figure 16, and outputs it as the final image.

As a result, by combining image 321b, which consists of blue components, and image 321r, which consists of red components, an image 322 is output that is composed of the display surface portions of a smartphone 331 and a display 333, which are displaying an image consisting of red and blue components, and does not contain a document 332 consisting of gray components.

As a result, it becomes possible to capture an image while restricting the image capture to only include information consisting of gray components, such as confidential documents, written on the white surface of the subject 321.

<Image processing by the imaging device shown in Figure 14>
Next, the imaging process performed by the imaging device 101 in Figure 14 will be explained with reference to the flowchart in Figure 17.

In step S71, the blue filter 311b and red filter 311r of the mask 311 modulate the blue and red components of the light from the subject surface 321, respectively, and cause them to be incident on the image sensor 312. At this time, gray components, which contain equal amounts of blue and red components, are not modulated even when transmitted through the mask 311, and therefore pass through the mask 311 as is and are incident on the image sensor 312.

In step S72, the image sensor 312 captures an image consisting of light from the subject surface 321, which includes blue and red light that has been modulated by passing through the mask 311, and gray light that contains equal amounts of unmodulated blue and red light, and outputs the captured image to the reconstruction unit 313.

In step S73, the reconstruction unit 313 reconstructs a final image (reconstructed image) consisting of the blue component and a final image (reconstructed image) consisting of the red component based on the captured image obtained by the modulated light output from the image sensor 312, and outputs them to the output unit 314. At this time, the gray component image, which is not modulated and contains equal amounts of blue and red components, is not reconstructed.

In step S74, the output unit 314 combines the final image consisting of the red component with the final image consisting of the blue component. For example, the output unit 314 may output the result of combining the final image consisting of the red component with the final image consisting of the blue component.

In step S75, the output unit 314 performs signal processing on the image obtained by combining the reconstructed image consisting of the red component and the reconstructed image consisting of the blue component, and outputs it as an image signal.

Through the above series of processes, the principle of a lensless imaging device is applied, and a mask 311 consisting of filters that transmit at least two specific light components is used to create a transmission region 41 and a light-shielding region 42 for each light component, thereby modulating the blue and red components.

This allows for the reconstruction of a final image consisting of modulated blue and red components, thereby appropriately limiting the information from the unmodulated gray component that is included during image acquisition, and enabling the acquisition of an image composed solely of blue and red light.

Furthermore, while the above has described an example of reconstructing the final image of the blue and red components, it is also possible to reconstruct the final image of other components. For example, by configuring a mask 311 consisting of filters that transmit blue and green components, the blue and green components can be modulated, and the modulated blue and green components can be reconstructed, preventing the reconstruction of a gray component that contains equal amounts of both blue and green components. Alternatively, by configuring a mask 311 consisting of filters that transmit red and green components, the red and green components can be modulated, and the modulated red and green components can be reconstructed, preventing the reconstruction of a gray component that contains equal amounts of both red and green components.

Furthermore, as described above, in a mask 311 consisting of a filter that transmits blue and red components equally, other color components cannot be reconstructed if the color contains blue and red components equally. For example, an image consisting of a purple component, which is an intermediate color containing blue and red components equally, can also be prevented from being reconstructed, thereby limiting the information of an image consisting of purple components. Similarly, by configuring the mask 311 to consist of a filter that transmits blue and green components, the blue and green components can be modulated, and the modulated blue and green components can be reconstructed, so that other color components containing both blue and green components equally cannot be reconstructed. For example, a cyan component, which is an intermediate color containing both blue and green components equally, can be prevented from being reconstructed. Furthermore, by configuring a mask 311 consisting of a filter that transmits red and green components equally, the red and green components can be modulated, the modulated red and green components can be reconstructed, and other color components that contain both red and green components equally can not be reconstructed. For example, the yellow component, which is an intermediate color containing both red and green components equally, can not be reconstructed.

Furthermore, filtering may be performed not only on one of the red, green, and blue components, but also on two specific wavelength components, so that the final image is reconstructed based on those two specific wavelength components, thereby restricting the reconstruction of images containing wavelength components that equally contain each of the two specific components.

<<6. Application Examples of the Third Embodiment>>
In the above, we have described an example in which two types of components at specific wavelengths are filtered, and the final image of those two types of components at specific wavelengths is reconstructed, so that an image of wavelength components that equally contain both components at specific wavelengths is not reconstructed.

However, it is also possible to filter out each of the RGB components (red, green, and blue components) and reconstruct the final image for each of the RGB components, while leaving the gray component, which includes each of the RGB components, unreconstructed.

Figure 18 shows an example of a mask 351 configuration that filters each of the RGB components (red, green, and blue components) and reconstructs the final image for each of the RGB components, while preventing other components, such as the gray component, which contain each of the RGB components equally, from being reconstructed.

The mask 351 is provided, for example, in the imaging device 301 of Figure 14, in place of the mask 311, in front of the image sensor 312.

The mask 351 consists of a red filter 351r that transmits the red component, a green filter 351g that transmits the green component, and a blue filter 351b that transmits the blue component, each arranged in a pseudo-random pattern.

In Figure 18, the area on the mask 351 where the blue filter 351b is placed is depicted as a dark gray area, the area where the green filter 351g is placed is depicted as a gray area of intermediate density between dark gray and light gray, and the area where the red filter 351r is placed is depicted as a light gray area. Furthermore, in the mask 351, the blue filter 351b, green filter 351g, and red filter 351r each have equal transmittance.

In the mask 351, the red filter 351r, the green filter 351g, and the blue filter 351b transmit the red, green, and blue components, respectively, thereby modulating each component before they are incident on the image sensor 312.

As a result, the reconstruction unit 313 reconstructs the final images of the modulated red, green, and blue components captured by the image sensor 312 and outputs them to the output unit 314.

The output unit 314 synthesizes and outputs the final images of the reconstructed red, green, and blue components.

Here, for example, a component in which red, green, and blue components are equally synthesized, such as a gray component, is transmitted equally through the red filter 351r, the green filter 351g, and the blue filter 351b, respectively.

As a result, components such as the gray component, which are an equal blend of red, green, and blue components, will be incident on the image sensor 312 without being modulated even if they pass through the mask 351.

Therefore, the reconstruction unit 313 cannot reconstruct the final image of a component where the red, green, and blue components are evenly combined, such as the gray component, based on the image captured by the image sensor 312.

As a result, by using the mask 351, it becomes possible to reconstruct the final image for each of the RGB components, while the final image containing each of the RGB components equally, such as the gray component, cannot be reconstructed. Therefore, it becomes possible to limit the inclusion of images consisting of components with equally distributed RGB components, such as the gray component.

<<7. Examples of execution by software>>
Incidentally, the series of processes described above can be executed by hardware, but they can also be executed by software. When the series of processes are executed by software, the programs that make up the software are installed from a storage medium onto a computer that has dedicated hardware built in, or onto a general-purpose computer that can perform various functions by installing various programs.

Figure 19 shows an example of a general-purpose computer configuration. This personal computer has a built-in CPU (Central Processing Unit) 1001. An input/output interface 1005 is connected to the CPU 1001 via a bus 1004. A ROM (Read Only Memory) 1002 and a RAM (Random Access Memory) 1003 are connected to the bus 1004.

The input/output interface 1005 is connected to an input unit 1006 consisting of input devices such as a keyboard and mouse for the user to input operation commands, an output unit 1007 for outputting images of the processing operation screen and processing results to a display device, a storage unit 1008 consisting of a hard disk drive for storing programs and various data, and a communication unit 1009 consisting of a LAN (Local Area Network) adapter for performing communication processing via a network such as the Internet. A drive 1010 is also connected to a removable storage medium 1011, such as a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini Disc)), or a semiconductor memory, for reading and writing data.

The CPU 1001 executes various processes according to the program loaded from the ROM 1002, or from a removable storage medium 1011 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and installed in the storage unit 1008. The RAM 1003 also appropriately stores data necessary for the CPU 1001 to execute various processes.

In a computer configured as described above, the CPU 1001 loads, for example, a program stored in the memory unit 1008 into the RAM 1003 via the input/output interface 1005 and the bus 1004, and executes it, thereby performing the series of processes described above.

The program executed by the computer (CPU 1001) can be provided by recording it on a removable storage medium 1011, such as a packaged media. The program can also be provided via wired or wireless transmission media, such as a local area network, the internet, or digital satellite broadcasting.

In a computer, a program can be installed in the storage unit 1008 via the input/output interface 1005 by inserting the removable storage medium 1011 into the drive 1010. Alternatively, the program can be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the storage unit 1008. Furthermore, the program can be pre-installed in the ROM 1002 or the storage unit 1008.

Furthermore, the programs executed by the computer may be programs that are processed chronologically in the order described herein, or they may be programs that are processed in parallel or at necessary times, such as when a call is made.

Furthermore, the CPU 1001 in Figure 19 implements the functions of the reconstruction unit 113 and output unit 114 in Figure 4, the reconstruction unit 213 and output unit 214 in Figure 9, and the reconstruction unit 313 and output unit 314 in Figure 14.

Furthermore, in this specification, a system means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure or not. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device in which multiple modules are housed in one enclosure, are both considered systems.

Furthermore, the embodiments described herein are not limited to those described above, and various modifications are possible without departing from the spirit of this disclosure.

For example, this disclosure can take the form of cloud computing, where a single function is shared and processed collaboratively by multiple devices via a network.

Furthermore, each step described in the flowchart above can be performed by a single device, or it can be divided and performed by multiple devices.

Furthermore, if a single step includes multiple processes, those processes can be executed by a single device or shared among multiple devices.

Furthermore, this disclosure can also be structured as follows:

<1> A mask is formed in a predetermined pattern in which multiple transmission regions are transmitted for each of the multiple components of the incident light, and the incident light is transmitted with each of the multiple components modulated.
An image sensor that captures a transmitted image consisting of light that has passed through the mask,
An imaging apparatus comprising: a reconstruction unit that reconstructs an image corresponding to the incident light based on the transmitted image.
<2> The imaging apparatus described in <1> wherein the plurality of transmission regions are formed in a pseudo-random pattern.
<3> The imaging apparatus according to <1> or <2> wherein the plurality of transmission regions are formed with equal transmittance.
<4> The imaging apparatus according to any one of <1> to <3>, wherein the plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in multiple directions.
<5> The imaging apparatus described in <4> wherein the plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in two directions.
<6> The imaging apparatus according to <5>, wherein the plurality of transmission regions are formed from a linear polarizing filter that transmits vertical polarization components and a horizontal polarizing filter that transmits horizontal polarization components.
<7> The imaging apparatus according to <6>, wherein the reconstruction unit reconstructs a vertical polarization component image consisting of the vertical polarization component and a horizontal polarization component image consisting of the horizontal polarization component.
<8> The imaging device described in <7>, wherein the vertical polarization component image and the horizontal polarization component image are combined and output.
<9> The imaging apparatus described in <4> wherein the plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in three directions.
<10> The imaging apparatus according to <9>, wherein the plurality of transmission regions are formed from a linear polarizing filter that transmits a vertical polarization component, a horizontal polarizing filter that transmits a horizontal polarization component, and a 45-degree polarizing filter that transmits a 45-degree polarization component.
<11> The imaging apparatus according to <10>, wherein the reconstruction unit reconstructs an image consisting of a composite image of light intensity and linear polarization degree, and a polarization angle.
<12> The imaging apparatus described in <11>, which outputs a reconstructed image obtained by combining the light intensity and the degree of linear polarization, and an image obtained by combining the polarization angle.
<13> The imaging apparatus according to any one of <1> to <3>, wherein the plurality of transmission regions are formed by a linear polarization filter that transmits polarization components in multiple directions and an ND (Neutral Density) filter.
<14> The imaging apparatus described in <13>, wherein the plurality of transmission regions are formed by a linear polarization filter that transmits polarization components in two directions and the ND filter.
<15> The imaging apparatus according to any one of <1> to <3>, wherein the plurality of transmission regions are formed from wavelength filters that transmit a plurality of wavelength components.
<16> The imaging apparatus according to <15>, wherein the plurality of transmission regions are formed by the wavelength filter that transmits two or three types of wavelength components.
<17> The imaging apparatus according to <16>, wherein the plurality of transmission regions are formed from wavelength filters that transmit two or three types of wavelength components from among a blue filter that transmits blue wavelength components, a red filter that transmits red wavelength components, and a green filter that transmits green wavelength components.
<18> The imaging apparatus according to <17>, wherein the reconstruction unit reconstructs images of the two or three wavelength components that form the plurality of transmission regions.
<19> The imaging apparatus described in <18>, wherein the reconstructed images of the two or three wavelength components are synthesized and output.
<20> A mask in which multiple transmission regions that transmit incident light component by component are formed in a predetermined pattern,
Image sensor and
In a method for operating an imaging device equipped with a reconstruction unit,
The mask modulates each component of the incident light and transmits it as modulated light.
The image sensor captures a transmitted image consisting of light that has passed through the mask.
A method for operating an imaging device, comprising the step of the reconstruction unit reconstructing an image corresponding to the incident light based on the transmitted image.

101 Imaging device, 111 Mask, 111h Horizontal polarizing filter, 111v Vertical polarizing filter, 112 Image sensor, 113 Reconstruction unit, 114 Output unit, 201 Imaging device, 211 Mask, 211h Horizontal polarizing filter, 211v Vertical polarizing filter, 211s 45-degree polarizing filter, 212 Image sensor, 213 Reconstruction unit, 214 Output unit, 221 Mask, 222 Optical element, 223 Image sensor, 251 Mask, 251h Horizontal polarizing filter, 251v Vertical polarizing filter, 251n ND filter, 301 Imaging device, 311 Mask, 311b Blue filter, 311r Red filter, 312 Image sensor, 313 Reconstruction unit, 314 Output unit

Claims (12)

A linear polarization filter that transmits incident light for each of the three polarization components is formed, and multiple transmission regions are formed in a predetermined pattern, and a mask that modulates each of the three polarization components of the incident light and transmits it,
An image sensor that captures a transmitted image consisting of light that has passed through the mask,
The system includes a reconstruction unit that reconstructs an image consisting of a composite image of the light intensity and linear polarization degree corresponding to the incident light, and an image consisting of the polarization angle, based on the transmitted image.
The aforementioned plurality of transmission regions are formed from a linear polarizing filter that transmits the polarization component in the vertical direction, a horizontal polarizing filter that transmits the polarization component in the horizontal direction, and a 45-degree polarizing filter that transmits the polarization component in the 45-degree direction.
Imaging device. The imaging apparatus according to claim 1, wherein the plurality of transmission regions are formed in a pseudo-random pattern. The imaging apparatus according to claim 1, wherein the plurality of transmission regions are formed with equal transmittance. The imaging apparatus according to claim 1 , wherein a reconstructed image consisting of the light intensity and the linear polarization degree, and an image consisting of the polarization angle are combined and output. The imaging apparatus according to claim 1, wherein the plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in multiple directions and an ND (Neutral Density) filter. The imaging apparatus according to claim 5 , wherein the plurality of transmission regions are formed from a linear polarization filter that transmits polarization components in two directions and the ND filter. The imaging apparatus according to claim 1, wherein the plurality of transmission regions are formed from wavelength filters that transmit a plurality of wavelength components. The imaging apparatus according to claim 7 , wherein the plurality of transmission regions are formed from the wavelength filter that transmits two or three types of wavelength components. The imaging apparatus according to claim 8, wherein the plurality of transmission regions are formed from wavelength filters that transmit two or three types of wavelength components from among a blue filter that transmits blue wavelength components, a red filter that transmits red wavelength components, and a green filter that transmits green wavelength components. The imaging apparatus according to claim 9 , wherein the reconstruction unit reconstructs images of the two or three wavelength components that form the plurality of transmission regions. The imaging apparatus according to claim 10 , wherein the reconstructed images of the two or three wavelength components are synthesized and output. A mask formed by a linear polarization filter that transmits incident light for each of the three polarization components, with multiple transmission regions formed in a predetermined pattern,
Image sensor and
It comprises a reconstruction unit ,
In a method for operating an imaging device in which the plurality of transmission regions are formed by a linear polarization filter that transmits a vertical polarization component, a horizontal polarization filter that transmits a horizontal polarization component, and a 45-degree polarization filter that transmits a 45-degree polarization component ,
The mask modulates and transmits each of the three deflection components of the incident light,
The image sensor captures a transmitted image consisting of light that has passed through the mask,
The reconstruction unit reconstructs an image based on the transmitted image, consisting of an image obtained by combining the light intensity and linear polarization degree corresponding to the incident light, and an image consisting of the polarization angle.
A method for operating an imaging device, including the device itself.