JP7769271B2 - Imaging device, imaging method, and program - Google Patents

Description

The present invention relates to imaging devices, imaging methods, and program technologies.

Conventionally, there is a hyperspectral image measurement technology based on compressed sensing theory called compressed spectral imaging. One implementation method is a technology called CASSI (Coded Aperture Snapshot Spectral Imaging), which combines a coded aperture mask and a dispersive optical element (see, for example, Non-Patent Document 1).

In addition, with regard to such CASSI, a method for designing a coded aperture mask within the CASSI optical system has been proposed as a technique for improving the measurement accuracy of hyperspectral images (see, for example, non-patent document 2).

On the other hand, a technology has been proposed to improve measurement accuracy by limiting the wavelength range information acquired through devising the design of the coded aperture mask in CASSI and the formulation of the optimization problem in the reconstruction process from the compressed signal to the original signal (see, for example, non-patent document 3).

Compressive Coded Aperture Spectral Imaging: An Introduction [Arce et al., 2013] Compressive spectral imaging approach using adaptive coded apertures [Zhang et al., 2020] Rank minimization code aperture design for spectrally selective compressive imaging[Arguello et al., 2013]

Generally, the above-mentioned CASSI is formulated as equation (1).

In equation (1), g represents the compressed signal, f represents the original signal, and Φ is an observation matrix (also called a compression matrix) that represents the CASSI observation process. In this way, the observation process (observation matrix) for obtaining the compressed signal in compressed sensing is closely related to the performance of the subsequent original signal estimation.

However, the only degree of design freedom in the CASSI observation process is the design of the coded aperture mask. Therefore, it is difficult to design an ideal observation matrix for reconstructing the original signal. The CASSI observation process (Φ) is a process of encoding → dispersion → accumulation, but the dispersion and accumulation processes are inherent to optical elements and there is no degree of design freedom.

Furthermore, conventional technologies that limit the wavelength range to be acquired only allow for the choice of whether or not to acquire each wavelength range, and are unable to consider the importance of each. In addition, because the wavelength range is limited uniformly for the input signal, it is not possible to acquire information on different wavelength ranges for spatially different regions.

In addition, in conventional technology, in order to limit the wavelength range to be acquired, pixels containing only information from the limited wavelength range are extracted from the compressed signal to solve the reconstruction problem, which reduces the number of available compressed signal elements and is expected to reduce reconstruction accuracy. Therefore, a conventional approach has been to suppress the decline in reconstruction accuracy by supplementing compressed signal information through multiple captures, but the disadvantage is that the cost of measurement increases.

In consideration of the above circumstances, the present invention aims to provide technology that can improve the measurement accuracy of hyperspectral images using CASSI.

One aspect of the present invention is an imaging device that measures hyperspectral images using compressed sensing, and includes a CASSI observation system that includes: an encoding unit that encodes and outputs an input signal by applying a coded aperture mask to the input signal; a dispersing unit that wavelength-disperses and outputs the input signal coded by the encoding unit; and a measurement unit that captures the input signal wavelength-dispersed by the dispersing unit; an attention weight setting unit that sets an attention weight for each pixel captured by the CASSI observation system; and a mask generation unit that generates the coded aperture mask based on the attention weight for each pixel set by the attention weight setting unit.

One aspect of the present invention is an imaging method for measuring hyperspectral images using compressed sensing, comprising: an imaging step of imaging an input signal using a CASSI observation system that includes: an encoding unit that encodes the input signal by applying a coded aperture mask to the input signal and outputs the encoded signal; a dispersion unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measurement unit that images the input signal wavelength-dispersed by the dispersion unit; and a modulation step of modulating the output signal of the CASSI observation system using a multiplication mask.

One aspect of the present invention is a program for causing a computer to function as the above-mentioned imaging device.

This invention makes it possible to improve the measurement accuracy of hyperspectral images using CASSI.

1 is a block diagram showing a configuration of an imaging apparatus 1A according to a first embodiment. FIG. 2 is a conceptual diagram showing a processing flow by the imaging device 1A of the first embodiment. 1A and 1B are diagrams illustrating an example of an effect achieved by the imaging device 1A of the first embodiment. FIG. 10 is a block diagram showing the configuration of an imaging device 1B according to a second embodiment. FIG. 10 is a conceptual diagram showing a processing flow by an imaging device 1B according to a second embodiment. FIG. 10 is a conceptual diagram showing the flow of processing when the imaging device 1B of the second embodiment generates a coded aperture mask and a multiplication mask. FIG. 10 is a conceptual diagram showing a flow in which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model in the imaging device 1B of the second embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
First Embodiment
1 is a block diagram showing the configuration of an image pickup apparatus 1A according to the first embodiment. The image pickup apparatus 1A includes an optical system 110, a CASSI observation system 120, a modulation unit 130, and a reconstruction processing unit 140.

The imaging device 1A is configured using a processor such as a CPU (Central Processing Unit) and memory. The imaging device 1A functions as a device that receives light as an input signal and outputs an estimated signal for a hyperspectral image by having the processor execute a program. Of the components of the imaging device 1A, part of the CASSI observation system 120, the modulation unit 130, and the reconstruction processing unit 140 are implemented by having the processor execute a program. Note that some or all of the functions of the imaging device 1A that perform electrical signal processing may be implemented using hardware such as an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), or an FPGA (Field Programmable Gate Array). The above program may be recorded on a computer-readable recording medium. Examples of computer-readable recording media include portable media such as flexible disks, magneto-optical disks, ROMs, CD-ROMs, and semiconductor storage devices (e.g., SSDs: Solid State Drives), as well as storage devices such as hard disks and semiconductor storage devices built into computer systems. The above program may also be transmitted via a telecommunications line.

The optical system 110 is composed of a lens that forms an image. The optical system 110 may include multiple lenses.

The CASSI observation system 120 has the function of encoding and measuring an input signal using CASSI. More specifically, the CASSI observation system 120 includes, for example, an encoding unit 121, a dispersing unit 122, and a measuring unit 123. The configuration of the CASSI observation system 120 is described below, but this is intended to mean that the CASSI observation system 120 of the first and second embodiments may be similar to conventional CASSI observation systems.

The encoding unit 121 applies a coded aperture mask to the input signal to output a coded input signal. The coded aperture mask may be a liquid crystal on silicon (LCOS) or digital mirror device (DMD). The coded aperture mask may be composed of discrete values of 0 and 1 (hereinafter sometimes represented as {0,1}) or continuous values ranging from 0 to 1 (hereinafter sometimes represented as [0,1]). For example, the coded aperture mask may be generated using a data-driven approach in which the coded aperture is modeled as a variable parameter and determined by end-to-end input/output optimization, including a reconstruction model. Alternatively, the coded aperture mask may be generated using a theoretical approach based on compressed sensing theory, which exploits the incoherence of the basis during sparse image transformation.

The dispersion unit 122 is, for example, a prism. The dispersion unit 122 receives the output signal from the encoding unit 121, disperses the input signal by wavelength, and outputs the result.

The measurement unit 123 is, for example, an FPA (Focal Plane Array) array sensor. The measurement unit 123 receives the output signal of the dispersion unit 122 from each sensor for each pixel, integrates it in the wavelength direction, and outputs it. The measurement unit 123 outputs a compressed signal (image) as the measurement result.

The modulation unit 130 receives the output signal of the CASSI observation system 120, modulates it with a multiplication mask, and outputs the result. For example, the multiplication mask can be generated using a data-driven approach, in which it is modeled as a variable parameter, similar to the coded aperture mask described above, and determined by end-to-end input/output optimization including the coded aperture mask and the reconstruction model.

The reconstruction processing unit 140 inputs the output signal of the modulation unit 130 and performs reconstruction processing on it, thereby outputting an estimated result (estimated signal) of the hyperspectral image.

Figure 2 is an image diagram showing the processing flow by the imaging device 1A. Figure 2 shows how the signal conversion at each step is parameterized (modeled). Note that in the signal conversion in Figure 2, solid lines represent conversion based on the inherent parameter values (fixed) of optical elements (prisms, sensors, etc.), while dashed lines represent conversion based on designable variable parameters.

First, the imaging device 1A inputs wavelength data (Spectral Data Cube) D0 of light in the imaging target space to the CASSI observation system 120 via the optical system 110 (step S1).

Next, in the CASSI observation system 120, the encoding unit 121 applies a coded aperture mask M1 to the input wavelength data (input signal) and outputs the coded wavelength data D1 to the dispersion unit 122 (step S2). This process means that each pixel of the 3D (spatial and wavelength) input signal is covered with a coded aperture mask.

Next, in the CASSI observation system 120, the dispersion unit 122 wavelength-disperses the encoded wavelength data D1 and outputs it to the measurement unit 123 (step S3). This process involves tilting the wavelength axis using a dispersion optical element (prism).

Next, in the CASSI observation system 120, the measurement unit 123 inputs the wavelength-dispersed wavelength data D2 and accumulates the wavelength data D2 in the wavelength direction for each pixel, thereby generating compressed signal data D3 as the measurement result of the target space and outputting it to the modulation unit 130 (step S4). This process involves accumulating each tilted wavelength signal for each pixel. Through these steps, a 2D compressed signal is obtained.

Next, the modulation unit 130 generates modulated compressed signal data D4 by modulating the 2D compressed signal data D3 input from the CASSI observation system 120 using a multiplication mask (step S5). The modulation unit 130 outputs the generated modulated compressed signal data D4 to the reconstruction processing unit 140.

Next, the reconstruction processing unit 140 performs reconstruction processing on the modulated compressed signal data D4 input from the modulation unit 130, thereby generating and outputting a hyperspectral image IMG as an estimation result (step S6). For example, the reconstruction processing unit 140 performs reconstruction processing by inputting the modulated compressed signal data D4 into a reconstruction model.

Here, the reconstruction model is generated by, for example, learning in advance the correlation between the desired signal data and the modulated compressed signal data (i.e., the observation matrix) using a machine learning method such as a neural network. For example, the reconstruction model can be constructed using a DUN (Deep Unrolled Network), which implements an iterative optimization algorithm such as ADMM (Alternating Direction Method of Multiplier) or ISTA (Iterative Shrinkage Thresholding Algorithm) using deep learning, and can be learned by end-to-end optimization of variable parameters including those of the encoding unit 121 and the modulation unit 130.

Figure 3 is a diagram showing an example of the effect achieved by the imaging device 1A of the first embodiment. Figure 3 shows a comparison between the image estimation accuracy using a conventional configuration that does not use a multiplication mask (only a coded aperture mask) and the image estimation accuracy using the configuration of this embodiment that uses the above-mentioned multiplication mask (a combination of a coded aperture mask and a multiplication mask). The vertical axis of the graph shown in Figure 3 represents the intensity of the PSNR (Peak Signal-to-Noise Ratio). As is clear from Figure 3, the estimation results of this embodiment have a larger PSNR value than the conventional configuration, indicating improved image quality.

According to the imaging device 1A of the first embodiment configured in this manner, the accuracy of measuring hyperspectral images by CASSI can be improved by performing modulation processing using a multiplication mask on the measured signal in the CASSI observation system that measures hyperspectral images.

More specifically, because each pixel of the coded aperture mask used in the CASSI observation system either blocks or attenuates the corresponding pixel information in the input signal (attenuating in the case of [0,1]), in conventional configurations, signal conversion for each wavelength signal within the same pixel in the input signal was limited to a signal conversion common to the pixel (a conversion that multiplies {0,1} or [0,1] by a scalar). In contrast, the imaging device 1A of the first embodiment includes not only a coded aperture mask that acts directly on the input signal before dispersion processing, but also a modulation unit 130 that multiplies and modulates the measurement signal resulting from dispersion processing and integration processing by the mask, thereby enabling different signal conversions for each wavelength signal within the same pixel in the input signal.

This makes it possible to design observation matrices with a high degree of freedom when measuring hyperspectral images, and by appropriately designing the two masks, the coded aperture mask and the multiplication mask, it is possible to improve the accuracy of the original signal that is ultimately obtained.

Furthermore, the design of the CASSI observation matrix uses various optical elements and is implemented in hardware. In contrast, the modulation unit 130 of this embodiment processes signals after signal compression and can be implemented in software. Therefore, there are fewer physical constraints on the implementation of the modulation unit 130, and advantages include ease of handling in terms of conversion speed and representation of continuous values.

Furthermore, in the imaging device 1A of the first embodiment, the modulation unit 130 does not add or modify anything to the conventional imaging process using CASSI. Therefore, with the imaging device 1A of this embodiment, the estimation accuracy of the original signal can be improved easily and without complicating the device.

Second Embodiment
4 is a block diagram showing the configuration of an image pickup device 1B according to the second embodiment. In FIG. 4, the same components as those in FIG. 1 are denoted by the same reference numerals, and a description thereof will be omitted. The image pickup device 1B differs from the image pickup device 1A according to the first embodiment in that it does not include a modulation unit 130, but includes a focus weight setting unit 150 and a mask generation unit 160.

The attention weight setting unit 150 sets a weight (hereinafter referred to as "attention weight") indicating the degree to which the observation data should be focused on for each pixel captured by the CASSI observation system 120. For example, the attention weight setting unit 150 accepts input for setting the attention weight from an operation input unit (not shown), and based on the input information, generates attention weight information indicating the attention weight for each pixel and outputs it to the mask generation unit 160.

More specifically, the attention weight is a value within the continuous range of [0, 1], and the attention weight setting unit 150 sets a 3D (spatial and wavelength) attention weight for each pixel.

Note that the attention weight may be set by any method other than the method described above. For example, attention weight information may be received from another device via communication, or pre-stored attention weight information may be read from a storage unit (not shown).

The mask generation unit 160 generates a coded aperture mask based on the attention weight information input from the attention weight setting unit 150. The mask generation unit 160 sets the generated coded aperture mask in the coding unit 121 of the downstream CASSI observation system 120.

Figure 5 is an image diagram showing the processing flow by imaging device 1B. First, in imaging device 1B, attention weight setting unit 150 performs processing to set attention weights for each pixel (step S201) and outputs attention weight information indicating the settings to mask generation unit 160. For example, attention weight setting unit 150 may accept an operation to select an attention area in an image with a rectangle, or may accept an operation to input an annotation map that associates information on whether each area in the image is an attention area or not.

Next, the mask generation unit 160 inputs the attention weight information from the attention weight setting unit 150 and generates a coded aperture mask based on the attention weight information (step S202). For example, the mask generation unit 160 can generate a coded aperture mask using an image processing model based on a CNN (Convolutional Neural Network) or a DNN (Deep Neural Network), such as U-NET. The mask generation unit 160 sets the generated coded aperture mask in the coding unit 121 of the downstream CASSI observation system 120.

The subsequent processing is the same as the processing flow when the multiplication mask is not applied in the first embodiment. Specifically, the CASSI observation system 120 acquires (images) observation data using the coded aperture mask set in step S202, and the reconstruction processing unit 140 performs reconstruction processing on the acquired observation data to output an image that is an estimation result of the original signal. In Figure 5, processes similar to those performed by the imaging device 1A in the first embodiment are denoted by the same symbols as in Figure 2.

Note that Figure 5 describes a case in which the mask generation unit 160 generates only a coded aperture mask by setting the attention weights, but the mask generation unit 160 may also generate the multiplication mask of the first embodiment in addition to the coded aperture mask. In this case, the function of the mask generation unit 160 to generate the multiplication mask can be realized by machine learning using a deep learning model.

Figure 6 is an image diagram showing the processing flow in this case. In this case, the imaging device 1B may further include the modulation unit 130 of the first embodiment and be configured to modulate the observation data of the CASSI observation system 120 and then perform reconstruction processing.

Figure 7 is an image diagram showing the process by which the mask generation unit 160 generates a multiplication mask and the reconstruction processing unit 140 learns a reconstruction model. As described above, the function of the mask generation unit 160 to generate a multiplication mask and the reconstruction processing unit 140 can be realized by a deep learning model. For example, the mask generation unit 160 and the reconstruction processing unit 140 can construct a multiplication mask generation model and a reconstruction model specialized for reconstructing the region of interest by learning the MSE (Mean Squared Error) weighted by the attention weight set on a pixel-by-pixel basis as the loss. In other words, by expressing the attention weight as a real value [0, 1] and designing the loss using this value, it is possible to measure hyperspectral images taking into account the level of importance. For example, the loss is defined as in the following equation (2):

As described above, the imaging device 1B of the second embodiment can configure a reconstruction model specialized for the region of interest by setting attention weights indicating the weight of the degree of attention for each pixel in the captured image for the CASSI observation system that measures the hyperspectral image, and generating a coded aperture mask based on the set attention weight information. Therefore, the imaging device 1B of the second embodiment can improve the measurement accuracy of hyperspectral images using CASSI.

Furthermore, the imaging device 1B of the second embodiment is equipped with the modulation unit 130 of the first embodiment, and the mask generation unit 160 generates a multiplication mask based on a generative model (DNN) trained to perform end-to-end optimization, including a coded aperture mask based on the attention weight information set by the attention weight setting unit 150, and the modulation unit 130 modulates the observation data using the generated multiplication mask, thereby further improving the measurement accuracy of hyperspectral images by CASSI.

More specifically, the imaging device 1B of the second embodiment weights the coded aperture mask used in the CASSI observation system 120 for the wavelength range of interest on a pixel-by-pixel basis (setting the weight of interest), and then generates the coded aperture mask using a DNN (Deep Neural Network) with the weight of interest for each pixel and wavelength range as input. In other words, by designing pixel-wise weights and learning a coded aperture mask generation model and reconstruction model that take this into account, it is possible to acquire measurement images in which the wavelength region of interest is spatially varied.

This allows the importance (weight) of each wavelength range to be set as a value within a continuous range of [0, 1] rather than as a discrete value of {0, 1}, making it possible to set different importance of wavelength ranges for each pixel during imaging. Unlike conventional methods, the imaging device 1B of the second embodiment can utilize all pixel information from the acquired compressed signal, so the reconstruction results from a single image can be expected to have sufficient performance, thereby reducing measurement costs.

The above describes in detail an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this embodiment and also includes designs that do not deviate from the gist of the present invention.

The present invention is applicable to imaging devices that measure hyperspectral images using the CASSI observation system.

1A, 1B...imaging device, 110...optical system, 120...CASSI observation system, 121...encoding unit, 122...dispersion unit, 123...measurement unit, 130...modulation unit, 140...reconstruction processing unit, 150...weight setting unit, 160...mask generation unit

Claims (8)

An imaging device that measures hyperspectral images using compressed sensing,
a CASSI observation system including: an encoding unit that applies a coded aperture mask to an input signal to encode the input signal and output the encoded signal; a dispersing unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measuring unit that captures the input signal wavelength-dispersed by the dispersing unit;
an attention weight setting unit that sets an attention weight for each pixel captured by the CASSI observation system;
a mask generation unit that generates the coded aperture mask based on the attention degree weight for each pixel set by the attention weight setting unit;
An imaging device comprising: a modulation unit that modulates the output signal of the CASSI observation system using a multiplication mask;
The imaging device according to claim 1 . the mask generation unit generates the multiplication mask in addition to the coded aperture mask based on the weight of the attention degree, and sets the generated multiplication mask in the modulation unit.
The imaging device according to claim 2 . the attention weight setting unit is capable of setting an arbitrary value between 0 and 1 as the weight of the attention degree for each pixel.
The imaging device according to claim 1 . a reconstruction processing unit that receives an output signal from the modulation unit, performs reconstruction processing on the output signal using a reconstruction model that is pre-constructed by a neural network , and outputs an estimation result of a hyperspectral image;
The imaging device according to claim 2 . the mask generation unit constructs the coded aperture mask and the multiplication mask such that end-to -end input/ output of the mask generation unit, the CASSI observation system, the modulation unit, and the reconstruction processing unit is optimized by the reconstruction model .
The imaging device according to claim 5 . An imaging method for measuring a hyperspectral image by compressed sensing, comprising:
an imaging step of imaging an input signal using a CASSI observation system including: an encoding unit that applies a coded aperture mask to the input signal to encode the input signal and output the encoded signal; a dispersing unit that wavelength-disperses the input signal encoded by the encoding unit and outputs the wavelength-dispersed input signal; and a measuring unit that images the input signal wavelength-dispersed by the dispersing unit;
a modulating step of modulating the output signal of the CASSI observation system by a multiplication mask;
An imaging method comprising: A program for causing a computer to function as the imaging device described in any one of claims 1 to 6.