JP7825540B2 - Spectroscopic imaging system, measurement system - Google Patents

Description

The present invention relates to a spectroscopic imaging system that disperses and detects incident light.

When observing substances or objects using a camera or microscope, the reflected or transmitted light from the substance or object is detected by the camera's or microscope's detector (such as an image sensor) and imaged. The detector captures all light of wavelengths to which it is sensitive. For example, a Si CMOS image sensor is sensitive to light of wavelengths around 0.4 to 1 μm, and integrates and detects light of that wavelength. In contrast, spectroscopic imaging obtains images at specific wavelengths by dispersing reflected or transmitted light and detecting light of specific wavelengths with a detector. Multispectral imaging, which uses a few to a dozen wavelengths (bands)—that is, several to a dozen corresponding images—is sometimes called multispectral imaging, while hyperspectral imaging, which uses more than this, is sometimes called hyperspectral imaging. Generally, different types of substances or objects absorb different wavelengths, so images obtained using spectroscopic imaging can be used to classify the type of substance or object and determine the amount of moisture they contain.

Multispectral imaging can be achieved by methods such as placing multiple types of optical filters on an image sensor that only allow light of specific wavelengths to pass through. For example, a color image sensor is equipped with color filters that pass red, green, and blue, respectively, so it can be considered a type of multispectral imaging. On the other hand, when using a similar method in hyperspectral imaging, increasing the number of optical filters reduces spatial resolution, so a method is sometimes used in which light dispersed by a spectroscopic element such as a diffraction grating is irradiated onto the image sensor. However, because light is dispersed in the x or y direction and irradiated onto an image sensor consisting of a two-dimensional array, the remaining dimension is one, meaning it becomes a line sensor, and the entire image cannot be captured at once.

In contrast, a compressed sensing technique is known that uses optical filters but significantly reduces the number of filters to the number of bands desired. This technique does not use optical filters that only allow light of specific wavelengths to pass, but instead uses multiple types of optical filters that allow light of various wavelengths to pass but with different transmittance depending on the wavelength. The intensity of the light that passes through each filter is detected by an image sensor, and the original light spectrum is reconstructed from this intensity information. To increase the accuracy of the reconstructed spectrum, it is best for the wavelength dependence of transmittance to differ as much as possible between different types of filters. Although this technique also uses multiple types of filters, while spatial resolution is inferior to images that do not use filters, it can significantly improve spatial resolution compared to using filters for the number of bands desired.

Patent Document 1 listed below addresses the issue of "providing a spectroscopic device capable of detecting the light intensity for each wavelength to be detected with high accuracy." It describes the following technology: "Spectroscopic device 1 detects the light intensity of each of multiple wavelengths to be detected. Spectroscopic device 1 includes multiple optical filters having multiple transmission wavelength bands with transmittances equal to or greater than a threshold transmittance in a wavelength band of interest that includes the multiple wavelengths to be detected. Spectroscopic device 1 includes detection units 11 and 17 that detect the light intensity of each of the multiple wavelengths to be detected based on light that has passed through the multiple optical filters. The multiple optical filters have multiple transmission wavelength bands that are different from one another." (See Abstract).

Japanese Patent Application Laid-Open No. 2019-138670

When using spectral imaging to acquire images of moving objects or substances, or moving bodies, if the subject moves relatively significantly between the time the detector acquires one piece of information (image information or line information) and the time it acquires the next piece of information, it is preferable to acquire image information at high speed. Therefore, methods using optical filters are more suitable than methods that disperse light using diffraction gratings, etc. Furthermore, when observing objects or substances of a specific type, a large number of bands is generally required, so hyperspectral imaging methods are preferable to multispectral methods.

When performing hyperspectral imaging using filters with compressed sensing techniques, a spectrum is reconstructed from the light intensity transmitted through multiple types of filters, but the spectrum of the light before passing through the filters used for reconstruction must be approximately identical. However, with conventional technology such as that disclosed in Patent Document 1, it is difficult to determine whether the spectrum of the light before passing through the filters used for reconstruction is approximately identical.

The present invention was developed in consideration of the above-mentioned problems, and aims to provide a spectral imaging technology that can acquire image information with a large number of bands accurately and quickly.

The spectroscopic imaging system of the present invention has multiple optical filters with different light transmittances for each wavelength, determines whether the incident light entering each of the optical filters has the same spectrum, and calculates the spectrum of the incident light using the optical filters that are determined to have the same spectrum.

The spectroscopic imaging system of the present invention makes it possible to acquire image information with a large number of bands accurately and quickly. Other issues, configurations, advantages, and other aspects of the present invention will become clear from the description of the embodiments below.

1 is a configuration diagram illustrating an outline of a spectral imaging system according to a first embodiment. 1 is a bird's-eye view showing an enlarged view of a part of the chip 2. FIG. FIG. 2 is an enlarged top view of a part of the chip 2. The relationship between the spectrum I of the incident light 1, the effective transmittance T, and the output P of the image sensor 7 is shown. 10A and 10B are schematic diagrams showing examples in which the spectrum of light incident on each pattern is different. The following shows a modification of equation (1) when the spectrum differs for each pattern. The results of decomposing equation (2) into two independent equations are shown below. FIG. 10 is a top view showing the configuration of a macrocell in the second embodiment. FIG. 10 is a configuration diagram of a spectral imaging system according to a third embodiment. FIG. 10 is a configuration diagram of a measurement system according to a third embodiment.

<Embodiment 1>
1 is a block diagram illustrating an outline of a spectroscopic imaging system according to a first embodiment of the present invention. The system includes: a chip 2 that converts incident light 1 into an electrical signal representing the light intensity; a computing device 3 that specifies a driving method for the chip 2 and receives and performs calculations on output signals from the chip 2; and a storage device 4 that stores the calculation results. A display that displays the calculation results may be connected as needed. The computing device 3 may be composed of multiple devices, and the incident light 1 may be focused by a lens and irradiated onto the chip 2.

Figure 2 is a bird's-eye view showing an enlarged portion of the chip 2. Nanostructures 5, each with dimensions comparable to or shorter than the wavelength of the incident light 1, are formed in an array to form the optical filter 6. An image sensor 7 is formed below the optical filter 6. Although the figure shows only a small number of holes making up the nanostructure 5, in reality, many holes are formed. The shape of the holes is not limited to being round or square.

The optical filter 6 may be provided with a substrate used in fabricating the optical filter 6 or an anti-reflection film formed on top of it. If the substrate used in fabricating the optical filter 6 is left, it must be able to transmit the incident light 1 well. Therefore, if the incident light 1 is visible light, the substrate may be, for example, quartz, or if the incident light 1 is infrared light, it may be, for example, quartz or silicon. Since the optical filter 6 must partially transmit the incident light 1, it is formed thin, with a film thickness of, for example, 100 nm to 500 nm, and made of, for example, silicon nitride, silicon, or silicon carbide. The optical filter 6 may be fabricated separately from the image sensor 7 and then bonded together, or it may be formed by depositing a film on the image sensor 7 and processing the nanostructures 5. The nanostructures 5 are formed so that, for example, circular or square-shaped holes penetrate the optical filter 6, with the pattern dimensions and spacing being comparable to or shorter than the wavelength of the incident light 1.

Figure 3 is a top view showing an enlarged portion of the chip 2. The nanostructures 5 are configured with nine different patterns repeatedly arranged. A set of these nine different patterns is called a macrocell. Each pattern may correspond to one pixel of the image sensor 7 directly below it, or may be formed on multiple pixels. The dimensions of each pattern of the nanostructures 5 are formed to be an integer multiple of the pixel size of the image sensor 7. In Figure 3, the nanostructures 5 are formed with nine different patterns, but any number of patterns may be used. In this embodiment, nine types will be described.

First, let's assume that the incident light 1 is spatially uniform. In this case, the spectrum of the incident light 1 is reconstructed using the output of the image sensor 7 directly below the nine types of nanostructures 5. If each pattern of the nanostructures 5 is arranged on multiple pixels of the image sensor 7, the output of a representative pixel may be used, or the average output of multiple pixels may be used. The spectrum I of the incident light 1, the effective transmittance T, and the output P of the image sensor 7 have the relationship shown in equation (1) in Figure 4.

The effective transmittance is not the transmittance of the optical filter 6 alone, but is the transmittance of all the materials present between the incident light 1 and the image sensor 7 multiplied by the sensitivity of the image sensor 7 (i.e., the efficiency with which the incident light is converted into output). n is the number of bands when the wavelength range to which the image sensor 7 responds is discretely expressed. The intensity I _m is the intensity of the incident light 1 at the mth wavelength, the transmittance T _lm is the effective transmittance at the mth wavelength of light passing through the lth optical filter, and P _l is the output of the image sensor 7 corresponding to the lth optical filter. The output P is a measured value, and the transmittance T is measured in advance. The intensity I of the incident light 1 is calculated using this information, but because the number of bands n is greater than the number of types of optical filters 6, it cannot be calculated analytically. Therefore, I satisfying equation (1) is estimated by search, i.e., the spectrum I is reconstructed.

Figure 5 is a schematic diagram showing an example in which the incident light spectrum for each pattern is different. Correct reconstruction is not possible unless the spectrum of the light incident on the optical filters (nine types in this case) used for reconstruction is roughly the same, but as shown in Figure 5, the spectrum may differ for each pattern. In this case, equation (1) can be rewritten as equation (2) in Figure 6. The incident light spectrum is I or J, and spectrum I is irradiated onto the 1st to Kth filters, and spectrum J is irradiated onto the K+1st to 9th filters. In Figure 5, K=4.

Equation (2) can be decomposed into two independent equations, equations (3) and (4), shown in FIG. 7. Both equations (3) and (4) have the same form as equation (1), and in the case of equation (3), the spectrum I is reconstructed using the outputs P ₁ to P _K of the image sensor corresponding to K types of optical filters. The important point here is that the value of K is known, that is, the boundary at which the spectrum changes is known. Therefore, the problem according to the present invention can be solved by recognizing the boundary and dynamically selecting the optical filter to be used for reconstruction according to the boundary.

In embodiment 1, the outputs (pixel values) of the image sensor detection elements corresponding to the same type of optical filter are compared between adjacent macrocells in Figure 3. If the comparison reveals that the outputs are roughly the same, it is assumed that incident light with the same spectrum is being irradiated. On the other hand, if the comparison reveals a difference in output, it is assumed that incident light with a different spectrum is being irradiated. In other words, there is a boundary somewhere between the two types of optical filters being compared. By comparing all optical filters in this way, it is possible to determine the boundary, and as a result, it is possible to provide spectral imaging that shows the correct spectrum. Note that when comparing the outputs of the image sensor detection elements, outputs that have been corrected for individual differences between the detection elements are used.

According to this embodiment, spectral information and spatial information can be acquired at the same time. That is, the computing device 3 creates data with spectral information attached for each pixel of the image sensor 7 (or for multiple pixels if one optical filter 6 is arranged across multiple pixels), thereby creating data that describes both the spatial information represented by the image and the spectral information for each pixel. This makes it possible to obtain information seamlessly by loading it onto a fast-moving vehicle. Therefore, by combining it with a drone or the like that knows its own coordinates using techniques such as satellite communication, it is possible to embed spectral information in a map and provide it.

In this embodiment, the amount of data acquired is compressed. In other words, the amount of information acquired is small compared to the spectral and spatial information obtained by reconstruction, so the load on storage is small. Therefore, the acquired data is stored in storage on the mobile object, and the spectra can be reconstructed later to obtain spectral and spatial information. In this case, the computing device on the mobile object can be small, making it suitable for use in drones and other devices that require lightweight design.

<Embodiment 2>
In a second embodiment of the present invention, an example of a configuration is described in which the accuracy of the reconstructed spectrum is improved by changing the internal structure of the macrocell from that of the first embodiment, while using the same device configuration ( FIG. 1 ) and chip structure ( FIG. 2 ) as the first embodiment.

Figure 8 is a top view showing the configuration of a macrocell in this embodiment 2. When reconstructing a spectrum using compressed sensing, the outputs of image sensors corresponding to different optical filters within the macrocell are used. In this case, the more different types of optical filters are installed within the macrocell, the higher the accuracy of the reconstructed spectrum. On the other hand, the more different types of optical filters are installed within the macrocell, the larger the area occupied by the macrocell, and therefore the higher the probability that different spectra will be irradiated within the macrocell. Therefore, in this embodiment 2, the same type of pattern is arranged within the macrocell. Figure 8 shows an example.

By comparing the image sensor outputs corresponding to optical filters of the same type of pattern within the macrocell, it is possible to detect whether a boundary between different spectra exists within the macrocell, and dynamically select the type of optical filter to use for reconstruction based on that boundary. When identifying the boundary, the outputs of filters of the same type are compared, but different types of optical filters are selected to be used for reconstructing the spectrum. In embodiment 2, even if there are multiple patterns of the same type within the macrocell and the spectra of the incident light irradiated onto them are approximately the same, one of those patterns of the same type is selected when reconstructing the spectrum, or the average output of filters of the same type is taken and used.

<Embodiment 3>
9 is a configuration diagram of a spectral imaging system according to a third embodiment of the present invention. The spectral imaging system shown in Fig. 9 includes an image sensor 7 having an optical filter 6 made of a nanostructure 5 directly thereon, and an image sensor 8 that does not have an optical filter 6 made of a nanostructure 5.

Image sensor 8 has high resolution because it does not have a macrocell configured with optical filter 6. Image sensor 8 identifies the pattern boundary where the spectra differ as described in embodiment 1, and dynamically selects the optical filter 6 to be used for spectral reconstruction based on that boundary, taking into account the parallax between image sensors 7 and 8. Image sensors 7 and 8 are preferably the same type, but may be different types. Even if an expensive near-infrared image sensor that is sensitive to longer wavelengths than a Si CMOS image sensor is used as image sensor 7, it is possible to prevent system costs from increasing by using an inexpensive Si CMOS image sensor (i.e., a sensor that can detect a narrower range of light wavelengths than image sensor 7) as image sensor 8.

<Fourth Embodiment>
10 is a configuration diagram of a measurement system according to a third embodiment of the present invention. The measurement system is configured by mounting a spectral imaging system 110 and a measuring instrument 120 on a mobile object 100. The spectral imaging system 110 is the spectral imaging system described in the first to third embodiments. The measuring instrument is a device that measures the geographical coordinates of the mobile object 100, and can be configured by, for example, a GPS (Global Positioning System).

As the mobile object 100 moves, it acquires a spectrum for each pixel using the spectroscopic imaging system 110, and stores the spectrum in storage 4, associating it with the geographical coordinates for each position of the mobile object 100. This makes it possible to generate a map that describes the spectrum for each position of the mobile object 100. This process may be performed by the computing device 3 or by a computing device provided in the mobile object 100.

<Modifications of the present invention>
The present invention is not limited to the above-described embodiments and includes various modifications. For example, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment, without departing from the spirit of the invention. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with another configuration.

For ease of understanding, the position, size, range, etc. of each component shown in the drawings, etc. of this specification may not represent the actual position, size, range, etc. Therefore, the present invention is not necessarily limited to the position, size, range, etc. disclosed in the drawings, etc.

In the above embodiment, the range of light wavelengths that image sensor 7 can detect does not necessarily completely encompass the range of light wavelengths that image sensor 8 can detect. In other words, it is sufficient if image sensor 8 can be used to identify the boundary line of a filter into which light of the same spectrum is incident.

In the above embodiment, the optical filters 6 having the same light transmittance for each light wavelength means that the dependence of light transmittance on light wavelength can be considered to be the same, assuming that there are minute differences due to manufacturing variations. For example, if the calculation device 3 receives the results of an image sensor detecting light of the same spectrum that has passed through different optical filters 6 and can treat these results as the same signal level, then it can be said that the optical filters 6 have the same light transmittance for each light wavelength.

In the above embodiments, if there is variation in the detection signal for each detection element of the image sensor, the calculation device 3 may correct that variation before performing the processes described in the above embodiments. If the variation has been corrected by the time the image sensor outputs the detection signal, that detection signal may be used as is.

The present invention relates to a spectral imaging device and a spectral imaging system using the same, which can be used to classify the types of objects and substances, and to determine the amount of moisture contained, in locations such as factory production lines, farms, and forests.

1... incident light 2... chip 3... arithmetic unit 4... storage 5... nanostructure 6... optical filter 7... image sensor 8... image sensor

Claims (11)

A spectroscopic imaging system that detects incident light by dispersing it,
a first image sensor for detecting the intensity of the incident light;
a macrocell disposed on the first image sensor;
a computing unit that calculates the spectrum of the incident light using the intensity detected by the first image sensor;
Equipped with
the macrocell has a plurality of optical filters each having a different light transmittance for each light wavelength;
the arithmetic unit is configured to calculate a spectrum of the incident light for each of one or more pixels of the first image sensor, using the light transmittance for each light wavelength of each of the optical filters and a result of detection by the first image sensor of the intensity of the incident light that has passed through each of the optical filters;
the calculation device determines whether the incident light beams incident on the optical filters have the same spectrum;
The computing device calculates the spectrum of the incident light using the optical filters that have been determined to have the same spectrum as each other. the arithmetic unit identifies a boundary between the optical filter onto which the incident light having the same spectrum is incident and the optical filter onto which the incident light having a different spectrum is incident by comparing the intensities of the incident light that has passed through two or more of the optical filters having the same light transmittance for each light wavelength;
The spectral imaging system according to claim 1 , wherein the calculation device determines the optical filter having the same spectrum as the incident light based on the identified boundary. The computing device divides the optical filters into one or more groups into which the incident light having the same spectrum is incident based on the identified boundary;
The spectral imaging system according to claim 2, wherein the arithmetic unit calculates the spectrum of the incident light for each spectrum possessed by the incident light by using the optical filters belonging to each of the groups to calculate the spectrum of the incident light for each of the groups. The spectroscopic imaging system according to claim 1, wherein the arithmetic unit calculates the spectrum of the incident light using the results of detection by the first image sensor of the intensity of the incident light that has passed through each of the plurality of optical filters, each having a different light transmittance for each light wavelength. the spectroscopic imaging system comprises a plurality of the macrocells;
At least one of the plurality of macrocells has, as the optical filter,
a plurality of different optical filters having different optical transmittances for each optical wavelength;
a plurality of homogeneous optical filters having the same light transmittance for each light wavelength;
and
the calculation device calculates the spectrum of the incident light using the plurality of different optical filters belonging to the same macrocell;
The spectroscopic imaging system according to claim 2 , wherein the arithmetic unit identifies the boundary by comparing the intensities of the incident light that has passed through a plurality of the same optical filters that belong to the same macrocell. The spectral imaging system according to claim 5, characterized in that, when the macrocell has a plurality of the heterogeneous optical filters but does not have a plurality of the homogeneous optical filters, the calculation device identifies the boundary using the optical filters in each of two adjacent macrocells that have the same light transmittance for each light wavelength. the calculation device selects one of the intensities of the incident light that have passed through the plurality of optical filters of the same type that belong to the same macrocell, or calculates an average of the intensities of the incident light;
The spectroscopic imaging system according to claim 6, wherein the calculation device calculates the spectrum of the incident light using the selected intensity or the average as the intensity of the incident light that has passed through the homogeneous optical filter. the spectroscopic imaging system further comprises a second image sensor;
the second image sensor does not have the macrocell;
The spectral imaging system according to claim 2 , wherein the arithmetic unit determines the boundary using the incident light incident on the second image sensor. 9. The spectral imaging system according to claim 8, wherein the range of light wavelengths that the second image sensor can detect is narrower than the range of light wavelengths that the first image sensor can detect. the computing device creates an image of the surrounding space of the spectroscopic imaging system using the result of the first image sensor detecting the incident light;
2. The spectroscopic imaging system according to claim 1, wherein the computing device generates data describing the calculated spectrum for each of one or more pixels of the image. A moving object equipped with the spectral imaging system according to any one of claims 1 to 10.
a measuring instrument for measuring the geographic coordinates of the moving object;
Equipped with
The mobile object acquires the spectrum while moving, and stores the spectrum in association with geographical coordinates for each location, thereby generating a map describing the spectrum for each location.