JP7760904B2 - Spectroscopic imaging method, spectroscopic imaging device, and computer program - Google Patents

Description

The present invention relates to a spectroscopic imaging method, a spectroscopic imaging device, and a computer program.

Patent Document 1 discloses the configuration of a spectroscopic camera equipped with a wavelength-tunable optical filter that functions as a spectroscopic imaging device. A spectroscopic imaging method using a spectroscopic imaging device can obtain a spectral spectrum by, for example, performing calibration using a white tile and then capturing images of all wavelengths over a specified exposure time.

Japanese Patent Application Laid-Open No. 2019-45599

However, when capturing images over a relatively wide field of view, it is difficult to place white tiles in the same position as the object across the wide field of view. Furthermore, to obtain the optimal spectral spectrum, i.e., the captured image, trial and error is required while changing the exposure time, or all wavelengths are captured with a fixed exposure time, which can result in poor image quality and a long capture time. Therefore, there is a demand for a simple and fast spectral capture method.

The spectroscopic imaging method includes the steps of: setting wavelengths for spectroscopic imaging of an object; placing the object and a white reference within an imaging area where the object will be imaged; identifying the position of the white reference and setting it as a reference area; taking preliminary images of the reference area at multiple wavelengths with a predetermined exposure time to obtain intensity values for each wavelength; calculating the exposure time for actual imaging based on the maximum intensity value obtained in the preliminary images and the target intensity value for actual imaging; taking actual images of the object using the exposure time for actual imaging; generating a processed image based on the intensity spectrum of all pixels in the image of the object relative to the intensity spectrum of the reference area; and displaying the processed image.

The spectroscopic imaging device includes a spectral filter, an optical sensor that receives light transmitted through the spectral filter, and a control unit that controls the transmission wavelength of the spectral filter. The control unit performs processing including setting the wavelength for spectroscopic imaging of the object, placing the object and a white reference within an imaging area where the object will be imaged, identifying the position of the white reference and setting it as a reference area, performing preliminary imaging of the reference area for multiple wavelengths at a predetermined exposure time to obtain intensity values for each wavelength, calculating the exposure time for actual imaging based on the maximum intensity value obtained in the preliminary imaging and the target intensity value for actual imaging, performing actual imaging of the object using the exposure time for actual imaging, generating a processed image based on the intensity spectrum of all pixels in the image of the object relative to the intensity spectrum of the reference area, and displaying the processed image.

The computer program generates a processed image and includes the following steps: setting wavelengths for spectroscopic photography of an object; placing the object and a white reference within an imaging area where the object will be imaged; identifying the position of the white reference and setting it as a reference area; taking preliminary images of the reference area at multiple wavelengths with a predetermined exposure time to obtain intensity values for each wavelength; calculating the exposure time for the actual imaging based on the maximum intensity value obtained in the preliminary imaging and the target intensity value for the actual imaging; taking actual images of the object using the exposure time for the actual imaging; generating a processed image based on the intensity spectrum of all pixels in the captured image of the object relative to the intensity spectrum of the reference area; displaying the processed image; and storing the processed image in a memory unit.

FIG. 1 is a schematic diagram showing the configuration of a spectroscopic imaging device. 1 is a flowchart showing a spectroscopic imaging method. FIG. 2 is a schematic diagram showing the configuration of a display unit. 1 is a graph illustrating a part of a spectroscopic imaging method. FIG. 2 is a diagram illustrating a part of a spectroscopic imaging method. FIG. 2 is a diagram illustrating a part of a spectroscopic imaging method. FIG. 2 is a diagram illustrating a part of a spectroscopic imaging method. FIG. 2 is a diagram illustrating a part of a spectroscopic imaging method. FIG. 2 is a diagram illustrating a part of a spectroscopic imaging method.

The configuration of the spectroscopic imaging device 100 will be explained with reference to Figure 1.

As shown in FIG. 1, the spectroscopic imaging device 100 is capable of, for example, capturing an image of an object P for each wavelength and obtaining a spectroscopic spectrum from the captured image. The spectroscopic imaging device 100 includes, for example, an imaging unit 10, a processing unit 20, and a display unit 30.

The imaging unit 10 includes an incident optical system 40 onto which light from the object P is incident, a bandpass filter (BPF) 50, a spectral filter 60 that spectrally separates the incident light, and an optical sensor 70 that captures the light spectrally separated by the spectral filter 60.

The incident optical system 40 is equipped with, for example, an autofocus mechanism. The incident optical system 40 is also configured, for example, as a telecentric optical system, and guides the light 41 of the first wavelength number to the spectral filter 60 so that the optical axis and the chief ray are parallel or approximately parallel.

The spectral filter 60 is, for example, a wavelength-selective filter, and uses a Fabry-Perot filter that can change the transmission wavelength band.

The spectral filter 60 is a wavelength-tunable interference filter that includes a pair of substrates 61, 62, a pair of opposing reflective films 63, 64, and a gap changer 65 that can change the gap dimension between these reflective films 63, 64. The gap changer 65 is configured, for example, by an electrostatic actuator. The wavelength-tunable interference filter is also called an etalon. This spectral filter 60 is placed on the optical path of light incident on the optical sensor 70.

The spectral filter 60 changes the gap dimension of the reflective films 63 and 64 by changing the voltage applied to the gap change unit 65 under the control of the control unit 21 that constitutes the processing unit 20, thereby changing the output wavelength λi (i = 1, 2, ..., N), which is the wavelength of the light that passes through the reflective films 63 and 64.

The optical sensor 70 is, for example, a CCD (Charge Coupled Device), and is an imaging device that photoelectrically converts the light 42 of the second wavelength that has passed through the spectral filter 60 to obtain an electrical signal representing the object P.

The spectroscopic imaging device 100 sequentially receives instructions for multiple measurement bands (multi-band) from the control unit 21 at the spectroscopic filter 60, which sequentially changes the transmission wavelength range of the spectroscopic filter 60. In this way, the spectroscopic imaging device 100 captures images of the object P with sensitivity to multiple wavelength bands.

The processing unit 20 includes a control unit 21, a memory unit 22, an arithmetic processing unit 23, an imaging wavelength setting unit 24, a reference area setting unit 25, an exposure time setting unit 26, a number of images to be captured setting unit 27, and an image generation unit 28. The processing unit 20 also performs various processes by executing a control program as a computer program.

The control unit 21 is configured with one or more processors and, for example, controls the operation of the spectroscopic imaging device 100 by operating in accordance with a control program stored in the memory unit 22.

The storage unit 22 is configured with memories such as RAM (Random Access Memory) and ROM (Read Only Memory). The RAM is used for temporary storage of various data, and the ROM stores control programs and control data for controlling the operation of the spectroscopic imaging device 100. The storage unit 22 also stores images captured by the optical sensor 70 for each transmitted wavelength, as well as processed images.

The calculation processing unit 23 executes various processes using the data and parameters stored in the memory unit 22. The calculation processing unit 23 also performs various calculation processes based on information from the captured image.

The imaging wavelength setting unit 24 sets the imaging wavelength that will allow the optimal image to be captured based on the object P selected by the user. Specifically, for example, when capturing images of NDVI (Normalized Difference Vegetation Index: an index that indicates the distribution and activity of vegetation), wavelengths of 680 nm and 800 nm are set. Furthermore, for example, when capturing images of moisture, a wavelength of around 970 nm is set. In this way, the imaging wavelength setting unit 24 reads and sets the optimal wavelength from the memory unit 22 according to the object P to be captured. Furthermore, when capturing images of the entire wavelength range, the imaging wavelength setting unit 24 sets wavelengths of, for example, 620 nm to 1100 nm.

The reference area setting unit 25 sets all or part of a white reference 32 (see Figure 3), such as a white tile, specified by the user as the reference area 33. Specifically, a white tile, for example, approximately 20 cm x 20 cm, is placed within the shooting area 31 (see Figure 3) to be photographed, and a portion or area of the white tile is determined, thereby setting that portion as the white reference 32 or reference area 33.

The exposure time setting unit 26 sets the exposure time for each wavelength to be photographed according to the object P. Specifically, the calculation processing unit 23 calculates the exposure time for the actual photographing based on the maximum intensity value obtained in the preliminary photographing and the target intensity value for the actual photographing, and the exposure time setting unit 26 sets the calculated value as the exposure time.

The number-of-images setting unit 27 sets the number of frames to be captured for each wavelength during actual imaging, i.e., the number of images to be captured. Specifically, the number-of-images setting unit 27 fixes the exposure time for each wavelength and increases the number of frames to be captured for each wavelength to generate an optimal frame number table.

The image generation unit 28 generates a processed image from the captured image taken under the above conditions. Specifically, the processed image is generated based on the intensity spectrum of all pixels in the captured image of the object P relative to the intensity spectrum of the reference area 33.

The display unit 30 is, for example, a liquid crystal display or organic EL display, and has both display and input functions. The display unit 30 displays images of the object P to be photographed, white tiles, and processed images.

The input function is configured, for example, by a touch panel provided on the surface of the display unit 30. However, input is not limited to a touch panel; input may also be made using a mouse, directional keys, numeric keys, etc.

Next, a spectroscopic imaging method using the spectroscopic imaging device 100 of this embodiment will be described with reference to Figures 2 to 9. Note that steps S11 to S14 are preparatory steps for performing preliminary imaging, and steps S15 to S17 are actual steps for performing actual imaging.

As shown in FIG. 2, in step S11, the imaging wavelength setting unit 24 sets the imaging wavelength for the object P. Specifically, as shown in FIG. 3, the user selects the type of object P on the display unit 30. In this case, moisture (970 nm) is selected as a pre-registered detection item. By selecting the item, the imaging wavelength is automatically set.

In this embodiment, by selecting an item, the process can be carried out automatically from preliminary shooting to actual shooting. As shown in Figure 3, the user can input the wavelength they wish to shoot, or the exposure time for shooting. Furthermore, if they wish to shoot across the entire wavelength range, they can select 620 nm to 1100 nm.

In step S12, the reference area setting unit 25 sets the reference area 33. First, as shown in FIG. 3, the user places the object P to be photographed and the white reference 32 within the photographing area 31. Next, for example, the user touches the white reference 32 displayed on the touch panel to identify the white reference 32. Note that the white reference 32 may be identified by mouse operation, or the reference area 33 may be surrounded by mouse operation.

In step S13, the image capture unit 10 performs preliminary image capture. Specifically, when the user touches the image capture start button 34 on the display unit 30, the image capture unit 10 performs preliminary image capture at a wavelength set with a short, fixed exposure time so as not to cause intensity saturation, specifically so as not to cause overexposure due to excessive brightness, and acquires the intensity of the reference area 33.

In step S14, the exposure time setting unit 26 sets the exposure time for the actual image capture so that the wavelength with the maximum intensity obtained in the preliminary image capture will have the targeted intensity value. Specifically, as shown in FIG. 4, the exposure time setting unit 26 causes the calculation processing unit 23 to calculate the exposure time for the actual image capture based on the maximum intensity value (S0max) obtained in the preliminary image capture and the targeted intensity value (S1) for the actual image capture. In other words, the exposure time is set so that the image will basically be bright at all wavelengths.

For example, if the exposure time is constant and the number of images taken for each wavelength is changed, the exposure time and number of images taken can be calculated using the following formulas (1) and (2). The exposure time for the actual image capture is exp1. The exposure time for the preliminary image capture is exp0. The target intensity value is S1. The maximum intensity value for the preliminary image capture is S0max. Furthermore, the number of images taken for the actual image capture is n(λ). The intensity value for each wavelength in the preliminary image capture is S0(λ).

In other words, the method for calculating the exposure time for actual shooting is to set the target intensity value for the wavelength with the maximum intensity value in preliminary shooting of reference area 33 based on (exposure time for preliminary shooting) x (target intensity value for actual shooting / maximum intensity value in preliminary shooting).

For other wavelengths, the intensity value will be smaller at the same exposure time, so an optimal frame number table is automatically generated by increasing the number of images taken for each wavelength (number of frames acquired), assuming the exposure time for the actual image capture.

For example, if one image is taken at a time and the exposure time for each wavelength is changed, the exposure time and number of images taken can be calculated using the following equations (3) and (4). The exposure time for the actual image capture is assumed to be exp1(λ).

In other words, the exposure time for the actual shooting is calculated by setting it based on (exposure time for preliminary shooting) x (target intensity value for actual shooting / maximum intensity value for preliminary shooting) so that the targeted intensity value is achieved for each wavelength captured in the preliminary shooting.

In this way, the exposure time setting unit 26 uses the spectral data shown in Figure 4 to determine the exposure time required for actual shooting, whether to increase the number of shots taken and average them out because the wavelength λ is too dark, and so on. The exposure time setting unit 26 also sets the exposure time for actual shooting so that the intensity of the reference area 33 does not saturate. Once the preliminary shooting is completed by step S14, the process moves to step S15 for actual shooting.

In step S15, the photographing unit 10 performs actual photographing. Specifically, the control unit 21 causes the optical sensor 70 to photograph the object P for each wavelength using the exposure time set in the above process. This allows the processing unit 20 to obtain the reflectance spectrum of each pixel.

In step S16, the image generation unit 28 causes the calculation processing unit 23 to generate an image. Specifically, a processed image is generated based on the intensity spectrum of all pixels in the captured image of the object P relative to the intensity spectrum of the reference area 33.

In step S17, the processed image is displayed. Here, we will explain an example of image processing using second-order differentiation to image the moisture 202 contained in an object 201, such as a shishito pepper, as shown in Figures 5 to 9. Figure 5 shows a sample image displayed in the imaging area 31 to be photographed. Second-order differentiation is a process commonly performed to make it easier to detect the peak wavelength of light absorption by a substance. When using second-order differentiation, images are taken at a minimum of three wavelengths (see Figures 6, 7, and 8).

As described above, actual shooting is performed based on the set exposure time and frame table, and multiple frames from the captured image are averaged. This is then used to generate and display a second-order differential image. If the three wavelengths are λ0, λ1, and λ2 in ascending order, the second-order differential image at λ1 can be calculated as the λ0 (920 nm) image + λ2 (1020 nm) image - 2 x λ1 (970 nm) image.

The NDVI image can be calculated by (800 nm image - 680 nm image) / (800 nm image + 680 nm image).

As a result, the moisture 202 contained in the object 201 can be detected, and a processed image representing the moisture 202, i.e., the characteristic parts corresponding to the substance, can be displayed on the display unit 30.

As described above, the spectroscopic imaging method of this embodiment includes the steps of setting the wavelengths for spectroscopic imaging of the object P, placing the object P and the white reference 32 within the imaging area 31 where the object P will be imaged, identifying the position of the white reference 32 and setting it as the reference area 33, performing preliminary imaging of the reference area 33 for multiple wavelengths at a predetermined exposure time to obtain intensity values for each wavelength, calculating the exposure time for the actual imaging based on the maximum intensity value S0max obtained in the preliminary imaging and the target intensity value S1 for the actual imaging, performing actual imaging of the object P using the exposure time for the actual imaging, generating a processed image based on the intensity spectrum of all pixels in the captured image of the object P relative to the intensity spectrum of the reference area 33, and displaying the processed image.

With this method, by identifying the white reference 32 and setting the reference area 33 before the actual shooting, calibration can be performed even when the white reference 32 is difficult to recognize, such as when shooting with a wide field of view. Furthermore, because the exposure time for the actual shooting is calculated based on the preliminary shooting, the object P can be photographed under optimal shooting conditions (e.g., shooting wavelength, exposure time for each wavelength, etc.) during the actual shooting. Therefore, compared to repeated trial and error or shooting all wavelengths with a fixed exposure time, shooting can be done easily and quickly. In addition, high-quality processed images can be obtained.

Furthermore, in the spectroscopic imaging method of this embodiment, the step of calculating the exposure time for actual imaging sets the wavelength with the maximum intensity value in preliminary imaging of the reference area 33 based on (exposure time for preliminary imaging) x (target intensity value for actual imaging / maximum intensity value in preliminary imaging) so that the targeted intensity value is achieved. For other wavelengths, it is preferable to generate an optimal frame number table in which the number of imaging frames is increased for each wavelength, based on the exposure time for actual imaging. According to this method, the exposure time for actual imaging is calculated from the preliminary imaging based on the above formula, so that the object P can be imaged under optimal imaging conditions in the actual imaging. Specifically, the exposure time for each wavelength is fixed, and the number of imaging frames for each wavelength is changed.

Furthermore, in the spectroscopic imaging method of this embodiment, the step of calculating the exposure time for actual imaging is preferably set based on (exposure time for preliminary imaging) x (target intensity value for actual imaging/maximum intensity value for preliminary imaging) so that the targeted intensity value is achieved for each wavelength captured in the preliminary imaging. According to this method, the exposure time for actual imaging is calculated from the preliminary imaging based on the above formula, so that the object P can be captured under optimal imaging conditions in the actual imaging. Specifically, the number of imaging frames for each wavelength is one, and the exposure time for each wavelength is changed.

The spectroscopic imaging device 100 of this embodiment includes a spectral filter 60, an optical sensor 70 that receives light transmitted through the spectral filter 60, and a control unit 21 that controls the transmission wavelength of the spectral filter 60. The control unit 21 performs processes including setting the wavelength for spectroscopic imaging of the object P, placing the object P and a white reference 32 within an imaging area 31 where the object P will be imaged, identifying the position of the white reference 32 and setting it as a reference area 33, performing preliminary imaging of the reference area 33 for multiple wavelengths at a predetermined exposure time to obtain intensity values for each wavelength, calculating the exposure time for the actual imaging based on the maximum intensity value obtained in the preliminary imaging and the target intensity value for the actual imaging, performing actual imaging of the object P using the exposure time for the actual imaging, generating a processed image based on the intensity spectrum of all pixels in the captured image of the object P relative to the intensity spectrum of the reference area 33, and displaying the processed image.

With this configuration, by identifying the white reference 32 and setting the reference area 33 before actual shooting, calibration can be performed even in cases where it is difficult to place white tiles in the same positions as the object across a wide field of view, such as when shooting with a wide field of view. Furthermore, because the exposure time for actual shooting is calculated based on the preliminary shooting, the object P can be photographed under optimal shooting conditions (e.g., shooting wavelength, exposure time for each wavelength, etc.) during actual shooting. Therefore, compared to repeated trial and error or shooting all wavelengths with a fixed exposure time, shooting can be done easily and quickly. In addition, high-quality processed images can be obtained.

The computer program of this embodiment is a computer program for generating a processed image, and includes the steps of: setting wavelengths for spectroscopic imaging of object P; placing object P and a white reference 32 within an imaging area 31 in which object P will be imaged; identifying the position of the white reference 32 and setting it as a reference area 33; performing preliminary imaging of the reference area 33 for multiple wavelengths at a predetermined exposure time to obtain intensity values for each wavelength; calculating the exposure time for the actual imaging based on the maximum intensity value obtained in the preliminary imaging and the target intensity value for the actual imaging; performing actual imaging of object P using the exposure time for the actual imaging; generating a processed image based on the intensity spectrum of all pixels in the captured image of object P relative to the intensity spectrum of the reference area 33; displaying the processed image; and storing the processed image in the memory unit 22.

This computer program allows calibration to be performed even when the white reference 32 is difficult to recognize, such as when photographing a wide field of view, by identifying the white reference 32 and setting the reference area 33 before the actual photographing. Furthermore, since the exposure time for the actual photographing is calculated based on the preliminary photographing, the object P can be photographed under optimal photographing conditions (e.g., photographing wavelength, exposure time for each wavelength, etc.) during the actual photographing. Therefore, compared to repeated trial and error or photographing all wavelengths with a fixed exposure time, photographing can be performed easily and in a short time. In addition, high-quality processed images can be obtained.

Furthermore, in the computer program of this embodiment, displaying the processed image preferably involves selecting a substance contained in the object P and displaying characteristic portions corresponding to the substance. This computer program extracts and displays characteristic portions corresponding to the substance, making it easier to visually recognize the analysis results.

Below, we will explain some variations of the above-mentioned embodiment.

As described above, the spectroscopic imaging device 100 includes all of the imaging unit 10, processing unit 20, and display unit 30, but this is not limited to this. At least some of the imaging unit 10, processing unit 20, and display unit 30 may be connected to the spectroscopic imaging device 100 via a cable, or may be connected to the spectroscopic imaging device 100 wirelessly.

10...imaging unit, 20...processing unit, 21...control unit, 22...storage unit, 23...arithmetic processing unit, 24...imaging wavelength setting unit, 25...reference area setting unit, 26...exposure time setting unit, 27...number of images set unit, 28...image generation unit, 30...display unit, 31...imaging area, 32...white reference, 33...reference area, 34...imaging start button, 40...incident optical system, 41...light of first wavelength number, 42...light of second wavelength number, 50...bandpass filter, 60...spectral filter, 61, 62...pair of substrates, 63, 64...pair of reflective films, 65...gap changing unit, 70...optical sensor, 100...spectrophotography device, 201...object, 202...moisture.

Claims (6)

setting a wavelength for spectrophotography of the object;
placing the object and a white reference within an imaging area in which the object is imaged;
Identifying the position of the white reference and setting it as a reference area;
a step of taking a preliminary image of the reference area for each of a plurality of wavelengths at a predetermined exposure time and acquiring an intensity value for each wavelength;
calculating an exposure time for actual imaging based on the intensity values for each of the plurality of wavelengths obtained by the preliminary imaging and a target intensity value for actual imaging;
performing an actual photograph of the object for each of the plurality of wavelengths using an exposure time for the actual photograph;
generating a first processed image for each of the plurality of wavelengths based on the intensity spectrum of all pixels of the captured image of the object relative to the intensity spectrum of the reference area;
displaying a second processed image that expresses a characteristic portion of the object using the first processed images for each of the plurality of wavelengths ;
A spectroscopic imaging method comprising: The spectroscopic imaging method according to claim 1,
The step of calculating the exposure time for actual shooting includes:
In the preliminary photographing of the reference area, a wavelength at which the intensity value is maximum is set based on (exposure time of the preliminary photographing) × (target intensity value in the actual photographing/maximum intensity value in the preliminary photographing) so as to achieve a targeted intensity value;
For other wavelengths, a spectroscopic imaging method is used to generate an optimum frame number table in which the number of imaging frames is increased for each wavelength, based on the exposure time in the actual imaging. The spectroscopic imaging method according to claim 1,
The step of calculating the exposure time for actual shooting includes:
A spectroscopic photography method in which the intensity values for each wavelength captured in the preliminary photography are set based on (exposure time of the preliminary photography) x (target intensity value in the actual photography/intensity value for each wavelength in the preliminary photography) so as to achieve the targeted intensity value. A spectral filter;
a photosensor that receives light transmitted through the spectral filter;
a control unit that controls the transmission wavelength of the spectral filter;
A spectroscopic imaging device comprising:
The control unit
Setting a wavelength for spectroscopic imaging of the object;
placing the object and a white reference within an imaging area in which the object is imaged;
Identifying the position of the white reference and setting it as a reference area;
taking a preliminary photograph of the reference area for each of a plurality of wavelengths at a predetermined exposure time, and acquiring an intensity value for each wavelength;
calculating an exposure time for actual imaging based on the intensity values for each of the plurality of wavelengths obtained by the preliminary imaging and a target intensity value for actual imaging;
performing actual photography of the object for each of the plurality of wavelengths using an exposure time for the actual photography;
generating a first processed image for each of the plurality of wavelengths based on the intensity spectrum of all pixels of the captured image of the object relative to the intensity spectrum of the reference area;
displaying a second processed image that expresses a characteristic portion of the object using the first processed images for each of the plurality of wavelengths ;
A spectroscopic imaging device that performs processing including the steps of: 1. A computer program for generating a processed image, comprising:
Setting a wavelength for spectroscopic imaging of the object;
placing the object and a white reference within an imaging area in which the object is imaged;
Identifying the position of the white reference and setting it as a reference area;
taking a preliminary photograph of the reference area for each of a plurality of wavelengths at a predetermined exposure time, and acquiring an intensity value for each wavelength;
calculating an exposure time for actual imaging based on the intensity values for each of the plurality of wavelengths obtained by the preliminary imaging and a target intensity value for actual imaging;
performing actual photography of the object for each of the plurality of wavelengths using an exposure time for the actual photography;
generating a first processed image for each of the plurality of wavelengths based on the intensity spectrum of all pixels of the captured image of the object relative to the intensity spectrum of the reference area;
displaying a second processed image that expresses a characteristic portion of the object using the first processed images for each of the plurality of wavelengths ;
storing the first processed image or the second processed image in a storage unit;
A computer program comprising: 6. A computer program according to claim 5,
A computer program for displaying the second processed image, the computer program selecting a substance contained in the object and displaying a characteristic portion corresponding to the substance.