JP5929338B2 - Imaging device, spectral information creation method - Google Patents

Description

  The present invention relates to an image pickup apparatus including an image pickup device that converts light into an electrical signal, a region division filter that is disposed on the light input side of the image pickup device, and an image processing unit that processes an output signal of the image pickup device.

  Attempts have been made to acquire not only two-dimensional image information but also spectral information (spectral information) for the imaging system. Spectroscopes are widely used as devices for acquiring spectral information. Since the spectroscope acquires the light incident on the light receiving element by using a prism or a diffraction grating to obtain a plurality of wavelength components, spatial information (light incident direction) cannot be obtained. On the other hand, in recent years, a technique using a two-dimensional distribution of spectrum information has attracted attention, and an image capturing apparatus capable of acquiring spectrum information using various methods is provided (for example, Patent Document 1). -3)).

  In Patent Document 1, a wavelength selection filter (bandpass filter, low-pass / high-pass filter) is used to switch the wavelength selection filter, arrange the wavelength selection filter in a minute region, or provide a liquid crystal wavelength tunable filter as the wavelength selection filter. There is described a spectroscopic camera head that acquires light intensity of a desired wavelength by, for example, dynamically switching a transmission wavelength.

  In Patent Document 2, spectral information is obtained by dividing an incident light beam into two optical paths (or polarization components) and Fourier-transforming a detected intensity signal obtained by giving a phase difference to one of them to cause interference. A spectroscopic device is disclosed. When Fourier spectroscopy is used in this way, the detection position is scanned two-dimensionally (or one-dimensionally), or different phase differences are dynamically given to two orthogonal polarization components to acquire image data in time series. There is a need to.

  In Patent Document 3, the spectrum of incident light is spatially separated to acquire one-dimensional spatial information and spectral information with a two-dimensional image sensor, and further, the one-dimensional spatial information is scanned in a direction orthogonal thereto. Techniques for obtaining dimensional (image) spectral information are described.

  However, in the method using the wavelength selective filter disclosed in Patent Document 1, it is necessary to use a narrow-band filter in order to obtain a spectrum, the signal detection intensity is lowered, a long exposure time is required, and However, since a strong incident angle characteristic appears, there is a problem that parallel light must be incident. Further, since it is necessary to switch the wavelength selection filter a plurality of times in order to acquire signals at a plurality of wavelengths, it is difficult to perform high-speed imaging in real time. Furthermore, since an active device such as a liquid crystal tunable filter is required, there are more restrictions in terms of reliability such as weather resistance and vibration resistance compared to a conventional luminance camera.

  Further, in the method using Fourier spectroscopy described in Patent Document 2, in order to divide into two optical paths and give a phase difference, it is necessary to create parallel light beams as much as possible, and a narrow slit and a small opening are required. However, there is a problem that the angle dependency is not large. Similarly to the problem of Patent Document 1, since an active device such as a liquid crystal phase shifter is required, there are more restrictions in terms of reliability such as weather resistance and vibration resistance compared to conventional luminance cameras.

  Further, in the method using the dispersion element described in Patent Document 3, as in the case of using Fourier spectroscopy, there is a problem that it is necessary to transmit a narrow slit in order to obtain spectral resolution, and the light amount is reduced. is there. In addition, a scanning mechanism is required, and measurement time is required, and it is difficult to reduce the size of the apparatus.

  In view of the above problems, an object of the present invention is to provide an imaging apparatus that can suppress a decrease in light amount and can detect spectral information with high accuracy without using a specific active device.

The present invention includes an image pickup device that converts light into an electrical signal, an area division filter that is disposed on the light input side of the image pickup device, and an image processing unit that processes an output signal of the image pickup device. In the apparatus, the region dividing filter has two or more regions that cause a luminance difference between adjacent unit pixels due to different transmittances for light of a specific wavelength, and is arranged in a predetermined pattern. Light generated from the first output signal output from each image sensor for the specific wavelength and the second output signal output from the image sensor for the second specific wavelength is incident on the image sensor. First reference data for extracting the first specific wavelength component and the second specific wavelength component, the second output signal output by each image sensor for the second specific wavelength, and the third specific Each image sensor outputs with respect to wavelength Second reference data for extracting a second specific wavelength component and a third specific wavelength component when light is incident on the imaging device, and generated from the third output signal, and the image The processing unit generates the first specific wavelength component and the second specific wavelength component by applying the first reference data to each output signal of the unit pixel when light is incident on the imaging device, A second specific wavelength component and a third specific wavelength component are generated by applying the second reference data, and the first specific wavelength component and the second specific wavelength generated by the first reference data A component, and the second specific wavelength component and the third specific wavelength component generated by the second reference data are synthesized .

  It is possible to provide an imaging apparatus that can suppress a decrease in the amount of light and can detect spectral information with high accuracy without using a specific active device.

It is an example of the schematic block diagram of the imaging device of this embodiment. It is an example of the figure explaining the structure of an area | region division filter. It is another example of the figure explaining the structure of an area | region division filter. It is another example of the figure explaining the structure of an area | region division filter. It is another example of the figure explaining the structure of an area | region division filter. It is another example of the figure explaining the structure of an area | region division filter. FIG. 3 is an example of a diagram schematically illustrating specific wavelengths λ1 and λ2 and reference data 1; It is an example of the figure explaining input light (lambda) 1 and output spectral information. It is an example of the figure explaining input light (lambda) 2 and output spectral information. It is an example of the figure explaining input light (lambda) 3 and output spectral information. FIG. 5 is an example of a diagram illustrating output spectral information when an input light wavelength λ1 and an input light wavelength λ2 obtained by overlapping two specific wavelengths used for calibration are input. FIG. 4 is an example of a diagram schematically illustrating specific wavelengths λ1, λ2, and λ3 and reference data 1 and 2; It is an example of the figure explaining the output spectral information by input light (lambda) 1 and the reference data 1. FIG. It is an example of the figure explaining the output spectral information by input light (lambda) 2 and the reference data 2. FIG. It is an example of the figure explaining the output spectral information by input light (lambda) 3 and the reference data 1 and 2. FIG. FIG. 5 is an example of a diagram illustrating output spectral information when an input light wavelength λ1 and an input light wavelength λ2 obtained by overlapping two specific wavelengths used for calibration are input. It is an example of the figure which shows the RAW image (part) of the image pick-up element with respect to the specific wavelength in a calibration. It is an example of the figure which shows the histogram of the sensor pixel value (luminance) with respect to a single wavelength. It is an example of the figure which shows the wavelength dependence of contrast which plotted contrast I (x) with respect to the wavelength. It is an example of the figure which shows the spectrum of the produced | generated spectrum light. It is a figure which shows an example of the spectral image by the calibration data. It is a figure which shows an example of the spectral image by the calibration data. It is an example of the figure which synthesize | combined two spectral information outputs.

  Hereinafter, modes for carrying out the present invention will be described.

[Configuration example]
FIG. 1 shows an example of a schematic configuration diagram of an imaging apparatus according to the present embodiment. The image pickup apparatus 100 converts the image pickup lens 11, the image pickup element 13, the area division filter 12 disposed closer to the image pickup lens 11 than the image pickup element 13, and a signal from the image pickup element 13 into output spectral information and outputs the converted image. Part 14. Note that the imaging result of the imaging device 13 is referred to as spectral image information, and the processing result of the image processing unit 14 is referred to as output spectral information.

  The angle of view (focal length), F-number, and the like of the imaging lens 11 vary depending on the application, and the specifications of these lenses affect the sensor sensitivity, the brightness of the photographic subject, and the subject depth. The imaging apparatus 100 according to the present embodiment can cope with the imaging lens 11 having various specifications by performing calibration in image processing to be described later.

  Examples of the imaging lens 11 include industrial use, in-vehicle use (LDW (Lane Departure Warning), ADB (detecting the position of the preceding vehicle and the oncoming vehicle from the image data, and preventing the driver of these vehicles from being dazzled). The lens (viewing angle ± 20 to 40 °, F1.4 to 2.0) generally used in the above-mentioned function). Further, an enlargement lens such as a macro lens used in measurement applications can also be used. However, the application of the imaging apparatus 100 is not limited to these.

  The imaging device 13 is an electronic component that converts an optical signal into an electrical signal, and a general image sensor such as a CCD or a CMOS can be used.

  The image processing unit 14 has a plurality of reference data corresponding to two or more specific wavelengths in advance. Using these reference data, a difference image of adjacent pixels of the image sensor 13 is generated and output. A method for generating reference data will be described later. The image processing unit 14 includes a memory, an arithmetic circuit, an input / output unit, and the like, and converts luminance information received by each pixel of the image sensor 13 through the region division filter 12 into output spectral information.

  The region dividing filter 12 has two or more regions having different transmittances that cause a luminance difference in adjacent pixels of the image sensor 13 corresponding to a specific wavelength.

[Region division filter]
FIG. 2 is an example of a diagram illustrating the configuration of the area division filter 12. The area division filter 12 in FIG. 2 has a structure in which four areas having different light transmittances according to wavelengths are arranged vertically and horizontally. In the figure, the four regions are arranged so that the vertices are in contact with each other. These four areas are referred to as one block area. Each region has a different transmittance of light of wavelengths λ1, λ2, λ3, and λ4, and specifically, selectively transmits light of each wavelength.

  One region of the region dividing filter 12 is arranged corresponding to one pixel of the image sensor 13. For this reason, the image sensor 13 into which the light that has passed through the block region is incident acquires four different pieces of spectral information. The imaging element 13 into which the light that has passed through the block region is incident acquires it as one piece of spectral image information.

  In FIG. 2, one region is arranged so as to cover the whole of one pixel except for a deviation that may occur in manufacturing, but one region covers a part of one pixel. May be arranged. Further, it is not limited that one region is arranged so as to cover the whole of two or more pixels. That is, one area can cover an integer number of elements.

  FIG. 3 is another example of a diagram illustrating the configuration of the area division filter 12. FIG. 3 has a structure in which two regions having different light transmittances depending on the wavelength are arranged vertically and horizontally. The two regions have different transmittances at two wavelengths λ1 and λ2. Two rectangular regions of the region dividing filter 12 constitute one block region, and one region of the region dividing filter 12 is arranged corresponding to one pixel of the image sensor 13.

  The imaging element 13 into which the light that has passed through the block region is incident acquires it as one piece of spectral image information. In FIG. 3, spectral image information 1 and 2 are shown corresponding to two block areas.

  In FIG. 3, the regions corresponding to the transmittance of λ1 are arranged in a row in the horizontal direction, the regions corresponding to the transmittance of λ2 are arranged in a row in the horizontal direction, the region corresponding to the transmittance of λ1 and the transmittance of λ2. The areas corresponding to are alternately arranged in the vertical direction. Such an arrangement pattern is referred to as a horizontal stripe pattern.

  FIG. 4 is another example of a diagram illustrating the configuration of the area division filter 12. Similar to FIG. 3, it has a structure in which two regions having different light transmittances according to wavelengths are arranged vertically and horizontally. In FIG. 4, the areas corresponding to the transmittance of λ1 are arranged in a line in the vertical direction, the areas corresponding to the transmittance of λ2 are arranged in a line in the vertical direction, and the areas corresponding to the transmittance of λ1 and the transmittance of λ2 are arranged. The areas corresponding to are alternately arranged in the left-right direction. Such an arrangement pattern is referred to as a vertical stripe pattern.

  FIG. 5 is another example of a diagram illustrating the configuration of the area division filter 12. Similar to FIG. 3, it has a structure in which two regions having different light transmittances according to wavelengths are arranged vertically and horizontally. In FIG. 5, the region corresponding to the transmittance of λ1 and the region corresponding to the transmittance of λ2 are alternately arranged both vertically and horizontally. In other words, the region corresponding to the transmittance of λ1 and the region corresponding to the transmittance of λ2 are alternately arranged in the horizontal direction, and are also arranged alternately in the vertical direction. Such an arrangement pattern is referred to as a checker pattern.

  FIG. 6 is another example of a diagram illustrating the configuration of the area division filter 12. In FIG. 6, two regions having different light transmittances according to wavelengths are arranged, but the shape of the region is not rectangular. The region corresponding to the transmittance of λ 1 is arranged with an oblique inclination with respect to the column direction and the row direction of the image sensor 13. The same applies to the region corresponding to the transmittance of λ2, and the region corresponding to the transmittance of λ1 and the region corresponding to the transmittance of λ2 form a structure that should be called an oblique stripe. The inclination of the stripe is two vertical dimensions of the pixel with respect to one width dimension of the pixel of the image sensor 13. That is, if the pixel is square, the slope is 2. The inclination is not limited to 2.

  In the area dividing filter 12 as shown in FIG. 6, the area corresponding to the transmittance of λ1 covers the pixels, and the area corresponding to the transmittance of λ2 covers the pixels. Becomes equal. A region of the region dividing filter 12 corresponding to four pixel blocks is a block region. Therefore, the spectral image information output by the four pixels shown in the figure can be used as one spectral image information.

  2 to 6, the pattern pitch is almost the same as the size of each pixel of the image sensor 13. That is, the vertical and horizontal lengths (in the direction in which the transmittance differs with respect to the wavelength) of one region of the region division pattern is approximately the same as each pixel of the image sensor 13.

[Structure of Region Dividing Filter 12]
The area dividing filter 12 includes a portion where a light shielding film such as AL or Cr that blocks or controls light transmission is formed and a portion where it is not formed. Alternatively, light shielding films having different transmittances may be patterned depending on the balance of light control. Here, although a metal film is mentioned as the light shielding film, it may be an organic film such as a resist containing a pigment dye as long as it can block and control light transmission.

  Also, a wire grid polarizer that can be formed by a photolithography process or an interference exposure process, or a photonic crystal polarizer formed by an autocloning method can be used.

  Furthermore, in order to adjust the light transmittance with respect to a specific wavelength, a filter divided into regions using a dielectric multilayer film is preferable, and such a filter is excellent in environmental resistance.

[Reference data]
Generation of reference data for image processing will be described. Generating reference data is called calibration of the image sensor 13. In the calibration, a process of measuring the ratio of each component from the spectral image information when the light having a specific wavelength that has passed through the region dividing filter 12 reaches the image sensor 13 is performed. This is performed for a plurality of lights having specific wavelengths.

  The calibration can be performed manually by a developer or the like, or can be performed by a calibration device (not shown). In the present embodiment, the description will be made assuming that the calibration apparatus performs.

  The calibration device first generates a specific wavelength by irradiating a flat band illumination with uniform brightness to, for example, a dielectric bandpass filter that transmits only the specific wavelength. Light of a specific wavelength is incident on the image sensor 13 to which the area division filter 12 is attached. At least two specific wavelengths λ1 and λ2 are generated. In addition to using a bandpass filter to enter the specific wavelength, other means capable of entering the specific wavelength may be used. For example, a multi-wavelength light source system in which white light is separated and synthesized by a spectroscopic element and a specific wavelength (including an arbitrary spectral shape) is incident can be used.

  The calibration device irradiates the entire image sensor 13 with light of a specific wavelength λ 1 having a uniform intensity, and records output data of each pixel of the image sensor 13 at that time as reference image data 1. Next, similarly, the light of the specific wavelength λ 2 with uniform intensity is irradiated and recorded as the reference image data 2.

  For each pixel, the pixel values themselves of the reference image data 1 and the reference image data 2 can be considered to be proportional to the sensitivity of the λ1 component and λ2 component at the pixel position. , The ratio of the sensitivity of λ1 and λ2 is measured.

  Based on this ratio, a method for generating image data composed of only the wavelength λ1 component and the wavelength λ2 component will be described.

The wavelength λ1 component of the incident light is L (λ1), the wavelength λ2 component is L (λ2), the sensitivity of the wavelength λ1 component at the light receiving pixel i is a (λ1) i, and the sensitivity of the wavelength λ2 component is a (λ2) i. To do. The intensity I _i of light that passes through the region dividing filter 12 and enters each pixel i can be expressed by the following equation (1). That is, the sum of the product of sensitivity and intensity at each of the wavelengths λ1 and λ2.

Here, as shown in FIG. 2, assuming that the same light is incident on adjacent 2 × 2 pixel positions, Equation (1) holds for these four pixels (0 to 3). It can express like Formula (2) and Formula (3).

Here, the incident intensity (I _i ) can be acquired as an image signal output from each pixel, and the component sensitivities (a (λ1) i and a (λ2) i) for the wavelengths λ1 and λ2 of each pixel are measured values ( Reference image data) is known, and only the wavelengths λ1 and λ2 components L (λ1) and L (λ2) of the incident light are unknown amounts.

In Expression (3), the number of unknown quantities L (λ1) and L (λ2) is “2”, whereas the number of constraints is “4”, and M is not a square matrix. For this reason, since the inverse matrix of M cannot be obtained as it is, the pseudo inverse matrix M ⁺ of the matrix M is used.

Depending on the pixel position of interest, there is a 2 × 2 pixel area to be referred to which has a small coverage (coverage area) of the area dividing filter 12, but a solution that minimizes the error can be obtained by a pseudo inverse matrix.

Since the sensitivity a (λ1) i and the sensitivity a (λ2) i are known, the above pseudo inverse matrix M ⁺ can be calculated in advance from 2 × 2 pixel values of the reference image data for all pixel positions. it can. Then, if it is recorded in a non-volatile memory or the like in the image processing unit 14 in the imaging apparatus 100, the image signal [I = (I0, I1, I2, I3)] and the simple weighted sum of each component of the pseudo inverse matrix, the wavelength λ1 and λ2 components L = [L (λ1), L (λ2)] at all pixel positions can be obtained. it can. Therefore, the pseudo inverse matrix obtained from the component sensitivity (a (λ1) i, a (λ2) i) or the component sensitivity (a (λ1) i, a (λ2) i) is the reference data. In other words, Expression (4) is an expression for obtaining a component of a specific wavelength in the input image using the sensitivity of the specific wavelength that is the generation source of the reference data. Since the component of the specific wavelength is recorded in association with the unit pixel, image processing can be performed in the same manner as a RAW image.

  Therefore, when the peak of the wavelength of the input image is not between the two specific wavelengths λ1 and λ2, the component L becomes almost zero. On the contrary, when the peak of the wavelength of the input image is between the two specific wavelengths λ1 and λ2, there is a risk that the input image is forcibly decomposed into components L [L (λ1), L (λ2)].

  The calculation of Equation (5) for obtaining the pseudo inverse matrix is somewhat complicated, but this process can be executed only once by a computer (for example, a calibration device) outside the image pickup apparatus and recorded only when the image pickup apparatus 100 is manufactured. It does not affect the processing time during shooting.

  In this method, the ratio of incident light components is determined based on an actual captured image of light having a specific wavelength. Therefore, the reference data includes elements other than the area coverage such as the wavelength characteristics of the region division filter 12 and light leakage at the filter division boundary, and the correction processing of the present embodiment means that correction of these characteristics is also performed at the same time. Have.

  Further, since the wavelength component (spectral information) obtained by this method indicates how much the reference wavelength is included, the actual brightness information can be obtained from the RAW image data of the image sensor 13. it can.

[Obtaining output spectral information]
Acquisition of output spectral information using reference data will be described.
(i) When having one reference data (for comparison, not a preferred example of this embodiment)
a1. As shown in FIG. 7, the calibration apparatus generates specific wavelengths λ1 and λ2, and measures spectral image information from the specific wavelengths λ1 and λ2 to obtain spectral information (performs calibration).
a2. Reference data 1 is generated from the result of calibration. That is, the calibration apparatus generates a pseudo inverse matrix from a (λ 1) i and a (λ 2) i and records it as reference data 1 in the memory of the image processing unit 14.
a3. When the imaging apparatus 100 images a subject having an input light wavelength λ1 as shown in FIG. 8A, the image processing unit 14 performs signal processing based on the pseudo inverse matrix M + (reference data) and the image signal in Expression (4). I do. FIG. 8B schematically shows the signal processing result of the image processing unit 14. The wavelength λ1 can be detected from the output result.
a4. FIGS. 9A and 9B show the result of signal processing by the image processing unit 14 for the input light wavelength λ2. Also in this case, the image processing unit 14 can detect the wavelength λ2.

  Next, consider a case where an image of an object having an input light wavelength λ3 in FIG. 10A that is different from the specific wavelength used by the imaging apparatus 100 for calibration is captured. In this case, the wavelength λ3 correlated with λ1 and λ2 as shown in FIG. 10B can be detected according to the difference between the specific wavelengths λ1, λ2 and the input light wavelength λ3. For example, when the input light wavelength λ3 is a median wavelength of λ1 and λ2, the output spectral information of the input light wavelength λ3 is obtained assuming that each of λ1 and λ2 has about half the sensitivity. Actually, it is output as an L (λ1) component and an L (λ2) component.

  However, as shown in FIG. 11A, consider a case where a subject is imaged in which two specific wavelengths (input light wavelength λ1 and input light wavelength λ2) used for calibration have the same intensity. As shown in FIG. 11 (b), depending on the intensity, the detection result is the same as when the input light wavelength λ3 is input. Actually, the L (λ1) component and the L (λ2) component are respectively detected. That is, since the intensity of the input light wavelength λ1 and the input light wavelength λ2 is smaller than that of FIG. 10A, the output result is almost the same as that of the input light wavelength λ3 of FIG.

  As described above, if only one reference data corresponding to two specific wavelengths is provided, an incorrect output result may be obtained depending on the wavelength (spectral shape) of the input light, and sufficient detection accuracy of spectral information can be obtained. Absent.

(ii) Case of having two reference data Therefore, in this embodiment, a plurality of reference data is generated by the calibration described above.
b1. The calibration apparatus generates specific wavelengths λ1, λ2, and λ3 (λ1 <λ3 <λ2) as shown in FIG. 12, generates reference data 1 from the specific wavelengths λ1 and λ3 to obtain spectral information, and generates specific wavelengths. Reference data 2 is generated from λ2 and λ3.
a2. Reference data 1 and 2 are generated from the calibration result. Reference data 1 is a pseudo inverse matrix generated from a (λ1) i and a (λ3) i, and reference data 2 is a pseudo inverse matrix generated from a (λ3) i and a (λ2) i. is there. The calibration device records the reference data 1 and 2 in the memory of the image processing unit 14.
a3. The imaging apparatus 100 images a subject having an input light wavelength λ1 as shown in FIG. 13A, and the image processing unit 14 performs signal processing by applying the image signal and the reference data 1 to Expression (4).

  FIG. 13B schematically shows the signal processing result of the image processing unit 14. The upper part of FIG. 13B shows the signal processing result based on the reference data 1, and the lower part of FIG. 13B shows the signal processing result based on the reference data 2. Although the reference data 1 is generated from the specific wavelengths λ1 and λ3, the reference data 2 is generated from the specific wavelengths λ3 and λ2, and therefore the wavelength λ1 is not detected from the reference data 2. That is, the wavelength λ1 is detected only from the reference data 1. For this reason, the wavelength λ1 can be detected.

  The same applies to the case where a subject having an input light wavelength λ2 as shown in FIG. Although the reference data 1 is generated from the specific wavelengths λ1 and λ3, the reference data 2 is generated from the specific wavelengths λ3 and λ2, and therefore the wavelength λ2 is not detected from the reference data 1. That is, the wavelength λ2 is detected only from the reference data 2. Therefore, the wavelength λ2 can be detected.

  The same applies to the case where a subject having an input light wavelength λ3 as shown in FIG. Since the reference data 1 is generated from the specific wavelengths λ1 and λ3 and the reference data 2 is generated from the specific wavelengths λ3 and λ2, the wavelength λ3 is detected from both the reference data 1 and 2. For this reason, the wavelength λ3 can be detected.

  When an object having the same intensity of the input light wavelength λ1 and the input light wavelength λ2 in which two of the specific wavelengths used for calibration as shown in FIG. 16A are overlapped is imaged as follows: Become. Since the reference data 1 is generated from the specific wavelengths λ1 and λ3, the input light wavelength λ1 can be detected from the input light wavelength λ1 and the input light wavelength λ2. Since the reference data 2 is generated from the specific wavelengths λ3 and λ2, the input light wavelength λ2 can be detected from the input light wavelength λ1 and the input light wavelength λ2.

  Therefore, an output result as shown in FIG. 16B is obtained. Unlike FIG. 11B, the wavelengths λ1 and λ2 can be detected without detecting the wavelength λ3.

  In FIG. 12, two reference data (images) corresponding to the three specific wavelengths λ1 to λ3 are generated. However, increasing the specific wavelength used for the reference data increases the reference data and increases the resolution of the spectral information. be able to.

  Thus, by having a plurality of reference data corresponding to two or more specific wavelengths, it is possible to output highly accurate spectral information regardless of the wavelength (spectral shape) of the input light. That is, it is possible to improve the detection accuracy of spectral information in the imaging apparatus 100 having the region dividing filter 12.

  Further, by combining output spectral information obtained from a plurality of pieces of reference data, it is possible to provide the imaging apparatus 100 that can output a spectral image with a large amount of spectral information.

[Example of application to equipment]
An application example of the imaging apparatus 100 described so far will be described.
The image sensor 13 is produced as a CMOS image sensor substrate, for example, by mounting a sensor die on the FPC substrate and wire bonding. The area dividing filter 12 is formed by depositing AL (several hundred nm) on a glass substrate and then forming a checker pattern as shown in FIG. 5 by a photolithography process. That is, according to the checker pattern, a rectangular area with and without an AL light shielding film is formed vertically and horizontally.

  The area division filter 12 is fixed so that the image pickup element 13 and the filter pattern have a one-to-one correspondence, and can display image information by being attached to the image pickup lens 11, the sensor drive board, and a control PC (Personal Computer). did. The applicant experimentally realized the image processing unit 14 in the imaging apparatus 100 of the present embodiment with a sensor drive board and a control PC. In the case of shipping as the imaging apparatus 100, it can be realized using an FPGA or the like.

  FIG. 17 shows a RAW image (part) of the image sensor 13 for a specific wavelength in calibration necessary for generating a spectral image. The specific wavelength used for calibration was a single wavelength of 450 nm, 550 nm, and 650 nm from the left in FIG. These single-wavelength lights are created using a halogen lamp and a bandpass filter.

  First, the calibration apparatus performs calibration at wavelengths of 450 nm and 550 nm to create reference data 1. Next, calibration is performed at a wavelength of 550 nm and a wavelength of 650 nm, and reference data 2 is created. Reference data 1 and 2 are stored in the control PC.

  Since each rectangular area of the area dividing filter 12 (the pattern of the area without the AL light shielding film and the area with the AL light shielding film) is arranged corresponding to each pixel of the image sensor 13, a light / dark pattern corresponding to the filter pattern ( A pattern of an area without the AL light shielding film and a region with the AL light shielding film) is output. Here, a case is shown in which the area division filter 12 is a checker pattern, but a light and dark pattern image corresponding to the filter pattern is output in the same manner for a stripe pattern.

  As shown in the RAW image (part) of FIG. 17, the contrast of the light and dark pattern changes depending on the wavelength (it is difficult to understand in the figure, but the contrast decreases as the wavelength increases). This depends on the optical characteristics resulting from the filter pattern and the arrangement. The optical characteristics are affected by the spectral transmittance, scattering, and diffraction phenomenon caused by the pattern of the filter.

  FIG. 18 shows a histogram of sensor pixel values (luminance) for a single wavelength. The luminance histogram of FIG. 18 shows two peaks at each wavelength. These two peaks correspond to the bright part and the dark part in FIGS. 18A to 18C, it can be seen that the two peak differences (peak intervals) change depending on the wavelength (the longer the specific wavelength, the smaller the peak difference).

  The peak on the high luminance side is the luminance information of the bright pattern, and the peak on the low luminance side is the dark luminance information of the dark pattern. Here, a histogram for a plurality of pixels (several hundreds to thousands) is shown, but two peaks are obtained in the same manner in the brightness / darkness information of one pixel unit.

When the brightness value of the light pattern is Ib and the brightness value of the dark pattern is Id, the contrast (difference) I (x) = (Ib−Id) / (Id + Id) of the light / dark pattern brightness value is as follows.
450 nm: I (x) = (63-41) / (63 + 41) = about 0.212
550 nm: I (x) = (60−42) / (60 + 42) = about 0.176
650 nm: I (x) = (57−47) / (57 + 47) = about 0.096
FIG. 19 shows the wavelength dependence of contrast in which I (x) is plotted against wavelength.

  That is, the spectral information of the adjacent pixels can be obtained by calculating the difference between the luminance values of the adjacent pixels in the light / dark pattern. For example, when the imaging apparatus 100 captures a single wavelength and obtains light / dark pattern RAW data, the contrast I (x) of the light / dark pattern luminance value is calculated, and the known I (x) of 450, 550, and 650 nm is calculated. A single wavelength can be estimated from which one is closer.

  Next, the imaging apparatus 100 of the present embodiment photographed a standard white plate irradiated with spectrum light having two peaks of a single wavelength of 450 nm and 650 nm having the same intensity. A wavelength tunable light source ELS (manufactured by Nikon) was used for generation of the spectrum light.

  FIG. 20 shows the spectrum of the generated spectral light, FIG. 21A shows an example of a spectral image based on the calibration data 1, and FIG. 22A shows an example of a spectral image based on the calibration data 2. It is. The calibration data 1 is generated from single wavelengths of wavelengths 450 nm and 550 nm, and the calibration data 2 is generated from single wavelengths of wavelengths 550 nm and 650 nm. For this reason, as described with reference to FIG. 16B, the output spectral information of light having a wavelength of 650 nm cannot be obtained with the output spectral information based on the calibration data 1. With the output spectral information based on the calibration data 2, output spectral information of light having a wavelength of 450 nm cannot be obtained.

  In FIG. 21A, two wavelength components of L (λ: 450 nm) and L (λ: 550 nm) are obtained for each pixel block (in this case, two adjacent unit pixels) by reference data 1. Yes. However, the wavelength component at 550 nm is almost zero. In FIG. 22A, two wavelength components of L (λ: 550 nm) and L (λ: 650 nm) are obtained for each pixel block (in this case, two adjacent unit pixels) by the reference data 2. Yes. However, the wavelength component of L (λ: 550 nm) is almost zero.

FIG. 21B shows the spectral information output result of the area surrounded by the broken line in FIG. For adjacent pixels in the area surrounded by the broken line, the image processing unit 14
[L (λ: 450 nm) −L (λ: 550 nm)] / [L (λ: 450 nm) + L (λ: 550 nm)]
Calculate The calculation result is taken on the X axis, and the frequency corresponding to the calculation result (number of calculations) is counted. As shown in the figure, a frequency peak is obtained at about 0.9. That is, when the calculation result falls within a certain range, the count is increased by one to obtain the frequency.

FIG. 22B shows the spectral information result of the area surrounded by the broken line in FIG. For adjacent pixels in the area surrounded by the broken line, the image processing unit 14
[L (λ: 550 nm) −L (λ: 650 nm)] / [L (λ: 550 nm) + L (λ: 650 nm)]
Calculate This calculation result is taken on the X axis, and the frequency corresponding to the calculation result is counted. As shown in the figure, a frequency peak is obtained at about -0.9.

  Since it is known that the spectrum light used for the reference has an intensity peak at a wavelength of 450 nm and a wavelength of 650 nm, the peak of FIG. 21B corresponds to the wavelength of 450 nm, and the peak of FIG. 22B is the wavelength of 650 nm. Corresponding to Therefore, the spectral information output result enables extraction of the wavelength. By obtaining the spectral information output results of FIGS. 21 and 22 in advance, spectral information of an unknown spectrum can be obtained.

  FIG. 23 is a combination of the spectral information of FIGS. 21 (b) and 22 (b). Comparing FIG. 23 with FIG. 20, the same result as the incident spectrum having peaks at 450 and 650 taken this time is obtained.

  Note that the horizontal axis in FIGS. 21B and 22B is represented by −1 to 1, but the actual image output is, for example, displayed in 256 gradations. For example, +1 Can be easily presented as visible information by representing the black scale with black (0) and -1 with the gray scale of white (255). Or you may display by the color table according to a spectral wavelength.

  As described above, the imaging apparatus 100 having the area division filter 12 according to the present embodiment prepares a plurality of reference data corresponding to two or more specific wavelengths, and the brightness of an image photographed through the area division filter 12 Spectral information can be detected with high accuracy by calculating a pattern using reference data and extracting a difference in luminance value between adjacent pixels.

DESCRIPTION OF SYMBOLS 11 Image pickup lens 12 Area | region division filter 13 Image sensor 14 Image processing part 100

JP 2005-057541 A Japanese Patent Laid-Open No. 2005-31007

Optical Alliance 1999.11 p4-9

Claims (7)

In an imaging device comprising: an imaging device that converts light into an electrical signal; a region division filter that is disposed on the light input side of the imaging device; and an image processing unit that processes an output signal of the imaging device.
In the region division filter, two or more regions that cause a luminance difference between adjacent unit pixels due to different transmittances for light of a specific wavelength are arranged in a predetermined pattern.
Light generated from the first output signal output by each image sensor for the first specific wavelength and the second output signal output by each image sensor for the second specific wavelength is transmitted to the image sensor. First reference data for extracting the first specific wavelength component and the second specific wavelength component when incident,
Light generated from the second output signal output by each image sensor for the second specific wavelength and the third output signal output by each image sensor for the third specific wavelength is transmitted to the image sensor. Second reference data for extracting the second specific wavelength component and the third specific wavelength component when incident, and
The image processing unit generates the first specific wavelength component and the second specific wavelength component by applying the first reference data to each output signal of the unit pixel when light is incident on the image sensor. Applying the second reference data to generate a second specific wavelength component and a third specific wavelength component;
The first specific wavelength component and the second specific wavelength component generated by the first reference data, and the second specific wavelength component and the third specific wavelength component generated by the second reference data An image pickup apparatus that synthesizes specific wavelength components . The image processing unit includes a first difference between the second specific wavelength component with respect to the first specific wavelength component between adjacent unit pixels, and a first difference between the first specific wavelength component and the second specific wavelength component. sum,
In addition, the second difference between the third specific wavelength component and the second specific wavelength component between the adjacent unit pixels and the second sum of the second specific wavelength component and the third specific wavelength component are calculated. And
A value obtained by dividing the first difference by the first sum and the number of calculations are output in association with each other, and a value obtained by dividing the second difference by the second sum and the number of calculations are output in association with each other.
The imaging apparatus according to claim 1. The frequency value obtained by counting up the value obtained by dividing the first difference by the first sum and the value obtained by dividing the second difference by the second sum for each predetermined range. Output,
The imaging apparatus according to claim 2. The image processing unit has the first specific wavelength known in advance to a value obtained by dividing the first difference by the first sum and a value obtained by dividing the second difference by the second sum. values, the second specific wavelength values, and outputs the third specific wavelength value, the frequency value in association with a
The imaging apparatus according to claim 3.   In the area division filter, two or more areas having different transmittances with respect to a specific wavelength causing a luminance difference between adjacent unit pixels are formed by a portion where the light shielding film is formed and a portion where the light shielding film is not formed. The image pickup apparatus according to claim 1, wherein the image pickup apparatus is an image pickup apparatus. The region dividing filter includes two parts having different transmittances for a specific wavelength that causes a luminance difference between adjacent unit pixels depending on a part where a dielectric multilayer film transmitting a specific wavelength is formed and a part where the dielectric multilayer film is not formed. The above region is formed,
The imaging apparatus according to any one of claims 1 to 4, wherein Spectral information of an imaging device comprising: an imaging device that converts light into an electrical signal; a region division filter that is disposed on the light input side of the imaging device; and an image processing unit that processes an output signal of the imaging device. In the creation method,
In the region division filter, two or more regions that cause a luminance difference between adjacent unit pixels due to different transmittances for light of a specific wavelength are arranged in a predetermined pattern.
Light generated from the first output signal output by each image sensor for the first specific wavelength and the second output signal output by each image sensor for the second specific wavelength is transmitted to the image sensor. First reference data for extracting the first specific wavelength component and the second specific wavelength component when incident,
Light generated from the second output signal output by each image sensor for the second specific wavelength and the third output signal output by each image sensor for the third specific wavelength is transmitted to the image sensor. Using the second reference data to extract the second specific wavelength component and the third specific wavelength component when incident,
The image processing unit generates the first specific wavelength component and the second specific wavelength component by applying the first reference data to each output signal of the unit pixel when light is incident on the image sensor. Applying the second reference data to generate a second specific wavelength component and a third specific wavelength component;
The first specific wavelength component and the second specific wavelength component generated by the first reference data, and the second specific wavelength component and the third specific wavelength component generated by the second reference data A spectral information generation method characterized by combining specific wavelength components .