JP6864604B2 - Imaging device - Google Patents

Description

The present invention relates to an imaging device.

Imaging devices mounted on devices such as IoT (Internet of things) devices and wearable devices are required to be thin and cost-reduced in order to be incorporated in a limited space. Therefore, for example, as in Patent Document 1, an imaging method capable of obtaining an object image without using a lens has been proposed. Further, for example, Patent Document 2 proposes an "imaging device that enables a finger authentication device having a small thickness that can be installed in a portable information device" using a microlens array.

U.S. Patent Application Publication No. 2015/0219808 Japanese Unexamined Patent Publication No. 2011-203792

The above-mentioned Patent Document 1 is an imaging method in which a special grid pattern is arranged in front of an image sensor, and an image of an object is obtained by solving an inverse problem by signal processing using an oblique shadow pattern of the grid pattern received by the image sensor. .. Although this imaging method makes it possible to reduce the thickness of the imaging device, there is a problem that the calculation for obtaining an image becomes complicated.

Further, Patent Document 2 described above states that "the front-stage microlens array and the diaphragm array are opposed to each other, the diaphragm of the diaphragm array is provided near the focal point of the light from the subject of the front-stage microlens array, and the subsequent-stage microlens By "focusing light at the same time by the array", it is an imaging method that "the optical system used for the finger vein recognition device can be made smaller and a compact and thin finger vein recognition device can be realized". This imaging method makes it possible to reduce the thickness of a close-up image pickup device, but it is thin in that a distance is required for focusing by a lens and a space for arranging two lens arrays is required. It is the limit of conversion.

An object of the present invention is to provide an image pickup apparatus having a simple structure, being thin, and having a wide field of view during close-up photography.

The present application includes a plurality of means for solving at least a part of the above problems, and examples thereof are as follows.

One aspect of the present invention is an image pickup apparatus, which is a modulator that modulates the light intensity using a compound eye lattice pattern having a plurality of basic lattice patterns, and converts the light transmitted through the modulator into image data. It includes a plurality of image sensors to be output and an image processing unit that performs image processing of a plurality of image data output from the plurality of image sensors.

According to the present invention, it is possible to provide an image pickup apparatus having a simple structure, being thin, and having an enlarged field of view at the time of close-up photography.

Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is a figure which shows the structural example of the image pickup apparatus which concerns on 1st Embodiment. It is a figure which shows the structural example of a modulator. It is a figure which shows the other structural example of a modulator. It is a figure which shows the state of taking a picture of the object of the outside world using an image pickup apparatus. It is a flowchart which shows an example of the image processing of the image processing unit. It is a figure explaining that the projection image from the front surface to the back surface of a lattice substrate by oblique incident parallel light causes in-plane deviation. It is a figure explaining the generation of moire fringes and the frequency spectrum when the axes of the lattices on both sides of a lattice substrate are aligned, (a) shows the case of incident angle: 0, and (b) shows the case of incident angle: + θ. (C) shows the case of the incident angle: −θ. It is a figure which shows an example of the lattice substrate which arranged the front surface lattice and the back surface lattice by shifting the axis. It is a figure explaining the generation of moire fringes and the frequency spectrum when the lattices on both sides of a lattice substrate are staggered, (a) shows the case of incident angle: 0, and (b) shows the case of incident angle: + θ. (C) shows the case of the incident angle: −θ. It is a figure which shows an example of a lattice pattern. It is a figure which shows another example of a lattice pattern. It is a figure which shows still another example of a lattice pattern. It is a figure explaining the angle which light from each point which constitutes an object makes with respect to an image sensor. It is a figure explaining that the front side grid pattern is projected when the object is at an infinite distance. It is a figure which shows the example of the moire fringe which is generated when an object is at an infinite distance. It is a figure explaining that the front side grid pattern is enlarged when the object is at a finite distance. It is a figure which shows the example of the moire fringe which is generated when an object is at a finite distance. It is a figure which shows the example of the moire fringe which corrected the back side grid pattern when the object is at a finite distance. It is a figure which shows the configuration example of the image pickup apparatus which realizes the back side grid pattern by image processing. It is a figure which shows the structural example of a modulator. It is a flowchart which shows an example of the image processing of the image processing unit. It is a figure which shows the configuration example of the image pickup apparatus which realizes the time division fringe scan. It is a figure which shows the example of the lattice pattern in the time division fringe scan, (a) shows the case of a phase (ΦF, Φ = 0), and (b) shows the case of a phase (ΦF, Φ = π / 2). , (C) show the case of phase (ΦF, Φ = π), and (d) show the case of phase (ΦF, Φ = 3π / 2). It is a figure which shows the structural example of the modulator which realizes a time-divided fringe scan, (a) shows the case of a phase (ΦF, Φ = 0), and (b) shows the case of a phase (ΦF, Φ = π / 2). The case is shown, (c) shows the case of the phase (ΦF, Φ = π), and (d) shows the case of the phase (ΦF, Φ = 3π / 2). It is a flowchart which shows an example of the image processing of the image processing unit which realizes the time division fringe scan. It is a figure which shows the configuration example of the image pickup apparatus which realizes a space division fringe scan. It is a figure which shows the example of the lattice pattern in the space division fringe scan. It is a figure explaining the angle of view θ _max2. It is a figure explaining the range and the field of view of the lattice pattern which the light from a certain point which constitutes an object irradiates when the object is in a short distance. It is a figure which shows the configuration example of the image pickup apparatus which realizes the wide field of view by the combination of the grid pattern of a compound eye and a plurality of image sensors. It is a figure which shows an example of the lattice pattern of a compound eye. It is a figure explaining the grid pattern of a compound eye and the field of view by a plurality of image sensors. It is a flowchart which shows an example of the image processing of the image processing unit. It is a figure explaining an example of the synthesis method. It is a figure explaining an example of the synthesis method. It is a flowchart which shows other example of the image processing of the image processing unit. It is a figure explaining the arrangement example of an image sensor. It is a figure explaining another arrangement example of an image sensor. It is a figure explaining the method of optimizing the size of an image sensor. It is a figure explaining an example of a gaze direction. It is a figure explaining another example of a gaze direction. It is a figure which shows an example of the concentric lattice pattern projected on the rectangular light receiving surface. It is a figure which shows an example of the elliptical lattice pattern projected on the rectangular light receiving surface. It is a figure which shows another example of the concentric lattice pattern projected on the rectangular light receiving surface. It is a figure which shows the configuration example of the finger vein imaging apparatus using the grid pattern of a compound eye. It is sectional drawing which shows the structural example of the liquid crystal display element as a modulator. It is a figure which shows the configuration example of the finger vein imaging apparatus using a monocular lattice pattern.

In the following embodiments, when necessary for convenience, the description will be divided into a plurality of sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one is the other. There is a relationship of some or all modifications, details, supplementary explanations, etc.

In addition, in the following embodiments, when referring to the number of elements (including the number, numerical value, quantity, range, etc.), except when explicitly stated and when the number is clearly limited to a specific number in principle. , The number is not limited to the specific number, and may be more than or less than the specific number.

Furthermore, it goes without saying that, in the following embodiments, the components (including element steps and the like) are not necessarily essential unless otherwise specified or when it is clearly considered to be essential in principle. No.

Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of a component or the like, the shape, etc., is substantially the same, except when it is clearly stated or when it is considered that it is not clearly the case in principle. It shall include those similar to or similar to. This also applies to the above numerical values and ranges.

Further, in all the drawings for explaining the embodiment, the same members are in principle the same reference numerals, and the repeated description thereof will be omitted. Hereinafter, each embodiment of the present invention will be described with reference to the drawings.

[First Embodiment]
<Principle of shooting infinite objects>
FIG. 1 is a diagram showing a configuration example of the image pickup apparatus 101 according to the first embodiment. The image pickup apparatus 101 acquires an image of an object in the outside world without using a lens for forming an image as provided in a normal camera, and is composed of a modulator 102, an image sensor 103, and an image processing unit 106. Has been done.

FIG. 2 is a diagram showing a configuration example of the modulator 102. The modulator 102 is closely fixed to the light receiving surface of the image sensor 103 (FIG. 1), and the lattice substrate 102a and the first lattice pattern 104 and the second lattice formed on the upper and lower surfaces of the lattice substrate 102a. It is composed of a pattern 105. The lattice substrate 102a is made of a transparent material such as glass or plastic. Hereinafter, the image sensor 103 side of the lattice substrate 102a is referred to as a back surface, and the facing surface, that is, the image target side is referred to as a front surface.

These grid patterns 104 and 105 are composed of concentric grid patterns in which the spacing of the grid patterns, that is, the pitch becomes narrower in inverse proportion to the radius from the center toward the outside. The lattice patterns 104 and 105 are formed by depositing metals such as aluminum and chromium by, for example, a sputtering method used in a semiconductor process. Light and shade are added by the pattern with metal vapor deposition and the pattern without metal deposition.

The formation of the lattice patterns 104 and 105 is not limited to vapor deposition, and may be formed by adding shading by printing with, for example, an inkjet printer. Further, although visible light has been described as an example here, for example, in the case of photographing with far infrared rays, the lattice substrate 102a is made of a material transparent to far infrared rays such as germanium, silicon, and chalcogenide. That is, a material transparent to the wavelength to be photographed can be used for the lattice substrate 102a, and a material such as a metal that blocks the wavelength to be photographed can be used for the lattice patterns 104 and 105. ..

Although the method of forming the lattice patterns 104 and 105 on the lattice substrate 102a in order to realize the modulator 102 has been described here, the present invention is not limited to this. FIG. 3 is a diagram showing another configuration example of the modulator 102. The modulator 102 can also be realized by forming the lattice patterns 104 and 105 on a thin film and holding them by a support member 102b provided instead of the lattice substrate 102a.

The image sensor 103 includes, for example, a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Sensor) image sensor. Pixels 103a, which are light receiving elements, are regularly arranged in a grid pattern on the surface of the image sensor 103. The light transmitted through the grid patterns 104 and 105 is received by the image sensor 103 after the intensity of the light is modulated by the grid patterns. The image sensor 103 converts the optical image received by the pixel 103a into an image signal which is an electric signal. The image signal (also referred to as “image data”) output from the image sensor 103 is image-processed by the image processing unit 106 and output to the image display unit 107 or the like. The image processing unit 106 may store the processed image data in a storage device (not shown) included in the image pickup device 101, or output the processed image data to an external host computer or recording medium.

The image pickup device 101 may include, for example, a processor, a memory, a communication device, a processing circuit, and the like. Further, the image pickup apparatus 101 may include an input / output interface for connecting to an external device such as USB (Universal Serial Bus) and HDMI (High-Definition Multimedia Interface). The image processing unit 106 may be realized by, for example, a dedicated image processing circuit or a processor that executes a program.

FIG. 4 is a diagram showing a state in which an object in the outside world is photographed using the image pickup apparatus 101 of FIG. FIG. 4 shows an example in which the subject 401 is photographed by the image pickup apparatus 101 and displayed on the image display unit 107. When the subject 401 is photographed, the surface of the modulator 102, specifically the surface of the lattice substrate 102a on which the first lattice pattern 104 is formed, faces the subject 401. It is said.

Subsequently, an outline of image processing by the image processing unit 106 will be described. FIG. 5 is a flowchart showing an example of image processing of the image processing unit 106.

First, the image processing unit 106 performs a two-dimensional Fourier transform calculation such as a fast Fourier transform (FFT) for each RGB (Red Green Blue) component of the color on the moire fringe image output from the image sensor 103. The frequency spectrum is obtained by the development process according to (501). Subsequently, the image processing unit 106 acquires an image by cutting out data in a necessary frequency region from the frequency spectrum processed in step 501 (502) and then calculating the intensity of the frequency spectrum (503). Then, the image processing unit 106 performs noise removal processing (504) on the obtained image, and subsequently performs contrast enhancement processing (505) and the like. After that, the image processing unit 106 adjusts the color balance of the image (506) and outputs it as a captured image. As a result, the image processing by the image processing unit 106 is completed.

Subsequently, the imaging principle in the image pickup apparatus 101 will be described.

First, the concentric grid patterns 104 and 105 whose pitch becomes finer in inverse proportion to the radius from the center are defined as follows. In a laser interferometer or the like, it is assumed that a spherical wave close to a plane wave and a plane wave used as reference light interfere with each other. When the radius from the reference coordinate which is the center of the concentric circle is r and the phase of the spherical wave there is Φ (r), the spherical wave uses the coefficient β that determines the magnitude of the curve of the wave surface.

と表せる。

Can be expressed as.

Despite the spherical wave, it is represented by the square of the radius r because it is a spherical wave close to a plane wave and can be approximated only by the lowest order of expansion. When a plane wave interferes with light having this phase distribution,

のような干渉縞の強度分布が得られる。これは、

The intensity distribution of the interference fringes is obtained. this is,

を満たす半径位置で明るい線を持つ同心円の縞となる。縞のピッチをｐとすると、

It becomes a concentric stripe with a bright line at a radius position that satisfies. Let p be the pitch of the stripes

が得られ、ピッチは、半径に対して反比例して狭くなっていくことがわかる。このような縞を持つプレートは、フレネルゾーンプレートやガボールゾーンプレートと呼ばれる。本実施形態では、式（２）で定義される強度分布に比例した透過率分布をもった格子パターンを、図１に示した格子パターン１０４，１０５として用いる。

Is obtained, and it can be seen that the pitch becomes narrower in inverse proportion to the radius. Plates with such stripes are called Fresnel zone plates or Gabor zone plates. In the present embodiment, the lattice patterns having the transmittance distribution proportional to the intensity distribution defined by the equation (2) are used as the lattice patterns 104 and 105 shown in FIG.

FIG. 6 is a diagram for explaining that the projected image from the front surface to the back surface of the lattice substrate due to the obliquely incident parallel light causes in-plane deviation. It is assumed that parallel light is incident on a modulator 102 having a thickness t having lattice patterns 104 and 105 formed on both sides at an angle θ _0. Geometrically, with the refraction angle in the modulator 102 as θ, the light multiplied by the transmittance of the lattice on the front surface is incident on the back surface with a deviation of δ = t · tan θ, and is assumed to be the center of the two concentric lattices. If they are formed together, the transmittance of the lattice on the back surface will be shifted by δ and multiplied. At this time,

のような強度分布が得られる。この展開式の第４項が、２つの格子のずれの方向にまっすぐな等間隔の縞模様を、２つの格子の重なり合った領域一面に作ることがわかる。このような縞と縞の重ね合わせによって相対的に低い空間周波数で生じる縞は、モアレ縞と呼ばれる。このようにまっすぐな等間隔の縞は、検出画像の２次元フーリエ変換によって得られる空間周波数分布に鋭いピークを生じる。その周波数の値からδの値、すなわち光線の入射角θを求めることが可能となる。このような全面で一様に等間隔で得られるモアレ縞は、同心円状の格子配置の対称性から、ずれの方向によらず同じピッチで生じることは明らかである。このような縞が得られるのは、格子パターンをフレネルゾーンプレートまたはガボールゾーンプレートで形成したことによるものであるが、全面で一様に等間隔なモアレ縞が得られるのであればどのような格子パターンを使用してもよい。

The intensity distribution is obtained. It can be seen that the fourth term of this expansion formula creates a striped pattern at equal intervals straight in the direction of deviation of the two grids over the overlapping region of the two grids. The fringes generated at a relatively low spatial frequency by superimposing such fringes are called moire fringes. Such straight, evenly spaced fringes produce sharp peaks in the spatial frequency distribution obtained by the two-dimensional Fourier transform of the detected image. From the value of the frequency, it is possible to obtain the value of δ, that is, the incident angle θ of the light beam. From the symmetry of the concentric lattice arrangement, it is clear that the moire fringes obtained uniformly over the entire surface at equal intervals occur at the same pitch regardless of the direction of deviation. Such fringes are obtained by forming the lattice pattern with a Fresnel zone plate or a Gabor zone plate, but any lattice can be obtained if moire fringes that are uniformly and evenly spaced on the entire surface can be obtained. Patterns may be used.

Here, from equation (5), only the components with sharp peaks

のように取り出すと、そのフーリエスペクトルは、

When taken out as, the Fourier spectrum is

のようになる。ここで、Ｆはフーリエ変換の演算を表し、ｕおよびｖは、ｘ方向およびｙ方向の空間周波数座標、括弧を伴うδはデルタ関数である。この結果から、検出画像の空間周波数スペクトルにおいて、モアレ縞の空間周波数のピークがｕ＝±δβ／πの位置に生じることがわかる。その様子を図７に示す。

become that way. Here, F represents the operation of the Fourier transform, u and v are the spatial frequency coordinates in the x and y directions, and δ with parentheses is the delta function. From this result, it can be seen that the peak of the spatial frequency of the moire fringes occurs at the position of u = ± δβ / π in the spatial frequency spectrum of the detected image. The situation is shown in FIG.

FIG. 7 is a diagram for explaining the generation of moire fringes and the frequency spectrum when the axes of the lattices on both sides of the lattice substrate are aligned. In FIG. 7, from left to right, a layout diagram of the light beam and the modulator 102, a moire fringe, and a schematic diagram of the spatial frequency spectrum are shown, respectively. FIG. 7 (a) is a vertical incident, FIG. 7 (b) is a case where a light ray is incident from the left side at an angle θ, and FIG. 7 (c) is a case where a light ray is incident from the right side at an angle θ. Each is shown.

The axes of the first lattice pattern 104 formed on the front surface side of the modulator 102 and the second lattice pattern 105 formed on the back surface side are aligned. In FIG. 7A, since the shadows of the first grid pattern 104 and the second grid pattern 105 match, moire fringes do not occur.

In (b) of FIG. 7 and (c) of FIG. 7, the same moire occurs because the deviations between the first lattice pattern 104 and the second lattice pattern 105 are equal, and the peak positions of the spatial frequency spectra also match. From the spatial frequency spectrum, it becomes impossible to determine whether the incident angle of the light beam is in the case of (b) of FIG. 7 or in the case of (c) of FIG. An example of how to avoid this is shown in FIG.

FIG. 8 is a diagram showing an example of a lattice substrate arranged with the axes of the front surface lattice and the back surface lattice shifted. As shown in FIG. 8, the two grid patterns 104 and 105 are preliminarily shifted relative to the optical axis so that the shadows of the two grid patterns are shifted and overlapped with respect to the light rays perpendicularly incident on the modulator 102. It is necessary to keep it. When the relative deviation of the shadows of the two lattices with respect to the vertically incident plane wave on the axis is δ ₀ , the deviation δ caused by the plane wave at the incident angle θ is

のように表せる。このとき、入射角θの光線のモアレ縞の空間周波数スペクトルのピークは周波数のプラス側では、

Can be expressed as. At this time, the peak of the spatial frequency spectrum of the moire fringes of the light beam having an incident angle θ is on the positive side of the frequency.

の位置となる。画像センサの大きさをＳ、画像センサのｘ方向およびｙ方向の画素数を共にＮとすると、２次元フーリエ変換による離散画像の空間周波数スペクトルは、−Ｎ／（２Ｓ）から＋Ｎ／（２Ｓ）の範囲で得られる。このことから、プラス側の入射角とマイナス側の入射角を均等に受光することを考えれば、垂直入射平面波（θ＝０）によるモアレ縞のスペクトルピーク位置は、原点（ＤＣ：直流成分）位置と、例えば＋側端の周波数位置との中央位置、すなわち、

It becomes the position of. Assuming that the size of the image sensor is S and the number of pixels in the x and y directions of the image sensor is N, the spatial frequency spectrum of the discrete image by the two-dimensional Fourier transform is from -N / (2S) to + N / (2S). It can be obtained in the range of. From this, considering that the positive-side incident angle and the negative-side incident angle are received evenly, the spectral peak position of the moire fringes due to the vertically incident plane wave (θ = 0) is the origin (DC: DC component) position. And, for example, the center position with the frequency position at the + side end, that is,

の空間周波数位置とするのが妥当である。したがって、２つの格子の相対的な中心位置ずれは、

It is appropriate to use the spatial frequency position of. Therefore, the relative center misalignment of the two grids is

とするのが妥当である。

Is appropriate.

FIG. 9 is a diagram for explaining the generation of moire fringes and the frequency spectrum when the lattices on both sides of the lattice substrate are arranged in a staggered manner. Similar to FIG. 7, from left to right, a layout diagram of the light beam and the modulator 102, moire fringes, and a schematic diagram of the spatial frequency spectrum are shown, respectively. 9 (a) shows a light ray vertically incident, FIG. 9 (b) shows a light ray incident at an angle θ from the left side, and FIG. 9 (c) shows a light ray incident at an angle θ from the right side. The cases to be done are shown respectively.

The first grid pattern 104 and the second grid pattern 105 are arranged in advance by shifting by _{δ 0.} Therefore, moire fringes also occur in FIG. 9A, and peaks appear in the spatial frequency spectrum. As described above, the shift amount δ ₀ is set so that the peak position appears in the center of the spectral range on one side from the origin. At this time, in FIG. 9B, the deviation δ is further increased, and in FIG. 9C, the deviation δ is decreasing. Therefore, unlike FIG. 7, the deviation δ is different from that of FIG. 9B. The difference from (c) in FIG. 9 can be discriminated from the peak position of the spectrum. The spectral image of this peak is a bright spot showing a luminous flux at infinity, and is nothing but an image taken by the image pickup apparatus 101 of FIG.

Assuming that the maximum angle of incidence of parallel light that can be received is θ _max ,

より、撮像装置１０１にて受光できる最大画角は、

Therefore, the maximum angle of view that can be received by the image pickup device 101 is

で与えられる。

Given in.

By analogy with imaging using a general lens, considering that _{parallel light with an angle of view θ max} is focused at the edge of the image sensor and received, the effective focal length of the image pickup device 101 without a lens is ,

に相当すると考えることができる。

Can be considered to be equivalent to.

Here, from equations (13) and (14), it can be seen that the angle of view can be changed by the thickness t of the modulator 102 and the size S of the image sensor 103. Therefore, for example, if the modulator 102 has the configuration shown in FIG. 3 and has a function (for example, a mechanical structure) that can change the length of the support member 102b, it is possible to change the angle of view at the time of shooting. Become.

Although the fast Fourier transform has been described as an example as a method for calculating the spatial frequency spectrum from the moire fringes, the present invention is not limited to this, and can be realized by using a discrete cosine transform (DCT: Discrete Cosine Transform) or the like. Therefore, it is possible to further reduce the amount of calculation.

Further, the transmittance distributions of the lattice patterns 104 and 105 have been described on the assumption that they have sinusoidal characteristics as shown in the equation (2), but such a component is used as the fundamental frequency component of the lattice pattern. It may be included. For example, it is possible to binarize the transmittance of the lattice pattern as shown in FIG. 10 (a diagram showing an example of the lattice pattern), and further, as shown in FIG. 11 (a diagram showing another example of the lattice pattern). It is also possible to increase the transmittance by widening the width of the region having a high transmittance by changing the duty of the lattice region having a high transmittance and the region having a low transmittance. As a result, effects such as suppressing diffraction from the lattice pattern can be obtained, and deterioration of the photographed image can be reduced.

Further, the lattice patterns 104 and 105 may be realized by phase modulation instead of transmittance modulation. For example, as shown in FIG. 12 (a diagram showing still another example of the lattice pattern), by configuring the lattice substrate 102a with the cylindrical lens 1201, an intensity modulation pattern as shown in the figure is generated on the image sensor 103. Therefore, it is possible to take an image as in the discussion so far. As a result, the light amount loss due to the shielding portion of the lattice pattern 104 can be reduced, the light utilization efficiency can be improved, and the effect of suppressing diffraction from the lattice pattern can be obtained. Although it was realized with a lens in FIG. 12, it can also be realized with a phase modulation element having the same effect.

In all of the above explanations, the incident light rays are incident at one incident angle at the same time, but in order for the image pickup apparatus 101 to actually act as a camera, the incident rays are incident at a plurality of incident angles at the same time. Must be assumed. Light with such a plurality of incident angles already overlaps the images of the plurality of front-side grids when they are incident on the back-side grid pattern. If these images generate moire fringes with each other, there is a concern that the moire fringes become noise that hinders the detection of the moire fringes with the second lattice pattern 105, which is a signal component. However, in reality, the overlap of the images of the first grid pattern 104 does not cause the peak of the moire image, and the peak is caused only by the overlap with the second grid pattern 105 on the back surface side. The reason will be explained below.

First, the major difference is that the overlap of the shadows of the first grid pattern 104 on the surface side due to the light rays having a plurality of incident angles is not a product but a sum. Regarding the overlap between the shadow of the first grid pattern 104 and the second grid pattern 105 due to the light of one incident angle, the light intensity distribution which is the shadow of the first grid pattern 104 and the second grid pattern 105 By multiplying the transmittance of, the light intensity distribution after passing through the second lattice pattern 105 on the back surface side can be obtained.

On the other hand, the overlap of shadows due to light having a plurality of angles incident on the first lattice pattern 104 on the surface side is not a product but a sum because of the overlap of light. In the case of sum

のように、もとのフレネルゾーンプレートの格子の分布に、モアレ縞の分布を乗算した分布となる。したがって、その周波数スペクトルは、それぞれの周波数スペクトルの重なり積分で表される。そのため、たとえモアレのスペクトルが単独で鋭いピークをもったとしても、実際上、その位置にフレネルゾーンプレートの周波数スペクトルのゴーストが生じるだけである。つまり、スペクトルに鋭いピークは生じない。したがって、複数の入射角の光を入れても検出されるモアレ像のスペクトルは、常に表面側の第１の格子パターン１０４と裏面側の第２の格子パターン１０５との積のモアレだけであり、第２の格子パターン１０５が単一である以上、検出されるスペクトルのピークは１つの入射角に対して１つだけとなるのである。

As shown above, the distribution is obtained by multiplying the distribution of the grid of the original Fresnel zone plate by the distribution of moiré fringes. Therefore, the frequency spectrum is represented by the overlap integral of each frequency spectrum. Therefore, even if the moiré spectrum alone has a sharp peak, in practice, only a ghost of the frequency spectrum of the Fresnel zone plate occurs at that position. That is, no sharp peaks occur in the spectrum. Therefore, the spectrum of the moire image that can be detected even when light with a plurality of incident angles is input is always only the moire of the product of the first lattice pattern 104 on the front surface side and the second lattice pattern 105 on the back surface side. As long as the second grid pattern 105 is single, the detected spectrum has only one peak for one incident angle.

Here, the correspondence between the parallel light, which has been described so far for detection, and the light from an actual object will be schematically described with reference to FIG.

FIG. 13 is a diagram illustrating an angle formed by light from each point constituting the object with respect to the image sensor. Strictly speaking, the light from each point constituting the subject 401 is a spherical wave from a point light source, and is referred to as a modulator 102 and an image sensor 103 of the imaging device 101 in FIG. 1 (hereinafter, referred to as a lattice sensor integrated substrate 1301 in FIG. 13). Incident in. At this time, when the grid sensor integrated substrate 1301 is sufficiently small or sufficiently far from the subject 401, the incident angles of the light illuminating the grid sensor integrated substrate 1301 from each point can be regarded as the same.

Since the spatial frequency displacement Δu of the moire with respect to the minute angular displacement Δθ obtained from the equation (9) is 1 / S or less, which is the minimum resolution of the spatial frequency of the image sensor, the condition that Δθ can be regarded as parallel light is

のように表せる。この条件下であれば、無限遠の物体を本実施形態の撮像装置が撮像可能である。

Can be expressed as. Under this condition, the image pickup apparatus of this embodiment can image an object at infinity.

<Principle of shooting finite-distance objects>
Here, the imaging of an object at infinity described so far will be described again. FIG. 14 is a diagram illustrating that the front grid pattern is projected when the object is at an infinite distance. FIG. 14 shows how the first lattice pattern 104 on the front surface side is projected onto the back surface. The spherical wave from the point 1401 constituting the object at infinity becomes a plane wave while propagating a sufficiently long distance, irradiates the first lattice pattern 104 on the surface side, and the projected image 1402 is projected on the lower surface. To. In this case, the projected image has almost the same shape as the first grid pattern 104. As a result, by multiplying the projected image 1402 by the transmittance distribution of the lattice pattern on the back surface side (corresponding to the second lattice pattern 105 in FIG. 2), FIG. 15 (generated when the object is at an infinite distance). It is possible to obtain linear moire fringes at equal intervals as shown in (figure showing an example of moiré fringes).

On the other hand, imaging of a finite-distance object will be described. FIG. 16 is a diagram illustrating that the front grid pattern is enlarged when the object is at a finite distance. FIG. 16 shows how the first lattice pattern 104 on the front surface side is projected onto the back surface. The spherical wave from the point 1601 constituting the object irradiates the first lattice pattern 104 on the surface side, and the projected image 1602 is projected on the lower surface. In this case, the projected image is magnified almost uniformly. The enlargement ratio α is determined by using the distance d from the first grid pattern 104 to the point 1601.

のように算出できる。

It can be calculated as follows.

Therefore, if the transmittance distribution of the lattice pattern on the back surface side designed for parallel light is multiplied as it is, it is shown in FIG. 17 (a diagram showing an example of moire fringes generated when an object is at a finite distance). As such, linear moire fringes at equal intervals do not occur. However, if the second grid pattern 105 is magnified to match the shadow of the uniformly magnified first grid pattern 104 on the surface side, then again with respect to the magnified projected image 1602, FIG. 18 (object). As shown in the figure showing an example of moire fringes in which the backside lattice pattern is corrected when is at a finite distance), linear moire fringes at equal intervals can be generated. For this purpose, the correction can be performed by setting the coefficient β of the second lattice pattern 105 to β / α ^2.

This makes it possible to selectively develop the light from the point 1601 at a distance that is not necessarily infinity. This makes it possible to focus on an arbitrary position for shooting.

<Simplified configuration>
Next, a method for simplifying the configuration of the above-mentioned modulator 102 will be described. In the modulator 102, the first lattice pattern 104 and the second lattice pattern 105 having the same shape are formed on the front surface and the back surface of the lattice substrate 102a so as to be offset from each other. Then, the angle of the parallel light incident by the image processing is detected from the spatial frequency spectrum of the moire fringes to develop the image. The second grid pattern 105 on the back surface side is an optical element that closely adheres to the image sensor 103 and modulates the intensity of the incident light, and is the same grid pattern regardless of the incident light. Therefore, the configuration of the image pickup apparatus 101 may be changed as follows. The points different from FIG. 1 will be mainly described.

FIG. 19 is a diagram showing a configuration example of an image pickup apparatus 101 that realizes a backside grid pattern by image processing. The image pickup apparatus 101 includes a modulator 1901 that omits the second grid pattern 105, and an image processing unit 1902 that includes an intensity modulation unit 1903 that performs processing corresponding to the second grid pattern 105.

FIG. 20 is a diagram showing a configuration example of the modulator 1901. The modulator 1901 is composed of a lattice substrate 102a and a first lattice pattern 104 formed on the surface of the lattice substrate 102a. Compared with FIG. 2, the lattice pattern formed on the lattice substrate 102a can be reduced by one surface. As a result, the manufacturing cost of the modulator can be reduced, and the light utilization efficiency can be further improved.

FIG. 21 is a flowchart showing an example of image processing of the image processing unit 1902. The difference from the flowchart of FIG. 5 is the process of step 2101 before step 501. Since the processes of steps 501 to 506 are the same as the corresponding processes of FIG. 5, the description thereof will be omitted here.

In the process of step 2101, the intensity modulation unit 1903 described above generates a moire fringe image corresponding to the image output from the image sensor 103 passing through the lattice pattern 105 on the back surface side. Specifically, since the calculation corresponding to the equation (5) may be performed, the intensity modulation unit 1903 generates the grid pattern 105 on the back surface side and multiplies the image of the image sensor 103. Further, if the lattice pattern 105 on the back surface side is a binarized pattern as shown in FIG. 10 or 11, the intensity modulation unit 1903 simply sets the value of the image sensor 103 in the region corresponding to black to 0. Good. This makes it possible to suppress the computational load of multiplication operations and the scale of multiplication circuits.

In this case, the pitch of the pixels 103a (FIG. 20) of the image sensor 103 is fine enough to sufficiently reproduce the pitch of the first grid pattern 104, or the pitch of the first grid pattern 104 is the pixel 103a. It needs to be rough enough to be reproduced at the pitch of. When the grid pattern is formed on both sides of the grid substrate 102a, it is not always necessary that the pitch of the grid pattern can be resolved by the pixels 103a of the image sensor 103, and only the moire image can be resolved. However, when the grid pattern is reproduced by image processing, the grid pattern and the resolution of the image sensor 103 need to be the same.

Further, as described above, the intensity modulation unit 1903 has realized the processing corresponding to the second grid pattern 105. However, the present invention is not limited to this, and since the second grid pattern 105 is an optical element that closely adheres to the sensor and modulates the intensity of the incident light, the sensitivity of the sensor is effectively adjusted to the transmittance of the second grid pattern 105. It can also be realized by adding and setting.

<Noise cancellation>
In the explanation so far, the discussion has focused on the equation (6) in which only the components having sharp peaks are extracted from the equation (5), but in reality, the terms other than the fourth term of the equation (5) are noise. It becomes. Therefore, noise cancellation based on fringe scan is effective.

First, in the interference fringe intensity distribution of the equation (2), assuming that the initial phase of the first lattice pattern 104 is Φ _F and the initial phase of the second lattice pattern 105 is Φ _B , the equation (5) is:

のように表せる。ここで、三角関数の直交性を利用し、

Can be expressed as. Here, using the orthogonality of trigonometric functions,

式（１９）のように式（１８）をΦ_Ｆ、Φ_Ｂに関して積分すると、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。前述の議論から、これをフーリエ変換すれば、空間周波数分布にノイズのない鋭いピークを生じることになる。

When the equation (18) is _{integrated with respect to Φ F} and Φ _B as in the equation (19), the noise term is canceled and a term of a constant multiple of a single frequency remains. From the above discussion, if this is Fourier transformed, a sharp peak without noise will be generated in the spatial frequency distribution.

Here, equation (19) is shown in the form of integral, but in reality, the same effect can be obtained by calculating the sum of the combinations of _{Φ F} and Φ _B. Φ _F and Φ _B may be set so as to divide the angle between 0 and 2π into equal parts, and divide into 4 equal parts such as {0, π / 2, π, 3π / 2}, {0, π / 3 , 2π / 3} may be divided into three equal parts.

Further, equation (19) can be simplified. In equation (19), Φ _F and Φ _B are calculated so that they can be changed independently, but the noise term can be canceled even if the same phase is applied to the initial phases of _{Φ F} = Φ _{B, that is, the lattice patterns 104 and 105.} .. _{If Φ F} = Φ _B = Φ in equation (19),

となり、ノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。また、Φは０〜２πの間の角度を等分するように設定すればよく、｛０、π／２、π、３π／２｝のように４等分すればよい。

Therefore, the noise term is canceled and a term that is a constant multiple of a single frequency remains. Further, Φ may be set so as to divide the angle between 0 and 2π into equal parts, and may be divided into four equal parts such as {0, π / 2, π, 3π / 2}.

Further, the noise term can be canceled by using the orthogonal phases of {0, π / 2} without equal division, which can be further simplified. First, if the second grid pattern 105 is performed by the image processing unit 1902 as in the configuration of FIG. 19, a negative value can be handled by the grid pattern 105, so that the equation (18) is:

となる（Φ_Ｆ＝Φ_Ｂ＝Φ）。格子パターン１０５は既知であるため、この式（２１）から格子パターン１０５を減算し、Φ＝｛０、π／２｝の場合について加算すれば、

(Φ _F = Φ _B = Φ). Since the grid pattern 105 is known, if the grid pattern 105 is subtracted from this equation (21) and added for the case of Φ = {0, π / 2},

のようにノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。

The noise term is canceled and a term that is a constant multiple of a single frequency remains.

Further, as described above, the first grid pattern 104 and the second grid pattern 105 are separated from each other by shifting the two developed images generated in the spatial frequency space by _{δ 0 in advance.} However, this method has a problem that the number of pixels of the developed image is halved. Therefore, a method of avoiding duplication of developed images without shifting _{δ 0 will be described.} In the fringe scan of equation (19), instead of cos,

のようにｅｘｐを用い複素平面上で演算する。これによりノイズ項がキャンセルされ単一周波数の定数倍の項が残ることになる。式（２３）中のｅｘｐ（２ｉβδｘ）をフーリエ変換すれば、

Calculate on the complex plane using exp as in. As a result, the noise term is canceled and a term that is a constant multiple of a single frequency remains. If exp (2iβδx) in equation (23) is Fourier transformed,

となり、式（７）のように２つのピークを生じず、単一の現像画像を得られることが判る。このように、格子パターン１０４，１０５をずらす必要もなくなり、画素数を有効に使用可能となる。

Therefore, it can be seen that a single developed image can be obtained without generating two peaks as in the equation (7). In this way, it is not necessary to shift the grid patterns 104 and 105, and the number of pixels can be effectively used.

The configuration for performing the noise canceling method based on the above fringe scan will be described with reference to FIGS. 22 to 27. In the fringe scan, it is necessary to use at least a plurality of patterns having different initial phases as the grid pattern 104. To realize this, there are a method of switching patterns by time division (FIGS. 22 to 25) and a method of switching patterns by spatial division (FIGS. 26 to 27).

FIG. 22 is a diagram showing a configuration example of an image pickup apparatus 101 that realizes time-division fringe scanning. The imaging device 101 has a modulator 2201 instead of the modulator 102 of FIG. Further, the image pickup apparatus 101 has a modulator control unit 2202 and an image processing unit 2203.

FIG. 23 is a diagram showing an example of a grid pattern in a time-division fringe scan. The modulator 2201 is composed of, for example, a liquid crystal display element capable of electrically switching and displaying a plurality of initial phases shown in FIG. 23 (that is, the lattice pattern can be changed). In the patterns of FIGS. 23 (a) to 23 (d), the initial phase Φ _F or the phase difference Φ is {0, π / 2, π, 3π / 2}, respectively. FIG. 24 shows an example of electrode arrangement in the liquid crystal display element of the modulator 2201 that realizes this.

FIG. 24 is a diagram showing a configuration example of a modulator 2201 that realizes time division fringe scanning. The modulator 2201 is provided with concentric electrodes so as to divide one cycle of the lattice pattern into four, the electrodes are connected every four from the inside, and four electrodes are pulled out from the outer peripheral portion as drive terminals. ing. By temporally switching the voltage states applied to these four electrodes between the two states of “0” and “1” by the modulator control unit 2202, the initial phase Φ _F or Φ of the lattice pattern can be changed in FIG. 24 (a). It is possible to switch between {0, π / 2, π, 3π / 2} as in (d). It should be noted that, in FIG. 24, the shaded electrode to which "1" is applied blocks light, and the electrode to which "0" is applied, which is shown in white, transmits light.

FIG. 25 is a flowchart showing an example of image processing of the image processing unit 2203 that realizes time-division fringe scanning. The difference from the flowchart of FIG. 21 is the processing of steps 2501 to 2504 before step 501. Since the processes of steps 501 to 506 are the same as the corresponding processes of FIG. 21, the description thereof will be omitted here.

First, the image processing unit 2203 resets the addition result at the beginning of the fringe scan calculation (2501). Next, in the case corresponding to the equation (20), the image processing unit 2203 sets the same initial phase as the grid pattern 104 used for photographing (2502), and generates a grid pattern 105 having the initial phase. , Multiplies the output image of the image sensor 103 (2101). The image processing unit 2203 adds the multiplication result for each pattern of each initial phase (2503). The image processing unit 2203 repeats the above steps 2502 to 2503 for the number of patterns of all initial phases (NO in 2504). When the processing is completed for all the phases (YES in 2505), the image processing unit 2203 advances the processing to step 501. Although the above flowchart has been described by taking the equation (20) as an example, it can be similarly applied to the equation (19), the equation (22), and the equation (23).

FIG. 26 is a diagram showing a configuration example of the image pickup apparatus 101 that realizes the spatial division fringe scan. The imaging device 101 has a modulator 2601 instead of the modulator 102 of FIG. Further, the image pickup apparatus 101 has an image division unit 2602 and an image processing unit 2203.

FIG. 27 is a diagram showing an example of a grid pattern in the spatial division fringe scan. The modulator 2601 has a plurality of initial phase patterns arranged two-dimensionally, for example, the initial phase ΦF or Φ in FIG. 27 is {0, π / 2, π, 3π / 2}, respectively. The image dividing unit 2602 divides the output image of the image sensor 103 into a region image corresponding to the pattern arrangement of the modulator 2601, and sequentially transmits the image to the image processing unit 2203. In the example of FIG. 26, the image sensor output is divided into 2 × 2 regions. Since the fringe scan based on equation (20) requires four phases, the modulator 2601 has a 2 × 2 pattern arrangement. Since the fringe scan based on the equation (22) can be realized in two phases, the modulator 2601 can be realized in a 1 × 2 pattern arrangement, and the image sensor output is also divided into 1 × 2 regions accordingly. Since the subsequent processing of the image processing unit 2203 is equivalent to the processing of FIG. 22 which is a time-division fringe scan, the description thereof will be omitted.

By using this spatial division fringe scan, it is not necessary to electrically switch the grid pattern as in the time division fringe scan modulator 2201, and the modulator can be manufactured at low cost. However, when the spatial division fringe scan is used, the image is divided and the resolution is effectively lowered. Therefore, when it is necessary to increase the resolution, time-division fringe scanning is suitable.

<Problems in close-up photography>
Next, problems during close-up photography will be described. First, redefine the angle of view. In the above description, the peak position u (Equation (9)) of the spatial frequency spectrum of the moire fringes of the light beam having the incident angle θ is larger than the upper limit frequency N / (2S) determined by the pixel pitch of the image sensor 103. In some cases, the angle of view θ _max was defined (Equation (13)). On the other hand, when β is small, or the pixel pitch (S / N) of the image sensor 103 is small, and the peak position u is smaller than the upper limit frequency, the definition of the angle of view is different.

FIG. 28 is a diagram for explaining the _{angle of view θ max2.} And reference coordinates of concentric grating pattern constituting the first grating pattern 104, theta _max2 is the slope of the straight line passing through the edges of the image sensor 103, the maximum angle of view can be developed. Therefore, the angle of view θ _max2 is determined by using the size S of the image sensor 103 and the thickness t of the modulator in imaging at a distance d away from the first grid pattern.

のように表せる。また、視野の直径Ａは、θ_ｍａｘ２を用いて、

Can be expressed as. Further, the diameter A of the visual field is set to θ _max2 by using θ max2.

となる。

Will be.

Here, at the time of close-up photography, there is a problem that the field of view is narrowed due to the scattering angle from the object to be imaged and the CRA (Chief Ray Angle) characteristic of the image sensor.

First, the limitation of the field of view due to the scattering angle from the object to be imaged will be described. FIG. 29 is a diagram illustrating a range and a field of view of a grid pattern irradiated by light from a certain point constituting the object when the object is at a short distance. FIG. 29 shows the spread angle (scattering angle) θ _s of light from a certain point of an object. When the scattered light from the points 2901 to 2903 constituting the object irradiates the first lattice pattern 104, the diameter B of the irradiation region in the first lattice pattern 104 is determined by using the _{scattering angle θ s.}

となる。ｄは点２９０１乃至点２９０３から第１の格子パターン１０４までの距離である。このように、接写時には第１の格子パターン１０４のうち照射領域しか使用することができない。なお、現実には散乱光強度は散乱角が大きくなるに従い徐々に減衰するものであるが、図２９では簡単化のため、照射領域のみ散乱光が到達しているとしている。

Will be. d is the distance from the points 2901 to 2903 to the first grid pattern 104. As described above, at the time of close-up photography, only the irradiation region of the first lattice pattern 104 can be used. In reality, the scattered light intensity gradually attenuates as the scattering angle increases, but in FIG. 29, for the sake of simplicity, it is assumed that the scattered light reaches only the irradiation region.

As described above, imaging is possible when the incident light passes through the reference coordinates of the concentric grid patterns constituting the first grid pattern 104. Therefore, while it is possible to image the points 2901 and 2902 in which the scattered light passes through the reference coordinates, it is not possible to image the points 2903 and the like that do not pass through the reference coordinates. As shown in FIG. 29, when the θ _s ≦ θ _max2 is the angle of view is determined by the scattering angle theta _s, the diameter _{A s} field of view,

となる。

Will be.

Next, the limitation of the field of view due to the CRA characteristics of the image sensor will be described. Depending on the arrangement of the wiring or the like on the front surface or the back surface of the light receiving element and the design of the microlens array which is usually arranged on the front surface of the light receiving element, the amount of received light of the image sensor has an angle dependence called CRA characteristic. Therefore, assuming that the incident light can be received up _{to the incident angle θ i based on the CRA characteristics, the angle of view is determined by θ i} in the imaging using an image sensor such that _{θ i} ≤ θ _{max 2,} and the field of view A _i Is

となる。ただし、θ_ｓ≦θ_ｉの場合の視野は式（２８）となる。

Will be. However, _{the field of view in the case of θ s} ≤ θ _i is given by Eq. (28).

The angle of incidence θ _i should be determined by an angle that is half-width of the CRA characteristic, but it is not limited to this, and if the angle is designed to be smaller than half-width of half-value, peripheral illumination is less likely to occur, and it is more than half-width of half-value. If you design it at a large angle, the field of view will be large.

<How to expand the field of view during close-up photography>
Therefore, a method of expanding the field of view during close-up photography will be described.

FIG. 30 is a diagram showing a configuration example of an imaging device that realizes a wide field of view by combining a compound eye grid pattern and a plurality of image sensors. Unlike FIG. 19, the image pickup apparatus 101 includes a modulator 3001 in which a compound eye grid pattern 3003 is formed, an image sensor 3004, and an image processing unit 3006 including an image synthesis unit 3007. The compound eye grid pattern 3003 is configured by arranging a plurality of basic patterns (basic grid patterns) 3002 composed of concentric grid patterns in which the pitch of stripes becomes narrower from the center to the edges. The image sensor 3004 is configured by arranging a plurality of image sensors 3005.

FIG. 31 is a diagram showing an example of a compound eye grid pattern. In this example, the compound eye grid pattern 3003 is realized by arranging the basic patterns 3002 in 3 × 3 without overlapping each other. In this case, each image sensor 3005 is arranged in 3 × 3 so as to correspond to each basic pattern 3002.

FIG. 32 is a diagram illustrating a grid pattern of the compound eyes shown in FIG. 30 and a field of view by a plurality of image sensors. Among the plurality of basic patterns 3002 and a plurality of image sensors 3005, a combination n-th field by the n-th basic pattern 3002 and the n-th image sensor 3005 _{A n,} the angle theta _n, n-th and n + 1 th _{Let n be} the distance between the centers of the basic pattern 3002, and N be the number of the plurality of basic patterns 3002 and the plurality of image sensors 3005 in the one-dimensional direction. At this time, the composite field of view _AN is

となる。つまり、基本パターン３００２と画像センサ３００５の組み合わせ（基本ユニットともいう）を複数用いることで、それぞれの組み合わせで撮像できる視野を合成し拡大することができる。

Will be. That is, by using a plurality of combinations (also referred to as basic units) of the basic pattern 3002 and the image sensor 3005, it is possible to synthesize and enlarge the field of view that can be imaged by each combination.

However, if the distance d from the compound-eye grating pattern 3003 to the imaging object is short, the field of view of the basic pattern 3002 is reduced, a gap in the synthesis field A _N. To obtain a field of view continuous gap does not occur, _{A n} is as long or _{q n,} may be set to d and _{q n} which satisfies _{A n = 2d · tanθ n ≧} q n.

Subsequently, image processing for imaging using the above-mentioned image sensor 3004 and compound eye grid pattern 3003 will be described with reference to FIGS. 33 to 35.

FIG. 33 is a flowchart showing an example of image processing of the image processing unit 3006. The difference from the flowchart of FIG. 21 is the process of step 3301 added between step 504 and step 505. The image processing unit 3006 performs the processes of steps 2101 to 504 for each of the sensor images output by the plurality of image sensors 3005. In step 3301, the image synthesizing unit 3007 of the image processing unit 3006 rearranges the output images of step 504 according to the arrangement of each image sensor 3005 and synthesizes them.

34 and 35 are diagrams illustrating an example of the synthesis method of step 3301. _{A N} in FIG. 34 is a synthetic field _{A N} in FIG. 32, _{A n} and _{A n + 1} is the field _{A n} and field _{A n + 1} by n-th and (n + 1) th basic pattern 3002 in FIG. 32. Further, the output image of step 504 in FIG. 33 is represented as an output image 3401 and an output image 3402. FIG. 35 shows a composite image of the output image 3401 and the output image 3402. In step 3301, n-th and (n + 1) th by the number of pixels corresponding to the center distance q _n of the basic pattern 3002, obtained one of the composite image by summing shifting at least one of the output image 3401 and the output image 3402 Be done.

Next, the brightness correction after composition will be described with reference to FIG. 35. The area 3501 indicates a region where the _{field of view An} and the field of view _{An + 1} of the output image 3401 and the output image 3402 overlap. The brightness of the region 3501 after composition is the sum of the output image 3401 and the output image 3402. Therefore, it is necessary to correct the brightness of the area 3501 in order to eliminate the difference in brightness between the area 3501 and the other areas where the visual fields do not overlap and to obtain a smooth brightness distribution. When the brightness of the region 3501 of the output image 3401 and the brightness of the region 3501 of the output image 3402 match, it can be corrected by halving the brightness of the combined region 3501.

However, since the actual brightness has a distribution that attenuates from the center to the edge due to the influence of CRA or the like, the brightness of the region 3501 of the output image 3401 and the region 3501 of the output image 3402 do not match, and the brightness after synthesis. Even if is set to 1/2, the brightness difference cannot be corrected. Therefore, of either the output image 3401 or the output image 3402, the _{region 3501 in which the field of view An} and the field of view _{An + 1} overlap is cut out and combined, or both images are pixels corresponding _{to q n / 2 from the edge.} All you have to do is add the images that have been cut out by the number and have no overlapping parts. As a result, the difference in brightness can be reduced without correcting the brightness.

FIG. 36 is a flowchart showing another example of image processing of the image processing unit 3006. The difference from the flowchart of FIG. 33 is the processing of step 3601 and step 3602 before step 501. In FIG. 36, step 3301 is omitted. Image synthesis unit 3007 of the image processing unit 3006, in step 3601, a plurality of sensor image output from the plurality of image sensors 3005, n-th and the number of pixels corresponding to the center distance _{q n} (n + 1) th basic pattern 3002 only Shift and add them together to synthesize. In step 3602, the image processing unit 3006 performs intensity modulation on the composite image of step 3601 using a second compound eye grid pattern composed of a plurality of patterns corresponding to each of the plurality of basic patterns 3002.

Although the configuration related to the compound eye grid pattern of FIG. 30 has been described with reference to FIG. 19, it is also possible to cancel noise by fringe scanning by similarly applying this configuration to other configurations such as FIG. 22.

Further, in FIG. 32 and the like, the basic pattern 3002 and the image sensor 3005 have been described by focusing on the arrangement in the one-dimensional direction, but this configuration can also be applied to the two-dimensional arrangement by extending the arrangement in the orthogonal direction as well. is there.

Further, the number of basic patterns 3002 is not limited to those shown in the figure. Since the field of view is determined by the number of basic patterns 3002, it may be appropriately changed according to the size of the imaging target.

Further, since the field of view becomes wider when the distance from the basic pattern 3002 to the imaging target is increased, the number of basic patterns 3002 can be reduced in this way.

Further, FIG. 37 is a diagram illustrating an arrangement example of each image sensor 3005. FIG. 37 shows an arrangement in which the directions (wiring directions) of the image sensors 3005 are aligned. In a general image sensor, wiring for transmitting a signal comes out from the side of the light receiving surface. In order to arrange a plurality of image sensors 3005, it is necessary to leave a space between the image sensors 3005 so that the wirings do not interfere with each other or the wirings overlap with the light receiving surface 3701, and the light receiving surfaces may not be brought close to each other. is there.

FIG. 38 is a diagram illustrating another arrangement example of the image sensor. FIG. 38 shows an arrangement in which the orientation (wiring orientation) of each image sensor 3005 faces the outside of the image sensor 3004. The position of the wiring differs depending on the type of the image sensor, but by arranging the orientation of at least some of the image sensors 3005 to be different from the orientation of the other image sensors 3005, the restriction of the distance between the light receiving surfaces due to the wiring is avoided. be able to.

According to the above method and configuration, by using a plurality of combinations of the image sensor 3005 and the basic pattern 3002, it is possible to synthesize and enlarge the field of view that can be imaged by each.

According to the present embodiment, it is possible to provide an image pickup apparatus having a simple structure, being thin, and having a wide field of view at the time of close-up photography.

[Second Embodiment]
<Optimization of image sensor size>
In this embodiment, a method of efficiently using pixels and reducing the amount of calculation by optimizing the size of the image sensor 3005 will be described.

FIG. 39 is a diagram illustrating a method of optimizing the size of the image sensor. The size and number of light receiving surfaces of the image sensor 3005 required for imaging using the compound eye grid pattern 3003 will be described with reference to FIG. 39. In the following description, the image sensor 3005 corresponds to the image sensor 3901 or the image sensor 3903.

Region C, when the angle theta _n is determined by theta _s or _{θ i (θ n ≦ θ max2} ), indicates a region on the light receiving surface of the image sensor 3901 required to obtain a visual field _{A n.} A ray having an incident angle of θ _n is incident on the region C on the image sensor 3901. In order to obtain the field of view _An , the size S of the light receiving surface of the image sensor 3901 may be set to C or more, and when S and C are equal, S can be minimized and the light receiving surface can be used efficiently. Further, by minimizing the light receiving surface of the image sensor 3901, the number of pixels other than the region C unnecessary for development can be reduced, so that the amount of calculation can be reduced.

Here, C uses the angle of view θ _n and the distance t from the compound eye grid pattern 3003 to the image sensor 3901.

と求められる。よって、ｔを小さくし、Ｃを小さくすることで、Ｓを小さくすることができる。

Is required. Therefore, S can be reduced by reducing t and C.

_{Further, S} can be reduced by reducing the scattering angle θ s of the imaging target, or by using an image sensor 3901 having a small _{incident angle θ i that} can be received by the image sensor 3901 determined by the CRA characteristics. However, in this case, the field of view _An also becomes small. Therefore, as described in the first embodiment, it is necessary to widen the field of view by increasing the number of combinations of the image sensor 3005 and the basic pattern 3002.

According to the above method and configuration, it is possible to optimize the size and number of the image sensors 3005 and reduce the amount of calculation.

In FIG. 39, the image sensor 3901 is arranged so as to have a one-to-one correspondence with the basic pattern 3902, but for example, even if one image sensor 3903 that receives the light rays of the _{field of view An + 1} and the field of view _{An + 2 is arranged.} Good. In this case, the number of image sensors 3005 can be reduced. Further, the image sensor 3901 and the image sensor 3903, which have different sizes, may be arranged in combination.

[Third Embodiment]
<Design of gaze direction>
In this embodiment, the gaze direction in imaging using the compound eye grid pattern 3003 will be described.

FIG. 40 is a diagram illustrating an example of a gaze direction of imaging by the compound eye grid pattern 3003. A _n represents the field of view can be obtained by the n-th basic pattern 4001 when angle by the equation (25) is determined. Further, here, the reference coordinates of the concentric grid patterns constituting the basic pattern 4001 and the center coordinates of the light receiving surface of the image sensor 3005 do not match. The center coordinates of the light receiving surface of the image sensor 3005 are the intersections of the diagonal lines connecting the four corners of the light receiving surface. As described in the first embodiment, the angle of view determined by the equation (25) is the slope of a straight line passing through the edge of the light receiving surface of the image sensor 3004 and the reference coordinates of the concentric grid pattern constituting the basic pattern 4001. Become. Thus, gaze direction is the vector connecting the two points and the center of the reference coordinates and the field A _n of concentric grating pattern constituting the basic pattern 4001. That is, the gaze direction can be changed by setting the reference coordinates of the concentric grid patterns constituting the basic pattern 4001 and the positions of the center coordinates of the light receiving surface of the image sensor 3004 at the time of design.

FIG. 41 is a diagram illustrating another example of the gaze direction, and shows a configuration for changing the gaze direction for each basic pattern. The difference between the configuration of FIG. 40 and the configuration of FIG. 41 is the position of the reference coordinates of each concentric grid pattern constituting the basic pattern 4101 and the basic pattern 4102. In FIG. 41, these reference coordinates are aligned in the direction in which the distance l between the centers of the concentric grid patterns of the basic pattern 4101 and the basic pattern 4102 approaches. By changing the center coordinates of the concentric grid patterns for each basic pattern in this way, it is possible to change the gaze direction of imaging by each basic pattern. Depending on the type of the image sensor 3005, the center coordinates of the light receiving surface of the adjacent image sensor 3004 may not be brought close to each other due to reasons such as the arrangement of the wiring protruding from the light receiving surface. Even in such a case, by changing the position of the reference coordinates of the basic pattern 3002, it is possible to adjust the field of view A _N in the imaging target, it is possible to obtain a continuous visual field A _N.

According to the above method / configuration, the gaze direction can be changed by setting the reference coordinates of the concentric grid patterns constituting the basic pattern 3002, and even when there are restrictions on the arrangement method of the image sensor 3005. It is possible to direct the field of view to the imaging target.

[Fourth Embodiment]
In this embodiment, imaging using an elliptical grid pattern as the basic pattern 3002 will be described.

The light receiving surface of the image sensor 3005 may be rectangular depending on the type. FIG. 42 is a diagram showing an example of a concentric grid pattern projected on a rectangular light receiving surface. In FIG. 42, a concentric grid pattern in which the pitch of the stripes becomes narrower from the center to the edges is used as the basic pattern 3002. In this case, the number of projected circles differs between the long side direction and the short side direction of the light receiving surface, and a grid pattern having a finer pitch is projected in the long side direction.

As described in the first embodiment, the fineness of the pitch of the concentric lattice pattern is determined by the coefficient β, and the resolution after development is determined by β as shown in the equation (12). That is, the resolution after development depends on the fineness of the pitch of the pattern projected on the light receiving surface. Therefore, when the fineness of the pitch in the long side direction and the pitch in the short side direction do not match as in the projection pattern of FIG. 42 (the finer pitch pattern on the outside in the long side direction does not exist in the short side direction), , The resolution after development will not match. For the above reasons, when a grid pattern concentric with respect to a rectangular light receiving surface is used, there arises a problem that the resolution in the short side direction is lower than that in the long side direction.

FIG. 43 is a diagram showing an example of an elliptical grid pattern projected on a rectangular light receiving surface. In FIG. 43, an elliptical grid pattern in which the pitch of the stripes becomes narrower from the center to the edges is used as the basic pattern 3002. The major axis direction and the minor axis direction of the ellipse are set so as to coincide with the long side direction and the short side direction of the rectangle, respectively. Further, the lengths of the long axis and the short axis of the ellipse may be set according to the aspect ratio of the long side and the short side of the rectangle, or may be set regardless of the aspect ratio. Also, the pitch of the stripes in the minor axis of the ellipse is set to be smaller than the pitch of the corresponding stripes in the major axis (that is, the minimum value of the pitch of the stripes in the minor axis of the ellipse is long. Does not match the minimum pitch of the corresponding stripes in the axial direction). In this case, the number of ellipses projected in the short side direction is increased as compared with FIG. 42, and the number of ellipses obtained in the long side direction and the short side direction are the same. That is, by using the elliptical grid pattern, it is possible to prevent a decrease in resolution in the short side direction even when the image sensor 3005 having a rectangular light receiving surface is used.

Further, the diffraction of the incident light in the concentric lattice pattern in which the pitch of the stripes becomes narrower from the center to the edge causes a factor of lowering the contrast of the projection pattern, which causes a decrease in resolution and noise. Since the effect of diffraction depends on the pitch of the stripes of the lattice pattern, when the lattice pattern of FIG. 43 is used, the effect of diffraction differs between the long side direction and the short side direction. On the other hand, the pitch of the stripes in the minor axis and the pitch of the corresponding stripes in the major axis match (that is, the minimum value of the pitch of the stripes in the minor axis and the minimum pitch of the corresponding stripes in the major axis direction). By using an elliptical grid pattern (which matches the value) and aligning the pitch of the stripes of the grid pattern projected on the rectangular light receiving surface in the long side direction and the short side direction of the light receiving surface, the long side direction and the short side direction are aligned. The effect of diffraction can be made constant in the side direction.

Note that FIG. 44 is a diagram showing another example of the concentric grid pattern projected on the rectangular light receiving surface. FIG. 4443 shows a case where the pitch of the pattern in the long side direction and the pattern in the short side direction is made constant by using a concentric grid pattern. By using a concentric grid pattern, it is possible to make the influence of diffraction in the long side direction and the short side direction constant, but in that case, the area where the pattern is projected on the light receiving surface is small and limited. Therefore, the pixels on the light receiving surface cannot be used efficiently. On the other hand, by using the elliptical grid pattern, the pixels of the entire light receiving surface can be used, and the number of points for sampling fringes having the same pitch as the concentric grid pattern can be increased, so that the image quality can be improved. it can.

According to the above method and configuration, when the image sensor 3005 having a rectangular light receiving surface is used, it is possible to prevent a decrease in resolution in the short side direction.

[Fifth Embodiment]
In this embodiment, a configuration for realizing a thin finger vein recognition device capable of close-up photography will be described. Hereinafter, the finger vein imaging device that constitutes a part of the finger vein recognition device will be described.

In order to realize the finger vein authentication device, it is necessary to secure a field of view for obtaining an image necessary for authentication in the finger vein image pickup device. FIG. 45 is a diagram showing a configuration example of a finger vein imaging device using a compound eye grid pattern. The finger vein imaging device includes a modulator 4501 in which a compound eye grid pattern 4502 including a plurality of basic patterns 4503 is formed, a plurality of image sensors 4504, and a plurality of light sources 4505.

FIG. 45 shows a configuration required to obtain _{a continuous field of view AN} = 12 mm × 10 mm when the distance d from the imaging target to the compound eye grid pattern 4502 is 2 mm. Each basic pattern 4503 is arranged in 2 × 2 with an interval of center-to-center distance qc = 7.5 mm and qr = 5.0 mm, and each image sensor 4504 is also arranged in the number and arrangement of each basic pattern 4503. Correspondingly, 2 × 2 are arranged.

Here, when d = 2 mm and the angle of view θ _i obtained from the CRA characteristics of each image sensor 4504 is θ _i = 50 °, the formula is based on N = 2, qc = 7.5 mm and qr = 5.0 mm. (30) than it can be _seen that a N = 12mm × 10mm, and the visual field of the object is obtained.

Further, in the configuration of FIG. 45, since the light receiving surface of the image sensor 4504 is rectangular, the pattern constituting the basic pattern 4503 is an elliptical lattice pattern, and as described in the fourth embodiment, the developed image. It is possible to prevent the resolution from decreasing in the direction of the short side of.

Further, a near infrared light source can be used as the light source 4505. In FIG. 45, the light source 4505 irradiates the finger to be imaged from below. Of course, the light source 4505 may be arranged above the finger, and the light transmitted through the finger may be acquired by the image sensor 4504.

Further, in order to perform the time division fringe scan described in the first embodiment, the modulator 4501 may be configured by the liquid crystal display element as shown in FIG. 24. FIG. 46 is a cross-sectional view showing a configuration example of a liquid crystal display element as the modulator 4501. The liquid crystal display element is mainly composed of a liquid crystal layer 4601, two polarizing plates 4602 arranged on the upper side and a lower side of the liquid crystal layer 4601, and two glass substrates 4603 arranged so as to sandwich the liquid crystal layer 4601. .. As the polarizing plate used for a general liquid crystal display or the like, a polarizing plate corresponding to visible light is used, but as the polarizing plate 4602 used for the modulator 4501, it is necessary to use a polarizing plate corresponding to the wavelength of the light source 4505. The two polarizing plates 4602 have transmission and absorption characteristics depending on the polarization direction of the light having the wavelength of the light source 4505.

Further, FIG. 47 is a diagram showing a configuration example of a finger vein imaging device using a monocular lattice pattern. Although the configuration using the compound eye grid pattern 4502 has been described with reference to FIG. 45, a finger vein imaging device may be used by using the monocular grid pattern 4702 and the image sensor 4703. In that case, in order to obtain the image required for authentication by setting a large distance d from the subject to the grid pattern 4702 and expanding the field of view determined by the equation (26), the equation (28) or the equation (29). The field of view can be secured. The image sensor 4703 may be configured by arranging a plurality of image sensors as shown in FIG. 45.

According to the above configuration, it is possible to secure the field of view required for the finger vein recognition device even in close-up photography and to realize a thin finger vein recognition device.

The present invention has been described above using a plurality of embodiments. Of course, the present invention is not limited to the above embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail in order to explain the present invention in an easy-to-understand manner, and is not necessarily limited to the one including all the described configurations.

Further, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add / delete / replace a part of the configuration of each embodiment with another configuration.

Further, each of the above configurations, functions, processing units, processing means and the like may be realized by hardware by designing a part or all of them by, for example, an integrated circuit. Further, each of the above configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a memory, a hard disk, a recording device such as an SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD (Digital Versaille Disc). Can be placed.

In addition, the control lines and information lines indicate those that are considered necessary for explanation, and do not necessarily indicate all the control lines and information lines in the product. In practice, it can be considered that almost all configurations are interconnected.

The present invention is not limited to the imaging device and the imaging method, but may be in various aspects such as an imaging system, a developing method, a computer-readable program, an image processing circuit, a finger vein authentication device, a finger vein imaging device, and a finger vein authentication method. Can be provided.

101 ... Imaging device, 102 ... Modulator, 102a ... Lattice substrate, 102b ... Support member, 103 ... Image sensor, 103a ... Pixel, 104 ... First lattice pattern, 105 ... Second lattice pattern, 106 ... Image processing unit , 107 ... Image display unit, 401 ... Subject, 1201 ... Cylindrical lens, 1301 ... Lattice sensor integrated substrate, 1401 ... Point, 1402 ... Projected image, 1601 ... Point, 1602 ... Projected image, 1901 ... Modulator, 1902 ... Image processing Unit, 1903 ... Intensity modulator, 2201 ... Modulator, 2203 ... Image processing unit, 2202 ... Modulator control unit, 2601 ... Modulator, 2602 ... Image division unit, 2901 ... Point, 2902 ... Point, 2903 ... Point, 3001 ... Modulator, 3002 ... Basic pattern, 3003 ... Compound eye grid pattern, 3004 ... Image sensor, 3005 ... Image sensor, 3006 ... Image processing unit, 3007 ... Image compositing unit, 3401 ... Output image, 3402 ... Output image, 3501 ... Area , 3701 ... light receiving surface, 3901 ... image sensor, 3902 ... basic pattern, 3903 ... image sensor, 4001 ... basic pattern, 4002 ... basic pattern, 4101 ... basic pattern, 4102 ... basic pattern, 4501 ... modulator, 4502 ... compound eye lattice Pattern, 4503 ... Basic pattern, 4504 ... Image sensor, 4505 ... Light source, 4601 ... Liquid crystal layer, 4602 ... Plate plate, 4603 ... Glass substrate, 4702 ... Lattice pattern, 4703 ... Image sensor

Claims (8)

A modulator that modulates the light intensity using a compound eye grid pattern with multiple basic grid patterns,
A plurality of image sensors that convert the light transmitted through the modulator into image data and output it,
An image processing unit that performs image processing of a plurality of image data output from the plurality of image sensors, and an image processing unit.
Equipped with
The basic grid pattern and the image sensor are composed of a plurality of basic units having a one-to-one correspondence.
The plurality of basic units have different fields of view.
An imaging device characterized by this. The imaging device according to claim 1.
The gaze direction can be set by the reference coordinates of the concentric patterns constituting the basic lattice pattern and the center coordinates of the image sensor.
An imaging device characterized by this. A modulator that modulates the light intensity using a compound eye grid pattern with multiple basic grid patterns,
A plurality of image sensors that convert the light transmitted through the modulator into image data and output it,
An image processing unit that performs image processing of a plurality of image data output from the plurality of image sensors, and an image processing unit.
Equipped with
The size of each of the plurality of image sensors is the distance from the compound eye lattice pattern to the image sensor, the number of the image sensors, the scattering angle of light from the image pickup object, or the incident light intensity of the image sensor. Determined based on the angle of view determined by the angle dependence,
An imaging device characterized by this. The imaging device according to claim 3.
The angle of view determined by the number of basic lattice patterns, the scattering angle of light from the imaging object, or the angle dependence of the incident light intensity of the image sensor, and the distance from the imaging object to the compound eye lattice pattern. The field of view is determined based on
An imaging device characterized by this. A modulator that modulates the light intensity using a compound eye grid pattern with multiple basic grid patterns,
A plurality of image sensors that convert the light transmitted through the modulator into image data and output it,
An image processing unit that performs image processing of a plurality of image data output from the plurality of image sensors, and an image processing unit.
Equipped with
The basic lattice pattern is composed of a plurality of elliptical patterns.
The plurality of elliptical patterns have the same minimum pitch between the plurality of elliptical patterns in the major axis direction and the minor axis direction.
An imaging device characterized by this. A modulator that modulates the light intensity using a compound eye grid pattern with multiple basic grid patterns,
A plurality of image sensors that convert the light transmitted through the modulator into image data and output it,
An image processing unit that performs image processing of a plurality of image data output from the plurality of image sensors, and an image processing unit.
Equipped with
The basic lattice pattern is composed of a plurality of elliptical patterns.
The lengths of the major axis and the minor axis of the plurality of elliptical patterns are determined according to the aspect ratio of the light receiving surface of the image sensor.
An imaging device characterized by this. The imaging device according to claim 1.
It is equipped with a light source that irradiates the object to be imaged with near-infrared light.
The image sensor is an image pickup apparatus characterized in that the near-infrared light emitted by the light source has a light receiving sensitivity. The imaging device according to claim 7.
The modulator is a liquid crystal display element.
A first polarizing plate between the image pickup object and the liquid crystal layer and a second polarizing plate between the liquid crystal layer and the image sensor are provided.
The first polarizing plate and the second polarizing plate have transmission and absorption characteristics with respect to the wavelength of the light source.
An imaging device characterized by this.

Images