JP4843640B2 - 3D information generation apparatus and 3D information generation program - Google Patents

Description

  The present invention relates to a three-dimensional information generation apparatus and a three-dimensional information generation program for generating three-dimensional information of a subject imaged by an integral photography method.

  In general, as one of the stereoscopic image display methods that allow a stereoscopic image to be viewed freely from an arbitrary viewpoint, integral photography (hereinafter referred to as integral photography) using a convex lens group or a pinhole group arranged in a plane is used. IP) systems are known (for example, Patent Document 1).

Here, the concept of the IP method will be described with reference to FIG. FIGS. 12A and 12B are explanatory diagrams for explaining the concept of the conventional IP system. FIG. 12A shows a state at the time of imaging, and FIG. As shown in FIG. 12A, a conventional IP image stereoscopic image capturing apparatus (stereoscopic information generating apparatus) 100 includes a lens group including a plurality of minute convex lenses 101 ₁ ,..., 101 _n arranged in the same plane. 101 and an imaging plate 102 such as a CCD (Charge Coupled Device) are arranged in parallel. Here, when the subject H is picked up by the stereoscopic image pickup device 100 from the observer O side, the subject H passes through the lens group 101 and the number of minute convex lenses 101 ₁ ,..., 101 _n constituting the lens group 101. Only images (hereinafter referred to as “element images”) 103 ₁ ,..., 103 _n are captured on the imaging plate 102.

  Then, the stereoscopic image capturing apparatus 100 clips each element image from the image captured by the imaging plate 102 by the image clipping unit 104. Further, if each element image is displayed as it is, the depth is reversed, so that the stereoscopic image capturing apparatus 100 converts the element image into point symmetry by the inversion processing unit 105, so Element image).

On the other hand, as shown in FIG. 12B, a stereoscopic image display apparatus 200 according to the conventional IP system includes a lens group 201 composed of a plurality of minute convex lenses 201 ₁ ,..., 201 _n arranged in the same plane, 12 (a) elemental image ₁₀₃ 1 captured by the imaging plate 102, ..., 103 elemental image ₂₀₃ 1 converted _n in point symmetry with, ..., 203 display _{panel n} display (display device) 202 parallel to the It is arranged. As the display plate 202, an LCD (Liquid Crystal Display) or the like that can display a high-definition image is usually used.

12A shows a state in which the light beam emitted from the subject H stops at the imaging plate 102, the light beam traveling to the right side of the imaging plate 102 in FIG. 12A. 12B is reproduced as a light beam traveling from the display plate 202 through the lens group 201 in FIG. That is, when the observer O observes the display plate 202 through the lens group 201, a state equivalent to a state in which the subject H is present at the position of the stereoscopic reproduction image M and light is emitted therefrom is reproduced. It becomes possible to visually recognize the image.
JP 2002-228974 A (paragraphs 0023-0028, FIG. 1)

  In the conventional method described above, the stereoscopic image capturing device acquires the direction of the light beam emitted from the subject as an element image, and the stereoscopic image display device reproduces the direction of the light beam from the subject by the element image, thereby An image was generated. Therefore, in the stereoscopic image capturing apparatus, it is necessary to correctly acquire the direction of light rays emitted from each point of the subject. This means that the subject image (element image) generated by each lens (microconvex lens) constituting the lens group must be acquired on the imaging plate with high resolution.

  However, since the number of pixels of the imaging plate is finite, the number of pixels allocated to the subject imaged by each lens is very small. For example, if the number of pixels of the image sensor is 1000 (horizontal) × 1000 (vertical) pixels and the number of lenses constituting the lens group is 50 (horizontal) × 50 (vertical), each lens is generated. The number of pixels constituting the image to be set is very small, 20 (horizontal) × 20 (vertical) pixels.

  For this reason, in the conventional method, when the subject to be imaged exists at a position away from the lens group, sufficient resolution for recognizing the subject in each elemental image cannot be obtained, and the stereoscopic image of the subject is obtained. There is a problem that the three-dimensional information (element image) necessary for reproducing the image cannot be generated.

  The present invention has been made to solve the above-described problems, and can generate high-resolution three-dimensional information (element image) even with an imaging board (imaging device) having a small number of pixels. An object of the present invention is to provide a three-dimensional information generation apparatus and a three-dimensional information generation program.

  The present invention was created to achieve the above object, and first, the three-dimensional information generation apparatus according to claim 1, an elemental image obtained by imaging a subject via a plurality of element lenses In the three-dimensional information generation device that generates the three-dimensional information of the subject in the graphy, an opening is provided in the center of the imaging surface, and a plurality of element lenses are arranged in a two-dimensional manner in the peripheral area of the opening, and the element lens A first image pickup unit provided with a first image pickup element that picks up an optical image of a subject at a position opposite to the first image pickup unit provided with an imaging lens at the opening, and the imaging range facing the imaging lens An image pickup unit having a second image pickup unit provided with a second image pickup element that picks up an optical image having a higher resolution than the first image pickup unit at a position that is the same as the image pickup range of the entire element lens of In the department An image processing unit that generates stereoscopic information from the imaged optical image, and the image processing unit further includes a first element image extraction unit, a second element image extraction unit, an element image generation unit, It was set as the structure provided with.

  According to such a configuration, the three-dimensional information generation device is configured such that, in the first image capturing unit of the image capturing unit, the subject in the peripheral region of the opening via the plurality of element lenses (for example, convex lenses) by the first image sensor. An optical image of is taken. The optical image picked up by the first image pickup device is an image in which a plurality of element images in integral photography are picked up. In the first image capturing unit of the image capturing unit, the three-dimensional information generating device captures an optical image of the subject through the imaging lens (for example, a convex lens) by the second image sensor in the aperture region. .

  By setting the optical image captured by the second image sensor to a higher resolution than the optical image captured by the first image sensor, the optical image captured by the second image sensor is placed at the center of the imaging unit. If the element lens is arranged and the resolution is higher than the optical image picked up by the first image pickup device, it can be used as a high-resolution element image.

  Then, the three-dimensional information generation device extracts the optical image for each element lens captured by the first image capturing unit as the first element image of the subject by the first element image extraction unit. Then, the three-dimensional information generation device extracts the optical image captured by the second image capturing unit as the second element image by the second element image extraction unit.

  Then, the three-dimensional information generation apparatus is configured to cause the element image generation unit to adjacently extract the second element image extracted by the second element image extraction unit and the first element image captured by the first image sensor. The three-dimensional information is generated by sequentially shifting by an amount corresponding to the parallax with the adjacent element image. Accordingly, element images are sequentially generated at the resolution of the second image sensor, and an element image having a higher resolution than that of the element image captured by the first image sensor can be generated as stereoscopic information.

  The stereoscopic information generating apparatus according to claim 2 is the stereoscopic information generating apparatus according to claim 1, wherein the element image generating means includes a parallax detecting means, a parallax extending means, a first shift processing means, A two-shift processing unit, an image expansion unit, and a complement processing unit are provided.

  According to this configuration, the stereoscopic information generating apparatus detects the parallax for each first element image extracted by the first element image extracting unit by the parallax detecting unit. Then, the stereoscopic information generating apparatus extends the parallax detected by the parallax detection unit to the image size of the second element image by the parallax expansion unit. The parallax thus extended can be used as parallax for the second element image.

Then, the three-dimensional information generation device uses the first shift processing means to convert the second element image into a central element image of the central area of the imaging unit, which is a virtual area in which an element lens is virtually arranged in the opening, and the central element The image is sequentially shifted based on the parallax expanded by the parallax expansion unit. As a result, an expanded element image in the central region is generated.
Then, the stereoscopic information generating apparatus sequentially shifts the expanded element image in the central area generated by the first shift processing unit based on the parallax expanded by the parallax expansion unit by the second shift processing unit.

Then, the three-dimensional information generation apparatus expands the first element image extracted by the first element image extraction unit to the image size of the second element image by the image expansion unit. Then, the three-dimensional information generating apparatus, the complementary processing unit, a first element image is expanded by the image expansion means, by complementing with extended element image in the second shift processing means, extended elements in the peripheral region Generate an image.

  Further, the stereoscopic information generating apparatus according to claim 3 is the stereoscopic information generating apparatus according to claim 2, wherein the parallax detecting unit is configured so that the parallax detecting unit is adjacent to the element lens in which the parallax corresponding to the central area is virtually arranged. Are detected by the parallax between the actual element lenses closest to the central region.

  According to such a configuration, the stereoscopic information generating apparatus obtains the parallax of the expanded element image corresponding to the central area from the parallax of the element image captured by the actual element lens closest to the central area. Errors with respect to parallax can be minimized.

  The three-dimensional information generation program according to claim 4 is provided with an opening at the center of the imaging surface, two-dimensionally arranging a plurality of element lenses in a peripheral region of the opening, and facing the element lenses. A first image pickup unit provided with a first image pickup device for picking up an optical image of a subject at a position, an imaging lens provided in the opening, and the imaging range facing the imaging lens; An image pickup unit including a second image pickup unit provided with a second image pickup element that picks up an optical image having a higher resolution than the first image pickup unit at a position that is the same as the image pickup range of the entire element lens of In order to generate the three-dimensional information of the subject in integral photography from the optical image of the subject, the computer includes a first element image extraction means, a second element image extraction means, a parallax detection means, a parallax expansion means, a center Frequency component image generating means, image expansion means and a peripheral region element image generating means, to function as a.

  According to such a configuration, the three-dimensional information generation program extracts the optical image for each element lens captured by the first image capturing unit as the first element image of the subject by the first element image extraction unit. Then, the three-dimensional information generation program extracts the optical image captured by the second image capturing unit as the second element image by the second element image extraction means.

  Then, the three-dimensional information generation program detects the parallax for each first element image extracted by the first element image extraction unit by the parallax detection unit. Then, the stereoscopic information generation program expands the parallax detected by the parallax detection unit to the image size of the second element image by the parallax expansion unit.

  Then, the three-dimensional information generation program uses the central area element image generation means to set the second element image as the central element image of the central area of the imaging unit, which is a virtual area in which the element lens is virtually arranged in the opening, The element image is shifted based on the disparity expanded by the disparity extending means, thereby generating an expanded element image in the central region.

Furthermore, the three-dimensional information generation program expands the first element image extracted by the first element image extraction unit to the image size of the second element image by the image expansion unit. The three-dimensional information generation program shifts the first element image expanded by the image expansion unit based on the parallax expanded by the parallax expansion unit by the peripheral region element image generation unit, thereby Generate an extended element image of.

The present invention has the following excellent effects.
According to the first and fourth aspects of the invention, in order to sequentially generate the peripheral three-dimensional information (element image) based on the element image with high resolution captured through the imaging lens provided in the opening, Even when the number of pixels of one image sensor is small, it is possible to generate high-resolution three-dimensional information in integral photography. For this reason, according to the present invention, even when the subject to be imaged exists at a position away from the lens group, the resolution of each elemental image can be increased, and a high-resolution stereoscopic image is reproduced. Three-dimensional information can be generated.

  According to the second aspect of the present invention, when the element image captured through the element lens around the opening is expanded, the element image supplemented by the high-resolution element image captured through the imaging lens at the opening is supplemented. Therefore, the expanded element image corresponding to the periphery of the opening can be reproduced more accurately.

  According to the third aspect of the present invention, the parallax of the element image in the central area corresponding to the opening can be replaced with the most similar parallax among the parallaxes of the actual element lens. Can be minimized, and highly accurate three-dimensional information can be generated.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[Configuration of stereoscopic information generating apparatus]
First, the configuration of the three-dimensional information generation apparatus according to the present invention will be described with reference to FIG. FIG. 1 is a block diagram illustrating a configuration of a three-dimensional information generation apparatus according to an embodiment of the present invention.

  The three-dimensional information generating apparatus 1 generates an element image obtained by imaging a subject through a plurality of element lenses as three-dimensional information of the subject in integral photography. Here, the three-dimensional information generation apparatus 1 includes an imaging unit 10 and an image processing unit 20.

  The imaging unit 10 images the subject H with a plurality of lenses, and includes a first image imaging unit 11 and a second image imaging unit 12.

  The first image capturing unit 11 captures an optical image of the subject H with a compound eye lens (a plurality of convex lenses [element lenses] arranged in the same plane). The image captured by the first image capturing unit 11 is output to the image processing unit 20.

  The second image capturing unit 12 captures an optical image of the subject H by an imaging lens (for example, a convex lens) provided in the opening 13 at the center of the image capturing surface of the image capturing unit 10. The second image capturing unit 12 captures an optical image having a higher resolution than the individual optical images captured by the first image capturing unit 11. The image captured by the second image capturing unit 12 is output to the image processing unit 20.

  Here, the configuration of the imaging unit 10 will be described in detail with reference to FIG. FIG. 2 is a configuration diagram illustrating a configuration of the imaging unit of the three-dimensional information generation apparatus according to the embodiment of the present invention, where (a) is a schematic side view of the imaging unit, and (b) is a schematic front view of the imaging unit. is there.

  The first image capturing unit 11 is provided with an opening 13 at the center of the imaging surface of the image capturing unit 10, and a plurality of element lenses 14 are arranged in a two-dimensional manner as a compound eye lens in a region (peripheral region) other than the opening 13. In addition, a first imaging element 15 that captures an optical image of the subject H is provided at a position facing the plurality of element lenses 14. The image captured by the first image capturing unit 11 includes a plurality of element images captured by the first image sensor 15. The first image sensor 15 may be configured as a single image sensor as a whole, or a minute image that captures an optical image via the element lens 14 at a position corresponding to each element lens 14. A plurality of image sensors (element image sensors: not shown) may be arranged. Further, FIG. 2B shows an example in which the element lenses 14 are arranged in a delta arrangement, but they may be arranged in a square lattice pattern.

  The second image capturing unit 12 includes an imaging lens 16 in the opening 13 and a second imaging element 17 that captures an optical image of the subject H at a position facing the imaging lens 16. The second image capturing unit 12 has the second image sensor 17 disposed at a position where the image capturing range is the same as the image capturing range of the first image capturing unit 11.

Here, with reference to FIG. 3 (refer to FIG. 2 as appropriate), the positional relationship for arranging the first image capturing unit 11 and the second image capturing unit 12 so that the imaging range is the same will be described. To do. FIG. 3 is an explanatory diagram for explaining the positional relationship between the first image capturing unit and the second image capturing unit.
As shown in FIG. 3, in order to make the imaging range of the first image capturing unit 11 and the second image capturing unit 12 the same, the range in which the element lenses 14L and 14R at both ends of the first image capturing unit 11 are imaged As long as the imaging range of the imaging lens 16 of the second image capturing unit 12 is the same.

Here, assuming that the range of an image captured by the first imaging element 15 by one element lens 14 is W ₁ and the distance from the element lens 14 to the first imaging element 15 is g ₁ , the angle of view θ of the element lens 14. ₁ can be expressed by the following equation (1).
_{θ 1 = 2ArcTan ((W 1} /2) g 1) ... (1) formula

Therefore, when the pitch of the element lenses 14 is p and the number of element lenses 14 (here, the number in the horizontal direction) is k, the first image capturing unit 11 captures an image of a subject space at a distance L from the element lens 14. The range W _all can be expressed by the following equation (2).
W _all = (k−1) p + Lθ ₁ (2)

On the other hand, if the range of the image captured by the second imaging element 17 by the imaging lens 16 is W ₂ and the distance from the imaging lens 16 to the second imaging element 17 is g ₂ , the angle of view θ of the imaging lens 16 ₂ can be expressed by the following equation (3).
_{θ 2 = 2ArcTan ((W 2} /2) g 2) ... (3) the formula

Accordingly, the imaging range W _obj of the second image capturing unit 12 in the subject space existing at a distance L from the imaging lens 16 can be expressed by the following equation (4).
W _obj = Lθ ₂ (4)

Therefore, in order to make the imaging range W _{all of} the first image imaging unit 11 and the imaging range W _{obj of} the second image imaging unit 12 the same, from the imaging lens 16 according to the equations (1) to (4) above. the distance g ₂ to the second image sensor 17 may be set to a distance shown in the following equation (5).

As described above, the second image capturing unit 12 captures an image of the subject in a state where the image is focused on the position of the distance L from the imaging lens 16. Note that although shown a convex lens as an example as an imaging lens 16, even when the distance L is changed, to the distance g ₂ satisfies the expression (5), an imaging lens A zoom lens may be used as 16. The imaging lens 16 may be provided with an aperture stop mechanism (not shown).

  Thereby, the second image capturing unit 12 can capture an image in the imaging range of the first image capturing unit 11 with high resolution. Returning to FIG. 1, the description of the configuration of the three-dimensional information generation apparatus will be continued. Note that FIG. 2 is appropriately referred to for the configuration of the imaging unit 10.

  The image processing unit 20 generates stereoscopic information (element image) of the subject H in integral photography from the optical image captured by the imaging unit 10. Here, the image processing unit 20 includes peripheral element image extraction means 21, center element image extraction means 22, and element image generation means 23.

  Peripheral element image extraction means (first element image extraction means) 21 is an element image of the subject H (first element image) for each predetermined area from the optical image captured by the first image sensor 15 of the first image capturing unit 11. ) Is extracted (cut out). The predetermined area is an image area corresponding to each element lens 14, and the number of pixels is determined in advance by the resolution of the first image sensor 15. The peripheral element image extracting unit 21 may extract the element image to be extracted as a circular area, or may be extracted as an area having a shape such as a rectangle or a hexagon. In addition, when the first image sensor 15 is configured by an element image sensor (not shown) corresponding to each element lens 14, the peripheral element image extraction unit 21 uses the elements captured by the individual element image sensors as elements. It may be extracted as an image.

  Here, it is assumed that the peripheral element image extraction unit 21 performs the inversion process so that the extracted element images are point-symmetric with respect to the center of each image. This can prevent reverse display of the depth when displaying the element image. The element image (peripheral element image) extracted by the peripheral element image extraction unit 21 is stored in a storage unit such as a memory (not shown) and is referred to by the element image generation unit 23.

The central element image extraction means (second element image extraction means) 22 is an element image (second element image) of the subject H in a predetermined area from the optical image captured by the second image sensor 17 of the second image capturing unit 12. Are extracted (cut out). The second element image is an image obtained by increasing the resolution of the element image captured on the assumption that the element lens 14 is arranged at the center of the imaging unit 10. Specifically, when the distance from the element lens 14 to the first image sensor 15 described in FIG. 3 is g ₁ and the distance from the imaging lens 16 to the second image sensor 17 is g ₂ , the central element image is displayed. extracting means 22, as shown in FIG. 4, the size of the extracted element image peripheral element image extracting means 21 (e.g., the number of horizontal pixels) to W _S, the magnitude is expressed by the following equation (6) is cutting out areas of W _L from the optical image captured by the second image sensor 17.
W _L = W _S (g ₂ / g ₁ ) (6) formula

  Here, the center element image extraction unit 22 performs the inversion process so that the extracted element images are point-symmetric with respect to the respective image centers. This can prevent reverse display of the depth when displaying the element image.

  4 shows an example in which the central element image extracting unit 22 extracts the element image in a circular region, but the central element image extracting unit 22 may extract the region as a region such as a rectangle or a hexagon. The same (similar shape) area as the element image extracted in 21 is extracted.

  The element image extracted by the central element image extraction unit 22 is stored in a storage unit such as a memory (not shown), and the element image generation unit 23 as a high-resolution central element image (central high resolution element image). Referenced by.

  The element image generation means 23 is based on the element image (peripheral element image) extracted by the peripheral element image extraction means 21 and the element image (center high resolution element image) extracted by the central element image extraction means 22. All element images are converted into element images (high resolution element images) having the same resolution as the central high resolution element image. Here, the element image generation unit 23 includes a parallax information generation unit 231, a central region element image generation unit 232, and a peripheral region element image generation unit 233.

  The disparity information generating unit 231 assumes that element lenses are arranged in the entire imaging unit 10 and corresponds to the image size of the center high-resolution element image extracted by the center element image extracting unit 22 with respect to the disparity between the element lenses. Is generated as a parallax. Here, the parallax information generation unit 231 includes a parallax detection unit 231a and a parallax extension unit 231b.

The parallax detection unit 231a detects a parallax indicating a deviation for each element image adjacent to the element image (peripheral element image) extracted by the peripheral element image extraction unit 21 in units of pixels. This parallax can be obtained by block matching or the like.
Note that the parallax detection unit 231a not only provides the actual parallax of the element image extracted by the peripheral element image extraction unit 21, but also the element lens 14 when the element lens 14 is virtually disposed at the position of the imaging lens 16. Also generates parallax of the elemental image virtually captured.

  Here, referring to FIG. 5 (refer to FIG. 1 as appropriate), the parallax detection unit 231a virtually captures an image when the element lens 14 is virtually disposed at a position corresponding to the position of the imaging lens 16. A method for generating parallax (parallax information) of an element image will be described. FIG. 5 is an explanatory diagram for explaining a method of calculating a virtual parallax of the parallax detection unit.

  As shown in FIG. 5A, the imaging unit 10 has an imaging lens 16 at the center and a plurality of element lenses 14 at the periphery thereof. Therefore, the parallax detection means 231a virtually arranges an element lens at the position of the imaging lens 16 as shown in FIG. 5B (dotted circle in the figure). And the parallax detection means 231a calculates the parallax of the element image virtually imaged with respect to the element lens virtually arranged. Here, the parallax detection unit 231a corresponds to the virtually arranged elemental lens with the parallax of the elemental image corresponding to the closest existing element lens that is the same as the adjacent direction of the virtually arranged elemental lenses. Calculated as parallax of element image.

  For example, as shown in FIG. 5C, when the element lens 14a virtually arranged is adjacent to the element lens 14b virtually arranged in the left horizontal direction, the parallax detection unit 231a The parallax of the virtually arranged element lenses 14a and 14b is generated by averaging the parallax for each pixel with respect to the element image captured by the closest existing element lens 14c and 14d corresponding to the adjacent direction.

  The parallax is obtained by sequentially calculating parallax in the peripheral direction with reference to the central element lens among the virtually arranged element lenses. Here, as an example, the example of obtaining the parallax in the left horizontal direction has been described. However, in the case of obtaining the parallax of the element lens that exists in the right horizontal direction rather than the virtually arranged central element lens, the actual that is closest to the right horizontal direction is described. What is necessary is just to obtain | require parallax between the element lenses to perform. In addition, even when the adjacent direction is the vertical direction, the diagonal vertical direction, or the like, the parallax between the closest existing element lenses may be obtained correspondingly to the adjacent direction.

  Hereinafter, in order to distinguish the parallax detected (calculated) by the parallax detection unit 231a, the parallax of the peripheral area corresponding to the element image captured by the actual element lens is referred to as “corresponding parallax”, and the area of the opening 13 (center area). The parallax corresponding to the element image captured by the element lens closest to the element image virtually captured by the virtual element lens in FIG. Returning to FIG. 1, the description of the configuration of the three-dimensional information generation apparatus will be continued.

  The parallax extending unit 231b extends the parallax for each pixel detected by the parallax detecting unit 231a to parallax with respect to the image size of the central high-resolution element image extracted by the central element image extracting unit 22. That is, the parallax extending unit 231b generates an extended parallax (extended parallax) by allocating the parallax for each pixel in the image data corresponding to the image size of the central high-resolution element image at a ratio according to the equation (6). . The extended parallax generated by the parallax extension means 231b is stored in a storage means such as a memory (not shown) and is referred to by the central area element image generation means 232 and the peripheral area element image generation means 233.

  Here, with reference to FIG. 6 (refer to FIG. 1 as appropriate), a method of generating disparity information of the disparity information generating unit 231 will be described. FIG. 6 is an explanatory diagram schematically illustrating a method in which the disparity information generating unit generates disparity information. As illustrated in FIG. 6, the parallax detection unit 231a generates a shift between adjacent element images (a) and (b) as a parallax image illustrated in (c), and uses the shift direction as parallax information. Note that this shift direction is associated with each pixel.

  Then, the parallax extending unit 231b extends the parallax image shown in (c) to a parallax image having the image size of the central high-resolution element image. For example, when the central high-resolution element image is composed of four times as many pixels as the element image (peripheral element image) extracted by the peripheral element image extraction unit 21, as shown in FIG. The parallax for each pixel is assigned to 4 times the pixel position.

  Then, the parallax expansion unit 231b changes the parallax assigned to each pixel position according to the image size. For example, when the central high-resolution element image has twice the number of pixels in both the vertical and horizontal directions of the peripheral element image, as shown in FIG. 6, the parallax expansion unit 231b doubles the parallax for each pixel in both vertical and horizontal directions. Thus, the extended parallax image (extended parallax information) shown in (d) is generated. As a result, the parallax detected by the parallax detection unit 231a is expanded to a parallax image corresponding to the image size of the central high-resolution element image. Returning to FIG. 1, the description of the configuration of the three-dimensional information generation apparatus will be continued.

  The center area element image generation means 232 virtually captures an image with the element lens 14 when it is assumed that the element lens 14 is arranged instead of the imaging lens 16 in the area corresponding to the opening 13 (center area). An image obtained by increasing the resolution of the element image (center region high resolution element image) is generated. The central area element image generation means 232 sequentially generates a central area high resolution element image from the central element image (central high resolution element image) extracted by the central element image extraction means 22. Here, the central area element image generation means 232 includes a shift processing means 232a and an interpolation processing means 232b.

  The shift processing unit 232a shifts each pixel of the element image that has been increased in resolution by an amount corresponding to the disparity based on the disparity information (expanded disparity information) expanded by the disparity expanding unit 231b. An image is generated. The shift processing unit 232a sequentially performs shift processing based on the center element image (center high-resolution element image) extracted by the center element image extracting unit 22.

  The interpolation processing unit 232b is configured to fill in the missing pixel due to the shift processing by interpolation on the high-resolution element image generated by the shift processing unit 232a. The interpolation processing unit 232b obtains the pixel value of the missing pixel by, for example, linearly interpolating the surrounding pixel values.

  Here, with reference to FIG. 7 (refer to FIG. 1 as appropriate), the processing performed by the central region element image generation means 232 will be specifically described. FIG. 7 is an explanatory diagram for explaining a mechanism in which the central region element image generation unit generates an element image adjacent to a certain element image with a high resolution. FIG. 7A shows the extended parallax information generated by the parallax information generating unit 231 described with reference to FIG.

  As shown in FIG. 7, when it means that the difference area of the extended parallax information in (a) has “two pixels in the left direction”, the shift processing unit 232 a In each pixel of the element image that has already been increased in resolution, the region corresponding to the difference region of the extended parallax information in (a) is shifted leftward by two pixels, so that the adjacent extended image shown in (c) is displayed. Generated element images.

  Note that the shift processing unit 232a captures the second image when the extended parallax information has a parallax such as “two pixels in the left direction” at the right end of the extended parallax image as shown in FIG. Of the images captured by the unit 12, the image is shifted from the outside of the central high-resolution element image.

  As a result, the central region element image generation means 232 can generate a high-resolution element image (high resolution element image [stereoscopic information]) corresponding to the region (center region) corresponding to the opening 13.

  The peripheral area element image generation means 233 generates an image (peripheral area high resolution element image) having a higher resolution from the element image captured by the element lens 14 provided in the periphery of the opening 13. The peripheral region element image generation unit 233 sequentially generates expanded element images adjacent to the periphery on the basis of the outermost element image generated by the central region element image generation unit 232. Here, the peripheral area element image generation unit 233 includes a shift processing unit 233a, an image expansion unit 233b, a complementing processing unit 233c, and an interpolation processing unit 233d.

  The shift processing unit 233a generates adjacent expanded element images by shifting each pixel of the element image having a high resolution by the amount of parallax based on the parallax information expanded by the parallax expansion unit 231b. It is. The shift processing unit 233a sequentially performs shift processing with reference to the outermost element image among the center element images (center high-resolution element images) extracted by the center element image extraction unit 22. This shift processing means 233a has the same function as the shift processing means 232a, except that the element image serving as a reference is different.

  The image expansion means 233b extends the element image (peripheral element image) extracted by the peripheral element image extraction means 21 to the image size of the central high resolution element image extracted by the central element image extraction means 22. That is, the image expansion unit 233b uses the ratio of the element image (peripheral element image) extracted by the peripheral element image extraction unit 21 in the image data for the image size of the central high-resolution element image according to the expression (6). An extended image is generated by assigning with. At this stage, pixels to which no extended image has been assigned are in a missing state.

  The complement processing means 233c complements missing pixels in the expanded peripheral element image generated by the image expansion means 233b with the high resolution element image generated by the shift processing means 233a.

  The interpolation processing means 233d fills the missing pixels by interpolation for the pixels that are not yet complemented by the complementing processing means 233c. The interpolation processing unit 233d obtains the pixel value of the missing pixel by, for example, linearly interpolating the surrounding pixel values. The pixel that is not complemented by the complement processing unit 233c is, for example, a pixel in the high-resolution element image generated by the shift processing unit 233a corresponding to the pixel to be complemented does not exist by the shift process. This is the case.

  Here, with reference to FIG. 9 (refer to FIG. 1 as appropriate), processing performed by the peripheral region element image generation unit 233 will be specifically described. FIG. 9 is an explanatory diagram for explaining a mechanism in which the peripheral area element image generation unit generates an element image adjacent to a certain element image with a high resolution. 9A to 9D also describe a process for generating extended parallax information by the parallax information generating unit 231 described with reference to FIG.

  As shown in FIG. 9, when the shift processing means 233a means that it has a parallax of “two pixels in the left direction” in the difference area of the extended parallax information in (d), In each pixel of the element image that has already been increased in resolution, the region corresponding to the difference region of the extended disparity information in (d) is shifted leftward by two pixels, so that the adjacent expanded image shown in (f) is displayed. Generated element images.

  Then, the image expansion means 233b assigns each pixel of the peripheral element image shown in (b) to the image data corresponding to the image size of the central high resolution element image at a ratio according to the expression (6) ( The extended image shown in g) is generated. For example, when the central high-resolution element image is composed of four times as many pixels as the element image (peripheral element image) extracted by the peripheral element image extraction unit 21, as shown in FIG. Is a state in which only one-fourth of all the pixels are allocated and three-fourths of the pixels are missing.

  Then, the complementary processing unit 233c assigns the corresponding pixel in the high-resolution element image shown in (f) generated by the shift processing unit 233a to the missing three-fourth pixels, and is expanded. The high-resolution element image shown in (h) is generated by complementing the peripheral element image. In this way, the complement processing unit 233c assigns pixels with priority over the extended image shown in (g) generated by the image expansion unit 233b over the element image shown in (f) generated by the shift processing unit 233a. As a result, a more accurate element image can be generated. It should be noted that also in this complementing process, if there is a missing pixel, the image is complemented by the interpolation processing means 233d.

  Accordingly, the peripheral region element image generation unit 233 can generate a high-resolution element image (high resolution element image [stereoscopic information]) for the peripheral region (peripheral region) of the opening 13.

  As described above, the three-dimensional information generation device 1 uses the imaging lens provided in the opening 13 to increase the resolution of peripheral element images with reference to an element image having a higher resolution than the element image captured by the element lens. can do. As a result, the stereoscopic information generating apparatus 1 increases the resolution of all element images of the stereoscopic information necessary for reproducing the stereoscopic image of the subject even when the resolution of the element image captured by the element lens is low. be able to.

  Note that the image processing unit 20 of the three-dimensional information generation apparatus 1 can be realized by a general computer equipped with a CPU and a memory (not shown). At this time, the image processing unit 20 of the three-dimensional information generation apparatus 1 operates by a three-dimensional information generation program that causes a computer to function as each of the above-described units.

[Operation of 3D information generator]
Next, the operation of the three-dimensional information generation apparatus according to the present invention will be described with reference to FIGS. FIG. 10 is a flowchart showing the overall operation of the three-dimensional information generation apparatus according to the embodiment of the present invention. FIG. 11 is a flowchart showing the operation of the element image generation means of the three-dimensional information generation apparatus according to the embodiment of the present invention.

[Overall operation]
First, the three-dimensional information generation device 1 images the subject H by the imaging unit 10 (step S1). Specifically, in step S <b> 1, the imaging unit 10 of the three-dimensional information generation apparatus 1 uses the first imaging element 15 to subject the subject via the plurality of minute convex lenses [element lens 14] of the first image imaging unit 11. An optical image of H is taken. In addition, the imaging unit 10 of the three-dimensional information generation device 1 captures an optical image of the subject H by the second imaging element 17 via the imaging lens 16 of the second image imaging unit 12.

Thereafter, the stereoscopic information generating apparatus 1 generates stereoscopic information of the subject from the optical image captured by the imaging unit 10 in the image processing unit 20.
First, the three-dimensional information generation apparatus 1 cuts out an image corresponding to each element lens 14 from the optical image picked up by the first image pickup device 15 in step S1 by the peripheral element image extraction unit 21, and extracts the cut out image. By performing inversion processing so as to be point-symmetric with respect to the center, it is extracted as an element image (peripheral element image) (step S2). A plurality of element images extracted in step S2 are extracted by the number of element lenses 14.

  In addition, the three-dimensional information generation apparatus 1 uses the center element image extraction unit 22 to measure the distance from the element lens 14 to the first image sensor 15 from the optical image captured by the second image sensor 17 in step S1, and the imaging lens. The image having the size shown in the equation (6) is cut out in accordance with the ratio with the distance from 16 to the second image sensor 17, and the cut image is inverted so as to be point-symmetric with respect to the center. Is extracted as an element image (high resolution element image) (step S3).

  The high resolution element image extracted in step S3 has a higher resolution than the element image extracted in step S2. In addition, the high-resolution element image extracted in step S3 is an image obtained by increasing the resolution of the element image captured by the central element lens when the element lens is virtually arranged in the opening 13 (center height). Corresponds to a resolution element image). Note that step S2 and step S3 do not need to be operated in this order, and may be operated in a different order or may be operated in parallel.

  Then, the three-dimensional information generation device 1 converts all element images based on the plurality of peripheral element images extracted in step S2 and the central high-resolution element image extracted in step S3 by the element image generation unit 23. Then, it is converted into an element image (high resolution element image) having the same resolution as the central high resolution element image (high resolution processing: step S4). The high resolution processing in step S4 will be described in detail with reference to FIG.

[High resolution processing]
(Parallax information generation step)
First, the stereoscopic information generating apparatus 1 uses the parallax (parallax) indicating the deviation for each element image adjacent to each peripheral element image extracted in step S2 (FIG. 10) by the parallax detecting unit 231a of the parallax information generating unit 231. Information) is detected in units of pixels (step S11).

  In step S11, parallax (corresponding parallax) is detected as a difference between elemental images captured by each elemental lens using parallax corresponding to an actual elemental lens. Further, the parallax of the element lens virtually arranged at the position of the imaging lens 16 is the parallax of the element image corresponding to the existing element lens that is adjacent in the same direction and closest to the opening 13 (nearest parallax). Detect as.

(Center area high resolution step)
Then, the stereoscopic information generating apparatus 1 expands the parallax information (nearest parallax) detected in step S11 to parallax with respect to the image size of the high-resolution element image extracted in step S3 (FIG. 10) by the parallax expansion unit 231b. Thus, an extended parallax (nearest extended parallax) is generated (step S12).

  Thereafter, the three-dimensional information generation device 1 uses the shift processing unit 232a of the central region element image generation unit 232 to each pixel of the high-resolution element image extracted in step S3 (FIG. 10), the latest extension generated in step S12. By shifting by the amount of parallax, an extended element image adjacent to the high resolution element image is generated (step S13).

  Here, whether or not the three-dimensional information generation apparatus 1 has generated an element image (high resolution element image) with a higher resolution corresponding to the area (center area) corresponding to the opening 13 by the center area element image generation unit 232. Is determined (step S14). Then, when all the high-resolution element images in the central area are generated (Yes in step S14), the three-dimensional information generation device 1 advances the operation to step S15.

  On the other hand, when all the high-resolution element images for the central region have not been generated (No in step S14), the three-dimensional information generation device 1 returns to step S12 and is adjacent to the already generated high-resolution element image. Using the image as a generation target, high-resolution element images for the central region are sequentially generated.

(Steps for increasing the resolution of the peripheral area)
Next, the stereoscopic information generating apparatus 1 converts the parallax information (corresponding parallax) detected in step S11 to parallax with respect to the image size of the high-resolution element image extracted in step S3 (FIG. 10) by the parallax expansion unit 231b. By expanding, an extended parallax (corresponding extended parallax) is generated (step S15).

  Thereafter, the stereoscopic information generating apparatus 1 uses the shift processing unit 233a of the peripheral region element image generation unit 233 to convert each pixel of the already expanded high resolution element image adjacent to the expansion target to the parallax ( By shifting by the amount of parallax based on the corresponding parallax, adjacent shifted high resolution element images are generated (step S16).

  Then, the three-dimensional information generation device 1 expands the element image adjacent to the element image in the central region that has already been increased in resolution among the peripheral element images extracted in step S2 (FIG. 10) by the image expansion unit 233b. As an object, an expanded peripheral element image is generated by expanding to the image size of the central high resolution element image (step S17).

  Then, the three-dimensional information generating apparatus 1 supplements the missing pixel in the peripheral element image expanded in step S17 with the shifted high resolution element image generated in step S16 by the complement processing unit 233c (step S18). ). At this time, the three-dimensional information generation device 1 supplements the pixels that have not been complemented in step S18 by interpolation using the interpolation processing unit 233d.

  Here, the three-dimensional information generating apparatus 1 determines whether or not all the element images corresponding to the peripheral area of the opening 13 have been increased in resolution by the peripheral area element image generating means 233, that is, all the high-resolution element images of the peripheral area. It is determined whether or not it has been generated (step S19). Then, when the three-dimensional information generation device 1 has generated all the high-resolution element images in the peripheral area (Yes in step S19), the operation ends.

  On the other hand, when all the high-resolution element images for the peripheral area have not been generated (No in step S19), the three-dimensional information generation device 1 returns to step S15 and is adjacent to the already generated high-resolution element image. Using the image as a generation target, high-resolution element images for the peripheral area are sequentially generated.

  Through the above operation, the three-dimensional information generation device 1 captures the element image captured through the element lens disposed in the peripheral region of the opening 13 through the imaging lens 16 provided in the opening 13. The resolution can be increased based on the high-resolution element image. As a result, the stereoscopic information generating apparatus 1 can increase the resolution of the element image even when the resolution of the individual element images in the first image sensor 15 of the first image imaging unit 11 is low. It is possible to generate stereoscopic information (element image) that can reproduce a high-definition stereoscopic image in the photography system.

It is a block diagram which shows the structure of the three-dimensional information generation apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the imaging part of the three-dimensional information generation apparatus which concerns on embodiment of this invention, Comprising: (a) is a schematic side view of an imaging part, (b) is a schematic front view of an imaging part. It is explanatory drawing for demonstrating the positional relationship of a 1st image imaging part and a 2nd image imaging part. It is explanatory drawing for demonstrating the relationship between the element image extracted by the periphery element image extraction means, and the element image extracted by the center element image extraction means. It is explanatory drawing for demonstrating the method of calculating the virtual parallax of a parallax detection means. It is explanatory drawing which showed typically the method in which a parallax information generation means produces | generates parallax information. It is explanatory drawing for demonstrating the mechanism in which a center area | region element image production | generation means produces | generates the element image adjacent to a certain high-resolution element image. It is a figure which shows the other example of parallax information. It is explanatory drawing for demonstrating the mechanism in which a periphery area | region element image generation means produces | generates the element image adjacent to a certain high-resolution element image. It is a flowchart which shows the whole operation | movement of the three-dimensional information generation apparatus which concerns on embodiment of this invention. It is a flowchart which shows operation | movement of the element image generation means of the three-dimensional information generation apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the concept of the conventional IP system, Comprising: (a) is at the time of imaging, (b) has shown the state at the time of a display.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Three-dimensional information generation apparatus 10 Imaging part 11 1st image imaging part 12 2nd image imaging part 13 Aperture part 14 Element lens 15 First imaging element 16 Imaging lens 17 Second imaging element 20 Image processing part 21 Peripheral element image extraction means (First element image extraction means)
22 Central element image extraction means (second element image extraction means)
23 element image generation means 231 disparity information generation means 231a disparity detection means 231b disparity expansion means 232 central region element image generation means 232a shift processing means (first shift processing means)
232b Interpolation processing means 233 Peripheral region degree element image generation means 233a Shift processing means (second shift processing means)
233b Image expansion means 233c Complement processing means 233d Interpolation processing means

Claims (4)

In a three-dimensional information generation device that generates an element image obtained by imaging a subject through a plurality of element lenses as three-dimensional information of the subject in integral photography,
An opening is provided in the center of the imaging surface, the plurality of element lenses are arranged two-dimensionally in a peripheral region of the opening, and an optical image of the subject is captured at a position facing the element lens. A first image capturing unit provided with an image sensor;
An imaging lens is provided in the opening, and the resolution is higher than that of the first image imaging unit at a position facing the imaging lens and the imaging range is the same as the imaging range of the entire element lens of the first image imaging unit. An image pickup unit having a second image pickup unit provided with a second image pickup device for picking up an optical image;
An image processing unit that generates the three-dimensional information from an optical image captured by the imaging unit,
The image processing unit
First element image extraction means for extracting an optical image of each element lens imaged by the first image imaging unit as a first element image of the subject;
Second element image extraction means for extracting the optical image captured by the second image capturing unit as a second element image;
The second element image extracted by the second element image extraction means is equivalent to the parallax between the first element image captured by the first image sensor and the adjacent element image adjacent to the first element image. Element image generation means for generating the three-dimensional information by sequentially shifting;
A three-dimensional information generating apparatus comprising: The element image generation means includes
Parallax detection means for detecting parallax for each first element image extracted by the first element image extraction means;
Parallax expansion means for expanding the parallax detected by the parallax detection means to the image size of the second element image;
The second element image is set as a central element image of a central area of the imaging unit, which is a virtual area in which the element lens is virtually arranged in the opening, and the central element image is expanded by the parallax expansion unit. First shift processing means for generating an extended element image in the central region by sequentially shifting based on parallax;
The element image expanded in the central region generated by the first shift processing unit is sequentially shifted based on the parallax expanded by the parallax expansion unit, so that the peripheral region of the opening is expanded. Second shift processing means for generating an element image;
Image expansion means for expanding the first element image extracted by the first element image extraction means to the image size of the second element image;
Complementary processing means for generating an element image expanded in the peripheral area by complementing the first element image expanded by the image expansion means with the element image expanded by the second shift processing means;
The three-dimensional information generation device according to claim 1, comprising:   The parallax detection means detects a parallax corresponding to the central area based on a parallax between the actual element lenses closest to the central area and having the same adjacent direction of the virtually arranged element lenses. The three-dimensional information generation device according to claim 2, wherein: A first imaging element that provides an opening in the center of the imaging surface, and in which a plurality of element lenses are two-dimensionally arranged in a peripheral region of the opening, and that captures an optical image of a subject at a position facing the element lens A first imaging unit provided with an imaging lens at the opening, and facing the imaging lens at a position where the imaging range is the same as the imaging range of the entire element lens of the first image imaging unit, A second image pickup unit provided with a second image pickup device that picks up an optical image having a higher resolution than that of the first image pickup unit, and an optical image of a subject imaged by the image pickup unit. In order to generate 3D information of the subject,
First element image extraction means for extracting an optical image of each element lens imaged by the first image imaging unit as a first element image of the subject;
Second element image extraction means for extracting an optical image captured by the second image capturing unit as a second element image;
Parallax detection means for detecting parallax for each first element image extracted by the first element image extraction means;
Parallax expansion means for expanding the parallax detected by the parallax detection means to the image size of the second element image;
The second element image is set as a central element image of a central area of the imaging unit, which is a virtual area in which the element lens is virtually arranged in the opening, and the central element image is expanded by the parallax expansion unit. A central area element image generating means for generating an extended element image in the central area by shifting based on parallax;
Image expansion means for expanding the first element image extracted by the first element image extraction means to the image size of the second element image;
A peripheral area element image that generates an extended element image of the peripheral area of the opening by shifting the first element image expanded by the image expansion means based on the parallax expanded by the parallax expansion means Generating means,
A three-dimensional information generation program characterized by being made to function as.

Images