JP4129938B2 - Panorama image creation method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a panorama image creation method, and more particularly to a panorama image creation method of an image taken by a digital camera, an all-round image creation method for realizing walk-through in multimedia, and a wide range for an immersive display. The present invention relates to a panoramic image creation method suitable for a visual field panoramic image creation method.
[0002]
[Prior art]
Conventionally, a part of the image is used as a method for creating a panoramic image, and the images to be joined are photographed so that 1/3 to 1/2 of the image overlap, and the affine transformation is performed based on the information of the overlap portion. Alignment is performed by parameter calculation by means of correlation and correlation calculation by plane projection conversion. Subsequent adjustments such as brightness and hue between images are performed. Examples of such research include S. Peleg and J. Herman “Panoramic Mosaics by Manifold Projection”, CVPR, June 1997, pp 338-343 and H. Shum and R. Szelski “Panoramic Mosaics” Microsoft Research Technical Report There are many studies such as MSR-TR-97-23. Currently, several types of panorama creation software, such as Quicktime VR, VideoBrush, and PhotoVista, are commercially available or attached as digital camera appendices.
[0003]
[Problems to be solved by the invention]
For example, when synthesizing a panoramic image from the two images of FIGS. 3 and 4, if the alignment of the two images is performed by correlation calculation by affine transformation or planar projection transformation, the ground in the foreground that is not familiar with these calculation methods The alignment may be broken due to the influence of. Specifically, the foreground ground cannot be joined well as shown in the circle of FIG. 5, or the far-view portion that pursues the ground consistency is not joined well at the joint as shown in the circle of FIG. Or the entire screen is curved. FIG. 5 and FIG. 6 are extreme examples, but there are many cases in which these are unnatural synthesis in the middle or synthesis itself is impossible. This is because the correlation calculation of the alignment that is valid only for the former is applied to the distant view perpendicular to the line of sight and the ground parallel to the line of sight including the latter.
[0004]
SUMMARY OF THE INVENTION An object of the present invention is to provide a panoramic image creation method for creating a panoramic image with high consistency at a joint portion from a plurality of images including a distant view portion and a near view portion. The foreground portion refers to a relatively flat front portion in front of the distant view portion, and refers to the ground portion.
[0005]
[Means for Solving the Problems]
In order to achieve the above object, in the present invention, separate processing is performed on the distant view portion and the near view portion. The far-field part at the top of the screen is aligned by the conventional method, and the ground part is converted into an image as if the real ground was seen from above by geometric transformation, and the transformed image is connected and viewed from the viewpoint again. Perform geometric transformation as shown. Thereby, panorama composition with high consistency is performed at the joint portion of the ground portion.
[0007]
That is , according to the present invention, there is provided a panoramic image creation method that is executed by a computer to create a panoramic image,
For each of a plurality of images captured so as to be continuous in the horizontal direction by the discriminating means in the computer, a distant view portion including things existing sufficiently far away, and a relatively flat front portion in front of the distant view portion A step of determining
Projecting the front portion onto a plane perpendicular to the image and including an intersection line of the distant view portion and the front portion by projection means in the computer ;
Replacing the image of the front part projected on the surface with the front part in the captured image by the replacement means in the computer ;
Joining by matching the position of the overlap between the adjacent images by joining means in the computer ;
Performing inverse transformation of the projection with respect to the joined front portion by inverse transformation means in the computer ;
A panoramic image creation method is provided.
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the panoramic image creation method of the present invention will be described below with reference to the drawings. FIG. 2 is a schematic diagram of representation of a distant view and the ground. Things that exist sufficiently far away, such as mountain scenery, are represented by a distant view plane 10, and a ground that is composed of a relatively flat ground or water surface is represented by a ground 12 that has a plurality of parallel lines toward the photographer. Yes. Positions B and C in FIG. 2 are camera positions for capturing an image serving as a basis for creating a panoramic image. Position A corresponds to an intermediate position between positions B and C.
[0009]
FIG. 7 is an ideal panoramic view that can be seen when the line of sight is directed vertically from the position A in FIG. 2 to the distant view plane. FIG. 3 is an image obtained when a landscape is imaged by a camera with a slightly small horizontal angle of view, with the imaging direction set in parallel to the ground line and parallel to the ground from position B in FIG. FIG. 4 is an image obtained by capturing an image of a landscape with a camera with a slightly small horizontal angle of view, with the image capturing direction set in parallel to the ground line and horizontally from the position C in FIG. 3 and 4 are taken from a position B and a position C with a line of sight perpendicular to the distant view portion 10 in FIG. 2, the distant view images in FIG. 3 and FIG. 4 cover the same range as FIG. As for the parallel lines of the ground portion 12, FIG. 3 and FIG. 4 also have a pattern that spreads evenly downward as in FIG. 7, but the spread is not as great as FIG.
[0010]
FIG. 1 is a flowchart of an embodiment for creating a panoramic image from two images according to the panoramic image creating method of the present invention. The panoramic image creation method of the present invention will be described with reference to this flowchart. When a panoramic image as shown in FIG. 2 is created from the two images shown in FIGS. 3 and 4, first, in step S1, a distant view portion and a ground portion in the two images are respectively determined. Step S1 is a preparatory stage for performing a conversion different from the distant view portion on the ground portion in the subsequent step. In step S2, relative affine parameters are obtained for the distant view image, and rotation and size unification are performed between the two images. In addition, a parallel movement component is calculated. In step S3, the ground portion in the image is converted to create an image suitable for panoramic synthesis. In step S4, the two images are joined and aligned based on the parallel movement component obtained in step S2, thereby aligning the images. In step S5, the inverse transformation of the transformation performed in step S3 is performed only on the ground portion. In this way, a panoramic image as shown in FIG. 7 is obtained.
[0011]
Next, the ground portion conversion method in step S3 will be described. FIG. 8 is a schematic explanatory view showing a cross section of FIG. 2 when the imaging direction of the camera installed at a point of height M above position B in FIG. 2 is horizontal. In FIG. 8, the ground portion is represented by a line segment (EE ′), and the distant view portion is represented by a line segment (DE ′). The vertical angle of view of the camera is θv, the viewing distance is L, and the length of the original image in the vertical direction is H.
[0012]
A projection of the ground portion of FIG. 2 on the vertical plane of the original landscape is a vertical in-plane ground portion 14 (FE ′) represented in a form that is submerged in the basement as shown in FIG. When the vertical in-plane ground portion 14 (EE ′) is viewed from the viewpoint at the height M at the position B, the ground portion 12 (EE ′) seems to be projected in the same plane as the distant view portion 10. Looks like. That is, the point on the vertical in-plane ground portion 14 (FE ′) and the point on the ground portion 12 (EE ′) are in a one-to-one relationship, and are perpendicular to the viewpoint at the height M on the position B. A point where a straight line connecting one point J on the in-plane ground portion 14 (FE ′) intersects the ground portion 12 (EE ′) can be set as G. The captured image is a two-dimensional plane, and a ground portion that is originally a horizontal plane that faces the imager is also captured as such a vertical plane.
[0013]
FIG. 9 is a schematic view of a bird's eye view of the horizontal surface of the ground portion of FIG. A region where the point G exists in FIG. 8 is indicated by a black portion in FIG. This corresponds to creating an image of the ground portion of the original image in FIG. The blacked portion in FIG. 9 is a trapezoid whose width is shortened by a length of (L−y ′) · H / L according to the distance y ′ in the direction of the photographer from the boundary between the ground portion and the distant view portion. Become. As described above, the correspondence between the ground portion of the original image and the trapezoidal ground image is calculated. This trapezoidal image is reprojected onto the original ground portion. At this time, this reprojection is a multi-viewpoint projection of a black portion, and for example, the reprojection is performed so that the shape of the trapezoidal ground image 16 remains as shown in FIG. By projecting the ground portion once onto a horizontal plane and re-projecting the trapezoidal area created by the projection onto the vertical plane, the images of the ground portions of FIGS. 3 and 4 are converted into trapezoidal ground images as shown in FIGS. Replace with 16. These ground lines are parallel.
[0014]
The above-described projection and replacement of the image of the ground portion can be expressed by a mathematical formula by setting a coordinate system on the captured image. In FIG. 3, v is the length of the captured image in the vertical direction, h is the length of the captured image in the horizontal direction, and the parameter m corresponding to M in FIG. It is the length with the center line. Further, a parameter corresponding to K in FIG. 9 is set as k, and a parameter corresponding to L in FIG. 9 is set as l. If a coordinate system is defined on the captured image such that the upper side of the ground portion is the x-axis and the y-axis is perpendicular to the midpoint of the upper side, the ground portion can be represented by coordinates (x, y). This ground portion (x, y) is converted by the following equation. By this conversion, the coordinates (x ′, y ′) of the trapezoidal ground image 16 shown in FIG. 10 are obtained, and a trapezoidal image in which the lines of the ground portion are parallel is obtained.
x ′ = {l− (y / p) · k} · x / l
y ′ = y
However,
p = (v / 2) -m, 0 ≦ y ≦ P
k = p / tan (θv / 2)}
Parameter corresponding to L in FIG. 8: l
l = (v / 2) · {1 / tan (θv / 2)}
[0015]
In step S2, the size of the distant view image at the top of the screen is integrated in advance, and a parallel movement component is calculated. In step S3, FIGS. 3 and 4 are converted into images in which the lines of the ground portion are parallel as shown in FIGS. 10 and 11, and then these two images are joined in step S4. At this time, the ground portion has a parallel line as shown in FIG. Accordingly, at this time, the ground image is also matched between the two images, and when the positioning is performed based on the parallel movement component of the distant view portion obtained in advance, the ground portion can be well connected.
[0016]
Further, in step S5, only the ground portion shown in FIGS. 10 and 11 is converted back to the original image scale by performing inverse transformation of the transformation performed for alignment. That is, in step S5, an operation for returning the width of the ground portion to the horizontal width of the image is performed. Inverse transformation is the same as in FIG. 3 except that the line segment that divides the image into equal left and right intervals is the y-axis, and the trapezoidal ground portion is regarded as a projected image projected on the ground portion of the original image. Reprojecting on the captured image. Further, as shown in FIG. 13, taking into account the horizontal angle of view θh after joining with respect to the image of the ground portion, for example, by returning only the inner side of the thick line in FIG. It is also possible to obtain a panoramic image having a desired horizontal angle of view by selecting a range to return to. As a result, the horizontal angle of view of the ideal panoramic image shown in FIG. 7 can also be created. It is possible to create a single panoramic image made up of a plurality of images by joining the adjacent images together in the images taken so that the above operations are continued in the horizontal direction.
[0017]
In the above embodiment, after the conversion is performed on the ground portion, the distant view portion and the ground portion are joined, and the screens are aligned based on the information of the distant view portion. However, the ground portion and the distant view portion are completely independent, and the operations for aligning the ground portion of the two images and the aligning the distant view portion of the two images are performed separately, and then the ground portion and the distant view portion are pasted. It is also possible to create a panoramic image by performing the matching operation. In addition, the steps relating to the panoramic image creation method of the present invention can be manually performed, for example, by selecting a distant view portion and a foreground portion by visually observing a person who performs image processing, and can also be automated by a computer. Is possible. In addition, it is desirable that the image that is the basis of the panoramic image is an image captured by a digital camera. For example, the present invention also uses an image that is created by processing an image recorded on a photosensitive material or the like into a digital image. Is a preferred embodiment.
[0018]
【The invention's effect】
As described above, according to the present invention, a single panoramic image is formed by joining a plurality of images obtained by photographing a landscape composed of a sufficiently distant object and a relatively flat ground surface or water surface. When creating a panoramic image from two images by transforming the ground portion so as to maintain the consistency of the joint, with respect to correlation calculation by affine transformation or planar projection transformation performed on a distant view portion Only the video information of things in the distant view at the top of the screen can be referred to when determining the alignment of the joint at the time, and correlation calculation by affine transformation or planar projection transformation can be applied to the ground portion.
[0019]
Further, in addition to the above configuration, for each transformation before joining, a plane with the intersection between the distant scene and the ground / water surface as a vertical plane with respect to the image based on a preset viewpoint corresponding to the ground / water surface. After creating a projected image and performing panoramic joining on the image obtained by replacing the projected ground part with the original ground part, the ground part is subjected to the reverse operation of the above projection, thereby distortion at the joint part. And the effect of reducing inconsistencies.
[Brief description of the drawings]
FIG. 1 is a flowchart of an embodiment for creating a panoramic image from two images according to a panoramic image creating method of the present invention.
FIG. 2 is a schematic diagram of a representation of a distant view and the ground.
FIG. 3 is a captured image when a landscape is imaged from a position B in FIG. 2, and is an explanatory diagram showing a coordinate system necessary for explaining the conversion of the ground portion.
4 is an image obtained by capturing a landscape from a position C in FIG. 2;
FIG. 5 is an example of joining failure in which matching is not achieved at the joint portion of the ground portion.
FIG. 6 is an example of joining failure in which matching is not achieved at a joint portion in a distant view portion.
7 is an ideal panoramic image captured from a position A in FIG.
8 is a schematic explanatory view showing a cross section of FIG. 2 when the imaging direction of a camera installed at a point of height M above position B in FIG. 2 is horizontal.
9 is a schematic view of the horizontal surface of the ground portion of FIG.
10 is a joined image after ground portion conversion, which is taken from a point at a height M above position B in FIG. 2;
FIG. 11 is a joined image after ground part conversion, picked up from a point of height M above position C in FIG. 2;
12 is a panoramic image obtained by joining a captured image from a point of height M above position B in FIG. 2 and a captured image from the point of height M above position C in FIG. 2;
FIG. 13 is an explanatory diagram showing adjustment of the ground portion according to the horizontal angle of view before reverse conversion.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 Distant view part 12 Ground part 14 Vertical in-plane part 16 Ground image after projection (trapezoidal ground image)

Claims (1)

A panorama image creation method executed by a computer to create a panorama image,
For each of a plurality of images captured so as to be continuous in the horizontal direction by the discriminating means in the computer, a distant view portion including things existing sufficiently far away, and a relatively flat front portion in front of the distant view portion A step of determining
Projecting the front portion onto a plane perpendicular to the image and including an intersection line of the distant view portion and the front portion by projection means in the computer ;
Replacing the image of the front part projected on the surface with the front part in the captured image by the replacement means in the computer ;
Joining by matching the position of the overlap between the adjacent images by joining means in the computer ;
Performing inverse transformation of the projection with respect to the joined front portion by inverse transformation means in the computer ;
A method for creating a panoramic image.

Images