JP3915664B2 - Image processing apparatus, image processing method used therefor, and program therefor - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus, an image processing method used therefor, and a program therefor, and more particularly to an image processing method used for three-dimensional data generation from stereo images and orthographic transformation.
[0002]
[Prior art]
Conventionally, in digital photogrammetry and cartographic work, three-dimensional data [hereinafter referred to as DEM (Digital Evelopment Map) data, based on stereo images obtained from cameras installed on the ground or images taken from satellites or aircraft, Is generated by stereo matching, and the orthorectified image is generated by using the stereo matching to generate an ortho image and applied to drawing or the like.
[0003]
Here, in the case of a normal camera, an ortho image is an object that is photographed with an inclination for central projection, as shown in FIG. 11 (a), as shown in FIG. 11 (b). Is an image obtained by orthogonally projecting a plurality of images, grasping a wide range of objects, and creating a drawing or the like.
[0004]
As an example of applying the above method, there is a technique using the above method when performing digital image measurement on the ground (for example, see Patent Documents 1 and 2). In addition, as an example of a system in which the above technique is applied to aerial photographs and satellite images, there is a technique for stereo-matching aerial photographs and satellite images to perform orthogonal conversion (see, for example, Patent Document 3).
[0005]
FIG. 12 is a block diagram illustrating a configuration example of a conventional image processing apparatus. As shown in FIG. 12, a conventional image processing apparatus 5 performs stereo matching processing on a plurality of images to extract DEM data, and an orthorectifier that orthorectifies images using the DEM data. And a conversion means 13.
[0006]
In the image processing apparatus 5 having the above configuration, an operation as shown in FIG. 13 is performed. That is, image data obtained from a plurality of photographs is given to the stereo processing means 11, and DEM data is automatically extracted by performing stereo matching as in the conventional system (step S41 in FIG. 13).
[0007]
The DEM data obtained in this way is given to the orthorizing means 13 together with the image data, and an ortho image is generated by moving the pixels in the image based on the DEM information and the standardized shooting angle information (FIG. 13). Step S42).
[0008]
[Patent Document 1]
JP 2000-182059 A (3rd to 7th pages, FIGS. 1 and 2)
[Patent Document 2]
Japanese Patent Laid-Open No. 11-351865 (pages 4, 5 and 1)
[Patent Document 3]
Japanese Unexamined Patent Publication No. 2000-348160 (pages 2, 3 and 1)
[0009]
[Problems to be solved by the invention]
However, in the conventional ortho image generation method described above, an error caused by mismatching of corresponding points is included in the DEM data obtained by stereo matching, and pixels having erroneous DEM data at the time of orthonormal transformation are included. Is moved to an incorrect position, and there is a problem that the image quality of the ortho image generated by the above method is low.
[0010]
In addition, in the conventional ortho image generation method, when an imaging target in an image includes many discontinuous shapes such as an aerial image capturing a city area, pixels are not captured in a concealed area such as a road hidden by a building or the like. Therefore, the corresponding pixel cannot be moved in the ortho image, and as shown in FIG. 14, a missing region in which the pixel is lost is generated in the concealed region, so that the pixel is lost in the ortho image generated by the above method. There is a problem in that many regions are included, resulting in an unnatural image.
[0011]
Since the conventional ortho image generation method has the above-described problems, there is a demand for a processing method that can be used for drawing and CG (Computer Graphics) generation and that can obtain a more natural ortho image.
[0012]
Accordingly, an object of the present invention is to solve the above-described problems, and to be able to obtain a more natural ortho image that is less affected by noise included in DEM data and less uncomfortable due to a defective area. And an image processing method used therefor, a program image processing apparatus thereof, an ortho image automatic shaping method used therefor, and a program thereof.
[0013]
[Means for Solving the Problems]
An image processing apparatus according to the present invention is an image processing apparatus that generates a three-dimensional data by performing a stereo matching process on a plurality of images, and a stereo processing unit that extracts the three-dimensional data by the stereo matching process; A DEM smoothing unit that applies a smoothing filter to the three-dimensional data extracted by the stereo processing unit to remove noise from the three-dimensional data and smoothes the three-dimensional data, and is smoothed by the DEM smoothing unit. The image using dimensional data All pixels included in The Individually And orthorectifying means for orthogonal transformation.
[0014]
Another image processing apparatus according to the present invention is an image processing apparatus that generates a three-dimensional data by performing a stereo matching process on a plurality of images, and a stereo processing unit that extracts the three-dimensional data by the stereo matching process An orthorectifying means for orthorectifying the image using the three-dimensional data, a projective transforming means for projectively transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, and an orthorectification obtained by the orthorectifying means. Image synthesizing means for synthesizing the image and the projective transformation image obtained by the projective transformation means to generate an interpolated ortho image in which the missing pixels are interpolated.
[0015]
An image processing method according to the present invention is an image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images in an image processing device. A step of extracting dimensional data, a step of applying a smoothing filter to the extracted three-dimensional data to remove noise of the three-dimensional data and smoothing, and using the smoothed three-dimensional data The image All pixels included in The Individually And an orthophoto conversion step.
[0016]
Another image processing method according to the present invention is an image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images, and extracting the three-dimensional data by the stereo matching processing; Orthorectifying the image using the three-dimensional data, projecting transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, an ortho image obtained by orthographic transformation of the image, and the projection And a step of generating an interpolated ortho image in which the missing pixels are interpolated by synthesizing the projected conversion image obtained by the conversion.
[0017]
A program for an image processing method according to the present invention is a program for an image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images. The computer program stores the three-dimensional data by the stereo matching processing. A process of extracting, a process of applying a smoothing filter to the extracted three-dimensional data to remove noise of the three-dimensional data and smoothing, and the image using the smoothed three-dimensional data All pixels included in The Individually And orthogonal transformation processing is executed.
[0018]
Another image processing method program according to the present invention is a program of an image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images. Obtained by processing for extracting data, processing for orthorectifying the image using the three-dimensional data, processing for projective transformation of the image according to the orientation angle with respect to the orthographic projection plane, and orthographic transformation of the image The generated ortho image and the projective transformation image obtained by the projective transformation are combined to generate an interpolated ortho image in which the missing pixels are interpolated.
[0019]
That is, the image processing apparatus of the present invention performs stereo matching processing on a plurality of images to extract three-dimensional data [hereinafter referred to as DEM (Digital Elevation Map) data], and smoothes the DEM data. A DEM smoothing means for removing noise by applying a normalization filter, an orthorectifying means for orthogonally transforming an image using DEM data, and a projective transforming means for projectively transforming an image according to an orientation angle with respect to an orthographic projection plane And an image synthesizing means for synthesizing the projected transformation image and the ortho image to generate an interpolated ortho image.
[0020]
In the image processing apparatus according to the present invention, the stereo matching process is automatically performed on a plurality of images with the above-described configuration, and height information at each point in the image based on the obtained parallax information, that is, DEM. After extracting the data and applying a smoothing filter to remove noise in the DEM data, orthorectification is generated by using the DEM data obtained by smoothing the image, and the orthographic projection plane is generated. Is combined with a projective transformation image obtained by projective transformation with respect to a plane parallel to the axis, and an interpolated ortho image is generated by interpolating the missing pixels.
[0021]
Thus, in the image processing apparatus of the present invention, noise in the DEM data is reduced and used for orthorectification, and the pixel defect area due to concealment is interpolated with the projective transformation image, thereby reducing the image quality due to noise and the defect area. It is possible to generate a more natural ortho image with less sense of incongruity.
[0022]
DETAILED DESCRIPTION OF THE INVENTION
Next, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to the first embodiment of the present invention. In FIG. 1, an image processing apparatus 1 performs stereo matching processing on a plurality of images (aerial image data a) to extract three-dimensional data [hereinafter referred to as DEM (Digital Elevation Map) data] b. A DEM smoothing means 12 for removing noise by applying a smoothing filter (not shown) to the extracted DEM data b, and an image (aerial image data a) using the smoothed DEM data c. An orthorectifying means 13 for conversion and a recording medium 14 for storing a program for realizing each means of the image processing apparatus 1 are configured.
[0023]
Each means of the image processing apparatus 1 generally operates as follows. That is, the following processing is performed by the computer that mainly configures the image processing apparatus 1 executing the program of the recording medium 14.
[0024]
First, the stereo processing means 11 performs stereo matching processing on a plurality of image data (aerial image data a), and extracts DEM data b. The DEM smoothing means 12 generates a smoothed DEM data c by applying a smoothing filter to the DEM data b obtained by the stereo processing means 11.
[0025]
The orthorectifying means 13 performs an orthogonal transformation of the image (aerial image data a) using the smoothed DEM data c smoothed by the DEM smoothing means 12 to generate an ortho image d. Here, as described above, in the case of an ordinary camera, an ortho image is an object that is photographed with an inclination for central projection, as shown in FIG. 11 (a), as shown in FIG. 11 (b). This is an image obtained by orthogonal projection with respect to a predetermined plane, and is an indispensable image for connecting a plurality of images to grasp a wide range of objects and creating a drawing or the like.
[0026]
FIG. 2 is a flowchart showing the operation of the image processing apparatus 1 according to the first embodiment of the present invention. The operation of the image processing apparatus 1 according to the first embodiment of the present invention will be described with reference to FIGS. Note that the processing shown in FIG. 2 is realized by a computer that mainly configures the image processing apparatus 1 executing the program of the recording medium 14.
[0027]
First, the stereo processing means 11 automatically performs a stereo matching process on a plurality of images (aerial image data a) to extract DEM data (step S1 in FIG. 2). There is no restriction | limiting in particular in the stereo matching process used here, For example, it can implement | achieve by the method disclosed by Unexamined-Japanese-Patent No. 3-167678.
[0028]
Next, the DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to generate smoothed DEM data c (step S2 in FIG. 2). The smoothing filter used here is not particularly limited as long as it has an effect of reducing noise. For example, a median filter, a multi-value morphological filter, or the like may be used. It is also possible to apply a combination of these filters.
[0029]
The orthorectifying means 13 uses the smoothed DEM data c smoothed by the DEM smoothing means 12 and orthorectifies the image (aerial image data a) with respect to a certain reference plane to create an ortho image d. (Step S3 in FIG. 2). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface in the case of using a satellite image or an aerial photograph.
[0030]
As described above, in the present embodiment, by using the smoothed DEM data for the orthorectification process, it is possible to generate an ortho image d in which the influence of noise included in the original DEM data b is reduced. .
[0031]
FIG. 3 is a block diagram showing the configuration of the image processing apparatus according to the second embodiment of the present invention. In FIG. 3, the second embodiment of the present invention is the same as the first embodiment of the present invention shown in FIG. 1 except that the DEM smoothing means 12 is omitted and a projective transformation means 21 and an image composition means 22 are added. The same components are denoted by the same reference numerals. The operation of the same component is the same as that of the first embodiment of the present invention.
[0032]
That is, the image processing apparatus 2 performs stereo matching processing on a plurality of images (aerial image data a) to extract DEM data b, and uses the extracted DEM data b to generate an image (aerial image). Orthorectification means 13 for orthographic transformation of the image data a), projective transformation means 21 for projective transformation of the image (aerial image data a) according to the rotation angle with respect to the orthographic projection plane, and a projective transformation image (projection transformation aerial image e ) And the ortho image d to generate an interpolated ortho image f in which the missing pixels are interpolated, and a recording medium 14 for storing a program for realizing each means of the image processing apparatus 2 It is composed of
[0033]
Each means of the above-described image processing apparatus 2 generally operates as follows. That is, the following processing is performed by the computer that mainly configures the image processing apparatus 2 executing the program of the recording medium 14.
[0034]
As described above, the stereo processing means 11 performs stereo matching processing on a plurality of image data (aerial image data a), and extracts DEM data b. The orthorectifying means 13 orthorectifies the image (aerial image data a) using the DEM data a obtained by the stereo processing means 11 described above.
[0035]
The projective conversion means 21 performs projective conversion on the image (aerial image data a) according to the rotation angle to the orthographic projection plane obtained by the orientation calculation. The image synthesizing means 22 interpolates the ortho image d obtained by the orthorectifying means 13 and the image (projective transformation aerial image e) transformed by the projective transform means 21 so that the missing pixel is replaced. An interpolated ortho ortho image f is generated.
[0036]
FIG. 4 is a flowchart showing the operation of the image processing apparatus 2 according to the second embodiment of the present invention. The operation of the image processing apparatus 2 according to the second embodiment of the present invention will be described with reference to FIGS. Note that the processing shown in FIG. 4 is realized by a computer that mainly configures the image processing apparatus 2 executing the program of the recording medium 14.
[0037]
First, the stereo processing means 11 automatically performs stereo matching processing on a plurality of images (aerial image data a) to extract DEM data b (step S11 in FIG. 4). There is no restriction | limiting in particular in the stereo matching process used here, For example, it can implement | achieve by the method disclosed by Unexamined-Japanese-Patent No. 3-167678.
[0038]
Next, using the DEM data b obtained by the stereo processing means 11, the orthorectifying means 13 orthorectifies the image (aerial image data a) with respect to a certain reference plane to create an ortho image d. (FIG. 4, step S12). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface in the case of using a satellite image or an aerial photograph.
[0039]
Projection conversion means 21 is an image (projection conversion aviation) obtained by projective conversion of any one of the plurality of images (aerial image data a) used in the stereo processing means 11 according to the rotation angle to the orthographic projection plane. An image e) is generated (step S13 in FIG. 4). This is an operation of approximating an image on a plane and reprojecting it on an orthographic projection plane with the viewpoint fixed. The image used at that time may be any image of the images (aerial image data a) used in the stereo processing means 11, but an image with a small rotation angle with respect to the orthographic projection plane is desirable.
[0040]
Finally, the image synthesizing means 22 superimposes the orthoimage d obtained by the orthorectifying means 13 and the projective transformation aerial image e obtained by the projective transformation conversion means 21 to interpolate to generate an interpolated orthoimage f (FIG. 4 step S14). Specifically, a process of interpolating using pixels of the projective transformation aerial image e at the corresponding position is performed on the pixels of the missing region where no value is obtained in the ortho image d.
[0041]
As described above, in this embodiment, the image (aerial image data a) is projectively converted in accordance with the rotation angle with respect to the orthographic projection plane, and the pixels in the corresponding region of the image (projection-converted aerial image e) are orthoimage d. Therefore, the missing area of the ortho image d can be interpolated. As a result, when the ortho image d is applied to CG or the like, the uncomfortable feeling caused by the missing area is reduced. Can do.
[0042]
FIG. 5 is a block diagram showing the configuration of the image processing apparatus according to the third embodiment of the present invention. In FIG. 5, the third embodiment of the present invention has the same configuration as that of the second embodiment of the present invention shown in FIG. 3 except that the DEM smoothing means 12 is added, and the same components are the same. The code | symbol is attached | subjected. The operation of the same component is the same as that of the second embodiment of the present invention.
[0043]
That is, the image processing device 3 performs stereo matching processing on a plurality of images (aerial image data a), and extracts stereo noise by applying a smoothing filter to the DEM data b and stereo processing means 11 that extracts DEM data b. DEM smoothing means 12 to be removed, orthorectifying means 13 for orthorectifying the image (aerial image data a) using the smoothed DEM data c, and an orientation angle for the image (aerial image data a) with respect to the orthographic projection plane The projection conversion means 21 for performing projective conversion according to the above, the image synthesis means 22 for synthesizing the projection-transformed aerial image e and the ortho image d, and generating an interpolated ortho image f in which the missing pixels are interpolated; The recording medium 14 stores a program for realizing each means.
[0044]
Each means of the image processing apparatus 3 operates roughly as follows. That is, the following processing is performed by the computer that mainly configures the image processing apparatus 3 executing the program of the recording medium 14.
[0045]
The stereo processing means 11 performs stereo matching processing on a plurality of image data (aerial image data a), and extracts DEM data b. The DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to generate smoothed DEM data c.
[0046]
The orthorectifying means 13 orthorectifies the image (aerial image data a) using the smoothed DEM data c smoothed by the DEM smoothing means 12 to create an ortho image d.
[0047]
The projective conversion means 21 performs projective conversion of the image (aerial image data a) according to the rotation angle to the orthographic projection plane that has been determined. The image synthesizing means 22 interpolates the ortho image d obtained by the orthorectifying means 13 and the image (projective transformation aerial image e) transformed by the projective transform means 21 so that the missing pixel is replaced. An interpolated ortho ortho image f is generated.
[0048]
FIG. 6 is a flowchart showing the operation of the image processing apparatus 3 according to the third embodiment of the present invention. The operation of the image processing apparatus 3 according to the third embodiment of the present invention will be described with reference to FIGS. Note that the processing shown in FIG. 6 is realized by a computer that mainly configures the image processing apparatus 3 executing the program of the recording medium 14.
[0049]
First, the stereo processing means 11 automatically performs stereo matching processing on a plurality of images (aerial image data a) to extract DEM data b (step S21 in FIG. 6). There is no restriction | limiting in particular in the stereo matching process used here, For example, it can implement | achieve by the method disclosed by Unexamined-Japanese-Patent No. 3-167678.
[0050]
Next, the DEM smoothing means 12 applies a smoothing filter to the DEM data b obtained by the stereo processing means 11 to create smoothed DEM data c (step S22 in FIG. 6). The smoothing filter used here is not particularly limited as long as it has an effect of reducing noise. For example, a median filter, a multi-value morphological filter, or the like may be used. It is also possible to apply a combination of these filters.
[0051]
The orthorectifying means 13 orthorectifies the image (aerial image data a) with respect to a certain reference plane using the smoothed DEM data c smoothed by the DEM smoothing means 12 to create an ortho image d. (Step S23 in FIG. 6). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface in the case of using a satellite image or an aerial photograph.
[0052]
Subsequently, the projective transformation means 21 is an image obtained by projective transformation of any one of the plurality of images (aerial image data a) used in the stereo processing means 11 according to the rotation angle to the orthographic projection plane. (Projection conversion aerial image e) is generated (step S24 in FIG. 6). This is an operation of approximating an image on a plane and reprojecting it on an orthographic projection plane with the viewpoint fixed. As an image used at this time, an arbitrary image of the images (aerial image data a) used in the stereo processing unit 11 may be used, but an image having a small rotation angle with respect to the orthographic projection plane is desirable.
[0053]
Finally, the image synthesizing means 22 interpolates the orthoimage d obtained by the orthorectifying means 13 and the projected transformation aerial image e obtained by the projective transformation converting means 21 so as to interpolate the missing pixels. The interpolated ortho image f is generated (step S25 in FIG. 6). Specifically, a process of interpolating using pixels of the projective transformation aerial image e at the corresponding position is performed on the pixels of the missing region where no value is obtained in the ortho image d.
[0054]
As described above, since the smoothed smoothed DEM data c is used in the orthorectification process in the present embodiment, the ortho image d in which the influence of noise included in the original DEM data b is reduced is generated. Can do.
[0055]
In this embodiment, the image (aerial image data a) is projectively converted in accordance with the rotation angle with respect to the orthographic projection plane, and pixels in the corresponding region of the image (projection-transformed aerial image e) are deleted regions of the ortho image d. Therefore, the missing area of the ortho image d can be interpolated. As a result, when the ortho image d is applied to CG or the like, the uncomfortable feeling caused by the missing area can be reduced.
[0056]
FIG. 7 is a block diagram showing a configuration of an image processing apparatus according to the fourth embodiment of the present invention. 7, the fourth embodiment of the present invention has the same configuration as that of the third embodiment of the present invention shown in FIG. 5 except that a stereo pair setting means 42 is added, and the same components are the same. The code | symbol is attached | subjected. The operation of the same components is the same as that of the third embodiment of the present invention.
[0057]
That is, the image processing apparatus 4 includes a stereo pair setting unit 42 that selects a pair of images from a plurality of images (aerial image data a), an ortho image generation unit 41 that generates an ortho image d, and an image (aerial image). The projection transformation means 21 for projecting transformation of the data a) according to the orientation angle with respect to the orthographic projection plane, the projection transformation aerial image e and the ortho image d are synthesized to generate an interpolated ortho image f in which the missing pixels are interpolated. The image synthesizing unit 22 and a recording medium 14 for storing a program for realizing each unit of the image processing apparatus 2 are configured.
[0058]
Here, the ortho image generation means 41 performs a stereo matching process on a plurality of images (aerial image data a) and extracts the DEM data b, and a smoothing filter on the extracted DEM data b. Is applied to the DEM smoothing means 12 for removing noise and the orthorectifying means 13 for orthogonally transforming the image (aerial image data a) using the smoothed smoothed DEM data c.
[0059]
Each means of the image processing apparatus 4 operates generally as follows. That is, the following processing is performed by the computer that mainly configures the image processing apparatus 4 executing the program of the recording medium 14.
[0060]
The stereo pair setting means 42 determines a plurality of stereo image pairs obtained by selecting two of the three or more image data groups obtained by photographing the same spot, and sets it as a processing unit. The stereo processing means 11 performs a stereo matching process on each stereo image pair given by the stereo pair setting means 42 to extract DEM data b.
[0061]
The DEM smoothing means 12 smoothes the DEM data b obtained by the stereo processing means 11 by applying a smoothing filter. The orthorectifying means 13 orthorectifies the image (aerial image data a) using the smoothed DEM data c smoothed by the DEM smoothing means 12 described above.
[0062]
The projective conversion means 21 performs projective conversion of the image (aerial image data a) according to the rotation angle to the orthographic projection plane that has been determined. The image synthesizing means 22 interpolates the orthoimages d obtained by the orthorectifying means 13 and the image (projective transformation aerial image e) transformed by the projective transformation means 21 by superimposing the missing pixels. Is generated as an interpolated ortho image f.
[0063]
FIG. 8 is a flowchart showing the operation of the image processing apparatus 4 according to the fourth embodiment of the present invention. The operation of the image processing apparatus 4 according to the fourth embodiment of the present invention will be described with reference to FIGS. Note that the processing shown in FIG. 8 is realized by a computer that mainly configures the image processing apparatus 4 executing the program of the recording medium 14.
[0064]
First, the stereo pair setting means 42 determines a plurality of stereo image pairs obtained by selecting two of three or more image data groups obtained by photographing the same spot, and sets them as a processing unit (step S31 in FIG. 8). For example, when four images are obtained, one set of two of these sets and another set of stereo images are set with the remaining two. Although there are six combinations that can be selected, it is not always necessary to set all of them as stereo image pairs, and a plurality of stereo image pairs may be determined so that all images are used to reduce the concealment area.
[0065]
Next, the stereo processing means 11 performs a stereo matching process for each stereo image pair set by the stereo pair setting means 42, and extracts DEM data b (step S32 in FIG. 8). There is no restriction | limiting in particular in the stereo matching process used here, For example, it can implement | achieve by the method disclosed by Unexamined-Japanese-Patent No. 3-167678.
[0066]
Subsequently, the DEM smoothing unit 12 applies a smoothing filter to each DEM data b obtained by the stereo processing unit 11 to perform a smoothing process (step S33 in FIG. 8). The smoothing filter used here is not particularly limited as long as it has an effect of reducing noise. For example, a median filter, a multi-value morphological filter, or the like may be used. It is also possible to apply a combination of these filters.
[0067]
The orthorectifying means 13 orthorectifies the image (aerial image data a) with respect to a reference plane using each DEM data (smoothed DEM data c) smoothed by the DEM smoothing means 12, An ortho image d is created (step S34 in FIG. 8). The orthographic projection plane at this time is, for example, a plane parallel to the ground surface in the case of using a satellite image or an aerial photograph.
[0068]
The projective transformation means 21 is an image (projection-transformed aerial image) obtained by projective transformation of any one of the plurality of images (aerial image data a) used in the stereo processing means 11 according to the rotation angle to the orthographic projection plane. e) is generated (step S35 in FIG. 8). This is an operation of approximating an image (aerial image data a) with a plane and reprojecting it on an orthographic projection plane with the viewpoint fixed. As an image used at that time, an arbitrary image among the images used in the stereo processing unit 11 may be used, but an image having a small rotation angle with respect to the orthographic projection plane is desirable.
[0069]
Finally, the image synthesizing means 22 superimposes each ortho image d obtained by the orthorectifying means 13 and the projective transformation aerial image e obtained by the projective transformation converting means 21 so as to interpolate. An interpolated ortho ortho image f is generated (step S36 in FIG. 8). Specifically, interpolation is performed using the corresponding pixels in the other orthoimages for the pixels in the omission area where no value is obtained in the orthoimage d. Interpolation processing is performed on the pixels in the missing area remaining in this processing using the corresponding pixels in the projective transformation image.
[0070]
In the above-described fourth embodiment of the present invention, the processing for a plurality of stereo pairs is repeatedly performed in the stereo processing means 11, the DEM smoothing means 12, and the orthorectifying means 13. For example, the ortho image generating means described above is used. It is also possible to provide a plurality of 41 and perform processing for a plurality of stereo pairs in parallel.
[0071]
As described above, in the present embodiment, since the smoothed smoothed DEM data c is used for the orthorectification process, an ortho image in which the influence of noise included in the original DEM data is reduced can be generated.
[0072]
Further, in this embodiment, ortho images are generated by mutually complementing from a plurality of stereo pairs with different line-of-sight directions, and in addition, the image is projectively converted according to the rotation angle with respect to the orthographic projection plane, and the corresponding region of the image is displayed. Since pixels are substituted in the missing area of the ortho image, the omission area of the ortho image can be interpolated, and as a result, when the ortho image is applied to CG, the uncomfortable feeling caused by the omission area is reduced. It becomes possible.
[0073]
The present invention is not limited to each of the first to fourth embodiments of the present invention described above, and can be implemented under various conditions. For example, the target image is not limited to an aerial photograph or a satellite image, and can be applied to an image obtained by stereo shooting on the ground or indoors. The image may be taken using two cameras or taken twice using a monocular camera, and the image data may be taken directly with a digital camera or scanned from an analog photo. Even if it is, it can be implemented.
[0074]
Subsequently, the operation of the embodiment of the present invention will be described using a specific example. In this description, a case will be described in which an ortho image parallel to the ground surface is generated from a stereo aerial photograph according to the third embodiment of the present invention.
[0075]
Stereo aerial photographs are treated by treating overlapping portions of images continuously taken from an aircraft as stereo images, and are first converted into digital data using a scanner. Subsequently, orientation work is performed to determine camera parameters at the time of shooting. Normally, even in aerial photographs taken in a direction close to vertical, it is difficult to grasp the camera parameters at the time of shooting due to the effects of changes in the attitude of the aircraft, etc. The coordinate values in the image are read, and parameters such as the camera rotation angle between the left and right images are determined using the coordinate pairs of the corresponding points. There is no restriction | limiting in particular in the orientation method used here, For example, you may use the value calculated by plotting software.
[0076]
After the camera parameters are determined, as shown in FIG. 9, the left and right images are re-projected onto a common parallel plane as necessary. This is an operation called parallelization, and the same object in the left and right images after parallelization comes on the same scanning line.
[0077]
Further, the stereo processing unit 11 automatically performs stereo matching processing on the left and right images that have been made parallel, and obtains a correspondence between the left and right images to generate dense parallax images. There is no restriction | limiting in particular in the stereo matching process used here, For example, it can implement | achieve by the method disclosed by Unexamined-Japanese-Patent No. 3-167678.
[0078]
If the parallax image is obtained, the position in the three-dimensional coordinate system corresponding to each pixel is calculated in combination with the determined camera parameters, and DEM data is extracted. As a result, the obtained DEM data includes many spike-like isolated points having discontinuous three-dimensional coordinates as noise due to the influence of pixels or the like that are mismatched during stereo matching processing.
[0079]
In order to remove the influence of these isolated points, the DEM smoothing means 12 applies a smoothing filter to the DEM data obtained by the stereo processing means 11 to perform a smoothing process. The smoothing filter used here is not particularly limited as long as it has an effect of reducing noise. For example, a median filter that replaces the median of pixel values in a small area centered on the target pixel as a representative value is used. For example, the above isolated noise can be efficiently removed while maintaining an edge depending on the actual shape.
[0080]
In addition, a multi-value morphological filter configured by combining a maximum value filter and a minimum value filter for a small region can be expected to have the same effect as described above. It is also possible to apply a combination of a plurality of such filters.
[0081]
Subsequently, using the smoothed DEM data that has been smoothed by the DEM smoothing means 12 and from which the noise has been removed, and the camera parameters that have been standardized, the orthorectifying means 13 corrects the original aerial image with respect to the ground surface. Projective transformation is performed to create an ortho image.
[0082]
Specifically, first, the plane coordinates in the ground coordinate system (for example, the plane rectangular coordinate system) when the three-dimensional coordinates at all points in the DEM data are projected on the ground surface, and the coordinates in the original image Ask for action. Then, according to the obtained correspondence, the original aerial image pixels are mapped while being interpolated into an image corresponding to the ground surface.
[0083]
Further, the projective conversion means 21 performs an operation of reprojecting the aerial image data on a plane parallel to the ground surface by projective conversion. As described above, the optical axis of the camera at the time of aerial photography is not completely perpendicular to the ground surface, and since the ground surface is photographed from an oblique direction, it is difficult to superimpose with the ortho image. For this reason, the projective transformation is applied to the aerial image, and the operation of reprojecting onto the projection plane parallel to the ground without changing the camera center position is applied. By this operation, an image corresponding to a complete vertical photograph can be generated.
[0084]
A specific method of projective transformation is, for example, a method disclosed in Japanese Patent Laid-Open No. 8-567 using the rotation angle of the camera coordinate system with respect to the ground surface and the focal length among the camera parameters at the time of shooting. It can be carried out.
[0085]
Finally, the image synthesizing unit 22 synthesizes the ortho image generated by the orthorizing unit 13 and the projective transformation image generated by the projective transformation unit 21 to generate an interpolated ortho image. Specifically, after superposing the ortho image and the projective transformation image, the pixel of the projective transformation image at the corresponding position is cited for the pixel of the omission area in the ortho image where the value is not obtained due to the influence of concealment. Then, an interpolation ortho image is generated by performing an interpolation process.
[0086]
In this case, the image from which the pixel is cited is an image of the left and right projection-transformed images that has a principal point close to the target position, that is, the one that faces the target point as much as possible. By doing so, it is possible to reduce the case where the building wall surface that falls down due to plane approximation is used for the concealment region, and to improve the image quality.
[0087]
Further, by blurring the boundary portion or the quoting portion at the time of quoting, it is possible to suppress the influence of the generated edge, and to generate an interpolated ortho image with less sense of incongruity.
[0088]
Next, a specific example in the case where an ortho image parallel to the ground plane is generated from four stereo aerial photographs according to the example of the fourth embodiment will be described.
[0089]
For example, as shown in FIG. 10, four stereo aerial photographs are taken with a large overlap between two adjacent shooting courses # 1 and # 2, and each pair of images shooting the same point is taken. Can be obtained by selecting one set from each course. However, the photographing method is not limited to this, and four images # 1 to # 4 obtained by individually photographing may be used. The obtained aerial photograph group is first converted into digital data using a scanner.
[0090]
Next, the stereo pair setting means 42 sets a stereo pair from the four stereo aerial photographs converted into digital data. By referring to metadata or the like added to the image, two stereo pairs are obtained by combining two images in the same shooting course. However, the pair setting method is not limited to this, and if all four images # 1 to # 4 are used, image pairs may be set between different courses. Further, each image # 1 to # 4 may be used a plurality of times to generate a maximum of 6 stereo pairs.
[0091]
For each set stereo pair, as in the third embodiment of the present invention, orientation work for determining camera parameters at the time of photographing, parallelization, stereo matching processing, extraction of DEM data, and smoothing of DEM data are performed. Is called.
[0092]
Subsequently, using the DEM smoothing means 12 smoothed by the above DEM smoothing means 12 and noise-removed smoothed DEM data and the camera parameters that have been standardized, the orthorectifying means 13 generates the original aerial image for each aerial photograph. Orthographic transformation is performed on the ground surface to create an ortho image.
[0093]
Specifically, first, when the three-dimensional coordinates at all points in the DEM data are projected onto the ground surface, the plane coordinates in the ground coordinate system (for example, the plane rectangular coordinate system) and the coordinates in the original image Ask for action. Then, according to the obtained correspondence, the original aerial image pixels are mapped while being interpolated into an image corresponding to the ground surface.
[0094]
Further, the projective transformation means 21 performs an operation of reprojecting the aerial image data of each stereo pair onto a plane parallel to the ground surface by projective transformation. As described above, the optical axis of the camera at the time of aerial photography is not completely perpendicular to the ground surface, and since the ground surface is photographed from an oblique direction, it is difficult to superimpose with the ortho image. For this reason, projective transformation is applied to the aerial image, and the operation of reprojecting onto a projection plane parallel to the ground without changing the camera center position is applied. By this operation, an image corresponding to a complete vertical photograph can be generated.
[0095]
A specific method of projective transformation is, for example, a method described in Japanese Patent Application Laid-Open No. 8-567 using the rotation angle and focal length of the camera coordinate system with respect to the ground surface among the camera parameters at the time of shooting. Can do this.
[0096]
Finally, the image synthesizing unit 22 synthesizes each ortho image generated by the above orthorizing unit 13 and each projective transformation image generated by the projective transformation unit 21 to generate an interpolated ortho image. Do.
[0097]
Specifically, after each ortho image and each projective transformation image are overlapped, first, a plurality of ortho images are synthesized by interpolating pixels. In other words, the value is used for the point where a pixel is obtained only for a certain ortho image, and if the value is obtained for multiple ortho images, the main point is located close to the target position. Are synthesized using pixel values in an image to be photographed, that is, an image in which the pixel is photographed with the pixel facing as much as possible.
[0098]
Further, with respect to pixels in the omission region in the ortho image in which a value is not obtained due to the influence of concealment, an interpolation ortho image is generated by quoting the pixel of the projective transformation image at the corresponding position and performing an interpolation process. At this time, the image from which the pixel is cited is an image of the plurality of projective transformation images having a principal point close to the target position, that is, the one facing the target point as much as possible. As a result, it is possible to reduce the case where the building wall surface that falls down due to the plane approximation is used for the concealment region, and to improve the image quality. Further, by blurring the boundary portion or the quoting portion at the time of quoting, it is possible to suppress the influence of the generated edge, and to generate an interpolated ortho image with less sense of incongruity.
[0099]
As described above, according to the present invention, noise included in the original DEM data can be reduced by using the smoothed DEM data for the orthorectification process, and the image quality of the ortho image is reduced due to the noise. Therefore, a natural ortho image with less noise can be generated.
[0100]
In addition, the present invention interpolates a missing region of an ortho image by projective transforming the image in accordance with a rotation angle with respect to the orthographic projection plane, and substituting a corresponding region pixel of the image in a missing region of the ortho image. Therefore, it is possible to generate an ortho image with less sense of incongruity due to the missing area.
[0101]
【The invention's effect】
As described above, the image processing apparatus of the present invention extracts three-dimensional data by stereo matching processing in an image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data. The three-dimensional data is included in the three-dimensional data by applying a smoothing filter to remove and smooth the noise of the three-dimensional data, and orthorectifying the image using the smoothed three-dimensional data. There is an effect that it is possible to obtain a more natural ortho image that is not easily affected by noise and that is less uncomfortable due to a defective region.
[0102]
According to another image processing apparatus of the present invention, in an image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, the three-dimensional data is extracted by stereo matching processing. The image is orthographically transformed, and the image is projectively transformed according to the orientation angle with respect to the orthographic projection plane, and the ortho image obtained by the orthographic transformation of the image and the projective transformation image obtained by the projective transformation are synthesized. By generating an interpolated ortho image in which the missing pixels are interpolated, it is possible to obtain a more natural ortho image that is less affected by noise included in the three-dimensional data and that has less uncomfortable feeling caused by the defective area. The effect is obtained.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a flowchart showing the operation of the image processing apparatus according to the first embodiment of the present invention.
FIG. 3 is a block diagram showing a configuration of an image processing apparatus according to a second embodiment of the present invention.
FIG. 4 is a flowchart showing the operation of the image processing apparatus according to the second embodiment of the present invention.
FIG. 5 is a block diagram showing a configuration of an image processing apparatus according to a third embodiment of the present invention.
FIG. 6 is a flowchart showing the operation of the image processing apparatus according to the third embodiment of the present invention.
FIG. 7 is a block diagram showing a configuration of an image processing apparatus according to a fourth embodiment of the present invention.
FIG. 8 is a flowchart showing an operation of the image processing apparatus according to the fourth embodiment of the present invention.
FIG. 9 is a schematic diagram showing stereo shooting geometry of the present invention.
FIG. 10 is a schematic diagram showing an example of aerial photography taken from four directions in the embodiment of the present invention.
FIGS. 11A and 11B are schematic diagrams showing the difference between a central projection image and an orthogonal image. FIGS.
FIG. 12 is a block diagram illustrating a configuration of an image processing apparatus according to a conventional example.
FIG. 13 is a flowchart illustrating an operation of an image processing apparatus according to a conventional example.
FIG. 14 is a schematic diagram showing the occurrence of a missing area in an ortho image.
[Explanation of symbols]
1, 2, 3, 4 image processing apparatus
11 Stereo processing means
12 DEM smoothing means
13 Orthorectification means
14 Recording media
21 Projective transformation means
22 Image composition means
41 Ortho-image generation means
42 Stereo pair setting means

Claims (13)

An image processing apparatus for generating three-dimensional data by performing stereo matching processing on a plurality of images, wherein the stereo processing means extracts the three-dimensional data by the stereo matching processing, and is extracted by the stereo processing means and DEM smoothing means for smoothing to remove noise of the three-dimensional data by applying a smoothing filter to the three-dimensional data, the image using the three-dimensional data smoothed by the DEM smoothing means An image processing apparatus comprising orthorectifying means for orthogonally transforming all contained pixels individually .   Projection conversion means for projective conversion of the image according to the orientation angle with respect to the orthographic projection plane, and a missing pixel by combining the ortho image obtained by the orthorectification means and the projection conversion image obtained by the projection conversion means The image processing apparatus according to claim 1, further comprising: an image synthesizing unit that generates an interpolated ortho image obtained by interpolating.   Stereo pair setting means for selecting a pair of images from the plurality of images, wherein the stereo processing means, the orthorectifying means, and the projective transformation means are for the image pair selected by the stereo pair setting means. The image processing apparatus according to claim 2, wherein each processing is performed.   An image processing apparatus that performs stereo matching processing on a plurality of images to generate three-dimensional data, wherein stereo processing means for extracting the three-dimensional data by the stereo matching processing, and using the three-dimensional data, the stereo processing means Orthorectification means for orthographic transformation of the image, projection transformation means for projective transformation of the image according to the orientation angle with respect to the orthographic projection plane, orthoimage obtained by the orthorectification means, and the projection transformation means An image processing apparatus, comprising: an image synthesizing unit that generates an interpolated ortho image in which a missing pixel is interpolated by synthesizing a projective conversion image.   The image synthesizing means interpolates pixels for which values are not obtained in a plurality of ortho-images, and has a photographing main axis closest to the point in an image having values for pixels with which values are obtained in duplicate. By selecting and synthesizing an ortho image and interpolating using a corresponding pixel from a projective transformation image having a photographing principal axis closest to the point for a pixel for which no value is obtained in any of the plurality of generated ortho images The image processing apparatus according to claim 4, wherein the interpolated ortho image is synthesized. An image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images in an image processing device, wherein the image processing device extracts the three-dimensional data by the stereo matching processing; Applying a smoothing filter to the extracted three-dimensional data to remove and smooth the noise of the three-dimensional data; and using the smoothed three-dimensional data, all pixels included in the image An image processing method comprising the step of performing orthogonal transformation individually .   Projection conversion of the image according to the orientation angle with respect to the orthographic projection plane is performed on the image processing apparatus, and the ortho image obtained by orthographic conversion of the image and the projection conversion image obtained by the projective transformation are combined. 7. An image processing method according to claim 6, further comprising: generating an interpolated ortho image in which the missing pixel is interpolated.   The image processing apparatus includes a step of selecting a pair of images from the plurality of images, and extracting the three-dimensional data, orthonormal transformation of the image, and projective transformation for the selected image pair. The image processing method according to claim 7, wherein:   An image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images in an image processing device, wherein the image processing device extracts the three-dimensional data by the stereo matching processing; Orthorectifying the image using the three-dimensional data, projecting transforming the image in accordance with an orientation angle with respect to an orthographic projection plane, an ortho image obtained by orthographic transformation of the image, and the projection And a step of generating an interpolated ortho image in which the missing pixels are interpolated by synthesizing with the projective transformation image obtained by the conversion.   The step of generating the interpolated ortho image interpolates pixels for which values are not obtained in a plurality of ortho images, and is closest to the point in the image having values for pixels that are obtained with duplicate values. Select and synthesize an ortho image with a shooting principal axis, and interpolate using the corresponding pixel from the projective transformation image with the shooting principal axis closest to that point for pixels for which no value is obtained in any of the multiple generated ortho images The image processing method according to claim 9, wherein the interpolated ortho image is synthesized. A program of an image processing method for generating three-dimensional data by performing stereo matching processing on a plurality of images, the computer extracting the three-dimensional data by the stereo matching processing, and the extracted 3-dimensional data to apply a smoothing filter processing for smoothing and removing noise of the three-dimensional data, orthographic and individually all the pixels included in the image using the three-dimensional data smoothed A program for executing conversion processing.   A program of an image processing method for generating a three-dimensional data by performing a stereo matching process on a plurality of images, the computer extracting the three-dimensional data by the stereo matching process; Obtained by orthographic conversion of the image, orthographic image obtained by orthographic transformation of the image, and processing of projecting transformation of the image according to the orientation angle with respect to the orthographic projection plane. A program for executing a process of generating an interpolated ortho image in which a missing pixel is interpolated by synthesizing with a projective transformation image.   When executing the process of generating the interpolated ortho image on the computer, an image having a value with respect to a pixel whose value is obtained redundantly while interpolating pixels whose values are not obtained in a plurality of ortho images Of these, an orthoimage having the photographing principal axis closest to the point is selected and synthesized, and a projected transformation image having the photographing principal axis closest to the point is obtained for a pixel for which no value is obtained in any of a plurality of generated orthoimages. 13. The program according to claim 12, wherein the interpolated ortho image is synthesized by performing interpolation using corresponding pixels.

Images