JP7558082B2 - Appropriate Image Selection System - Google Patents

Description

This invention relates to photogrammetry, and more specifically to an appropriate image selection system that can select images that are more suitable for stereoscopic viewing of features.

When creating a topographical map or a three-dimensional terrain model (a terrain model composed of three-dimensional coordinates), terrain measurements are usually taken over a wide area. When taking topographical measurements over a wide area, photogrammetry is sometimes used to map (model) the terrain using aerial photographs taken from an aircraft. This type of photogrammetry involves preparing a pair of different aerial photographs (so-called stereo pair photographs) of the same location, and using stereoscopic vision (also called stereoscopic vision) to identify the same object in both photographs and use the difference in the object's position in the photographs (parallax) to determine the three-dimensional coordinates of the object on the ground.

The coordinates of features captured in aerial photographs are found by a forward intersection method based on the camera center point (principal point). Therefore, the camera specifications, such as the coordinates of the principal point and shooting direction when each stereo pair photograph was taken, the focal length of the camera, the deviation of the principal point position, and various distortions (radial distortion, asymmetric distortion, etc.), are known. The camera principal point coordinates and shooting direction when the photograph was taken are called "exterior orientation parameters", while the focal length of the camera, the deviation of the principal point position, and various distortions are called "interior orientation parameters", and the exterior orientation parameters and interior orientation parameters are collectively called orientation parameters.

As explained above, exterior orientation parameters are known in order to determine the coordinates of terrestrial objects using aerial photogrammetry, but these exterior orientation parameters are not necessarily determined by measurement, etc. For example, if an aircraft is equipped with a Global Navigation Satellite System (GNSS) and an Inertial Measurement Unit (IMU), the exterior orientation parameters can be obtained directly, but if such measuring equipment is not installed, the exterior orientation parameters can be obtained by aerial triangulation.

However, photography for photogrammetry can also involve multiple high-lap photography, in which the same location is photographed from multiple directions, as shown in Figure 5. Figure 5 is a model diagram that shows a schematic diagram of the situation in which multiple high-lap photography is performed, Figure 6(a) is an image diagram showing a 3 x 3 multiple-photographed image obtained by multiple high-lap photography, and Figure 6(b) is an image diagram showing a 5 x 5 multiple-photographed image obtained by multiple high-lap photography.

In this way, when there are many images taken of the same location, the results obtained may differ depending on the stereo pair photos selected, and in some cases, the desired results may not be obtained. Therefore, Patent Document 1 proposes a technology for automatically selecting suitable stereo pair image candidates.

JP 2017-49883 A

The technology disclosed in Patent Document 1 automatically selects appropriate stereo pair images using a "skewer vector" based on the view point of the user performing stereoscopic viewing, and an "image vector" based on the principal point of photography. Therefore, the technology disclosed in Patent Document 1 allows the user to perform stereoscopic viewing favorably, but on the other hand, there is room for improvement in terms of avoiding occlusion.

For example, in cases where it is desired to ascertain the three-dimensional coordinates of a road edge, occlusion may occur due to road trees covering the road edge. When photographing from the sky (especially directly above), an aerial photograph may be obtained that includes road trees covering the road edge, and if such an aerial photograph is used as a stereo pair image, it is not possible to obtain the appropriate three-dimensional coordinates of the road edge. In contrast, an aerial photograph taken from an oblique angle may include the road edge, and if this aerial photograph is used as a stereo pair image, it is possible to obtain the appropriate three-dimensional coordinates of the road edge. In other words, if it is expected that occlusion problems will occur, it is desirable to select as the stereo pair images those aerial photographs with the least amount of occlusion from the many aerial photographs obtained by multiple high-speed overlap photography, etc.

The objective of the present invention is to solve the problems inherent in the prior art, namely, to provide an appropriate image selection system that can select images with minimal occlusion from acquired aerial photographs as stereo pair images.

The present invention focuses on the point that aerial photographs with minimal occlusion are selected by utilizing the angular difference between a "normal vector" based on a terrain model and a "line of sight vector" based on the principal point of photography, and is an invention based on a previously unconventional idea.

The appropriate image selection system of the present invention is a system for selecting appropriate images to be used for stereoscopic viewing of a target area, and is equipped with an image storage means, a terrain model storage means, a normal vector calculation means, a line of sight vector calculation means, an angle calculation means, and an image selection means. The image storage means is a means for storing a plurality of images obtained by photographing the target area with some overlapping, and the terrain model storage means is a means for storing a terrain model that represents the target area in three dimensions. The normal vector calculation means is a means for calculating a "normal vector" at a gaze point set within the target area based on a terrain model, and the line of sight vector calculation means is a means for calculating a "line of sight vector" consisting of the principal point of photography when the image was acquired and the gaze point. The angle calculation means calculates the angle difference between the normal vector at the gaze point and the line of sight vector related to the image, and the image selection means selects an appropriate image suitable for the gaze point based on the angle difference calculated by the angle calculation means.

The appropriate image selection system of the present invention can also use a terrain model that has multiple grid points with coordinates and multiple divided regions.

The appropriate image selection system of the present invention may further include a gradient calculation means and an area extraction means. The gradient calculation means is a means for calculating the gradient of a divided area of a terrain model, and the area extraction means is a means for extracting a divided area whose gradient exceeds a predetermined threshold value. In this case, a three-dimensional coordinate point related to the divided area extracted by the area extraction means (a lattice point of this divided area, or a constituent point of a feature polygon or the like included in the divided area) is set as the point of interest.

The appropriate image selection system of the present invention may further include a two-dimensional map information storage means and a feature extraction means. The two-dimensional map information storage means is a means for storing two-dimensional map information in which features of a target area are assigned feature classifications, and the feature extraction means is a means for extracting features (e.g., road edges, sidewalk edges, and the outer edge of a median strip) that have been assigned feature classifications based on the two-dimensional map information. In this case, a three-dimensional coordinate point related to a divided area that includes the features extracted by the feature extraction means (e.g., road edges, sidewalk edges, and the outer edge of a median strip) is set as the gaze point.

The appropriate image selection system of the present invention has the following advantages.
(1) Since the three-dimensional coordinates of the object are calculated after selecting an image with minimal occlusion, appropriate results can be obtained even for features that are easily shaded, such as road edges.
(2) Since no judgment by the user is required, the effort required by the user is reduced and human errors due to subjectivity, etc. are avoided, making it possible to obtain efficient and highly accurate results.

A model diagram showing a schematic of the "normal vector" and "line of sight vector." 1 is a block diagram showing the main configuration of a suitable image selection system according to the present invention; FIG. 2 is a flowchart showing the flow of main processes when the appropriate image selection system of the present invention is used. 4 is a flowchart showing a main process flow in which a gaze point is automatically set by a viewpoint setting means. FIG. 13 is a model diagram showing a schematic diagram of a situation in which multiple high-lap photography is performed. FIG. 1A is an image diagram showing a 3×3 multiplex image obtained by multiple high-lap photography, and FIG. 1B is an image diagram showing a 5×5 multiplex image obtained by multiple high-lap photography.

An example of an implementation of the appropriate image selection system of the present invention is explained below with reference to the drawings.

1. Overall Overview One of the technical features of the present invention is to select an aerial photograph with less occlusion by utilizing the angular difference between a "normal vector" and a "line-of-sight vector." Here, we will explain the normal vector VN and line-of-sight vector VS, and the procedure for selecting an aerial photograph, with reference to FIG.

When calculating the normal vector VN and the line of sight vector VS, a gaze point PV is set as described below. This gaze point PV is a point for which the user wishes to determine the coordinates, such as a road edge, and it is particularly advisable to set a point where occlusion may occur, such as a road edge covered by roadside trees.

As shown in FIG. 1, the normal vector VN is set in the normal direction at the gaze point PV, and is a vector of an arbitrary length (for example, unit length) starting from the gaze point PV. The normal direction at the gaze point PV can be set by using a three-dimensional terrain model such as a DSM (Digital Surface Model) or a DTM (Digital Terrain Model). Usually, a three-dimensional terrain model is composed of a plurality of lattice points to which three-dimensional coordinates are assigned, and a plurality of divided regions (hereinafter referred to as "meshes"). A mesh including the gaze point PV (or the gaze point PV is a lattice point) is extracted, and the direction perpendicular to the inclination (attitude) of the mesh can be set as the normal direction at the gaze point PV. Alternatively, the normal vector VN can be set based on the normal direction determined by various conventional methods, such as determining the normal direction using a triangulated irregular network (TIN) formed from random data.

On the other hand, the line of sight vector VS is a vector whose origin is the photographing principal point PM and whose end point is the gaze point PV, as shown in FIG. 1. Then, once the normal vector VN and the gaze vector VS are obtained, the angle between the normal vector VN and the gaze vector VS (hereinafter referred to as the "vector angle") is obtained. In principle, only one normal vector VN is set for one gaze point PV, but the same number of gaze vectors VS can be set for one gaze point PV as the number of photographing principal points PM. In other words, multiple vector angles (as many as the number of photographing principal points PM) can be obtained for one gaze point PV. Therefore, the line of sight vector VS that gives the smallest vector angle is selected, and an aerial photograph associated with this line of sight vector VS is selected. A small vector angle means that the normal vector VN is generally directed toward the photographing principal point PM and the gaze vector VS is generally directed toward the gaze point PV, so the aerial photograph associated with that line of sight vector VS can be considered an image with little occlusion (i.e., suitable for stereoscopic viewing).

2. Appropriate Image Selection System Next, the appropriate image selection system of the present invention will be described in detail with reference to the drawings. Fig. 2 is a block diagram showing the main components of the appropriate image selection system 100 of the present invention. As shown in this figure, the appropriate image selection system 100 of the present invention is configured to include a normal vector calculation means 101, a line of sight vector calculation means 102, an angle calculation means 103, an image selection means 104, an image storage means 109, and a 3D topographical model storage means 110, and may further include a gradient calculation means 105, an area extraction means 106, a feature extraction means 107, a gaze point setting means 108, a 2D map information storage means 111, a shooting parameter storage means 112, and the like.

The normal vector calculation means 101, line of sight vector calculation means 102, angle calculation means 103, image selection means 104, gradient calculation means 105, area extraction means 106, feature extraction means 107, and gaze point setting means 108 that constitute the appropriate image selection system 100 can be manufactured as dedicated devices, or a general-purpose computer device can be used. This computer device has a processor such as a CPU and memories such as ROM and RAM, and some also include input means such as a mouse and keyboard and a display. Examples of such a computer device include portable terminal devices such as tablet computers (iPad (registered trademark) and smartphones, as well as personal computers (PCs) and servers.

The image storage means 109, 3D terrain model storage means 110, 2D map information storage means 111, and shooting parameters storage means 112 can be stored in a computer device or in a database server. When stored in a database server, they can be placed on a local network (LAN: Local Area Network) or can be a cloud server that stores data via the Internet (i.e., wireless communication).

Below, we will explain in detail each of the main elements that make up the appropriate image selection system 100.

(3D terrain model storage means and 2D map information storage means)
The 3D terrain model storage means 110 is a means for storing a three-dimensional terrain model (DSM, DTM, etc.) of a target range (hereinafter referred to as the "target area"). On the other hand, the 2D map information storage means 111 is a means for storing two-dimensional map information of the target area. Here, the two-dimensional map information is two-dimensional terrain information such as a topographical map that does not have height information, and is map information that includes information on features (hereinafter referred to as "feature information"). The feature information includes information on the planar position and shape of features such as polygons and polylines (hereinafter referred to as "feature figures"), and attribute information on the features. Furthermore, the attribute information on the features includes the types of features such as roads, road edges, sidewalk edges, the outer edge of a median strip (hereinafter simply referred to as the "median strip edge"), buildings, the outer edge of buildings, parks, and schools (hereinafter referred to as "feature classification"). In other words, feature figures are arranged on a plane within the target area, and attribute information on the features (including feature classification) is associated with these feature figures.

(Image storage means and photographing data storage means)
The image storage means 109 is a means for storing a plurality of images obtained by photographing a target area with some overlapping, and can store multiple images obtained by multiple high-lap photography as shown in Fig. 6. The photography specification storage means 112 stores the photography principal point PM when each image stored in the image storage means 109 is acquired, and can also store other exterior orientation parameters and interior orientation parameters. The photography principal point PM is stored in the photography specification storage means 112 in association (linked) with the image related to the photography principal point PM.

(Point of interest setting means)
The gaze point setting means 108 is a means for setting a gaze point PV. The gaze point setting means 108 can be designed so that a user can set the gaze point PV by operating a pointing device or a keyboard while visually viewing a terrain model or an image, or can be designed so that the gaze point PV is set automatically without user operation. The process of the gaze point setting means 108 automatically setting the gaze point PV will be described below.

As mentioned above, the gaze point PV is a point for which the user seeks to determine the coordinates, such as a road edge, and is preferably set to a point where occlusion may occur, such as a road edge covered by roadside trees. Therefore, a mesh for which occlusion is predicted (hereinafter referred to as a "specific mesh") is extracted from the 3D terrain model. Specifically, the gradient calculation means 105 reads out the 3D terrain model from the 3D terrain model storage means 110 and calculates the mesh gradient (hereinafter referred to as a "mesh gradient"), and the area extraction means 106 extracts a mesh that exceeds a predetermined threshold (hereinafter referred to as a "gradient threshold") as a specific mesh (Figure 1). Note that the mesh gradient can be calculated in advance and stored in the 3D terrain model storage means 110, in which case it is sufficient to simply read it out from the 3D terrain model storage means 110 without calculating the mesh gradient. The mesh gradient is calculated as an angle with the horizontal plane, and similarly, the gradient threshold is set as an angle with the horizontal plane. Usually, when there is no obstruction such as roadside trees, the feature (i.e., mesh) is generally horizontal or has a gentle slope. In contrast, a mesh that is significantly inclined from the horizontal plane is not in a normal state, i.e., the presence of an obstruction such as a roadside tree is suspected. When a specific mesh is extracted by the area extraction means 106, the gaze point setting means 108 sets the grid point of the specific mesh as the gaze point PV (FIG. 1). Alternatively, in a case where the 3D terrain model storage means 110 stores polygons, polylines, etc. of features separately (independently) from the 3D terrain model, a three-dimensional constituent point (hereinafter referred to as a "3D constituent point") of a polygon, etc. included in the specific mesh can be set as the gaze point PV. In other words, any point related to the specific mesh (a constituent point or a point included in the specific mesh) that has three-dimensional coordinates can be set as the gaze point PV.

In addition, the gaze point PV may be set for a feature that has been assigned a specific feature classification (hereinafter referred to as a "specific feature"). In this case, the feature extraction means 107 obtains the plane coordinates of the specific feature by consulting the 2D map information storage means 111 (Figure 1). For example, when the user specifies a road edge as a specific feature, the feature extraction means 107 obtains the plane coordinates of the road edge from the 2D map information storage means 111. Of course, the user can also specify a sidewalk edge or a median strip edge as a specific feature instead of a road edge. The gaze point setting means 108 then sets a mesh corresponding to the plane coordinates of the specific feature (for example, a road edge) as a specific mesh, and sets a 3D constituent point such as a grid point of the specific mesh or a feature polygon contained in the specific mesh as a gaze point PV (Figure 1). At this time, the gradient calculation means 105 may calculate a mesh gradient for the mesh corresponding to the plane coordinates of the specific feature obtained by the feature extraction means 107, and the area extraction means 106 may extract it as a specific mesh.

(Normal vector calculation means)
The normal vector calculation means 101 is a means for calculating a normal vector VN. More specifically, the normal vector calculation means 101 reads out a mesh related to the gaze point PV set by the gaze point setting means 108 from the 3D terrain model storage means 110, and calculates the normal vector VN based on the inclination (attitude) of the mesh. Note that the normal vector calculation means 101 can calculate the normal vector VN at the timing when the gaze point setting means 108 sets the gaze point PV, or can calculate the normal vector VN in advance for each 3D constituent point such as a mesh lattice point or a feature polygon.

(Gaze vector calculation means)
The line-of-sight vector calculation means 102 is a means for calculating a line-of-sight vector VS. More specifically, the line-of-sight vector calculation means 102 reads out the photographing principal point PM from the photographing parameter storage means 112, and calculates the line-of-sight vector VS based on the photographing principal point PM and the gaze point PV set by the gaze point setting means 108. As described above, the line-of-sight vector calculation means 102 can calculate the line-of-sight vector VS for the number of photographing principal points PM. Of course, it is also possible to use a specification in which the line-of-sight vector VS is calculated for all photographing principal points PM (i.e., all images), or it is also possible to use a specification in which the line-of-sight vector VS is calculated after selecting images that meet specific conditions, such as selecting only images that include the gaze point PV within the angle of view.

(Angle Calculation Means)
The angle calculation means 103 is a means for calculating a vector angle based on the normal vector VN and the line of sight vector VS. The angle calculation means 103 can also calculate an inner product based on the normal vector VN and the line of sight vector VS (hereinafter referred to as a "vector inner product") instead of (or in addition to) the vector angle. However, when calculating the vector inner product, the calculation is performed after one of the normal vector VN and the line of sight vector VS is reversed (multiplied by a negative value).

(Image Selection Means)
The image selection means 104 is a means for selecting an image suitable for stereoscopic viewing (hereinafter referred to as an "appropriate image") from the image storage means 109 based on the vector included angle (vector inner product). When making a judgment based on the vector included angle, an image associated with a small value of the vector included angle (i.e., associated with the line-of-sight vector VS) is selected, and when making a judgment based on the vector inner product, an image associated with a large value of the vector inner product (i.e., associated with the line-of-sight vector VS) is selected. At this time, the top two images in the stereo pair can be selected as the appropriate images, or all images whose vector included angle is below a predetermined threshold (exceeding a predetermined threshold in the case of a vector inner product) can be selected as the appropriate images, or images positioned at the top (e.g., 1st to 6th place) can be selected as the appropriate images.

(Processing flow)
Next, the main process flow when using the appropriate image selection system 100 of the present invention will be described with reference to Fig. 3. In Fig. 3 and the flow diagram of Fig. 4 described later, the action to be performed is shown in the center column, the things necessary for that action are shown in the left column, and the things resulting from that action are shown in the right column.

First, the gaze point PV is set by the gaze point setting means 108 (Step 210 in FIG. 3). As described above, the gaze point setting means 108 can be designed to allow the user to set the gaze point PV by operating a pointing device or a keyboard while visually viewing a terrain model or an image, or can be designed to set the gaze point PV automatically without user operation. The process of automatically setting the gaze point PV by the gaze point setting means 108 will be described below with reference to FIG. 4.

When a specific feature (e.g., a road edge) is specified by a user operation, the feature extraction means 107 acquires the planar coordinates of the specific feature from the 2D map information storage means 111 (Step 201 in FIG. 4). Then, the gradient calculation means 105 calculates the mesh gradient for the mesh corresponding to the planar coordinates (or reads out a mesh previously calculated by the gradient calculation means 105) (Step 202 in FIG. 4), and the area extraction means 106 extracts a mesh that exceeds the gradient threshold as a specific mesh (Step 203 in FIG. 4). When the specific mesh is extracted by the area extraction means 106, the gaze point setting means 108 sets a three-dimensional coordinate point related to the specific mesh (a lattice point of the specific mesh or a 3D constituent point such as a feature polygon included in the specific mesh) as the gaze point PV (Step 210 in FIG. 4).

When the gaze point PV is set by the gaze point setting means 108, the normal vector calculation means 101 calculates the normal vector VN for the gaze point PV (Step 220 in FIG. 3), and the line-of-sight vector calculation means 102 calculates the line-of-sight vector VS (Step 230 in FIG. 3). When the normal vector VN is calculated in advance for each 3D constituent point such as a mesh lattice point or a feature polygon, it is sufficient to simply read out the normal vector VN corresponding to the gaze point PV. Then, when the normal vector VN and the line-of-sight vector VS are obtained, the angle calculation means 103 calculates the vector angle (vector inner product) (Step 240 in FIG. 3). The line-of-sight vector calculation means 102 repeatedly calculates the line-of-sight vector VS while changing the photographing principal point PM, and the angle calculation means 103 also repeatedly calculates the vector angle. At this time, it is possible to calculate the line-of-sight vector VS and the vector angle for all photographic principal points PM, and it is also possible to calculate the line-of-sight vector VS and the vector angle for photographic principal points PM related to an image in which the gaze point PV is included within the angle of view.

When the vector angle is calculated by the angle calculation means 103, the image selection means 104 selects the appropriate image from the image storage means 109 based on the vector angle. However, it is possible that the gaze point PV is not included in the appropriate image selected by the image selection means 104. Therefore, the image selected by the image selection means 104 based on the vector angle is temporarily set as a "provisional image" (Step 250 in FIG. 3), and it is determined whether the gaze point PV is included in this provisional image (Step 260 in FIG. 3). That is, if the gaze point PV is included in the provisional image (Yes in Step 260 in FIG. 3), this provisional image is determined as the appropriate image (Step 270 in FIG. 3), while if the gaze point PV is not included in the provisional image (No in Step 260 in FIG. 3), the shooting principal point PM is changed and a series of processes (Steps 230 to 260 in FIG. 3) are repeatedly executed. However, in cases where the line of sight vector VS and the vector angle are calculated for the principal point PM of an image in which the gaze point PV is included within the angle of view, the provisional image setting process (Step 250 in FIG. 3) and the provisional image determination process (Step 260 in FIG. 3) can be omitted.

The appropriate image selection system of the present invention can be particularly useful for managing various facilities, including road facilities, and as map information used in automated driving. Furthermore, the present invention can provide highly accurate step information that is useful for elderly people and wheelchair users, and can also be effectively used in disaster prevention planning. The appropriate image selection system of the present invention is not only applicable to industry, but is also expected to make a significant contribution to society.

100 Appropriate image selection system of the present invention 101 Normal vector calculation means 102 Line of sight vector calculation means 103 Angle calculation means 104 Image selection means 105 Gradient calculation means 106 Area extraction means 107 Feature extraction means 108 Point of interest setting means 109 Image storage means 110 3D topographical model storage means 111 2D map information storage means 112 Photographing specifications storage means PM Photographing principal point PV Point of interest VN Normal vector VS Line of sight vector

Claims (5)

1. A system for selecting suitable images for use in stereoscopic viewing of a target area, comprising:
an image storage means for storing a plurality of images obtained by photographing the target area with some overlapping;
a terrain model storage means for storing a terrain model that represents the target area in three dimensions;
a normal vector calculation means for calculating a normal vector at a gaze point set within the target area based on the terrain model;
a gaze vector calculation means for calculating a gaze vector consisting of a photographing principal point when the image was acquired and the gaze point;
An angle calculation means for calculating an angular difference between the normal vector at the gaze point and the line-of-sight vector related to the image;
and an image selection means for selecting the appropriate image suitable for the gaze point based on the angle difference calculated by the angle calculation means.
A system for selecting an appropriate image. The terrain model includes a plurality of grid points to which coordinates are assigned, and a plurality of divided regions.
2. The system for selecting suitable images according to claim 1. gradient calculation means for calculating a gradient of the divided region;
and a region extraction means for extracting the divided region in which the gradient exceeds a predetermined threshold value,
a three-dimensional coordinate point relating to the divided area extracted by the area extraction means is set as the gaze point;
3. The system for selecting suitable images according to claim 2. a two-dimensional map information storage means for storing two-dimensional map information in which feature classifications are assigned to features in the target area;
and a feature extraction means for extracting the features to which the specific feature classification is assigned based on the two-dimensional map information,
The three-dimensional coordinate point related to the divided area including the feature extracted by the feature extraction means is set as the gaze point.
4. The system for selecting suitable images according to claim 3. a two-dimensional map information storage means for storing two-dimensional map information in which feature classifications are assigned to features in the target area;
and a feature extraction means for extracting a road edge, a sidewalk edge, or a median strip edge based on the two-dimensional map information,
the three-dimensional coordinate point relating to the divided area including the road edge, the sidewalk edge, or the median strip edge extracted by the feature extraction means is set as the gaze point;
4. The system for selecting suitable images according to claim 3.

Images