JP7658939B2 - Comparison image search device, method and program - Google Patents

Description

The present invention relates to a comparison image search device, method, and program for searching for comparison images suitable for use in image-based camera pose estimation.

Camera pose estimation based on images is called VPS (Visual Positioning System), and is used as a core technology in various fields such as robotics and augmented reality. Patent Document 1 discloses a technology related to VPS that matches a 3D point cloud map with a query image captured later and estimates the posture of the query image in the map. Patent Document 2 discloses a technology that estimates the approximate camera posture of the query image in the 3D mesh map, renders a 2D comparison image from the 3D mesh based on the estimated approximate camera posture, and matches the comparison image with the query image in order to estimate the posture of the query image using a general-purpose 3D mesh map instead of a 3D point cloud map.

Patent Publication No. 2021-174285 Patent Application No. 2022-063059

When comparing a query image with images stored in a database, it is necessary to extract images suitable for comparison from the database. However, the accuracy of methods for extracting images based on the similarity of the images is insufficient. Patent Document 2 also assumes the use of GPS (Global Positioning System) information and geomagnetic sensors, but these can only be used in limited locations.

In view of the problems with the conventional technology, the present invention aims to provide a comparison image search device, method, and program capable of searching for comparison images suitable for use in image-based camera pose estimation.

To achieve the above object, the present invention is a comparison image search device that repeatedly estimates the camera position and orientation of the query image in the three-dimensional map for each round by performing image matching between a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a three-dimensional map at one or more camera positions and/or images taken in advance at one or more camera positions and orientations in the real world that is the subject of the three-dimensional map, and repeats the process of estimating the camera position and orientation of the query image in the three-dimensional map for each round by performing image matching between the current comparison image set selected from the reference image set and the query image based on the camera position and orientation estimated in the previous round until a convergence determination is obtained, and outputs the comparison image set used in the round for which the convergence determination is obtained. Also, the present invention is characterized by a method and a program corresponding to the device.

According to the present invention, the camera position and orientation are updated and estimated through repeated processing until a convergence judgment is obtained, and at this time, a comparison image set selected from a reference image set based on this camera position and orientation is updated and used, thereby finally outputting an appropriate image set to be compared with the query image.

1 is a functional block diagram of a comparison image search device according to an embodiment. 1 is a flowchart of an operation of a comparison image search device according to an embodiment. FIG. 13 is a schematic representation of the rendering of a comparison image from a three-dimensional map. FIG. 13 is a schematic diagram illustrating the rendering of multiple comparison images to obtain a comparison image set. FIG. 13 is a schematic diagram of an estimated camera pose. 10 is a diagram illustrating each process up to calculation of an estimated camera posture based on corresponding point pairs obtained by an image matching unit. FIG. FIG. 13 is a schematic diagram showing a state when the result of convergence determination is NG. 10 is a diagram illustrating a method for determining a pair of corresponding points between three-dimensional coordinates of a three-dimensional map and two-dimensional coordinates of a comparison image. FIG. 5 is a diagram illustrating a method (5-1) for selecting an optimal comparison image. FIG. 5 is a diagram illustrating a method (5-2) for selecting an optimal comparison image. FIG. 5 is a diagram illustrating a method (5-4) for selecting an optimal comparison image. FIG. 13 is a diagram showing an example of the relationship between two-dimensional coordinates (u _j , v _j ) and (width, height). FIG. 1 is a diagram illustrating an example of a hardware configuration of a general computer device.

Figure 1 is a functional block diagram of an optimal comparison image search device 10 for image matching with a 3D map according to an embodiment. The comparison image search device 10 includes a comparison image rendering unit 11, an initial comparison image selection unit 12, an image matching unit 13, an estimated camera posture calculation unit 14, an optimal comparison image selection unit 15, and a convergence determination unit 16.

Figure 2 is a flowchart of the operation of the comparison image search device 10 according to the embodiment. Below, the details of the operation of each functional unit of the comparison image search device 10 in Figure 1 will be explained while explaining each step in Figure 2.

The flow in Figure 2 has a structure in which the processes in steps S3 to S7 are repeated until the convergence test in step S7 is satisfied. This number of repetitions is referred to as N (N=1 is the first repetition, and N increases with each repetition as N=1, 2, 3, ...), and the current number of repetitions in the explanation of the flow in Figure 2 is an arbitrary N, and will be explained separately from the previous N-1th iteration and the future N+1th iteration. Here, steps S1 and S2 belong only to the first N=1 iteration.

(1) Step S1...Comparison image rendering unit 11
In step S1, the comparison image rendering unit 11 renders a set of m comparison images PS={P(i)|i=1, 2, ..., m} (reference image set PS) as a process of constructing an entire database of reference images, and then proceeds to step S2.

Specifically, the comparison image rendering unit 11 renders multiple comparison images PS from a three-dimensional map M that is prepared in advance as input data. The rendered comparison images PS are output to an initial comparison image selection unit 12, which will be described later, and are used for image matching with the query image. The rendered comparison images PS are also output to an optimal comparison image selection unit 15, which will be described later, and become candidates for the comparison images to be selected.

Figure 3 shows a schematic diagram of the rendering of a comparison image P(i) from a 3D map M. For each comparison image P(i) (i = 1, 2, ..., m) belonging to the comparison image set PS, it is possible to render each comparison image P(i) using existing techniques in 3D CG (computer graphics) by providing external parameters EX(i) (corresponding to the camera position and orientation) and internal parameters IN(i) in the world coordinates of the 3D map M. This rendering is possible by preparing, for example, polygon vertex coordinates and polygon textures for the structures that make up the map in the 3D map M.

In this way, as shown in Figure 4, by rendering each of the comparison images P(i) belonging to the comparison image set PS, a comparison image set PS consisting of multiple m images can be obtained. (Note that Figure 4 shows comparison images P(1) to P(13), which are part of the comparison image set PS, as a schematic example.)

The rendering position given by the external parameters EX(i) may be randomly determined at the ground level where the query image Q is assumed to be captured, or in a space without structures (such as a road in the real world). (Each of the external parameters EX(i) may be set comprehensively within the spatial range in which the 3D map M is defined, so that the position and orientation of the query image Q can be estimated by matching it with the set of comparison images PS.) The internal parameters IN(i) may be randomly selected from a predetermined setting, or the same internal parameters as those of the query image Q may be used for all comparison images P(i) (i=1, 2, ..., m).

In addition, for all or part of the rendering images constituting the comparison image set PS, instead of using rendering images generated by rendering in the comparison image rendering unit 11, real photographs used when generating the 3D map M (real photographs taken in the real world, such as a street corner, which is the subject of the 3D map M) may be used. In addition, for the comparison image set PS as a reference image database, instead of executing the processing of the comparison image rendering unit 11 as internal processing in the comparison image retrieval device 10, the comparison image set PS corresponding to the 3D map M may be prepared in advance on an external server or the like, and the comparison image set PS may be read from this external server by the comparison image retrieval device 10 (read by the initial comparison image selection unit 12 and the optimal comparison image selection unit 15), thereby omitting the processing of the comparison image rendering unit 11.

(2) Step S2: First comparison image selection unit 12
In step S2, the initial comparison image selection unit 12 selects the initial comparison image set PS[1] for N=1 from the comparison image set PS obtained by the comparison image rendering unit 11 (i.e., selects it as PS[1] ⊂ PS), and proceeds to step S3.

Specifically, the initial comparison image selection unit 12 selects comparison images suitable for image matching with the query image Q in the image matching unit 13 (first time for N=1), described later, from among the multiple comparison images (the entire comparison image set PS) rendered by the initial comparison image rendering unit 11, to obtain a set PS[1]. The selected comparison images are output to the image matching unit 13, described later, for the first processing for N=1. Here, only one comparison image may be selected, or multiple comparison images may be selected.

(3) Step S3...Image matching unit 13
In step S3, the image matching unit 13 performs image matching between the set of comparison images PS[N] corresponding to the number of repetitions N and the query image Q to obtain corresponding point pairs, and then the process proceeds to step S4.

Specifically, the image matching unit 13 performs image matching between the comparison image set PS[1] selected by the initial comparison image selection unit 12 and the query image Q the first time N=1, and from the second time onwards when N≧2, performs image matching between the comparison image set PS[N] selected by the optimal comparison image selection unit 15 (described later) and the query image Q.

As described above, the matching target set PS[N] differs depending on the number of times N, but the image matching process itself is common regardless of the number of times N, and a corresponding point pair between the two-dimensional coordinates of the query image Q and the two-dimensional coordinates of the comparison image P(i)∈PS[N] is obtained. The obtained corresponding point pair is output to an estimated camera posture calculation unit 14, which will be described later. For image matching, the technology described in Non-Patent Document 1 or the technology described in Patent Document 2, mentioned above, may be used.
[Non-patent Document 1] "Wang, Qianqian, et al. "Learning feature descriptors using camera pose supervision." European Conference on Computer Vision. Springer, Cham, 2020."

When multiple comparison images are input, image matching is performed with each comparison image against one query image, and the results are output to the estimated camera posture calculation unit 14 described below. For example, when there are five comparison images, image matching is performed five times, and corresponding point pairs for five times are output to the estimated camera posture estimation unit 14 described below. However, corresponding point pairs obtained as a result of performing image matching multiple times may be aggregated as the result of one time and output to the estimated camera posture estimation unit 14 described below.

(4) Step S4... Estimated camera attitude calculation unit 14
In step S4, the estimated camera posture calculation unit 14 calculates an estimated camera posture of the query image Q from the result (corresponding point pair) in the image matching unit 13, and the process proceeds to step S5.

Specifically, the estimated camera pose calculation unit 14 obtains corresponding point pairs between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the query image based on the corresponding point pairs between the query image PS and the comparison image P(i)∈PS[N] obtained by the image matching unit 13, and uses them to calculate the estimated camera pose of the query image Q. When the results of multiple image matching operations are input from the image matching unit 13, the estimated camera poses are calculated for the multiple operations, and the better camera pose (the camera pose determined to be the best) is adopted as the estimated camera pose. The method of determining the better camera pose may be, for example, to apply Ransac (random sample voting) described in Non-Patent Document 4 (described later) when estimating the camera pose, and use the one with the largest number of calculated inliers.

In this way, the image matching unit 13 can obtain image matching results integrated into a single result or image matching results from multiple iterations, but in either case, the estimated camera pose calculation unit 14 can obtain one estimated camera pose corresponding to one external parameter EX[N] as the Nth result.

The calculated estimated camera pose of the query image Q is output to an optimum comparison image selection unit 15, which will be described later, and is used to select an optimum comparison image (a set of comparison images PS[N+1] for the next N+1th iteration) from among the multiple comparison images PS rendered by the initial comparison image rendering unit 11. The estimated camera pose of the query image Q is found by first calculating a corresponding point pair between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the query image, and then solving a PnP (Perspective-n-Point) problem, which is a technique described in Non-Patent Document 2.
[Non-Patent Document 2]: "Lepetit, Vincent, Francesc Moreno-Noguer, and Pascal Fua. "Epnp: An accurate o (n) solution to the pnp problem." International journal of computer vision 81.2 (2009): 155."

Figure 5 shows a schematic diagram of the estimated camera pose of the calculated query image Q. As in Figure 3, which explains rendering, the Nth estimated camera pose can be estimated as an external parameter EX[N], which is the position and pose of the camera in the world coordinate system in which the 3D map M is defined.

The corresponding point pairs of the 3D coordinates of the 3D map and the 2D coordinates of the query image can be found via the 2D coordinates of the comparison image based on the corresponding point pairs of the 2D coordinates of the comparison image and the 2D coordinates of the query image, and the corresponding point pairs of the 3D coordinates of the 3D map and the 2D coordinates of the comparison image.

Figure 6 shows a schematic diagram of each process p1 to p4 up to the calculation of the estimated camera pose based on the corresponding point pairs obtained by the image matching unit 13. Process p1 is the result of the image matching unit 13, and for the result of this process p1 (corresponding point pairs of the two-dimensional coordinates of the comparison image P(i) and the two-dimensional coordinates of the query image Q), as shown in process p2, the two-dimensional coordinates of the comparison image P(i) among the corresponding point pairs are determined as corresponding three-dimensional coordinates by referring to the original three-dimensional map M. Therefore, by comparing the results of processes p1 and p2, the three-dimensional coordinates of the corresponding point pairs that correspond to the two-dimensional coordinates of the query image Q are determined as shown in process p3. This allows the camera pose of the query image Q to be estimated by solving the PnP problem shown in process p4.

(5) Step S5... Optimal Comparison Image Selection Unit 15
In step S5, optimal comparison image selection unit 15 selects an optimal set of comparison images PS[N+1] for the next time (N+1th time), and then the process proceeds to step S6.

Specifically, the optimal comparison image selection unit 15 selects an optimal set of comparison images PS[N+1] for the next N+1th iteration from among the multiple comparison images (comparison image set PS) rendered by the comparison image rendering unit 11 based on the estimated camera pose of the query image Q estimated by the estimated camera pose calculation unit 14 (the current estimated camera pose estimated as the external parameter EX[N]). The selected optimal comparison image set PS[N+1] is output to the convergence determination unit 16, which will be described later, and used for convergence determination. (Note that if the result of the convergence determination is that the process is not to be terminated in the current Nth iteration but is to be repeated for the next N+1th iteration, the set PS[N+1] obtained in the current Nth iteration will be used as the target for matching with the query image Q in the image matching unit 13 in step S3 of the next N+1th iteration.)

The comparison image set PS[N+1] may be selected as one or more comparison images. Details of the processing performed by the optimal comparison image selection unit 15 will be described later as an embodiment.

(6) Steps S6, S7, S8...Convergence determination unit 16
In step S6, the convergence determination unit 16 performs a convergence determination, and then the process proceeds to step S7. In step S7, if the determination result in step S6 is convergence, the process proceeds to step S8, and if the determination result in step S6 is convergence, the process returns to step S3, and the above-mentioned processes from step S3 onwards are repeated as the next (N+1)th process. In step S8, the comparison image PS[N+1] (or PS[N]) is output as the search result in the comparison image search device 10, and the flow in FIG. 2 ends.

Specifically, the convergence judgment unit 16 performs a convergence judgment on the comparison image set PS[N+1] selected by the optimal comparison image selection unit 15. If the convergence judgment result is OK (converged judgment), the latest selected comparison image set PS[N+1] is output. If the convergence judgment result is NG (not converged judgment), the process proceeds to "steps S6 → S7 → S3", and in the functional block configuration, the process returns to the image matching unit 13, and image matching is performed again using the latest selected comparison image PS[N+1] (as the N+1th time), and the estimated camera posture calculation unit 14 calculates the estimated camera posture of the query image Q. After that, the optimal comparison image selection unit 15 reselects a comparison image based on the estimated camera posture (external parameter EX[N+1]) of the query image calculated by the estimated camera posture calculation unit 14, and the comparison image set PS[(N+1)+1]=PS[N+2] is output again to the convergence judgment unit 16.

As described above with regard to the flow structure of FIG. 2, this cycle is repeated until the convergence determination unit 16 determines that the result is OK. Convergence may be determined when the most recent (Nth) comparison image PS[N+1] is determined to be the same as the comparison image PS[N] selected immediately before (N-1th). Identicality may be determined by whether the overlap rate (rate at which identical images exist) of the image sets PS[N+1] and PS[N] is equal to or greater than a threshold value.

Alternatively, it may be determined that convergence has occurred if the change between the estimated camera position and orientation (external parameters EX[N]) of the latest (Nth) query image Q and the position and orientation (external parameters EX[N-1]) of the estimated camera orientation of the previous (N-1th) query image Q is within a threshold. The threshold may be set freely, for example, to 1 m for the position. However, in the case of N=1, since there is no previous (N-1th) result, it is always determined as NG and branched to the image matching unit 13. (Note that in the case of convergence determination based on the overlap ratio of sets PS[N+1], PS[N] rather than the position and orientation, it is possible that the convergence determination is obtained by chance at the first time of N=1.) Note that, regarding the threshold determination of the change, the estimated camera position and orientation are composed of a position element (translation component) and a posture element (rotation element), but the changes of both the position element and the posture element may be used as the target of the threshold determination, or only one of them (for example, only the position) may be used as the target of the threshold determination.

Figure 7 is a schematic diagram showing the state when the result of the convergence judgment is NG. It is determined that convergence has not occurred because it is determined that the current Nth comparison image set PS[N+1] does not match the previous N-1th comparison image set PS[N].

Above, we have explained each step in Figure 2 and each functional block in Figure 1. Below, we will further explain an example of the processing of each functional block in Figure 1.

(2) The initial comparison image selection unit 12 selects the comparison image set PS as follows:
(2-1) You may use the Image Retrieval technology described in Non-Patent Document 3,
(2-2) When performing image matching using the technique described in Non-Patent Document 1 or the technique described in Patent Document 2, the Ransac (random sample voting) described in Non-Patent Document 4 may be applied to select a comparison image having a large number of values (inliers) other than the calculated outliers, or
(2-3) After performing image matching using the technology described in Non-Patent Document 1 or the technology described in Patent Document 2, the camera posture of the query image may be estimated using the technology described in Non-Patent Document 2. Then, Ransac described in Non-Patent Document 4 may be applied to select a comparison image with a large number of calculated inliers.
[Non-Patent Document 3]: "Revaud, J., Almazan, J., Rezende, RS, & Souza, CRD (2019). Learning with average precision: Training image retrieval with a listwise loss. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 5107-5116)."
[Non-Patent Document 4]: "MA Fischler and RC Bolles, ``Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography,'' Communications of the ACM 24(6):381-395 (1981).''

(4) When the estimated camera attitude calculation unit 14 obtains a pair of corresponding points between the three-dimensional coordinates of the three-dimensional map M and the two-dimensional coordinates of the comparison image P(i),
(4-1) As shown in FIG. 8, the camera attitude position Pos(i) of the comparison image P(i) (given as a translation component in the external parameters EX(i) consisting of translation components and rotation components during rendering in the comparison image rendering unit 11, and in the case of an actual image rather than a rendered image, similar external parameters EX(i) may be associated and acquired in advance at the time of shooting) and the two-dimensional coordinates of the comparison image P(i), and the extension of the straight line connecting the position Pos(i) of the camera attitude of the comparison image P(i) and the two-dimensional coordinates of the comparison image P(i) may be obtained from the point of intersection with the three-dimensional map M (a structure therein);
(4-2) When the method described in Patent Document 2 is used for image matching, the three-dimensional coordinates of the three-dimensional map may be converted into two-dimensional coordinates on the comparison image P(i) based on the camera attitude of the comparison image P(i).

(5) Various embodiments of the method for selecting the optimal comparison image (set PS[N+1]) in the optimal comparison image selection unit 15 will be described below.
(5-1) For example, as shown in Fig. 9, a comparison image located within a range R close to the position of the estimated camera pose of the query image Q (the translation component of the external parameter EX[N]) may be selected. The coordinates ( _XQ , _YQ , _ZQ ) of the estimated pose of the query image Q in the 3D map can be calculated using the estimated camera pose of the query image Q (rotation matrix _RQ and translation vector _tQ ) as follows:

This can be expressed as:
The coordinates (Xi _{, Yi} , _Zi ) of the camera posture of the comparison image P(i) in the three-dimensional map are the values used when the comparison image is rendered by the comparison image rendering unit 11 (values that are linked in advance at the time of shooting in the case of a real-life image). The distance _dl between the two points of the coordinates ( _XQ , _YQ , _ZQ ) of the estimated posture of the query image Q and the coordinates ( _Xi , _Yi , _Zi ) of the camera posture of the comparison image P(i) may be calculated, and if the distance _dl is smaller, it may be determined that the distance is closer. In addition, the number of comparison images (set PS[N+1]) to be selected may be one with the smallest _dl , or multiple images with the smallest distance _dl may be selected.

(5-2) For example, as shown in FIG. 10, a comparison image may be selected in which the rotation of the estimated camera pose of a query image Q (the rotation component of the external parameter EX[N]) is similar to the rotation of the camera pose of a comparison image. The determination of whether the rotation of the camera poses is similar may be performed by taking a difference d _θ between a rotation angle θ _Q in the quaternion representation of the estimated camera pose of the query image Q and a rotation angle θ _i in the quaternion representation of the camera pose of the comparison image P(i), and determining that the difference d _θ is smaller. In addition, the number of comparison images (set PS[N+1]) to be selected may be one with the smallest d _θ , or multiple images with the smallest d_θ may be selected.

(5-3) For example, a weighted average is calculated using the distance d _l between the position of the estimated posture of the query image obtained by the method of (5-1) and the position of the posture of the comparison image, and the difference d _θ between the rotation angle in the quaternion representation of the posture of the query image obtained by the method of (5-2) and the rotation angle in the quaternion representation of the posture of the comparison image.

A comparison image with a small rotation angle may be selected. Here, w _l and w _θ are adjustable parameters, and by reducing w _l , the influence of d _l increases, so that a comparison image located closer to the position of the query image is more likely to be selected, and by reducing w _θ , the influence of d _θ increases, so that a comparison image with a smaller difference from the rotation angle of the query image is more likely to be selected. In addition, the number of comparison images (set PS[N+1]) to be selected may be one with the smallest W, or multiple images with the smallest d _θ may be selected.

(5-4) For example, as shown in FIG. 11, if the camera pose of a comparison image exists within the field of view V of the estimated camera pose of a query image Q, that comparison image may be selected. If the following expressions (3) and (4) are satisfied simultaneously, it may be determined that the camera pose of a comparison image P(j)∈PS[N+1] exists within a rendering image (i.e., within the field of view V) using the estimated pose of the query image Q (pose estimated as the external parameter EX[N]). (In other words, if either expression (3) or (4) does not hold, it is determined that the comparison image P(j) is not "P(j)∈PS[N+1]".)
0≦ _uj ≦width …(3)
0≦ _vj ≦height …(4)

where (width, height) are the width and height in the image size of the query image Q, and (u _j , v _j ) are the position of the camera pose of the comparison image P(j) (each candidate image P(j) to be determined as to whether it belongs to PS[N+1]) in the camera coordinate system of the query image Q. An example of the relationship between the two-dimensional coordinates (u _j , v _j ) and (width, height) is shown in FIG. 12. (u _j , v _j ) can be converted from the coordinates (X j , Y _j , Z _j ) in the three-dimensional map M of the camera pose of the comparison image P(j) as a candidate, using the perspective projection matrix P (matrix P equivalent to information on the internal parameters of the query image Q) _.

However, the projection matrix P is calculated by using the camera rotation matrix R, translation vector t, and internal parameters K when rendering the comparison image.
P = K[R t] …(6)
If no camera pose of a comparison image is present within the field of view of the estimated camera pose of the query image, a comparison image may be selected using any of the methods (5-1), (5-2), or (5-3).

As described above, according to each embodiment and example of the present invention, by repeatedly calculating the estimated camera attitude of the query image and selecting the optimal comparison image, it is possible to select a comparison image suitable for image matching, and therefore it is possible to select the optimal comparison image from multiple candidates prepared in advance even in a location where a GPS or a geomagnetic sensor cannot be used. The repeated processing is as described above, but in one embodiment, for example, the following result may be obtained.
●At N=1 (first time), a comparison image set PS[1] is selected from the reference image set PS, an estimated camera pose EX[1] is estimated from the set PS[1], and a comparison image set PS[2] for the next time N=2 is selected from the reference image set PS. Since there is little overlap between the sets PS[1] and PS[2], it is determined that convergence has not occurred.
●At N=2, estimate the estimated camera pose EX[2] from the set PS[2], select a comparison image set PS[3] for the next N=3 from the reference image set PS, and determine that convergence has occurred since there is a large overlap between the sets PS[2] and PS[3], and output PS[3] as the result. (Note that it is also possible to output PS[N] (PS[2] in this example) instead of PS[N+1] (PS[3] in this example) as the result.)

In particular, when the reference image set PS prepared initially is dense and spans a large number of images (for example, when the intervals of the external parameters EX(i) corresponding to the position and orientation of each image P(i) are dense, such as 1 cm intervals and 1° intervals for all three axis rotations, resulting in a large number of images constituting the image set PS), it is expected that a more appropriate comparison image set PS[N] will be selected, as the comparison image set PS[N] selected changes each time N=1, 2, 3, ... from PS[1] → PS[2] → PS[3] → ... through the repetition of each embodiment or example of the present invention, and ultimately convergence to an appropriate set PS[N] will occur.

Below, we explain various supplementary, alternative, and additional examples.

<1> The comparative image search device 10 of this embodiment can contribute to improving the accuracy of self-location estimation of a robot as an elemental technology of VPS, etc., even in situations where GPS, etc. are not used. This improves the convenience of remote robots, does not require leaders to travel to remote sites, and reduces carbon dioxide emissions by saving energy resources required for user movement, making it possible to contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs) "Take urgent action to combat climate change and its impacts."

(14) FIG. 13 is a diagram showing an example of the hardware configuration of a general computer device 70. The comparison image search device 10 can be realized as one or more computer devices 70 having such a configuration. When the comparison image search device 10 is realized by two or more computer devices 70, information required for processing may be sent and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main memory device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary memory device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, and a bus BS for transmitting and receiving data between them.

Each functional unit of the comparison image search device 10 can be realized by a CPU 71 and/or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, the display 76 also operates in conjunction, and when communication-related processing related to data transmission and reception is performed, the communication interface 75 also operates in conjunction. Processing results by the comparison image search device 10 may be displayed and output on the display 76.

10...Comparison image search device, 11...Comparison image rendering unit, 12...Initial comparison image selection unit, 13...Image matching unit, 14...Estimated camera posture calculation unit, 15...Optimal comparison image selection unit, 16...Convergence judgment unit

Claims (9)

by performing image matching between a query image and a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a 3D map at one or more camera positions and/or images obtained by photographing in advance at one or more camera positions and/or in the real world that is the subject of the 3D map, estimating the camera position and orientation of the query image in the 3D map for each round repeatedly;
In each iteration, image matching is performed between the current set of comparison images selected from the set of reference images and the query image based on the camera position and orientation estimated as the previous one, thereby estimating the current camera position and orientation of the query image in the 3D map until a convergence determination is obtained;
a comparison image search device that outputs a set of comparison images used in the round in which the convergence judgment was obtained. The comparison image search device according to claim 1, characterized in that the convergence judgment is obtained when it is judged that there is little change between the current comparison image set and the previous comparison image set. The comparative image search device according to claim 1, characterized in that the convergence judgment is obtained when it is judged that there is little change between the position and/or orientation in the camera position and orientation estimated for the current time and the position and/or orientation in the camera position and orientation estimated for the previous time. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the reference image set those images that are determined to be close to the camera position and orientation estimated for the previous image. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the reference image set those images whose pose is determined to be close to the pose of the camera position and pose estimated as the previous one. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the reference image set those whose position and orientation are determined to be close to the camera position and orientation estimated as the previous image. The comparison image search device according to claim 1, characterized in that the current set of comparison images is determined by selecting from the reference image set those images that are determined to be located within the camera field of view at the camera position and orientation estimated as the previous one. by performing image matching between a query image and a comparison image set for each round selected from a reference image set including images obtained by rendering in advance from a 3D map at one or more camera positions and/or images obtained by photographing in advance at one or more camera positions and/or in the real world that is the subject of the 3D map, estimating the camera position and orientation of the query image in the 3D map for each round repeatedly;
In each iteration, image matching is performed between the current set of comparison images selected from the set of reference images and the query image based on the camera position and orientation estimated as the previous one, thereby estimating the current camera position and orientation of the query image in the 3D map until a convergence determination is obtained;
A comparison image retrieval method comprising: outputting a set of comparison images used in the round in which the convergence judgment was obtained. A program that causes a computer to function as a comparative image search device according to any one of claims 1 to 7.

Images