JP7290599B2 - Point cloud processing system, method and program - Google Patents

Description

The present invention relates to a point cloud processing apparatus, method and program capable of effectively coping with scale drift.

SLAM (Simultaneous Localization and Mapping, simultaneous execution of self-position estimation and map creation) and SfM (Structure from Motion) can be used to construct 3D point clouds, etc., and there are various conventional methods as related technologies.

Non-Patent Document 1 discloses SLAM using an omnidirectional image, and it is said that the more an image with a wider angle of view is used, the more stable the result of SLAM is. In Non-Patent Document 2, a point cloud map is generated by linking information from a plurality of cameras. In Non-Patent Document 3, a landmark portion is divided into a plurality of parts in an omnidirectional image, converted into central projections, and compared.

In Patent Literature 1 (“Camera Posture Estimation Apparatus/Program”), a degree of reliability is set for position estimation of an image, and the pose of the camera is estimated so that the error is minimized.

JP 2011-118724 A

"OpenVSLAM: A Versatile Visual SLAM Framework", ACM Multimedia 2019 Open Source Software Competition, https://arxiv.org/abs/1910.01122, "Collaborative monocular SLAM with multiple Micro Aerial Vehicles", 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems "City-scale landmark identification on mobile devices", CVPR 2011

In point cloud construction by SLAM and SfM, the problem is that scale drift occurs and accuracy decreases, but the above conventional techniques could not effectively deal with this scale drift. Scale drift is a phenomenon in which the scale gradually becomes smaller or larger when performing SLAM or SfM, and is an unavoidable phenomenon that tends to occur due to accumulation of computational errors. Scale drift is likely to occur when the field of view of the image is narrow and the feature points are insufficient, and the area used for calculation is biased.

In Non-Patent Document 1, the results of SLAM are stabilized by using an image with a wide angle of view. It causes scale drift. Also, if a wide angle of view is used directly, the distortion of the image feature amount becomes large, and the matching performance with other images deteriorates. In addition, in the case of a wide angle of view, similar images are only partially similar to normal photographs, so similar image search performance is also reduced.

Although there is a method of linking information from a plurality of cameras to generate a point cloud map, as in Non-Patent Document 2, increasing the number of cameras and increasing the amount of images increases the amount of processing. Here, even if a plurality of cameras are fixed and measured, the positions of the cameras are scattered, and the generation of the map is not stable. Moreover, it is impossible to completely fix the position of the camera, and calibration is difficult because it shifts over time.

In Non-Patent Document 3, a specific location is cut out from the omnidirectional image for comparison, but it is used only to narrow down the location of the query image, and is not intended for position calculation in point cloud construction. , and Non-Patent Document 3 also cannot deal with scale drift.

Even in the method of Patent Document 1, scale drift occurs when the distribution of the feature quantity is biased in one image, and the position estimation accuracy decreases.

SUMMARY OF THE INVENTION In view of the above problems of the prior art, it is an object of the present invention to provide a point cloud processing apparatus, method, and program capable of effectively coping with scale drift.

In order to achieve the above object, the present invention is a point cloud processing system that obtains divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges. A dividing unit extracts feature points from each frame image, performs matching of the feature points between different frame images, and extracts the position and rotation of each of the split images obtained by dividing the frame images whose positions and rotations have not been estimated. estimating the position and rotation of each of the divided images using the matching result between the estimated frame image and imposing a condition that each of the divided images share a position and a part. Also, the present invention is characterized by being a method and a program corresponding to the point cloud processing system.

According to the present invention, by using divided images obtained by dividing an image into a plurality of visual field ranges and estimating the position and rotation of each divided image under the condition that each divided image shares a position, a single It is possible to effectively suppress the influence of scale drift compared to the case of estimating the position and rotation of the entire image.

1 is a configuration diagram of a point cloud processing system according to an embodiment; FIG. 1 is a functional block diagram of a point cloud processing system according to one embodiment; FIG. 4 is a flowchart of the operation of a point cloud processing system according to one embodiment; FIG. 10 is a diagram showing an example of division of a visual field range applied by a division unit when a frame image is configured as an omnidirectional image; FIG. 10 is a diagram showing an example of division of a visual field range applied by a division unit when a frame image is configured as an omnidirectional image; Generating a reference image _I0 and a related image _Ii as normal images without distortion by a perspective projection model (pinhole camera) in the visual field range divided by the dividing unit, and obtaining the corresponding initial rotation value _Ri . It is a figure for demonstrating a process. It is a figure which shows typically the effect of this embodiment with a comparison. It is a figure which shows the example of the hardware constitutions in a common computer apparatus.

FIG. 1 is a configuration diagram of a point cloud processing system according to one embodiment. A point cloud processing system 10 includes a terminal 1 and a server 2 that can communicate with each other via a network NW such as the Internet. The terminal 1 is a mobile device that can be configured as a mobile device such as a smartphone or an in-vehicle device. It is obtained from In one embodiment, the point cloud processing system 10 includes only the terminal 1 and does not include the server 2, so that the terminal 1 alone may process the point cloud. In one embodiment, the point cloud processing system 10 includes a terminal 1 and a server 2, and the terminal 1 and the server 2 may share the processing and perform processing related to the point cloud.

FIG. 2 is a functional block diagram of the point cloud processing system 10 according to one embodiment. The point cloud processing system 10 includes an image acquisition unit 11, a division unit 12, and an estimation unit 13. The estimation unit 13 further includes a matching unit 31, a position rotation estimation unit 32, an outlier exclusion unit 33, and an optimal estimation unit . The point cloud processing system 10 processes the point cloud of the surrounding environment of the terminal 1, and the image acquisition unit 11 acquires the image of the surrounding environment. 11 is installed as a camera (a camera whose position and orientation is to be estimated). All or part of the dividing unit 12 and the estimating unit 13 may be provided in either the terminal 1 or the server 2 .

FIG. 3 is a flowchart of the operation of the point cloud processing system 10 according to one embodiment. The operation of the flowchart in FIG . Compare with one or more past time images F(tk) (k≧1) (match point clouds extracted from the past time image whose position and orientation have been estimated and the current time image whose position and orientation have not been estimated) As a result, the position and orientation of the terminal 1 at the current time t (position P _i =P _i (t) and rotation Q _i =Q _i (t) to be described later) are output as estimation results.

Hereinafter, the processing contents of each functional block in FIG. 2 will be described while describing each step in FIG.

In step S1, the image acquisition unit 11 configured as a camera installed in the terminal 1 captures a frame of the environment in which the terminal 1 exists (fields around the terminal 1, etc.) at the current time t. After acquiring the image F(t) and outputting it to the dividing unit 12, the process proceeds to step S2.

It is preferable that the camera of the image acquisition unit 11 is configured to be capable of wide-angle shooting by including a fisheye lens or the like. The captured frame image F(t) may be, for example, an omnidirectional image (an image obtained by photographing a global area) or an omnidirectional image (an image obtained by photographing a hemispherical area).

In step S2, the dividing unit 12 divides the frame image F(t) obtained in step S1 into a plurality of visual field ranges to obtain one reference image _I0 and one or more related images _Ii (1≤i≤ N-1, N≧2) are obtained, and an initial rotation value R _i (1≦i≦N-1) corresponding to each related image I _i is obtained, and the obtained reference image I ₀ and the related image After outputting I _i and the corresponding initial rotation value R _i to the estimator 13, proceed to step S3.

The frame image F(t) is a projected image such as perspective projection, circular fisheye, equirectangular projection, etc. In order to cover a wide range, it is composed of one or more images (fisheye image, etc.). It is The dividing unit 12 divides the visual field range in the frame image F(t) configured as a fisheye image or the like into a plurality of predetermined visual field ranges (which may partially overlap), and applies a predetermined projection method. By doing so, the reference image I ₀ and the related image I _i to generate

4 and 5 are diagrams showing examples of division of the visual field range applied by the division unit 12 when the frame image is configured as an omnidirectional image and an omnidirectional image, respectively. In both FIGS. 4 and 5, it is assumed that the depth direction on the paper surface is the vertical downward direction in the three-dimensional space, and the frontward direction on the paper surface is the vertical upward direction. Figure 4 divides the 180° horizontal front range captured by the fisheye lens FL on the front side into three viewing ranges f1, f2, and f3 that allow mutual overlap, and divides the 180° horizontal rear range captured by the fisheye lens BL on the rear side. , By dividing into three fields of view f4, f5, and f6 that allow mutual overlap, two circular fisheye images (front side and back side) representing the omnisphere can be overlapped. 6 shows an example of dividing into six visual field ranges f1 to f6. FIG. 5 illustrates a 360° horizontal field of view in a full-dome circular fisheye image configured to capture a horizontal hemispherical area (e.g., a hemispherical area above the ground). It shows an example of dividing into six viewing ranges f11, f12, f13, f14, f15, and f16. In FIG. 4, the two fish-eye lenses FL and BL face horizontally forward and backward, whereas in FIG. 5 one fish-eye lens faces vertically upward.

It should be noted that in step S2, the division unit 12 may divide the frame image F(t) into a plurality of visual field ranges by setting in advance a predetermined division method that does not depend on the time t.

FIG. 6 shows a normal perspective projection model (pinhole camera) with an appropriate angle of view (for example, about 90 degrees diagonally) with no distortion (or with little distortion) in the visual field range divided by the dividing unit 12. FIG. 10 is a diagram for explaining the process of generating a reference image I ₀ and a related image I _i as images and acquiring the corresponding initial rotation value R _i ;

In FIG. 6, the example EX1 shows the viewing ranges S0 _and S _i corresponding to the reference image _I0 and the related image _Ii on the spherical surface SP centered on the camera viewpoint C. FIG. Since the visual field ranges S ₀ and S _i can be defined with rotations R ₀ and R _i (which can be given as rotation matrices, respectively) indicating the positions of the ranges on the spherical surface SP, these are the initial rotation values. do it. In particular, the field of view of one or more associated images I _i as a relative rotation with respect to the rotation R ₀ of the field range S ₀ of the reference image I ₀ (that is, assuming that the rotation R ₀ is no rotation) A rotation R _i corresponding to the range Si may be set as an initial value. In example EX1, the arrow representing the direction of rotation schematically shows that the rotation R _i is determined based on the rotation R ₀ . (Therefore, the rotation initial value R _i is also determined as a fixed value according to a predetermined division method in which the division unit 12 divides the frame image F(t) into a plurality of visual field ranges.)

As shown schematically in examples EX1 and EX2 in FIG. 6, a normal image with an appropriate angle of view (for example, about 90 degrees diagonal) with no (or little) distortion due to a perspective projection model (pinhole camera) To generate the reference image I ₀ and the associated image I _i as By projecting the fisheye image F1 shown in example EX1 onto the spherical surface SP ( _F2 is the projection of F1 ₎ , the spherical image S ₀ and S _i (using the same reference numerals for field of view and spherical image for convenience of explanation) can be obtained. Then, as shown in example EX2, by projecting the spherical images S 0 and S i onto planes I ₀ and _{I i} of central projection centering on the camera viewpoint C, the reference images I ₀ and S _i as normal images are obtained. An associated image I _i (for convenience of explanation, the same reference numerals are used for the image plane for projection and the resulting projected image) can be obtained.

There are various projection methods such as a circular fisheye, and an example EX3 shows an example of a 180-degree circular fisheye as a method different from the method shown in the example EX1 of FIG. The dividing unit 12 can generate the reference image _I0 and the related image _Ii and acquire the initial rotation value _Ri by a technique corresponding to the individual projection method applied to the frame image F(t). can. If Ri has already been optimized and calculated in step S6, that value may be used as the initial rotation value _Ri . (For example, at the initial time t=1, the optimization calculation in step S6 has not been performed even once, so a predetermined rotation initial value R _i (t=1) corresponding to the projection method is used. At time t≧2, for example, the value R _i (t-1) optimized in step S6 at time t-1 immediately before may be used as the initial rotation value at time t.)

Returning to the description of each step in FIG. 3, in step S3, an image at current time t whose position and orientation have not been estimated (the reference image I ₀ and the related image I _i obtained in step S2 from the frame image F(t)) and an image at past time tk (k≧1) whose position and orientation have been estimated (reference image I ₀ and related image I _i obtained in step S2 for past time tk from frame image F(tk)) After the matching unit 31 performs feature point matching, the process proceeds to step S4.

As described above, the feature point matching by the matching unit 31 is performed between the current time t and the past time tk with respect to the reference image _I0 and the related image _Ii with no (or little) distortion, and the image at the current time t Matching results in one or more past images for are stored. The matching result is referred to and used in the processing of the position/rotation estimating unit 32 in step S4, the processing of the outlier excluding unit 33 in step S5, and the processing of the optimum estimating unit 34 in step S6, which will be described later.

The matching unit 31 may extract feature points and feature amounts (SIFT features, etc.) from the reference image _I0 and the related image _Ii , and then perform matching using the feature points and feature amounts. For example, for each feature point at the current time t, matching between all feature points at one past time tk is obtained as a matching result with the maximum similarity of the feature amount. good. If the maximum degree of similarity is determined to be small by threshold determination, the matching result may be determined as no match.

In step S4, the position/rotation estimator 32 determines the position P _i and the rotation Q _i of the reference image I ₀ and the related image I _i at the current time t (0≤i≤N-1, and the reference image if i=0). After estimating the position and rotation of the image _I0 , and the position and rotation of the associated image _Ii if i≧1, go to step S5. (Note that the estimated rotation Q _i is different from the initial rotation value R _i obtained for each related image I _i in step S2, as is clear from the assignment of another variable notation. .)

The position/rotation estimating unit 32 uses the matching result obtained in step S3 (that is, the matching result between the point group formed by the image at the current time t and the point group formed by the image at the past time tk) to estimate the epipolar By using any existing method such as geometry or bundle adjustment, the position P _i and rotation Q _i of each image can be estimated in the same way as in general SfM. (As will be described below, with respect to images at past time tk, the positions and orientations (positions and rotations) of reference image _I0 and related image _Ii have already been estimated in step S6 at past time tk (excluded in step S5). Therefore, in step S4, the positions P _i and rotations Q _i can be estimated.)

In step S5, the outlier removing unit 33 extracts the matching result obtained in step S3, the position P _i and the rotation Q _i for the reference image I ₀ and the related image I _i at the current time t obtained in step S4. , is used to determine an image corresponding to an outlier among the reference image I ₀ and the related image I _i at the current time t, and the determined image is processed by the optimum estimation unit 34 in the next step S6. Exclude it from the object and proceed to step S6. (If no exclusion determination is made in step S5, no image is excluded.)

The outlier exclusion unit 33 determines that an image corresponding to the following (1) or (2) should be excluded and excludes it.

(1) For each image I _i (0≦i≦N−1, reference image I ₀ if i=0, related image I _i if i≧1), the average position P _avg of the entire N images to calculate The average position P _avg may be calculated as, for example, the arithmetic mean ΣP _i /N or the median value of each position P _i =(x _i, y _i, z _i ). An image with a large divergence from the average position P _avg is determined by threshold determination with respect to the absolute value of the difference |P _i −P _avg |, and an image determined to have a large deviation is excluded.

(2) In the matching result obtained in step S3, an image that is determined to have a small number of matched feature points by threshold determination is excluded.

Hereinafter, for the sake of explanation, let the index set representing the remaining images not excluded in step S5 be U (U ⊂ {0, 1, 2, ..., N-1}), and the remaining images not excluded be I It is written as _j (j∈U). The exclusion in step S5 can improve the accuracy of the final position and rotation estimates obtained in the next step S6.

In step S6, the optimum estimating unit 34 estimates the position and rotation of each image I _j (j∈U) remaining without being excluded in step S5 as an optimized image by the following procedures 1 to 4. Then, it is regarded as an output from the point cloud processing system 10, and the process returns to step S1. In the returned step S1, the same processing is continuously executed for the next current time t+1.

₍ Procedure 1) Rotation inverse matrix _{invR j (} = _R Calculate the average (arithmetic mean) within the set U of ( _{invR j} )·Q _j (j∈U) multiplied by j ⁻¹ ). Let this average value be Q _avg . Note that (invR ₀ )·Q ₀ =Q ₀ because R ₀ is non-rotated with respect to the reference image I ₀ (j=0).

(Step 2) Let the average position P _avg (the average of the positions based on the remaining images not excluded by step S5) and the average rotation Q _avg (obtained in step 1) be the position and rotation of the reference image I ₀ (set as the initial value for subsequent bundle adjustments). (Note that even if the reference image I ₀ is excluded as an outlier in step S5, the optimization calculation in step S6 is performed and this setting is performed. This setting applies only to the reference image I ₀ itself. This is because it is the premise of the setting for images other than the reference image _I0 , not for the setting for

(Procedure 3) Solve a nonlinear minimization problem by bundle adjustment with variables P _avg , Q _avg , and R _j (j∈U, j≧1). A general method such as Ceres, a library, or the like may be used to solve the problem. At this time, the position of the related image is calculated as P _j =P _avg and the rotation Q _j =R _j ·Q _avg . The variables at this time are P _{avg ,} R _{j ,} Q _avg (jεU, j≧1), and the bundle adjustment also adjusts the relative rotation R _j (j≧1 ) with respect to the reference image I ₀ . Note that R ₀ is fixed without rotation and is excluded from the optimization calculation.

(Procedure ₄ ) As a result of solving in the above procedure 3, optimized _{P avg} , Q _avg , R _j (j∈U, j≧1) are calculated. The final estimation result is obtained with the rotation Q _j =R _j ·Q _avg (0≦k≦N−1). Here, as described above, since R ₀ is fixed at no rotation and is not subject to optimization calculation, rotation Q ₀ =Q _avg .

As described above, in step S6, the optimum estimating unit 34 determines the three feature points matched in step S3 in each image I _j (j∈U) at the current time t for each image I _j (j∈U). Using the position of the dimensional world coordinates and the camera position and orientation (position and rotation of the image) of each image I _j (j∈U) as unknown parameters, the 3D world coordinates of the feature points are calculated by bundle adjustment, which is an existing method. These unknown parameters are estimated to minimize the error when reprojecting to each image plane coordinate. At this time, in the bundle adjustment of the present embodiment, the camera position and orientation of each image I _j (j∈U) are not changed freely, but are changed under the constraints described in procedure 3. A final estimation result can be obtained as in procedure 4 by calculating the one that minimizes the reprojection error. Moreover, in the bundle adjustment of this embodiment, by setting the initial values of the variables in the procedure 3 in the procedures 1 and 2, the calculation can be stably performed.

Note _{that the} bundle adjustment by the optimal estimation unit 34 in step S6 is performed on one or more Each image I _j =I _j (tk) (j∈U=U(tk)) at the past time tk (k≧1) of is also added before performing. (In this case, each image I _j (j ∈ U) at one or more past times tk (k ≥ 1) is also optimized by imposing the same constraint (procedure 3), so that one or more past The camera position and orientation of each image I _j (j∈U) at time tk (k≧1) are also optimized again to obtain updated values.)

In step S6, bundle adjustment is performed on all images (all images up to the current time t) whose positions and rotations are determined periodically (for example, at predetermined frame intervals). The position and rotation may be further optimized.

With the above-described flow of FIG. 3, scale drift can be effectively suppressed, and the position and rotation can be estimated by the optimum estimator 34 . In particular, the optimum estimation unit 34 uses the reference image I ₀ and the related image I _i obtained by dividing the wide-angle image into a plurality of viewing ranges (especially, the position (the position finally estimated as P _avg ) is shared (has the same position) in the reference image I ₀ and the related image I _i , and only the rotations are different from each other). Scale drift can be effectively suppressed by using feature points without bias. Although the case of performing real-time positioning has been described as an example in FIG. 3, this embodiment can also be applied to the case of creating a map based on this positioning result. (In the case of map generation, after estimating the position of the image, the operation of saving the surrounding point cloud as a map may be added.) Positioning may be performed off-line instead of in real time. (The flow in FIG. 3 may be changed to an arbitrary order instead of the order according to each time t of the video.)

FIG. 7 is a diagram schematically showing the effect of this embodiment together with the comparison. As a contrast, in example EX11, objects that should be at the same position are grouped together at different angles of view, and are erroneously estimated as different positions. Example EX12 is an application example of the present embodiment, and it can be seen that image positions are calculated in a fan shape from one location. Moreover, the effect of shortening the calculation time can be obtained. In FIG. 6, the fan-like shape represents the estimated position and rotation of the cone (the degree of opening of the quadrangle on the bottom of the cone corresponds to the angle of view). There are 6 common trajectories in example EX11 and example EX12, but in example EX11 there are 3 trajectories with different positions, 1 that is scattered and cannot be seen as a line, and the position was unknown There are 2 lines and 6 lines, and in example EX12, these 6 lines are neatly overlapped in a fan shape and can be seen as one line. (The reason why two tracks are visible in the example EX12 is because one trajectory makes one round.) In the example EX12, the influence of scale drift is reduced by stable position calculation. , In example EX11, due to the influence of scale drift, the position calculations, which should be at regular intervals, gradually narrow and widen instead of at regular intervals.

FIG. 8 is a diagram showing an example of hardware configuration in a general computer device 70. As shown in FIG. Each of the terminal 1 and the server 2 in the point cloud processing system 10 can be implemented as one or more computer devices 70 having such a configuration. When the point cloud processing system 10 is realized by two or more computer devices 70, information necessary for processing may be transmitted and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, and a GPU (graphics processing unit) as a dedicated processor that executes part or all of the execution instructions of the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM 73 as a main storage device that provides a work area to CPU 71 (and GPU 72), ROM 74 as an auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input by mouse, keyboard, touch panel, etc. It has a camera 78 that constitutes the video acquisition unit 11 and a bus BS for exchanging data therebetween.

Each functional unit of the terminal 1 and the server 2 in the point cloud processing system 10 can be realized by the CPU 71 and/or the GPU 72 that reads and executes a predetermined program corresponding to the function of each unit from the ROM 74 . Both the CPU 71 and the GPU 72 are a kind of arithmetic unit (processor). Here, when display-related processing is performed, the display 76 further operates in conjunction, and when communication-related processing relating to data transmission/reception is performed, the communication interface 75 further operates in conjunction. The estimation results and the like by the point cloud processing system 10 may be displayed on the display 76 and output.

10…Point cloud processing system, 1…Terminal, 2…Server
11... video acquisition unit, 12... division unit, 13... estimation unit, 31... matching unit, 32... position/rotation estimation unit, 33... outlier exclusion unit, 34... optimum estimation unit

Claims (11)

a dividing unit that obtains divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. an estimating unit that estimates the position and rotation of each of the split images under the condition that
The plurality of visual field ranges are set to correspond to a reference image and one or more related images set by rotating the reference image by a predetermined amount,
The dividing unit obtains divided images as a reference image and a related image,
The estimating unit optimizes the position shared by the reference image and the related image and the rotation of the related image with respect to the reference image using the matching result, and calculates the position of the reference image and the related image. and estimate the rotation,
The estimating unit optimizes at least the three-dimensional world coordinates of the matched feature points of the reference image and the related images, and the positions and rotations of the reference image and the related images, by bundle adjustment. estimating the position and rotation of the image and related images;
In the estimation unit, when optimizing by the bundle adjustment,
adopting the average (P _avg ) of the position of the reference image and the related image estimated by the matching result between the frame image whose position and rotation have been estimated as the initial value of the position of the reference image;
As an initial value of the rotation of the reference image, the rotation (Q ₀ ) of the reference image estimated by the matching result between the frame image whose position and rotation have been estimated, and the estimated related image A point cloud processing system characterized by adopting a rotation (Q _j ) converted to a rotation in a reference image ((invR _j )·Q _j ) and an average (Q _avg ) of the rotation (Q j ). a dividing unit that obtains divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. an estimating unit that estimates the position and rotation of each of the split images under the condition that
The plurality of visual field ranges are set to correspond to a reference image and one or more related images set by rotating the reference image by a predetermined amount,
The dividing unit obtains divided images as a reference image and a related image,
The estimating unit optimizes the position shared by the reference image and the related image and the rotation of the related image with respect to the reference image using the matching result, and calculates the position of the reference image and the related image. and estimate the rotation,
The estimating unit optimizes at least the three-dimensional world coordinates of the matched feature points of the reference image and the related images, and the positions and rotations of the reference image and the related images, by bundle adjustment. estimating the position and rotation of the image and related images;
In the estimation unit, when optimizing by the bundle adjustment,
A common position (P _avg ) in a reference image and a related image, a rotation (Q _avg ) of the reference image, and a relative rotation (R _j ) with respect to the rotation (Q _avg ) of the reference image and the rotation of the related image (Q A point cloud processing system, characterized in that a relative rotation (R _j ) specifying _j =R _j ·Q _avg ) and a relative rotation (R j ) are variables . a dividing unit that obtains divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. an estimating unit that estimates the position and rotation of each of the split images under the condition that
The estimating unit calculates the position of each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated with the frame image whose position and rotation have been estimated. A point cloud processing system characterized by estimating as an initial value, and excluding those divided images for which the position estimated as the initial value corresponds to an outlier from targets for position and rotation estimation. a dividing unit that obtains divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. an estimating unit that estimates the position and rotation of each of the split images under the condition that
In the estimating unit, for each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, and the frame image whose position and rotation have been estimated, the divided image is obtained by using the matching result. 1. A point group processing system characterized by excluding those determined to have a small number of matched feature points from targets for position and rotation estimation. 5. The point cloud processing system according to claim 1, wherein each frame image is configured as a wide-angle image. Each frame image is configured as a fisheye image,
The dividing unit is characterized in that, by applying a central projection method from a camera viewpoint to a frame image configured as the fisheye image, divided images divided into a plurality of visual field ranges are generated as perspective projection model images. The point cloud processing system according to any one of claims 1 to 5 . A dividing step of obtaining divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. estimating the position and rotation of each of the sub-images subject to the condition that they share
The plurality of visual field ranges are set to correspond to a reference image and one or more related images set by rotating the reference image by a predetermined amount,
obtaining divided images as a reference image and a related image in the dividing step;
In the estimating step, the positions of the reference image and the related images are obtained by optimizing the position shared by the reference image and the related images and the rotation of the related images with respect to the reference image using the matching result. and estimate the rotation,
In the estimating step, at least the three-dimensional world coordinates of the matched feature points of the reference image and the related images, and the positions and rotations of the reference image and the related images are optimized by bundle adjustment to obtain the reference estimating the position and rotation of the image and related images;
In the estimating step, when optimizing by the bundle adjustment,
adopting the average (P _avg ) of the position of the reference image and the related image estimated by the matching result between the frame image whose position and rotation have been estimated as the initial value of the position of the reference image;
As an initial value of the rotation of the reference image, the rotation (Q ₀ ) of the reference image estimated by the matching result between the frame image whose position and rotation have been estimated, and the estimated related image A computer-implemented point cloud processing method characterized in that it employs a transformation of rotations (Q _j ) to rotations in a reference image ((invR _j )·Q _j ) and an average (Q _avg ) of the rotations (Q j ). A dividing step of obtaining divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. estimating the position and rotation of each of the sub-images subject to the condition that they share
The plurality of visual field ranges are set to correspond to a reference image and one or more related images set by rotating the reference image by a predetermined amount,
obtaining divided images as a reference image and a related image in the dividing step;
In the estimating step, the positions of the reference image and the related images are obtained by optimizing the position shared by the reference image and the related images and the rotation of the related images with respect to the reference image using the matching result. and estimate the rotation,
In the estimating step, at least the three-dimensional world coordinates of the matched feature points of the reference image and the related images, and the positions and rotations of the reference image and the related images are optimized by bundle adjustment to obtain the reference estimating the position and rotation of the image and related images;
In the estimating step, when optimizing by the bundle adjustment,
A common position (P _avg ) in a reference image and a related image, a rotation (Q _avg ) of the reference image, and a relative rotation (R _j ) with respect to the rotation (Q _avg ) of the reference image and the rotation of the related image (Q a relative rotation (R _j ) specifying _j =R _j ·Q _avg ), and a computer-implemented point cloud processing method as a variable. A dividing step of obtaining divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. estimating the position and rotation of each of the sub-images subject to the condition that they share
In the estimating step, for each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, the position and the position of the frame image whose position and rotation have been estimated are used to determine the position. estimating as an initial value, and excluding those divided images for which the position estimated as the initial value corresponds to an outlier from the position and rotation estimation target Point cloud processing method. A dividing step of obtaining divided images obtained by dividing each frame image captured while moving in the environment and changing the position and rotation into a plurality of visual field ranges;
extracting feature points from each frame image, performing feature point matching between different frame images,
For each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, using the result of matching between the frame image whose position and rotation have been estimated, each of the divided images is divided. estimating the position and rotation of each of the sub-images subject to the condition that they share
In the estimating step, for each of the divided images obtained by dividing the frame image whose position and rotation have not been estimated, and the frame image whose position and rotation have been estimated, the divided image is obtained by using the result of matching. 1. A point group processing method executed by a computer, characterized in that, out of these, those determined to have a small number of matching feature points are excluded from targets for position and rotation estimation. A program that causes a computer to function as the point cloud processing system according to any one of claims 1 to 6 .

Images