JP7343446B2 - Model generation device, method and program - Google Patents

Description

The present invention relates to a model generation device, method, and program for generating a three-dimensional model from a point group.

Point clouds (three-dimensional point clouds) are extracted from images and the like, and are used for various purposes, such as creating three-dimensional models of objects existing in the real world and maps of the real world. High-precision modeling or wide-area modeling usually requires the use of point clouds consisting of a large number of points, so when working with point clouds or 3D models related to point clouds, it is There is a need for processing, and conventional techniques include, for example, the techniques disclosed in Patent Documents 1 to 3.

In Patent Document 1, when a three-dimensional model is generated using the visual volume intersection method, an accurate A method has been proposed in which conversion processing is performed only on some representative points among a large number of voxel points, and conversion on other voxel points is performed approximately by complementing the conversion results of the representative points.

Patent Document 2 proposes a method for converting a point group composed of points on an object surface into a voxel model, and then using an octree method to efficiently generate a three-dimensional model that reflects the topological structure. . Patent Document 3 proposes a method that applies a voxel model in order to quickly extract the shape of an empty area (an empty area for arranging other parts, etc.) within a product in three-dimensional shape data of the product shape. has been done.

Japanese Patent Application Publication No. 2018-133059 Japanese Patent Application Publication No. 2000-222600 Japanese Patent Application Publication No. 2018-005859

However, in the conventional techniques described above, it has been proposed to efficiently perform various high-load calculations on the premise that the point clouds and three-dimensional models to be processed are accurate. . For example, in Patent Document 1, if there is noise in the silhouette image when applying the visual volume intersection method, an inaccurate voxel model may be obtained, but no consideration was given to dealing with such a case. Patent Documents 2 and 3 also assume that the point cloud and three-dimensional shape data to be processed are accurate, and do not take into account the case where the point cloud, etc. to be processed is inaccurate. There wasn't.

In other words, in the conventional technology, the influence of noise, etc. is reduced by taking into consideration the case where the point cloud to be processed, etc., erroneously includes incorrect points that are not located in the original object, etc. due to the influence of noise, etc. No consideration was given to efficient processing.

In view of the above problems of the prior art, an object of the present invention is to provide a model generation device, method, and program capable of generating an efficient three-dimensional model even when a point group contains noise.

In order to achieve the above object, the present invention is a model generation device that constructs a point cloud of an object photographed in a plurality of images from a plurality of images, and extracts each point belonging to the point cloud. a construction unit that acquires information about the camera that captured the image as correspondence information between each point and the camera; and a construction unit that generates an initial voxel model from the point group, and generates an initial voxel model from the plurality of images from among each voxel belonging to the initial voxel model. By evaluating voxels that intersect on a line segment connecting the position of the camera that photographed the image and each point that the camera corresponds to based on the correspondence information as having a low probability of corresponding to the object being photographed. , a selection unit that generates a voxel model of the point group by selecting voxels that are determined to have a high evaluated probability from among voxels belonging to the initial voxel model. Further, the present invention is characterized in that it is a method or program compatible with the model generation device.

According to the present invention, by using the positional relationship between each point of a point cloud and the position of the camera that captured the plurality of images when constructing the point cloud, voxels are efficiently created as a three-dimensional model with reduced noise influence. A model can be generated.

FIG. 1 is a functional block diagram of a model generation device according to an embodiment. FIG. 3 is a diagram schematically showing processing in a sorting section. FIG. 3 is a diagram illustrating a flowchart of the operation of the model generation device according to an embodiment. 1 is a diagram showing an example of a hardware configuration in a general computer device.

FIG. 1 is a functional block diagram of a model generation device according to an embodiment. The model generation device 10 includes a construction section 1 and a sorting section 2, and the sorting section 2 includes an initial setting section 21, an evaluation section 22, and a post-processing section 23. As for its overall operation, the model generation device 10 operates at different camera positions for targets (objects, real-world fields such as street corners modeled as maps (3D topographic maps), etc.) for which 3D models are to be generated. A construction unit 1 reads a plurality of images taken as input, and a selection unit 2 outputs a voxel model as a three-dimensional model. The general operation of the model generation device 10 is as follows.

The construction unit 1 constructs a three-dimensional point cloud from a plurality of input images, and outputs this point cloud to the selection unit 2. The construction unit 1 also provides information on one or more images corresponding to each point constituting the point cloud (information belonging to the point cloud) as one of the information accompanying the point cloud obtained when constructing the point cloud. For each point, information indicating in which of the plurality of input images the point was photographed (observed) and extracted is also output to the selection unit 2. The construction unit 1 also outputs information on the camera position and orientation of each of the input plurality of images to the selection unit 2 as one of the information accompanying the point cloud obtained when constructing the point cloud. do.

For each point in the point cloud obtained from the construction section 1, the sorting section 2 extracts information accompanying the point cloud obtained from the construction section 1 (information on the corresponding image of each point belonging to the point cloud and information on the point cloud extracted). By using information such as information on the position and orientation of the camera in each image, points estimated to be caused by noise are excluded, and the remaining points are output as a voxel model.

In the sorting unit 2, in order to select and output the voxel model, the initial setting unit 21 sets the voxel space, uses the point cloud obtained from the construction unit 1 as an (initial) voxel model, and selects the point group that belongs to the initial voxel model. Using the information accompanying the point cloud obtained from the construction unit 1 for each voxel, the evaluation unit 22 evaluates whether or not it is a voxel caused by noise, and for voxels that are determined to be not caused by noise by the evaluation. By further performing post-processing in the post-processing unit 23, a final selected voxel model can be output.

FIG. 2 is a diagram schematically showing the processing in the sorting unit 2 divided into examples EX1 to EX3. Example EX1 shows a schematic example of the initial voxel model V1 set by the initial setting unit 21, Example EX2 shows a schematic example of one of the evaluations by the evaluation unit 22, and Example EX3 shows the initial voxel model V1 set by the initial setting unit 21. A schematic example of a final voxel model V2 obtained by being selected from the initial voxel model V1 through processing by the processing unit 23 is shown. By excluding voxels that are estimated to be caused by noise, etc., a final voxel model V2 (V2⊂V1) is obtained as a subset of the initial voxel model V1. Note that the example EX2 will be referred to as a schematic example in the detailed description of the evaluation unit 22 described later.

FIG. 3 is a diagram showing a flowchart of the operation of the model generation device 10 according to one embodiment. Regarding each step in FIG. 3, steps S1 and S2 are the processing of the construction unit 1, step S3 is the processing of the initial setting unit 21, and steps S41, S42, and S43 repeated by steps S4 and S44 are the processing of the evaluation unit 22. Steps S5, S6, and S7 are the processes of the post-processing section 23. Hereinafter, details of the processing of each part of the model generation device 10 will be explained while explaining each step in FIG.

In step S1, the construction unit 1 constructs a point cloud from the plurality of input images and outputs it to the sorting unit 2, and then the process proceeds to step S2. Any existing method may be used to construct this point cloud, such as SfM (Structure from Motion). In SfM, for example, feature points (two-dimensional image coordinates) and local features are obtained from each image using SIFT features, etc., and feature point correspondence (local features are determined to match) between two mutually different images is determined. By obtaining information on the 3D world coordinates of each feature point in the image using principles such as triangulation (correspondence between feature points), the 3D world coordinates of each point belonging to the point group are calculated, and the points groups can be constructed. When constructing a point cloud using SfM, an existing method such as bundle adjustment may be used to minimize the error.

The point cloud constructed in step S1 is constructed as including inaccurate points due to the influence of noise, etc., but in this embodiment, inaccurate points are estimated by sorting through the processing of the sorting unit 2. It is possible to obtain a three-dimensional model (voxel model) in which the influence of noise is efficiently reduced.

In step S2, each information is acquired as information incidentally obtained when the construction unit 1 constructs the point cloud in step S1 (information already obtained as intermediate output during the construction of the point cloud in step S1). After outputting information on the corresponding images of points and information on the position and orientation of the camera of each image to the sorting unit 2, the process proceeds to step S2.

In other words, each point belonging to a point cloud can be differentiated between different images by (1) each feature point being extracted from one of the images, and (2) determining that the feature points of the different images match the local features. (3) By applying principles such as triangulation, it is determined that the positions of corresponding feature points match between different images, and their 3D world coordinates are determined. There is. Therefore, for each point belonging to the point cloud, information on the corresponding image of each point is already obtained when constructing the point cloud, as information on which image (two or more images) the point was extracted from. There is. For example, regarding a certain point p=(X,Y,Z) belonging to the constructed point cloud, feature points (u _A ,v _A ),(u Information on the corresponding images is obtained as information indicating that the same points are extracted as _B , v _B ) and (u _C , v _C ), respectively.

Furthermore, as is known in the field of three-dimensional computer graphics, when multiple images are input and a point cloud is constructed using principles such as triangulation, the position and orientation of the camera that took each image is also It is calculated at the same three-dimensional world coordinates as each point belonging to . By providing the camera parameters of the camera that took each image as known, a point cloud is constructed by applying principles such as triangulation, and the position and orientation of the camera that took each image is also calculated. be able to.

In step S3, the initial setting unit 21 sets an initial voxel model including noise using the point cloud constructed by the construction unit 1 in step S1, and then the process proceeds to step S4. For example, the 3D world coordinate system in which the point cloud is defined is divided into voxel grids of a predetermined size, and the point cloud is modeled in 3D as a set of voxel grids in which points belonging to the point cloud exist. An initial voxel model can be obtained.

In step S3, the initial setting unit 21 also sets the initial value of the existence probability VP(k) for each voxel V(k) (k is the ID of the voxel) belonging to the initial voxel model as shown in equation (1) below. For example, assign the maximum value of 1.
VP(k)=1 …(1)

Note that the value of the existence probability VP(k) of each voxel V(k) is a value in the range of 0 to 1, as shown in equation (2) below, and can be updated in subsequent steps S4 to S44. Become. In this embodiment, voxels with a lower existence probability VP(k) are defined as having a higher probability of being caused by noise, and are targets whose values can be updated. .
0≦VP(k)≦1 …(2)

As shown in the figure, steps S4 and S44 are performed in steps S41, S42, and S43 ( This is a control step representing repeating each processing step) by the evaluation unit 21.

In step S4, unprocessed images (unprocessed in subsequent steps S41, S42, and S43) among the N images are set as processing targets, and then the process proceeds to step S41. For the explanation of subsequent steps S41, S42, S43, and S44, let the N images input to the construction unit 1 be P(i) (i=1,2,...,N), and in the i-th step S4, Assume that image P(i) is set to be processed.

In step S41, as shown in equation (3), the coordinates of the image P(i) to be processed correspond to the voxel V(k), that is, the camera position (pos(i) )), the evaluation unit 22 reduces its existence probability VP(k) by a certain ratio r (0<r<1, for example, r=0.9). After performing the evaluation, the process proceeds to step S42. Here, the information on the camera position pos(i) can be obtained from the accompanying information when the construction unit 1 constructs the point cloud in step S2. Also, instead of reducing the existence probability VP(k) by a constant rate r, as in the expression (3') instead of the expression (3), the existence probability VP(k) can be reduced to a constant value r (0<r <1, for example r=0.1).
if (pos(i)∈V(k)) VP(k)=r*VP(k) …(3)
if (pos(i)∈V(k)) VP(k)=VP(k)-r …(3')

The significance of the process in step S41 is as follows. That is, since it is assumed that the object does not exist at the photographed position, step S41 lowers the probability of its existence. (However, this process depends on the size of the voxel, so you can use an appropriate r value. For example, if a piece of voxel is large (about 5 m), there is a high possibility that an object exists within that range. , if the voxel is small (e.g. 50 cm), there is a low possibility that the object exists unless the image was taken near a wall.When creating a 3D model, it is necessary to pass through a large space away from the object. (The method in step S41 is preferable because images are often taken using

Note that equation (3) applies the postscript assignment expression when the condition indicated by if() is true (if the value inside the parentheses () of if is true), that is, the existence probability VP( This means updating the value of k) to the value calculated as the value on the right side of the assignment formula using the current value of existence probability VP(k) (value before update), and hereafter Similar notation is used in each expression.

In step S42, as shown in equation (4), the voxel V(k) as observed from the image P(i) to be processed (as explained below, the voxel V(k) is not directly unobserved but indirectly observed through the belonging point), i.e. at least one point ( If a point p(k) is observed from the image P(i), the evaluation unit 22 performs an evaluation so that its existence probability is increased by a certain value a (a>0, for example, a=0.5). After doing so, the process proceeds to step S43. In addition, if the value of the existence probability VP(k) exceeds 1 due to the update of equation (4), then equation (5) is further applied, and the maximum value of the existence probability is set to 1, and the value from the range of equation (2) is Prevent deviations. In addition, instead of increasing the existence probability by a constant value a by adding the existence probability as shown in expression (4') instead of formula (4), the existence probability can be multiplied by a certain percentage a (a>1, for example, a=1.1 ) may be increased.
if (a certain point p(k) is observed from image P(i)) VP(k)=VP(k)+a …(4)
if (a certain point p(k) is observed from image P(i)) VP(k)=a*VP(k) …(4')
if(VP(k)>1) VP(k)=1 …(5)

Note that if two or more points p(k1), p(k2),... belonging to a certain voxel V(k) are observed from the image P(i), they are increased using equations (4) and (4'). The process may be applied repeatedly the same number of times as the number of observed points p(k1), p(k2), . . . .

Here, the information as to whether the point p(k) of the point group located within the voxel V(k) is observed from the image P(i) is obtained by the construction unit 1 in step S2. Information on the corresponding image of each point can be obtained from the accompanying information when constructing the point cloud. In other words, if point p(k) corresponds to image P(i) (point p(k) is extracted from image P(i)), point p(k) corresponds to image P(i). This corresponds to what is observed from image P(i) as being photographed at P(i).

As step S42 is executed for each image P(i) by repeating control steps S4 and S44, actual objects correspond to voxels V(k) that are observed from a larger number of images P(i). If there is a high probability that it exists, its existence probability VP(k) will be determined as a larger value. Furthermore, when adopting an embodiment in which equations (4) and (4') are applied repeatedly the same number of times as the number of observed points p(k1), p(k2),... described above, the number of observed points p(k1), p(k2),... For the voxel V(k) to which the observed points p(k1), p(k2),... belong, the existence probability V(k) is determined as a larger value.

In step S43, as shown in equation (6), the camera position pos(i) of image P(i) and the point q(i )) and the voxel V(k) located on the line segment L(pos(i),q(i)), that is, the line segment L(pos(i),q(i )) intersects V(k) (excluding the voxel to which the observation point q(i) as the end point of the line segment L and the camera position pos(i) belong), its existence probability VP(k) is constant. After the evaluation unit 22 performs evaluation so as to decrease by the ratio w (0<w<1, for example, w=0.5), the process proceeds to step S44. In addition, instead of reducing the existence probability VP(k) by a constant rate w by multiplying the existence probability VP(k) as in the expression (6') instead of the expression (6), the existence probability VP(k) can be reduced to a constant value w(0 <w<1, for example w=0.1).
if(line segment L(pos(i),q(i)) intersects voxel V(k)) VP(k)=w*VP(k) …(6)
if(line segment L(pos(i),q(i)) intersects voxel V(k)) VP(k)=VP(k)-w …(6')

In addition, ``the line segment L(pos(i),q(i)) intersects the voxel V(k)'' means ``the observed camera position pos(i) of the image P(i)''. The voxel V(k) is located between the point q(i) and the point q(i).

Here, the information of the point q(i) observed from the image P(i) and the information of the camera position pos(i) of the image P(i) are as explained in steps S41 and S42. It can be acquired from the accompanying information when constructing the point cloud in the constructing unit 1 in S2. There is one or more point q(i) observed from image P(i), but there is one or more point q(i) for a certain voxel V(k) according to equation (6) or (6'). When the probability VP(k) is subject to a decreasing update, the decreasing update according to equation (6) or (6') can be applied repeatedly for the number of the one or more points q(i). In other words, by cumulatively applying the decrease update the same number of times as the number of line segments L(pos(i),q(i)) that intersect the voxel V(k), the higher the number, the higher the existence probability VP(k) may be made smaller. As another embodiment, when one or more points q(i) with respect to a certain voxel V(k) are subject to a decreasing update of the existence probability VP(k) according to equation (6) or (6'), 1 The reduction update may be applied only once.

In the example EX2 of FIG. 2 described above, a schematic example of voxels whose existence probability is reduced according to equation (6) in step S43 is shown as a two-dimensional arrangement. That is, in example EX2, three examples of points q(i) observed from image P(i) are shown as circles (〇), and camera position pos(i) of image P(i) is set as camera C. Shown schematically. The three straight lines connecting the camera C and the three observation points q(i) are shown as dashed arrows, and the voxels Vb (indicated in white in the figure) intersect with any of the three straight lines. is the target of decreasing and updating the existence probability according to equation (6). On the other hand, formula (6) is not applied to a plurality of voxels Va (shown in gray in the figure) that do not intersect with any of the three straight lines.

As schematically shown in example EX2 in FIG. , points belonging to the point group as feature points extracted from image P(i)) (assuming that there is no object between them (looking in the direction of observation point q(i) from image P(i)) In this case, the observation point q(i) is the first visible point in the foreground and corresponds to the object surface), which effectively reduces the probability of the existence of voxels that are assumed to be affected by noise. can.

In step S44, it is determined whether or not the processing of steps S41, S42, and S43 described above has been completed for all N images P(i) (i=1,2,...,N), and whether or not it has been completed is determined. If the process is not completed, the process returns to step S4, sets the next unprocessed image P(i+1) as the processing target, and repeats the same process.

In step S5, the post-processing unit 23 deletes voxels V(k) whose existence probability VP(k) is less than or equal to a predetermined threshold TH as being affected by noise, etc., and then proceeds to step S6. move on. Here, the existence probability VP(k) is updated by the repeated processing in steps S4, S41, S42, S43, and S44 by the evaluation unit 22 as explained above, and the existence probability VP(k) is By performing a threshold value determination in step S5, voxels V(k) that are assumed to be affected by noise and do not correspond to the real object captured in the image are identified as belonging to the model. It is possible to delete it as if it does not exist. In addition, as another embodiment of step S5, instead of deleting voxels V(k) whose existence probability VP(k) is less than or equal to a predetermined threshold TH, A predetermined percentage of lower voxels (eg, lower 10%) of voxels V(k) may be deleted.

In step S6, the post-processing unit 23 further scans the connections between voxels for the remaining voxels that were not deleted in step S5 to find connection relationships, and identifies voxels that are connected to each other but are not connected to the outer periphery. After deleting it, proceed to step S7.

The outer periphery in step S6 may be set in three-dimensional world coordinates as the minimum rectangular parallelepiped that includes the initial voxel model set in step S3. Such voxels that are not connected to the outer periphery mean that they are floating in the air, and can be determined to be affected by noise or the like and deleted in step S6.

In step S7, the post-processing unit 23 outputs the voxels remaining after deletion in steps S5 and S6 as a final voxel model, and the flow in FIG. 3 ends.

As described above, according to the present embodiment, even when the point cloud contains noise and it is difficult to obtain an appropriate three-dimensional model using existing methods, the selection unit 2 selects points that are assumed to be caused by noise. By efficiently excluding the noise, an appropriate three-dimensional model with reduced noise influence can be obtained. Furthermore, according to the present embodiment, even for objects such as planes with flat patterns and few features, the sorting unit 2 can efficiently exclude points in areas other than the object, thereby creating an appropriate three-dimensional model. Obtainable. (For example, if there is an almost pure white wall in a certain area, it is difficult for feature points to occur on it, but when the wall is viewed from the side rather than the front, if a point cloud occurs outside the area of the wall, the line of sight is By decreasing the existence probability of the voxel, a space is generated that is determined not to correspond to the object.In other words, by decreasing the existence probability of the space in front of the white wall and disappearing, the area of the white wall with few feature points is created. (It is possible to appropriately obtain the result that .

(1) Each of the plurality of images input to the construction unit 1 may be taken with a monocular camera or a stereo camera. In the case of a stereo camera, the images may be treated as two images taken by two monocular cameras.

(2) As explained in equations (1) to (6), the evaluation unit 22 evaluated the voxels as existence probabilities of 0 or more and 1 or less, but it was assumed that the voxels were highly influenced by noise. Similar evaluation may be performed using any other arbitrary score value in which the value decreases as the number of points increases.

(3) FIG. 4 is a diagram showing an example of the hardware configuration of a general computer device 70. The model generation device 10 can be realized as one or more computer devices 70 having such a configuration. Note that when the model generation device 10 is implemented using two or more computer devices 70, information necessary for processing may be sent and received via a network. The computer device 70 includes a CPU (central processing unit) 71 that executes predetermined instructions, a GPU (graphics processing unit) as a dedicated processor that executes some or all of the instructions executed by the CPU 71 instead of the CPU 71 or in cooperation with the CPU 71. ) 72, RAM 73 as a main storage device that provides a work area for the CPU 71 (and GPU 72), ROM 74 as an auxiliary storage device, communication interface 75, display 76, input interface 77 that accepts user input from a mouse, keyboard, touch panel, etc. It includes a camera 78 and a bus BS for exchanging data between them.

Each functional unit of the model generation device 10 can be realized by the CPU 71 and/or the GPU 72, which read a predetermined program corresponding to the function of each unit from the ROM 74 and execute it. Note that both the CPU 71 and the GPU 72 are a type of arithmetic unit (processor). Here, when display-related processing is performed, the display 76 also operates in conjunction, and when communication-related processing regarding data transmission and reception is performed, the communication interface 75 further operates in conjunction. The processing results etc. by the model generation device 10 may be displayed on the display 76 and output. All or part of the plurality of images input to the construction unit 1 may be photographed by the camera 78.

10...Model generation device, 1...Construction section, 2...Selection section, 21...Initial setting section, 22...Evaluation section, 23...Post-processing section

Claims (7)

A point cloud of the object photographed in the image is constructed from multiple images, and information about the camera that photographed the image from which each point belonging to the point cloud is extracted is stored as correspondence information between each point and the camera. A construction part to be obtained as,
Generating an initial voxel model from the point cloud,
Among the voxels belonging to the initial voxel model, the voxels that intersect the line segment connecting the position of the camera that captured the plurality of images and each point that the camera corresponds to based on the correspondence information, By evaluating the probability of falling under the target as low,
A model generation device comprising: a selection unit that generates a voxel model of the point group by selecting voxels that are determined to have a high evaluated probability from among voxels belonging to the initial voxel model. In the sorting section, further,
Among the voxels belonging to the initial voxel model, the voxels corresponding to the positions of the cameras that took the plurality of images are evaluated as having a low probability of corresponding to the object being photographed,
The model generation device according to claim 1, wherein the voxel model of the point group is generated by the sorting from voxels belonging to the initial voxel model. In the sorting section, further,
For each voxel belonging to the initial voxel model, the greater the number of cameras that correspond to each point of the point group located within the voxel based on the correspondence information , the higher the probability that the point corresponds to the object being photographed. By evaluating it as high,
3. The model generation device according to claim 1, wherein the voxel model of the point group is generated by the sorting from voxels belonging to the initial voxel model. In the sorting unit, after the sorting from the voxels belonging to the initial voxel model, further:
4. A voxel model of the point group is generated by deleting, from the selected voxel model, voxels that do not have a connection relationship with an outer periphery that includes the initial voxel model. The model generation device according to any one of the above. The selection unit evaluates, with respect to voxels that intersect on the line segment, the greater the number of intersecting line segments, the lower the probability that the voxel corresponds to the object being photographed. 5. The model generation device according to any one of 1 to 4. A point cloud of the object photographed in the image is constructed from multiple images, and information about the camera that photographed the image from which each point belonging to the point cloud is extracted is stored as correspondence information between each point and the camera. a construction stage to obtain as;
Generating an initial voxel model from the point cloud,
Among the voxels belonging to the initial voxel model, the voxels that intersect the line segment connecting the position of the camera that captured the plurality of images and each point that the camera corresponds to based on the correspondence information, By evaluating the probability of falling under the target as low,
A model generation method comprising: a selection step of generating a voxel model of the point group by selecting voxels that are determined to have a high evaluated probability from among voxels belonging to the initial voxel model. A program for causing a computer to function as a model generation device according to any one of claims 1 to 5.

Images