JP7724429B2 - 3D model generation device, 3D model generation method, and 3D model generation program - Google Patents

Description

The present invention relates to a 3D model generation device, a 3D model generation method, and a 3D model generation program that use a processor to execute a process for generating a 3D model based on point cloud data acquired by performing a 3D reconstruction process on a target location.

Conventionally, 3D restoration technology is known that generates point cloud data as 3D spatial information about a target location based on photographed images of the target location. Furthermore, technology is known that generates a 3D model of a target location from point cloud data of the target location (see Patent Document 1).

Japanese Patent Application Laid-Open No. 2022-142994

With conventional technology, even users who are not familiar with the algorithms for generating 3D models from point cloud data can easily and quickly generate 3D models suitable for radio wave propagation simulations.

However, 3D models generated from point cloud data can be used for a variety of purposes, not just radio wave propagation simulations. In particular, it is possible to measure the distance between two opposing walls based on a 3D model of a target location. It is also possible to combine multiple 3D models to generate a larger 3D model. For this reason, there is a demand for technology that can easily measure the distance between walls at a target location and combine multiple 3D models, but conventional technology does not take these needs into consideration.

The primary objective of the present invention is to provide a 3D model generation device, a 3D model generation method, and a 3D model generation program that can generate 3D models that allow for easy measurement of inter-surface distances and the combining of multiple 3D models.

The three-dimensional model generation device of the present invention is a three-dimensional model generation device that uses a processor to execute processing for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location , and when setting coordinate axes along four wall surfaces of the target location based on the distribution of normal directions for the entire point cloud of the target location that forms a rectangle in a plan view, the processor acquires, as normal angles, radial angles when the normal directions of each point constituting the point cloud are expressed in cylindrical coordinates, and multiplies the normal angles of each point by four to acquire quadrupled values of each point, and calculates estimates that exclude the influence of outliers from the quadrupled values of each point. a conormal clustering process is performed to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range, as a conormal cluster; and a rectangularization process is performed to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model in which the target location is represented by a set of multiple rectangular planes.

Furthermore, the three-dimensional model generation method of the present invention is a three-dimensional model generation method in which a processor executes a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, and the method includes the steps of: when setting coordinate axes along four wall surfaces of the target location based on the distribution of normal directions for the entire point cloud of the target location that forms a rectangle in a plan view, acquiring a radial angle when the normal directions of each point constituting the point cloud are expressed in cylindrical coordinates as a normal angle, multiplying the normal angle of each point by four to acquire a quadrupled value of each point, and excluding the influence of outliers from the quadrupled value of each point; using the above formula, estimate an optimal value of the quadruple value, multiply the optimal value of the quadruple value by 1/4 to obtain a rotation angle that defines the direction of the coordinate axis, rotate the provisionally set coordinate axis by the rotation angle to set the normal coordinate axis, perform a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range, and perform a rectangularization process to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

Furthermore, a three-dimensional model generation program of the present invention is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, and when setting coordinate axes along the four wall surfaces of the target location based on the distribution of normal directions for the entire point cloud of the target location that forms a rectangle in a plan view, acquires the radial angle when the normal direction of each point that constitutes the point cloud is expressed in cylindrical coordinates as the normal angle, multiplies the normal angle of each point by four to acquire the quadrupled value of each point, and estimates the quadrupled value of each point to exclude the influence of outliers. a conormal clustering process is performed to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range, as a conormal cluster; and a rectangularization process is performed to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model in which the target location is represented by a set of multiple rectangular planes.

The present invention allows for the creation of a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure the distance between surfaces. It also allows for the easy combination of multiple 3D models. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

Overall configuration of a three-dimensional model generation system according to the present embodiment. Block diagram showing the general configuration of a user terminal and a server Flow diagram showing the procedure for generating a 3D model performed by the server Flow diagram showing the preprocessing steps performed on the server FIG. 10 is an explanatory diagram showing the state of the point cloud at the restoration location before and after the coordinate axis setting process performed by the server. FIG. 10 is an explanatory diagram showing the status of the first and second coordinate axis setting processes performed by the server; FIG. 10 is an explanatory diagram showing the procedure for setting a second coordinate axis performed by the server; FIG. 10 is an explanatory diagram showing the procedure for setting a second coordinate axis performed by the server; Flow diagram showing the clustering process performed by the server An explanatory diagram showing the status of conormal clustering processing performed on the server. An explanatory diagram showing an overview of the conormal clustering process performed on the server An explanatory diagram showing an overview of the individual clustering processing performed by the server An explanatory diagram showing the status of the axis rotation process performed by the server. An explanatory diagram showing the status of sub-clustering processing performed on the server An explanatory diagram showing the status of sub-clustering processing performed on the server An explanatory diagram showing the status of sub-clustering processing performed on the server An explanatory diagram showing the status of sub-clustering processing performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server An explanatory diagram showing the sub-clustering process performed on the server Flow diagram showing the procedure for rectangularization processing performed on the server An explanatory diagram showing the state of rectangularization processing performed on the server. FIG. 10 is an explanatory diagram showing a three-dimensional model generated by rectangularization processing performed on a server. FIG. 10 is an explanatory diagram showing a three-dimensional model generated by rectangularization processing performed on a server. An explanatory diagram showing an overview of the alignment process performed by the server FIG. 10 is an explanatory diagram showing an overview of snap processing between first and second groups performed by the server; An explanatory diagram showing an overview of the intra-group snap processing performed on the server An explanatory diagram showing the status of floor plan generation processing performed on the server. An explanatory diagram showing the state of dimension line insertion processing performed on the server FIG. 10 is an explanatory diagram showing a viewing screen displayed on a user terminal.

A first invention made to solve the above problems is a 3D model generation device that uses a processor to execute processing to generate a 3D model based on point cloud data acquired by performing 3D reconstruction processing on a target location, wherein the processor, when setting coordinate axes along four wall surfaces of the target location based on the distribution of normal directions for the entire point cloud of the target location that forms a rectangle in a plan view, acquires, as normal angles, radial angles when the normal directions of each point constituting the point cloud are expressed in cylindrical coordinates, multiplies the normal angles of each point by four to acquire quadrupled values of each point, and removes the influence of outliers from the quadrupled values of each point. an optimal value of the quadruple value is estimated using an estimation method that excludes the quadruple value, the optimal value of the quadruple value is multiplied by 1/4 to obtain a rotation angle that defines the direction of the coordinate axis, the provisionally set coordinate axis is rotated by the rotation angle to set the normal coordinate axis, a conormal clustering process is performed to extract, from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range, and a rectangularization process is performed to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

This allows for the creation of a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure the distance between surfaces. It also makes it easy to combine multiple 3D models. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

In this way, coordinate axes are set based on the distribution of normal directions for the entire point cloud of the restoration location. Compared to methods that set coordinate axes based on a specific plane, errors due to distortions that occur in the point cloud of the restoration location are dispersed, making it possible to appropriately set coordinate axes that follow the wall surfaces of the target location. Furthermore, even if the point cloud of the restoration location primarily represents furniture such as desks and shelves placed at the restoration location and there is no large-area wall surface to serve as a reference, coordinate axes can still be appropriately set.

This allows for easy and appropriate setting of coordinate axes based on the distribution of normal directions across the entire point cloud at the restoration location.

A second invention is a three-dimensional model generation device that uses a processor to execute a process for generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location, wherein the processor sets coordinate axes along the wall surfaces of the target location for the point cloud of the target location, executes a conormal clustering process to extract from the point cloud of the target location a set of points whose normal direction coincides with a predetermined direction based on the coordinate axes within a predetermined tolerance range as a conormal cluster, executes an individual clustering process before the conormal clustering process to extract planes of any direction from the point cloud of the target location as individual clusters , and executes a rectangularization process on the individual clusters and the conormal clusters to generate the three-dimensional model that represents the target location as a set of multiple rectangular planes .

As a result, because individual clustering processing can extract planes in any direction, it compensates for the drawback of conormal clustering processing, which can only extract planes in a specified direction. On the other hand, while individual clustering processing cannot detect small planes, conormal clustering processing can detect even small planes as long as they are in a specified direction, so the drawback of individual clustering processing is compensated for by conormal clustering processing. As a result, by combining both conormal clustering processing and individual clustering processing, it is possible to appropriately extract the required clusters.

In a third aspect of the present invention, the processor divides the conormal cluster into a plurality of sub-clusters corresponding to the shape of the conormal cluster, and converts the conormal cluster and the sub-clusters into the rectangular plane.

This allows for improved fidelity of the 3D model of the target location.

A fourth aspect of the present invention is a three-dimensional model generation method for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, using a processor. When setting coordinate axes along four wall surfaces of the target location based on the distribution of normal directions of the entire point cloud of the target location that forms a rectangle in a plan view, the method acquires, as normal angles, radial angles when the normal directions of each point constituting the point cloud are expressed in cylindrical coordinates, and multiplies the normal angles of each point by four to acquire quadrupled values of each point. The method then uses an estimation method that excludes the influence of outliers on the quadrupled values of each point. the optimal value of the quadruple value is estimated, the optimal value of the quadruple value is multiplied by 1/4 to obtain a rotation angle that defines the direction of the coordinate axes, the provisionally set coordinate axes are rotated by the rotation angle to set the normal coordinate axes, a conormal clustering process is performed to extract from the point cloud of the target location a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axes within a predetermined tolerance range, and a rectangularization process is performed to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

As with the first invention, this allows for the creation of a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure inter-surface distances. It also allows for the easy combination of multiple 3D models. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

A fifth aspect of the present invention is a three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, the program comprising: when setting coordinate axes along four wall surfaces of the target location based on a distribution state of normal directions for the entire point cloud of the target location that forms a rectangle in a plan view, acquiring a radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates as a normal angle; multiplying the normal angle of each point by four to acquire a quadrupled value of each point; and using an estimation method that excludes the influence of outliers on the quadrupled value of each point. the optimal value of the quadruple value is estimated, the optimal value of the quadruple value is multiplied by 1/4 to obtain a rotation angle that defines the direction of the coordinate axis, the provisionally set coordinate axis is rotated by the rotation angle to set the normal coordinate axis, a conormal clustering process is performed to extract , from the point cloud of the target location, a set of points whose normal direction coincides with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range, and a rectangularization process is performed to convert the conormal cluster into a rectangular plane, thereby generating the three-dimensional model that represents the target location as a set of multiple rectangular planes.

As with the first invention, this allows for the creation of a 3D model in which the parallelism of parallel planes at the reconstruction location is ensured. This makes it easy to measure inter-surface distances. It also allows for the easy combination of multiple 3D models. Furthermore, since a moderately abstract 3D model is generated, the processing load on the processor can be reduced.

Embodiments of the present invention will be described below with reference to the drawings.

Figure 1 is a diagram showing the overall configuration of a 3D model generation system according to this embodiment.

This system comprises a user terminal 1 and a server 2 (a 3D model generation device and a drawing creation and display device). The user terminal 1 and server 2 are connected via a network such as a LAN or the Internet.

The user terminal 1 comprises a device main body 11 and a camera 12. The user terminal 1 can be configured as a tablet terminal or a notebook PC.

Camera 12 is a visible camera, i.e., a monocular camera that detects visible light to capture an image of a subject, and outputs the captured image, for example, an RGB color image. For the 3D reconstruction performed by server 2, in addition to camera 12 (visible camera), user terminal 1 may be equipped with sensors such as a depth camera that measures the distance to the subject and an IMU (Inertial Measurement Unit) that detects three-dimensional angular velocity and acceleration.

The user (worker) walks through the target location while holding the device main body 11 of the user terminal 1. At this time, the camera 12 of the user terminal 1 captures images of the target location, and captured images of each point in the target location are acquired sequentially. Note that while an example has been shown in which a user holding the device main body 11 captures captured images by walking through the target location, it is also possible to acquire captured images of the target location using a self-propelled robot or the like equipped with the user terminal 1.

The server 2 is composed of a PC. Based on the captured images received from the user terminal 1, the server 2 generates point cloud data as a 3D reconstruction result, which is 3D spatial information about the target location. The server 2 also generates a 3D model of the target location based on the point cloud data of the target location. The server 2 also generates a floor plan (2D layout diagram) of the target location based on the 3D model of the target location. The server 2 also displays the generated point cloud data, 3D model, and floor plan of the target location on the user terminal 1.

Next, we will explain the general configuration of the user terminal 1 and server 2. Figure 2 is a block diagram showing the general configuration of the user terminal 1 and server 2. Figure 3 is a flow diagram showing the steps in the 3D model generation process.

In addition to the camera 12, the user terminal 1 is equipped with a display 13 (display unit), an input device 14, a memory unit 15, a processor 16 (CPU), and a communication unit 17.

The display 13 displays various operation screens related to shooting and editing operations, a setting screen for setting various processing conditions, and a viewing screen for checking the generated 3D reconstruction results (point cloud data) and 3D models. The input device 14 is used by the user to perform input operations. If the user terminal 1 is configured as a tablet terminal, a touch panel display is provided that integrates a touch panel as the input device 14 and a display panel as the display 13.

The memory unit 15 stores programs executed by the processor 16. The memory unit 15 also stores images captured by the camera 12. The captured images are transmitted to the server 2 at appropriate times.

The processor 16 executes programs stored in the memory unit 15 to perform various processes related to capturing images using the camera 12, displaying images on the display 13, and obtaining operation information using the input device 14.

The communication unit 17 communicates with the server 2. Specifically, the communication unit 17 transmits to the server 2 images captured by the camera 12 and operation information acquired through user operation of the input device 14. The communication unit 17 also receives various screen display information transmitted from the server 2.

The server 2 includes a communication unit 21, a memory unit 22, and a processor 23.

The communication unit 21 communicates with the user terminal 1. Specifically, the communication unit 21 receives captured images and operation information sent from the user terminal 1. The communication unit 21 also sends display information for various screens to be displayed on the user terminal 1 to the user terminal 1.

The storage unit 22 stores programs executed by the processor 23. The storage unit 22 also stores point cloud data generated by the processor 23, data on 3D and 2D models, and floor plans (layout drawings).

The processor 23 performs various processes by executing programs stored in the memory unit 22. In this embodiment, the processor 23 performs 3D reconstruction processing, 3D model generation processing, drawing generation processing, drawing display processing, and the like.

In the 3D reconstruction process, the processor 23 generates point cloud data as 3D spatial information about the target location based on the captured image received from the user terminal 1. The 3D reconstruction process can be achieved, for example, by determining the camera position and orientation in the world coordinate system at the time of capture using the SLAM (Simultaneous Localization and Mapping) method, and then superimposing the depth information obtained from the depth camera on the world coordinate system based on this camera position and orientation.

In the 3D model generation process, the processor 23 generates a 3D model of the target location based on the point cloud data generated in the 3D reconstruction process. In the 3D model generation process, the point cloud of the target location is sequentially subjected to preprocessing (ST101), clustering (ST102), and rectangularization (ST103), to generate a 3D model (see Figure 3).

In pre-processing, the processor 23 performs processes on the point cloud of the target location generated by the 3D reconstruction process, such as estimating the normals of each point that makes up the point cloud, extracting the floor surface, setting coordinate axes, and extracting flat surfaces.

In the clustering process, the processor 23 detects a plane (a set of points distributed in a plane at a predetermined density) from the point cloud of the target location, and extracts the set of points that make up that plane as a cluster. In the clustering process, the processor 23 also divides the cluster into multiple sub-clusters.

In the rectangularization process, the processor 23 converts the clusters and sub-clusters obtained in the clustering process into rectangular planes, and generates a three-dimensional model that represents the target location as a collection of multiple rectangular planes.

The drawing generation process includes floor plan generation and dimension line insertion.

In the floor plan generation process, the processor 23 generates a floor plan (2D layout diagram) of the restoration location based on the 3D model generated in the 3D model generation process. Specifically, the floor and ceiling portions are removed from the 3D model, and the 3D model is orthogonally projected onto the XY plane (horizontal plane), i.e., orthogonally projected in the Z direction, to generate the floor plan.

In the dimension line insertion process, the processor 23 extracts a representative plane from the three-dimensional model and inserts dimension lines related to that representative plane into the plan view. For example, a wall surface at the restoration location is extracted as a representative plane, and a dimension line representing the inter-surface distance between two opposing wall surfaces is inserted into the plan view.

Drawing display processing includes floor plan display processing and measurement display processing.

In the floor plan display process, the processor 23 displays the floor plan generated in the drawing generation process on the display 13 of the user terminal 1.

In the measurement display process, the processor 23 executes a measurement process related to the specified measurement object in response to a user operation to specify the measurement object, and displays the measurement results on the display 13 of the user terminal 1. Specifically, the inter-plane distance between two planes is measured based on a three-dimensional model, and this inter-plane distance is displayed. In addition, the parallelism of the two planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed.

Next, we will explain the pre-processing (ST101 in Figure 3) performed by server 2. Figure 4 is a flow diagram showing the pre-processing procedure.

First, server 2 performs normal estimation processing (ST201). In normal estimation processing, the normals of each point that constitutes the point cloud at the restoration location are estimated.

Next, floor surface estimation processing is performed (ST202). In floor surface estimation processing, a horizontal plane that will become the floor surface is extracted from the point cloud at the restored location. Specifically, points where the dot product of the normal vector at each point that makes up the point cloud and the Z-direction reference vector [0,0,1] exceeds a predetermined threshold (e.g., 0.9) are extracted as horizontal surfaces. A histogram of the z coordinates of each point that makes up the point cloud is also obtained, and the first and second peaks correspond to the floor and ceiling surfaces, with the smaller z coordinate being recognized as the floor surface.

Next, coordinate axis setting processing is performed (ST203). In the coordinate axis setting processing, coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in directions that follow the wall surfaces of the target location.

Next, flat surface extraction processing is performed (ST204). In flat surface extraction processing, local features are calculated for the point cloud at the restoration location, and flat surfaces are extracted based on these local features. Principal component analysis is used to calculate the local features. First, a covariance matrix is calculated from the neighboring points of each point, and the eigenvalues of this covariance matrix are calculated. The eigenvalues are sorted in descending order and, assuming λ1, λ2, and λ3, points where (λ2 - λ3)/λ1>0.3 are extracted. Subsequent processing, such as clustering, is performed on the extracted flat surfaces.

In addition to the above, preprocessing also removes noise (isolated points) from the point cloud at the restoration location and downsamples the point cloud. In downsampling, the voxel size is set to 0.01 (1 cm), for example. The entire point cloud is also translated and rotated so that the floor is at z=0, the center of the point cloud is the center, and the maximum vertical plane is in the X-axis direction.

Next, we will explain the coordinate axis setting process performed by server 2. Figure 5 is an explanatory diagram showing the state of the point cloud at the restoration location before and after the coordinate axis setting process is performed. Figure 6 is an explanatory diagram showing the state of the first and second coordinate axis setting processes. Figures 7 and 8 are explanatory diagrams showing the outline of the second coordinate axis setting process.

In the coordinate axis setting process, the coordinate axes (X-axis, Y-axis, and Z-axis) of a three-dimensional Cartesian coordinate system are set for the point cloud data of the restoration location acquired by the three-dimensional restoration process. In this embodiment, the coordinate axes (X-axis and Y-axis) are set for the point cloud of the restoration location in a direction along the wall surface of the target location. Note that the vertical direction of the point cloud can be set based on the detection data during image capture by the IMU installed in the user terminal 1, so the Z-axis is known. In other words, the direction perpendicular to the floor surface acquired in the floor surface estimation process is set as the Z-axis.

In the example shown in Figure 5, the restoration location is a room with four walls constructed to form a rectangle in plan view. Specifically, the restoration location is a conference room, with desks, chairs, etc. arranged inside the room. Note that the example shown in Figure 5 shows the point cloud generated for the restoration location as viewed from directly above, but the point cloud that makes up the ceiling has been deleted from the point cloud of the restoration location so that the interior of the room appears.

In the example shown in Figure 5(A), the coordinate axes for the point cloud at the restoration location are provisionally set at an angle relative to the walls of the restoration location. In this state, subsequent processing such as clustering cannot be performed properly. Therefore, in this embodiment, as shown in Figure 5(B), the coordinate axes (X-axis and Y-axis) are set to follow the walls of the restoration location. In other words, the coordinate axes (X-axis and Y-axis) are set in two directions perpendicular to the four walls that form a rectangle in plan view.

There are two types of coordinate axis setting processes: first coordinate axis setting process and second coordinate axis setting process, and either the first coordinate axis setting process or the second coordinate axis setting process is performed. That is, a determination is made as to whether the first coordinate axis setting process is appropriate, and if it is determined that the first coordinate axis setting process is appropriate, the first coordinate axis setting process is performed. If it is determined that the first coordinate axis setting process is inappropriate, the second coordinate axis setting process is performed.

In the first coordinate axis setting process, the plane with the largest area detected from the point cloud of the restoration location is used as the reference plane, and the coordinate axes are set to follow this reference plane. At this time, a vertical plane is extracted from the point cloud of the target location. Specifically, points are extracted where the dot product of the normal of each point that makes up the point cloud and the vertical normal [0,0,1] based on the coordinate axis is less than a threshold value (e.g., 0.5).

In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions across the entire point cloud of the restoration location, which forms a rectangle in plan view, and the coordinate axes are set based on the estimation results. In a typical indoor space, the four wall surfaces are constructed to form a rectangle in plan view. Furthermore, many objects placed indoors, such as desks and shelves, are positioned parallel to the walls. For this reason, the directions of the four wall surfaces of the restoration location can be estimated based on the distribution of normal directions across the entire point cloud of the restoration location.

In addition, when determining whether the first coordinate axis setting process is appropriate, a determination is made as to whether the plane with the largest area detected from the point cloud of the restored location is appropriate as a reference plane. For example, a determination is made as to whether the plane with the largest area is sufficiently large compared to the overall size of the target location. Here, if the first coordinate axis setting process is determined to be appropriate, the first coordinate axis setting process is executed, and if the first coordinate axis setting process is determined to be inappropriate, the second coordinate axis setting process is executed.

In the example shown in Figure 6(A), the plane with the largest area is determined to be appropriate as the reference plane, and the first coordinate axis setting process is determined to be appropriate, so the first coordinate axis setting process is executed. This allows the coordinate axes (X-axis and Y-axis) to be appropriately set for the point cloud at the restoration location.

On the other hand, in the example shown in Figure 6 (B), the plane with the largest area is inappropriate as a reference plane, and when the first coordinate axis setting process is performed, the coordinate axes (X-axis and Y-axis) are inappropriately set for the point cloud of the restoration location. In other words, in the first coordinate axis setting process, the distortion of the reference plane has an effect on the entire image, and there is a significant error between the direction of the coordinate axes (X-axis and Y-axis) and the direction of the wall surface of the restoration location.

In contrast, the second coordinate axis setting process allows the coordinate axes to be set appropriately for the entire wall surface, as shown in Figure 6(C). In this case, errors due to distortion occurring in the point cloud at the restored location are dispersed, and the coordinate axes are set so that the error between the direction of the coordinate axes (X-axis and Y-axis) and the direction of the wall surface at the restored location is minimized.

In the second coordinate axis setting process, the directions of the four wall surfaces (vertical surfaces) of the restoration location are estimated based on the distribution of normal directions for the entire point cloud of the restoration location, which is rectangular in plan view, and the coordinate axes are set based on these estimation results. Note that in the second coordinate axis setting process, because horizontal coordinate axes (X-axis and Y-axis) are set, only points whose normal directions are roughly horizontal, i.e., points representing vertical surfaces, are extracted as processing targets from among the points that make up the point cloud of the target location.

Specifically, as shown in Figure 7(A), if the normal vectors of each point that makes up the point cloud at the restoration location are positioned so that their starting points are located at the origin of the coordinate axes, the end points of the normal vectors of each point, i.e., the points representing the normal directions of each point, will be lined up on a circumference (sphere), making it possible to grasp the distribution of the normal directions of each point.

The example shown in Figure 7(B) represents the distribution of the normal directions of each point that makes up the point cloud of a restoration location where four wall surfaces are arranged to form a rectangle in a plan view. Here, the normal direction of each point that makes up the point cloud is expressed in cylindrical coordinates (r, θ, z), and the normal angle θ (the azimuth angle of the normal vector) is expressed by the radial angle of the cylindrical coordinates. Furthermore, the points that represent the normal directions of each point that makes up the point cloud are concentrated in four locations, spaced 90 degrees apart, corresponding to the directions of the four wall surfaces.

Next, as shown in Figure 8, the normal angle θ of each point on the cylindrical coordinate system is multiplied by four to obtain the quadrupled value 4θ. Here, the four concentrated distribution points (see Figure 7(B)) where points representing the normal directions of each point that make up the point cloud are gathered are aggregated into a single concentrated distribution point.

Next, the optimal value of the quadruple value 4θ is estimated. Here, the points representing the normal directions of each point that makes up the point cloud contain many outliers. Therefore, an estimation method that removes the influence of outliers is applied to the quadruple value 4θ of each point to estimate the optimal value of the quadruple value 4θ. One possible estimation method is to use, for example, the RANSAC (Random Sample Consensus) method to remove outliers, then find the center of gravity in cylindrical coordinates and obtain the radial angle of the center of gravity.

Once the optimal value of the quadruple value 4θ is found in this way, the rotation angle φ that defines the direction of the coordinate axes is found by multiplying that optimal value by 1/4. The rotation angle φ obtained here represents the inclination of the typical normal angle with respect to the provisionally set horizontal coordinate axes (X-axis and Y-axis), i.e., the inclination of the four wall surfaces of the restored location. Therefore, by rotating the provisionally set horizontal coordinate axes (X-axis and Y-axis) by the rotation angle φ, the correct coordinate axes are set to align with the four wall surfaces of the restored location. The rotation angle φ can be found in the range of -45 degrees to 45 degrees.

Furthermore, by determining the rotation angle φ, the directions of the two horizontal coordinate axes (X-axis and Y-axis) are determined, but the direction of the X-axis and Y-axis is indeterminate. Therefore, the X-axis and Y-axis should be set so that the point cloud of the restoration location becomes horizontal after rotation. At this time, a bounding box (circumscribed rectangle) should be set for the point cloud of the restoration location, and a determination should be made as to whether the point cloud of the restoration location is horizontal or vertical. Here, if the point cloud of the restoration location is vertical, the X-axis and Y-axis are set after rotating it by an additional 90 degrees from the rotation angle φ.

In this way, in the second coordinate axis setting process, coordinate axes are set based on the distribution of normal directions for the entire point cloud at the restoration location. Compared to the first coordinate axis setting process, which sets coordinate axes based on a specific plane, errors in the point cloud are dispersed, making it possible to appropriately set coordinate axes that follow the wall surfaces of the target location. Furthermore, because coordinate axes are set with high precision for the point cloud, clusters can be appropriately extracted in the clustering process, which is performed based on the coordinate axes.

However, depending on the restoration location, there may not be a plane with an area suitable for use as a reference plane. In such cases, it may be difficult to set the coordinate axes using the first coordinate axis setting process, which sets the coordinate axes using the plane with the largest area as the reference plane. For example, if the restoration location is a large room (such as a warehouse, office, or store) with many shelves, desks, and other furniture, and the images mainly depict furniture such as shelves and desks, the generated point cloud for the restoration location will contain almost no points representing the walls of the room. In this case, the point cloud for the restoration location does not contain a vertical plane with an area suitable for use as a reference plane.

On the other hand, furniture such as shelves and desks placed in a room are generally arranged parallel to the room's walls, and the normal direction of each point that makes up the point cloud representing furniture such as shelves and desks is often perpendicular to or parallel to the room's walls. For this reason, even if the point cloud of the restoration location contains few points that represent the room's walls, the second coordinate axis setting process can appropriately set horizontal coordinate axes (X-axis and Y-axis) that are assumed to follow the walls of the target location. Furthermore, even if there is distortion in the point cloud representing furniture such as shelves and desks, the distortion is appropriately absorbed, allowing the coordinate axes to be appropriately set.

Next, the clustering process (ST102 in Figure 3) performed by server 2 will be described. Figure 9 is a flow diagram showing the steps of the clustering process. Figure 10 is an explanatory diagram showing the status of the conormal clustering process. Figure 11 is an explanatory diagram showing an overview of the conormal clustering process.

As shown in Figure 9(A), in the clustering process, individual clustering is first performed (ST301). Next, conormal clustering is performed (ST302).

In individual clustering processing, a plane (a set of points distributed in a plane at a specified density) in an arbitrary direction is detected from the point cloud of the target location, and the set of points that make up that plane is extracted as an individual cluster. In individual clustering processing, a plane in an arbitrary direction is detected using an estimation method that eliminates the influence of outliers, such as RANSAC.

In conormal clustering processing, a group of planes (a set of points distributed in a planar pattern at a specified density) in a specified direction based on the coordinate axes is detected from the point cloud of the target location (the specified direction is described below). In other words, a set of points where the specified direction and the point normal direction of the point cloud match within a specified tolerance range are detected as a group of planes. Furthermore, each plane is extracted from the group of planes as a conormal cluster. A total of 21 specified directions are set, for example, in increments of 22.5 degrees.

The specified direction can be expressed as the components of a normal vector, for example: [1.,0.,0.], [0.,1.,0.], [0.7071,0.7071,0.], [-0.7071,0.7071,0.], [0.9239,0.3827,0.], [-0.3827,0.9239,0.], [0.3827,0.9239,0.], [-0.9239,0.3827,0.], [0.9239,0.,0.3827], [0.,0.9239,0.3827], [-0.9239, 21 possible settings are possible: [0.,0.3827],[0.,-0.9239,0.3827],[0.7071,0.,0.7071],[0.,0.7071,0.7071],[-0.7071,0.,0.7071],[0.,-0.7071,0.7071],[0.3827,0.,0.9239],[0.,0.3827,0.9239],[-0.3827,0.,0.9239],[0.,-0.3827,0.9239],[0.,0.,1.].

In the conormal clustering process (ST302 in Figure 9(A)), each step is performed according to the procedure shown in Figure 9(B).

First, a conormal surface extraction process is performed (ST401). In this process, a plane (conormal surface) is extracted from the point cloud of the restoration location, which is a collection of points whose normal direction coincides with a predetermined direction based on the coordinate axes within an allowable range. Specifically, for each point that makes up the point cloud of the restoration location, points where the dot product of the point normal and the set normal exceeds a threshold value (e.g., 0.985) are extracted as conormal surfaces. Here, a point normal refers to the normal of each point that makes up the point cloud. Furthermore, a set normal refers to a normal (one of 21 possible directions) in a predetermined direction based on the coordinate axes. In other words, a group of planes whose normal is the set normal are extracted as conormal surfaces in the predetermined direction.

Next, density-based clustering processing is performed (ST402). In density-based clustering processing, the conormal planes extracted in the conormal plane extraction processing are grouped using a density-based clustering (DBSCAN: Density-based spatial clustering of applications with noise) technique, and conormal clusters are generated for each specified direction.

Next, the smallest cluster discarding process is performed (ST403). In the smallest cluster discarding process, of the conormal clusters obtained in the density-based clustering process, those conormal clusters whose constituent points are less than a predetermined threshold are discarded. The threshold is set to 200 points, for example. In this case, if voxelization was performed in 1 cm increments in the preprocessing, for example, clusters with an area less than 200 cm2 will generally be discarded.

Next, the necessity of axis rotation processing is determined (ST404). If it is determined that axis rotation processing is necessary (Yes in ST404), the coordinate axes of the target cluster are rotated (axis rotation processing) (ST405).

Next, sub-clustering processing is performed (ST406). In sub-clustering processing, the conormal clusters obtained in the density-based clustering processing (ST402) that have undergone the minimum cluster discarding processing (ST403) are divided into sub-clusters. If axial rotation processing has been performed, once the sub-clustering processing is completed, the clusters are rotated in the opposite direction to the initial axial rotation.

Next, a cluster group generation process is performed (ST407). In the cluster group generation process, conormal clusters (including those decomposed into sub-clusters) are grouped for each predetermined direction to generate conormal cluster groups. Conormal cluster groups are generated for each predetermined direction (set normal) based on the coordinate axes.

In the conormal clustering process, each time a conormal cluster is detected, the points included in that conormal cluster are excluded from the point cloud to be processed thereafter. At this time, points included in the conormal cluster, i.e., points included in the bounding box when the conormal cluster was detected—specifically, the axis-aligned bounding box (AABB) or oriented bounding box (OBB)—are excluded. This prevents the same point from being included in multiple conormal clusters. Furthermore, since points included in conormal clusters are successively excluded from the point cloud at the target location as conormal clusters are detected, controlling the order in which conormal clusters are detected allows for prioritization of detected plane directions. For example, if priority is given to detecting vertical planes, the process can begin with detecting conormal clusters for vertical planes and finish with detecting conormal clusters for horizontal planes.

Also, while Figure 9(B) shows sub-clustering processing in the conormal clustering process, sub-clustering processing is also performed in the individual clustering process. Specifically, sub-clustering processing is performed on points included in the oriented bounding boxes detected as individual clusters in the individual clustering process.

The example shown in Figure 10 is a case where conormal clustering processing has been performed on the point cloud of the restored location shown in Figure 5(B). In conormal clustering processing, the point cloud of the restored location is decomposed into conormal planes, and these conormal planes are grouped and extracted as conormal clusters. The conormal clusters are then further grouped by specified direction (set normal). Note that in Figure 10, the conormal clusters are drawn in different colors (densities).

The example shown in Figure 10(A) is a cluster whose normal direction is the X direction [1,0,0], i.e., a group of clusters with conormal planes parallel to one wall surface. The example shown in Figure 10(B) is a cluster whose normal direction is the Y direction [0,1,0], i.e., a group of clusters with conormal planes parallel to the other wall surface. The example shown in Figure 10(C) is a cluster whose normal direction is the Z direction [0,0,1], i.e., a group of clusters with conormal planes parallel to the floor surface.

In the example shown in Figure 11(A), the object to be restored is a cylinder with two planes joined in an L-shape and arranged inside. The example shown in Figure 11(B) is a group of clusters whose normal direction extracted from the point cloud of the target location shown in Figure 11(A) is a predetermined direction [0,1,0]. In this case, there are three clusters belonging to the group. The example shown in Figure 11(C) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [-1,1,0]. In this case, there are two clusters belonging to the group. The example shown in Figure 11(D) is a group of clusters whose normal direction extracted from the point cloud of the target location is a predetermined direction [0,0,1]. In this case, there is one cluster belonging to the group.

Next, we will explain the individual clustering process performed on server 2. Figure 12 is an explanatory diagram showing the status of the individual clustering process.

In the example shown in Figure 12(A), partitions and tables are arranged in a room. One partition 51 is arranged in a different orientation from the other partitions and tables. The coordinate axes (X-axis and Y-axis) are set to follow the orientations of the partitions and tables other than partition 51. In this case, because partition 51 deviates from the set orientation based on the coordinate axes, when conormal clustering processing is performed, partition 51 is not properly represented in the 3D model after rectangularization processing, as shown in Figure 12(B). In other words, there is no continuity in the rectangular plane representing partition 51, and partition 51 is decomposed into rectangular planes 52 in two specified directions (e.g., 22.5 degrees and 45 degrees).

Therefore, in this embodiment, an individual clustering process is performed before the conormal clustering process, in which planes of any direction are individually extracted from the point cloud of the target location. As a result, in the 3D model after the rectangularization process, as shown in Figure 12 (C), the rectangular planes 53 representing the partitions 51 are arranged on a single plane, improving the shape to be closer to the actual shape.

In the individual clustering process, a plane of arbitrary direction is extracted as an individual cluster from the point cloud of the target location. At this time, an oriented bounding box is set for the point cloud of the restored location, and the points contained in that oriented bounding box are extracted as an individual cluster. In addition, the individual clustering process uses an estimation method, such as the RANSAC technique, to eliminate the influence of outliers.

In addition, the individual clustering process uses a planar patch detection technique. In planar patch detection, planar patches are detected based on the coplanarity angle. The coplanarity angle (coplanarity deg) defines the angle between the normal vector and the vector drawn from the center of the surface to a point cloud constituent point. A coplanarity angle of 90 degrees is the strictest, allowing only points that are completely on the plane. On the other hand, decreasing the coplanarity angle allows points that are outside the plane. The coplanarity angle is set to, for example, 85 degrees. This plane detection technique is described in the paper "A. Araujo and M. Oliveira, A robust statistics approach for plane detection in unorganized point clouds, Pattern Recognition, 2020."

In individual clustering, planes of any direction can be detected by setting oriented bounding boxes. A probabilistic search is performed using the RANSAC method. Therefore, while individual clustering can detect large planes from the point cloud at the restored location, it may not be able to detect small planes. On the other hand, conormal clustering cannot detect planes that deviate significantly from the specified direction based on the coordinate axes, but it can also detect small planes.

In this way, individual clustering processing can extract planes in any direction, which compensates for the drawback of conormal clustering processing, which can only extract planes in a specified direction. On the other hand, while individual clustering processing cannot detect small planes, conormal clustering processing can detect even small planes as long as they are in a specified direction, so the drawback of individual clustering processing is compensated for by conormal clustering processing. As a result, by combining both conormal clustering processing and individual clustering processing, it is possible to appropriately extract the required clusters.

In addition, in this embodiment, the individual clustering process is performed before the conormal clustering process, and first, large planes in any direction are detected by the individual clustering process. Next, the conormal clustering process is performed. At this time, the points that make up the individual clusters obtained by the individual clustering process are excluded from the point cloud of the target location, and then the conormal clustering process is performed. This prevents the points that make up the individual clusters detected by the individual clustering process from being re-detected by the conormal clustering process. This makes it possible to avoid the generation of incorrect clusters caused by forcibly decomposing planes in a direction that deviates from the specified direction based on the coordinate axes into planes in the specified direction.

In addition, while conormal clustering may detect many small planes, it is limited to planes in a specific direction based on the coordinate axes, so the 3D model obtained after rectangularization processing is generated in a neat and orderly manner without being overly complex.

In the individual clustering process, planes of any direction can be detected by setting a oriented bounding box for the point cloud. However, the normal direction obtained from the oriented bounding box is based on random numbers assigned by RANSAC and is therefore of low accuracy. Therefore, it is advisable to redo the plane estimation for each point included in the oriented bounding box to obtain the correct normal direction.

In the coordinate axis setting process, the coordinate axes (X-axis and Y-axis) are set so that they are parallel to the walls of a room that is rectangular in plan view. However, in rooms that are not rectangular in plan view, some of the walls may not be parallel to the coordinate axes. In such cases, even conormal clustering process may not be able to extract some of the walls. On the other hand, individual clustering process can detect planes in any direction that deviates from the specified direction based on the coordinate axes, so it can be used to detect the walls of rooms that are not rectangular in plan view.

Note that if there is an object, such as partition 51 in the example shown in Figure 12, that is positioned in a direction significantly different from the wall of the room and the many other objects that are positioned parallel to that wall, such an object will be an outlier in the second coordinate axis setting process based on the distribution of normal directions, and will not affect the setting of the coordinate axes.

Next, we will explain the axis rotation processing performed by server 2. Figure 13 is an explanatory diagram showing the status of the axis rotation processing.

In the example shown in Figure 13(A), a long, thin handrail 61 is placed diagonally along the wall. The handrail 61 extends in a direction significantly different from the direction of the coordinate axes. In contrast, in conormal clustering processing, one cluster is set in the point cloud representing the handrail, and this cluster is then divided into multiple sub-clusters in sub-clustering processing. At this time, a bounding box (circumscribing rectangle) with sides aligned with the coordinate axis direction is set, and the cluster set for the entire handrail 61 is decomposed into bounding boxes aligned with the coordinate axis direction. For this reason, when rectangularization processing is performed after clustering processing, the handrail 61 is represented in a stepped shape made up of multiple rectangular planes 62, as shown in Figure 13(B), which is significantly different from the actual shape.

Therefore, in this embodiment, before entering the sub-clustering process, an axis rotation process is performed to rotate the coordinate axes relative to the target cluster. As a result, as shown in Figure 13 (C), multiple rectangular planes 63 representing the handrail 61 are aligned straight in the direction in which the handrail 61 extends, improving the state to be closer to the actual shape.

The necessity of axial rotation processing is also determined, and if necessary, axial rotation processing is performed. The necessity of axial rotation processing is determined based on the fitting rate of an oriented bounding box that is set in an arbitrary direction relative to the point cloud of the cluster. In other words, axial rotation processing is performed if the fitting rate of the oriented bounding box is high. Furthermore, when axial rotation processing is performed, the orientation of the sides of the rectangular plane corresponding to the axially rotated cluster in the 3D model after rectangularization processing differs from that of other rectangular planes, which can make the 3D model appear cluttered and unsightly. Therefore, it is recommended to adjust the threshold for the fitting rate of the oriented bounding box to prevent excessive axial rotation processing. This allows for the generation of a 3D model that is orderly and easy to view.

In the axis rotation process, the angle by which the coordinate axes are rotated relative to the cluster is set based on the axial tilt of the oriented bounding box set for the cluster's point cloud.

Specifically, first, the cluster's point cloud is rotated so that the normal points upward. Next, an axis-aligned bounding box and an oriented bounding box are set for the cluster's point cloud. Next, the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box onto the XY plane (horizontal plane) are calculated. Next, if the ratio of the orthogonal projection areas of the axis-aligned bounding box and the oriented bounding box exceeds a predetermined threshold (e.g., 1.5), it is determined that axial rotation is necessary. If it is determined that axial rotation is necessary, the coordinate axes are rotated relative to the cluster according to the inclination of the oriented bounding box. When the sub-clustering process is complete, the cluster's point cloud is subjected to an axial rotation in the opposite direction to the initial rotation.

Next, we will explain the sub-clustering process performed by server 2. Figures 14, 15, 16, and 17 are explanatory diagrams showing the status of the sub-clustering process. Note that in Figures 14, 15, 16, and 17, the sub-clusters are drawn in different colors (densities).

In the subclustering process, the conormal clusters obtained in the density-based clustering process are divided into multiple subclusters after the smallest clusters are discarded.

The example shown in Figure 14 is a cluster whose normal direction is the X direction [1,0,0], i.e., a group of conormal clusters with conormal planes parallel to one wall. When subclustering processing is performed on the conormal cluster shown in Figure 14(A), the conormal cluster is divided into subclusters as shown in Figure 14(B).

The example shown in Figure 15 is a group of clusters whose normal direction is the Y direction [0,1,0], i.e., clusters with conormal planes parallel to the other wall surface. When subclustering processing is performed on the conormal cluster shown in Figure 15(A), the conormal cluster is divided into subclusters as shown in Figure 15(B).

The example shown in Figure 16 is a group of clusters whose normal direction is the Z direction [0,0,1], i.e., clusters with conormal planes parallel to the floor surface. When subclustering processing is performed on the conormal cluster shown in Figure 16(A), the conormal cluster is divided into subclusters as shown in Figure 16(B).

The example shown in Figure 17 is a group of clusters whose normal direction is the Z direction [0,0,1], similar to the example shown in Figure 16, but is drawn as viewed from a different direction than the example shown in Figure 16.

Next, the sub-clustering process will be explained. Figures 18, 19, 20, 21, 22, 23, 24, and 25 are explanatory diagrams showing the sub-clustering process. Note that in the examples shown in Figures 18 to 25, the clusters are assumed to be on the XY plane.

Clusters (conormal clusters and individual clusters) are planar and irregular in shape, and often have non-uniform density and holes. Therefore, in this embodiment, clusters are divided into multiple sub-clusters that correspond to the shape of the cluster. This makes it possible to generate a 3D model that faithfully represents the shape of the cluster.

If a cluster is, for example, L-, T-, or U-shaped, simply setting a bounding box (circumscribing rectangle) for that cluster would result in the L-, T-, or U-shape being represented as a simple rectangle, resulting in a significantly insufficient ability to represent the shape. On the other hand, if the model is too faithful to the shape of the cluster and results in a rectangular model with a sawtooth-like outline, this is undesirable for the original purpose, and it is desirable to abstract it appropriately. Therefore, in this embodiment, one of the following modes is used: 2-division mode, N-division mode, or 2-stage division mode.

First, we will explain the two-division mode. As shown in Figure 18, in the two-division mode, two bounding boxes BB1 and BB2 (circumscribing rectangles) are set for cluster C so that cluster C is divided into two. In the two-division mode, the division position that maximizes the amount of area reduction due to division is searched for. Specifically, the division position that maximizes the difference (S0 - S1 - S2) between the area S0 of the bounding box BB0 that corresponds to the entire point cloud of the cluster before division and the sum of the areas S1 and S2 of the first and second bounding boxes BB1 and BB2 that correspond to the point cloud after division is searched for. In the example shown in Figure 18, the division position is searched for in the X direction.

The example shown in Figure 18(A) is when cluster C is L-shaped. In this case, compared to division position a shown in Figure 18(B-1) and division position c shown in Figure 18(B-3), division position b shown in Figure 18(B-2) is optimal as it maximizes the amount of area reduction (S0-S1-S2) due to division. Therefore, cluster C is divided into two sub-clusters SC at division position b. In this case, the L-shape of cluster C is represented by two rectangular planes set corresponding to the two bounding boxes BB1 and BB21 during the rectangularization process.

Here, if a cluster is divided into sub-clusters even if the area reduction due to division is small, the resulting 3D model will be complex and difficult to see. Therefore, if the area reduction due to division (S0-S1-S2) is less than a predetermined threshold, division is not performed. The threshold is set to, for example, 0.25 ^m2 .

In this way, in bisection mode, two bounding boxes (circumscribing rectangles) are set to divide the cluster in half along the coordinate axis, and the division position along the coordinate axis is searched for to maximize the area reduction of the bounding boxes due to the division. This divides the cluster so that the total area of the bounding boxes is reduced, making it possible to generate a 3D model that faithfully represents the shape of the cluster.

In the example shown in Figure 18, division positions are searched for in the X direction, but division positions may also be searched for in the Y direction. Alternatively, division positions may be searched for in both the X and Y directions, and the one that results in the largest amount of area reduction due to division may be used.

Next, we will explain an example in which a single search in one direction (X direction) in two-division mode cannot set an appropriate division position. The example shown in Figure 19(A) is a case in which cluster C is T-shaped.

In this case, the area of the bounding box is reduced for division position a shown in Figure 19 (B-1) and division position c shown in Figure 19 (B-3). However, division position a shown in Figure 19 (B-1), division position b shown in Figure 19 (B-2), and division position c shown in Figure 19 (B-3) are not optimal because they cannot represent the T-shape of the cluster.

Next, we will explain an example in which an appropriate division position can be set by searching twice in one direction (X direction) in the two-division mode. The example shown in Figure 20(A) is a case in which cluster C is T-shaped, similar to the example shown in Figure 19(A).

In this case, first, as shown in Figure 20(B), the point cloud of cluster C is divided a first time, and bounding boxes BB1 and BB2 are set. Next, as shown in Figures 20(C-1) and (C-2), the point cloud of bounding box BB2 is divided a second time, and bounding boxes BB21 and BB22 are set. As a result, as shown in Figure 20(D), three bounding boxes BB1, BB21, and BB22 are set for the point cloud of cluster C, and the point cloud of cluster C is divided into three sub-clusters SC. In this case, the T-shape of cluster C is represented by three rectangular planes set corresponding to the three bounding boxes BB1, BB21, and BB22 during the rectangularization process.

In this way, by repeating the search and division in one direction twice, a T-shaped cluster is expressed. In the example shown in Figure 20, the search in the X direction is performed twice and division is performed twice. On the other hand, the second search may be performed in a different direction (Y direction) from the first search. Alternatively, the first search may be performed in the Y direction and the second search in the X direction.

Next, we will explain an example where an appropriate split position cannot be set in the two-split mode. The example shown in Figure 21(A) is when cluster C is U-shaped.

In this case, whether it is division position a shown in Figure 21 (B-1), division position b shown in Figure 21 (B-2), or division position c shown in Figure 21 (B-3), the area of bounding boxes BB1 and BB2 is not reduced, and the U-shape of cluster C cannot be represented, so it is not optimal. Furthermore, the results are the same even if the search direction is changed.

Next, we will explain the N-division mode. As shown in Figure 22, in the N-division mode, a search box SB (such as SB1 in the figure) is set for cluster C, and a division position that allows for division into two is searched for within that search box SB. If no division position is found within the search box SB, the width of the search box SB (search length) is gradually widened to L, 2L, and 3L. Furthermore, if a division position that allows for division into two is found within the search box SB, a bounding box BB (such as BB1 in the figure) is set so that the point cloud is divided at that division position. Next, a search for a division position using the search box SB is performed on the remaining points of cluster C, and this process is repeated throughout cluster C.

The example shown in Figure 22(A) is a case where cluster C is U-shaped. In this case, first, as shown in Figure 22(B-1), a search box SB1 with a width of search length L is set, and it is determined whether there is a division position by dividing the area into two, and it is determined that there is no division position. Next, as shown in Figure 22(B-2), a search box SB2 with a width of search length 2L is set, and it is determined whether there is a division position by dividing the area into two, and it is determined that there is no division position. Next, as shown in Figure 22(B-3), a search box SB3 with a width of search length 3L is set, and it is determined whether there is a division position by dividing the area into two, and it is determined that there is a division position. Therefore, as shown in Figure 22(C), the point cloud included in search box SB3 is divided into two. This results in setting a bounding box BB1.

Next, as shown in Figure 23(A), a division position is searched for within the remaining portion of cluster C after removing the bounding box BB1. First, as shown in Figure 23(B-1), a search box SB1 with a width of search length L is set, and a determination is made as to whether or not there is a division position resulting from bisection. It is determined that there is no division position. Next, as shown in Figure 23(B-2), a search box SB2 with a width of search length 2L is set, and a determination is made as to whether or not there is a division position resulting from bisection. It is also determined that there is no division position. Next, as shown in Figure 23(B-3), a search box SB3 with a width of search length 3L is set, and a determination is made as to whether or not there is a division position resulting from bisection. It is determined that there is a division position. Therefore, as shown in Figure 23(C), the point cloud included in search box SB3 is bisected. This results in the setting of bounding box BB21.

Next, as shown in Figure 24(A), a division position is searched for in the portion remaining after removing bounding boxes BB1 and BB21 from cluster C. First, as shown in Figure 24(B-1), a search box SB1 with a width of search length L is set, and it is determined whether there is a division position resulting from bisection. It is determined that there is no division position. Next, as shown in Figure 24(B-2), a search box SB2 with a width of search length 2L is set, and it is determined whether there is a division position resulting from bisection. It is also determined that there is no division position. Next, as shown in Figure 24(B-3), a search box SB3 with a width of search length 3L is set, and it is determined whether there is a division position resulting from bisection. It is also determined that there is no division position. At this point, since there are no points beyond search box SB3 in the X direction, it is determined that the search has ended. Therefore, as shown in Figure 24(C), a bounding box BB22 is set for the portion remaining after removing bounding boxes BB1 and BB21 from cluster C. As a result, three bounding boxes BB1, BB21, and BB22 are set for cluster C, and cluster C is divided into three sub-clusters SC. In this case, the U-shape of cluster C is expressed by three rectangular planes set corresponding to the three bounding boxes BB1, BB21, and BB22 in the rectangularization process.

In this way, in N-division mode, the width (search length) of the search box SB (such as SB1 in the figure) is gradually increased by the expansion width L, and a bisection process is attempted; if possible, that is, when a division position is found within the search box SB, a bisection process is performed at that division position and one bounding box BB (such as BB1 in the figure) is set. The same process is then repeated for the remaining point clouds, and this process is repeated until a bounding box BB is set for the entire point cloud of cluster C.

In addition, in N-division mode, a search box SB is set when searching for a division position, and the division position is searched for while gradually widening the width (search length) of the search box SB to L, 2L, and 3L. Therefore, only one division position can be set for each set search box SB. As a result, it may not be possible to express shapes smaller than the width of the search box SB, but this prevents the cluster from being randomly subdivided, allowing the shape of the cluster to be expressed at an appropriate level of abstraction. The expansion width L when gradually widening the width of the search box SB can be set appropriately depending on the restoration location, etc., but may be set to 0.5 m, for example.

Note that the examples shown in Figures 22, 23, and 24 are for clusters that are U-shaped, but in N-division mode, even for clusters that are not U-shaped, multiple bounding boxes (sub-clusters) can be appropriately set to divide the cluster.

Next, we will explain the two-stage division mode. In the two-stage division mode, the cluster is divided into two stages with different search directions. That is, the first stage of division divides the cluster in a first direction (X or Y direction), and the second stage of division divides the cluster in a second direction different from the first direction.

In the example shown in Figure 25(A), cluster C has a complex shape. In this case, as shown in Figures 25(B-1) and 25(C-1), a single division in only one direction, either the X or Y direction, does not allow for the appropriate bounding box to be set for the cluster. On the other hand, as shown in Figures 25(B-2) and 25(C-2), when a second division is performed in a direction different from the first division, cluster C is appropriately divided into multiple sub-clusters SC. Here, in the example shown in Figures 25(B-1) and (B-2), division is first performed in the X direction, and then in the Y direction. Also, as shown in Figures 25(C-1) and (C-2), division is first performed in the Y direction, and then in the X direction.

Here, in the two-stage division mode, as in the two-stage division mode, a threshold value for the amount of area reduction used to determine whether or not division is possible is set, and if the amount of area reduction due to division is less than the threshold value, division is not performed. Furthermore, if the threshold value for the second stage division is set smaller than that for the first stage division, division becomes more likely to occur. Therefore, it is preferable to set separate threshold values for determining whether or not division is possible for the first and second stages. For example, the threshold value may be set to 0.25 ^m2 for the first division and 0.05 ^m2 for the second division.

In this embodiment, the sub-clustering process that divides clusters (conormal clusters or individual clusters) obtained by the conormal clustering process or the individual clustering process into multiple sub-clusters can be performed in two-division mode, N-division mode, or two-stage division mode, but these processes are not limited to the application of dividing clusters into sub-clusters (sub-clustering).

That is, a clustering process may be performed in which a plane (a set of points distributed on a plane at a specified density) is extracted from the point cloud of the target location, and that plane is then divided into clusters using two-division mode, N-division mode, or two-stage division mode. This type of clustering process can be used, for example, to create a drawing (front view, plan view, etc.) of a single object to be restored at the target location from the point cloud of that object. Specifically, it may be used, for example, to create a rough sketch of the exterior of a building.

Next, the rectangularization process (ST103 in Figure 3) performed by server 2 will be described. Figure 26 is a flow diagram showing the steps of the rectangularization process. Note that this process is performed on a group basis of conormal clusters and individual clusters generated in the clustering process. Figure 27 is an explanatory diagram showing the status of the rectangularization process. Figures 28 and 29 are explanatory diagrams showing the three-dimensional model (rectangle model) generated in the rectangularization process.

The rectangularization process first converts the clusters obtained in the clustering process (see ST102 in Figure 3) into rectangular planes (initial rectangularization process) (ST501). Here, the process is performed on individual clusters and their sub-clusters, and conormal clusters and their sub-clusters.

In the initial rectangularization process, the cluster is first rotated around the origin so that the normal points upward at [0,0,1]. This causes the rotated cluster to extend horizontally, simplifying subsequent processing. Next, the bounding box and center of gravity of the cluster are obtained, and the z coordinate of the bounding box is replaced with the z coordinate of the center of gravity. This results in a horizontal rectangular plane. Next, the rectangular plane is rotated in the opposite direction to the initial rotation.

In the example shown in Figure 27, similar to the example shown in Figure 11, the object to be restored is a cylinder with two planes joined in an L-shape. In the initial rectangularization process, as shown in Figure 27(A), clusters extracted from the point cloud of the object to be restored are converted into rectangular planes. Next, as shown below, a process is performed to adjust the position and shape of the rectangular planes. In the example shown in Figure 27(A), the end of one rectangular plane protrudes from the other rectangular plane. In this case, by cutting off the protruding portion of one rectangular plane, the shape of the rectangular planes is adjusted so that two adjacent rectangular planes share a side, as shown in Figure 27(B).

As shown in Figure 26, alignment processing is first performed to adjust the position and shape of rectangular planes (ST502). In alignment processing, rectangular planes belonging to the same group are targeted, and multiple rectangular planes that should be located on one plane are aligned on the same plane.

Next, a first inter-group snapping process is performed (ST503). In the first inter-group snapping process, two rectangular planes from different groups are targeted, and if there is a gap between the two rectangular planes, the gap is filled in, adjusting the shape of the rectangular planes so that the two rectangular planes share a side.

Next, a second inter-group snapping process is performed (ST504). In the second inter-group snapping process, two rectangular planes from different groups are targeted, and if there is a protrusion between the two rectangular planes, the protruding portion is removed, thereby adjusting the shape of the rectangular planes so that the two rectangular planes share a side.

Next, intra-group snap processing is performed (ST505). In intra-group snap processing, rectangular planes belonging to the same group are targeted, and the shape of the rectangular planes is adjusted by filling in the gaps between the two rectangular planes and cutting out the portion of one rectangular plane that extends beyond the other.

Note that the groups in the alignment process, the first and second inter-group snapping processes, and the intra-group snapping process represent groups of rectangular models (rectangle groups) when clusters are converted into rectangular planes by the initial rectangularization process, and correspond to groups of clusters (conormal clusters and individual clusters).

When sub-clustering is not performed, rectangularization is performed on the individual clusters and conormal clusters obtained by the clustering process, as shown in Figure 28. On the other hand, when sub-clustering is performed, rectangularization is performed on the individual clusters and their sub-clusters, and the conormal clusters and their sub-clusters, as shown in Figure 29.

As shown in Figure 28, when sub-clustering processing is not performed, the fidelity of the objects in the restored location is low, and for example, a U-shaped desk placed in the center of a room is not sufficiently represented. On the other hand, as shown in Figure 29, when sub-clustering processing is performed, the fidelity of the objects in the restored location is high, and for example, a U-shaped desk placed in the center of a room is sufficiently represented.

Next, we will explain the alignment process performed by server 2. Figure 30 is an explanatory diagram showing the alignment process.

The alignment process involves aligning adjacent rectangular planes that belong to the same group (conormal rectangle group). The rectangular planes of each cluster obtained by the initial rectangle creation process may not be located on the same plane, even though they actually belong to a single plane (for example, a wall, floor, or ceiling). If this is left unchecked, they will not be represented as a single line when drawn on a floor plan (2D layout diagram). This is why alignment processing is performed.

When alignment processing is performed, for example, the unnatural state of having a step between rectangular planes that make up indoor wall surfaces, i.e., rectangular planes that should be located on the same plane, is eliminated. In the example shown in Figure 30(A), there is a step between rectangular planes RP1 and RP2. When alignment processing is performed, as shown in Figure 30(B), the step between rectangular planes RP1 and RP2 disappears, and rectangular planes RP1 and RP2 are aligned on the same plane.

In the alignment process, first, the rectangular planes are rotated so that their normals point upward. Next, the proximity between rectangular planes is determined, and adjacent planes are grouped together. Next, the z coordinates of the rectangular planes in each group are weighted averaged, and the z coordinate of the rectangular plane is replaced with this average value. At this time, the weighting is set according to the number of points that make up the original cluster. Next, the rectangular planes are rotated in the opposite direction to the initial rotation. This results in rectangular planes belonging to the same group being aligned on the same plane.

Next, we will explain the snap processing between the first and second groups performed on server 2. Figure 31 is an explanatory diagram showing the status of the snap processing between the first and second groups.

The first inter-group snapping process targets two rectangular planes from different groups (conormal rectangle groups), and if there is a gap between the two rectangular planes, it adjusts the shape of the rectangular planes so that the two planes share an edge. Specifically, it joins one edge (the adjacent edge) of the two rectangular planes to the other face (the adjacent face) to eliminate the gap between the two rectangular planes.

When the first inter-group snapping process is performed, for example, the unnatural state of having gaps between the rectangular planes representing each face (top face, side face, etc.) of a box-shaped object is eliminated. In the example shown in Figure 31 (A-1), there are gaps between the rectangular planes RP1, RP2, and RP3. When the first inter-group snapping process is performed, as shown in Figure 31 (A-2), the gaps between the rectangular planes RP1, RP2, and RP3 disappear, and the shape is adjusted so that the rectangular planes RP1, RP2, and RP3 share a common side.

In the first inter-group snapping process, the rectangular planes are first rotated so that their normals point upward. Next, a proximity determination is made between the rectangular planes. If the rectangular planes intersect, that is, if one of the rectangular planes protrudes beyond the other, this process is skipped and the process moves on to the next, second inter-group snapping process. If there is a gap between the rectangular planes, the edge of one rectangular plane (the adjacent edge) is extended to join with the face of the other rectangular plane (the adjacent face). At this time, the adjacent face is extended if necessary. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

The second inter-group snapping process targets two rectangular planes from different groups (conormal rectangle groups), and if one of the two rectangular planes protrudes beyond the other, a process is performed to adjust the shape of the rectangular plane so that the two rectangular planes share an edge. Specifically, a process is performed to remove the protruding portion. In this case, if the amount of protrusion is large and the protruding edge is not close to the surface of the other side, the second inter-group snapping process is not performed.

When the second inter-group snapping process is performed, for example, the unnatural state in which one of the rectangular planes representing each face (top face, side face, etc.) of a box-shaped object protrudes from the other is eliminated. In the example shown in Figure 31 (B-1), rectangular plane RP1 protrudes from rectangular plane RP2. When the second inter-group snapping process is performed, the protruding portion of rectangular plane RP1 is removed, and the shape is adjusted so that rectangular plane RP1 and rectangular plane RP2 share a side, as shown in Figure 31 (B-2).

In the second inter-group snapping process, first, the rectangular planes are rotated so that their normals point upward. Next, a proximity determination is made between the rectangular planes. If the rectangular planes do not intersect, that is, if one of the rectangular planes does not protrude beyond the other, this process is skipped. Next, the protruding portion of one of the rectangular planes (the adjacent side) is cut off. Next, the rectangular planes are rotated in the opposite direction to the initial rotation.

Next, we will explain the intra-group snap processing performed on Server 2. Figure 32 is an explanatory diagram showing the status of intra-group snap processing.

Intra-group snapping processes target two adjacent rectangular planes that belong to the same group (conormal rectangle group) and adjust the shape of the rectangular planes.

In the example shown in Figure 32 (A-1), of the two rectangular planes RP1 and RP2 that belong to the same group, one side of the rectangular plane RP1 slightly extends into the other rectangular plane RP2. In this case, as shown in Figure 32 (A-2), a process is performed to remove the extending portion of one rectangular plane RP1, correcting the sides of the two rectangular planes RP1 and RP2 to be in contact.

In the example shown in Figure 32 (B-1), of the rectangular planes RP1, RP2, and RP3 that belong to the same group, there is a small gap between the rectangular plane RP1 and the rectangular planes RP2 and RP3. In this case, as shown in Figure 32 (B-2), processing is performed to extend one of the rectangular planes, RP1, to eliminate the gap between the rectangular planes, thereby correcting the edges of the three rectangular planes RP1, RP2, and RP3 to be in contact.

Next, we will explain the floor plan generation process performed by server 2. Figure 33 is an explanatory diagram showing the status of the floor plan generation process.

On server 2, the floor and ceiling portions, i.e., the rectangular planes representing the floor and ceiling, are deleted from the 3D model (rectangular model), and then the 3D model is orthogonally projected onto the XY plane (horizontal plane) to generate a floor plan (2D layout drawing) (floor plan generation process). Note that the floor and ceiling portions in the 3D model can be determined based on the z coordinate.

As shown in Figure 33(A), the 3D model is composed of multiple rectangular planes. Note that in the 3D model shown in Figure 33(A), the rectangular planes that make up the ceiling have been removed so that the interior of the room where the model is restored can be seen. As shown in Figure 33(B), the floor plan depicts the walls of the room where the model is restored, as well as furniture such as desks, chairs, and shelves arranged within the room.

In this way, in this embodiment, the majority of objects in the restoration location in the 3D model are represented by rectangular planes aligned along horizontal coordinate axes (X-axis and Y-axis), and the shapes of the objects to be restored are represented fairly faithfully, making it possible to obtain a reasonably abstract, orderly, and easy-to-read plan view (2D layout diagram).

Next, we will explain the dimension line insertion process performed on server 2. Figure 34 is an explanatory diagram showing the status of the dimension line insertion process.

Server 2 extracts a representative plane from the three-dimensional model and inserts dimension lines related to that representative plane into the floor plan (dimension line insertion process). At this time, in the three-dimensional model (rectangular model), for example, two planes that are spaced apart are extracted as representative planes. For example, in the case of a room that is rectangular in plan view, the walls of the room are extracted. Furthermore, dimension lines representing the inter-plane distance between two opposing planes are inserted into the floor plan as dimension lines related to the representative planes. In the example shown in Figure 34, dimension lines representing the inter-plane distance between two pairs of wall surfaces that face each other in the directions of the coordinate axes (X-axis and Y-axis) in a room that is rectangular in plan view are inserted into the floor plan.

In this case, a combination of a predetermined number of planes may be selected starting from the combination with the largest inter-plane distance. Alternatively, a combination of planes with an area equal to or greater than a predetermined threshold may be selected. The threshold may be set based on the overall size of the restoration location. This makes it possible to avoid cluttering the plan view by inserting numerous dimension lines into the plan view.

Next, we will explain the drawing display process performed by the server 2. Figure 35 is an explanatory diagram showing the viewing screens 101 and 201 displayed on the user terminal 1.

On the user terminal 1, viewing screens 101 and 201 are displayed on the display 13 based on control by the server 2. Figure 35 (A) shows the viewing screen 101 in plan view display mode, and Figure 35 (B) shows the viewing screen 201 in measurement display mode.

As shown in Figure 35 (A), the viewing screen 101 in plan view display mode displays a plan view 111 (two-dimensional layout drawing) generated by the drawing generation process. In the example shown in Figure 35 (A), a plan view 111 with dimension lines inserted is displayed. A CAD mode may also be provided to allow editing of the plan view 111, such as erasing unnecessary lines or adding necessary lines.

In measurement display mode, the viewing screen 201 displays a drawing that visualizes a three-dimensional model. In the example shown in Figure 35(B), a three-dimensional layout drawing 211 is displayed that visualizes a three-dimensional model (rectangular model) made up of multiple rectangular planes as viewed from a specified viewpoint.

The viewing screen 201 also displays measurement results related to the specified measurement object in response to a user operation to specify the measurement object on the three-dimensional layout diagram 211. In the example shown in Figure 35(B), when the user specifies two wall surfaces to be measured using a predetermined operation (e.g., a drag operation), a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed.

In the example shown in Figure 35(B), a three-dimensional layout drawing 211 is displayed as a layout drawing that visualizes a three-dimensional model, but a two-dimensional layout drawing (floor plan) may also be displayed. Furthermore, on the viewing screen 201, when the user selects a plane to be measured (e.g., a wall surface of a room), the selected plane may be highlighted on the layout drawing. For example, the drawing color of the selected plane on the layout drawing may change.

Furthermore, in the three-dimensional layout diagram 211, rectangular planes may be drawn in different colors according to their normal direction. For example, rectangular planes whose normal direction is the X direction may be drawn in red, rectangular planes whose normal direction is the Y direction may be drawn in green, and of rectangular planes whose normal direction is the Z direction, rectangular planes representing the floor may be drawn in gray, and rectangular planes representing surfaces other than the floor may be drawn in blue.

In response to user operations on the user terminal 1, the server 2 performs a measurement and display process for the inter-surface distance and a measurement and display process for the parallelism, and a display box 212 for the inter-surface distance and a display box 213 for the parallelism are displayed on the viewing screen 201 of the user terminal 1.

In the measurement and display process for inter-surface distances, when the user selects two planes on the three-dimensional layout diagram 211, the inter-surface distance between the two selected planes is measured based on the three-dimensional model, and the inter-surface distance is displayed in the display box 212 on the viewing screen 201.

At this time, the user simply designates one point on each of the two planes (e.g., the walls of a room) for which the inter-plane distance will be measured. The user simply selects one point on each plane as appropriate, and the direction connecting the two points designated by the user may deviate from the normal to the two planes. In this embodiment, the parallelism of two planes arranged parallel to each other at the reconstruction location is ensured in the 3D model using conormal clustering processing. Therefore, once the two planes to be measured are selected by the user, the inter-plane distance between the two planes, i.e., the separation distance in the normal direction of the two planes, can be easily measured.

In this embodiment, the inter-plane distance between two planes is measured, but the distance between one plane and a line parallel to that plane may also be measured. For example, the distance between a wall of a room and the edge (straight line) of a piece of furniture, such as a desk, placed parallel to that wall may also be measured.

In addition, in the parallelism measurement and display process, when the user selects two planes on the 3D layout drawing, the parallelism of the two selected planes is calculated based on the point cloud data, and the parallelism of the two planes is displayed in the display box 213 on the viewing screen 201. In actual construction, there are cases where two planes that should be parallel are not constructed parallel; in such cases, the user can easily check the parallelism of the two planes by selecting the two planes to be measured.

In this embodiment, when the user selects points on two parallel planes, the inter-plane distance between the two planes is measured and displayed on the viewing screen 201. However, when the user selects two points, the distance between the two points, i.e., the length of the straight line connecting the two points, may also be measured and displayed on the viewing screen 201.

In this embodiment, a 3D model is obtained in which the parallelism of parallel planes at the reconstruction location is ensured. This not only makes it easy to measure inter-surface distances, but also makes it easy to combine 3D models. As a specific example, consider the case where partially obtained 3D models of a square building are combined to create a 3D model of the entire building. In a 3D model generated by the 3D model generation device of the present invention, the coordinate axes are set in accordance with the building's wall surfaces, and the constituent planes are limited to specific directions. Therefore, when combining, only parallel translation and three types of rotation (90-degree, 180-degree, and -90-degree rotations around the Z axis) need to be considered, making it easier to combine models compared to models in which the orientation of the constituent planes is not limited.

This embodiment is expected to be widely used as a fundamental technology for creating digital twins of real spaces. For example, the technology disclosed herein can be used to generate 3D models of factories, warehouses, stores, etc., and for a variety of applications, including equipment and inventory management, renovations, layout change planning, and acquiring spatial data for simulations.

As mentioned above, the embodiments have been described as examples of the technology disclosed in this application. However, the technology in this disclosure is not limited to these, and can also be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made. Furthermore, it is possible to combine the components described in the above embodiments to create new embodiments.

The 3D model generation device, 3D model generation method, and 3D model generation program according to the present invention have the effect of generating 3D models that allow for easy measurement of inter-surface distances and the combination of multiple 3D models, and are useful as 3D model generation devices, 3D model generation methods, and 3D model generation programs that use a processor to execute a process for generating a 3D model based on point cloud data acquired by performing a 3D reconstruction process on a target location.

1: User terminal 2: Server 12: Camera 13: Display 14: Input device 16: Processor 51: Partitions 52, 53: Rectangular plane 61: Handrails 62, 63: Rectangular plane 101: Viewing screen 111: Floor plan (2D layout diagram)
201: Viewing screen 211: 3D layout diagram 212: Display box for inter-surface distance 213: Display box for parallelism BB, BB0, BB1, BB2, BB21, BB22: Bounding box C: Cluster L: Enlargement width RP1, RP2, RP3: Rectangular plane SB, SB1, SB2, SB3: Search box SC: Sub-cluster θ: Normal angle φ: Rotation angle

Claims (5)

A three-dimensional model generation device that uses a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location,
The processor:
When setting coordinate axes along the four walls of a target location based on the distribution of normal directions of the entire point cloud of the target location that forms a rectangle in a plan view,
a radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates is obtained as a normal angle, the normal angle of each point is multiplied by four to obtain a quadrupled value of each point, an optimal value of the quadrupled value of each point is estimated using an estimation method that excludes the influence of outliers, and the optimal value of the quadrupled value is multiplied by one-fourth to obtain a rotation angle that defines the direction of the coordinate axes;
Rotating the provisionally set coordinate axes by the rotation angle to set the regular coordinate axes;
execute a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal directions coincide with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range as a conormal cluster;
performing a rectangularization process to convert the conormal cluster into a rectangular plane;
A three-dimensional model generating device that generates the three-dimensional model in which a target location is expressed by a set of a plurality of the rectangular planes. A three-dimensional model generation device that uses a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location,
The processor:
Set a coordinate axis along the wall of the target location for the point cloud of the target location,
executes a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal directions coincide with a predetermined direction based on the coordinate axes within a predetermined tolerance range, as a conormal cluster;
Before the conormal clustering process,
An individual clustering process is performed to extract a plane in any direction from the point cloud of the target location as an individual cluster;
performing a rectangularization process on the individual clusters and the conormal clusters;
A three-dimensional model generating device that generates the three-dimensional model in which a target location is represented by a set of a plurality of rectangular planes . The processor:
Dividing the conormal cluster into a plurality of sub-clusters corresponding to the shape of the conormal cluster;
3. The three-dimensional model generating device according to claim 1 , wherein the conormal clusters and the sub-clusters are transformed into the rectangular plane. A three-dimensional model generation method for generating a three-dimensional model based on point cloud data acquired by performing three-dimensional reconstruction processing on a target location, the method comprising:
When setting coordinate axes along the four walls of a target location based on the distribution of normal directions of the entire point cloud of the target location that forms a rectangle in a plan view,
a radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates is obtained as a normal angle, the normal angle of each point is multiplied by four to obtain a quadrupled value of each point, an optimal value of the quadrupled value of each point is estimated using an estimation method that excludes the influence of outliers, and the optimal value of the quadrupled value is multiplied by one-fourth to obtain a rotation angle that defines the direction of the coordinate axes;
Rotating the provisionally set coordinate axes by the rotation angle to set the regular coordinate axes;
execute a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal directions coincide with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range as a conormal cluster;
performing a rectangularization process to convert the conormal cluster into a rectangular plane;
A three-dimensional model generation method, characterized in that the three-dimensional model is generated by expressing a target location as a set of a plurality of the rectangular planes. A three-dimensional model generation program that causes a processor to execute a process of generating a three-dimensional model based on point cloud data acquired by performing a three-dimensional reconstruction process on a target location,
When setting coordinate axes along the four walls of a target location based on the distribution of normal directions of the entire point cloud of the target location that forms a rectangle in a plan view,
a radial angle when the normal direction of each point constituting the point cloud is expressed in cylindrical coordinates is obtained as a normal angle, the normal angle of each point is multiplied by four to obtain a quadrupled value of each point, an optimal value of the quadrupled value of each point is estimated using an estimation method that excludes the influence of outliers, and the optimal value of the quadrupled value is multiplied by one-fourth to obtain a rotation angle that defines the direction of the coordinate axes;
Rotating the provisionally set coordinate axes by the rotation angle to set the regular coordinate axes;
execute a conormal clustering process to extract, from the point cloud of the target location, a set of points whose normal directions coincide with a predetermined direction based on the normal coordinate axis within a predetermined tolerance range as a conormal cluster;
performing a rectangularization process to convert the conormal cluster into a rectangular plane;
A three-dimensional model generation program, characterized by generating the three-dimensional model in which a target location is represented by a set of a plurality of the rectangular planes.