JP7613690B2 - 3D point cloud processing device, 3D point cloud processing method, and 3D point cloud processing program - Google Patents

Description

The disclosed technology relates to a 3D point cloud processing device, a 3D point cloud processing method, and a 3D point cloud processing program.

Non-Patent Document 1 discloses a technology that uses LiDAR (Light Detection and Ranging) and stereo images to output a depth image by upsampling a sparse point cloud obtained by LiDAR based on the disparity estimation results of the stereo images.

Non-Patent Document 2 discloses a technology that uses supervised deep learning to output a depth image by upsampling the LiDAR point cloud from LiDAR and stereo images.

Non-Patent Document 3 discloses a technology that uses semi-supervised deep learning to output a depth image by upsampling the LiDAR point cloud from LiDAR and stereo images. The technology disclosed in Non-Patent Document 3 makes it possible to remove the influence of projection errors and perform inference even if they exist.

Maddern, Will, and Paul Newman. "Real-time probabilistic fusion of sparse 3D LIDAR and dense stereo." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016. Park, Kihong, Seungryong Kim, and Kwanghoon Sohn. "High-precision depth estimation with the 3d lidar and stereo fusion." 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2018. Cheng, Xuelian, et al. "Noise-aware unsupervised deep lidar-stereo fusion." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.

The problem with Non-Patent Document 1 is that the projected LiDAR points are processed assuming that they are the correct depth values, and do not take into account cases where projection errors may occur.

The problem with Non-Patent Document 2, like Non-Patent Document 1, is that it does not take into account that projection errors exist in the data, and that high-density depth images are required for learning, but it is not easy to obtain such data.

The problem with Non-Patent Document 3 is that a large number of correctly registered pairs of LiDAR measurements and stereo images are required for learning, and obtaining such data is not easy.

As mentioned above, the problem with Non-Patent Documents 1 and 2 is that they "do not take into account the projection error of LiDAR data." Also, the problem with Non-Patent Document 3 is that it "requires a large amount of training data."

Furthermore, all of Non-Patent Documents 1 to 3 use a process in which "upsampled data is created based on the results of disparity estimation." Disparity estimation using stereo reduces distance measurement accuracy over long distances. In contrast, LiDAR has high distance measurement accuracy even over long distances, but this characteristic is not utilized.

The disclosed technology has been made in consideration of the above points, and aims to provide a 3D point cloud processing device, a 3D point cloud processing method, and a 3D point cloud processing program that can accurately upsample 3D point clouds obtained by measurement.

A first aspect of the present disclosure is a three-dimensional point cloud processing device, which includes an input processing unit that receives a first image and a second image, the relationship of which at least is determined in advance, and a three-dimensional point cloud on the surface of an object, the relationship of which at least is determined in advance between the shooting position and the measurement position, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, and a neighborhood selection unit that selects, for each pixel position on the first image, based on the pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, from depth values of three-dimensional points corresponding to multiple neighboring pixel positions, so that the consistency between the first image and the second image is high.

A second aspect of the present disclosure is a three-dimensional point cloud processing method, in which an input processing unit receives a first image and a second image for which at least the relationship between the shooting positions has been determined in advance, and a three-dimensional point cloud on the surface of an object for which at least the relationship between the shooting positions and the measurement positions has been determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, and a neighborhood selection unit selects, for each pixel position on the first image, based on the pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, the depth value from the depth values of three-dimensional points corresponding to multiple neighboring pixel positions, so that the consistency between the first image and the second image is high.

The third aspect of the present disclosure is a 3D point cloud processing program that causes a computer to function as the 3D point cloud processing device of the first aspect.

The disclosed technology makes it possible to upsample 3D point clouds obtained through measurement with high precision.

FIG. 2 is a schematic block diagram of an example of a computer that functions as the three-dimensional point cloud processing device of the present embodiment. A figure showing an example of measurement points measured by a LiDAR sensor and scenes captured by a first camera and a second camera. 1 is a block diagram showing a configuration of a three-dimensional point cloud processing apparatus according to an embodiment of the present invention; FIG. 1A is a diagram showing an example of a first image, FIG. 1B is a diagram showing an example of a second image, and FIG. 1C is a diagram showing an example of the result of projecting a three-dimensional point cloud onto the first image. FIG. 13 is a diagram for explaining a method for extracting a neighborhood set. 4 is a flowchart showing a three-dimensional point cloud processing routine of the three-dimensional point cloud processing device of the present embodiment.

Below, an example of an embodiment of the disclosed technology will be described with reference to the drawings. Note that the same reference symbols are used for identical or equivalent components and parts in each drawing. Also, the dimensional ratios in the drawings have been exaggerated for the convenience of explanation and may differ from the actual ratios.

<Outline of this embodiment>
In this embodiment, a sparse 3D point cloud obtained by LiDAR measurement is upsampled using a first image and a second image captured by a camera as clues.

Specifically, the LiDAR projection is not assumed to be correct, and the optimal depth value is selected from the 3D points projected around the pixel based on an energy function consisting of a cost function and a smoothing term. This method does not require training data and correctly upsamples the 3D point cloud measured by LiDAR even if there is an error in the LiDAR projection information.

Furthermore, this embodiment is characterized by upsampling by assigning the 3D point cloud measured by LiDAR to pixels, so that measurement accuracy is maintained over long distances, unlike methods that perform disparity estimation.

As such, this embodiment is characterized by the fact that it does not use learning data, that it is possible to perform correct upsampling even when there is a projection error between the LiDAR and the camera due to occlusion caused by vehicle movement or differences in sensor position, and that it is possible to suppress loss of accuracy over long distances by upsampling the 3D point cloud measured by the LiDAR without estimating distance based on image parallax.

<Configuration of the 3D point cloud processing device according to this embodiment>
FIG. 1 is a block diagram showing the hardware configuration of a three-dimensional point cloud processing apparatus 10 according to the present embodiment.

As shown in FIG. 1, the 3D point cloud processing device 10 has a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM (Random Access Memory) 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I/F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

The CPU 11 is a central processing unit that executes various programs and controls each part. That is, the CPU 11 reads out a program from the ROM 12 or storage 14, and executes the program using the RAM 13 as a working area. The CPU 11 controls each of the above components and performs various calculation processes according to the program stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a three-dimensional point cloud processing program for upsampling a three-dimensional point cloud. The three-dimensional point cloud processing program may be one program, or may be a group of programs composed of multiple programs or modules.

The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is composed of a HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

The input unit 15 includes a pointing device such as a mouse, and a keyboard, and is used to perform various inputs including a first image and a second image for which the relationship between at least the shooting positions is determined in advance, and a three-dimensional point cloud P on the surface of an object for which the relationship between at least the shooting positions and the measurement positions is determined in advance. For example, the three-dimensional point cloud P measured by the LiDAR sensor 50, the first image taken by the first camera 52 (see FIG. 4(A)), and the second image taken by the second camera 54 (see FIG. 4(B)) as shown in FIG. 2 are input to the input unit 15. The relationship between the shooting positions of the first camera 52 and the second camera 54 is determined in advance, and the relationship between the shooting position of the first camera 52 and the measurement position of the LiDAR sensor 50 is determined in advance.

The first image is a distortion-corrected RGB or grayscale image, and the second image is a distortion-corrected RGB or grayscale image. The three-dimensional point cloud P is a collection of three-dimensional points measured by the LiDAR sensor 50. Each three-dimensional point is a three-dimensional vector, and when the three-dimensional point cloud P contains N three-dimensional points, the three-dimensional point cloud P becomes a collection of three-dimensional vectors with N elements.

The input unit 15 also receives input of an internal parameter K_1 of the first camera 52, an internal parameter K_2 of the second camera 54, a rotation matrix R_C between the first camera 52 and the second camera 54, a translation vector T_C between the first camera 52 and the second camera 54, a projection matrix R_L between the first camera 52 and the LiDAR sensor 50, and a translation vector T_L between the first camera 52 and the LiDAR sensor 50.

The internal parameters K_1 and K_2 of the first camera 52 and the second camera 54 are 3x3 camera internal parameter matrices. The rotation matrix R_C between the first camera 52 and the second camera 54 is a 3x3 rotation matrix. The translation vector T_C between the first camera 52 and the second camera 54 is a three-dimensional vector. The projection matrix R_L between the first camera 52 and the LiDAR sensor 50 is a 3x3 rotation matrix. The translation vector T_L between the first camera 52 and the LiDAR sensor 50 is a three-dimensional vector.

The display unit 16 is, for example, a liquid crystal display, and displays various information including the results of upsampling the three-dimensional point cloud P measured by the LiDAR sensor 50. The display unit 16 may employ a touch panel system and function as the input unit 15.

The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

Next, the functional configuration of the 3D point cloud processing device 10 will be described. Figure 3 is a block diagram showing an example of the functional configuration of the 3D point cloud processing device 10.

As shown in FIG. 3, the 3D point cloud processing device 10 functionally comprises an input processing unit 20, a neighborhood extraction unit 22, a neighborhood selection unit 24, and a smoothing unit 26.

The input processing unit 20 projects each of the three-dimensional points of the three-dimensional point group P onto the first image based on the first image, the second image, and the three-dimensional point group P received by the input unit 15, and determines the pixel position on the first image corresponding to each three-dimensional point (see FIG. 4(C)). FIG. 4(C) shows an example in which one of the three-dimensional points on the surface of an object represented by dots is erroneously projected onto a pixel position within an area representing a white object on the first image.

Specifically, each 3D point of the 3D point group P is projected onto the first image, and the 3D points projected outside the area of the first image are removed from the 3D point group P. A set Q_1 is obtained that is made up of elements that are combinations of pixel positions and depth values on the first image that correspond to each 3D point of the 3D point group P after the removal. The points indicated by each element of set Q_1 are projected onto the second image, and the elements indicating points projected outside the area of the second image are removed from set Q_1, updating it to set Q_1 after removal.

For example, for each of the 3D points included in the 3D point group P, the projection matrix R_L and translation vector T_L are applied according to the following formula to obtain the 3D point group P_1. The 3D point group P_1 is a collection of N 3D vectors.

P_1=R_L P+T_L

The three-dimensional point group P_1 is projected onto the first image of the first camera 52 using the internal parameter K_1, and those projected outside the region of the first image are removed to obtain a set Q_1 of elements consisting of (x coordinate in the first image, y coordinate in the first image, depth value of the three-dimensional point). For example, (x, y, d) calculated according to the following formula for each three-dimensional point p (∈P_1) is set as an element of the set Q_1.
(X, Y, d) = K_1 p
x=X/d
y = Y/d
The number of points projected within the region of the first image is denoted as M_1. Note that the correspondence between each element of the set Q_1 and each 3D point of the 3D point group P before the transformation is maintained.

Then, for each of the 3D points included in the 3D point group P_1, the projection matrix R_C and translation vector T_C are applied according to the following formula to obtain the 3D point group P_2. The 3D point group P_2 is a collection of N 3D vectors.

P_2=R_C P_1+T_C

The three-dimensional point cloud P_2 is projected onto the second image of the second camera 54 using the internal parameter K_2, and those projected outside the area of the second image are removed from the three-dimensional point cloud P_2 to obtain a set of elements Q_2 consisting of (x coordinate in the second image, y coordinate in the second image, depth value of the three-dimensional point). The number of points projected within the area of the second image is denoted as M_2. Note that the correspondence between each element of the set Q_2 and each element of the three-dimensional point cloud P before transformation is maintained.

Elements of set Q_1 that are also included in set Q_2 are extracted using the correspondence with the 3D point group P, and set Q_1 is updated so that it becomes a set consisting of elements that are included in both set Q_1 and set Q_2.

The neighborhood extraction unit 22 extracts depth values of three-dimensional points corresponding to multiple neighboring pixel positions for each pixel position on the first image based on the pixel position on the first image corresponding to each three-dimensional point of the three-dimensional point group P.

Specifically, for each pixel in the first image, elements that are included in set Q_1 and have a nearby pixel position are extracted. As a result, a neighborhood set Q_1_xy is extracted for each pixel position (x, y). The neighborhood set Q_1_xy for each pixel position (x, y) is a subset of set Q_1. Furthermore, for pixel positions (x, y) for which no elements with nearby pixel positions were extracted, the neighborhood set Q_1_xy is set as an empty set.

Note that the definition of neighborhood is arbitrary, but for example, neighborhood may be within a certain radius from the pixel position (Figure 5), or K neighborhoods from the pixel position. Figure 5 shows an example in which elements having pixel positions that exist within a search radius from the pixel position of interest are extracted as neighborhood set Q_1_xy.

In the belief propagation unit 32 described later, the pixel position (x, y) is assigned the depth value of one of the elements in the neighborhood set Q_1_xy.

For each pixel position (x, y) in the first image, the neighborhood selection unit 24 selects a depth value from the depth values of the elements of the neighborhood set Q_1_xy such that there is high consistency between the first image and the second image and the distance between the pixel position (x, y) and the pixel position of the element is short.

Specifically, the neighborhood selection unit 24 includes a cost calculation unit 30 and a belief propagation unit 32.

For each pixel position (x, y) on the first image, the cost calculation unit 30 projects a point represented by using the pixel position (x, y) and the depth value of the element for each element included in the neighborhood set Q_1_xy onto the second image, and calculates the distance between the pixel value of the pixel position (x, y) on the first image and the pixel value of the pixel position where the point is projected on the second image. For each pixel position (x, y) on the first image, the cost calculation unit 30 calculates a cost function represented by using the distance between the pixel position (x, y) on the first image and the pixel position of the element, and the distance between the pixel value of the pixel position (x, y) on the first image calculated above and the pixel value of the pixel position where the point is projected on the second image for each element included in the neighborhood set Q_1_xy.

Specifically, the cost calculation unit 30 calculates a cost function for each element q = (q_x, q_y, d) included in the neighborhood set Q_1_xy. The cost function evaluates the likelihood that the pixel position (x, y) has the depth value d of element q.

First, it is derived where on the second image a pixel position (x, y) of the first image is projected when it has a depth value of d. This can be obtained by inversely projecting a point (x, y, d) expressed using the pixel position (x, y) of the first image and the depth value d by the inverse matrix of the internal parameter K_1, applying the projection matrix R_C and the translation vector T_C to perform coordinate transformation into the coordinate system of the second image of the second camera 54, and then projecting onto the second image plane by the internal parameter K_2. The pixel position on the second image obtained in this way is assumed to be (x', y'). For example, the coordinates (x', y') of the pixel position (x, y) projected onto the second image using the depth value d are calculated according to the following formula.
(X', Y', d') = K_2 (R_C K_1 ^-1 (d×x, d×y, d) + T_C)
x'=X'/d'
y'=Y'/d'
As a result, the cost function c_xyq when the pixel position (x, y) has a depth value of q can be calculated as follows.

c_xyq=w_S Stereo (I_1, I_2, x, y, x', y') + w_D Distance (x, y, q_x, q_y)

Stereo is a function that evaluates the difference in pixel values at pixel position (x, y) in the first image and pixel position (x', y') in the second image. There are methods that take the difference in pixel values, the Hamming distance between images that have been census transformed, and the difference in the gradient of pixel values (see Non-Patent Document 3), and the weighting and window size are arbitrary.

Distance is the distance between the pixel position (x, y) of the first image and the pixel position (q_x, q_y) of the element, and can be the l1 distance, l2 distance, or Huber distance, or a truncated value of the distance can be used.

w_S and w_D are the respective weights. c_xyq means that when a depth value d is assigned to a pixel position (x, y), the first image and the second image are highly consistent, and it is easy to assign the depth value of a 3D point projected near the pixel position (x, y).

For each pixel position (x, y) on the first image, the belief propagation unit 32 selects a depth value for a three-dimensional point based on a cost function and an energy function expressed using the difference from the depth value selected for an adjacent pixel.

Specifically, the belief propagation unit 32 defines an energy function that takes into account a smoothing term in addition to the cost function described above, and selects the depth value d(x, y) that minimizes the energy function for each pixel position (x, y) of the first image by belief propagation. The energy function E is defined by the following formula.

norm(∇d(x,y)) means the norm of the difference between adjacent depth values d(x,y) (such as l1 distance, l2 distance, or Huber distance), and a truncated value of the norm may be used. The smaller the difference in depth values between adjacent pixels, the smaller the value of norm(∇d(x,y)).
For example, norm(∇d(x, y)) is the sum of the norms (such as l1 distance, l2 distance, or Huber distance) of the selected depth value differences calculated for the pixel positions adjacent to the right and below the pixel position of interest. A specific example of how to calculate the difference with the pixel position adjacent to the right is shown below. The calculation of the difference with the pixel position adjacent below is similar.
At a pixel position (x, y) of interest in the first image, a depth value d is selected from Q_1_xy, and at an adjacent pixel position (x+1, y), a depth value d' is selected from Q_1_x+1_y. Then, the difference between them (d-d') is derived and its norm is calculated.

By minimizing the energy function E, it is possible to determine the optimal depth value d(x, y) for each pixel position (x, y), taking into account the stereo error, the proximity of the projection position, and the smoothness of the depth value at each pixel position.

The energy function E can be minimized by a technique called belief propagation, more specifically, loopy belief propagation.

As described above, the belief propagation unit 32 determines a combination of depth values d(x, y) selected from the neighborhood set Q_1_xy for each of all pixel positions (x, y) by minimizing the energy function E according to the following formula:

d(x,y)=argminE

The smoothing unit 26 smoothes the selected depth value d(x, y) for each pixel position (x, y) on the first image.

Here, the depth value d(x, y) is derived by assigning depth values of three-dimensional points projected to nearby pixel positions by the belief propagation unit 32, and is therefore a discontinuous map. Therefore, the smoothing unit 26 according to this embodiment smooths using the Variational method, which is a smoothing method for continuous functions, generates the smoothed depth value d_v(x, y), and outputs it as a depth image on the display unit 16.

Specifically, the following energy function E_V is minimized using the Variational method.

norm1 and norm2 are distances such as the l1 distance, l2 distance, or Huber distance. G is the ADT (Anisotropic Diffusion Tensor) or B-ADT (Binary Anisotropic Diffusion Tensor) described in Non-Patent Document 4, which weights the smoothing term norm2 (∇d_V(x, y)) for each position in the image.

The minimization of the energy function E_V can be performed by the first order primal dual algorithm as described in Non-Patent Document 4. This allows the combination of depth values d_V(x,y) for all pixel positions (x,y) after smoothing to be found according to the following formula:

d_V(x,y)=argmin E_V

[Non-Patent Document 4] Yao, Yasuhiro, et al. "Discontinuous and Smooth Depth Completion with Binary Anisotropic Diffusion Tensor." IEEE Robotics and Automation Letters 5.4 (2020): 5128-5135.

<Action of the 3D point cloud processing device according to this embodiment>
Next, the operation of the three-dimensional point cloud processing device 10 will be described.

Figure 6 is a flowchart showing the flow of 3D point cloud processing by the 3D point cloud processing device 10. The CPU 11 reads out a 3D point cloud processing program from the ROM 12 or storage 14, expands it in the RAM 13, and executes it to perform 3D point cloud processing. In addition, the 3D point cloud P measured by the LiDAR sensor 50, the first image taken by the first camera 52, and the second image taken by the second camera 54 are input to the 3D point cloud processing device 10. In addition, the internal parameters K_1 of the first camera 52, the internal parameters K_2 of the second camera 54, the rotation matrix R_C between the first camera 52 and the second camera 54, the translation vector T_C between the first camera 52 and the second camera 54, the projection matrix R_L between the first camera 52 and the LiDAR sensor 50, and the translation vector T_L between the first camera 52 and the LiDAR sensor 50 are input to the 3D point cloud processing device 10.

In step S100, the CPU 11, as the input processing unit 20, acquires the first image, the second image, and the three-dimensional point cloud received by the input unit 15.

In step S102, the CPU 11, as the input processing unit 20, calculates sets Q_1 and Q_2 based on the first and second images and the three-dimensional point cloud.

In step S104, the CPU 11, as the neighborhood extraction unit 22, extracts a neighborhood set Q_1_xy for each pixel position (x, y) on the first image based on the pixel positions on the first image corresponding to each 3D point of the 3D point group P.

In step S106, the CPU 11, as the cost calculation unit 30, calculates a cost function for each element q = (q_x, q_y, d) included in the neighborhood set Q_1_xy for each pixel position (x, y) on the first image.

In step S108, the CPU 11, as the belief propagation unit 32, selects, for each pixel position (x, y) on the first image, one of the depth values of each element included in the neighborhood set Q_1_xy based on a cost function and an energy function expressed using the difference with the depth value selected for the adjacent pixel.

In step S110, the CPU 11, as the smoothing unit 26, smoothes the selected depth values for each pixel position on the first image, displays the smoothed depth values as a depth image on the display unit 16, and ends the 3D point cloud processing routine.

As described above, the 3D point cloud processing device according to this embodiment selects a depth value for each pixel position on the first image based on the pixel position on the first image onto which each of the 3D points of the 3D point cloud measured by the LiDAR sensor is projected, from the depth values of the 3D points corresponding to multiple nearby pixel positions, so that there is high consistency between the first image and the second image and the distance to the nearby pixel positions is short. This allows the 3D point cloud obtained by measurement to be upsampled with high precision.

<Modification>
The present invention is not limited to the above-described embodiment, and various modifications and applications are possible without departing from the spirit and scope of the present invention.

For example, the example described above uses a LiDAR sensor to measure and obtain a three-dimensional point cloud, but the present invention is not limited to this. A sensor other than a LiDAR sensor may be used to measure a three-dimensional point cloud.

Although the first image and the second image are taken by different cameras, the present invention is not limited to this example. As long as the relationship between the photographing positions is determined in advance, the first image and the second image may be taken by the same camera.

In addition, various processes that the CPU reads and executes in each of the above embodiments by reading the software (programs) may be executed by various processors other than the CPU. Examples of processors in this case include a dedicated electrical circuit that is a processor having a circuit configuration designed specifically to execute a specific process, such as a GPU (Graphics Processing Unit), a PLD (Programmable Logic Device) whose circuit configuration can be changed after manufacture, such as an FPGA (Field-Programmable Gate Array), and an ASIC (Application Specific Integrated Circuit). In addition, the three-dimensional point cloud processing may be executed by one of these various processors, or may be executed by a combination of two or more processors of the same or different types (for example, multiple FPGAs, a combination of a CPU and an FPGA, etc.). More specifically, the hardware structure of these various processors is an electrical circuit that combines circuit elements such as semiconductor devices.

In addition, in each of the above embodiments, the 3D point cloud processing program is described as being pre-stored (installed) in the storage 14, but this is not limiting. The program may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The program may also be downloaded from an external device via a network.

The following notes are further provided with respect to the above embodiment.

(Additional Note 1)
A three-dimensional point cloud processing device,
Memory,
at least one processor coupled to the memory;
Including,
The processor,
receiving a first image and a second image, the relationship between at least the shooting positions of which is previously determined, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is previously determined, and determining pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud;
a 3D point cloud processing device configured to select, for each pixel position on the first image, a depth value from depth values of 3D points corresponding to a plurality of neighboring pixel positions based on a pixel position on the first image corresponding to each of the 3D points of the 3D point cloud, such that consistency between the first image and the second image is high.

(Additional Note 2)
A non-transitory storage medium storing a program executable by a computer to perform three-dimensional point cloud processing,
The three-dimensional point cloud processing includes:
receiving a first image and a second image, the relationship between at least the shooting positions of which is previously determined, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is previously determined, and determining pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud;
A non-transitory storage medium that selects, for each pixel position on the first image based on a pixel position on the first image corresponding to each three-dimensional point of the three-dimensional point cloud, the depth value from depth values of three-dimensional points corresponding to multiple neighboring pixel positions so as to increase consistency between the first image and the second image.

Reference Signs List 10 3D point cloud processing device 14 Storage 15 Input unit 16 Display unit 20 Input processing unit 22 Neighborhood extraction unit 24 Neighborhood selection unit 26 Smoothing unit 30 Cost calculation unit 32 Belief propagation unit 50 LiDAR sensor 52 First camera 54 Second camera

Claims (8)

an input processing unit that receives a first image and a second image, the relationship between at least the shooting positions of which is determined in advance, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud;
a neighborhood selection unit that selects, for each pixel position on the first image, a depth value from depth values of three-dimensional points corresponding to a plurality of neighboring pixel positions based on a pixel position on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, such that consistency between the first image and the second image is high;
A three-dimensional point cloud processing device comprising: The neighborhood selection unit is
For each pixel location on the first image,
For each of the three-dimensional points corresponding to the plurality of neighboring pixel positions, a point consisting of the pixel position on the first image and a depth value of the three-dimensional point is projected onto the second image, and a distance between a pixel value of the pixel position on the first image and a pixel value of the pixel position of the point projected onto the second image is calculated;
2. The three-dimensional point cloud processing device according to claim 1, further comprising: a cost function expressed using a distance between a pixel value of a pixel position on the first image and a pixel value of a pixel position of a point projected onto the second image, the cost function being expressed using a distance between a pixel value of a pixel position on the first image and a pixel value of a pixel position of a point projected onto the second image, the cost function being expressed using a cost function ... The neighborhood selection unit is
For each pixel location on the first image,
3. The three-dimensional point cloud processing device according to claim 2, wherein the depth value is selected based on the cost function and an energy function expressed using a difference between a depth value selected at a pixel position of interest on the first image and a depth value selected at an adjacent pixel position. The 3D point cloud processing device according to any one of claims 1 to 3, further comprising a smoothing unit that smoothes the depth value selected for each pixel position on the first image. The 3D point cloud processing device according to any one of claims 1 to 4, wherein the neighborhood selection unit selects, for each pixel position on the first image based on the pixel positions on the first image corresponding to each of the 3D points of the 3D point cloud, the depth value from the depth values of 3D points corresponding to multiple neighboring pixel positions, such that the consistency between the first image and the second image is high and the distance to the neighboring pixel positions is short. The input processing unit:
projecting each of the three-dimensional points of the three-dimensional point cloud onto the first image, removing three-dimensional points projected outside the area of the first image, and obtaining a set of elements that are combinations of pixel positions and depth values on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud after the removal;
projecting the points of each element of the set onto the second image, removing elements projected outside the region of the second image, and updating the set to the removed elements;
The neighborhood selection unit is
A 3D point cloud processing device as described in any one of claims 1 to 5, wherein, based on the pixel positions of the elements of the set on the first image, for each pixel position on the first image, a depth value of the elements of the set having a plurality of neighboring pixel positions is selected from the depth values of the elements of the set so that there is high consistency between the first image and the second image and the distance to the neighboring pixel positions is short. an input processing unit receives a first image and a second image, the relationship between at least the shooting positions of which is determined in advance, and a three-dimensional point cloud on a surface of an object, the relationship between at least the shooting positions and measurement positions of which is determined in advance, and determines pixel positions on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud;
A three-dimensional point cloud processing method, in which a neighborhood selection unit selects, for each pixel position on the first image, based on a pixel position on the first image corresponding to each of the three-dimensional points of the three-dimensional point cloud, the depth value from depth values of three-dimensional points corresponding to a plurality of neighboring pixel positions, so as to increase consistency between the first image and the second image. A three-dimensional point cloud processing program for causing a computer to function as a three-dimensional point cloud processing device according to any one of claims 1 to 6.