JP7557406B2 - Point group decoding device, point group decoding method and program - Google Patents

Description

The present invention relates to a point cloud decoding device, a point cloud decoding method, and a program.

Non-Patent Document 1 discloses a technique for encoding and decoding geometric information of a point cloud using inter-prediction. Such inter-prediction is composed of multiple component technologies.

Non-Patent Document 2 discloses a technique for matching small regions (nodes) divided by the Octree method between two chronologically consecutive point groups, and for achieving motion compensation on a node-by-node basis.

Non-Patent Document 3 discloses a technique in which, when a corresponding node is divided into eight child nodes, it is determined whether or not a point exists in each child node (occupancy information), and the occupancy information of the node in the previous point cloud (reference frame) in the time series is used as a prediction of the occupancy information of the node in the current point cloud (current frame) in the time series.

Such occupancy predictions are expressed in the following classifications: no pred, which indicates unpredictable; pred0, which indicates non-occupied; pred1, which indicates occupied; and predL, which indicates strong occupied.

For pred1 and predL, if the number of points in the child node is less than a fixed threshold, it is classified as pred1, otherwise it is classified as predL.

Here, when arithmetically coding the node occupancy information of the current frame, such predictions are used to select the context of each bit, and different occurrence probabilities are selected for each context, thereby improving coding performance.

Non-Patent Document 4 discloses a technique for improving the performance of coding based on Non-Patent Documents 2 and 3 by estimating and aligning the motion parameters (rotation, translation) of the entire point cloud between two time-series consecutive point clouds.

Exploratory model for inter-prediction in G-PCC, ISO/IEC JTC1/SC29/WG11 N18096 [PCC] On motion compensation for geometry coding in TM3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m42521 [PCC] How to use a predictive set of points for geometry coding in TMC3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m42520 [PCC] Global motion compensation for point cloud compensation in TM3, ISO/IEC JTC1/SC29/WG11 MPEG2018/m44751

However, the inter prediction in Non-Patent Documents 1 to 4 has a problem in that when the spatial density of point clouds is low and the correlation in the temporal direction is low, the prediction accuracy is low and the coding efficiency is reduced.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide a point cloud decoding device, a point cloud decoding method, and a program that can improve the inter-prediction accuracy of point clouds and improve coding efficiency.

The first feature of the present invention is a point cloud decoding device configured to decode a point cloud from a bit stream, comprising an upsampling unit configured to upsample points of reference geometric information, and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

The second feature of the present invention is a point cloud decoding method configured to decode a point cloud from a bit stream, comprising the steps of upsampling points of reference geometric information and predicting occupancy information using the upsampled reference geometric information.

The third feature of the present invention is a program for causing a computer to function as a point cloud decoding device configured to decode a point cloud from a bit stream, the point cloud decoding device having an upsampling unit configured to upsample points of reference geometric information, and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

The present invention provides a point cloud decoding device, a point cloud decoding method, and a program that can improve the inter-prediction accuracy of point clouds and improve coding efficiency.

FIG. 1 is a diagram illustrating an example of the configuration of a point cloud processing system 10 according to an embodiment. FIG. 2 is a diagram illustrating an example of functional blocks of a point group decoding device 200 according to an embodiment. FIG. 3A is a diagram illustrating an example of an overview of nodes for each layer in one embodiment. FIG. 3B is a diagram illustrating an example of an overview of nodes for each layer in one embodiment. FIG. 4 is a schematic diagram of upsampling to a location where the L1 distance is 1. FIG. 5 is a flowchart showing an example of a process for generating a model in the point group decoding device 200 according to an embodiment. FIG. 6 is a flowchart showing an example of a process for generating a model in the point group decoding device 200 according to an embodiment. FIG. 7 is a schematic diagram showing the case where upsampling is performed using a vector set obtained by maximum likelihood estimation. FIG. 8 is a diagram showing an example in which an azimuth angle is used as attribute information.

Hereinafter, the embodiments of the present invention will be described with reference to the drawings. Note that the components in the following embodiments can be appropriately replaced with existing components, and various variations including combinations with other existing components are possible. Therefore, the description of the following embodiments does not limit the contents of the invention described in the claims.

First Embodiment
A point cloud processing system 10 according to a first embodiment of the present invention will be described below with reference to Figures 1 to 8. Figure 1 is a diagram showing a point cloud processing system 10 according to the present embodiment.

As shown in FIG. 1, the point cloud processing system 10 has a point cloud encoding device 100 and a point cloud decoding device 200.

The point cloud encoding device 100 is configured to generate encoded data (bit stream) by encoding an input point cloud signal. The point cloud decoding device 200 is configured to generate an output point cloud signal by decoding the bit stream.

The input point cloud signal and the output point cloud signal are composed of position information and attribute information of each point in the point cloud. The attribute information is, for example, color information and reflectance of each point.

Here, such a bit stream may be transmitted from the point cloud encoding device 100 to the point cloud decoding device 200 via a transmission path. Also, the bit stream may be stored in a storage medium and then provided from the point cloud encoding device 100 to the point cloud decoding device 200.

(Point Cloud Decoding Device 200)
Hereinafter, the point group decoding device 200 according to this embodiment will be described with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the point group decoding device 200 according to this embodiment.

As shown in FIG. 2, the point cloud decoding device 200 includes a motion decoding unit 21, an arithmetic decoding unit 22, an octree decoding unit 23, a motion compensation unit 24, an upsampling unit 25, an occupancy prediction unit 26, a probability acquisition unit 27, and a model generation unit 28.

In this embodiment, the processing in the model generation unit 28 may be performed in advance before the decoding process is performed.

Below, we will explain each functional block of the point cloud decoding device 200 according to this embodiment. For specific processing in the motion decoding unit 21, the arithmetic decoding unit 22, the octree decoding unit 23, the motion compensation unit 24, the occupancy prediction unit 26, and the probability acquisition unit 27, for example, the methods described in the above-mentioned non-patent documents 1 to 4 can be used.
The motion decoding unit 21 is configured to decode coded motion parameters contained in the input bitstream, where the motion parameters are expressed as rotation matrices and translation vectors that perform a pose transformation on the entire point cloud or only a part of the point cloud.

The arithmetic decoding unit 22 is configured to decode octree data (bit string) using the input bit stream and the occurrence probability of the given data. Such data may be a binary value of 0 or 1, or may be something else. The occurrence probability of such data is provided by the probability acquisition unit 24.

Since the octree decoding unit 23 described later performs processing on a node-by-node basis, the arithmetic decoding unit 22 is configured to stop processing once it has decoded eight bits representing the occupancy information of the eight child nodes contained in one node.

The arithmetic decoding unit 22 is configured to resume processing after the octree decoding unit 23 completes processing on these eight bits.

In this way, the arithmetic decoding unit 22 and the octree decoding unit 23 are configured to alternate between processing until the entire input bitstream has been processed.

In this embodiment, the octree data in all or some of the nodes may be replaced with well-known quad tree data or binary tree data.

The octree decoding unit 23 is configured to decode geometric information from the bit string using a decoding method based on the well-known octree method.

In other words, when eight bits representing the occupancy information of eight child nodes contained in a node are given, the octree decoding unit 23 is configured to generate a child node at a position where the bit is 1.

The octree decoding unit 23 is configured to perform this processing hierarchically from the top to the bottom, and output the coordinates of the nodes in the bottom layer as geometric information representing a point cloud.

Since the octree method is a method for processing on a node-by-node basis, the octree decoding unit 23 is configured to stop processing once the generation process for the eight child nodes contained in the current node is completed.

The octree decoding unit 23 is configured to resume processing after the arithmetic decoding unit 22 decodes the eight bits representing the occupancy information of the eight child nodes of the node to be processed next.

In this way, the arithmetic decoding unit 22 and the octree decoding unit 23 are configured to alternate between processing until the entire input bitstream has been processed.

Here, nodes are represented as voxels that divide three-dimensional space, so it is possible to select reference geometric information that corresponds to the area of the currently processed node or its child nodes. This selection is a process of extracting only a portion of the reference geometric information that is represented as a point cloud and that is contained within the area.
In this embodiment, the octree decoding unit 23 is configured to output to the upsampling unit 25 the region of voxels represented by the eight child nodes of the node currently being processed.

In this embodiment, the octree method may be replaced with the well-known quadtree method or binary tree method in all or some of the nodes.
The motion compensation unit 24 is configured to perform posture transformation of the reference geometric information represented as a point cloud using the motion parameters, and output the reference geometric information. Such posture transformation may be performed on the entire point cloud, or only on a part of the point cloud.

In other words, the motion compensation unit 24 is configured to perform posture transformation using common motion parameters for all points in the case of the entire point cloud, and to perform posture transformation using individual motion parameters for some points in the case of only a portion of the point cloud. The motion parameters are provided by the motion decoding unit 21.

The upsampling unit 25 is configured to perform upsampling on the reference geometric information represented as a point cloud. Such reference geometric information may be pose-transformed by the motion compensation unit 24, or may not be pose-transformed by the motion compensation unit 24.

The upsampling unit 25 is configured to generate new points around each point of the reference geometric information using a given model. Such a model is provided by the model generation unit 28.

When the model requires attribute information for a point that is not shown (e.g., distance from the LiDAR sensor), the upsampling unit 25 generates the point by using or calculating such attribute information.

In other words, the upsampling unit 25 is configured to perform upsampling using a different model for each piece of attribute information. That is, the upsampling unit 25 is configured to change the upsampling method for a point depending on the attribute information of the point.

Here, if attribute information has been assigned to each point of the reference geometric information in advance, it is used as is, and if attribute information has not been assigned to each point of the reference geometric information in advance, it is newly calculated.

In this embodiment, the upsampling unit 25 is configured to select the reference geometric information using the area of voxels represented by the eight child nodes of the node output from the octree decoding unit 23.

The upsampling unit 25 is configured to perform upsampling on the selected reference geometric information.

However, such upsampling may be performed on the input reference geometric information or the reference geometric information output from the motion compensation unit 24.

The upsampling unit 25 may also be configured to receive syntax from the model generation unit 28 and, based on the syntax, determine information on whether to perform upsampling and the upsampling method.

The upsampling unit 25 may also be configured to distinguish between sparse and dense points in a given point cloud and to perform upsampling only on the sparse points.

Here, methods for determining sparse points may include, for example, whether the distance between each point and its nearest neighbor is equal to or greater than a threshold, whether the number of points located within a radius r [m] of each point is equal to or less than a threshold, whether the number of points located inside the voxel to which each point belongs is equal to or less than a threshold, or other methods.

The occupancy prediction unit 26 is configured to predict the occupancy information using a point cloud representing the reference geometric information upsampled by the upsampling unit 25.

That is, the occupancy prediction unit 26 obtains classifications such as no pred (unpredictable), pred0 (unoccupied), pred1 (occupied), and predL (strongly occupied) from reference geometric information expressed as a group of points in a voxel represented by eight child nodes corresponding to a bit string to be decoded by the arithmetic decoding unit 22. The region of such a voxel is provided by the octree decoding unit 23.

The occupancy prediction unit 26 is configured to output the prediction of the occupancy information of these eight child nodes to the probability acquisition unit 27.

In this embodiment, occupancy information is predicted based on the results of upsampling the reference geometric information contained in the voxel area represented by the eight child nodes of the node output from the octree decoding unit by the upsampling unit.
However, the occupancy prediction unit 26 may be configured to upsample the input reference geometric information or the reference geometric information output from the motion compensation unit 24, and then predict the occupancy information based on the result of selecting the reference geometric information included in the voxel area represented by the eight child nodes of the node output from the octree decoding unit 23.

In addition, in the octree method, it is expected that the number of points contained within a node will be greater in higher layers and less in lower layers. Therefore, the threshold th used to classify pred1 and predL may be set to th = λ x n instead of a fixed value. Here, n is the hierarchical number in the octree method (the lowest layer is set to 1, and the higher the layer, the larger the number), and λ is an arbitrary coefficient.

Figures 3A and 3B show an overview of the nodes for each layer. The nodes are actually represented as a three-dimensional grid, but here they are represented as a two-dimensional grid to simplify the illustration.

Figure 3A shows a schematic diagram of four nodes in the upper layer (n=4). The squares in each node represent the smallest unit that can represent a point, and a square filled with black indicates the presence of a point. Figure 3B shows a schematic diagram of four nodes in the lower layer (n=2).

Nodes in higher layers have a larger voxel volume and therefore may contain more points. On the other hand, nodes in lower layers are divisions of nodes in higher layers, so they contain relatively fewer points.

Therefore, when a fixed threshold value th is used, the higher the tier, the more likely it is to be classified as predL, and the lower the tier, the more likely it is to be classified as pred1. For example, when th = 3, all the nodes in the upper tier shown in Figure 3A are classified as predL, and all the nodes in the lower tier shown in Figure 3B are classified as pred1.

On the other hand, for example, if λ = 1 and th = λ × n, the upper and lower nodes in the upper left corner of Figures 3A and 3B are classified as predL, and the other nodes are classified as pred1.

In this way, by varying the threshold depending on the hierarchy, the effect is achieved of classifying nodes in the same hierarchy according to point density.

Alternatively, any statistical quantity may be used instead of the hierarchical number. For example, the average value m of the number of points in an occupied node in a certain hierarchical level may be counted, and the threshold value used in the next lower hierarchical level may be th = λ × m. Alternatively, the statistical quantity may be the median value of the number of points in an occupied node in a certain hierarchical level, or the cumulative moving average value from the top hierarchical level. This allows the threshold value to be adaptively changed in response to the density of the input point cloud.

The probability acquisition unit 27 is configured to acquire the occurrence probability of data corresponding to a node decoded by the arithmetic decoding unit 22, using the prediction of the occupancy information provided by the occupancy prediction unit 26. Such occurrence probability may be a fixed value or an adaptive variable value.

The probability acquisition unit 27 may be configured to use other information, such as occupancy information of neighboring nodes of the parent node of the node decoded by the arithmetic decoding unit 22, or occupancy information of eight child nodes that have a common parent node, when acquiring the occurrence probability.

The probability acquisition unit 27 is configured to output the acquired occurrence probability to the arithmetic decoding unit 22.

The model generator 28 is configured to generate a model for upsampling the points. The model generator 28 may be configured to generate such a model based on a regular method.

For example, the model generation unit 28 may be configured to generate a model that upsamples to six locations in three-dimensional space where the L1 distance is 1 based on the position of the point, or may be configured to generate a model that upsamples to the vertex positions of a geodesic dome when the point is placed at the center, or may be configured to generate a model that upsamples using another method.

Figure 4 shows a schematic diagram of upsampling to a point where the L1 distance is 1. Points are actually represented in three dimensions, but here they are represented in two dimensions to simplify the illustration.

In Figure 4, a black square represents a point in the reference frame, and a black circle represents a point upsampled from this point. In Figure 4, the point is upsampled to four locations on a two-dimensional grid where the L1 distance is 1.

The model generation unit 28 may also be configured to generate such a model based on maximum likelihood estimation using a training point cloud. Figure 5 shows a flowchart of the process in such a case.

As shown in FIG. 5, when a training point cloud is input in step S101, in step S102, the motion compensation unit 24 treats multiple chronologically consecutive point clouds measured by the LiDAR as a training point cloud, i.e., a pair of a reference frame and a current frame, and performs posture transformation of the reference frame.

In step S103, the model generation unit 28 matches each point in the reference frame with a point in the current frame by nearest neighbor search, and calculates a vector representing the position error of the corresponding points.

In step S104, the model generation unit 28 may quantize such vectors to increase their occurrence frequency.

For example, the model generation unit 28 may replace the vector direction and magnitude with representative values that are discretized to be spatially equally spaced, or may replace them with representative values of a group of vectors classified into k ways by the k-means method, or may use other methods.

In step S105, the model generation unit 28 counts the number of times the same vector appears and selects the top N vectors with the highest frequency of appearance. The model generation unit 28 may generate a model that upsamples a point to the destination indicated by the vector based on the position of the point.

In step S106, the model generation unit 28 outputs the selected vector set.

Figure 6 shows a flowchart of another case of the process for generating the above-mentioned model.

As shown in FIG. 6, in step S201, a training point cloud and vector set are input, and in step S202, the model generation unit 28 encodes the current frame using a reference frame without upsampling to obtain the code amount.

In step S203, the model generation unit 28 repeats the processes of steps S204 to S209 until the number of elements in the vector set becomes 0.

In step S204, the model generation unit 28 repeats the operations of steps S205 to S209 for each of the remaining vectors in the vector set.

In step S205, the model generation unit 28 upsamples the reference frame using the set of selected vectors and the one vector selected in step S204.

In step S206, the model generation unit 28 encodes the current frame using the upsampled reference frame and obtains the code amount.

In step S207, the model generation unit 28 determines whether the code amount obtained in step S206 is the lowest in history. If Yes, the process proceeds to step S208, and if No, the process proceeds to step S209.

In step S208, the model generation unit 28 adds the one vector selected in step S204 to the set of selected vectors.

In step S209, the model generation unit 28 excludes the one vector selected in step S204 from the vector set.

In step S210, the model generation unit 28 outputs the selected vector set.

When determining the number N of vectors, the model generation unit 28 may be configured to select N that maximizes the coding efficiency when the training point cloud is actually coded.

Alternatively, the model generation unit 28 may be configured to select the N vectors in order of increasing coding efficiency for the training point cloud, instead of selecting them in order of increasing frequency of occurrence.

In other words, the model generation unit 28 may be configured to investigate the encoding efficiency when all vectors that are candidates for selection are included in the model, and then include in the model one vector that has the greatest improvement effect. Then, the model generation unit 28 may be configured to exclude this vector from the candidates and repeat the same process until the encoding efficiency does not improve any more. In this way, the model generation unit 28 may select a set of vectors to be included in the model.

Alternatively, the model generation unit 28 may be configured to first select a set of N vectors with high frequency of occurrence in the process shown in FIG. 5, then select vectors one by one in descending order of frequency of occurrence, and perform the inner iterative process (steps S204 to S209) shown in FIG. 6, thereby selecting a set of vectors to be included in the model in only N iterations.

A schematic diagram of upsampling using a vector set obtained by maximum likelihood estimation is shown in Figure 7. Although points are actually represented in three dimensions, they are represented in two dimensions here to simplify the diagram.
In FIG. 7, a black square represents a point in the reference frame, and a black circle represents a point upsampled from that point.

As an example of the resulting vector set, consider v1 = (1, 1), v2 = (2, -2), v3 = (-1, 2), v4 = (-2, 0), and v5 = (-1, -2). In Figure 7, points are upsampled to the positions indicated by these vectors, based on the position of a point in the reference frame.

The model generation unit 28 may also be configured to create such a model individually for each piece of point attribute information.

Such attribute information is any feature of a point, such as the distance from the center, the elevation angle, or the azimuth angle when the coordinates of the point are expressed in a three-dimensional polar coordinate system. Alternatively, such attribute information may be the angle between vectors from a point to its two neighboring points, or the magnitude or direction of a vector from a point to the center of its two neighboring points. Alternatively, such attribute information may be information such as that described in Non-Patent Document 5 (Stoyanov, Todor et al., "Fast and accurate scan
registration through minimization of the distance between compact 3D NDT representations. ”, The International Journal of
As described in (Robotics Research 31.12 (2012): 1377-1393), a variance-covariance matrix calculated using points surrounding a point is subjected to singular value decomposition, and the classification may be a sphere, a plane, or a line obtained from the relationship between the singular values, or other features may be used.

The above-mentioned maximum likelihood estimation may be performed for each point group classified by such attribute information. That is, the model generation unit 28 may be configured to generate a different model for each piece of attribute information.

At this time, the attribute information values may be quantized to increase the frequency with which the attribute information matches between points in the reference frame and points in the current frame. For example, the distance from the center may be expressed in bins spaced at 10-meter intervals, and points that belong to the same bin may be considered to be at the same distance.

When classifying by attribute information, multiple pieces of attribute information may be combined to increase the number of classification patterns.

Figure 8 shows an example of using azimuth angle as attribute information. In the example in Figure 8, the black circle represents the origin, and the triangle and diamond represent points in the reference frame.

The position of a point in a reference frame is generally expressed in (x, y, z) coordinates relative to the origin in a three-dimensional Cartesian coordinate system, and can therefore be converted to a three-dimensional polar coordinate system representation.

The example in Figure 8 shows a case where the triangular points have an azimuth angle of 45°, and the diamond points have an azimuth angle of 150°. In this way, attribute information can be assigned to each point, and the points can be classified according to the attribute information.

The model generation unit 28 may also be configured to output information about whether or not upsampling is to be performed by the upsampling unit 25 and the upsampling method as syntax.

This syntax is information in which, for example, 0 represents a rule that no upsampling is performed, 1 represents a rule that upsampling is performed based on a regular method, and 2 represents a rule that upsampling is performed based on maximum likelihood estimation.

Furthermore, the above-mentioned point cloud encoding device 100 and point cloud decoding device 200 may be realized as a program that causes a computer to execute each function (each process).

In each of the above embodiments, the present invention has been described using the point cloud encoding device 100 and the point cloud decoding device 200 as examples, but the present invention is not limited to such examples and can be similarly applied to a point cloud encoding/decoding system having the functions of the point cloud encoding device 100 and the point cloud decoding device 200.

In addition, according to this embodiment, for example, it is possible to realize an improvement in the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

10... Point cloud processing system 100... Point cloud encoding device 200... Point cloud decoding device 21... Motion decoding unit 22... Arithmetic decoding unit 23... Octree decoding unit 24... Motion compensation unit 25... Upsampling unit 26... Occupancy prediction unit 27... Probability acquisition unit 28... Model generation unit

Claims (8)

1. A point cloud decoding device configured to decode a point cloud from a bitstream, comprising:
an upsampling unit configured to upsample points of the reference geometric information;
and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information. a model generator configured to generate a model representing how the points are upsampled;
The point cloud decoding device according to claim 1 , wherein the upsampling unit is configured to upsample the points using the model generated by the model generation unit. The point cloud decoding device according to claim 1 or 2, characterized in that the upsampling unit is configured to upsample the points using a model generated based on a regular method. The point cloud decoding device according to any one of claims 1 to 3, characterized in that the upsampling unit is configured to upsample the points using a model generated based on maximum likelihood estimation. The point cloud decoding device according to claim 4, characterized in that the maximum likelihood estimation estimates the positions and number of points to be upsampled in the training point cloud so as to maximize the coding efficiency. The point cloud decoding device according to any one of claims 1 to 5, characterized in that the upsampling unit is configured to change the upsampling method of the points depending on attribute information of the points. A point group decoding method in which a point group decoding device decodes a point group from a bit stream, comprising:
said point cloud decoding device upsampling points of reference geometric information;
and a step of predicting occupancy information by the point cloud decoding device using the upsampled reference geometric information. 1. A program for causing a computer to function as a point cloud decoding device configured to decode a point cloud from a bitstream, comprising:
The point group decoding device comprises:
an upsampling unit configured to upsample points of the reference geometric information;
and an occupancy prediction unit configured to predict occupancy information using the upsampled reference geometric information.

Images