JP7712890B2 - Mesh decoding device, mesh encoding device, mesh decoding method and program - Google Patents

Description

The present invention relates to a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program.

Non-Patent Document 1 discloses a technique for encoding the dynamic mesh defined in Non-Patent Document 3 by extending the technique in Non-Patent Document 2, which encodes a video-based point cloud.

Graziosi, Danillo Bracco. "Video-Based Dynamic Mesh Coding." 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021. ISO/IEC FDIS 23090-5:2020, Information technology -Coded representation of immersive media - Part 5: Visual Volumetric Video-based Coding (V3C) and Video-based Point Cloud Compression (V-PCC). CfP for Dynamic Mesh Coding, ISO/IEC JTC1/SC29/WG7 N00231, MPEG136 - OnLine. Rossignac, Jarek. "Edgebreaker: Connectivity compression for triangle meshes." IEEE transactions on visualization and computer graphics 5.1 (1999): 47-61.

However, the video-based mesh coding technology disclosed in Non-Patent Document 1 has a problem in that the number of polygons in a mesh is uniform, making it difficult to optimize for a wide range of bit rates, from low to high. Therefore, the present invention has been made in consideration of the above-mentioned problems, and aims to provide a mesh decoding device, mesh coding device, mesh decoding method, and program that can improve the efficiency of mesh coding.

The first feature of the present invention is a mesh decoding device comprising a mesh decoding unit configured to decode a mesh in each patch from an encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and a mesh post-processing unit configured to perform post-processing on the mesh of an adjacent patch.

The second feature of the present invention is a mesh coding device that includes a mesh generation unit configured to generate multiple meshes with different numbers of polygons for each patch.

The third feature of the present invention is a mesh decoding method, which includes the steps of decoding a mesh in each patch from an encoded bitstream, reconstructing the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and performing post-processing on the mesh of an adjacent patch.

The fourth feature of the present invention is a program for causing a computer to function as a mesh decoding device, the mesh decoding device comprising a mesh decoding unit configured to decode a mesh in each patch from an encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch, and a mesh post-processing unit configured to perform post-processing on the mesh of an adjacent patch.

The present invention provides a mesh decoding device, a mesh encoding device, a mesh decoding method, and a program that can improve mesh encoding efficiency.

FIG. 1 is a diagram showing an example of functional blocks of a mesh encoding device 100 according to an embodiment. FIG. 2 is a diagram showing an example of how a mesh is simplified and a texture image and UV coordinates are modified by a mesh generating unit 202 of a mesh decoding device 200 according to an embodiment. FIG. 3 is a diagram showing an example of how edges are reduced using edge collapse by the mesh generation unit 202 of the mesh decoding device 200 according to an embodiment. FIG. 4 is a diagram showing an example of functional blocks of a mesh decoding device 200 according to an embodiment. FIG. 5 is a diagram showing an example of how the problem of insufficient edges between adjacent patches is solved by the mesh post-processing unit 202 of the mesh decoding device 200 according to an embodiment. FIG. 6 is a diagram showing an example of how gaps at the boundary L1 are filled using a zippering algorithm by the mesh post-processing unit 202 of the mesh decoding device 200 according to an embodiment.

The following describes embodiments of the present invention with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

First Embodiment
Hereinafter, a mesh encoding device 100 and a mesh decoding device 200 according to this embodiment will be described with reference to Fig. 1 to Fig. 6. Fig. 1 is a diagram showing an example of a functional block of the mesh encoding device 100 according to this embodiment, and Fig. 2 is a diagram showing an example of a functional block of the mesh decoding device 200 according to this embodiment.

The mesh encoding device 100 is configured to encode the dynamic mesh defined in Non-Patent Document 3.

The characteristic of a dynamic mesh is that the number of vertices and the connectivity of triangles are different in the mesh of each frame.

In addition, to cover a range from low bit rates (e.g., about 2 Mbps) to high bit rates (e.g., about 32 Mbps), the anchor in Non-Patent Document 3 creates meshes of 5K, 10K, 15K, 20K, and 25K from the original mesh, preparing six levels of mesh in combination with the original mesh, and then selects the optimal mesh based on the bit rate. In general, for low bit rates, meshes with a small number of polygons are coded, and conversely, for high bit rates, meshes with a large number of polygons are coded.

As shown in FIG. 1, the mesh coding device 100 includes a segmentation unit 101, a mesh generation unit 102, and a mesh coding unit 103.

The segmentation unit 101 is configured to segment the mesh and generate patches.

Specifically, as shown in FIG. 1, the segmentation unit 101 is configured to generate patches based on "Texture map PNG" and "Mesh OBJ".

Note that "Texture map PNG" includes "Attribute maps" defined in Non-Patent Document 3, and "Mesh OBJ" includes "Connectivity Information", "Geometry Information", and "Mapping Information" defined in Non-Patent Document 3.

Here, the segmentation unit 101 may be configured to utilize the segmentation method defined in Non-Patent Document 1. In this case, the segmentation unit 101 is configured to divide the original mesh M _all into N patches according to the following formula (1):

The mesh generating unit 102 is configured to generate a plurality of meshes (including texture information) with different numbers of polygons for each patch. Here, the texture information is all or part of a "Texture map PNG".

For example, the mesh generation unit 102 is configured to generate meshes with 1/2, 1/4, and 1/8 the number of polygons from the original mesh, and set the variable f indicating the number of polygons to 0, 1, 2, or 3. Here, when the variable f is 0, it indicates the original number of polygons, when it is 1, it indicates 1/2 the number of polygons of the original mesh, when it is 2, it indicates 1/4 the number of polygons of the original mesh, and when it is 3, it indicates 1/8 the number of polygons of the original mesh.

The mesh generating unit 102 may also use a correspondence table that associates the variable f with the number of polygons. For example, when the variable f is 0, it indicates the original number of polygons, when it is 1, it indicates 90% of the number of polygons in the original mesh, when it is 2, it indicates 80% of the number of polygons in the original mesh, and when it is 3, it indicates 70% of the number of polygons in the original mesh.

Specifically, the mesh generator 102 may be configured to generate such multiple meshes in the following manner:

The mesh generating unit 102 is configured to carry out the following procedure when reducing the number of polygons F _i ^orig to 1, 1/2, 1/4, or 1/8 (or 100%, 90%, 80%, or 70%) of the original _number of polygons F i orig for one patch M i .

In step 1, the mesh generation unit 102 is configured to select edges to be deleted.

As shown in FIG. 2, the mesh generation unit 102 is configured to calculate the cost of an edge u v (one side of a triangle) and select the edge with the smallest cost from among the target edges.

Here, the cost is defined as the magnitude of the visual change when an edge is removed. In other words, polygon edges in flat areas or short, fine edges have a low cost.

Specifically, the mesh generation unit 102 is configured to calculate the cost of an edge, cost(u, v), using equation (2).

Here, T _u is the set of polygons that include vertex u, T _uv is the set of polygons that include vertices u and v, and ||u-v|| is the distance between vertices u and v (the length of edge uv).

In step 2, the mesh generation unit 102 is configured to simplify the mesh (polygon) by reducing the selected edges using an operation called edge collapse, as shown in FIG. 3.

Here, edge collapse is an operation that targets the edge between vertex _Vt and vertex _Vs in Fig. 3A as a target for deletion, and combines vertex _Vt and vertex _Vs (deletes vertex _Vt ) as shown in Fig. 3B. In this way, performing edge collapse once reduces the number of polygons by two.

The operation of refining the mesh (polygon) by changing the state in Figure 3(b) to the state in Figure 3(a) is called vetrex split.

The mesh generation unit 102 may be configured to repeat the operations shown in Figures 2 and 3 until the number of polygons in the original mesh is reduced to half.

Furthermore, the mesh generation unit 102 may be configured to repeatedly perform such operations until the number of polygons in the original mesh is reduced to 1/4, and then to 1/8 of the original number of polygons, and to store the meshes as patches M ² _i and M ³ _i .

Finally, the mesh generator 102 is configured to set the variable fi to 0, 1, 2, and 3 for ₁ , 1/2, 1/4, and 1/8 (or 100%, 90%, 80%, and 70%) of the ^original number of polygons F i orig, respectively.

The number of polygons F _i ^k of M _i ⁰ , M _i ¹ , M _i ² and M _i ³ is calculated by the following formula (3).

The patch encoding unit 103 is configured to select and encode an optimal mesh for each patch using RDO (Rate Distortion Optimization), encode the number of polygons in the selected optimal mesh, and insert the encoded number of polygons into a bit stream.

The patch encoding unit 103 is configured to select an optimal mesh from among a plurality of prepared meshes M _i ⁰ , M _i ¹ , M _i ² and M _i ³ , each having a different number of polygons in each patch.

For example, the patch encoding unit 103 is configured to calculate the cost (Cost(M _i ^f )) using the following equation (4), select the mesh having the smallest cost (Cost(M _i ^f )) as the optimal mesh using the following equation (5), and record the corresponding variable f _i ^opt .

Cost( _Mif )=Rate ⁽ _Mif )+λ・Distortion( _Mif ^{) (} 4)

Here, Rate( ) is a function for calculating the number of encoding bits, Distortion is a function for calculating the amount of error, and λ is a parameter that is determined in advance.

The patch encoder 103 is configured to put the selected optimal mesh and the corresponding variable f _i ^opt into the bitstream.

In this way, encoding the variable f _i (where f _i is 0, 1, 2, and 3) can improve encoding efficiency rather than directly encoding the number of polygons.

Furthermore, the patch encoding unit 103 can calculate the number of polygons of the selected mesh from the variable f _i using the following equation (6).

where F _i ^opt is the number of polygons of the selected mesh in the i-th patch, and F _i ^orig is the original number of polygons in the i-th patch.

The patch encoding unit 103 may be configured to encode the coordinates, connectivity, and texture information of the vertices of the optimal mesh selected for each patch using the method described in Non-Patent Document 1.

Specifically, the method in Non-Patent Document 1 takes a mesh as input and segments it into multiple patches.

The method described in Non-Patent Document 1 projects triangles onto a plane because edges connect the vertices of each patch.

Furthermore, since the projection of triangles is sparse compared to point clouds, the method of Non-Patent Document 1 uses rasterization to convert polygons into pixel data, which is then used as a Geometry Image as defined in Non-Patent Document 2.

Here, the Geometry Image is encoded using a video encoding technique as specified in Non-Patent Document 2. Also, Connectivity, which is information without a point group, is encoded using the Edgebreaker of Non-Patent Document 4.

For simplicity, the patch encoding unit 103 calculates the rate and distortion using only the geometry image. Also, since connectivity is lossless, distortion is zero.

Note that the patch encoding unit 103 may be configured to project each patch onto a plane and encode the rasterized geometry image using a video encoding technique when counting Rate(M ^f _i ) (Rate using only the Geometry image) in the above-mentioned formula (4).

In this video coding technology, it is possible to set a Q value that controls the amount of code for each patch. In this video coding technology, using a large Q value results in a small amount of code.

The patch encoding unit 103 can set the Q value for each patch in the following steps:

In step 1, the patch encoding unit 103 allocates the total code amount to each patch in proportion to the size of the patch, and sets an initial value Q1 ^f _i of the Q value based on the allocated code amount.

In step 2, the patch encoding unit 103 adjusts the initial value ^Q1f _i of the Q value to ^Q2f _i based on the number of polygons of _{the corresponding mesh Mf i} ^of the corresponding patch. Basically, when the number of polygons is large, the patch encoding unit 103 adjusts the Q value to a small value.

In step 3, since some patches are displayed and some are not displayed depending on the viewer's viewpoint or line of sight, the patch encoding unit 103 adjusts Q2 ^f _i to Q3 ^f _i , which is the final Q value, depending on the visibility of the corresponding patch. Basically, the patch encoding unit 103 adjusts the Q value to a larger value for patches that are not displayed.

As shown in FIG. 4, the mesh decoding device 200 includes a mesh decoding unit 201 and a mesh post-processing unit 202.

The mesh decoding unit 201 is configured to decode a mesh (specifically, vertex coordinates, connectivity, and texture information) in each patch in the manner described in Non-Patent Document 1, and to reconstruct the original number of polygons in the decoded mesh based on the variable f _i selected in each patch.

The mesh decoding unit 201 is configured to calculate the above-mentioned original polygon count (target polygon count) F _i ^orig by the following equation (7).

The mesh decoding unit 201 may be configured to perform the following steps after reconstructing the original polygon count F _i ^orig for one patch M _{i :}

As step 1, the mesh decoding unit 201 is configured to select vertices to be divided in a patch M _i .

Here, the mesh decoding unit 201 is configured to use the edge costs defined by the mesh generation unit 102 as described above.

As shown in FIG. 2, the mesh decoding unit 201 is configured to calculate the cost of an edge uv (one side of a triangle) and select the edge with the highest cost from within the patch M _i .

Here, the mesh decoding unit 201 is configured to add vertices to edges of polygons in non-flat regions or polygons with large areas.

The mesh decoding unit 201 is configured to select any one of the two vertices of the selected edge that is not a boundary vertex as the vertex to be operated on. If both vertices are boundary vertices, the mesh decoding unit 201 sets the cost of the edge to zero and repeats step 1.

In step 2, the mesh decoding unit 201 is configured to perform polygon refinement by splitting the vertices selected in step 1 using an operation called vertex split, as shown in FIG. 3.

According to vertex split, the vertex _Vs in FIG. 3B is the target of division, and is divided into vertices _Vt and _Vs. New edges are created between the vertices Vt and _Vs , between the vertices _Vl and _Vt , and between the vertices _{Vr and Vt.} The original edges e1 and e2 connected to the vertex _Vs are reconnected to the new vertex _Vt .

When splitting a vertex, the mesh decoding unit 201 is configured to set the difference vector between the coordinates of the two newly generated vertices to half the length of the edge uv shown in FIG. 2. Here, when a vertex split is performed once, the number of polygons increases by two.

The mesh decoding unit 201 is configured to repeatedly perform steps 1 and 2 described above until the original number of polygons is reached.

The mesh post-processing unit 202 is configured to perform post-processing on the meshes of adjacent patches.

Specifically, the mesh post-processing unit 202 is configured to perform the following steps:

In step 1, because the polygons at the boundary of the patch have insufficient edges, the mesh post-processing unit 202 is configured to perform the following processing for each boundary edge.

L1, which is the boundary of patch P1 shown in Figure 5, is assumed to be the edge to be processed. Here, it is assumed that patch P1 and patches P2/P3 are actually in contact with each other.

Here, the mesh post-processing unit 202 is configured to find the vertices D1 to D3 that are closest to the vertices V1 to V3 of the polygon of the neighboring patch P2/P3 from the boundary L1.

The mesh post-processing unit 202 is configured to add dotted lines L11 to L13 as new edges if the distance between the vertices D1 to D3 and the closest vertices V1 to V3 is within a threshold.

Note that once the mesh post-processing unit 202 has completed this operation for all boundary edges, it can resolve the problem of insufficient edges between adjacent patches.

In step 2, if a gap occurs at the boundary L1 between patch P1 and patches P2/P3 as shown in FIG. 5 due to lossy compression, the mesh post-processing unit 202 may be configured to fill such gaps using a zippering algorithm, similar to the technique defined in Non-Patent Document 1.

In such a case, the mesh post-processing unit 202 may be configured to find pairs of vertices where gaps occur (in the example of FIG. 6, the pair of vertices V1 and D1, the pair of vertices V2 and D2, and the pair of vertices V3 and D3), as shown in FIG. 6, and to average and merge the vertices of such pairs.

The above-mentioned mesh encoding device 100 and mesh decoding device 200 may be realized as a program that causes a computer to execute each function (each process).

In addition, according to this embodiment, for example, it is possible to realize an improvement in the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

100... mesh encoding device 101... segmentation unit 102... mesh generation unit 103... mesh encoding unit 200... mesh decoding device 201... mesh decoding unit 202... mesh post-processing unit

Claims (8)

A mesh decoding device, comprising:
a mesh decoder configured to decode a mesh in each patch from the encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch;
a mesh post-processing unit configured to perform post-processing on the mesh of the adjacent patch ;
The mesh decoding device is characterized in that the mesh decoding unit is configured to calculate the original number of polygons based on a variable indicating the number of polygons for each patch, and to repeatedly perform vertex split until the calculated original number of polygons is reached . 2. The mesh decoding device according to claim 1 , wherein the mesh decoding unit is configured to calculate a cost of an edge in the patch, and repeat an operation of increasing polygons starting from the edge with the largest cost. A mesh decoding device, comprising:
a mesh decoder configured to decode a mesh in each patch from the encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch;
a mesh post-processing unit configured to perform post-processing on the mesh of the adjacent patch;
The mesh decoding device according to claim 1, wherein the mesh post-processing unit is configured to create new edges for polygons that lack edges at the boundaries of the patches. The mesh decoding device according to claim 3, characterized in that the mesh post-processing unit is configured to find pairs of vertices that create gaps at the boundaries of the patches using a zippering algorithm, and to average and merge the vertices of the pairs . A mesh decoding method, comprising:
A step A of decoding a mesh for each patch from the encoded bitstream and reconstructing the original number of polygons in said mesh based on a variable indicating the number of polygons in each patch;
and B. performing post-processing on the mesh of the adjacent patch ,
In the step A, the original number of polygons is calculated based on a variable indicating the number of polygons for each patch, and vertex split is repeatedly performed until the calculated original number of polygons is reached . A mesh decoding method, comprising:
A step A of decoding a mesh for each patch from the encoded bitstream and reconstructing the original number of polygons in said mesh based on a variable indicating the number of polygons in each patch;
and step B performing post-processing on the mesh of the adjacent patch,
A mesh decoding method, characterized in that in the step B, new edges are created for polygons that lack edges at the boundaries of the patches. A program for causing a computer to function as a mesh decoding device, comprising:
The mesh decoding device comprises:
a mesh decoder configured to decode a mesh in each patch from the encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch;
a mesh post-processing unit configured to perform post-processing on the mesh of the adjacent patch ;
the mesh decoding unit is configured to calculate the original number of polygons based on a variable indicating the number of polygons for each patch, and to repeatedly perform vertex split until the calculated original number of polygons is reached . A program for causing a computer to function as a mesh decoding device, comprising:
The mesh decoding device comprises:
a mesh decoder configured to decode a mesh in each patch from the encoded bitstream and reconstruct the original number of polygons in the mesh based on a variable indicating the number of polygons in each patch;
a mesh post-processing unit configured to perform post-processing on the mesh of the adjacent patch;
The mesh post-processing unit is configured to create new edges for polygons that lack edges at the boundaries of the patches.