JP7615036B2 - High-Level Syntax for Immersive Video Coding - Google Patents

Description

In compression/decompression (codec) systems, compression efficiency and video quality are important performance criteria. For example, visual quality is an important aspect of the user experience in many video applications, and compression efficiency impacts the amount of memory storage required to store a video file and/or the amount of bandwidth required to transmit and/or stream the video content. A video encoder compresses the video information so that more information can be transmitted over a given bandwidth or stored in a given memory space, etc. The compressed signal or data is then decoded by a decoder, which decodes or decompresses the signal or data for display to the user. In most implementations, higher visual quality with greater compression is desired.

Immersive and 3D video coding standards, such as the developing MPEG 3DoF+ standard, encode both texture and depth representations of a scene and then use the coded depth to render intermediate view positions through view interpolation (also called view synthesis). Legacy 2D video coding standards can be used to code texture and depth separately. Most commonly used video coding standards can only support 8-bit or 10-bit input images, while many depth representations have 16-bit representations. Furthermore, a high level of syntax is required for coding efficiency. With regard to these and other considerations, current improvements are needed, which could become important as the desire to encode and decode immersive and 3D video becomes more widespread.

This application claims priority to U.S. Provisional Patent Application No. 62/820,760, filed March 19, 2019, and entitled "HIGH LEVEL SYNTAX FOR IMMERSIVE VIDEO CODING," which is incorporated by reference in its entirety.

The material described herein is presented by way of example and not limitation in the accompanying drawings. For simplicity and clarity of illustration, elements illustrated in the figures have not necessarily been drawn to scale. For example, dimensions of some elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference labels have been repeated among the figures to indicate corresponding or analogous elements.
FIG. 1 is a block diagram of an exemplary encoding system for encoding immersive video data. FIG. 2 is a block diagram of an exemplary decoding system for decoding immersive video data. FIG. 3 shows an exemplary piece-wise linear mapping between depth values of a higher bit-depth and depth values of a lower bit-depth. FIG. 4 shows one exemplary process for reducing the bit depth of a depth view using piecewise linear mapping. FIG. 5 illustrates an example generation of a piecewise linear mapping based on pixel counts of depth values in a depth view. FIG. 6 illustrates one example process for defining a piecewise linear mapping to map and demap depth views between higher and lower bit-depth values. FIG. 7 shows one example process for increasing the bit depth of a depth view using piecewise linear mapping. FIG. 8 shows the camera array and corresponding camera parameters for one exemplary encoding. FIG. 9 shows an exemplary process for encoding immersive video data including camera parameters. FIG. 10 shows an exemplary process for decoding immersive video data including camera parameters. FIG. 11 is an illustration of an example system 1100 for encoding immersive video data. FIG. 12 is an illustration of one exemplary system. FIG. 13 illustrates an example device constructed in accordance with at least some embodiments of the present disclosure.

One or more embodiments or implementations will now be described with reference to the enclosed drawings. While particular configurations and arrangements are described, it should be understood that this is done for illustrative purposes only. Those skilled in the art will appreciate that other configurations and arrangements may be used without departing from the spirit and scope of the present description. Those skilled in the art will appreciate that the techniques and/or arrangements described herein may also be used in a variety of other systems and applications other than those described herein.

Although the following description illustrates various implementations that may be manifested in architectures such as, for example, system-on-chip (SoC) architectures, implementations of the techniques and/or configurations described herein are not limited to a particular architecture and/or computing system, but may be implemented by any architecture and/or computing system for the same purpose. For example, various architectures using multiple integrated circuit (IC) chips and/or packages, and/or various computing devices such as set-top boxes, smartphones, and/or consumer electronic devices may implement the techniques and/or configurations described herein. Furthermore, although the following description may set forth many specific details, such as logical implementations, types and interrelationships of system components, logical partitioning/integration options, and the like, the claimed subject matter may be practiced without such specific details. In other instances, some material, such as, for example, control structures and complete software instruction sequences, may not be shown in detail so as not to obscure the material disclosed herein.

The material disclosed herein may be implemented in hardware, firmware, software, or any combination thereof. The material disclosed herein may also be implemented as instructions stored on a machine-readable medium that may be read and executed by one or more processors. A machine-readable medium may include any medium and/or mechanism for storing or transmitting information in a form readable by a machine (e.g., a computing device). For example, a machine-readable medium may include read-only memory (ROM), random access memory (RAM), magnetic disk storage media, optical storage media, flash memory devices, electrical, optical, acoustic, or other forms of propagated signals (e.g., carrier waves, infrared signals, digital signals, etc.), and the like.

References in the specification to "one implementation," "an implementation," "an example implementation," and the like indicate that the described implementation may include a particular feature, structure, or characteristic, but not all embodiments may necessarily include the particular feature, structure, or characteristic. Moreover, such phrases do not necessarily refer to the same implementation. Furthermore, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one of ordinary skill in the art to implement such feature, structure, or characteristic in connection with other embodiments, whether or not explicitly described herein.

The terms "substantially,""close,""approximately," and "about" generally mean within +/- 10% of a target value. For example, the terms "substantially equal,""approximately _equal ," and "approximately equal" mean that there is no accidental variation between those so described, unless otherwise specified in the express context of their use. In the art, such variation is typically no more than +/- 10% of a given target value. Unless otherwise specified, the use of the common adjectives "first,""second,""third," etc., to describe a common object is merely to indicate that different instances of a similar object are being referenced to describe the common object, and is not intended to imply that the objects so described must be in a given sequence, either temporally, spatially, in order, or in any other manner.

Methods, devices, apparatus, computing platforms, and articles of manufacture are described herein in connection with immersive and 3D video encoding.

As described, immersive and 3D video coding standards encode both texture and depth representations of a scene and can use the coded depth to render intermediate view positions through view interpolation or synthesis. As used herein, the term immersive video refers to a video that includes multiple pairs of texture and depth views of a scene. Conventional 2D video coding standards can be used to code texture and depth separately and typically support 8-bit or 10-bit input images. The techniques described herein improve the quality of immersive video coding by applying a piece-wise linear mapping of a depth view (e.g., a depth image, frame, or map) in a 16-bit representation to a 10-bit representation before encoding the depth in a 10-bit video coding standard, such as the HEVC Main 10 profile. Although described with respect to HEVC Main 10 for clarity of presentation, any suitable codec may be applied. Additionally, although described with respect to a mapping of 16-bit to 10-bit representation, the input and output bit depths may be any suitable bit depth. Additionally, a high level syntax is provided to efficiently describe additional camera parameters used in view interpolation. Such modifications may be made to HEVC video parameter set extensions and/or new profiles may be provided to indicate support for the 3DoF+ standard. Again, while described with respect to HEVC for clarity of presentation, such modifications and/or new profiles may be implemented with respect to any suitable codec.

The techniques described herein improve video quality and coding efficiency for the developing MPEG 3DoF+ standard. However, they may be applied to other immersive video codec systems that include texture and depth views and where depth is available at a higher bit representation than can be coded. In some embodiments, at the encoder, before coding the depthwise view, the 16-bit representation is mapped to a 10-bit representation using a piecewise linear mapping with metadata describing the piecewise linear mapping signaled in the bitstream. At the decoder, after decoding the 10-bit depth view, an inverse mapping is performed to map to a 16-bit representation, which is used for view synthesis. In particular, the 10-bit representation of depth is insufficient to fully represent the depth and may lead to visual artifacts when using the depth for view synthesis. Furthermore, rounding or discarding the least significant bits to represent a 16-bit bit depth as 10 bits also introduces artifacts. The disclosed technology mitigates or solves such problems using a described 16-bit to 10-bit mapping with metadata describing the piecewise linear mapping signaled in the bitstream. This results in improved objective and subjective video quality using the described technology (e.g., typical test conditions for the developing 3DoF+ standard).

Furthermore, the techniques described herein provide efficient signaling for carrying metadata of additional camera parameters in multi-camera immersive video systems. In some embodiments, a new HEVC immersive profile is provided to indicate support for the 3DoF+ standard. In some embodiments, a modification of the HEVC video parameter set (VPS) is proposed to indicate the use of an atlas. For example, an atlas may be formed such that a single image encoded with a video codec such as HEVC has many regions that combine data from multiple different camera positions, as described in Renaud Dore and Frank Suder, “Outperforming 3DoF+ anchors: first evidence”, M43504, July 2018. In particular, the techniques described reduce the overhead for carrying camera parameters in multi-camera systems, allowing more camera representations to be used in the bitstream, resulting in higher video quality.

FIG. 1 is a block diagram of an example encoding system 100 for encoding immersive video data, configured in accordance with at least some implementations of the present disclosure. FIG. 2 is a block diagram of an example decoding system 200 for decoding immersive video data, configured in accordance with at least some implementations of the present disclosure. That is, FIGS. 1 and 2 show block diagrams of an encoding system and a client system (e.g., a decoding system or decode system), respectively.

1, the encoding system 100 provides video compression, and the encoding system 100 may be part of a video encoding system implemented via a computing device, such as a computer or server system. For example, the encoding system 100 receives video, e.g., including texture and depth from various views (e.g., from viewpoints of a view or scene, respectively), including texture video 101 and depth video 102 from a selected view 142, texture video 103 and depth video 104 from one of the selected views 142, texture video 105 and depth video 106 from another one of the selected views 142 (as well as any additional texture videos and depth videos from other selected views 142).
Encoding system 100 encodes such texture and depth videos to generate a bitstream 131 that can be decoded by a decoder to generate decompressed versions of such texture and depth videos (as described below), and, optionally, generates a synthesized view of the scene represented by base view 141 and selected view 142. In particular, the synthesized view does not necessarily correspond to any such view, and thus may represent a view of the scene not encoded by encoding system 100.

For example, texture encoding module 111 receives texture video 101 (e.g., texture view), which may be of 8-bit or 10-bit bit depth, and texture encoding module 111 may encode texture video 101 using a standard-compliant codec and profile to generate a corresponding bitstream. Similarly, texture encoding module 113 receives texture video 103 of 8-bit or 10-bit bit depth, and texture encoding module 113 encodes texture video 103 using a standard-compliant codec and profile to generate a corresponding bitstream. And texture encoding module 115 receives texture video 105 of 8-bit or 10-bit bit depth, and encodes it to generate a corresponding bitstream. In particular, texture videos 101, 103, 105 have a bitdepth that matches the bitdepth used by texture encoding modules 111, 113, 115. Although described with respect to a bitdepth of 8 or 10 bits, any standard bitdepth may be used.

However, the depth videos 102, 104, 106 are at a bit depth that is not used by the standard encoding module. In particular, the depth encoding modules 112, 114, 116 also operate at a standard bit depth of 8 or 10 bits, but the depth videos 102, 104, 106 are received at a higher bit depth, such as 16 bits. As shown, before being encoded by the depth encoding modules 112, 114, 116, each depth video 102, 104, 106 is mapped by mapping modules 121, 123, 125, respectively, from a higher bit depth (e.g., 16 bits) to a lower bit depth that also conforms to a standard encoding profile (e.g., 10 bits or 8 bits with the 10 bits shown in FIG. 1). Such bit depth mapping provides, for example, a 10-bit or 8-bit depth view from a 16-bit depth view. Such bit depth mapping is applied in a piecewise linear fashion based on mapping parameters 161 (e.g., end points of line segments representing piecewise linear mapping functions) selected by a mapping parameter selection module 110 based on the depth videos 102, 104, 106, as described further below.

As shown, a resulting depth video (or depth view) including depth videos 122, 124, 126 is generated based on the depth videos 102, 104, 106, respectively, such that the depth videos 122, 124, 126 are 8-bit or 10-bit depth videos or depth views. As used herein, the terms depth video and depth view are used interchangeably to refer to a video having video frames that are depth frames that include per-pixel depth values, i.e., each pixel represents a depth value. Such low bit depth depth videos 122, 124, 126 are then encoded by depth encoding modules 112, 114, 116 using a standard-based codec and profile, such as HEVC Main 10, to generate a corresponding bitstream. Additionally, as used herein, texture view, texture video, and similar terms refer to color video with those pixels having a luma value and multiple color values, three colors, or similar representations. For example, such texture video may include any suitable video frames, video pictures, etc., in any suitable resolution, such as video graphics array (VGA), high definition (HD), full HD (e.g., 1080p), 4K resolution video, 8K resolution, etc. Such texture video may be in any suitable color space, such as YUV. During encoding, the depth video and texture video may be split into coding units, prediction units, transform units, etc., and encoded using intra and inter techniques known in the art.

The resulting bitstreams from the depth coding modules 112, 114, 116 (e.g., representatives of depth video) and the texture coding modules 111, 113, 115 (e.g., representatives of texture video) are received by a multiplexer 119, which multiplexes the bitstreams into a bitstream 131 for storage, transmission, etc., and ultimately for use by a decoder, as described further below. The bitstream 131 may be compatible with a video compression-decompression (codec) standard, such as, for example, HEVC. Although described herein with respect to HEVC, the disclosed techniques may be implemented with respect to any codec, such as AVC (Advanced Video Coding/H.264/MPEG-4 Part 10), VVC (Versatile Video Coding/MPEG-I Part 3), VP8, VP9, Alliance for Open Media (AOMedia) Video1 (AV1), VP8/VP9/AV1 family, etc. The encoding system 100 may be implemented via any suitable device, such as a server, a personal computer, a laptop computer, a tablet, a phablet, a smartphone, a digital camera, a gaming console, a wearable device, a display device, an all-in-one device, a two-in-one device, etc. For example, as used herein, a system, device, computer, or computing device may include any such device or platform.

As illustrated, in the encoding system 100 of FIG. 1, multiple views with texture and depth are available, where the texture is 8 or 10 bits, and the depth is 16 bits. The encoding system selects a subset of the views to encode. In the illustrated embodiment, an HEVC encoder is used. For the selected views, the texture is encoded in the available bit representation, and the depth is converted from a 16-bit representation, x, to a 10-bit representation, y. In conventional systems, such a conversion may be performed according to the following equation: y=round(1024*x/65536). Alternatively, truncation may be employed instead of rounding, and clipping may also be applied, for example to avoid values outside the allowed range, such as the following equation: y=Clip3(0,1023,int(1024*x÷65536+0.5). Such conventional techniques unfavorably lose information in the bit depth reduction.

In contrast, as shown in FIG. 1, the encoding system 100 converts the depth videos 102, 104, 106 from higher bit depths (e.g., 16-bit representations) to lower bit depths (e.g., 10-bit representations) via mapping modules 121, 123, 125 in the encoding system 100 using a piecewise linear mapping that is applied such that some ranges of depth values are represented with finer granularity than other ranges of depth values. Such bit depth mapping is described further below.

In some embodiments, the mapping parameter selection module 110 of the encoding system 100 selects parameters for a piecewise linear mapping. As shown, such mapping parameters 161 are provided to the mapping modules 121, 123, 125 for implementation in the piecewise linear mapping. Furthermore, the mapping parameters 161 (e.g., metadata representing the mapping parameters) are signaled and encoded via the metadata encoding module 117 and multiplexed with the encoded bitstream into the bitstream 131 via the multiplexer 119 for use by the decoder. In one embodiment, the mapping metadata can be conveyed using a supplemental enhancement information message or other high-level syntax structure in the video bitstream or system layer. In some embodiments, the mapping parameters 161 (e.g., mapping metadata) include or represent a list of (x,y) coordinates of the end points of the line segments, where x represents a 16-bit deep representation value and y represents a transformed 10-bit representation value.

Additionally, camera array parameters 151, the base view 141 used to capture and the selected view 142, as well as any other views of the scene, may be provided to the camera parameter encoding module 118 for encoding. The resulting bitstream is then provided to the multiplexer 119 for inclusion in the bitstream 131. The camera array parameters 151 include parameters representative of the camera during video capture, such as the camera's position (e.g., x, y, z information), the camera's orientation in space (e.g., yaw, pitch, and roll information, or quaternion information), and camera characteristics such as focal length, field of view, and Znear and Zfar, which indicate the physical distance from the camera, with the lowest and highest 16-bit values (0 and 65535) corresponding to disparity depth files, etc. The camera parameter encoding module 118 may advantageously encode such camera array parameters 151 to reduce redundancy and improve efficiency, as described further herein below.

2, the decoding system 200 may be implemented, for example, as a client system. Similar to the encoding system 100, the decoding system 200 may be implemented via any suitable device, such as, for example, a server, a personal computer, a laptop computer, a tablet, a phablet, a smartphone, a digital camera, a gaming console, a wearable device, a display device, an all-in-one device, a two-in-one device, and the like. For example, as used herein, a system, a device, a computer, or a computing device may include any such device or platform. The decoding system 200 receives the bitstream 131 from any source, and the decoding system 200 demultiplexes the bitstream 131 for decoding.

As shown, each demultiplexed bitstream from the demultiplexer 219 is provided to a corresponding module, including one of texture decoding modules 211, 213, 215, depth decoding modules 212, 214, 216, metadata decoding module 217, and camera parameter decoding module 218. As shown, the texture decoding module 211 receives and decodes the bitstream to generate a texture video 201 corresponding to the texture video 101. The texture decoding module 213 receives and decodes the bitstream to generate a texture video 203 corresponding to the texture video 103. And, the texture decoding module 215 receives and decodes the bitstream to generate a texture video 205 corresponding to the texture video 105. Note that the texture videos 201, 203, 205 correspond to the texture videos 101, 103, 105, but do not perfectly replicate them due to the lossy encoding/decoding process. Such decoding may be performed using any suitable technology or technique. In particular, such decoding provides texture videos 201, 203, 205 at a bit depth (e.g., 8-bit or 10-bit) suitable for use by view synthesis module 231 in generating a synthetic view 250 for use by a head mounted display (HMD) 251 (or other display device) based on head mounted display position information (HMD pos. info.) 252.

Furthermore, the metadata decoding module 217 receives the appropriate bitstream and decodes the mapping parameters 161 for use by the inverse mapping modules 221, 223, 225 of the decoding system 200. In particular, such decoding faithfully recreates the mapping parameters 161, which include a list of (x,y) coordinates or similar data structure of the line segment end points, where x represents a 16-bit deep representation value and y represents a transformed 10-bit representation value. For example, the inverse mapping modules 221, 223, 225 reverse the process performed by the mapping modules 121, 123, 125 to generate a higher bit depth video from a lower bit depth video.

表1：例示的なSEIシンタックスおよびセマティック In some embodiments, the line segment endpoint metadata is sent in a depth mapping SEI message or other high level syntax or system signaling. In some embodiments, more than one line segment endpoint set is signaled. In some embodiments, an SEI message lasts until it is canceled or until another SEI message with new parameters is sent to replace it. Exemplary SEI syntax and semantics are provided in Table 1 and in the description below.

Table 1: Example SEI syntax and semantics

In some embodiments, the SEI message parameters remain in coding order until canceled. In Table 1, the parameters may be implemented as follows:
dm_cancel_flag equal to 1 indicates that the persistence of any previous depth mapping SEI messages is canceled.
dm_num_mapping_sets specifies the number of linear mapping sets per piece that are signaled.
dm_num_segments[i] specifies the number of line segments in the i-th mapping set.
dm_seg_x[i][j] and dm_seg_y[i][j] specify the jth (x,y) line segment end points defined in the ith mapping set.

As explained further below, for each sample in the depth-coded image that uses the i-th mapping set, DecodedValue, the variable InverseMappedValue is determined using:
for(j=0;<jdm_num_segments[i];j++){
if(DecodedValue>=dm_seg_y[i][j]&&DecodedSampleValue<=
dm_seg_y[i][j+1])
InverseMappedValue=Clip3(0,65535,int(dm_seg_x[i][j]+0.5+
((DecodedValue-dm_seg_y[i][j])÷(dm_seg_x[i][j+1])-
dm_seg_x[i][j])
*(dm_seg_y[i][j+1]-dm_seg_y[i][j])))

For example, the previous code or pseudocode may be used to determine the variable InverseMappedValue for each sample in the depth-coded image using the ith mapping set, DecodedValue, so that 10-bit bit-depth values may be mapped to 16-bit bit-depth values for use by the decoding system in view synthesis.

Continuing to refer to FIG. 2, the depth decoding module 212 receives and decodes the bitstream to generate a depth video 222, which corresponds to the depth video 122, such that the depth video 222 is at a lower bit depth, such as 8 or 10 bits. In particular, the depth video 222 may be unsuitable for use by the view synthesis module 231 due to its low bit depth. The depth video 222 and the mapping parameters 161 are received by an inverse mapping module 221, which maps the depth video 222 to the depth video 202 (at a lower bit depth), such as 16 bits, as shown. The depth video 222 corresponds to the depth video 202, although it is not a perfect replication due to the lossy encoding/decoding process. In particular, the inverse mapping module 221 performs an inverse piece-wise linear mapping with respect to the piece-wise linear mapping performed by the mapping module 121. As shown, mapping parameters 161 are used in both contexts. Such mapping parameters 161 retain more information in some depth ranges of the depth map while losing more information in other depth ranges. Such piecewise linear mapping may be contrasted with rounding or truncation techniques, which in effect lose the same amount of information in all depth ranges.

Similarly, the depth decoding module 214 receives and decodes the bitstream to generate a depth video 224 corresponding to the depth video 124 such that the depth video 224 is at a lower bit depth (e.g., 8 or 10 bits). The depth video 224 and the mapping parameters 161 are received by an inverse mapping module 223, which maps the depth video 224 to the depth video 204 at a higher bit depth using an inverse piece-wise linear mapping technique described herein. Additionally, other views are processed similarly. For example, the depth decoding module 216 receives and decodes the bitstream to generate a depth video 226 corresponding to the depth video 126 such that the depth video 226 is at a lower bit depth. The depth video 226 and the mapping parameters 161 are used by an inverse mapping module 225 to map the depth video 226 to the depth video 206 at a higher bit depth using an inverse piece-wise linear mapping technique.

Furthermore, the camera parameter decoding module 218 receives the associated bitstream and decodes the camera array parameters 151 used by the view synthesis module 231. Such decoding faithfully reproduces the camera array parameters 151, including, for example, parameters representative of the cameras used by the view synthesis module 231 to generate the views. For example, the camera array parameters 151 may include camera positions (e.g., x, y, z information), camera orientations in space (e.g., yaw, pitch, roll information or quaternion information), and camera properties such as focal length, field of view, Znear, Zfar, etc.

The view synthesis module 231 receives the camera array parameters 151 and head mounted display position information 252, or other information, that represents a desired view within the playback scene, and the view synthesis module 231 generates the view 250. The view 250 may include any suitable data structure for the presentation of a scene via a display, such as a head mounted display 251. A user wears the head mounted display 251, and the user can move the head mounted display 251 spatially. Based on the position and orientation of the head mounted display 251, head mounted display position information 252 is generated to identify a desired virtual view within the scene. Alternatively, the position and orientation at which the virtual view is desired (e.g., a virtual view camera or viewport) may be provided by another system or device to generate the view 250. The view synthesis module 231 can generate the view 250 using any suitable technology or technique based on the texture videos 201, 203, 205, the depth videos 202, 204, 206, the camera array parameters 151, and the head mounted display position information 252 (or other virtual view data). The view 250 is then displayed to the user to provide an immersive viewer experience within the scene.

In the client system shown in Figure 2, a bitstream containing compressed depth and texture views is decoded (e.g., demultiplexed and decoded). View synthesis is performed based on a selected view position and orientation (e.g., identified by head mounted display movement or other view selection indicated by HMV position information) and uses the decoded texture video and the decoded, inverse bitmapped depth video to generate a view for the display viewport.
For example, the decoded 10-bit deep view may be converted to reconstruct a 16-bit representation (or a higher bit integer or floating point representation) using a piecewise inverse linear function. View synthesis may then be performed based on the selected view position and orientation (e.g., identified by head mounted display movement or other view selection indicated by HMV position information) using the reconstructed 16-bit deep representation data (or a higher bit integer or floating point representation) and the Znear and Zfar parameters.
In some embodiments, the client system may combine the inverse mapping and view synthesis operations and perform the inverse mapping calculations only when needed for the view synthesis operation.

As described, mapping modules 121, 123, 125 provide a piecewise linear mapping from higher bit depth values to lower bit depth values, and inverse mapping modules 221, 223, 225 reverse the process to provide higher bit depth values from lower bit depth values. We now turn to the application of piecewise linear mapping and piecewise inverse linear mapping. In one embodiment, a list of line segment endpoints is selected by the encoding system and transmitted via the bitstream. In some implementations, N+1 (x,y) data pairs are used to represent N line segments.

FIG. 3 illustrates an exemplary piecewise linear mapping 300 between higher and lower bit depth values configured in accordance with at least some implementations of the present disclosure. In the example of FIG. 3, the piecewise linear mapping 300 utilizes three line segments 307, 308, 309, represented by ( _x0 , _y0 ), ( _x1 , _y1 ), ( _x2 , _y2 ), and ( _x3 , _y3 ). However, any number of line segments may be used. In some embodiments, a piecewise linear function is used because it has a relatively low complexity for a decoder to perform the inverse mapping function and does not require a large amount of metadata for signaling. In some embodiments, the number of line segments of the piecewise linear function is limited to a predefined number to limit the decoder complexity and metadata signaling overhead.

In the illustrated example, line segment 307 is defined as the line segment between endpoint 303 ( _x0 , _y0 ) and endpoint 304 ( _x1 , _y1 ). Similarly, line segment 308 is defined as the line segment between endpoint 304 ( _x1 , _y1 ) and endpoint 305 ( _x2 , _y2 ), and line segment 309 is defined as the line segment between endpoint 305 ( _x2 , _y2 ) and endpoint 306 ( _x3 , _y3 ). As used herein, the term endpoint refers to the points that define the ends of a line segment.

In particular, the piecewise linear mapping 300 maps between high bit-depth values 302 and low bit-depth values 301. For example, the high bit-depth values 302 may be 16-bit depth values and the low bit-depth values 301 may be 8-bit or 10-bit depth values, as described herein. In such an embodiment, the high bit-depth values 302 of a depth view (e.g., any of the depth videos 102, 104, 106) have a bit-depth of 16 bits and therefore have a range of available values from 0 to 65535 (e.g., =2^16-1). Such high bit depth values 302 of the depth view are then forward mapped to low bit depth values 301 of the depth view (e.g., any of the depth videos 122, 124, 126), such that the low bit depth values 301 may be, for example, 8-bit deep or 10-bit deep. In an 8-bit depth context, the low bit depth values 301 have an available range of 0 to 255 (e.g., =2^8-1). In a 10-bit depth context, the low bit depth values 301 have an available range of 0 to 1023 (e.g., =2^10-1).

Furthermore, at least the endpoints 304, 305 are between the minimum and maximum values of the components of the high bit-depth value 302 and the low bit-depth value 301. For example, the range of the high bit-depth value 302 from 0 to 65535 may define the horizontal or x component of the piecewise linear mapping 300, and the range of the low bit-depth value 301 from 0 to 255 or 1023 may define the vertical or y component of the piecewise linear mapping 300. The endpoints 303, 304, 305, 306 are between (0,0) and (65535,255) or (65535,1023), inclusive, so that the endpoint 303 may be at (0,0) and the endpoint 306 may be at (65535,255) or (65535,1023), although other final endpoints within the available ranges may be defined. Intermediate endpoints, such as endpoints 304, 305, are then between such minimum and maximum endpoints. Thus, the horizontal or x-components of both endpoints 304, 305 exceed the minimum depth value (e.g., 0) of the higher bit depth range, and the vertical or y-components of both endpoints 304, 305 exceed the minimum depth value (e.g., 0) of the lower bit depth range. Similarly, the horizontal or x-components of both endpoints 304, 305 are less than the maximum depth value (e.g., 65535) of the higher bit depth range, and the vertical or y-components of both endpoints 304, 305 are less than the maximum depth value (e.g., 255 or 1023) of the lower bit depth range. As used herein, the term horizontal component of a point corresponds to a coordinate along the high bit depth value 302, and the term vertical component of a point corresponds to a coordinate along the low bit depth value 301.

As shown, endpoints 303, 304 define line segment 307, endpoints 304, 305 define line segment 308, and endpoints 305, 306 define line segment 309. Thus, line segments 307, 308 share endpoint 304, and line segments 308, 309 share endpoint 305. As described, line segment 307 has a greater slope than line segment 308, which in turn has a greater slope than line segment 309. Such a slope may be generated in response to a depth value pixel count in the depth view defined by endpoints 303, 304, 305, 306 and used to generate piecewise linear mapping 300, as described further below.

In some embodiments, forward mapping is performed by determining which of the line segments 307, 308, 309 corresponds to a particular high bit-depth value of the high bit-depth values 302, and then determining the mapped low bit-depth value by applying a linear mapping according to the end points of the line segments. For example, as shown for high bit-depth value 311, it is determined that the high bit-depth value 311 is within the line segment 308. The high bit-depth value 311 is then mapped to the low bit-depth value 312 using a linear mapping based on the line segment 308. That is, the high bit-depth value 311 is mapped to the low bit-depth value 312 by applying a linear function defined by the line segment 308 to the high bit-depth value 311 to determine the resulting low bit-depth value 312.

In some embodiments, for x in the range [x _i , x _i+1 ], high bit-depth values (e.g., 16-bit x values) are mapped to low bit-depth values (e.g., 8-bit or 10-bit values) according to equation (1) as follows:
y=round(y _i +(xx _i )/(x _i+1 -x _i ))*(y _i+1 -y _i )) (1)
Here, x is in the range [x _i ,x _i+1 ] (x is the high value of the bit depth), and [x _i ,x _i+1 ] and
A linear mapping is provided between [ _yi , yi ₊₁ ] (defining the line segment that falls within the range as the high bit-depth value, x). For example, for a high bit-depth value 311(x), a low bit-depth value 312(y) may be defined as y= _y2 +(( _xx1 )/( _x2 - _x1 ))*( _y2 - _y1 ).

Inverse mapping is performed in the decoding system 200 using a similar technique. In some embodiments, the inverse mapping is performed by determining which of the line segments 307, 308, 309 corresponds to a particular low bit-depth value of the low bit-depth values 302, and then determining the mapped high bit-depth value by applying a linear mapping (e.g., an inverse linear mapping) according to the end points of the line segments. For example, as shown for low bit-depth value 312, it is determined that the low bit-depth value 312 is within the line segment 308. The low bit-depth value 312 is then mapped to the high bit-depth value 311 using a linear mapping based on the line segment 308 by fitting the linear function defined by the line segment 308 to the low bit-depth value 312 to determine the resulting high bit-depth value 311.

In some embodiments, for y in the range [ _yi , _yi+1 ], low bit-depth values (e.g., 8-bit or 10-bit y values) are mapped to high bit-depth values (e.g., 16-bit values) according to the inverse of equation (1). For example, for a low bit-depth value 312(y), the high bit-depth value 311(x) may be determined as x= _x2 +(( _yy1 )/ _y2 - _y1 ))*( _x2 - _x1 ).

Such mapping and/or inverse mapping may be performed using any suitable computational technique or technique. In some embodiments, a lookup table may be used for either or both of the forward or inverse mapping functions to map a particular input value (x value) to a particular output value (y value) using the mapping table. In some embodiments, truncation may be used in the above equations instead of rounding. Clipping may also be applied to the above equations, for example, to avoid values outside of an acceptable range.

In particular, it is expected that the high bit-depth values 311 will lose precision as a result of mapping from the high bit-depth values 311 to the low bit-depth values 312. However, by applying the piecewise linear mapping 300 with the line segments 307, 308, 309, the precision loss can be advantageously varied across the depth values 302. For example, the line segments 307, 308, 309 with larger slopes lose less precision than the line segments 307, 308, 309 with smaller slopes. This is because the larger slope line segments have relatively lower bit-depth values (or bins) to map to compared to the smaller slope line segments. This can be visually seen with respect to the line segments 307, 309. The line segment 307 has a larger slope, and as a result, the high bit-depth values in the range from _x0 to _x1 have a relatively higher number of corresponding low bit-depth values in the range from _y0 to _y1 . This is in contrast to line segment 309, which covers an equal size of high bit-depth values from _x2 to _x3 (e.g., _x1 - _x0 = _x3 - _x2 ), but has smaller low bit-depth values from _y2 to _y3 (e.g., _y1 - _y0 > _y3 - _y2 ). Thus, the mapping for line segment 309 converts the same number of high bit-depth values to a smaller number of low bit-depth values, resulting in a larger loss of precision, as reflected in the inverse mapping from low to high bit-depth values. Such a line segment may be advantageous when the depth view has more objects and/or details (e.g., edges) with less depth compared to objects with more depth. Thereby, more details are preserved in the depth view by employing a piecewise linear mapping. The selection of endpoints 303, 304, 305, 306 that define line segments 307, 308, 309 for optimal mapping and precision loss is further described below. Note that equally spaced endpoints in the x dimension are shown merely for clarity of explanation. Such endpoint spacing is not actually applied as a constraint.

₃ , the slope of line segment 307 is greater than the slope of line segment 309. In some embodiments, such a slope difference may be generated (for a corresponding depth image) in response to a histogram showing more samples (e.g., pixel samples) in the range _x0 ... _x1 than in the range _x2 ...x3. That is, a first slope of the first range is greater than a second slope of the second range in response to a greater number of samples in the first range compared to the second range.

1 and 2, according to the technique described with respect to FIG. 3, mapping modules 121, 123, 125 map depth videos 102, 104, 106 (higher bit depth) to depth videos 122, 124, 126 (lower bit depth), and inverse mapping modules 221, 223, 225 map depth videos 222, 224, 226 (lower bit depth) to depth videos 202, 204, 206 (higher bit depth). For example, for each depth value, mapping is performed as described with respect to high bit depth value 311 and low bit depth value 312.

FIG. 4 illustrates one example process 400 for reducing the bit depth of a depth view using piecewise linear mapping, arranged in accordance with at least some implementations of the present disclosure. Process 400 may include one or more operations 401-406, as illustrated, performed, for example, by encoding system 100. Process 400 begins with operation 401, where bit depth reduction is initiated. Bit depth reduction may be performed, for example, by mapping modules 121, 123, 125, which map depth videos 102, 104, 106 (higher bit depths) to depth videos 122, 124, 126 (lower bit depths).

Processing continues at operation 402, where a depth view at a high or higher (compared to a lower target bit depth) bit depth is received. The depth view may have any suitable data structure, such as a depth frame or image having a depth value (at the higher bit depth) for each pixel thereof. In some embodiments, the depth views have depth values each having a bit depth of 16 bits. In such a context, each depth value may range from 0 to 65535, as described. For example, at operation 402, a first depth view having depth values at a high bit depth may be received such that the high bit depth has a large range of available values from a minimum value (e.g., 0) to a maximum value (e.g., 65535).

Processing continues at operation 403, where line segment endpoints are obtained to map a depth view at a higher bit depth to a lower bit depth, such that the line segment endpoints define line segments for the mapping, such as line segments 307, 308, and 309. Such line segment endpoints and line segments may be obtained or generated using any suitable technology or technique, such as those described with respect to FIGS. 5 and 6. In some embodiments, the line segments are determined by generating a histogram of depth pixel value counts per depth value range using at least a portion of one or more depth views (e.g., depth frames), and by recursively generating multiple line segments for the mapping, including first and second line segments, to minimize reconstruction errors corresponding to the mapping and inverse mapping.

Processing continues at operation 404, where the high or higher bit depth view is mapped to a lower bit depth to generate a depth view having a reduced bit depth. The lower bit depth may be any bit depth, such as 8 bits or 10 bits. Such mapping is performed according to the line segment endpoints obtained at operation 403. For example, for each pixel to the higher bit depth view, a determination may be made as to which of the multiple line segments the piecewise linear mapping of the pixel belongs to (e.g., such that the higher bit depth value is within the range of the bit depth values of the line segment). A linear mapping according to the appropriate line segment is then provided to map the higher bit depth value to the lower bit depth value. Such a process may be repeated serially or in parallel for all pixels of a higher bit depth view (e.g., one of depth videos 102, 104, 106) to generate a lower bit depth view (e.g., one of depth videos 122, 124, 126) of 8 bits (range 0 to 255) or 10 bits (range 0 to 1023).

For example, referring to FIG. 3, the depth values of a higher depth view may be mapped to depth values at a lower bit depth that is less than the higher bit depth to generate another depth view. Thus, the lower bit depth has a range of available values from a minimum value (e.g., 0) to a maximum value (e.g., 255 or 1,023),
A first line segment endpoint (e.g., (x1, _y1 )) and a second line segment endpoint (e.g., (x2, _y2 )) are used to define a line segment for mapping such that the horizontal component (e.g., _x1 ) and vertical component (e.g., _y1 ) of the first line segment endpoint exceed the minimum depth value (e.g., 0) of the higher bit depth and the minimum depth value (e.g., 0) of the lower bit depth, respectively. Furthermore, the horizontal component (e.g., _{x2 )} and vertical component (e.g., _y2 ) of the second line segment endpoint are less than the maximum depth value (e.g., ₆₅₅₃₅ ) of the higher bit depth and the maximum depth value (e.g., 255 or 1023) of the lower bit depth, respectively.

Processing continues at operation 405, where the lower bit depth views are output for encoding using a standard encoder implementing a standard codec and/or profile. In some embodiments, the lower bit depth views may be encoded using an HEVC Main Profile encoder. As shown, process 400 may end at operation 406. Process 400 may be performed serially or in parallel for any number of depth views. Such depth views may also be characterized as depth frames, depth pictures, etc. As described herein, such reduced bit depth views may be encoded using standard codecs and profiles by virtue of such codecs and profiles being able to handle the reduced bit depth.

As described with respect to operation 403, line segment endpoints may be generated in encoding system 100 to define a piecewise linear mapping from a higher bit depth to a lower bit depth. Such line segment endpoints are used in the mapping and transmitted to decoding system 100 for use in decoding. We now turn to generating line segments and line segment endpoints for use in the forward and reverse piecewise linear mappings.

5 illustrates the generation of an example piecewise linear mapping 300 based on pixel counts of depth values in a depth view, configured in accordance with at least some implementations of the present disclosure. As shown in FIG. 5, the range of high bit depth values 302 (e.g., from 0 to 65535) can be divided into a number of bins 505, such as histogram bins. Although illustrated with respect to 12 bins 505, any number can be used. For a particular portion of a depth view (e.g., one or more portions of depth views 102, 104, 106), the pixels are assigned to one of the bins 505 based on their depth values. That is, a pixel count is generated for each bin 505 based on the corresponding range of depth values. Such pixel counts generate a histogram 501 indicative of depth value pixel counts. Such pixel counts are then used to generate endpoints and corresponding line segments, as described further below.

5, line segment 307 has a slope that is greater than the slope of line segment 308 in response to a pixel count in bin 502 corresponding to line segment 307 having a greater value than the pixel count in bin 503 corresponding to line segment 308. Similarly, segment 3087 has a slope that is greater than the slope of line segment 309 in response to a pixel count in bin 503 having a greater value than the pixel count in bin 504 corresponding to line segment 308. Note again that equally spaced x-dimension endpoints are shown merely for clarity of illustration. Such endpoint spacing is not a constraint that is actually applied, but such endpoint spacing may be implemented.

In particular, the slopes of the line segments 307, 308, 309 and the coordinates of the endpoints 303, 304, 305, 306 are generated based on pixel counts of the corresponding bins 502, 503, 504, as generated by a histogram 501 or similar counting technique. The portion of the depth view 102, 104, 106 used to generate the histogram 501 may be any suitable portion. In one embodiment, the histogram 501 is generated for each frame. In some implementations, the histogram 501 is generated using a group of images from one of the depth views 102, 104, 106. In some implementations, the histogram 501 is generated based only on edge regions of a single frame or multiple frames. The corresponding line segments 307, 308, 309 and the coordinates of the endpoints 303, 304, 305, 306 are then used for mapping and inverse mapping, and the coordinates are represented in the bitstream 131 and transmitted.

FIG. 6 illustrates one example process 600 for defining a piecewise linear mapping to map and demap depth views between high and low bit depth values, configured in accordance with at least some implementations of the present disclosure. Process 600 may include one or more operations 601-606 as shown, performed, for example, by encoding system 100. Process 600 begins with operation 601, where piecewise linear mapping generation is initiated. For example, piecewise linear mapping model generation may be performed by mapping parameter selection module 110 to generate mapping parameters 161.

Processing continues at operation 602, where depth view data for mapping is acquired. As described, a portion of one or more depth directions 102, 104, 106 is used to generate a piecewise linear mapping. In some embodiments, a single frame is used to determine the piecewise linear mapping. In some embodiments, a preselected number of frames are used. In some embodiments, a group of images (e.g., as defined in the coding structure) is used. In some embodiments, only edge portions of a single frame, a preselected number of frames, or an edge portion of a group of images are used. Additionally, a single depth view may be used to generate mapping parameters for each view, and multiple views may be used to generate mapping parameters shared between views.

As described above, the encoding system 100 selects the piecewise linear mapping used such that a 10-bit representation of the depth is used instead of a 16-bit representation due to limitations of using a 10-bit video encoder, such as the HEVC Main 10 Profile. The same technique can be used to convert 16-bit data to 8-bit data for use with an 8-bit video codec, such as the HEVC Main Profile. Additionally, the described technique can be used to convert n-bit data to m-bit data, where n is greater than m. When rounding or truncating 16-bit data to 10-bit data, the quality of view synthesis is reduced due to the coarser representation of the depth data. Additional steps of encoding and decoding may further reduce the video quality. Video quality is more severely affected at the edges of objects (i.e., where the depth value changes between the object and the background). In some embodiments, the selection method in the encoding system 100 attempts to minimize the reduction in video quality associated with the reduction in the bit representation of the depth data. In some embodiments, a histogram is formed that counts the frequency of occurrence of particular depth values in the input depth view. The range of depth values that occur most frequently is assigned to the line segment with a higher slope, and the range of depth values that occur least frequently is assigned to the line segment with a lower slope. As explained, a higher slope means that a greater granularity of the representation is preserved, and similarly, a lower slope means that a lesser granularity of the representation is preserved.

In one embodiment, a histogram of depth values is determined separately for edge regions in the image by first applying an edge detection algorithm. Line segment endpoints may be selected based on the edge depth histogram to give a preference to providing a higher slope for the most frequently occurring depth values that occur at object edges. For example, edge regions may be determined for a depth image and an edge histogram may be generated for the edge regions. Then, in response to there being more edge samples in a first range of the histogram relative to the second range, a first slope of the first range is greater than a second slope of the second range. In some embodiments, edge regions of the depth view are determined and a histogram of depth pixel value counts is generated using pixels within the detected edge regions and excluding pixels outside the edge regions. Such a histogram may then be used to determine line segments for a piecewise linear mapping as described below.

The encoder may select line segments based on histograms of any number of views and images. In some embodiments, it is advantageous to apply the same set of depth mapping parameters for the entire encoded video sequence (e.g., an I-picture and a number of additional non-I-pictures) within the same random access period to avoid introducing inefficiencies in inter-frame prediction caused by using different sets of mapping parameters. In this case, it may be advantageous to use all images of all views within the random access period when selecting mapping parameters. In one embodiment, such histograms (using all samples or edge samples) are generated using all images of all views within the random access period or group of images.

Processing continues at operation 603, where pixel counts across an initial high depth value range are determined using the selected depth view data, and then continues at operation 604, where line segments are recursively generated (as defined by the line segment end points) to minimize reconstruction error.

Now, a technique for recursively determining an optimal piecewise linear mapping is described. Such a technique uses as input a histogram (either full frame, multiple frames, or edges only) of a higher bit depth map (e.g., 16-bit) and a desired number of segments in the mapping. Such a histogram and the number of input segments may be determined using any suitable technique or techniques. In an embodiment, the desired number of segments is limited to a certain default value. The following technique uses dynamic programming to determine optimal x and y coordinates for the endpoints (e.g., endpoints 303, 304, 305, 306) of segments (line segments 307, 308, 309) of a piecewise linear mapping (e.g., piecewise linear mapping 300).

In the following description, (i) the first segment of a piecewise linear mapping starts at (0,0), (ii) if a segment s starts at (i,k), then the previous segment s-1 ends at (i-1,q), where q is k or k-1, and (iii) the slope (number of output bins/number of input bins) of each segment is constrained to be less than or equal to 1 (<=1).

Let E(s,i,k) be the minimum reconstruction error when input bins [0...i] are mapped to output bins [0...k] using segment [0...s]. Let S(s,i,k) denote the corresponding optimal solution consisting of the start and end points of each of the s+1 segments. That is, S is the piecewise linear mapping that corresponds to the minimum reconstruction error. Note that the last segment (index s) ends at (i,k). Then, all possible starting positions (p,q) for this segment can be considered such that p is less than or equal to i and q is less than or equal to k (p<=i and q<=k).

In particular, this problem has the optimal substructure property such that the optimal solution S(s,i,k) with error E(s,i,k) has a segment s that starts at a particular (p,q). Furthermore, the previous segment s-1 in the optimal solution ends at (m,n), which can be (p-1,q) or (p-1,q-1). Then, the optimal solution S(s,i,k) needs to be constructed from the optimal solution for (s-1,m,n), denoted S'(s-1,m,n), with error E'(s-1,m,n) followed by segment s. (If S(s,i,k) does not contain S'(s-1,m,n), then the segment [0.s-1] in S(s,i,k) can be replaced with S'(s-1,m,n). This gives a new solution to (s,i,k) with an overall error lower than E(s,i,k), since S'(s-1,m,n) has the smallest possible error among all solutions to (s-1,m,n).)

The reconstruction error when segment s maps bin [pi] to bin [qk] is denoted as e(s,p,q,i,k), which gives a recursive solution for E(s,i,k), as shown in equation (2).
E(s,i,k)
=minimum over all feasible(p,q) of {E(s-1,m,n)+e(s,p,q,i,k)} if s>0
=e(0,0,i,k) if s=0
(2)
where (p,q) is the start point of the current segment s, and (m,n) is the end point of the previous segment s-1, where m is equal to p-1 (m=p-1), and n is either q or q-1.

A feasible (p,q) is defined as follows: The segments are assumed to be flat (single output bin) or monotonically increasing. Each previous segment must cover at least one input bin, so the current segment with index s must start at or above s (>=s). Each segment must have a slope less than or equal to 1. And each segment must cover at least one input bin. These constraints may be defined as shown in equation (3).
S<=p<=i
0<=q<=k
k-q+1<=i-p+1 (slope constraint)
s<=i (each segment must cover at least one input bin)
(3)

The error function e(s,p,q,i,k) for a segment s mapping bins [p..i] to bins [q..k] is defined as: First, we uniformly map the input bins to the output bins using the definition given in equation (4).
Let I=i-p+1 =number of input bins
Let J=k-q+1 =number of output bins
Let f = floor(I/J)
Let r=If*J
(4)

The first r output bins are then each assigned f+1 input bins: [0..f] for first, [f+1,2f+1] for second, etc. The remaining J-r output bins are each assigned f input bins.

The error model is then provided as follows: Error Model: Let each input bin span be in range D. The true values of pixels counted in input bin i may be assumed to be uniformly distributed over [(i-1)D..iD]. Next, assume that we map K input bins to one output bin. The inverse mapping then reconstructs the pixels to the values at the center of the KD-sized range of these input bins. The squared error se may then be given as shown in the Pseudo-Code provided by Equation (5) below.

If K is even: t represents the bin distance from the center of the interval, e.g., K=6, t={2,1,0,0,1,2}.
se=integral(over 0..D)1/D.(tD+x)^2dx
=t*t*D*D+t*D*D+D*D/3
For K=1:
se=2integral(over 0 to D/2)1/Dx^2dx=D*D/12
If K is odd and greater than 1: t represents the bin distance from the center of the interval, e.g., K=7, t={3,2,1,0,1,2,3}.
For t not equal to 0:
se=integral(over 0..D)1/D.((t-0.5)D+x)^2dx
=(t-0.5)^2*D*D+(t-0.5)*D*D+D*D/3
Total squared error for the output bin=sum_{over t}(histogram value*se(t))
For simplicity, D can be set to 1, D=1
(5)

Note that a problem can arise if the current segment has a start point whose y coordinate (output bin) matches the y coordinate of the endpoint of the previous segment. This causes more input bins to be mapped to the same output bin than if only the previous segment was used, affecting the reconstruction error of the previous segment. One solution is to attribute this additional reconstruction error to the current segment. Another solution is to not allow this situation, so that the current segment can only start in the new output bin. The optimization can be costly.

Using such a recursive technique, line segment endpoints are generated based on the input histogram data and a number of selected line segments.

Processing continues at operation 605, where the generated line segment endpoints are output for use in the forward mapping described with respect to mapping modules 121, 123, 125, for encoding into bitstream 131, and finally for use by decoding system 200 as described with respect to reverse mapping modules 221, 223, 225, and ends at operation 606. Description is now directed to the use of such line segment endpoints in the decoding process.

FIG. 7 illustrates one example process 700 for increasing the bit depth of a depth view using piecewise linear mapping, configured in accordance with at least some implementations of the present disclosure. Process 700 may include one or more operations 701-706, as illustrated, performed, for example, by decoding system 200. Process 700 begins with operation 701, where an increase in bit depth is initiated. The increase in bit depth may be performed, for example, by mapping modules 221, 223, 225, which map depth videos 222, 224, 226 (lower bit depths) to depth videos 202, 204, 206 (higher bit depths).

Processing continues at operation 702, where a depth view is received at a lower or lower bit depth (relative to a higher target bit depth) and decoded. The depth view may have any suitable data structure, such as a depth frame or image having a depth value (at the lower bit depth) for each pixel thereof. In some embodiments, the depth view has depth values with a bit depth of 8 bits each ranging from 0 to 255. In some embodiments, the depth view has depth values with a bit depth of 10 bits each ranging from 0 to 255. In such a context, each depth value may range from 0 to 1023. For example, at operation 702, a first depth view may be received having depth values at a low bit depth. Thus, the low bit depth has a large range of available values from a minimum value (e.g., 0) to a maximum value (e.g., 255 or 1023).

Processing continues at operation 703, where the line segment endpoints are decoded to map depth views at lower bit depths to higher bit depths, such that the line segment endpoints define line segments for mapping, such as line segments 307, 308, 309. Such line segment endpoints and line segments may be decoded using any suitable technique or technique, such as those described with respect to FIG. 2. For example, the line segment endpoints may be decoded by metadata decoding module 217 from a portion of the bit stream demultiplexed from bit stream 131. As described, the line segment endpoints and line segments defined thereby may be sought to minimize lower bit depths to higher bit depths in the inverse mapping used to increase depth values.

Processing continues at operation 704, where the low or lower bit depth view is mapped to a higher bit depth view to generate a depth view having an increased bit depth. The higher bit depth may be any bit depth, such as 16 bits. Such mapping is performed according to the line segment endpoints decoded at operation 703. For example, for each pixel of the lower bit depth view, a determination may be made as to which line segment of multiple line segments that piecewise linearly map the pixel is associated (e.g., such that the lower bit depth value is within the range of bit depth values of the line segment). A linear mapping according to the associated line segment is then provided to map the lower bit depth value to the higher bit depth value. Such processing may be repeated, in series or in parallel, for every pixel of a lower bit depth view (e.g., one of depth videos 222, 224, 226) to generate a higher bit depth view (e.g., one of depth videos 202, 204, 206), such as 8-bit (range 0 to 255m) or 10-bit (range 0 to 1023).

3, the depth values of a depth view may be mapped to depth values at a higher bit depth that is greater than the lower bit depth to generate a depth view, such that the higher bit depth has an available range of values from a minimum value (e.g., 0) to a maximum value (e.g., 65535). A first line segment end point (e.g., ( _x1 , _y1 )) and a second line segment end point (e.g., ( _x2 , _y2 )) are used to define a line segment for the mapping, such that the horizontal component (e.g., _x1 ) and vertical component (e.g., _y1 ) of the first line segment end point exceed the minimum depth value (e.g., 0) of the higher bit depth and the minimum depth value (e.g., 0) of the lower bit depth, respectively. Further, the horizontal component (e.g., _x2 ) and vertical component (e.g., _y2 ) of the second line segment end point are less than the maximum depth value of the higher bit depth (e.g., 65535) and the maximum depth value of the lower bit depth (e.g., 255 or 1023), respectively.

Processing continues at operation 705, where a depth view at a higher bit depth is output for view synthesis processing, as described herein. In some embodiments, the higher bit depth view may be used to generate view 250, which may also be generated based on a corresponding texture view, as well as other depth and texture view pairs. As shown, process 700 may end at operation 706. Process 700 may be performed serially or in parallel for any number of depth views. Such depth views may also be characterized as depth frames, depth pictures, etc. As described herein, such increased bit depth views may be advantageous and provide improved performance in the context of view synthesis.

1 and 2, the description now turns to the encoding and decoding of the camera array parameters 151 via the camera parameter encoding module 118 and the camera parameter decoding module 218, respectively. Such encoded camera array parameters may be transmitted between the encoding system 100 and the decoding system 200 using a camera information SEI message. In some embodiments, metadata describing the camera parameters, currently in the .json file and including additional metadata, may also be carried in the same or another SEI message. In some embodiments, it is advantageous to include depth mapping parameters in a separate SEI message, since these parameters may change more frequently than the camera parameters. In one embodiment, the 3DoFplus camera information SEI message represents data more compactly than a .json file, saving bitrate for data transmission. For each camera, the SEI message may include (x, y, z) position information and orientation information (based on yaw, pitch, roll, or quaternion parameters). In one embodiment, the yaw_pitch_roll flag indicates whether (yaw, pitch, roll) information is requested to be sent or they all have a default value of 0. For the (x, y, z) parameters, we typically have regular camera spacing. In some embodiments, bitrate savings can be achieved by signaling the granularity of the position signaled for each of x, y, and z. For example, spacing values in x, y, and z can be provided for camera positions to save bitrate.

Additionally, the videos described herein may include omnidirectional or normal perspective content. In some embodiments, the Camera Info SEI message includes a flag indicating which is used. For example, the Camera Info SEI message may include a flag indicating omnidirectional or normal perspective content. Additionally, in some implementations, all cameras in a multi-camera immersive media system have the same parameters. The parameters include field of view, Znear (the minimum physical depth achievable by the camera), and Zfar (the maximum physical depth achievable by the camera). In some embodiments, signaling overhead is reduced by sending flags to indicate common values of these parameters, which may be signaled for all views, or where the parameters are explicitly signaled for each view. For example, the Camera Info SEI message may include flags indicating common or different camera parameters among all views. In the former example, a single set of corresponding camera parameters is provided, and in the latter example, multiple sets of camera parameters are provided. In examples where a set of cameras share camera parameters, only a unique set of camera parameters may be provided. And the SEI message may include an indicator as to which view corresponds to which set of camera parameters so that redundant sets of camera parameters are not sent.

FIG. 8 illustrates one exemplary camera array 810 and corresponding camera parameters 800 for encoding, configured in accordance with at least some implementations of the present disclosure. For example, the camera array 810 may be used to capture a base view 141, a selected view 142, and other views, if necessary, of a scene. That is, the camera array 810 may be used for immersive video generation or capture. In the example of FIG. 8, the camera array 810 includes six cameras C1-C6 in a grid pattern, such that cameras C1-C3 are in a first row and cameras C4-C6 are in a second row. However, the camera array 810 may have any number of cameras with any spacing layout. In one embodiment, the camera array 810 includes a single row of cameras. Furthermore, the camera array 810 is shown such that the cameras C1-C6 are all aligned in the x-y plane. However, the cameras of the camera array 810 may be arranged in any manner.

Each of the cameras C1-C6 has corresponding camera parameters 811, 812, 813, 814, 815, 816, respectively. As shown, the camera parameters 811 may include the position of the camera C1, the orientation of the camera C1, and the imaging parameters of the camera C1. As shown, the position parameters include the coordinates of the camera C1 ( _x1 , _y1 , z1). The orientation parameters include the roll, pitch, yaw values (r1, p1, yaw1) or quaternion parameters that define the orientation of the camera C1. As shown with respect to the camera C1, each of the cameras C1-C6 may be positioned and oriented throughout 3D space, the positioning and orientation being characterized as 6-DOF motion 835. This indicates that each of the cameras C1-C6 can move in translation: forward/backward (e.g., z direction), up/down (e.g., y direction), right/left (e.g., x direction), and rotation: yaw (e.g., angle around y axis), roll (e.g., angle around x axis), and pitch (e.g., angle around z axis). The imaging parameters include one or more of a projection mode (PM1), a focal length (FL1), a field of view (FOV1), a Znear value (ZN1), and a Zfar value (ZF1). For example, the projection mode can indicate one of an omnidirectional content, a normal viewpoint content, or an orthogonal content for the camera. That is, the projection mode can indicate one of an omnidirectional projection mode, a normal viewpoint projection mode, or an orthographic projection mode.

Similarly, camera parameters 812, 813, 814, 815, 816 include similar position, orientation, and imaging parameters for each of cameras C2-C6, respectively. In particular, as represented by camera array parameters 151, such camera parameters 811, 812, 813, 814, 815, 816, or a portion thereof as represented by camera array parameters 151, are encoded into bitstream 131 and decoded by decoding system 200 for use in view synthesis. The techniques described herein provide efficient coding of camera parameters 812, 813, 814, 815, 816 of camera array 810.

As shown with respect to camera C1, system 110 may move throughout 3D space with motion characterized as 6-DOF motion 135. This indicates that system 110 may move in translation: forward/backward (e.g., in the x direction), up/down (e.g., in the y direction), right/left (e.g., in the z direction), and rotation: yaw (e.g., angle α about the z axis), roll (e.g., angle β about the y axis), and pitch (e.g., angle γ about the x axis). As will be appreciated, 3D content such as VR frames may be generated based on an estimated line of sight of the user of system 110 (e.g., along the forward direction of the x axis).

FIG. 9 illustrates an example process 900 for encoding immersive video data including camera parameters, arranged in accordance with at least some implementations of the present disclosure. Process 900 may include one or more operations 901-908, as illustrated, performed, for example, by encoding system 100. Process 900 begins with operation 901, where camera parameter encoding is initiated. Coding of the camera parameters may be performed, for example, by camera parameter encoding module 118 to encode camera parameters 800 for camera array 810, as represented by camera array parameters 151, into bitstream 131.

Processing continues at operation 902, where position, orientation, and imaging parameters are obtained for the cameras of the camera array. Such position, orientation, and imaging parameters may have any suitable data structure. Processing continues at operation 903, where a particular camera parameter type is selected for encoding. For example, each of the described camera parameters may be processed in turn.

Processing continues at decision operation 904, where a determination is made as to whether, for the selected camera parameter type, a particular parameter value is shared by each of the cameras represented by the camera parameters. If not, processing continues at operation 905, where a flag indicating a shared parameter value for the camera is set off (e.g., 0), and the individual parameters for each camera are coded separately.

If the parameter values are shared by each of the cameras for a camera parameter type (e.g., all cameras C1-C6 have the same parameter value for that parameter type), processing continues at operation 906, where a flag indicating a shared parameter value for the camera is set on (e.g., 1) and the shared value is encoded. In particular, operations 904-906 provide reduced overhead when each of the cameras has a shared parameter value. For example, if the focal length of each of the cameras is the same, the shared focal length flag may be set on (e.g., 1) and the flag and the shared focal length may be encoded. Such processing reduces overhead by not repeating the focal length for each camera. However, if the cameras do not share a focal length, the shared focal length flag is set off and the focal length of each camera is encoded. Note that such flags may be set for multiple cameras and then changed. For example, if cameras C1-C4 have the same parameter value and C5 and C6 are different, an on flag may be set and the parameter value set for camera C1 and C2-C3 may use the same parameter value until an off flag is set and the parameter values for C5 and C6 are then signaled individually. For example, a shared parameter value flag may indicate that the shared parameter value is used until the flag is switched off.

Such shared parameter value flags may be characterized simply as parameter value flags and may be labeled or identified based on their corresponding parameter values. For example, a shared projection mode flag (e.g., a projection mode flag or a projection mode value flag) may indicate that all cameras C1-C6 share the same projection mode (e.g., one of an omnidirectional projection mode, a normal perspective projection mode, or an orthographic projection mode) based on a projection mode flag having a first value (e.g., 1), or that all cameras do not have the same projection mode based on a projection mode flag having a second value (e.g., 0). A shared field of view flag (e.g., a field of view flag or a field of view shared value flag) may indicate that all cameras C1-C6 share the same field of view based on a field of view flag having a first value (e.g., 1), or that all cameras do not have the same field of view based on a projection mode flag having a second value (e.g., 0). In a similar manner, cameras with the same Znear and Zfar values may be signaled, either together or separately, to indicate that both Znear and Zfar are identical for all cameras C1-C6 using only a shared Znear flag (e.g., Znear flag, Znear shared value flag, or minimum physical depth flag) and a shared Zfar flag (e.g., Zfar flag, Zfar shared value flag, or maximum physical depth flag), or a shared Zdistances flag (e.g., Zdistances flag, Zdistances shared value flag, or physical depth flag). In all such cases of flagging, if the parameter value is shared by all cameras, an indicator of a single shared parameter value, a single shared parameter value, etc., is coded in the bitstream. If not, separate parameter values for each camera may be signaled.

Additionally, shared parameter value flags may be used for multiple camera parameter values. For example, a single shared image flag may be used to indicate all of the projection mode, focal length, field of view, Znear value, and Zfar value to be shared between the cameras. Such a single shared image flag may then be followed by parameter values for each of the projection mode, focal length, field of view, Znear value, and Zfar value, which are not signaled for the other cameras. Similarly, a single yaw, pitch, roll flag, or quaternion flag may be signaled to indicate yaw, pitch, and roll, or the quaternion parameters may be the same for each of the cameras C1-C6. In some embodiments, the values of the yaw, pitch, roll, or quaternion parameters (which are then shared by all the cameras) are signaled. In some embodiments, a second flag may indicate that the yaw, pitch, and roll, or the quaternion parameters are default values, such as 0, 0, 0 for yaw, pitch, and roll. In some embodiments, if the second flag is not used and a single yaw, pitch, and roll flag or a quaternion flag is set to on (e.g., 1), the default values are inferred by the decoder.

In some embodiments, the camera parameter values are not shared, but they may be predicted using a single value. For example, if the cameras C1-C6 are equally spaced in a row, a row position flag may be provided. Then, based on the row position flag, the shared z and y positions for the cameras may be signaled and used as the same values for all cameras C1-C6. Furthermore, the x position of camera C1 may be signaled using a constant x position interval value (e.g., x _C in FIG. 8 ) that is the same for all cameras. Thus, the x positions of cameras C2-C6 may be generated using the x position of camera C1 and a constant x position interval value (e.g., x position of camera CN=x ₁ +(N-1)*x _C ). And there is no explicit signaling of the positions of cameras C2-C6.

Similarly, if cameras C1-C6 are equally spaced in a grid (as shown in FIG. 8), a grid position flag may be provided. Based on the grid position flag, the shared z position for the cameras may then be signaled and used as the same value for all cameras C1-C6. Furthermore, the x and y positions of camera C1 may be signaled with a constant x position spacing value (e.g., x _C in FIG. 8) and a constant y position spacing value (e.g., y _C in FIG. 8) that are the same for all cameras. This allows the x and y positions of cameras C2-C6 to be generated using the x and y positions of camera C1 and a constant x position spacing value and a constant y position spacing value (C2 is at (( _x1 + _xC ), ( _y1 )), C4 is at (( _x1 ), ( _y1 + _yC ), C5 is at (( _x1 + _xC ), ( _y1 + _yC ))))) and there is no explicit signaling of the positions of cameras C2-C6.

Processing continues at decision operation 907, where a determination is made whether the camera parameter type selected at operation 903 is the last camera parameter type to be processed. If so, processing ends at operation 908. If not, processing continues at operation 903 until the last camera parameter type has been processed, as described above. Although described with respect to such recursive processing for clarity of presentation, process 900 can decode any number of camera parameter types in parallel.

As described above, the process 900 provides efficient signaling of camera parameters, including camera position ((x,y,z)), camera orientation ((r,p,yaw) or quaternion values), and camera imaging parameters (PM,FL,FOV,ZN,ZF). Such compression techniques can be combined in any suitable combination. Such compressed flags and values are encoded into a bitstream and subsequently decoded and used in view synthesis, as described with respect to FIG. 10. In some embodiments, such encoding provides a part of immersive video data encoding and includes determining a camera projection mode for each of a plurality of cameras of a camera array corresponding to immersive video generation or capture, generating a camera projection mode flag for the camera, the camera projection mode flag having a first value (e.g., 1) in response to each camera having a shared projection mode and a second value (e.g., 0) in response to any of the cameras having a projection mode other than the shared projection mode, and encoding the camera projection mode flag into the bitstream. In some embodiments, in response to a first camera projection mode flag having a first value, the bitstream further includes a single camera projection mode indicator indicating a shared projection mode. In some embodiments, in response to a camera projection mode flag having a second value, the bitstream further includes multiple camera projection mode indicators, one for each camera.

In some embodiments, the encoding further includes determining a minimum and maximum physical depth achievable by each camera, generating a physical depth flag for the camera, the physical depth flag indicating whether the camera has a shared minimum physical depth and/or a shared maximum physical depth, and encoding the physical depth flag into the bitstream. In some embodiments, the encoding further includes determining one of yaw, pitch, and roll parameters or quaternion parameters for each camera, generating a yaw, pitch, and roll flag or quaternion flag for the camera, the yaw, pitch, and roll flag or quaternion parameter flag indicating whether the camera has shared yaw, pitch, and roll parameters or shared quaternion parameters, and encoding the yaw, pitch, and roll flag or quaternion parameter flag into the bitstream. In some implementations, the yaw, pitch, and roll parameters or quaternion parameters indicate default yaw, pitch, and roll or default quaternion parameter flag values. In some embodiments, the encoding further includes determining a shared physical spacing value representing the shared physical spacing between the cameras and encoding the shared physical spacing value into the bitstream.

FIG. 10 illustrates an example process 1000 for decoding immersive video data including camera parameters, arranged in accordance with at least some implementations of the present disclosure. The process 1000 may include one or more operations 1001-1008 as shown, performed, for example, by the decoding system 200. The process 1000 begins with operation 1001, where camera parameter decoding is initiated. The camera parameter decoding may be performed, for example, by the camera parameter decoding module 218 to decode camera parameters 800 for the camera array 810, as represented by the camera array parameters 151 from the bitstream 131.

Processing continues at operation 1002, where a received bitstream including camera parameter flags and indicators is received and decoded using any suitable technology or technique. In particular, the bitstream may include any flags, parameters, indicators, etc., such as those described with respect to FIG. 9, that may be decoded to reconstruct. Processing continues at operation 1002, where a first camera parameter type is selected. For example, each of the described camera parameters may be processed in turn.

Processing continues at decision operation 1004, where a determination is made for the selected camera parameter type as to whether a specific or shared parameter value flag for the selected camera parameter type is set on (e.g., 1). If not, processing continues at operation 1005, where the camera parameter values are decoded for each of the cameras in the camera array. For example, because the cameras do not share specific or shared parameter values, the values for each camera are decoded separately.

If the specific or shared parameter value flag for the selected camera parameter type is set on (e.g., 1), processing continues at operation 1006, where an indicator or value of the specific or shared parameter value, etc. is decoded and set for all cameras in the camera array. That is, each camera is assigned the same value for the specific or shared parameter value (or, in the context of location information, the location of each camera is determined using the decoded information, as described below). Operations 1004-1006 provide for decoding of efficiently packed camera information, such that the shared camera information may be flagged and encoded only once. For example, if the focal length of each of the cameras in a camera array is the same, the shared focal length flag (or focal length flag, etc.) may be decoded to be on (e.g., 1). The shared or specific focal length, or indicator thereof, is then decoded and assigned for each camera. However, if the shared focal length flag is decoded to be off (e.g., 0), the focal length of each camera is decoded separately. Similar to encoding, in some embodiments, a sharing flag of 1 may be set and an indicator that applies may be sent until the flag is switched to 0 and new parameters are then sent. For example, each camera may be assigned a flag of 1 to indicate that sharing continues for that camera. The first camera with a different focal length is then flagged to 0 and new parameters are sent. Note that such flags may be set for many cameras and then changed.

As described with respect to the encoding, such shared parameter value flags may be characterized simply as parameter value flags and may be labeled or identified based on their corresponding parameter values. For example, a shared projection mode flag may indicate if all cameras share the same projection mode, a shared field of view flag may indicate if all cameras share the same field of view, and a shared Znear flag, a shared Zfar flag, or a shared Zdistances flag may indicate, respectively, that the minimum physical depth the cameras can achieve, the maximum physical depth the cameras can achieve, or both, are the same for all cameras. Again, as described with respect to the camera parameter encoding, shared parameter value flags may be used for multiple camera parameter values. For example, a single shared imaging flag may indicate that all projection modes, focal lengths, field of view, Znear values, and Zfar values are all the same for all cameras. When such a shared image flag is decoded as on, indicators or parameter values for each of the projection modes, focal lengths, field of view, Znear values, and Zfar values are decoded from the bitstream and applied to all cameras.

Similarly, a shared yaw, pitch, roll flag, or quaternion flag may be decoded and, when flagged as on, a single set of yaw, pitch, and roll, or quaternion parameters are decoded and applied to all cameras. Alternatively, a shared yaw, pitch, roll flag, or quaternion flag being on may indicate that such parameters are shared across all cameras and that the parameters are set to default values of 0, 0, 0 for yaw, pitch, and roll.

Further, in some embodiments, such flagging and parameter sharing may be used by the decoder to determine values at the decoder that are not explicitly coded. For example, the bitstream may be decoded to determine a row position flag that the camera is set to on. Based on the row position flag, the shared z and y positions of the cameras are decoded and assigned to all cameras C1-C6. Further, the x position of camera C1 (e.g., x ₁ in FIG. 8) and a constant x position spacing value (e.g., x _C ) are decoded. The camera positions in the x dimension for each of cameras C1-C6 may then be determined by the decoder such that camera C1 is at x ₁ and cameras C2-C6 are at positions offset by x _C relative to the previous camera in the row (e.g., camera CNx position=x ₁ +(N-1)*x _C , and there is no explicit signaling of the positions of cameras C2-C6). In some embodiments, the bitstream may be decoded and provided to determine that the grid position flag is on. Based on the grid position flags, the shared z-positions of the cameras are decoded and assigned to all cameras C1-C6. The x and y positions of camera C1 ( _x1 and _y1 , respectively) and a constant x-position interval value (e.g., _xC ) and a constant y-position interval value (e.g., _yC ) are decoded. Thus, the x and y positions of cameras C2-C6 can be generated using the x and y positions of camera C1 and the constant x-position and y-position interval values.

Processing continues at decision operation 1007, where a determination is made as to whether the camera parameter type selected at operation 1003 is the last camera parameter type to be processed. If so, processing ends at operation 1008. If not, processing continues at operation 1003 until the last camera parameter type has been processed, as described above. Although illustrated with respect to such recursive processing for clarity of presentation, process 1000 can decode any number of camera parameter types in parallel.

As described above, the process 1000 provides for decoding of camera parameters including camera position ((x,y,z)), camera orientation ((r,p,yaw) or quaternion values), and camera imaging parameters (PM,FL,FOV,ZN,ZF). In some embodiments, such decoding provides a part of immersive video data decoding and includes decoding the bitstream to determine a depth view, a texture view, and camera projection mode flags corresponding to a plurality of cameras of the camera array, the camera projection mode flags having a first value indicating that all of the cameras have a particular projection mode or a second value indicating that any of the cameras have a projection mode other than the particular projection mode, and generating a decoded view based on the depth view, the texture view, and the camera projection mode flag depth view. In some embodiments, in response to the first camera projection mode flag having the first value, the bitstream further includes a single camera projection mode indicator indicating the particular projection mode. In some embodiments, in response to the camera projection mode flag having the second value, the bitstream further includes a plurality of camera projection mode indicators, one for each camera. In some embodiments, the shared camera projection mode includes at least one of an omnidirectional projection mode, a normal perspective projection mode, or an orthographic projection mode.

In some embodiments, the bitstream further includes one of a yaw, pitch, roll flag, or a quaternion flag for the camera indicating whether the cameras have shared yaw, pitch, roll parameters, or shared quaternion parameters. In some embodiments, the bitstream further includes a physical depth flag for the camera, the physical depth flag indicating whether the cameras have a shared minimum physical depth and/or a shared maximum physical depth. In some embodiments, the decoding process is further responsive to the yaw, pitch, roll flag, or the quaternion flag indicating that the cameras have shared yaw, pitch, roll parameters, or shared quaternion parameters, and the yaw, pitch, roll parameters, or quaternion parameters for each of the cameras are default yaw, pitch, roll parameters, or default quaternion parameters. In some embodiments, the bitstream further includes a shared physical spacing value, and the decoding process further includes determining a physical spacing between the cameras using the shared physical spacing value.

The discussion now moves on to compliance with the 3DoF+ standard and support for atlases created to represent texture and depth as described herein.

表2：NALユニットヘッダの実施例 HEVC uses a 2-byte NAL (network abstraction layer) unit header, whose syntax is copied below, which includes 6 bits allocated for nuh_layer_id.

Table 2: Example of a NAL unit header

HEVC defines a base video parameter set (VPS) in Section 7 and a VPS extension in Annex F. Annex F describes the Independent Non-Base Layer Decoding (INBLD) capability, where independently coded layers may have nuh_layer_id values not equal to 0.

The base VPS includes the vps_max_layers_minus1 syntax element. The VPS extension includes the syntax elements splitting_flag, scalability_mask_flag[i], dimension_id_len_minus1[j], dimension_id[i][j], and direct_depency_flag[i][j]. These syntax elements are used to indicate the characteristics of each layer, including the derivation of the ScalabilityId variable and the dependencies between layers.

表3：スケーラビリティ次元に対するScalabilityIdマッピング The ScalabilityId variable can be mapped to a scalability dimension or tier characteristic as follows:

Table 3: ScalabilityId mapping for scalability dimensions

Annex G defines the multi-view main profile, where the enhancement layer is an inter-layer predicted from the base layer.

In some aspects, two atlases are created to represent texture and depth. These two atlases may be coded as separate layers in the same bitstream, each with a different value nuh_layer_id. The depth atlas layers are independent, and no inter-layer prediction can be made from the texture atlas layer.

A new "Immersive" HEVC profile is provided herein. In one embodiment, the profile allows for two independent layers, each individually conforming to the Main 10 (or Main) profile with INBLD (Independent Non-Base Layer Decoding) capability. In another embodiment, an additional profile is supported for each independent layer, specifically to allow for monochrome representation of depth, rather than the 4:2:0 chroma format required by the Main and Main 10 profiles.

The creation and definition of this profile allows bitstreams and decoders to demonstrate conformance to the 3DoF+ standard, which is necessary to ensure interoperability.

表4: スケーラビリティ次元に対するScalabilityIdのマッピングの修正 In some embodiments, a new entry in the scalability mask table is provided: AtlasFlag, which must be set to indicate that it is a coded image atlas by using one of the reserved values. The existing DepthLayerFlag may be used to distinguish between texture atlas layers and depth atlas layers.

Table 4: Correction to the mapping of ScalabilityId to the Scalability dimension

The Immersive Profile requires that texture atlas layers have DepthLayerFlag equal to 0 and AtlasFlag equal to 1, and that atlas layers have DepthLayerFlag equal to 1 and AtlasFlag equal to 1, as derived from the ScalabilityId value.

The Immersive Profile places no additional requirements on the values of nuh_layer_id for the two layers, and the VPS provides flexibility in how to signal the appropriate value of ScalabilityId.

The discussion now moves to the most bitrate-efficient way to use the VPS to appropriately set the ScalabilityId value for the immersive profile. In some implementations, a nuh_layer_id of 0 is used for texture atlas layers, and a nuh_layer_id of 1 is used for depth atlas layers.

In some embodiments, splitting_flag is set equal to 1 to indicate a simple mapping of bits in nuh_layer_id to multi-layer characteristics.

In some embodiments, scalability_mask_flag[i] is set equal to 00000010001 (each bit represents an array value) to indicate that the layer dimensions used are 0 (for DepthLayerFlag) and 4 (for AtlasFlag).

In some embodiments, dimension_id_len_minus1[1] is set equal to 0 to indicate that only one bit is required to represent the AtlasFlag.

In some embodiments, direct_dependency_flag[1][0] is set to 0 to indicate that the depth atlas layer does not depend on the texture atlas layer.

表5：シンタックス要素と変数 For the two layers, Table 5 below shows the values of the relevant syntax elements and variables. The table values highlighted in grey (i.e., DepthLayerFlag, AtlasFlag, ScalabilityId, and direct_dependency_flag for DepthAtlas) are required, while the values not highlighted (i.e., nuh_layer_id) can vary. In some embodiments, the DepthLayerFlag and AtlasFlag values are required in the immersive profile, while the nuh_layer_id value can vary.

Table 5: Syntax elements and variables

The description now turns to systems and devices for implementing the described techniques, encoders, and decoders. For example, any encoder or decoder described herein may be implemented via the system shown in FIG. 11, the system shown in FIG. 12, and/or the device shown in FIG. 13. In particular, the described techniques, encoders, and decoders may be implemented via any suitable device or platform, such as a personal computer, a laptop computer, a tablet, a phablet, a smartphone, a digital camera, a gaming console, a wearable device, a display device, an all-in-one device, or a two-in-one device.

The various components of the systems described herein may be implemented in software, firmware, and/or hardware, and/or any combination thereof. For example, the various components of the devices or systems described herein may be provided, at least in part, by the hardware of a computing system-on-chip (SoC), such as found in a computing system, such as a smartphone. Those skilled in the art will recognize that the systems described herein may include additional components not depicted in the corresponding figures. For example, the systems described herein may include additional components not shown for clarity.

While implementation of the example processes described herein may include performance of all of the acts shown in the order illustrated, the disclosure is not limited in this respect, and in various examples, implementation of the example processes described herein may include only a subset of the acts illustrated, acts performed in an order different from those illustrated, or additional acts.

Additionally, any one or more of the operations described herein may be performed in response to instructions provided by one or more computer program products. Such program products may include, for example, signal bearing media that provide instructions that, when executed by a processor, can provide the functionality described herein. A computer program product may be provided in any form of one or more machine-readable media. Thus, for example, a processor including one or more graphics processing units or processor cores may undertake one or more blocks of the exemplary processes herein in response to program code and/or instructions or instruction sets conveyed to the processor by one or more machine-readable media. In general, a machine-readable medium may carry software in the form of program code and/or instruction sets that can cause any of the devices and/or systems described herein to implement at least a portion of the device or system, or any other module or component described herein.

As used in any implementation described herein, the term "module" refers to any combination of software logic, firmware logic, hardware logic, and/or circuitry configured to provide the functionality described herein. Software may be embodied as a software package, code, and/or instruction set or instructions. And, "hardware," as used in any implementation described herein, may include, for example, hardwired circuitry, programmable circuitry, state machine circuitry, fixed function circuitry, execution unit circuitry, and/or firmware that stores instructions executed by programmable circuitry, either alone or in any combination. Modules, collectively or individually, may be embodied as circuitry that forms part of a larger system, such as, for example, an integrated circuit (IC), a system on a chip (SoC), etc.

FIG. 11 is an illustration of an example system 1100 for encoding immersive video data, configured in accordance with at least some implementations of the present disclosure. As shown in FIG. 11, the system 1100 may include one or more central processing units 1101, one or more graphics processing units 1102, a memory store 1103, a display 1104, and a transmitter 1105. As also shown, the graphics processing unit 1102 may include or implement a bit depth mapping module 1111, a camera parameter encoding module 1112, an encoder 1121, a decoder 1122, and/or a view synthesis module 1123. Such modules may be implemented to perform operations described herein. In the example system 1100, the memory store 1103 may store texture views, depth views, immersive video data, piecewise linear mapping parameters, bitstream data, synthetic view data, camera array parameters, or any other data or data structures described herein.

As shown, in some examples, the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are implemented via the graphics processing unit 1102. In other examples, one or more portions of the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are implemented via the central processing unit 1101 or an image processing unit (not shown) of the system 1100. In still other examples, one or more portions of the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are implemented via an image processing pipeline, a graphics pipeline, or the like.

The graphics processing unit 1102 may include any number and type of graphics processing units capable of providing the operations described herein. Such operations may be implemented by software, hardware, or a combination thereof. For example, the graphics processing unit 1102 may include dedicated circuitry for manipulating texture views, depth views, bitstream data, etc. obtained from the memory store 1103. The central processing unit 1101 may include any number and type of processing units or modules capable of providing control and other high-level functions for the system 1100 and/or providing any operations as described herein. The memory store 1103 may be any type of memory, such as volatile memory (e.g., static random access memory (SRAM), dynamic random access memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.). In a non-limiting example, the memory store 1103 may be implemented by a cache memory. In one embodiment, one or more portions of the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are executed via an execution unit (EU) of the graphics processing unit 1102. The EU may include programmable logic or circuitry, such as a logic core or cores capable of providing a wide array of programmable logic functions. In one embodiment, one or more portions of the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are implemented via dedicated hardware, such as fixed function circuitry. The fixed function circuitry may include dedicated logic or circuitry and provide a set of fixed function entry points to map to the dedicated logic for a fixed purpose or function. In some embodiments, one or more portions of the bit depth mapping module 1111, the camera parameter encoding module 1112, the encoder 1121, the decoder 1122, and/or the view synthesis module 1123 are implemented via an application specific integrated circuit (ASIC). The ASIC may include an integrated circuit customized to perform the operations described herein.

12 is an illustration of an example system 1200 configured in accordance with at least some embodiments of the present disclosure. In various implementations, the system 1200 may be a mobile device system, although the system 1200 is not limited in this context. For example, the system 1200 may be incorporated into a personal computer (PC), a laptop computer, an ultra-laptop computer, a tablet, a touchpad, a portable computer, a handheld computer, a personal digital assistant (PDA), a mobile phone, a mobile phone/PDA combination, a television, a smart device (e.g., a smartphone, a smart tablet, or a smart television), a mobile internet device (MID), a messaging device, a data communication device, a camera (e.g., a point-and-shoot camera, a super zoom camera, a digital single-lens reflex (DSLR) camera), a surveillance camera, a surveillance system including a camera, and the like.

In various embodiments, the system 1200 includes a platform 1202 coupled to a display 1220. The platform 1202 can receive content from a content device, such as a content services device 1230 or a content delivery device 1240, or from another content source, such as an image sensor 1219. For example, the platform 1202 can receive image data from the image sensor 1219 or any other content source, as described herein. A navigation controller 1250, including one or more navigation features, can be used to interact with the platform 1202 and/or the display 1220, for example. Each of these components is described in more detail below.

In various implementations, the platform 1202 may include any combination of the chipset 1205, the processor 1210, the memory 1212, the antenna 1213, the storage 1214, the graphics subsystem 1215, the application 1216, the image signal processor 1217, and/or the radio 1218. The chipset 1205 may provide intercommunication between the processor 1210, the memory 1212, the storage 1214, the graphics subsystem 1215, the application 1216, the image signal processor 1217, and/or the radio 1218. For example, the chipset 1205 may include a storage adapter (not shown) that may provide intercommunication with the storage 1214.

The processor 1210 may be implemented as a mixed instruction set computer or reduced instruction set computer processor, an x86 instruction set compatible processor, a multi-core, or any other microprocessor or central processing unit. In various implementations, the processor 1210 may be a dual-core processor, a dual-core mobile processor, etc.

Memory 1212 may be implemented as a volatile memory device, such as, but not limited to, random access memory (RAM), dynamic random access memory (DRAM), or static RAM (SRAM).

Storage 1214 may be implemented as non-volatile storage, such as, but not limited to, a magnetic disk drive, an optical disk drive, a tape drive, internal storage, attached storage, flash memory, battery-backed synchronous dynamic random access memory (SDRAM), and/or a network-accessible storage device. In various implementations, storage 1214 may include technology to increase storage performance enhanced protection for valuable digital media, for example, when multiple hard drives are included.

Image signal processor 1217 may be implemented as a specialized digital signal processor used for image processing, etc. In some examples, image signal processor 1217 may be implemented based on a single instruction multiple data or multiple instruction multiple data architecture, etc. In some examples, image signal processor 1217 may be characterized as a media processor. As described herein, image signal processor 1217 may be implemented based on a system on a chip architecture and/or based on a multi-core architecture.

The graphics subsystem 1215 can perform processing of images, such as still images or video, for display. The graphics subsystem 1215 can be, for example, a graphics processing unit or a visual processing unit. An analog or digital interface can be used to communicatively connect the graphics subsystem 1215 to the display 1220. For example, the interface can be any of High-Definition Multimedia Interface, DisplayPort, Wireless HDMI, and/or Wireless HD compliant technologies. The graphics subsystem 1215 can be integrated into the processor 1210 or the chipset 1205. In some implementations, the graphics subsystem 1215 can be a standalone device communicatively connected to the chipset 1205.

The graphics and/or video processing techniques described herein may be implemented in a variety of hardware architectures. For example, graphics and/or video capabilities may be integrated within a chipset. Alternatively, separate graphics and/or video processors may be used. In yet another implementation, graphics and/or video capabilities may be provided by a general-purpose processor, including a multi-core processor. In further embodiments, the capabilities may be implemented in a consumer electronic device.

The wireless device 1218 may include one or more radios capable of transmitting and receiving signals using various suitable wireless communication technologies. Such technologies may include communication over one or more wireless networks. Examples of wireless networks include, but are not limited to, wireless local area networks (WLANs), wireless personal area networks (WPANs), wireless metropolitan area networks (WMANs), cellular networks, and satellite networks. In communicating over such networks, the wireless device 1218 may operate according to one or more applicable standards in any version.

In various embodiments, the display 1220 may include any television-type monitor or display. The display 1220 may include, for example, a computer display screen, a touch screen display, a video monitor, a television-like device, and/or a television. The display 1220 may be digital and/or analog. In various implementations, the display 1220 may be a holographic display. The display 1220 may also be a transparent surface that may receive visual projections. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be visual overlays for mobile augmented reality (MAR) applications. Under control of one or more software applications 1216, the platform 1202 may display a user interface 1222 on the display 1220.

In various implementations, the content services device(s) 1230 may be hosted by a national, international, and/or independent service and thus may be accessible to the platform 1202 via, for example, the Internet. The content services device(s) 1230 may be connected to the platform 1202 and/or the display 1220. The platform 1202 and/or the content services device(s) 1230 may be connected to the network 1260 to communicate (e.g., send and receive) media information to and from the network 1260. The content delivery device(s) 1240 may also be connected to the platform 1202 and/or the display 1220.

Image sensor 1219 may include any suitable image sensor capable of providing image data based on a scene. For example, image sensor 1219 may include a semiconductor charge-coupled device-based sensor, a complementary metal oxide semiconductor-based sensor, an N-type metal oxide semiconductor-based sensor, etc. For example, image sensor 1219 may include any device capable of detecting information of a scene to generate image data.

In various implementations, the content services device 1230 may include a cable television box, a personal computer, a network, a telephone, an Internet-enabled device, or an appliance capable of delivering digital information and/or content, and other similar devices capable of unidirectionally or bidirectionally communicating content between a content provider and the platform 1202 and/or the display 1220 via the network 1260 or directly. It will be appreciated that content may be unidirectionally and/or bidirectionally communicated to and/or from any one of the components and content providers in the system 1200 via the network 1260. Examples of content may include any media information, including, for example, video, music, medical, and gaming information, etc.

The content services device 1230 can receive content, such as cable television programming, including media information, digital information, and/or other content. Examples of content providers may include any cable or satellite television, or over-the-air or Internet content provider. The examples provided are not intended to limit in any way implementations according to the present disclosure.

In various implementations, platform 1202 can receive control signals from a navigation controller 1250 having one or more navigation functions. The navigation functions of navigation controller 1250 can be used, for example, to interact with user interface 1222. In various embodiments, navigation controller 1250 can be a pointing device, which can be a computer hardware component (particularly a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems, such as graphical user interfaces (GUIs), as well as televisions and monitors, allow a user to control a computer or television and provide data using physical gestures.

Movement of navigation features of the navigation controller 1250 may be replicated on a display (e.g., display 1220) by movement of a pointer, cursor, focus ring, or other visual indicator displayed on the display. For example, under control of the software application 1216, navigation features located on the navigation controller 1250 may be mapped to virtual navigation features displayed on the user interface 1222, for example. In various embodiments, the navigation controller 1250 may not be a separate component, but may be integrated into the platform 1202 and/or the display 1220. However, this disclosure is not limited to the elements shown or in the context described herein.

In various implementations, the drivers (not shown) may include technology that, when enabled, allows a user to instantly turn the platform 1202 on or off like a television with the touch of a button after initial boot-up. Program logic allows the platform 1202 to stream content to a media adapter or other content service device 1230 or content delivery device 1240 even when the platform is turned "off." In addition, the chipset 1205 may include hardware and/or software support for, for example, 5.1 surround sound audio and/or high definition 7.1 surround sound audio. The drivers may include graphics drivers for an integrated graphics platform. In various embodiments, the graphics drivers may include a peripheral component interconnect (PCI) graphics card.

In various implementations, any one or more of the components shown in system 1200 may be integrated. For example, platform 1202 and content services device 1230 may be integrated, or platform 1202 and content delivery device 1240 may be integrated, or platform 1202, content services device 1230, and content delivery device 1240 may be integrated, for example. In various embodiments, platform 1202 and display 1220 may be an integrated unit. Display 1220 and content services device 1230 may be integrated, for example, or display 1220 and content delivery device 1240 may be integrated. These examples are not intended to limit the disclosure.

In various embodiments, system 1200 may be implemented as a wireless system, a wired system, or a combination of both. When implemented as a wireless system, system 1200 may include components and interfaces suitable for communicating over a wireless shared media, such as one or more antennas, transmitters, receivers, transceivers, amplifiers, filters, control logic, etc. One example of a wireless shared media may include a portion of a wireless spectrum, such as the RF spectrum. When implemented as a wired system, system 1200 may include components and interfaces suitable for communicating over a wired communication media, such as input/output adapters, physical connectors for connecting input/output (I/O) adapters to corresponding wired communication media, network interface cards (NICs), disk controllers, video controllers, audio controllers, etc. Examples of wired communication media may include wires, cables, metal leads, printed circuit boards (PCBs), backplanes, switch fabrics, semiconductor materials, twisted pair wires, coaxial cables, optical fibers, etc.

The platform 1202 can establish one or more logical or physical channels for communicating information. The information can include media information and control information. Media information can refer to any data that represents content intended for a user. Examples of content can include, for example, data from an audio conversation, a video conference, streaming video, an email message, a voicemail message, alphanumeric symbols, graphics, images, video, text, etc. Data from an audio conversation can be, for example, speech information, periods of silence, background noise, comfort noise, tones, etc. Control information can refer to any data that represents commands, instructions, or control words for an automated system. For example, control information can be used to route media information through a system or to instruct a node to process the media information in a predetermined manner. However, embodiments are not limited to the elements or context shown or described in FIG. 12.

As mentioned above, system 1200 may be embodied in a variety of physical styles or form factors. FIG. 13 illustrates an example small form factor device 1300 configured in accordance with at least some embodiments of the present disclosure. In some examples, system 1200 may be implemented via device 1300. In other examples, other systems, components, or modules described herein, or portions thereof, may be implemented via device 1300. In various embodiments, for example, device 1300 may be implemented as a mobile computing device having wireless capabilities. A mobile computing device may refer to any device having a processing system and a mobile power source or power source, such as, for example, one or more batteries.

Examples of mobile computing devices may include personal computers (PCs), laptop computers, ultra-laptop computers, tablets, touchpads, portable computers, handheld computers, personal digital assistants (PDAs), mobile phones, mobile phone/PDA combinations, smart devices (e.g., smartphones, smart tablets, or smart televisions), mobile internet devices (MIDs), messaging devices, data communications devices, cameras (e.g., point-and-shoot cameras, superzoom cameras, digital single lens reflex (DSLR) cameras), and the like.

Examples of mobile computing devices may also include computers configured to be implemented by automobiles or robots, or computers worn by a person, such as wrist computers, finger computers, ring computers, eyeglass computers, belt clip computers, armband computers, shoe computers, clothing computers, and other wearable computers. In various embodiments, for example, a mobile computing device may be implemented as a smartphone capable of executing computer applications as well as voice and/or data communications. Some embodiments may be described using a mobile computing device implemented as a smartphone as an example, although it will be understood that other embodiments may be implemented using other wireless mobile computing devices as well. The embodiments are not limited in this context.

As shown in FIG. 13, device 1300 may include a housing having a front 1301 and a back 1302. Device 1300 includes a display 1304, an input/output (I/O) device 1306, a color camera 1321, a color camera 1322, and an integrated antenna 1308. In some embodiments, color camera 1321 and color camera 1322 acquire planar images as described herein. In some embodiments, device 1300 does not include color cameras 1321 and 1322, and device 1300 acquires input image data from another device (e.g., any input image data described herein). Device 1300 may also include navigation functionality 1312. I/O device 1306 may include any suitable I/O device for inputting information into a mobile computing device. Examples of I/O devices 1306 may include an alphanumeric keyboard, numeric keypad, touchpad, input keys, buttons, switches, microphones, speakers, voice recognition devices, and software, etc. Information may also be entered into device 1300 via a microphone (not shown) or digitized by a voice recognition device. As shown, device 1300 may include color cameras 1321, 1322, and a flash 1310 integrated into the back 1302 (or elsewhere) of device 1300. In other examples, color cameras 1321, 1322, and a flash 1310 may be integrated into the front 1301 of device 1300, or both a front and back set of cameras may be provided. Color cameras 1321, 1322 and flash 1310 may be components of a camera module for generating color image data with IR texture correction that may be output to display 1304 and/or processed into images or streaming video that may be remotely communicated from device 1300 via antenna 1308, for example.

Various embodiments may be implemented using a combination of hardware elements, software elements, or both. Examples of hardware elements may include a processor, a microprocessor, a circuit, a circuit element (e.g., a transistor, a resistor, a capacitor, an inductor, etc.), an integrated circuit, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a digital signal processor (DSP), a field programmable gate array (PFGA), a logic gate, a register, a semiconductor device, a chip, a microchip, a chipset, etc. Examples of software may include a software component, a program, an application, a computer program, an application program, a system program, a machine program, an operating system software, a middleware, a firmware, a software module, a routine, a subroutine, a function, a method, a procedure, a software interface, an application program interface (API), an instruction set, a computing code, a computer code, a code segment, a computer code segment, a word, a bayou, a symbol, or any combination thereof. Determining which embodiment to implement using hardware and/or software elements may vary according to any number of factors, such as desired computational speed, power levels, thermal tolerance, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints.

One or more aspects of at least one embodiment may be implemented by representative instructions stored on a machine-readable medium that represent various logic within a processor, which, when read by a machine, causes the machine to create logic for performing the techniques described herein. Such representations, known as IP cores, are stored on tangible machine-readable media and supplied to various customers or manufacturing facilities for loading into manufacturing machines that actually produce the logic or processor.

While certain features described herein have been described with reference to various embodiments, this description is not intended to be construed in a limiting sense. Accordingly, various modifications of the embodiments described herein, as well as other embodiments, that are apparent to those skilled in the art to which this disclosure pertains, are deemed to be within the spirit and scope of the disclosure.

In one or more first embodiments, a method for encoding immersive video data includes receiving a first depth view having a first depth value at a first bit depth, the first bit depth including a first available range of values from a first minimum depth value to a first maximum depth value; mapping the first depth value to a second depth value at a second bit depth less than the first bit depth, the second bit depth including a second available range of values from a second minimum depth value to a second maximum depth value, using first and second line segment endpoints to define line segments for the mapping, the horizontal and vertical components of the first line segment endpoints exceeding the first and second minimum depth values, respectively, to generate a second depth view; and encoding the second depth view and the first and second line segment endpoints into a bitstream.

In one or more second embodiments, further relative to the first embodiment, the horizontal and vertical components of the second line segment endpoints are less than the first and second maximum depth values, respectively.

In one or more third embodiments, further to the first or second embodiment, the first line segment endpoint and the second line segment endpoint define a first line segment of the mapping, and the mapping further includes a second line segment defined by the second line segment endpoint and a third line segment endpoint.

In one or more fourth embodiments, further to any of the first to third embodiments, the first line segment has a first slope, and the second line segment has a second slope less than the first slope in response to a first depth value pixel count for the first line segment exceeding a second depth value pixel count for the second line segment.

In one or more fifth embodiments, further to any of the first to fourth embodiments, the method further includes determining at least the first and second line segments of the mapping, the determining being performed by generating a histogram of depth value pixel counts for each depth value range using at least a portion of the first depth view, and recursively generating a number of line segments for the mapping including the first and second line segments to minimize a reconstruction error corresponding to the mapping and the inverse mapping.

In one or more sixth embodiments, further to any of the first to fifth embodiments, the method further includes determining an edge region of the first depth view, and the histogram of depth value pixel counts is generated using pixels within the edge region and excluding pixels outside the edge region.

In one or more seventh embodiments, the method further includes, relative to any of the first to sixth embodiments, encoding texture views corresponding to the first and second depth views into a bitstream.

In one or more eighth embodiments, further to any of the first to seventh embodiments, the first bit depth is 16 bits, the second bit depth is one of 8 bits or 10 bits, and the bitstream is a High Efficiency Video Coding (HEVC) compliant bitstream.

In one or more ninth embodiments, a method for decoding immersive video data includes the steps of: decoding a bitstream to determine a first depth view having a first depth value at a first bit depth, and first and second line segment endpoints for mapping the first depth view to a second depth view, the first bit depth including a first available range of values from a first minimum value to a first maximum value; mapping the first depth value to a second depth value using the first and second line segment endpoints to generate a second depth view, the second depth value being at a second bit depth greater than the first bit depth, the second bit depth including a second available range of values from a second minimum value to a second maximum value, the horizontal and vertical components of the first line segment endpoints exceeding the first minimum value and the second minimum value, respectively; and generating a synthetic view based on the second depth view.

In one or more tenth embodiments, further to the ninth embodiment, the horizontal and vertical components of the second line segment endpoint are less than the first maximum depth value and the second maximum depth value, respectively.

In one or more eleventh embodiments, further to the ninth or tenth embodiment, the first line segment endpoint and the second line segment endpoint define a first line segment of the mapping, and the mapping further includes a second line segment defined by the second line segment endpoint and a third line segment endpoint.

In one or more twelfth embodiment, further to any of the ninth to eleventh embodiments, the first line segment has a first slope; and
The second line segment has a second slope that is less than the first slope.

In one or more thirteenth embodiments, the method further includes, relative to any of the ninth to twelfth embodiments, a step of decoding texture views corresponding to the first depth view and the second depth view from the bitstream, and the synthetic view is generated based on the texture views.

In one or more of the fourteenth embodiments, further to any of the ninth to thirteenth embodiments, the first bit depth is one of 8 bits or 10 bits, the second bit depth is 16 bits, and the bitstream is a High Efficiency Video Coding (HEVC) compliant bitstream.

In one or more fifteenth embodiments, a method for encoding immersive video data includes determining a camera projection mode for each of a plurality of cameras in a camera array capable of immersive video generation or capture, generating camera projection mode flags for the cameras, the camera projection mode flags having a first value responsive to each of the cameras having a particular projection mode and a second value responsive to any of the cameras having a projection mode other than the particular projection mode, and encoding the camera projection mode flags into a bitstream.

In one or more sixteenth embodiments, in response to a first camera projection mode flag having the first value, the bitstream further includes a single camera projection mode indicator indicating the particular projection mode, relative to the fifteenth embodiment.

In one or more of the seventeenth embodiments, further to the fifteenth or sixteenth embodiments, in response to the camera projection mode flag having the second value, the bitstream further includes a plurality of camera projection mode indicators, one for each of the cameras.

In one or more of the eighteenth embodiments, further to any of the fifteenth to seventeenth embodiments, the camera projection mode includes at least one of an omnidirectional projection mode, a normal viewpoint projection mode, or an orthographic projection mode.

In one or more nineteenth embodiments, further to any of the fifteenth to eighteenth embodiments, the method further includes determining a minimum and a maximum physical depth achievable by each of the cameras, generating a physical depth flag for the cameras, the physical depth flag indicating whether the cameras have a shared minimum physical depth and/or a shared maximum physical depth, and encoding the physical depth flag into the bitstream.

In one or more twentieth embodiments, further to any of the fifteenth to nineteenth embodiments, the method further includes the steps of determining one of yaw, pitch, and roll parameters or quaternion parameters for each of the cameras, generating yaw, pitch, and roll flags or quaternion flags for the cameras, the yaw, pitch, and roll flags or the quaternion flags indicating whether the cameras have shared yaw, pitch, and roll parameters or shared quaternion parameters, and encoding the yaw, pitch, and roll flags or the quaternion flags into the bitstream.

In one or more twenty-first embodiments, further to any of the fifteenth to twentieth embodiments, the yaw, pitch, and roll parameters or the quaternion parameters indicate default yaw, pitch, and roll or default quaternion parameter flag values.

In one or more twenty-second embodiments, further to any of the fifteenth to twenty-first embodiments, the method further includes determining a shared physical spacing value representing a shared physical spacing between the cameras, and encoding the shared physical spacing value into the bitstream.

In one or more twenty-third embodiments, a method for decoding immersive video data includes decoding a bitstream to determine a depth view, a texture view, and camera projection mode flags corresponding to a plurality of cameras of a camera array, the camera projection mode flags having a first value indicating that all of the cameras have a particular projection mode or a second value indicating that any of the cameras have a projection mode other than the particular projection mode; and generating a decoded view based on the depth view, the texture view, and the depth view of the camera projection mode flags.

In one or more of the twenty-fourth embodiments, further to the twenty-third embodiment, in response to the camera projection mode flag having the first value, the bitstream further includes a single camera projection mode indicator indicating the particular projection mode.

In one or more of the twenty-fifth embodiment, further to the twenty-third or twenty-fourth embodiment, in response to the camera projection mode flag having the second value, the bitstream further includes a plurality of camera projection mode indicators, one for each of the cameras.

In one or more of the twenty-sixth embodiment, further to any of the twenty-third to twenty-fifth embodiments, the shared camera projection mode includes at least one of an omnidirectional projection mode, a normal viewpoint projection mode, or an orthogonal projection mode.

In one or more of the twenty-seventh embodiment, further to any of the twenty-third to twenty-sixth embodiments, the bitstream further includes a physical depth flag for the camera, the physical depth flag indicating whether the camera has a shared minimum physical depth and/or a shared maximum physical depth.

In one or more twenty-eighth embodiments, further to any of the twenty-third to twenty-seventh embodiments, the bitstream further includes yaw, pitch, and roll flags or quaternion flags for the cameras, the yaw, pitch, and roll flags or the quaternion flags indicating whether the cameras have shared yaw, pitch, and roll parameters or shared quaternion parameters.

In one or more twenty-ninth embodiments, further to any of the twenty-third to twenty-eighth embodiments, the method further includes the step of setting the yaw, pitch, and roll parameters, or the quaternion parameters, for each of the cameras, to default yaw, pitch, and roll parameters, or default quaternion parameters, in response to a yaw, pitch, and roll flag, or a quaternion flag indicating that the cameras have shared yaw, pitch, and roll parameters, or shared quaternion parameters.

In one or more 30th embodiments, further to any of the 23rd to 28th embodiments, the bitstream further includes a shared physical spacing value, and the method further includes a step of determining a physical spacing between the cameras using the shared physical spacing value.

In one or more thirty-first embodiments, a device or system includes a memory and one or more processors for performing a method according to any one of the above embodiments.

In one or more thirty-second embodiments, at least one machine-readable medium containing a plurality of instructions causes a computing device to perform a method according to any one of the above embodiments in response to the instructions being executed on the computing device.

In one or more of the thirty-third embodiments, the apparatus includes means for carrying out a method according to any one of the above embodiments.

It will be recognized that the embodiments are not limited to the above embodiments, but can be practiced with modification and alteration without departing from the scope of the appended claims. For example, the above embodiments may include a particular combination of features. However, the above embodiments are not limited in this respect, and in various embodiments, the above embodiments may include taking on only a subset of such features, taking on a different order of such features, taking on different combinations of such features, and/or taking on additional features other than those expressly recited. The scope of the embodiments should therefore be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims (13)

1. A method for encoding immersive video data, comprising:
receiving a first depth view having a first depth value at a first bit depth;
the first bit depth comprises a first available range of values from a first minimum depth value to a first maximum depth value;
Steps and
mapping the first depth values to second depth values at a second bit depth less than the first bit depth to generate a second depth view;
the second bit depth comprises a second available range of values from a second minimum depth value to a second maximum depth value;
The mapping is performed by using a first line segment end point and a second line segment end point to define a line segment for the mapping;
horizontal and vertical components of the first line segment endpoints correspond to the first and second depth values and exceed the first and second minimum depth values, respectively;
Steps and
encoding a second depth view and the first and second line segment endpoints into a bitstream;
A method comprising: a horizontal component and a vertical component of the second line segment endpoint are less than the first maximum depth value and the second maximum depth value, respectively;
The method of claim 1. the first line segment endpoint and the second line segment endpoint define a first line segment of the mapping;
the mapping further includes a second line segment defined by the second line segment endpoint and a third line segment endpoint.
The method according to claim 1 or 2. the first line segment has a first slope; and
the second line segment having a second slope less than the first slope in response to a first depth value pixel count for the first line segment exceeding a second depth value pixel count for the second line segment.
The method according to claim 3. The method further comprises:
determining at least the first and second line segments of the mapping;
The said decision:
generating a histogram of depth value pixel counts for each depth value range using at least a portion of the first depth view;
recursively generating a number of line segments for the mapping including the first line segment and the second line segment to minimize a reconstruction error corresponding to the mapping and the inverse mapping;
carried out by
The method according to claim 3 or 4. The method further comprises:
determining an edge region of the first depth view;
the histogram of depth value pixel counts is generated using pixels within the edge region and excluding pixels outside the edge region.
The method according to claim 5. 1. A method for decoding immersive video data, comprising:
decoding the bitstream to determine a first depth view having a first depth value at a first bit depth and first and second line segment end points for mapping the first depth view to a second depth view;
the first bit depth comprises a first available range of values from a first minimum value to a first maximum value;
Steps and
mapping the first depth values to second depth values to generate a second depth view;
the second depth value is a second bit depth greater than the first bit depth;
the second bit depth comprises a second available range of values from a second minimum value to a second maximum value;
the mapping is performed by using the first line segment end point and the second line segment end point to define a line segment for the mapping;
horizontal and vertical components of the first line segment endpoints correspond to the first and second depth values and exceed the first and second minimum values, respectively;
Steps and
generating a synthetic view based on the second depth view;
A method comprising: the horizontal and vertical components of the second line segment endpoint are less than the first maximum depth value and the second maximum depth value, respectively;
The method of claim 7. the first line segment endpoint and the second line segment endpoint define a first line segment of the mapping;
the mapping further includes a second line segment defined by the second line segment endpoint and a third line segment endpoint.
9. The method according to claim 7 or 8. the first line segment has a first slope; and
the second line segment has a second slope that is less than the first slope.
10. The method of claim 9. The method further comprises:
decoding texture views corresponding to the first depth view and the second depth view from the bitstream;
the synthetic view is generated based on the texture view.
Steps, including:
11. The method according to any one of claims 7 to 10. Memory,
One or more processors for carrying out the method of any one of claims 1 to 11;
Including, the system. At least one machine-readable medium containing a plurality of instructions,
In response to instructions being executed on a computing device, causing the computing device to perform the method of any one of claims 1 to 11.
Machine-readable media.

Images