JP7524390B2 - VIDEO ENCODER, VIDEO DECODER, AND CORRESPONDING METHODS - Patent application - Google Patents

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims the benefit of U.S. Provisional Patent Application No. 62/786,167, entitled "Flexible Tiling in Video Coding," filed Dec. 28, 2018 by Ye-Kui Wang et al., which is incorporated herein by reference.

This disclosure relates generally to video coding, and more particularly to a flexible video tiling scheme that supports multiple tiles with different resolutions within the same picture.

The amount of video data required to depict even a relatively short video can be substantial, which can pose difficulties when the data is to be streamed or otherwise communicated across communication networks with limited bandwidth capacity. Therefore, video data is typically compressed before being communicated across modern telecommunications networks. The size of the video can also be an issue when the video is stored on a storage device, since memory resources may be limited. Video compression devices often use software and/or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data required to depict a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. With limited network resources and an ever-increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little sacrifice in image quality are desirable.

In one embodiment, the disclosure includes a method implemented in an encoder, the method including: partitioning a picture into a plurality of first level tiles by a processor of the encoder; partitioning a subset of the first level tiles into a plurality of second level tiles by the processor; assigning the first level tiles and the second level tiles to one or more tile groups by the processor such that each tile group includes several first level tiles, one or more contiguous sequences of second level tiles where each sequence of second level tiles is divided from a single first level tile, or a combination thereof; encoding the first level tiles and the second level tiles by the processor into a bitstream; and storing the bitstream in a memory of the encoder for communication to a decoder. Some streaming applications (e.g., virtual reality (VR) and teleconferencing) can be improved if a single image including multiple regions encoded at different resolutions can be sent. Some slicing and tiling mechanisms, such as raster scan-based slicing and/or tiling, may not support such functionality since tiles at different resolutions may be treated differently. For example, a tile at a first resolution may contain a single slice of data, while a tile at a second resolution may carry multiple slices of data due to differences in pixel density. To support this functionality, a flexible tiling scheme may be employed that includes first level tiles and second level tiles. The second level tiles are created by partitioning the first level tiles. This tiling scheme allows the first level tiles to contain one slice of data at the first resolution and the first level tiles including the second level tiles to contain multiple slices at the second resolution. Such a flexible tiling scheme allows an encoder/decoder (codec) to support pictures that include multiple resolutions, thus enhancing the capabilities of both the encoder and the decoder. This disclosure describes a mechanism for integrating tile groups into the flexible tiling scheme. A tile group may include a complete set of first level tiles and/or second level tiles partitioned from one or more first level tiles. This approach prevents second level tiles from a single first level tile from being split into different tile groups. Thus, the disclosed mechanism allows tiles of a flexible tiling scheme to be included in tile groups, which allows coding tools to be applied to various tiles based on the tile groups. By preventing second level tiles from a single first level tile from being split into different groups, the complexity of the resulting flexible tiling scheme with tile groups is reduced. Thus, the present disclosure further enhances the functionality of both the encoder and the decoder while reducing processor and/or memory resource usage.

In one embodiment, the disclosure includes a method implemented in an encoder, the method including: partitioning a picture into a plurality of first level tiles by a processor of the encoder; partitioning a subset of the first level tiles into a plurality of second level tiles by the processor; assigning the first level tiles and the second level tiles to one or more tile groups by the processor such that all second level tiles created from a single first level tile are assigned to the same tile group; encoding the first level tiles and the second level tiles into a bitstream by the processor; and storing the bitstream in a memory of the encoder for communication to a decoder. Some streaming applications (e.g., VR and teleconferencing) can be improved if a single image including multiple regions encoded at different resolutions can be sent. Some slicing and tiling mechanisms, such as raster scan-based slicing and/or tiling, may not support such functionality since tiles at different resolutions may be treated differently. For example, a tile at a first resolution may contain a single slice of data, while a tile at a second resolution may carry multiple slices of data due to differences in pixel density. To support this functionality, a flexible tiling scheme may be employed that includes first level tiles and second level tiles. The second level tiles are created by partitioning the first level tiles. This tiling scheme allows the first level tiles to contain one slice of data at the first resolution and the first level tiles including the second level tiles to contain multiple slices at the second resolution. Such a flexible tiling scheme allows the codec to support pictures that include multiple resolutions, thus enhancing the capabilities of both the encoder and the decoder. This disclosure describes a mechanism for integrating tile groups into the flexible tiling scheme. A tile group may include a complete set of first level tiles and/or second level tiles partitioned from one or more first level tiles. This approach prevents second level tiles from a single first level tile from being split into different tile groups. Thus, the disclosed mechanism allows tiles of a flexible tiling scheme to be included in tile groups, which allows coding tools to be applied to various tiles based on the tile groups. By preventing second level tiles from a single first level tile from being split into different groups, the complexity of the resulting flexible tiling scheme with tile groups is reduced. Thus, the present disclosure further enhances the functionality of both the encoder and the decoder while reducing processor and/or memory resource usage.

Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the first level tiles outside the subset include picture data at a first resolution and the second level tiles include picture data at a second resolution different from the first resolution.

Optionally, in any of the above aspects, another implementation of the aspect provides that each first level tile in the subset of first level tiles includes two or more complete second level tiles.

Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the first level tiles and the second level tiles are encoded according to a scan order, and encoding according to the scan order includes encoding the first level tiles in raster scan order, interrupting the raster scan order encoding of the first level tiles upon encountering one of the second level tiles, and encoding all successive second level tiles in raster scan order before continuing the raster scan order encoding of the first level tiles.

Optionally, in any of the above aspects, another implementation of the aspect provides that all second level tiles partitioned from a current first level tile are encoded before encoding any second level tiles partitioned from a subsequent second level tile.

Optionally, in any of the above aspects, another implementation of the aspect provides that each of the one or more tile groups is constrained such that all tiles in the assigned tile group cover a rectangular portion of the picture.

In one embodiment, the disclosure includes a method implemented in a decoder, the method including receiving, by a processor of the decoder via a receiver, a bitstream including a picture partitioned into a plurality of first level tiles, wherein a subset of the first level tiles are further partitioned into a plurality of second level tiles, and the first level tiles and the second level tiles are assigned to one or more tile groups such that each tile group includes several first level tiles, one or more contiguous sequences of second level tiles where each sequence of second level tiles is divided from a single first level tile, or a combination thereof; decoding, by the processor, the first level tiles and the second level tiles based on the one or more tile groups; and generating, by the processor, a reconstructed video sequence for display based on the decoded first level tiles and the second level tiles. Some streaming applications (e.g., VR and teleconferencing) can be improved when a single image including multiple regions encoded at different resolutions can be sent. Some slicing and tiling mechanisms, such as raster-scan based slicing and/or tiling, may not support such functionality since tiles at different resolutions may be treated differently. For example, a tile at a first resolution may contain a single slice of data, while a tile at a second resolution may carry multiple slices of data due to differences in pixel density. To support this functionality, a flexible tiling scheme may be employed that includes first level tiles and second level tiles. The second level tiles are created by partitioning the first level tiles. This tiling scheme allows the first level tiles to include one slice of data at the first resolution and the first level tiles, including the second level tiles, to include multiple slices at the second resolution. Such a flexible tiling scheme allows the codec to support pictures that include multiple resolutions, thus enhancing the capabilities of both the encoder and the decoder. This disclosure describes mechanisms for integrating tile groups into a flexible tiling scheme. A tile group may include a complete set of first level tiles and/or second level tiles partitioned from one or more first level tiles. This approach prevents second level tiles from a single first level tile from being split into different tile groups. Thus, the disclosed mechanism allows tiles of a flexible tiling scheme to be included in tile groups, which allows coding tools to be applied to various tiles based on the tile groups. By preventing second level tiles from a single first level tile from being split into different groups, the complexity of the resulting flexible tiling scheme with tile groups is reduced. Thus, the present disclosure further enhances the functionality of both the encoder and the decoder while reducing processor and/or memory resource usage.

In one embodiment, the disclosure includes a method implemented in a decoder, the method including receiving, by a processor of the decoder via a receiver, a bitstream including a picture partitioned into a plurality of first level tiles, where a subset of the first level tiles are further partitioned into a plurality of second level tiles, and the first level tiles and the second level tiles are assigned to one or more tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group; decoding, by the processor, the first level tiles and the second level tiles based on the one or more tile groups; and generating, by the processor, a reconstructed video sequence for display based on the decoded first level tiles and the second level tiles. Some streaming applications (e.g., VR and teleconferencing) can be improved if a single image including multiple regions encoded at different resolutions can be sent. Some slicing and tiling mechanisms, such as raster scan-based slicing and/or tiling, may not support such functionality since tiles at different resolutions may be treated differently. For example, a tile at a first resolution may contain a single slice of data, while a tile at a second resolution may carry multiple slices of data due to differences in pixel density. To support this functionality, a flexible tiling scheme may be employed that includes first level tiles and second level tiles. The second level tiles are created by partitioning the first level tiles. This tiling scheme allows the first level tiles to contain one slice of data at the first resolution and the first level tiles including the second level tiles to contain multiple slices at the second resolution. Such a flexible tiling scheme allows the codec to support pictures that include multiple resolutions, thus enhancing the capabilities of both the encoder and the decoder. This disclosure describes a mechanism for integrating tile groups into the flexible tiling scheme. A tile group may include a complete set of first level tiles and/or second level tiles partitioned from one or more first level tiles. This approach prevents second level tiles from a single first level tile from being split into different tile groups. Thus, the disclosed mechanism allows tiles of a flexible tiling scheme to be included in tile groups, which allows coding tools to be applied to various tiles based on the tile groups. By preventing second level tiles from a single first level tile from being split into different groups, the complexity of the resulting flexible tiling scheme with tile groups is reduced. Thus, the present disclosure further enhances the functionality of both the encoder and the decoder while reducing processor and/or memory resource usage.

Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the first level tiles outside the subset include picture data at a first resolution and the second level tiles include picture data at a second resolution different from the first resolution.

Optionally, in any of the above aspects, another implementation of the aspect provides that each first level tile in the subset of first level tiles includes two or more complete second level tiles.

Optionally, in any of the aforementioned aspects, another implementation of the aspect provides that the first level tiles and the second level tiles are decoded according to a scan order, and decoding according to the scan order includes decoding the first level tiles in raster scan order, interrupting the raster scan order encoding of the first level tiles upon encountering one of the second level tiles, and encoding all successive second level tiles in raster scan order before continuing the raster scan order decoding of the first level tiles.

Optionally, in any of the above aspects, another implementation of the aspect provides that all second level tiles partitioned from a current first level tile are decoded before decoding any second level tiles partitioned from a subsequent second level tile.

Optionally, in any of the above aspects, another implementation of the aspect provides that each of the one or more tile groups is constrained such that all tiles in the assigned tile group cover a rectangular portion of the picture.

In one embodiment, the present disclosure includes a video coding device comprising a processor, a receiver coupled to the processor, and a transmitter coupled to the processor, the processor, the receiver, and the transmitter configured to perform a method of any of the aforementioned aspects.

In one embodiment, the present disclosure includes a non-transitory computer-readable medium comprising a computer program product for use by a video coding device, the computer program product comprising computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, cause the video coding device to perform a method of any of the foregoing aspects.

In one embodiment, the present disclosure includes an encoder comprising partitioning means for partitioning a picture into a plurality of first level tiles and a subset of the first level tiles into a plurality of second level tiles, allocation means for assigning the first level tiles and the second level tiles to one or more tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group, encoding means for encoding the first level tiles and the second level tiles into a bitstream, and storage means for storing the bitstream for communication to a decoder.

Optionally, in any of the above aspects, another implementation of the aspect provides that the encoder is further configured to perform a method of any of the above aspects.

In one embodiment, the disclosure includes a decoder comprising: receiving means for receiving a bitstream including a picture partitioned into a plurality of first level tiles, where a subset of the first level tiles are further partitioned into a plurality of second level tiles, and the first level tiles and the second level tiles are assigned to one or more tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group; decoding means for decoding the first level tiles and the second level tiles based on the one or more tile groups; and generating means for generating a reconstructed video sequence for display based on the decoded first level tiles and second level tiles.

Optionally, in any of the above aspects, another implementation of the aspect provides that the decoder is further configured to perform any of the methods of any of the above aspects.

For clarity, any one of the above embodiments may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present disclosure.

These and other features will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings and claims.

For a more complete understanding of the present disclosure, reference is now made to the accompanying drawings and the following brief description, which are incorporated in conjunction with the detailed description, in which like reference numerals represent like parts.

4 is a flowchart of an exemplary method for coding a video signal. 1 is a schematic diagram of an example coding and decoding (codec) system for video coding. 1 is a schematic diagram illustrating an example video encoder. FIG. 2 is a schematic diagram illustrating an exemplary video decoder. 1 is a schematic diagram illustrating an exemplary bitstream including an encoded video sequence. FIG. 1 illustrates an example mechanism for creating an extractor track to combine multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. FIG. 1 illustrates an example mechanism for creating an extractor track to combine multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. FIG. 1 illustrates an example mechanism for creating an extractor track to combine multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. FIG. 1 illustrates an example mechanism for creating an extractor track to combine multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. FIG. 1 illustrates an example mechanism for creating an extractor track to combine multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. FIG. 1 illustrates an example video conferencing application in which multiple resolution pictures from different bitstreams are stitched together into a single picture for display. FIG. 2 is a schematic diagram illustrating an example flexible video tiling scheme capable of supporting multiple tiles with different resolutions within the same picture. FIG. 2 is a schematic diagram illustrating an example flexible video tiling scheme capable of supporting multiple tiles with different resolutions within the same picture. FIG. 2 is a schematic diagram illustrating an example flexible video tiling scheme capable of supporting multiple tiles with different resolutions within the same picture. FIG. 2 is a schematic diagram illustrating an example flexible video tiling scheme capable of supporting multiple tiles with different resolutions within the same picture. 1 is a schematic diagram of an example video coding device. 1 is a flowchart of an exemplary method for encoding an image by employing a flexible tiling scheme. 1 is a flowchart of an exemplary method for decoding an image by employing a flexible tiling scheme. 1 is a schematic diagram of an example system for coding a video sequence by employing a flexible tiling scheme.

First, an example implementation of one or more embodiments is provided below, but it should be understood that the disclosed system and/or method may be implemented using any number of techniques, whether currently known or in existence. The present disclosure should in no way be limited to the example implementations, drawings, and techniques shown below, including the example designs and implementations shown and described herein, and may be modified within the scope of the appended claims together with their full scope of equivalents.

Various acronyms are employed herein, such as coding tree block (CTB), coding tree unit (CTU), coding unit (CU), coded video sequence (CVS), Joint Video Experts Team (JVET), motion constrained tile set (MCTS), maximum transfer unit (MTU), network abstraction layer (NAL), picture order count (POC), raw byte sequence payload (RBSP), sequence parameter set (SPS), versatile video coding (VVC), and working draft (WD).

Many video compression techniques may be employed to reduce the size of video files with minimal loss of data. For example, video compression techniques may include performing spatial (e.g., intra-picture) prediction and/or temporal (e.g., inter-picture) prediction to reduce or remove data redundancy in a video sequence. For block-based video coding, a video slice (e.g., a video picture or a portion of a video picture) may be partitioned into video blocks, which may be referred to as tree blocks, coding tree blocks (CTBs), coding tree units (CTUs), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial prediction with reference to reference samples in neighboring blocks in the same picture. Video blocks in an inter-coded unidirectionally predictive (P) or bidirectionally predictive (B) slice of a picture may be coded by employing spatial prediction with reference to reference samples in neighboring blocks in the same picture or temporal prediction with reference to reference samples in other reference pictures. A picture may be referred to as a frame and/or an image, and a reference picture may be referred to as a reference frame and/or a reference image. Spatial or temporal prediction results in a predictive block that represents an image block. The residual data represents pixel differences between the original image block and the predictive block. Thus, an inter-coded block is coded according to a motion vector that points to a block of reference samples forming the predictive block, and the residual data that indicates the difference between the coded block and the predictive block. An intra-coded block is coded according to an intra-coding mode and the residual data. For further compression, the residual data may be transformed from the pixel domain to a transform domain. These result in residual transform coefficients, which may be quantized. The quantized transform coefficients may be initially arranged in a two-dimensional array. The quantized transform coefficients may be scanned to produce a one-dimensional vector of transform coefficients. To achieve even further compression, entropy coding may be applied. Such video compression techniques are described in more detail below.

To ensure that the encoded video may be accurately decoded, the video is encoded and decoded according to a corresponding video coding standard. Video coding standards include Advanced Video Coding (AVC), also known as International Telecommunications Union (ITU) Standardization Sector (ITU-T) H.261, International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Motion Picture Experts Group (MPEG)-1 Part 2, ITU-T H.262 or ISO/IEC MPEG-2 Part 2, ITU-T H.263, ISO/IEC MPEG-4 Part 2, ITU-T H.264 or ISO/IEC MPEG-4 Part 10, and High Efficiency Video Coding (HEVC), also known as ITU-T H.265 or MPEG-H Part 2. AVC includes extensions such as Scalable Video Coding (SVC), Multiview Video Coding (MVC) and Multiview Video Coding Plus Depth (MVC+D), and three-dimensional (3D) AVC (3D-AVC). HEVC includes extensions such as Scalable HEVC (SHVC), Multiview HEVC (MV-HEVC), and 3D HEVC (3D-HEVC). The ITU-T and ISO/IEC Joint Video Experts Team (JVET) has begun developing a video coding standard called Versatile Video Coding (VVC). VVC is included in the Working Drafts (WDs) including JVET-L1001-v5.

To code a video image, the image is first partitioned and the partition is coded into a bitstream. Various picture partitioning schemes are available. For example, an image may be partitioned into normal slices, dependent slices, tiles, and/or according to Wavefront Parallel Processing (WPP). For simplicity, HEVC restricts the encoder to only use normal slices, dependent slices, tiles, WPP, and combinations thereof when partitioning slices into groups of CTBs for video coding. Such partitioning may be applied to support maximum transmission unit (MTU) size alignment, parallel processing, and reduced end-to-end delay. The MTU indicates the maximum amount of data that can be transmitted in a single packet. If a packet payload exceeds the MTU, it is split into two packets through a process called fragmentation.

A normal slice, also simply called a slice, is a partitioned portion of an image that may be reconstructed independently from other normal slices in the same picture, despite some interdependencies due to loop filtering operations. Each normal slice is encapsulated within its own network abstraction layer (NAL) unit for transmission. Furthermore, intra-picture prediction (intra-sample prediction, motion information prediction, coding mode prediction) and entropy coding dependencies across slice boundaries may be disabled to support independent reconstruction. Such independent reconstruction supports parallelization. For example, normal slice-based parallelization employs minimal inter-processor or inter-core communication. However, each normal slice is independent, and each slice is associated with a separate slice header. The use of normal slices may incur significant coding overhead due to the bit cost of the slice header per slice and due to the lack of prediction across slice boundaries. Furthermore, normal slices may be employed to support alignment against MTU size requirements. In particular, because regular slices may be encapsulated in separate NAL units and coded independently, each regular slice should be smaller than the MTU in the MTU scheme to avoid breaking the slice into multiple packets. Thus, the goals of parallelization and MTU size matching may impose conflicting demands on slice layout within a picture.

Dependent slices are similar to normal slices, but have a shortened slice header, allowing for partitioning of picture tree block boundaries without breaking intra-picture prediction. Thus, dependent slices allow normal slices to be fragmented into multiple NAL units, which results in reduced end-to-end delay by allowing parts of a normal slice to be sent out before the coding of the entire normal slice is completed.

A tile is a partitioned portion of an image created by horizontal and vertical boundaries that create columns and rows of tiles. Tiles may be coded in raster scan order (right to left and top to bottom). The scan order of the CTBs is local within the tile. Thus, the CTB in the first tile is coded in raster scan order before proceeding to the CTB in the next tile. As with normal slices, tiles break intra-picture prediction dependencies as well as entropy decoding dependencies. However, tiles may not be included within individual NAL units, and therefore tiles may not be used for MTU size alignment. Each tile may be processed by one processor/core, and inter-processor/inter-core communication employed for intra-picture prediction between processing units that decode adjacent tiles may be limited to conveying shared slice headers (when adjacent tiles are in the same slice) and performing loop filter processing-related sharing of reconstructed samples and metadata. When more than one tile is included in a slice, the entry point byte offset for each tile, other than the initial entry point offset in the slice, may be signaled in the slice header. For each slice and tile, at least one of the following conditions should be fulfilled: 1) all coded treeblocks in a slice belong to the same tile, and 2) all coded treeblocks in a tile belong to the same slice.

In WPP, a picture is partitioned into a single row of CTBs. The entropy decoding and prediction mechanism may use data from CTBs in other rows. Parallel processing is enabled through parallel decoding of CTB rows. For example, the current row may be decoded in parallel with the previous row. However, the decoding of the current row is delayed by two CTBs from the decoding process of the previous row. This delay ensures that data related to the CTB above the current CTB and the CTBs above and to the right of the current CTB in the current row are available before the current CTB is coded. This approach looks like a wavefront when represented graphically. This staggered beginning allows parallelization with up to as many processors/cores as the picture contains CTB rows. Since intra-picture prediction between adjacent treeblock rows in a picture is allowed, inter-processor/inter-core communication to enable intra-picture prediction can be substantial. WPP partitioning takes into account the NAL unit size. Therefore, WPP does not support MTU size alignment. However, regular slices can be used with WPP to enforce MTU size alignment if necessary, with some coding overhead.

A tile may also include a motion constrained tile set. A motion constrained tile set (MCTS) is a tile set designed such that associated motion vectors are restricted to point to full sample locations inside the MCTS and fractional sample locations that require only full sample locations inside the MCTS for interpolation. Furthermore, the use of motion vector candidates for temporal motion vector prediction derived from blocks outside the MCTS is rejected. In this way, each MCTS may be decoded independently without the presence of tiles not included in the MCTS. A temporal MCTS supplemental enhancement information (SEI) message may be used to indicate the presence of the MCTS in the bitstream and to signal the MCTS. The MCTS SEI message provides additional information that may be used in MCTS sub-bitstream extraction (specified as part of the semantics of the SEI message) to generate a conforming bitstream for the MCTS. The information includes several extraction information sets, each of which defines several MCTSs and includes raw byte sequence payload (RBSP) bytes of a replacement video parameter set (VPS), sequence parameter sets (SPS), and picture parameter sets (PPS) to be used during the MCTS sub-bitstream extraction process. Upon extracting the sub-bitstreams according to the MCTS sub-bitstream extraction process, the parameter sets (VPS, SPS, and PPS) may be rewritten or replaced and slice headers may be updated, since one or all of the slice address related syntax elements (including first_slice_segment_in_pic_flag and slice_segment_address) may adopt different values in the extracted sub-bitstreams.

Various tiling schemes may be employed when partitioning a picture for further encoding. As a particular example, tiles may be assigned to tile groups, which in some examples may replace slices. In some examples, each tile group may be extracted independently of other tile groups. Thus, tile grouping may support parallelization by allowing each tile group to be assigned to a different processor. Tile groups may be assigned in raster scan order or may be constrained to form a rectangular shape of area within the picture. To support such tile groups, explicit tile identifier (ID) signaling may be used. In some systems, tile IDs are assigned to always be the same as the tile index. Explicit tile ID signaling allows the tile ID to be different from the tile index. Having explicit tile ID signaling supports extraction of the MCTS from the bitstream without the need to update the tile group header. Note that the signaling of explicit tile IDs and the corresponding use as addresses of tile groups may be specific to HEVC-style tile structure definitions and signaling. If the tile structure definition and/or signaling is modified, signaling of tile IDs via an explicit tile ID mechanism may be inaccurate and/or inapplicable in some instances. Tile grouping and explicit tile ID signaling may be employed, for example, when a decoder may not want to decode the entire image. As a particular example, a video coding scheme may be employed to support virtual reality (VR) video, which may be encoded according to the Omnidirectional Media Application Format (OMAF).

In a VR video, one or more cameras may record the environment around the camera. The user may then view the VR video as if the user were present in the same location as the camera. In a VR video, the picture encompasses the entire environment around the user. The user then sees a sub-portion of the picture. For example, the user may employ a head-mounted display that changes the sub-portion of the picture displayed based on the user's head movement. The portion of the video being displayed may be called a viewport.

Thus, a differentiating feature of omnidirectional video is that at any particular time only a viewport is displayed. This is in contrast to other video applications where the entire video may be displayed. This feature may be exploited to improve the performance of omnidirectional video systems, for example through selective delivery according to the user's viewport (or any other criteria such as recommended viewport timed metadata). Viewport-dependent delivery may be enabled, for example, by employing per-region packing and/or viewport-dependent video coding. The performance improvement may result in smaller transmission bandwidth, lower decoding complexity, or both, when compared to other omnidirectional video systems when employing the same video resolution/quality.

An exemplary viewport-dependent operation is an MCTS-based approach to achieve an effective equirectangle projection (ERP) resolution of 5000 samples (e.g., 5120×2560 luma samples) resolution (5K) using an HEVC-based viewport-dependent OMAF video profile. This approach is described in more detail below. In general, however, this approach partitions the VR video into tile groups and encodes the video at multiple resolutions. The decoder can indicate the viewport currently used by the user during streaming. The video server providing the VR video data can then transfer the tile groups associated with the viewport at high resolution and transfer the unseen tile groups at a lower resolution. This allows the user to view the VR video at high resolution without requiring the entire picture to be sent at high resolution. The unseen sub-portions are discarded, and therefore the user may not be aware of the lower resolution. However, if the user changes viewport, the lower resolution tile groups may be displayed to the user. The resolution of the new viewport may then be increased as the video progresses. To implement such a system, a picture should be created that contains both higher resolution and lower resolution tile groups.

In another example, a video conferencing application may be designed to transfer a picture containing multiple resolutions. For example, a video conference may include multiple participants. The participant currently speaking may be displayed in a higher resolution, while the other participants may be displayed in a lower resolution. To implement such a system, a picture should be created that includes both higher resolution and lower resolution tile groups.

Various flexible tiling mechanisms are disclosed herein to support creating pictures with sub-pictures coded at multiple resolutions. For example, video may be coded at multiple resolutions. Video may also be coded by employing sub-pictures at each resolution. The lower resolution sub-pictures are smaller than the higher resolution sub-pictures. To create a picture with multiple resolutions, a picture may be partitioned into first level tiles. Sub-pictures from the highest resolution may be included directly within the first level tiles. The first level tiles may further be partitioned into second level tiles that are smaller than the first level tiles. Thus, the smaller second level tiles can directly accept the lower resolution sub-pictures. In this manner, slices from each resolution may be compressed into a single picture via tile index relationships without requiring tiles of different resolutions to be dynamically re-addressed to use a coherent addressing scheme. The first level tiles and the second level tiles may be implemented as MCTSs and thus may accept motion constrained image data at different resolutions. This disclosure includes many aspects. As a particular example, a first level tile is divided into second level tiles. The first level tiles and second level tiles may then be included in a tile group. The tile group may be constrained to include an integer number of first level tiles, and/or one or more consecutive sequences of second level tiles, where each sequence of second level tiles is divided from a single first level tile. This approach may ensure that all second level tiles created from a single first level tile are assigned to the same tile group. In another particular example, a scan order for coding a flexible tiling scheme is described. In this example, the first level tiles are coded in raster scan order relative to picture and/or tile group boundaries. When a second level tile is encountered, the first level tile scan is interrupted. The consecutive sequence of second level tiles is then scanned in raster scan order relative to the first level tile from which such second level tile was partitioned. The scan order then proceeds to the next consecutive sequence of second level tiles, if any. If not, the first level tile traversal continues. This process continues until the tile group and/or picture, depending on the example, is encoded or decoded.

FIG. 1 is a flow chart of an exemplary operational method 100 of coding a video signal. In particular, a video signal is encoded in an encoder. The encoding process compresses the video signal by employing various mechanisms to reduce the video file size. The smaller file size allows the compressed video file to be transmitted towards a user while reducing the associated bandwidth overhead. A decoder then decodes the compressed video file to reconstruct the original video signal for display to the end user. The decoding process generally mirrors the encoding process to allow the decoder to consistently reconstruct the video signal.

In step 101, a video signal is input into the encoder. For example, the video signal may be an uncompressed video file stored in memory. As another example, the video file may be captured by a video capture device, such as a video camera, and may be encoded to support live streaming of the video. The video file may include both audio and video components. The video component includes a series of image frames that, when viewed in sequence, give the visual impression of motion. The frames include pixels that are represented in terms of light, referred to herein as luma components (or luma samples), and colors, referred to herein as chroma components (or color samples). In some examples, the frames may also include depth values to support three-dimensional viewing.

In step 103, the video is partitioned into blocks. Partitioning involves subdividing pixels in each frame into square and/or rectangular blocks for compression. For example, in High Efficiency Video Coding (HEVC) (also known as H.265 and MPEG-H Part 2), a frame may first be partitioned into coding tree units (CTUs), which are blocks of a predefined size (e.g., 64 pixels by 64 pixels). CTUs contain both luma and chroma samples. A coding tree may be employed to partition the CTUs into blocks and then recursively subdivide the blocks until a configuration is achieved that supports further encoding. For example, the luma component of a frame may be subdivided until each block contains relatively homogenous light values. Additionally, the chroma component of a frame may be subdivided until each block contains relatively homogenous color values. Thus, the partitioning mechanism varies depending on the content of the video frame.

In step 105, various compression mechanisms are employed to compress the image blocks partitioned in step 103. For example, inter-prediction and/or intra-prediction may be employed. Inter-prediction is designed to take advantage of the fact that objects in a common scene tend to appear in successive frames. Thus, a block showing an object in a reference frame does not need to be repeatedly described in adjacent frames. In particular, an object such as a table may remain in a constant position across multiple frames. Thus, the table may be described once and adjacent frames may refer back to the reference frame. Pattern matching mechanisms may be employed to match objects across multiple frames. Furthermore, a moving object may be depicted across multiple frames, for example due to object movement or camera movement. As a particular example, a video may show a car moving across the screen across multiple frames. Motion vectors may be employed to represent such movement. A motion vector is a two-dimensional vector that provides an offset from the coordinates of the object in a frame to the coordinates of the object in the reference frame. Thus, inter prediction can encode an image block in a current frame as a set of motion vectors that indicate an offset from a corresponding block in a reference frame.

Intra prediction encodes blocks within a common frame. Intra prediction exploits the fact that luma and chroma components tend to cluster within a frame. For example, green fragments within a portion of a tree tend to be located adjacent to similar fragments of green. Intra prediction employs multiple directional prediction modes (e.g., 33 in HEVC), planar mode, and direct current (DC) mode. The directional modes indicate that the current block is similar/same as samples of neighboring blocks in the corresponding direction. The planar mode indicates that a series of blocks (e.g., planes) along a row/column may be interpolated based on neighboring blocks at the edge of the row. The planar mode actually indicates a smooth transition of light/color across the row/column by employing a relatively constant gradient in changing values. The DC mode is employed for boundary smoothing and indicates that the block is similar/same as the average value associated with the samples of all neighboring blocks associated with the angular direction of the directional prediction mode. Thus, intra prediction blocks can represent image blocks as various prediction mode values that indicate relationships rather than actual values. Additionally, inter prediction blocks can represent image blocks as motion vector values rather than actual values. In either case, the prediction block may not exactly represent the image block in some cases; any differences are stored in the residual block. A transform may be applied to the residual block to further compress the file.

In step 107, various filtering techniques may be applied. In HEVC, filters are applied according to an in-loop filtering scheme. Block-based prediction as described above may result in the generation of a blocky image at the decoder. Furthermore, the block-based prediction scheme may encode a block and then reconstruct the encoded block for later use as a reference block. The in-loop filtering scheme iteratively applies noise suppression filters, deblocking filters, adaptive loop filters, and sample adaptive offset (SAO) filters to the block/frame. These filters mitigate such blocking artifacts so that the encoded file can be accurately reconstructed. Furthermore, these filters mitigate artifacts in the reconstructed reference block, such that the artifacts are less likely to generate additional artifacts in subsequent blocks that are encoded based on the reconstructed reference block.

Once the video signal has been segmented, compressed, and filtered, the resulting data is encoded in a bitstream at step 109. The bitstream includes the data described above, as well as any signaling data desired to support proper video signal reconstruction at the decoder. For example, such data may include segmentation data, prediction data, residual blocks, and various flags that provide coding instructions to the decoder. The bitstream may be stored in memory for transmission to the decoder upon request. The bitstream may also be broadcast and/or multicast to multiple decoders. Creation of the bitstream is an iterative process. Thus, steps 101, 103, 105, 107, and 109 may occur continuously and/or simultaneously across many frames and blocks. The order shown in FIG. 1 is presented for clarity and ease of explanation and is not intended to limit the video coding process to any particular order.

In step 111, the decoder receives the bitstream and begins the decoding process. In particular, the decoder employs an entropy decoding scheme to convert the bitstream into corresponding syntax and video data. In step 111, the decoder employs syntax data from the bitstream to determine a partition for the frame. The partition should match the result of the block partition in step 103. The entropy coding/decoding as employed in step 111 is now described. The encoder makes many choices during the compression process, such as selecting a block partitioning scheme from several possible options based on the spatial positioning of values in the input image. Signaling the exact choice may employ a number of bins. A bin, as used herein, is a binary value (e.g., a bit value that may vary depending on the context) that is treated as a variable. Entropy coding allows the encoder to discard any options that are not clearly viable for a particular case, leaving a set of acceptable options. Each acceptable option is then assigned a codeword. The length of the codeword is based on the number of allowable options (e.g., one bin for 2 options, two bins for 3-4 options, etc.). The encoder then encodes a codeword for the selected option. This scheme reduces the size of the codeword because the codeword is as large as desired to uniquely indicate a choice from a small subset of allowable options, as opposed to uniquely indicating a choice from a potentially large set of all possible options. The decoder then decodes the choices by determining the set of allowable options in a similar manner as the encoder. By determining the set of allowable options, the decoder can read the codeword and determine the selection made by the encoder.

In step 113, the decoder performs block decoding. In particular, the decoder employs an inverse transform to generate a residual block. Then, the decoder employs the residual block and a corresponding prediction block to reconstruct an image block according to the partition. The prediction block may include both intra-prediction blocks and inter-prediction blocks, as generated in the encoder in step 105. The reconstructed image block is then placed in a frame of the reconstructed video signal according to the partition data determined in step 111. The syntax for step 113 may also be signaled in the bitstream via entropy coding, as described above.

In step 115, filtering is performed on the frames of the reconstructed video signal in a manner similar to step 107 in the encoder. For example, noise suppression filters, deblocking filters, adaptive loop filters, and SAO filters may be applied to the frames to remove blocking artifacts. Once the frames have been filtered, the video signal may be output to a display in step 117 for viewing by an end user.

FIG. 2 is a schematic diagram of an exemplary coding and decoding (codec) system 200 for video coding. In particular, the codec system 200 provides functionality to support the implementation of the operational method 100. The codec system 200 is generalized to show components employed in both an encoder and a decoder. The codec system 200 receives and segments a video signal, as described with respect to steps 101 and 103 in the operational method 100, which results in a segmented video signal 201. The codec system 200 then compresses the segmented video signal 201 into a coded bitstream when acting as an encoder, as described with respect to steps 105, 107, and 109 in the method 100. When acting as a decoder, the codec system 200 generates an output video signal from the bitstream, as described with respect to steps 111, 113, 115, and 117 in the operational method 100. The codec system 200 includes a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, an intra-picture prediction component 217, a motion compensation component 219, a motion estimation component 221, a scaling and inverse transform component 229, a filter control analysis component 227, an in-loop filter component 225, a decoded picture buffer component 223, and a header formatting and context adaptive binary arithmetic coding (CABAC) component 231. Such components are coupled as shown. In FIG. 2, the black lines indicate the movement of data to be encoded/decoded, and the dashed lines indicate the movement of control data that controls the operation of the other components. All of the components of the codec system 200 may be present in an encoder. A decoder may include a subset of the components of the codec system 200. For example, the decoder may include an intra-picture prediction component 217, a motion compensation component 219, a scaling and inverse transform component 229, an in-loop filter component 225, and a decoded picture buffer component 223. These components are described below.

The partitioned video signal 201 is a captured video sequence that has been partitioned into blocks of pixels by a coding tree. The coding tree employs various partitioning modes to subdivide the blocks of pixels into smaller blocks of pixels. These blocks may then be further subdivided into smaller blocks. The blocks may be referred to as nodes on the coding tree. Larger parent nodes are partitioned into smaller child nodes. The number of times a node is subdivided is referred to as the depth of the node/coding tree. The partitioned blocks may possibly be contained within a coding unit (CU). For example, a CU may be a sub-part of a CTU that includes a luma block, a red differential chroma (Cr) block, and a blue differential chroma (Cb) block along with corresponding syntax instructions for the CU. Partitioning modes may include a binary tree (BT), a triple tree (TT), and a quad tree (QT), which are employed to partition a node into two, three, or four child nodes, respectively, of varying shapes depending on the partitioning mode employed. The segmented video signal 201 is forwarded to a general coder control component 211, a transform scaling and quantization component 213, an intra-picture estimation component 215, a filter control analysis component 227, and a motion estimation component 221 for compression.

The generic coder control component 211 is configured to make decisions related to the coding of images of a video sequence into a bitstream according to application constraints. For example, the generic coder control component 211 manages the optimization of bitrate/bitstream size versus reconstruction quality. Such decisions may be made based on storage space/bandwidth availability and image resolution requirements. The generic coder control component 211 also manages buffer utilization in light of transmission rate to mitigate buffer underrun and overrun issues. To manage these issues, the generic coder control component 211 manages segmentation, prediction, and filtering by other components. For example, the generic coder control component 211 may dynamically increase compression complexity for higher resolution and higher bandwidth usage, or may decrease compression complexity for lower resolution and bandwidth usage. Thus, the generic coder control component 211 controls other components of the codec system 200 to balance video signal reconstruction quality with bitrate issues. The generic coder control component 211 creates control data that controls the operation of other components. The control data is also forwarded to the header formatting and CABAC component 231 to be encoded in the bitstream to signal parameters for decoding at the decoder.

The partitioned video signal 201 is also sent to a motion estimation component 221 and a motion compensation component 219 for inter prediction. A frame or slice of the partitioned video signal 201 may be divided into multiple video blocks. The motion estimation component 221 and the motion compensation component 219 perform inter predictive coding of the received video block relative to one or more blocks in one or more reference frames to perform temporal prediction. The codec system 200 may perform multiple coding passes, for example, to select an appropriate coding mode for each block of video data.

The motion estimation component 221 and the motion compensation component 219 may be highly integrated, but are illustrated separately for conceptual purposes. Motion estimation performed by the motion estimation component 221 is a process of generating motion vectors that estimate motion for a video block. A motion vector may indicate, for example, the displacement of a coded object relative to a predictive block. A predictive block is a block that is found to closely match a block to be coded in terms of pixel difference. A predictive block may be referred to as a reference block. Such pixel difference may be determined by sum of absolute difference (SAD), sum of square difference (SSD), or other difference metric. HEVC employs several coded objects, including CTUs, coding tree blocks (CTBs), and CUs. For example, a CTU may be divided into CTBs, which may then be divided into CBs for inclusion in a CU. A CU may be coded as a prediction unit (PU), which includes predictive data, and/or a transform unit (TU), which includes transformed residual data for the CU. The motion estimation component 221 generates the motion vectors, PUs, and TUs by using the rate-distortion analysis as part of the rate-distortion optimization process. For example, the motion estimation component 221 may determine multiple reference blocks, multiple motion vectors, etc. for a current block/frame, and may select the reference block, motion vector, etc. with the best rate-distortion characteristics. The best rate-distortion characteristics balance both the quality of the video reconstruction (e.g., the amount of data lost due to compression) and the coding efficiency (e.g., the size of the final encoding).

In some examples, the codec system 200 may calculate values for sub-integer pixel positions of the reference picture stored in the decoded picture buffer component 223. For example, the video codec system 200 may interpolate values for quarter-pixel positions, eighth-pixel positions, or other fractional pixel positions of the reference picture. Thus, the motion estimation component 221 may perform motion searches for full pixel positions and fractional pixel positions and output motion vectors with fractional pixel accuracy. The motion estimation component 221 calculates motion vectors for the PUs of the video blocks in the inter-coded slices by comparing the positions of the PUs with the positions of the predictive blocks of the reference pictures. The motion estimation component 221 outputs the calculated motion vectors as motion data to the header formatting and CABAC component 231 for encoding and to the motion compensation component 219.

The motion compensation performed by the motion compensation component 219 may involve fetching or generating a predictive block based on a motion vector determined by the motion estimation component 221. Again, the motion estimation component 221 and the motion compensation component 219 may be functionally integrated in some examples. Upon receiving the motion vector for the PU of the current video block, the motion compensation component 219 may locate the predictive block to which the motion vector points. A residual video block is then formed by subtracting pixel values of the predictive block from pixel values of the current video block being coded to form pixel difference values. In general, the motion estimation component 221 performs motion estimation on the luma component, and the motion compensation component 219 uses the motion vector calculated based on the luma component for both the chroma and luma components. The predictive block and the residual block are forwarded to the transform scaling and quantization component 213.

The partitioned video signal 201 is also sent to an intra picture estimation component 215 and an intra picture prediction component 217. As with the motion estimation component 221 and the motion compensation component 219, the intra picture estimation component 215 and the intra picture prediction component 217 may be highly integrated, but are illustrated separately for conceptual purposes. The intra picture estimation component 215 and the intra picture prediction component 217 intra predict the current block relative to blocks in the current frame, as an alternative to the inter prediction performed by the motion estimation component 221 and the motion compensation component 219 between frames as described above. In particular, the intra picture estimation component 215 determines the intra prediction mode to be used to encode the current block. In some examples, the intra picture estimation component 215 selects an appropriate intra prediction mode for encoding the current block from the tested multiple intra prediction modes. The selected intra prediction mode is then forwarded to the header formatting and CABAC component 231 for encoding.

For example, the intra picture estimation component 215 may use a rate-distortion analysis to calculate rate-distortion values for various intra prediction modes tested and select an intra prediction mode with the best rate-distortion characteristics among the tested modes. The rate-distortion analysis generally determines the amount of distortion (or error) between a coding block and an original uncoded block that was coded to produce the coding block, as well as the bit rate (e.g., number of bits) used to produce the coding block. The intra picture estimation component 215 may calculate a ratio from the distortion and rate for various coding blocks to determine which intra prediction mode exhibits the best rate-distortion value for the block. In addition, the intra picture estimation component 215 may be configured to code the depth blocks of the depth map using a depth modeling mode (DMM) based on a rate-distortion optimization (RDO).

The intra-picture prediction component 217 may generate a residual block from the prediction block based on a selected intra-prediction mode determined by the intra-picture estimation component 215 when implemented on an encoder, or may read the residual block from the bitstream when implemented on a decoder. The residual block contains the difference in values between the prediction block and the original block, represented as a matrix. The residual block is then forwarded to the transform scaling and quantization component 213. The intra-picture estimation component 215 and the intra-picture prediction component 217 may operate on both the luma and chroma components.

The transform scaling and quantization component 213 is configured to further compress the residual block. The transform scaling and quantization component 213 applies a transform, such as a discrete cosine transform (DCT), a discrete sine transform (DST), or a conceptually similar transform, to the residual block to produce a video block comprising residual transform coefficient values. A wavelet transform, an integer transform, a subband transform, or other types of transforms may also be used. The transform may convert the residual information from a pixel value domain to a transform domain, such as a frequency domain. The transform scaling and quantization component 213 is also configured to scale the transformed residual information, for example, based on frequency. Such scaling involves applying a scale factor to the residual information, such that different frequency information is quantized with different granularity, which may affect the final visual quality of the reconstructed video. The transform scaling and quantization component 213 is also configured to quantize the transform coefficients to further reduce the bit rate. The quantization process may reduce the bit depth associated with some or all of the coefficients. The degree of quantization may be modified by adjusting a quantization parameter. In some examples, the transform scaling and quantization component 213 may then perform a scan of a matrix containing the quantized transform coefficients. The quantized transform coefficients are forwarded to the header formatting and CABAC component 231 for encoding in the bitstream.

The scaling and inverse transform component 229 applies the inverse operations of the transform scaling and quantization component 213 to support motion estimation. The scaling and inverse transform component 229 applies inverse scaling, transformation, and/or quantization to reconstruct a residual block in the pixel domain for later use as a reference block that may become a prediction block for another current block, for example. The motion estimation component 221 and/or the motion compensation component 219 may compute a reference block by adding the residual block back to the corresponding prediction block for use in motion estimation of a later block/frame. A filter is applied to the reconstructed reference block to mitigate artifacts created during the scaling, quantization, and transformation. Such artifacts may potentially cause inaccurate predictions (and may create additional artifacts) when subsequent blocks are predicted.

The filter control analysis component 227 and the in-loop filter component 225 apply filters to the residual block and/or the reconstructed image block. For example, a transformed residual block from the scaling and inverse transform component 229 may be combined with a corresponding prediction block from the intra-picture prediction component 217 and/or the motion compensation component 219 to reconstruct the original image block. A filter may then be applied to the reconstructed image block. In some examples, a filter may be applied to the residual block instead. As with the other components in FIG. 2, the filter control analysis component 227 and the in-loop filter component 225 may be highly integrated and implemented together, but are shown separately for conceptual purposes. The filters applied to the reconstructed reference block are applied to a particular spatial region and include multiple parameters to adjust how such filters are applied. The filter control analysis component 227 analyzes the reconstructed reference block and sets the corresponding parameters to determine where such filters should be applied. Such data is forwarded to the header formatting and CABAC component 231 as filter control data for encoding. The in-loop filter component 225 applies such filters based on the filter control data. The filters may include deblocking filters, noise suppression filters, SAO filters, and adaptive loop filters. Such filters may be applied in the spatial/pixel domain (e.g., to reconstructed pixel blocks) or in the frequency domain, depending on the example.

When operating as an encoder, the filtered reconstructed image blocks, residual blocks, and/or predictive blocks are stored in the decoded picture buffer component 223 for later use in motion estimation as described above. When operating as a decoder, the decoded picture buffer component 223 stores the reconstructed filtered blocks and forwards them towards the display as part of the output video signal. The decoded picture buffer component 223 may be any memory device capable of storing predictive blocks, residual blocks, and/or reconstructed image blocks.

The header formatting and CABAC component 231 receives data from various components of the codec system 200 and encodes such data into a coded bitstream for transmission towards the decoder. In particular, the header formatting and CABAC component 231 generates various headers for encoding control data such as general control data and filter control data. Furthermore, prediction data including intra prediction and motion data as well as residual data in the form of quantized transform coefficient data are all encoded in the bitstream. The final bitstream contains all the information desired by the decoder to reconstruct the partitioned original video signal 201. Such information may also include an intra prediction mode index table (also called a codeword mapping table), definitions of coding contexts for various blocks, an indication of the most probable intra prediction mode, an indication of partition information, etc. Such data may be encoded by employing entropy coding. For example, the information may be encoded by employing context adaptive variable length coding (CAVLC), CABAC, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioning entropy (PIPE) coding, or another entropy coding technique. Following entropy coding, the coded bitstream may be transmitted to another device (e.g., a video decoder) or archived for later transmission or retrieval.

FIG. 3 is a block diagram illustrating an example video encoder 300. Video encoder 300 may be employed to perform the encoding functions of codec system 200 and/or perform steps 101, 103, 105, 107, and/or 109 of method of operation 100. Encoder 300 segments an input video signal to obtain a segmented video signal 301 that is substantially similar to segmented video signal 201. Segmented video signal 301 is then compressed and encoded into a bitstream by components of encoder 300.

In particular, the partitioned video signal 301 is forwarded to an intra picture prediction component 317 for intra prediction. The intra picture prediction component 317 may be substantially similar to the intra picture estimation component 215 and the intra picture prediction component 217. The partitioned video signal 301 is also forwarded to a motion compensation component 321 for inter prediction based on a reference block in the decoded picture buffer component 323. The motion compensation component 321 may be substantially similar to the motion estimation component 221 and the motion compensation component 219. The prediction block and the residual block from the intra picture prediction component 317 and the motion compensation component 321 are forwarded to a transform and quantization component 313 for transform and quantization of the residual block. The transform and quantization component 313 may be substantially similar to the transform scaling and quantization component 213. The transformed and quantized residual block and the corresponding prediction block (together with associated control data) are forwarded to an entropy coding component 331 for coding into a bitstream. The entropy coding component 331 may be substantially similar to the header formatting and CABAC component 231.

The transformed and quantized residual block and/or the corresponding prediction block are also forwarded from the transform and quantization component 313 to the inverse transform and quantization component 329 for reconstruction into a reference block for use by the motion compensation component 321. The inverse transform and quantization component 329 may be substantially similar to the scaling and inverse transform component 229. An in-loop filter in the in-loop filter component 325 is also applied to the residual block and/or the reconstructed reference block, depending on the example. The in-loop filter component 325 may be substantially similar to the filter control analysis component 227 and the in-loop filter component 225. The in-loop filter component 325 may include multiple filters as described with respect to the in-loop filter component 225. The filtered block is then stored in the decoded picture buffer component 323 for use as a reference block by the motion compensation component 321. The decoded picture buffer component 323 may be substantially similar to the decoded picture buffer component 223.

FIG. 4 is a block diagram illustrating an example video decoder 400. The video decoder 400 may be employed to perform the decoding functions of the codec system 200 and/or to perform steps 111, 113, 115, and/or 117 of the method of operation 100. The decoder 400 receives a bitstream, e.g., from the encoder 300, and generates a reconstructed output video signal based on the bitstream for display to an end user.

The bitstream is received by the entropy decoding component 433. The entropy decoding component 433 is configured to perform an entropy decoding scheme, such as CAVLC, CABAC, SBAC, PIPE coding, or other entropy coding techniques. For example, the entropy decoding component 433 may employ header information to provide a context for interpreting additional data encoded as codewords in the bitstream. The decoded information includes any desired information for decoding a video signal, such as general control data, filter control data, partition information, motion data, prediction data, and quantized transform coefficients from the residual block. The quantized transform coefficients are forwarded to the inverse transform and quantization component 429 for reconstruction into the residual block. The inverse transform and quantization component 429 may be similar to the inverse transform and quantization component 329.

The reconstructed residual block and/or predictive block are forwarded to the intra picture prediction component 417 for reconstruction into an image block based on an intra prediction operation. The intra picture prediction component 417 may be similar to the intra picture estimation component 215 and the intra picture prediction component 217. In particular, the intra picture prediction component 417 employs a prediction mode to identify the location of a reference block in a frame and applies a residual block to the result to reconstruct an intra predicted image block. The reconstructed intra predicted image block and/or the residual block and the corresponding inter prediction data are forwarded via the in-loop filter component 425 to the decoded picture buffer component 423, which may be substantially similar to the decoded picture buffer component 223 and the in-loop filter component 225, respectively. The in-loop filter component 425 filters the reconstructed image block, the residual block, and/or the predictive block, and such information is stored in the decoded picture buffer component 423. The reconstructed image block from the decoded picture buffer component 423 is forwarded to the motion compensation component 421 for inter prediction. The motion compensation component 421 may be substantially similar to the motion estimation component 221 and/or the motion compensation component 219. In particular, the motion compensation component 421 employs a motion vector from a reference block to generate a prediction block and applies a residual block to the result to reconstruct an image block. The resulting reconstructed block may also be forwarded to the decoded picture buffer component 423 via the in-loop filter component 425. The decoded picture buffer component 423 continues to store additional reconstructed image blocks, which may be reconstructed into frames via the partition information. Such frames may also be placed into sequences. The sequences are output as a reconstructed output video signal towards a display.

5 is a schematic diagram illustrating an exemplary bitstream 500 including an encoded video sequence. For example, the bitstream 500 may be generated by the codec system 200 and/or the encoder 300 for decoding by the codec system 200 and/or the decoder 400. As another example, the bitstream 500 may be generated by the encoder in step 109 of the method 100 for use by the decoder in step 111.

The bitstream 500 includes a sequence parameter set (SPS) 510, multiple picture parameter sets (PPS) 512, a tile group header 514, and image data 520. The SPS 510 includes sequence data common to all pictures in a video sequence included in the bitstream 500. Such data may include picture size determination, bit depth, coding tool parameters, bit rate constraints, etc. The PPS 512 includes parameters specific to one or more corresponding pictures. Thus, each picture in the video sequence may reference one PPS 512. The PPS 512 may indicate coding tools, quantization parameters, offsets, picture-specific coding tool parameters (e.g., filter control), etc. available for tiles in the corresponding picture. The tile group header 514 includes parameters specific to each tile group in the picture. Thus, there may be one tile group header 514 for each tile group in the video sequence. The tile group header 514 may include tile group information, picture order count (POC), reference picture list, prediction weights, tile entry points, deblocking parameters, etc. Note that some systems refer to the tile group header 514 as a slice header and use such information to support slices rather than tile groups.

The image data 520 includes video data coded according to inter-prediction and/or intra-prediction, as well as corresponding transformed and quantized residual data. Such image data 520 is sorted according to a partition used to partition the image before encoding. For example, an image in the image data 520 is divided into tiles 523. The tiles 523 are further divided into coding tree units (CTUs). The CTUs are further divided into coding blocks based on the coding tree. The coding blocks may then be coded/decoded according to a prediction mechanism. An image/picture may include one or more tiles 523.

A tile 523 is a partitioned portion of a picture created by horizontal and vertical boundaries. The tile 523 may be rectangular and/or square. In particular, the tile 523 includes four sides connected at right angles. The four sides include two pairs of parallel sides. Furthermore, the sides in a parallel side pair are equal in length. Thus, the tile 523 may be any rectangular shape, where a square is a special case of a rectangle where all four sides are of equal length. A picture may be arranged in rows and columns of tiles 523. A tile row is a set of tiles 523 arranged horizontally adjacent to create a continuous line from the left boundary to the right boundary of the picture (or vice versa). A tile column is a set of tiles 523 arranged vertically adjacent to create a continuous line from the top boundary to the bottom boundary of the picture (or vice versa). The tiles 523 may or may not allow prediction based on other tiles 523, depending on the example. Each tile 523 may have a unique tile index within a picture. The tile index is a procedurally selected numeric identifier that may be used to distinguish one tile 523 from another tile 523. For example, the tile index may increase numerically in a raster scan order. The raster scan order is left-to-right and top-to-bottom. Note that in some examples, tiles 523 may also be assigned a tile identifier (ID). The tile ID is an assigned identifier that may be used to distinguish one tile 523 from another tile 523. In some examples, the calculation may employ the tile ID rather than the tile index. Furthermore, in some examples, the tile ID may be assigned to have the same value as the tile index. The tile index and/or tile ID may be signaled to indicate the tile group that includes the tile 523. For example, the tile index and/or tile ID may be employed to map picture data associated with the tile 523 to an appropriate location for display. A tile group is a related set of tiles 523 that may be extracted and coded separately, for example, to support display of regions of interest and/or to support parallel processing. Tiles 523 within a tile group may be coded without reference to tiles 523 outside the tile group. Each tile 523 may be assigned to a corresponding tile group, and thus a picture may contain multiple tile groups.

6A-6E illustrate an example mechanism 600 for creating an extractor track 610 for combining multiple resolution sub-pictures from different bitstreams into a single picture for use in virtual reality (VR) applications. The mechanism 600 may be employed to support example use cases of the method 100. For example, the mechanism 600 may be employed to generate a bitstream 500 for transmission from the codec system 200 and/or the encoder 300 to the codec system 200 and/or the decoder 400. As a particular example, the mechanism 600 may be employed for use with VR, OMAF, 360-degree video, etc.

In VR, only a portion of the video is displayed to the user. For example, the VR video may be shot to include a sphere surrounding the user. The user may employ a head mounted display (HMD) to view the VR video. The user may point the HMD towards a region of interest. The region of interest is displayed to the user, and other video data is discarded. In this way, at any given moment, the user sees only the portion of the VR video that the user selected. This approach mimics the user's perception, and thus allows the user to experience the virtual environment in a way that mimics the real environment. One of the problems with this approach is that although the entire VR video may be transmitted to the user, only the current viewport of the video is actually used, and the rest is discarded. To increase signaling efficiency for streaming applications, the user's current viewport may be transmitted at a higher first resolution, and other viewports may be transmitted at a lower second resolution. In this way, viewports that may be discarded occupy less bandwidth than viewports that may be seen by the user. When the user selects a new viewport, the lower resolution content may be shown until the decoder may request that a different current viewport be transmitted at the higher first resolution. To support this functionality, a mechanism 600 may be employed to create an extractor track 610 as shown in FIG. 6E. The extractor track 610 is a track of image data that encapsulates pictures at multiple resolutions for use as described above.

The mechanism 600 encodes the same video content in a first resolution 611 and a second resolution 612, as shown in FIG. 6A and FIG. 6B, respectively. As a specific example, the first resolution 611 may be 5120×2560 luma samples, and the second resolution 612 may be 2560×1280 luma samples. Pictures of the video may be partitioned into tiles 601 in the first resolution 611 and tiles 603 in the second resolution 612, respectively. In the illustrated example, the tiles 601 and 603 are each partitioned into a 4×2 grid. Furthermore, an MCTS may be coded for the location of each tile 601 and 603. The pictures in the first resolution 611 and the second resolution 612 each result in an MCTS sequence that represents the video over time at the corresponding resolution. Each coded MCTS sequence is stored as a subpicture track or a tile track. The mechanism 600 can then use the pictures to create segments to support viewport-adaptive MCTS selection. For example, each range of viewing orientations that causes a different selection of high-resolution and low-resolution MCTSs is considered. In the illustrated example, four tiles 601 containing MCTSs in a first resolution 611 and four tiles 603 containing MCTSs in a second resolution 612 are obtained.

The mechanism 600 can then create an extractor track 610 for each possible viewport-adaptive MCTS selection. FIGS. 6C and 6D show an exemplary viewport-adaptive MCTS selection. In particular, a set of selected tiles 605 and 607 are selected in a first resolution 611 and a second resolution 612, respectively. The selected tiles 605 and 607 are shown in shades of gray. In the illustrated example, the selected tile 605 is a tile 601 in the first resolution 611 that will be shown to the user, and the selected tile 607 is a tile 603 in the second resolution 612 that may be discarded but is retained to support display if the user selects a new viewport. The selected tiles 605 and 607 are then combined into a single picture that includes image data in both the first resolution 611 and the second resolution 612. Such pictures are combined to create the extractor track 610. FIG. 6E shows a single picture from a corresponding extractor track 610 for illustrative purposes. As shown, the picture in the extractor track 610 includes selected tiles 605 and 607 from a first resolution 611 and a second resolution 612. As mentioned above, FIGS. 6C-6E show a single viewport adaptive MCTS selection. To allow user selection of any viewport, an extractor track 610 should be created for each possible combination of selected tiles 605 and 607.

In the illustrated example, each selection of tiles 603 encapsulating content from a bitstream of a second resolution 612 includes two slices. A RegionWisePackingBox may be included in the extractor track 610 to create a mapping between the packed picture and a projected picture in equirectangular projection (ERP) format. In the presented example, the resolved bitstream from the extractor track 610 has a resolution of 3200x2560. Hence, a 4000 sample (4K) capable decoder may decode content in which the viewport is extracted from a coded bitstream having a 5000 sample 5K (5120x2560) resolution.

As shown, the extractor track 610 includes two rows of high resolution tiles 601 and four rows of low resolution tiles 603. Thus, the extractor track 610 includes two slices of high resolution content and four slices of low resolution content. Uniform tiling may not support such use cases. Uniform tiling is defined by a set of tile columns and a set of tile rows. The tile columns extend from the top of the picture to the bottom of the picture. Similarly, the tile rows extend from the left of the picture to the right of the picture. Although such a structure may be easily defined, this structure cannot effectively support advanced use cases, such as the use case described by the mechanism 600. In the illustrated example, different numbers of rows are employed in different sections of the extractor track 610. If uniform tiling were employed, the tiles on the right side of the extractor track 610 would have to be rewritten to accommodate two slices each. This approach is inefficient and computationally complex.

The present disclosure includes a flexible tiling scheme, as described below, that does not require tiles to be rewritten to include a different number of slices. The flexible tiling scheme allows a tile 601 to include content in a first resolution 611. The flexible tiling scheme also allows the tile 601 to be partitioned into smaller tiles that can each be directly mapped to tiles 603 in a second resolution 612. This direct mapping is more efficient because such an approach does not require tiles to be rewritten/readdressed when different resolutions are combined as described above.

FIG. 7 illustrates an example video conferencing application 700 that stitches together pictures of multiple resolutions from different bitstreams into a single picture for display. The application 700 may be employed to support an example use case of the method 100. For example, the application 700 may be employed in the codec system 200 and/or the decoder 400 to display video content from the bitstream 500 from the codec system 200 and/or the encoder 300. The video conferencing application 700 displays a video sequence to a user. The video sequence includes pictures displaying a speaking participant 701 and other participants 703. The speaking participant 701 is displayed in a higher/larger first resolution and the other participants 703 are displayed in a lower/smaller second resolution. To code such a picture, the picture should include a portion with a single row and a portion with three rows. To support such a scenario with uniform tiling, the picture is partitioned into a left tile and a right tile. The right tile is then rewritten/readdressed to contain three rows. Such readdressing incurs both compression and performance penalties. The flexible tiling scheme described below allows a single tile to be partitioned into smaller tiles and mapped to tiles in the sub-picture bitstreams associated with other participants 703. In this way, the speaking participant 701 may be mapped directly to a first level tile, and the other participants 703 may be mapped to second level tiles split off from the first tile without such rewriting/readdressing.

8A-8D are schematic diagrams illustrating an example flexible video tiling scheme 800 capable of supporting multiple tiles with different resolutions in the same picture. The flexible video tiling scheme 800 may be employed to support more efficient coding mechanisms 600 and applications 700. Thus, the flexible video tiling scheme 800 may be employed as part of the method 100. Furthermore, the flexible video tiling scheme 800 may be employed by the codec system 200, the encoder 300, and/or the decoder 400. The result of the flexible video tiling scheme 800 may be stored in the bitstream 500 for transmission between the encoder and the decoder.

As shown in FIG. 8A, a picture (e.g., a frame, image, etc.) may be partitioned into first level tiles 801, also referred to as level 1 tiles. As shown in FIG. 8B, the first level tiles 801 may be selectively partitioned to create second level tiles 803, also referred to as level 2 tiles. The first level tiles 801 and the second level tiles 803 may then be employed to create a picture having sub-pictures coded at multiple resolutions. The first level tiles 801 are tiles generated by completely partitioning the picture into a set of columns and a set of rows. The second level tiles 803 are tiles generated by partitioning the first level tiles 801.

As described above, in various scenarios, for example, in VR and/or teleconferencing, a video may be coded at multiple resolutions. The video may also be coded by employing slices at each resolution. The lower resolution slices are smaller than the higher resolution slices. To create a picture with multiple resolutions, the picture may be partitioned into first level tiles 801. Slices from the highest resolution may be included directly in the first level tiles 801. The first level tiles 801 may further be partitioned into second level tiles 803 that are smaller than the first level tiles 801. Thus, the smaller second level tiles 803 may directly accept slices with lower resolutions. In this way, slices from each resolution may be compressed into a single picture, for example, via a tile index relationship, without requiring tiles with different resolutions to be dynamically readdressed in order to use a coherent addressing scheme. The first level tiles 801 and the second level tiles 803 may be implemented as MCTS and may therefore accept motion constrained image data at different resolutions.

This disclosure includes many aspects. As a particular example, a first level tile 801 is divided into second level tiles 803. The second level tiles 803 may then each be constrained to contain a single rectangular slice of picture data (e.g., at a lower resolution). A rectangular slice is a slice that is constrained to maintain a rectangular shape and is therefore coded based on horizontal and vertical picture boundaries. Thus, a rectangular slice is not coded based on a raster scan group (which may contain CTUs in lines from left to right and lines from top to bottom and not maintain a rectangular shape). A slice is a spatially distinct region of a picture/frame that is coded separately from any other region in the same frame/picture. In another example, a first level tile 801 may be divided into two or more complete second level tiles 803. In such a case, the first level tile 801 may not contain a partial second level tile 803. In another example, the configuration of the first level tiles 801 and second level tiles 803 may be signaled in a parameter set in the bitstream, such as a PPS associated with the picture partitioned to create the tile. In one example, a partition indication, such as a flag, may be coded in the parameter set for each first level tile 801. The indication indicates which first level tiles 801 are further partitioned into second level tiles 803. In another example, the configuration of the second level tiles 803 may be signaled as a number of second level tile columns and a number of second level tile rows.

In another example, the first level tiles 801 and the second level tiles 803 may be assigned to tile groups. Such tile groups may be constrained such that all tiles in the corresponding tile group are constrained to cover a rectangular region of the picture (e.g., as opposed to raster scan). For example, some systems may add tiles to the tile group in raster scan order. This involves adding an initial tile in the current row, proceeding to add each tile in the row until the left picture boundary of the current row is reached, proceeding to the right boundary of the next row, and adding each tile in the next row until the final tile is reached, etc. This approach may result in non-rectangular shapes that extend beyond the picture. Such shapes may not be useful for creating pictures with multiple resolutions as described herein. Instead, the present example may constrain the tile groups such that any first level tile 801 and/or second level tile 803 may be added to the tile group (e.g., in any order), but the resulting tile group must be rectangular or square (e.g., including four sides connected at right angles). This constraint may ensure that second level tiles 803 that are separate from a single first level tile 801 are not placed in different tile groups.

In another example, data explicitly indicating the number of second level tile columns and the number of second level tile rows may be omitted from the bitstream when the first level tile width is less than twice the minimum width threshold and the first level tile height is less than twice the minimum height threshold, because first level tiles 801 that satisfy such conditions may not be split into two or more columns or one row, respectively, and therefore such information may be inferred by the decoder. In another example, partition indications indicating which first level tiles 801 are partitioned into second level tiles 803 may be omitted from the bitstream for some first level tiles 801. For example, such data may be omitted when a first level tile 801 has a first level tile width less than the minimum width threshold and a first level tile height less than the minimum height threshold. This is because a first-level tile 801 that satisfies such a condition would be too small to be divided into second-level tiles 803, and therefore such information could be inferred by the decoder.

In another aspect, the flexible video tiling scheme 800 may employ a tile group 805 as shown in FIG. 8C. The tile group 805 is shown as being separated by a bold line. The picture is partitioned into first level tiles 801. A subset of the first level tiles 801 is partitioned into second level tiles 803. The first level tiles 801 and the second level tiles 803 may then be assigned to a tile group 805. A tile group 805 is a related set of tiles that may be extracted and coded separately, for example, to support the display of a region of interest and/or to support parallel processing. The tile group 805 may be generated as a raster scan tile group or a rectangular tile group. A raster scan tile group includes tiles that are assigned in a raster scan order, proceeding from left to right and top to bottom. For example, the raster scan order proceeds from the first tile towards the right picture boundary, then from the left picture boundary in the next row towards the right picture boundary, etc., until the last tile is reached. In contrast, a rectangular tile group contains a rectangular subportion of a picture. Tile group 805 is a rectangular tile group, although in some examples raster scan tile groups may also be used.

In some aspects, the first level tiles 801 and the second level tiles 803 may be assigned to tile groups 805 such that each tile group 805 includes one or more consecutive sequences of several first level tiles 801 or second level tiles 803. As used herein, a sequence of second level tiles 803 is a group of second level tiles 803 split from a single first level tile 801 and therefore has consecutive tile indices. This approach allows all second level tiles 803 created from a single first level tile 801 to be assigned to the same tile group 805. In the example shown in FIG. 8C, one tile group 805 includes four first level tiles 801 and four other tile groups 805, each including second level tiles 803 split from a single first level tile 801. However, many combinations of tiles and tile groups 805 may be used depending on the type of video to be encoded and decoded.

In another aspect, the flexible video tiling scheme 800 may employ a scan order 807 as shown in FIG. 8D. The scan order 807 is the order in which tiles are coded at the encoder and/or decoded at the decoder or hypothetical reference decoder (at the encoder), depending on the example. In the illustrated scan order 807, the first level tile 801 is coded in raster scan order. When one of the second level tiles 803 is encountered, the raster scan order coding of the first level tile 801 is interrupted. All successive second level tiles 803 are then coded in raster scan order before continuing the raster scan order coding of the first level tile 801. This process continues until all tiles are coded. In the illustrated example, the second level tile 803 labeled Tile 1 is encountered first. Thus, the raster scan order coding of the first level tile 801 is interrupted and the successive second level tiles 803 labeled tiles 1 and 2 are coded. Once all the successive second level tiles 803 have been coded, the raster scan order coding of the first level tiles 801 continues. Thus, the first level tiles 801 labeled tiles 3 and 4 are then coded. The second level tile 803 labeled 5 is encountered. The successive second level tiles 803 are then coded. Thus, the tiles labeled 5 to 8 are coded. Once all the successive second level tiles 803 have been coded, the raster scan order coding of the first level tiles 801 continues again. This results in coding the first level tiles 801 labeled tiles 9 and 10. The second level tile 803 labeled 11 is encountered. Consecutive second level tiles 803 are then coded. Thus, tiles labeled 11 and 12 are coded. In more formal terms, the first level tiles 801 are coded in raster scan order relative to picture and/or tile group boundaries. Furthermore, all second level tiles 803 partitioned from the current first level tile 801 (e.g., a sequence of consecutive second level tiles 803) are coded before coding any second level tiles 803 partitioned from a subsequent second level tile 803. In addition, all second level tiles 803 partitioned from the current first level tile 801 are coded in raster scan order relative to the boundaries of the current first level tile 801.

As described above, the flexible video tiling scheme 800 supports merging sub-pictures from different bitstreams into a picture containing multiple resolutions. The following describes various embodiments that support such functionality. Generally, this disclosure describes methods for signaling and coding tiles in video coding that partition pictures in a more flexible manner than the tiling schemes in HEVC. More specifically, this disclosure describes several tiling schemes in which tile columns may not extend uniformly from the top to the bottom of the coded picture, and similarly tile rows may not extend uniformly from the left to the right of the coded picture.

For example, based on the HEVC tiling approach, some tiles should be further divided into multiple tile rows to support the functionality described in Figures 6A-6E and 7. Furthermore, depending on how the tiles are arranged, the tiles should be further divided into tile columns. For example, in Figure 7, in some cases participants 2-4 may be arranged below participant 1, which may be supported by dividing the tiles into columns. To satisfy these scenarios, the first level tiles may be divided into tile rows and tile columns of second level tiles as described below.

For example, the tile structure may be relaxed as follows: Tiles in the same picture are not required to be a specific number of tile rows. Furthermore, tiles in the same picture are not required to be a specific number of tile columns. For signaling flexible tiles, the following steps may be used: The first level tile structure may be defined by tile columns and tile rows as defined in HEVC. The tile columns and tile rows may be uniform or non-uniform in size. Each of these tiles may be referred to as a first level tile. A flag may be signaled to specify whether each first level tile is further split into one or more tile columns and one or more tile rows. If a first level tile is further split, the tile columns and tile rows may be either uniform or non-uniform in size. The new tiles resulting from the split of the first level tiles are referred to as second level tiles. The flexible tile structure may be limited to only second level tiles, and thus, in some examples, further splitting of any second level tiles is not allowed. In other examples, further division of the second level tiles may be applied to create subsequent level tiles in a manner similar to the creation of the second level tiles from the first level tiles.

In the above example, the picture may include zero or more first level tiles that are not further divided, and zero or more second level tiles. The first level tiles that are further divided may only be conceptually present and may not be counted in the total number of tiles in the picture. An exemplary scan order is defined as follows: For simplicity, a tile group may be constrained to include either some complete first level tiles, or a complete subset of the first level tiles. The first level tiles may be ordered according to a tile raster scan of the picture. When a first level tile that is further divided is referenced, the set of second level tiles resulting from such division may be referenced collectively. The second level tiles of any current first level tile may be referenced before referencing any second level tiles of a subsequent first level tile that comes after the current first level tile. The second level tiles of the current first level tile are referenced in raster scan order within the current first level tile. Any CTU within the current tile may be referenced in CTU raster scan order within the current tile.

For simplicity, when a first level tile is split into two or more second level tiles, the split may always use uniform sized tile columns and uniform tile rows. In such examples, it may not be necessary to signal a flag specifying whether the level 2 tile columns and level 2 tile rows are uniform. Furthermore, it may not be necessary for syntax elements to specify the tile row height and tile row width. In some examples, the first level tiles may be constrained to always use uniform sizes for the tile columns and tile rows. In such examples, it may not be necessary to signal a flag specifying whether the level 1 tile columns and level 1 rows are uniform. Furthermore, it may not be necessary for syntax elements to specify the tile row height and tile row width. In another example, the first level tiles and the second level tiles may be constrained to always use uniform sized tile columns and tile rows. In such examples, it may not be necessary to signal a flag specifying whether the level 1 and level 2 tile columns and rows are uniform. Additionally, syntax elements may not need to specify tile row height and tile row width.

The derivation of tile locations, sizes, indices, and scan order of flexible tiles defined by this approach is described below. In some examples, when such a flexible tile structure is used, a tile group may include only some complete first level tiles, or only a contiguous sequence of complete second level tiles of one single first level tile. Furthermore, when such a flexible tile structure is used, a tile group may be constrained to include one or more complete first level tiles. In this example, when a tile group includes a second level tile, all second level tiles arising from the division of the same first level tile should be included in the tile group. When such a flexible tile structure is used, it may be further constrained that the tile group includes one or more tiles, and all tiles together belong to a tile group that covers a rectangular area of the picture. In another aspect, when such a flexible tile structure is used, a tile group includes one or more first level tiles, and all tiles together belong to a tile group that covers a rectangular area of the picture.

In one example, signaling of flexible tiles may be as follows: The minimum tile width and minimum tile height are defined values. The first level tile structure may be defined by tile columns and tile rows. The tile columns and tile rows may be uniform or non-uniform in size. Each of these tiles may be referred to as a first level tile. A flag may be signaled to specify whether any of the first level tiles may be further divided. This flag may be absent when the width of each first level tile is less than or equal to twice the minimum tile width and the height of each first level tile is less than or equal to twice the minimum tile height. When absent, the value of the flag is inferred to be equal to 0.

In one example, the following applies for each first level tile: A flag may be signaled to specify whether the first level tile is further divided into one or more tile columns and one or more tile rows. The presence of the flag may be constrained as follows: If the first level tile width is greater than the minimum tile width or if the first level tile height is greater than the minimum tile height, then the flag is present/signaled. Otherwise, the flag is not present and the value of the flag is inferred to be equal to 0 indicating that the first level tile is not further divided.

If the first level tile is further divided, the number of tile columns and the number of tile rows for this division may be further signaled. The tile columns and tile rows may be either uniform or non-uniform in size. The tiles resulting from the division of the first level tiles are called second level tiles. The existence of the number of tile columns and the number of tile rows may be constrained as follows: When the first level tile width is smaller than twice the minimum tile width, the number of tile columns may not be signaled and the number of tile column values may be inferred to be equal to 1. The signaling may employ the _minus1 syntax element, such that the signaled syntax element value may be 0 and the number of tile columns is the value of the syntax element + 1. This approach may further compress the signaling data. When the first level tile height is smaller than twice the minimum tile height, the number of tile rows may not be signaled and the value of the number of tile rows may be inferred to be equal to 0. The signaled syntax element value may be 0 and the number of tile rows may be the syntax element value + 1 to further compress the signaling data. The tiles resulting from the partitioning of the first level tiles may be referred to as second level tiles. The flexible tile structure may be limited to only second level tiles such that further partitioning of any second level tiles is not allowed. In other examples, further partitioning of second level tiles may be applied in a manner similar to partitioning the first level tiles into second level tiles.

In one example, signaling of a flexible tile structure may be as follows: When a picture includes more than one tile, a signal such as a flag may be employed in a parameter set that is directly or indirectly referenced by the corresponding tile group. The flag may specify whether the corresponding tile structure is a uniform tile structure or a non-uniform tile structure (e.g., a flexible tile structure as described herein). The flag may be referred to as uniform_tile_structure_flag. When uniform_tile_structure_flag is equal to 1, HEVC-style uniform tile structure signaling is employed, e.g., by signaling num_tile_columns_minus1 and num_tile_rows_minus1 to indicate a single level of uniform tiles. When uniform_tile_structure_flag is equal to 0, the following information may also be signaled: The number of tiles in a picture may be signaled by the syntax element num_tiles_minus2, which indicates that the number of tiles in a picture (NumTilesInPic) is equal to num_tiles_minus2+2. This may result in bit savings during signaling, since by default pictures may be considered to be tiles. For each tile, except the last one, the addresses of the first coding block (e.g., CTU) and the last coding block of the tile are signaled. The address of the coding block may be the index of the block in the picture (e.g., the index of the CTU in the picture). The syntax elements for such coding blocks may be tile_first_block_address[i] and tile_last_block_address[i]. These syntax elements may be coded as ue(v) or u(v). When the syntax elements are coded as u(v), the number of bits used to represent each of the syntax elements is ceil(log2(maximum number of coding blocks in a picture)). The addresses of the first and last coding blocks of the last tile do not need to be signaled, but instead may be derived based on the picture size, the luma samples, and the collection of all other tiles in the picture.

In one example, rather than signaling the addresses of the first and last coding blocks of the tile, except for the last one, for each tile, the address of the first coding block of the tile, as well as the width and height of the tile may be signaled. In another example, rather than signaling the addresses of the first and last coding blocks of the tile, except for the last one, for each tile, the offset of the top left point of the tile (e.g., the top left of the picture) relative to the picture original, as well as the width and height of the tile may be signaled. In yet another example, rather than signaling the addresses of the first and last coding blocks of the tile, except for the last one, for each tile, the following information may be signaled: The width and height of the tile may be signaled. Also, the location of each tile may not be signaled. Instead, a flag may be signaled to specify whether the tile should be placed immediately to the right or immediately below the previous tile. If the tile can only be to the right or only below the previous tile, this flag may not be present. The top-left offset of the first tile may be set to always be at the origin/top-left of the picture (e.g. x=0 and y=0).

For signaling efficiency, a set of unique tile sizes (e.g., width and height) may be signaled. This list of unique tile sizes may be referenced by index from a loop that contains the signaling of each tile size. In some examples, the tile locations and tile sizes as derived from the signaled tile structure must be constrained for partitioning to ensure that no gaps or overlaps occur between any tiles.

The following constraints may also apply: The tile shape may be required to be rectangular (e.g., not raster scan shaped). The units of tiles in a picture must cover the picture without any gaps between tiles and without any overlap. When decoding is performed with only one core, for coding of a current coding block (e.g., CTU) that is not at the left edge of the picture, the left neighboring coding block must be decoded before the current coding block. When decoding is performed with only one core, for coding of a current coding block (e.g., CTU) that is not at the top edge of the picture, the top neighboring coding block must be decoded before the current coding block. When two tiles have tile indices (e.g., idx3 and idx4) that are adjacent to each other, one of the following is true: When two tiles share a vertical edge and/or a first tile has a top-left location at (Xa,Ya) with size (Wa and Ha representing its width and height) and a second tile has a top-left location at (Xb,Yb), then Yb=Ya+Ha.

The following constraints may also apply: When a tile has more than one left neighboring tile, the tile height must be equal to the sum of the heights of all its left neighboring tiles. When a tile has more than one right neighboring tile, the tile height must be equal to the sum of the heights of all its left neighboring tiles. When a tile has more than one above neighboring tile, the tile width must be equal to the sum of the widths of all its above neighboring tiles. When a tile has more than one below neighboring tile, the tile width must be equal to the sum of the widths of all its below neighboring tiles. In addition, the signaling of tile IDs, including the mapping between tile indexes and tile IDs, may be based on the number of tiles in the picture. Thus, the mapping may be based on the number of tiles in the picture, rather than based on tile columns and tile rows. For example, a loop may be employed (e.g., from the first index to the last index, where the first index is 0 and the last index is the number of tiles in the picture minus 1) such that each tile index is assigned a tile ID.

The following are specific example embodiments of the above aspects: The CTB raster and tile traversal process may be as follows: A list ColWidth[i], for i ranging from 0 to num_level1_tile_columns_minus1 inclusive, specifying the width of the i-th first level tile column in units of CTB, may be derived as follows:
if(uniform_level1_tile_spacing_flag)
for(i=0;i<=num_level1_tile_columns_minus1;i++)
ColWidth[i]=((i+1)*PicWidthInCtbsY)/(num_level1_tile_columns_minus1+1)-(i*PicWidthInCtbsY)/(num_level1_tile_columns_minus1+1)

else {
ColWidth[num_level1_tile_columns_minus1]=PicWidthInCtbsY (6-1)
for(i=0;i<num_level1_tile_columns_minus1;i++){
ColWidth[i]=tile_level1_column_width_minus1[i]+1
ColWidth[num_tile_level1_columns_minus1] -= ColWidth[i]
}
}

The list RowHeight[j], for j ranging from 0 to num_level1_tile_rows_minus1 inclusive, specifying the height of the jth tile row in CTB units, may be derived as follows:
if(uniform_level1_tile_spacing_flag)
for(j=0;j<=num_level1_tile_rows_minus1;j++)
RowHeight[j]=((j+1)*PicHeightInCtbsY)/(num_level1_tile_rows_minus1+1)-(j*PicHeightInCtbsY)/(num_level1_tile_rows_minus1+1)

else {
RowHeight[num_level1_tile_rows_minus1]=PicHeightInCtbsY (6-2)
for(j=0;j<num_level1_tile_rows_minus1;j++){
RowHeight[j]=tile_level1_row_height_minus1[j]+1
RowHeight[num_level1_tile_rows_minus1] -= RowHeight[j]
}
}

The list colBd[i], for i ranging from 0 to num_level1_tile_columns_minus1+1 inclusive, specifying the location of the i-th tile column boundary in units of CTB, may be derived as follows:
for(colBd[0]=0,i=0;i<=num_level1_tile_columns_minus1;i++)
colBd[i+1]=colBd[i]+ColWidth[i] (6-3)

The list rowBd[j], for j ranging from 0 to num_level1_tile_rows_minus1+1 inclusive, specifying the location of the jth tile row boundary in units of CTB, may be derived as follows:
for(rowBd[0]=0,j=0;j<=num_level1_tile_rows_minus1;j++)
rowBd[j+1]=rowBd[j]+RowHeight[j] (6-4)

The variable NumTilesInPic, which specifies the number of tiles in the picture with reference to the PPS, and the lists TileColBd[i], TileRowBd[i], TileWidth[i], and TileHeight[i], for i ranging from 0 to NumTilesInPic-1 inclusive, which specify the location of the i-th tile column boundary in units of CTBs, the location of the i-th tile row boundary in units of CTBs, the width of the i-th tile column in units of CTBs, and the height of the i-th tile column in units of CTBs, may be derived as follows:
for(tileIdx=0,i=0;i<NumLevel1Tiles;i++){
tileX=i%(num_level1_tile_columns_minus1+1)
tileY=i/(num_level1_tile_columns_minus1+1)
if(!level2_tile_split_flag[i]){ (6-5)
TileColBd[tileIdx]=colBd[tileX]
TileRowBd[tileIdx]=rowBd[tileY]
TileWidth[tileIdx]=ColWidth[tileX]
TileHeight[tileIdx]=RowHeight[tileY]
tileIdx++
}else{
for(k=0;k<=num_level2_tile_columns_minus1[i];k++)
colWidth2[k]=((k+1)*ColWidth[tileX])/(num_level2_tile_columns_minus1[i]+1)-(k*ColWidth[tileX])/(num_level2_tile_columns_minus1[i]+1)

for(k=0;k<=num_level2_tile_rows_minus1[i];k++)
rowHeight2[k]=((k+1)*RowHeight[tileY])/(num_level2_tile_rows_minus1[i]+1)-(k*RowHeight[tileY])/(num_level2_tile_rows_minus1[i]+1)

for(colBd2[0]=0,k=0;k<=num_level2_tile_columns_minus1[i];k++)
colBd2[k+1]=colBd2[k]+colWidth2[k]
for(rowBd2[0]=0,k=0;k<=num_level2_tile_rows_minus1[i];k++)
rowBd2[k+1]=rowBd2[k]+rowHeight2[k]
numSplitTiles=(num_level2_tile_columns_minus1[i]+1)*(num_level2_tile_rows_minus1[i]+1)

for(k=0;k<numSplitTiles;k++) {
tileX2=k%(num_level2_tile_columns_minus1[i]+1)
tileY2=k/(num_level2_tile_columns_minus1[i]+1)
TileColBd[tileIdx]=colBd[tileX]+colBd2[tileX2]
TileRowBd[tileIdx]=rowBd[tileY]+rowBd2[tileY2]
TileWidth[tileIdx]=colWidth2[tileX2]
TileHeight[tileIdx]=rowHeight2[tileY2]
tileIdx++
}
}
}
NumTilesInPic=tileIdx

A list CtbAddrRsToTs[ctbAddrRs] for ctbAddrRs ranging from 0 to PicSizeInCtbsY-1, inclusive, that specifies the conversion from CTB addresses in the CTB raster scan of a picture to CTB addresses in the tile scan may be derived as follows:
for(ctbAddrRs=0;ctbAddrRs<PicSizeInCtbsY;ctbAddrRs++){
tbX=ctbAddrRs%PicWidthInCtbsY
tbY=ctbAddrRs/PicWidthInCtbsY tileFound=FALSE
for(tileIdx=NumTilesInPic-1,i=0;i<NumTilesInPic-1 &&!tileFound;i++){ (6-6)
tileFound=tbX<(TileColBd[i]+TileWidth[i]) &&tbY<(TileRowBd[i]+TileHeight[i])
if(tileFound)
tileIdx=i
}
CtbAddrRsToTs[ctbAddrRs]=0
for(i=0;i<tileIdx;i++)
CtbAddrRsToTs[ctbAddrRs] += TileHeight[i]*TileWidth[i]
CtbAddrRsToTs[ctbAddrRs] += (tbY-TileRowBd[tileIdx])*TileWidth[tileIdx]+tbX-TileColBd[tileIdx]

}

A list CtbAddrTsToRs[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1 inclusive, which specifies the conversion from CTB addresses in the tile scan to CTB addresses in the CTB raster scan of the picture, may be derived as follows:
for(ctbAddrRs=0;ctbAddrRs<PicSizeInCtbsY;ctbAddrRs++) (6-7)
CtbAddrTsToRs[CtbAddrRsToTs[ctbAddrRs]]=ctbAddrRs

The list TileId[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1, inclusive, which specifies the conversion from CTB addresses to tile IDs in a tile traversal, and the list FirstCtbAddrTs[tileIdx] for tileIdx ranging from 0 to NumTilesInPic-1, inclusive, which specifies the conversion from tile IDs to CTB addresses in a tile traversal of the first CTB in the tile, can be derived as follows:
for(i=0,tileIdx=0;i<=NumTilesInPic;i++,tileIdx++){
for(y=TileRowBd[i];y<TileRowBd[i+1];y++) (6-8)
for(x=TileColBd[i];x<TileColBd[i+1];x++)
TileId[CtbAddrRsToTs[y*PicWidthInCtbsY+x]]=tileIdx
FirstCtbAddrTs[tileIdx]=CtbAddrRsToTs[TileColBd[tileIdx]]*PicWidthInCtbsY+TileColBd[tileIdx]]
}

The list NumCtusInTile[tileIdx], for tileIdx ranging from 0 to NumTilesInPic-1 inclusive, which specifies the conversion from tile index to the number of CTUs in the tile, can be derived as follows:
for(i=0,tileIdx=0;i<NumTilesInPic;i++,tileIdx++) (6-9)
NumCtusInTile[tileIdx]=TileColWidth[tileIdx]*TileRowHeight[tileIdx]

An example picture parameter set RBSP syntax is as follows:

An example picture parameter set RBSP semantic is: num_level1_tile_columns_minus1+1 specifies the number of level 1 tile columns that partition the picture. num_level1_tile_columns_minus1 must be in the range of 0 to PicWidthInCtbsY-1, inclusive. When not present, the value of num_level1_tile_columns_minus1 is inferred to be equal to 0. num_level1_tile_rows_minus1+1 specifies the number of level 1 tile rows that partition the picture. num_level1_tile_rows_minus1 must be in the range of 0 to PicHeightInCtbsY-1, inclusive. When not present, the value of num_level1_tile_rows_minus1 is inferred to be equal to 0. The variable NumLevel1Tiles is set equal to (num_level1_tile_columns_minus1+1)*(num_level1_tile_rows_minus1+1). When single_tile_in_pic_flag is equal to 0, NumTilesInPic must be greater than 1. uniform_level1_tile_spacing_flag is set equal to 1 to specify that the level 1 tile column borders, and similarly the level 1 tile row borders, are uniformly distributed across the picture. uniform_level1_tile_spacing_flag is equal to 0 to specify that the level 1 tile column borders, and similarly the level 1 tile row borders, are not uniformly distributed across the picture but are explicitly signaled using the syntax elements level1_tile_column_width_minus1[i] and level1_tile_row_height_minus1[i]. When not present, the value of uniform_level1_tile_spacing_flag is inferred to be equal to 1.

level1_tile_column_width_minus1[i]+1 specifies the width of the ith level 1 tile column in CTB units. level1_tile_row_height_minus1[i]+1 specifies the height of the ith tile level 1 row in CTB units. level2_tile_present_flag specifies that one or more level 1 tiles are split into more level 2 tiles. level2_tile_split_flag[i] specifies that the ith level 1 tile is split into two or more tiles. num_level2_tile_columns_minus1[i]+1 specifies the number of tile columns that partition the ith tile. num_level2_tile_columns_minus1[i] must be in the range 0 to ColWidth[i] inclusive. When not present, the value of num_level2_tile_columns_minus1[i] is inferred to be equal to 0. num_level2_tile_rows_minus1[i]+1 specifies the number of tile rows that divide the i-th tile. num_level2_tile_rows_minus1[i] must be in the range 0 to RowHeight[i], inclusive. When not present, the value of num_level2_tile_rows_minus1[i] is inferred to be equal to 0.

When level2_tile_split_flag[i] is equal to 1, the value (num_level2_tile_columns_minus1[i]+1)*(num_level2_tile_rows_minus1[i]+1) must be greater than 1. A picture may contain zero or more level 1 tiles with level2_tile_split_flag[i] equal to 0, and zero or more level 2 tiles. Level 1 tiles with level2_tile_split_flag[i] equal to 0 do not need to be counted in the total number of tiles in the picture. When such a tile is referenced, the set of level 2 tiles resulting from such a split may be referenced collectively.

The CTB raster and tile scan conversion process is invoked by passing the following variables: a list ColWidth[i] for i ranging from 0 to num_level1_tile_columns_minus1 inclusive, specifying the width of the ith level 1 tile column in CTB units; a list RowHeight[j] for j ranging from 0 to num_level1_tile_rows_minus1 inclusive, specifying the height of the jth level 1 tile row in CTB units; a variable NumTilesInPic, which specifies the number of tiles in the picture by reference to the PPS; a list TileWidth[i] for i ranging from 0 to NumTilesInPic inclusive that specifies the height of the ith tile in CTB units; a list TileHeight[i] for i ranging from 0 to NumTilesInPic inclusive that specifies the location of the ith tile column border in CTB units; a list TileColBd[i] for i ranging from 0 to NumTilesInPic inclusive that specifies the location of the ith tile row border in CTB units; a list TileRowBd[i] for j ranging from 0 to NumTilesInPic inclusive that specifies the location of the ith tile row border in CTB units; A list CtbAddrRsToTs[ctbAddrRs] for ctbAddrRs ranging from 0 to PicSizeInCtbsY-1 inclusive, which specifies the conversion from CTB addresses in the CTB raster scan of the picture to CTB addresses in the tile scan of the picture; a list CtbAddrTsToRs[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1 inclusive, which specifies the conversion from CTB addresses in the tile scan to CTB addresses in the CTB raster scan of the picture; a list CtbAddrTsToRs[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1 inclusive, which specifies the conversion from CTB addresses in the tile scan to tile IDs in the picture The following are derived: a list TileId[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1, a list NumCtusInTile[tileIdx] for tileIdx ranging from 0 to PicSizeInCtbsY-1 inclusive, which specifies the conversion from tile index to the number of CTUs in the tile, and a list FirstCtbAddrTs[tileIdx] for tileIdx ranging from 0 to NumTilesInPic-1 inclusive, which specifies the conversion from tile ID to the CTB address in the tile traversal of the first CTB in the tile.

An example tile group header semantic is as follows: A tile group may contain only some complete level 1 tiles, or only some complete level 2 tiles of one single level 1 tile. tile_group_address specifies the tile address of the first tile in the tile group, where the tile address is equal to TileId[firstCtbAddrTs] as specified by Equation 6-8, where firstCtbAddrTs is the CTB address in the tile traversal of the CTB of the first CTU in the tile group. The length of tile_group_address is Ceil(Log2(NumTilesInPic)) bits. The value of tile_group_address must be in the range of 0 to NumTilesInPic-1, inclusive, and the value of tile_group_address must not be equal to the value of tile_group_address of any other coded tile group NAL unit of the same coded picture. When tile_group_address is not present, it is inferred to be equal to 0.

The following is a second specific exemplary embodiment of the above aspect. An exemplary CTB raster and tile traversal process is as follows: A variable NumTilesInPic, which specifies the number of tiles in the picture with reference to the PPS, and lists TileColBd[i], TileRowBd[i], TileWidth[i], and TileHeight[i], for i ranging from 0 to NumTilesInPic-1 inclusive, which specify the location of the i-th tile column boundary in units of CTBs, the location of the i-th tile row boundary in units of CTBs, the width of the i-th tile column in units of CTBs, and the height of the i-th tile column in units of CTBs, are derived as follows:
for(tileIdx=0,i=0;i<NumLevel1Tiles;i++){
tileX=i%(num_level1_tile_columns_minus1+1)
tileY=i/(num_level1_tile_columns_minus1+1)
if(!level2_tile_split_flag[i]){ (6-5)
TileColBd[tileIdx]=colBd[tileX]
TileRowBd[tileIdx]=rowBd[tileY]
TileWidth[tileIdx]=ColWidth[tileX]
TileHeight[tileIdx]=RowHeight[tileY]
tileIdx++
}else{
if(uniform_level2_tile_spacing_flag[i]){
for(k=0;k<=num_level2_tile_columns_minus1[i];k++)
colWidth2[k]=((k+1)*ColWidth[tileX])/(num_level2_tile_columns_minus1[i]+1)-(k*ColWidth[tileX])/(num_level2_tile_columns_minus1[i]+1)


for(k=0;k<=num_level2_tile_rows_minus1[i];k++)
rowHeight2[k]=((k+1)*RowHeight[tileY])/(num_level2_tile_rows_minus1[i]+1)-(k*RowHeight[tileY])/(num_level2_tile_rows_minus1[i]+1)


}else{
colWidth2[num_level2_tile_columns_minus1[i]]=ColWidth[tileX])
for(k=0;k<=num_level2_tile_columns_minus1[i];k++){
colWidth2[k]=tile_level2_column_width_minus1[k]+1
colWidth2[k] -= colWidth2[k]
}
rowHeight2[num_level2_tile_rows_minus1[i]]=RowHeight[tileY])
for(k=0;k<=num_level2_tile_rows_minus1[i];k++){
rowHeigh2[k]=tile_level2_column_width_minus1[k]+1
rowHeight2[k] -= rowHeight2[k]
}
}
for(colBd2[0]=0,k=0;k<=num_level2_tile_columns_minus1[i];k++)
colBd2[k+1]=colBd2[k]+colWidth2[k]
for(rowBd2[0]=0,k=0;k<=num_level2_tile_rows_minus1[i];k++)
rowBd2[k+1]=rowBd2[k]+rowHeight2[k]
numSplitTiles=(num_level2_tile_columns_minus1[i]+1)*(num_level2_tile_rows_minus1[i]+1)

for(k=0;k<numSplitTiles;k++) {
tileX2=k%(num_level2_tile_columns_minus1[i]+1)
tileY2=k/(num_level2_tile_columns_minus1[i]+1)
TileColBd[tileIdx]=colBd[tileX]+colBd2[tileX2]
TileRowBd[tileIdx]=rowBd[tileY]+rowBd2[tileY2]
TileWidth[tileIdx]=colWidth2[tileX2]
TileHeight[tileIdx]=rowHeight2[tileY2]
tileIdx++
}
}
}
NumTilesInPic=tileIdx

An example picture parameter set RBSP syntax is as follows:

An example picture parameter set RBSP semantic is as follows: uniform_level2_tile_spacing_flag[i] is set equal to 1 to specify that the level 2 tile column boundary of the ith level 1 tile and similarly the level 2 tile row boundary of the ith level 1 tile are uniformly distributed across the picture. uniform_level2_tile_spacing_flag[i] may be set equal to 0 to specify that the level 2 tile column boundary of the ith level 1 tile and similarly the level 2 tile row boundary of the ith level 1 tile are not uniformly distributed across the picture but are explicitly signaled using the syntax elements level2_tile_column_width_minus1[j] and level2_tile_row_height_minus1[j]. When not present, the value of uniform_level2_tile_spacing_flag[i] is inferred to be equal to 1. level2_tile_column_width_minus1[j]+1 specifies the width of the jth level 2 tile column of the ith level 1 tile in CTB units. level2_tile_row_height_minus1[j]+1 specifies the height of the jth tile level 2 row of the ith level 1 tile in CTB units.

The following is a third specific exemplary embodiment of the above aspect: The CTB raster and tile traversal process is as follows: A list ColWidth[i], for i ranging from 0 to num_level1_tile_columns_minus1 inclusive, specifying the width of the i-th first level tile column in units of CTB, may be derived as follows:
for(i=0;i<=num_level1_tile_columns_minus1;i++)
ColWidth[i]=((i+1)*PicWidthInCtbsY)/(num_level1_tile_columns_minus1+1)-(i*PicWidthInCtbsY)/(num_level1_tile_columns_minus1+1)

The list RowHeight[j], for j ranging from 0 to num_level1_tile_rows_minus1 inclusive, specifying the height of the jth tile row in CTB units, may be derived as follows:
for(j=0;j<=num_level1_tile_rows_minus1;j++)
RowHeight[j]=((j+1)*PicHeightInCtbsY)/(num_level1_tile_rows_minus1+1)-(j*PicHeightInCtbsY)/(num_level1_tile_rows_minus1+1)

An example picture parameter set RBSP syntax is as follows:

The following is a fourth specific exemplary embodiment of the above aspect: An exemplary picture parameter set RBSP syntax is as follows:

An example picture parameter set RBSP semantic is as follows: Bitstream adaptation may require that the following constraints apply: The value MinTileWidth specifies the minimum tile width and shall be equal to 256 luma samples. The value MinTileHeight specifies the minimum tile height and shall be equal to 64 luma samples. The values of minimum tile width and minimum tile height may vary according to the profile and level definition. The variable Level1TilesMayBeFurtherSplit may be derived as follows:
Level1TilesMayFurtherBeSplit=0
for(i=0,!Level1TilesMayFurtherBeSplit &&i=0;i<NumLevel1Tiles;i++)
if((ColWidth[i]*CtbSizeY>=(2*MinTileWidth))||(RowHeight[i]*CtbSizeY>=(2*MinTileHeight)))

Level1TilesMayFurtherBeSplit=1

level2_tile_present_flag specifies that one or more level tiles are split into more tiles. When not present, the value of level2_tile_present_flag is inferred to be equal to 0. level2_tile_split_flag[i]+1 specifies that the i-th level 1 tile is split into two or more tiles. When not present, the value of level2_tile_split_flag[i] is inferred to be equal to 0.

The following is a fifth specific exemplary embodiment of the above aspect. Each tile location and each tile size may be signaled. A syntax for supporting such tile structure signaling may be as tabulated below: tile_top_left_address[i] and tile_bottom_right_address[i] are CTU indices in the picture, indicating the rectangular area covered by the tile. The number of bits for signaling these syntax elements should be sufficient to represent the maximum number of CTUs in the picture.

Each tile location and each tile size may be signaled. A syntax to support such tile structure signaling may be as tabulated below: tile_top_left_address[i] is the CTU index of the first CTU in the tile in the order of CTU raster scan of the picture. Tile width and tile height specify the size of the tile. By signaling the tile size common unit first, some bits may be saved when signaling these two syntax elements.

Alternatively, the signaling could be as follows:

In another example, each tile size may be signaled as follows: To signal a flexible tile structure, the location of each tile may not be signaled. Instead, a flag may be signaled to specify whether the tile should be placed immediately to the right or immediately below the previous tile. If the tile can only be to the right or only below the current tile, this flag may not be present.

The values of tile_x_offset[i] and tile_y_offset[i] may be derived by the following ordered steps:
tile_x_offset[0] and tile_y_offset[0] are set equal to 0.
maxWidth is set equal to tile_width[0], and maxHeight is set equal to tile_height[0].
runningWidth is set equal to tile_width[0] and runningHeight is set equal to tile_height[0].
lastNewRowHeight is set equal to 0.
TilePositionCannotBeInferred=false.
For i>0 the following applies:
Let the value isRight be set to:
If runningWidth+tile_width[i]<=PictureWidth, then isRight==1,
Otherwise, isRight==0.
Let the value isBelow be set to:
If runningHeight+tile_height[i]<=PictureHeight, then isBelow==1,
Otherwise, isBelow==0.
If isRight==1 && isBelow==1, then TilePositionCannotBeInferred=true.
If isRight==1 && isBelow==0, then:
right_tile_flag[i]=1,
tile_x_offset[i]=runningWidth,
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight,
lastNewRowHeight=(runningWidth==maxWidth) ? 0 : lastNewRowHeight is applied,
Otherwise, if isRight==0 && isBelow==1, then:
right_tile_flag[i]=0,
tile_y_offset[i]=runningHeight,
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1],
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight is applied,
Otherwise, if isRight==1 && isBelow==1 && right_tile_flag[i]==1, then:
tile_x_offset[i]=runningWidth,
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight,
lastNewRowHeight=(runningWidth==maxWidth) ? 0 : lastNewRowHeight is applied,
Otherwise (i.e., isRight==1 && isBelow==1 && right_tile_flag[i]==0),
tile_y_offset[i]=runningHeight,
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1],
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight is applied,
If right_tile_flag[i]==1, then:
runningWidth = runningWidth + tile_width[i] is applied,
If runningWidth>maxWidth, set maxWidth equal to runningWidth,
runningHeight is equal to tile_y_offset[i] + tile_height[i],
Otherwise (i.e. right_tile_flag[i]==0),
runningHeight = runningHeight + tile_height[i] is applied,
If runningHeight>maxHeight, set maxHeight equal to runningHeight,
runningWidth is equal to tile_x_offset[i] + tile_width[i].

The above can be written in pseudocode as follows:
tile_x_offset[0]=0
tile_y_offset[0]=0
maxWidth=tile_width[0]
maxHeight=tile_height[0]
runningWidth=tile_width[0]
runningHeight = tile_height[0]
lastNewRowHeight=0
isRight=false
isBelow=false
TilePositionCannotBeInferred=false
for(i=1;i<num_tiles_minus2+2;i++){
TilePositionCannotBeInferred=false
isRight=(runningWidth+tile_width[i]<=PictureWidth) ? true : false
isbelow=(runningHeight+tile_height[i]<=PictureHeight) ? true : false
if(!isRight && !isBelow)
// Error, this case should never occur.
if(isRight && isBelow)
TilePositionCannotBeInferred=true
if(isRight && !isBelow) {
right_tile_flag[i]=true
tile_x_offst[i]=runningWidth
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight
lastNewRowHeight=tile_y_offset[i]
}
else if(!isRight && isBelow){
right_tile_flag[i]=false
tile_y_offset[i]=runningHeight
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1]
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight

}
else if(right_tile_flag[i]) {
tile_x_offst[i]=runningWidth
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight
lastNewRowHeight=tile_y_offset[i]
}
else {
tile_y_offset[i]=runningHeight
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1]
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight

}
}
if(right_tile_flag[i]) {
runningWidth += tile_width[i]
if(runningWidth>maxWidth)maxWidth=runningWidth
runningHeight=tile_y_offset[i]+tile_height[i]
}
else {
runningHeight += tile_height[i]
if(runningHeight>maxHeight)maxHeight=runningHeight
runningWidth=tile_x_offset[i]+tile_width[i]
}

Below is one implementation in pseudocode that derives the size of the final tile.
tile_x_offset[0]=0
tile_y_offset[0]=0
maxWidth=tile_width[0]
maxHeight=tile_height[0]
runningWidth=tile_width[0]
runningHeight=tile_height[0]
lastNewRowHeight=0
isRight=false
isBelow=false
TilePositionCannotBeInferred=false
for(i=1;i<num_tiles_minus2+2;i++){
currentTileWidth=(i==num_tiles_minus2+1) ? (PictureWidth-runningWidth)%PictureWidth : tile_width[i]

currentTileHeight=(i==num_tiles_minus2+1) ? (PictureHeight-runningHeight)%PictureHeight : tile_Height[i]

isRight=(runningWidth+currentTileWidth<=PictureWidth)? true : false
isbelow=(runningHeight+currentTileHeight<=PictureHeight) ? true : false
if(!isRight && !isBelow)
// Error, this case should never occur.
if(isRight && isBelow)
TilePositionCannotBeInferred=true
if(isRight && !isBelow) {
right_tile_flag[i]=true
tile_x_offst[i]=runningWidth
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight
lastNewRowHeight=tile_y_offset[i]
}
else if(!isRight && isBelow){
right_tile_flag[i]=false
tile_y_offset[i]=runningHeight
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1]
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight

}
else if(right_tile_flag[i]) {
tile_x_offst[i]=runningWidth
tile_y_offset[i]=(runningWidth==maxWidth) ? 0 : lastNewRowHeight
lastNewRowHeight=tile_y_offset[i]
}
else {
tile_y_offset[i]=runningHeight
tile_x_offset[i]=(runningHeight==maxHeight) ? 0 : tile_x_offset[i-1]
lastNewRowHeight=(runningHeight==maxHeight && runningWidth==maxWidth) ? runningHeight : lastNewRowHeight

}
}
if(right_tile_flag[i]) {
runningWidth += currentTileWidth
if(runningWidth>maxWidth)maxWidth=runningWidth
runningHeight=tile_y_offset[i]+currentTileHeight
}
else {
runningHeight += currentTileHeight
if(runningHeight>maxHeight)maxHeight=runningHeight
runningWidth=tile_x_offset[i]+currentTileWidth
}

For further signaling bit savings, the number of unique tile sizes may be signaled to support tabulation of unit tile sizes. The tile sizes may then be referenced by index only.

The following is a sixth specific exemplary embodiment of the above aspect. An exemplary CTB raster and tile traversal process is as follows: A list TileId[ctbAddrTs] for ctbAddrTs ranging from 0 to PicSizeInCtbsY-1, inclusive, specifying the conversion from CTB addresses to tile IDs in a tile traversal, and a list FirstCtbAddrTs[tileIdx] for tileIdx ranging from 0 to NumTilesInPic-1, inclusive, specifying the conversion from tile IDs to CTB addresses in a tile traversal of the first CTB in a tile, may be derived as follows:
for(i=0,tileIdx=0;i<=NumTilesInPic;i++,tileIdx++){
for(y=TileRowBd[i];y<TileRowBd[i+1];y++) (6-8)
for(x=TileColBd[i];x<TileColBd[i+1];x++)
TileId[CtbAddrRsToTs[y*PicWidthInCtbsY+x]]=explicit_tile_id_flag ? tile_id_val[i] : tileIdx

FirstCtbAddrTs[tileIdx]=CtbAddrRsToTs[TileColBd[tileIdx]]*PicWidthInCtbsY+TileColBd[tileIdx]]

}

The set TileIdToIdx[tileId] for a set of NumTilesInPic tileId values that specifies the conversion from tile IDs to tile indexes may be derived as follows:
for(ctbAddrTs=0,tileIdx=0,tileStartFlag=1;ctbAddrTs<PicSizeInCtbsY;ctbAddrTs++){
if(tileStartFlag) {
TileIdToIdx[TileId[ctbAddrTs]]=tileIdx
tileStartFlag=0
}
tileEndFlag=ctbAddrTs==PicSizeInCtbsY-1||TileId[ctbAddrTs+1]!=TileId[ctbAddrTs]

if(tileEndFlag) {
tileIdx++
tileStartFlag=1
}
}

An example picture parameter set RBSP syntax and semantics are as follows:

In some examples, the semantics may be as follows: tile_id_len_minus1+1 specifies the number of bits used to represent the syntax elements tile_id_val[i] and tile_group_address in the tile group header with reference to the PPS. The value of tile_id_len_minus1 may be in the range of Ceil(Log2(NumTilesInPic) to 15, inclusive. In other examples, the semantics may be as follows: tile_id_len_minus1+1 may specify the number of bits used to represent the syntax elements tile_id_val[i] and tile_group_address in the tile group header with reference to the PPS that references the tile ID value. The value of tile_id_len_minus1 may be in the range of Ceil(Log2(NumTilesInPic) to 15, inclusive.

explicit_tile_id_flag may be set equal to 1 to specify that the tile ID for each tile is explicitly signaled. explicit_tile_id_flag may be set equal to 0 to specify that the tile ID is not explicitly signaled. tile_id_val[i] may specify the tile ID for the i-th tile in the picture. The length of tile_id_val[i] may be tile_id_len_minus1+1 bits. For any integer m in the range 0 to NumTilesInPic-1, inclusive, tile_id_val[i] may not be equal to tile_id_val[j] when i is not equal to j, and tile_id_val[i] may be less than tile_id_val[j] when j is greater than i.

An example tile group header RBSP syntax and semantics is as follows:

tile_group_address may specify the tile address of the first tile in the tile group. The length of tile_group_address may be tile_id_len_minus1+1 bits. The value of tile_group_address may be in the range of 0 to 2 ^{tile_id_len_minus1+1} -1, inclusive, and the value of tile_group_address may not be equal to the value of tile_group_address of any other coded tile group NAL unit of the same coded picture. tile_group_address may be inferred to be equal to 0 when not present.

FIG. 9 is a schematic diagram of an example video coding device 900. The video coding device 900 is suitable for implementing the disclosed examples/embodiments as described herein. The video coding device 900 comprises a downstream port 920, an upstream port 950, and/or a transceiver unit (Tx/Rx) 910 including a transmitter and/or a receiver for communicating data upstream and/or downstream over a network. The video coding device 900 also includes a processor 930 including a logic unit and/or a central processing unit (CPU) for processing data, and a memory 932 for storing data. The video coding device 900 may also include electrical components, optical-to-electrical (OE) components, electrical-to-optical (EO) components, and/or wireless communication components coupled to the upstream port 950 and/or the downstream port 920 for communication of data over an electrical communication network, an optical communication network, or a wireless communication network. Video coding device 900 may also include input and/or output (I/O) devices 960 for communicating data to and from a user. I/O devices 960 may include output devices, such as a display for displaying video data, speakers for outputting audio data, etc. I/O devices 960 may also include input devices, such as a keyboard, mouse, trackball, etc., and/or corresponding interfaces for interacting with such output devices.

The processor 930 is implemented by hardware and software. The processor 930 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and digital signal processors (DSPs). The processor 930 is in communication with the downstream port 920, the Tx/Rx 910, the upstream port 950, and the memory 932. The processor 930 comprises a coding module 914. The coding module 914 implements the disclosed embodiments described herein, such as the methods 100, 1000, and 1100, the mechanism 600, and/or the application 700, which may employ the bitstream 500 and/or the image partitioned according to the flexible video tiling scheme 800. The coding module 914 may also implement any other method/mechanism described herein. Additionally, the coding module 914 may implement the codec system 200, the encoder 300, and/or the decoder 400. For example, the coding module 914 can partition a picture into first level tiles and partition the first level tiles into second level tiles. The coding module 914 can also assign the first level tiles and the second level tiles to tile groups such that each tile group includes a contiguous sequence of several first level tiles or second level tiles of a single first level tile, such that all second level tiles created from a single first level tile are assigned to the same tile group. The coding module 914 can also encode and/or decode such tiles in a scan order such that the first level tiles are coded in raster scan order, and the second level tiles are coded in raster scan order within the first level tile from which such second level tiles are partitioned. The coding module 914 further supports employing such mechanisms to combine sub-pictures at different resolutions into a single picture for various use cases as described herein. Thus, the coding module 914 improves the functionality of the video coding device 900 and addresses problems specific to video coding techniques. Furthermore, the coding module 914 effects the transformation of the video coding device 900 into a different state. Alternatively, the coding module 914 may be implemented as instructions stored in the memory 932 (e.g., as a computer program product stored on a non-transitory medium) and executed by the processor 930.

Memory 932 comprises one or more memory types, such as a disk, a tape drive, a solid state drive, a read only memory (ROM), a random access memory (RAM), a flash memory, a ternary content addressable memory (TCAM), a static random access memory (SRAM), etc. Memory 932 may be used as an overflow data storage device to store such programs when such programs are selected for execution, and to store instructions and data read during program execution.

FIG. 10 is a flowchart of an example method 1000 for encoding an image by employing a flexible tiling scheme, such as flexible tiling scheme 800. Method 1000 may be employed by an encoder, such as codec system 200, encoder 300, and/or video coding device 900, when executing method 100, mechanism 600, and/or supporting application 700. Additionally, method 1000 may be employed to generate bitstream 500 for transmission to a decoder, such as decoder 400.

Method 1000 may begin when an encoder receives a video sequence including multiple images and determines, for example, based on user input, that the video sequence should be encoded into a bitstream. As an example, the video sequence, and therefore the images, may be encoded at multiple resolutions. In step 1001, a picture is partitioned into multiple first level tiles. A subset of the first level tiles may also be partitioned into multiple second level tiles. In some examples, the first level tiles outside the subset may include picture data at the first resolution. Additionally, the second level tiles may include picture data at a second resolution different from the first resolution. In some examples, each first level tile in the subset of first level tiles may include two or more complete second level tiles.

In step 1003, the first level tiles and the second level tiles are assigned to one or more tile groups. The assignment may be performed such that each tile group includes several first level tiles, one or more contiguous sequences of second level tiles where each sequence of second level tiles is divided from a single first level tile, or a combination thereof. As a particular example, the assignment may be performed such that all second level tiles created from a single first level tile are assigned to the same tile group. In some examples, each of the one or more tile groups may be constrained such that all tiles in the assigned tile group cover a rectangular portion of the picture.

In step 1005, the first level tiles and the second level tiles are encoded into the bitstream. For example, the first level tiles and the second level tiles may be encoded according to a scan order. In a particular example, encoding according to a scan order may include encoding the first level tiles in a raster scan order. Upon encountering one of the second level tiles, the raster scan order encoding of the first level tiles may be interrupted. The raster scan order encoding of the first level tiles may continue after all successive second level tiles have been encoded in raster scan order. For example, all second level tiles partitioned from the current first level tile may be encoded before encoding any second level tiles partitioned from the subsequent second level tile. In step 1007, the bitstream may be stored for communication towards a decoder.

11 is a flowchart of an example method 1100 for decoding an image by employing a flexible tiling scheme, such as flexible video tiling scheme 800. Method 1100 may be employed by a decoder, such as codec system 200, decoder 400, and/or video coding device 900, when executing method 100, mechanism 600, and/or supporting application 700. Additionally, method 1100 may be employed upon receiving bitstream 500 from an encoder, such as encoder 300.

Method 1100 may begin when a decoder begins receiving a bitstream of coded data representing a video sequence, for example as a result of method 1000. The bitstream may include video data from the video sequence coded at multiple resolutions. In step 1101, the bitstream is received. The bitstream includes a picture partitioned into multiple first level tiles. A subset of the first level tiles may be further partitioned into multiple second level tiles. In some examples, first level tiles outside the subset may include picture data at the first resolution. Additionally, the second level tiles may include picture data at a second resolution different from the first resolution. In another example, each first level tile in the subset of first level tiles may include two or more complete second level tiles. The first level tiles and the second level tiles are assigned to one or more tile groups. The allocation may be performed such that each tile group includes several first level tiles, one or more contiguous sequences of second level tiles where each sequence of second level tiles is divided from a single first level tile, or a combination thereof. As a particular example, the allocation may be performed such that all second level tiles created from a single first level tile are assigned to the same tile group. In some examples, each of the one or more tile groups may be constrained such that all tiles in the assigned tile group cover a rectangular portion of the picture.

In step 1105, the first level tiles and the second level tiles may be decoded based on one or more tile groups. In some examples, the first level tiles and the second level tiles are decoded according to a scan order. In a particular example, decoding according to the scan order may include decoding the first level tiles in a raster scan order. Upon encountering one of the second level tiles, the raster scan order encoding of the first level tiles may be interrupted. All successive second level tiles may then be encoded in raster scan order before continuing the raster scan order decoding of the first level tiles. For example, all second level tiles partitioned from the current first level tile may be decoded before decoding any second level tiles partitioned from the subsequent second level tile.

In step 1107, a reconstructed video sequence may be generated for display based on the decoded first level tiles and second level tiles.

12 is a schematic diagram of an example system 1200 for coding a video sequence by employing a flexible tiling scheme, such as flexible video tiling scheme 800. System 1200 may be implemented by an encoder and a decoder, such as codec system 200, encoder 300, decoder 400, and/or video coding device 900. Additionally, system 1200 may be employed when implementing methods 100, 1000, 1100, mechanism 600, and/or application 700. System 1200 may also encode data into a bitstream, such as bitstream 500, and decode such bitstream for display to a user.

The system 1200 includes a video encoder 1202. The video encoder 1202 comprises a partition module 1201 for partitioning a picture into a plurality of first level tiles and partitioning a subset of the first level tiles into a plurality of second level tiles. The video encoder 1202 further comprises an allocation module 1203 for allocating the first level tiles and the second level tiles to one or more tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group. The video encoder 1202 further comprises an encoding module 1205 for encoding the first level tiles and the second level tiles into a bitstream. The video encoder 1202 further comprises a storage module 1207 for storing the bitstream for communication towards a decoder. The video encoder 1202 further comprises a transmission module 1209 for transmitting the bitstream towards the decoder. The video encoder 1202 may be further configured to perform any of the steps of the method 1000.

The system 1200 also includes a video decoder 1210. The video decoder 1210 comprises a receiving module 1211 for receiving a bitstream including a picture partitioned into a plurality of first level tiles, a subset of the first level tiles being further partitioned into a plurality of second level tiles, and the first level tiles and the second level tiles being assigned to one or more tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group. The video decoder 1210 further comprises a decoding module 1213 for decoding the first level tiles and the second level tiles based on the one or more tile groups. The video decoder 1210 further comprises a generating module 1215 for generating a reconstructed video sequence for display based on the decoded first level tiles and the second level tiles. The video decoder 1210 may be further configured to perform any of the steps of the method 1100.

A first component is directly coupled to a second component when there are no intervening components between the first and second components, other than a line, trace, or another medium. A first component is indirectly coupled to a second component when there are intervening components between the first and second components, other than a line, trace, or another medium. The term "coupled" and variations thereof include both directly coupled and indirectly coupled. Use of the term "about" means a range that includes ±10% of the succeeding number unless otherwise specified.

It should also be understood that the steps of the exemplary methods described herein are not necessarily required to be performed in the order described, and the order of steps of such methods should be understood to be merely exemplary. Similarly, additional steps may be included in such methods, and some steps may be excluded or combined in methods consistent with various embodiments of the present disclosure.

Although several embodiments are provided in this disclosure, it may be understood that the disclosed systems and methods may be embodied in many other specific forms without departing from the spirit or scope of the disclosure. The examples should be considered as illustrative rather than limiting, and the intent should not be limited to the details given herein. For example, various elements or components may be combined or integrated in another system, or certain features may be excluded or not implemented.

In addition, the techniques, systems, subsystems, and methods described and illustrated in various embodiments as separate or distinct may be combined or integrated with other systems, components, techniques, or methods without departing from the scope of the present disclosure. Other examples of changes, substitutions, and alterations are ascertainable by one of ordinary skill in the art and may be made without departing from the spirit and scope of the present disclosure.

100 How it works
200 Coding and Decoding (Codec) Systems
201 Segmented Video Signal
211 General purpose coder control components
213 Transform Scaling and Quantization Components
215 Intra-picture Estimation Components
217 Intra-picture prediction components
219 Motion Compensation Components
221 Motion Estimation Components
223 Decoded Picture Buffer Components
225 In-loop filter components
227 Filter Control Analysis Components
229 Scaling and Inverse Transformation Components
231 Header Formatting and Context-Adaptive Binary Arithmetic Coding (CABAC) Components
300 Video Encoder
301 Segmented Video Signal
313 Transformation and Quantization Components
317 Intra-picture prediction components
321 Motion Compensation Components
323 Decoded Picture Buffer Components
325 In-loop filter components
329 Inverse Transformation and Quantization Components
331 Entropy Coding Components
400 Video Decoder
417 Intra-picture prediction components
421 Motion Compensation Components
423 Decoded Picture Buffer Components
425 In-loop filter components
429 Inverse Transformation and Quantization Components
433 Entropy Decoding Components
500 bitstream
510 Sequence Parameter Set (SPS)
512 Picture Parameter Set (PPS)
514 Tile Group Header
520 Image data
523, 601, 603, 605, 607 tiles
600 Mechanism
610 Extractor Truck
611 First Resolution
612 Second Resolution
700 Videoconferencing Applications
701 Talking Participants
703 other participants
800 Flexible Video Tiling Method
801 1st level tiles
803 Second Level Tiles
805 Tile Group
807 Traversal Order
900 Video Coding Device
910 Transceiver unit (Tx/Rx)
914 Coding Module
920 downstream ports
930 Processor
932 Memory
950 Upstream Ports
960 Input and/or Output (I/O) Devices
1000 How it works
1100 How it works
1200 System
1201 Division Module
1202 Video Encoder
1203 Allocation Module
1205 Encoding Module
1207 Memory Module
1209 Transmitter module, transmitter
1210 Video Decoder
1211 Receiving module, receiver
1213 Decryption Module
1215 Generation Module

Claims (19)

1. A method of encoding, comprising:
partitioning a picture into a plurality of first level tiles, the plurality of first level tiles being defined by a set of tile columns and a set of tile quantities;
partitioning a subset of the plurality of first level tiles into a plurality of second level tiles, the plurality of second level tiles being assigned to tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group;
encoding the plurality of first level tiles and the plurality of second level tiles into a bitstream;
encoding at least one parameter for decoding the plurality of first level tiles and the plurality of second level tiles into the bitstream. The method of claim 1 further comprising storing the bitstream in a memory of an encoder for communication to a decoder. The method of claim 1 or 2, wherein the at least one parameter is used to derive a number of tiles in the picture. 4. A method according to claim 1, wherein first level tiles outside the subset contain picture data at a first resolution and second level tiles contain picture data at a second resolution different from the first resolution. 5. The method of claim 1, wherein each first level tile in the subset of the plurality of first level tiles comprises two or more complete second level tiles. 6. The method of claim 1, wherein all second level tiles partitioned from a current first level tile are coded before coding any second level tiles partitioned from a subsequent second level tile. A method according to claim 1 , wherein each of the tile groups is constrained such that all tiles in an assigned tile group cover a rectangular portion of the picture. 1. A method of decoding, comprising:
receiving a bitstream including data representing a picture partitioned into a plurality of first level tiles, the plurality of first level tiles being defined by a set of tile columns and a set of tile rows, a subset of the plurality of first level tiles being further partitioned into a plurality of second level tiles, the plurality of second level tiles being assigned to tile groups such that all second level tiles created from a single first level tile are assigned to the same tile group;
and decoding the plurality of first level tiles and the plurality of second level tiles based on at least one parameter in the bitstream. Decoding the plurality of first level tiles and the plurality of second level tiles based on the at least one parameter in the bitstream includes:
deriving a number of tiles in a picture based on the at least one parameter;
decoding the first level tiles and the second level tiles based on the number of tiles in the picture;
The decoding method according to claim 8, comprising: first level tiles outside the subset contain picture data at a first resolution and second level tiles contain picture data at a second resolution different from the first resolution;
A decoding method according to claim 8 or 9. 11. The method of claim 8, wherein each first level tile in the subset of the plurality of first level tiles includes two or more complete second level tiles. 12. The method of claim 8, wherein all second level tiles partitioned from a current first level tile are decoded before decoding any second level tiles partitioned from a subsequent second level tile. 13. A method according to any one of claims 8 to 12, wherein each of the tile groups is constrained such that all tiles in an assigned tile group cover a rectangular portion of the picture. 1. A video coding device, comprising:
A method for controlling a signal processing unit (100) comprising: a processor; a receiver coupled to the processor; and a transmitter coupled to the processor, the processor, the receiver, and the transmitter being configured to perform a method according to any one of claims 1 to 13.
Video coding device. 14. A non-transitory computer-readable medium comprising a computer program for use by a video coding device, the computer program comprising computer-executable instructions stored on the non-transitory computer-readable medium that, when executed by a processor, causes the video coding device to perform a method according to any one of claims 1 to 13. a receiver configured to receive a bitstream for decoding;
a transmitter coupled to the receiver and configured to transmit the decoded image to a display;
An apparatus for decoding comprising: a memory unit coupled to at least one of the receiving unit or the transmitting unit, the memory unit configured to store instructions; and a processing unit coupled to the memory unit, the processing unit configured to execute the instructions stored in the memory unit to perform the method of any one of claims 1 to 13. An encoder comprising a processing circuit for carrying out the method according to any one of claims 1 to 7. A decoder comprising a processing circuit for performing the method according to any one of claims 8 to 13. A computer program comprising program code for carrying out the method according to any one of claims 1 to 13 when the computer program is run on a computer or processor.