JP7513809B2 - Encoder, decoder, and encoding and decoding methods with complex processing for flexibly sized image partitions - Patents.com - Google Patents

Description

This invention relates to video encoding and video decoding, and in particular to encoders, decoders, and encoding and decoding methods involving complex processing for flexibly sized image data.

H.265/HEVC (HEVC = High Efficiency Video Coding) is a video coder that already offers tools to enhance or even enable parallel processing at the encoder and/or decoder. For example, HEVC supports subdivision of an image into an array of tiles that are coded independently of each other. Another concept supported by HEVC is associated with WPP according to which CTU-columns or CTU-lines of an image are processed in parallel, for example striped, from left to right (CTU = coding tree unit). However, it would be preferable to have at hand a video coder that supports the parallel processing capabilities of a video encoder and/or video decoder even more efficiently.

In the following, an introduction to VCL partitioning according to the state of the art is described (VCL = Video Coding Layer).

In general, in video coding, the coding process of image samples requires smaller partitions. The samples are divided into several rectangular areas for joint processing such as prediction or transform coding. Thus, the image is partitioned into blocks of a certain size that is constant during the encoding of a video sequence. In the H.264/AVC standard, fixed-size blocks of 16x16 samples, so-called macroblocks, are used (AVC = Advanced Video Coding).

In the state-of-the-art HEVC standard (see Non-Patent Document 1), there is a maximum size of a coding tree block (CTB) or coding tree unit (CTU) of 64x64 samples. In further descriptions of HEVC, the more common term CTU is used for such kind of blocks.

The CTUs are processed in raster scan order, starting with the top left CTU and processing a line of CTUs in the image, down to the bottom right CTU.

[1] ISO/IEC, ITU-T. High efficiency video coding. ITU-T Rec ommendation H.265 | ISO/IEC 23008 10 (HEVC), edition 1, 2013; edition 2, 2014.

The coded CTU data is organized into a kind of container called a slice. Originally, in previous video coding standards, a slice meant a segment of an image that contained one or more consecutive CTUs. Slice is adopted for a segment of coded data. From another point of view, a complete image can also be defined as one large segment, and later, historically, the term slice is still used. Besides the coded image samples, a slice also contains additional information related to the coding process of the slice itself, which is placed into a so-called slice header.

According to the state of the art, the VCL (Video Coding Layer) also includes techniques for fragmentation and spatial partitioning. Such partitioning is used in video coding for various reasons, for example while processing load balancing in parallelization, CTU size matching in network transmission, error mitigation, etc.

Therefore, it is an object of the present invention to provide an improved concept for video encoding and video decoding.

The object of the invention is solved by the subject matter of the independent claims.

Preferred embodiments are provided in the dependent claims.

The present invention provides an improved concept for video encoding and video decoding.

The following describes in detail an embodiment of the present invention with reference to the drawings.

FIG. 1 shows a video encoder according to an embodiment. FIG. 2 shows a video decoder according to an embodiment. FIG. 3 shows a system according to an embodiment. FIG. 4 illustrates CTUs affected by replacing partial CTUs that generate tile boundaries. FIG. 5 shows luma samples per image compensation from a partial CTU. FIG. 6 shows the CTU grid alignment mismatch. FIG. 7 shows the proportion of CTUs affected by grid mismatch. FIG. 8 shows two sets of tile boundaries. FIG. 9 shows a sequence of corresponding CTUs in a reference image after a partial CTU in a current image. FIG. 10 shows a video encoder. FIG. 11 shows a video decoder. FIG. 12 shows the relationship between, on the one hand, the reconstructed signal, i.e. the reconstructed image, and, on the other hand, the combination of the predicted residual signal and the predicted signal signalled in the data stream. FIG. 13 illustrates image segmentation by slices in raster scan order. FIG. 14 shows image partitioning by tiles.

The following description of the drawings begins with the provision of an encoder and decoder of a block-based predictive encoder for coding images of video to form an example for a coding frame operation in which an embodiment of the present invention is incorporated. The respective encoders and decoders are described with reference to Figs. 10-12. In the following, a description of an embodiment of the present concept is provided together with an explanation of how such concept is incorporated in the encoder and decoder of Figs. 10 and 11, respectively, although the embodiments described in Figs. 1-3 and subsequent figures may also be used to form encoders and decoders that do not operate according to the coding frame operation underlying the encoder and decoder of Figs. 10 and 11.

10 shows a video encoder, an apparatus for predictively coding an image 12 into a data stream 14, exemplarily using a transform-based residual coding. The apparatus, or encoder, is indicated using the reference number 10. FIG. 11 shows a corresponding video decoder 20, i.e. an apparatus 20 configured to predictively decode an image 12' from a data stream 14, using a transform-based residual coding. The apostrophe is used to indicate that the image 12' as reconstructed by the decoder 20 is derived from the image 12 originally coded by the apparatus 10, with respect to the coding loss introduced by the quantization of the predictive residual signal. Although the embodiments of this application are not limited to this type of predictive residual coding, FIGS. 10 and 11 exemplarily use a transform-based predictive residual coding. This also applies to other details described with respect to FIGS. 10 and 11, as outlined below.

The encoder 10 is configured to subject the predicted residual signal to a spatial-to-spectral transformation and thus encode the resulting predicted residual signal into the data stream 14. Similarly, the decoder 20 is configured to decode the predicted residual signal from the data stream 14 and thus subject the resulting predicted residual signal to a spectral-to-spatial transformation.

Inside, the encoder 10 includes a prediction residual signal former 22 which generates a prediction residual signal 24 so as to measure the derivation of a prediction signal 26 from the original signal, i.e. from the image 12. The prediction residual signal former 22 is, for example, a subtractor which derives a prediction signal from the original signal, i.e. from the image 12. The encoder 10 further includes a transformer 28 which subjects the prediction residual signal 24 to a spatial-to-spectral transformation in order to obtain a spectral domain prediction residual signal 24' which is subjected to quantization by a quantizer 32 which is also included in the encoder 10. The quantized prediction residual signal 24'' is thus coded into the bitstream 14. For this purpose, the encoder 10 optionally includes an entropy coder 34 which codes the prediction residual signal such that the entropy is transformed and quantized into the data stream 14. The prediction signal 26 is generated by a prediction stage 36 of the encoder 10 on the basis of the prediction residual signal 24'' encoded into and decoded from the data stream 14. For this purpose, the prediction stage 36 internally includes, as shown in FIG. 10, an inverse quantizer 38 for inverse quantizing the prediction residual signal 24'' to obtain a spectral domain prediction residual signal 24''' corresponding to the signal 24' except for the quantization losses, and, following the inverse quantizer 38, an inverse transformer 40 for subjecting the latter prediction residual signal 24''' to an inverse transformation, i.e. a spectral to spatial transformation, to obtain a prediction residual signal 24'''' corresponding to the original prediction residual signal 24 except for the quantization losses. A combiner 42 of the prediction stage 36 recombines the prediction signal 26 and the prediction residual signal 24'''', for example by addition, to obtain a reconstructed signal 46, i.e. a reconstruction of the original signal 12. The reconstructed signal 46 corresponds to the signal 12'. A prediction module 44 of the prediction stage 36 generates the prediction signal 26 on the basis of the signal 46, for example using a spatial prediction, i.e. intra-image prediction and/or a temporal prediction, i.e. inter-image prediction.

Similarly, the decoder 20 includes components corresponding to the prediction stage 36 therein and interconnected in a manner corresponding to the prediction stage 36, as shown in FIG. 11. In particular, the entropy decoder 50 of the decoder 20 entropy decodes the quantized spectral domain prediction residual signal 24'' from the data stream. Moreover, the inverse quantizer 52, the inverse transformer 54, the combiner 56 and the prediction module 58 are interconnected in the manner described above for the modules of the prediction stage 36 and cooperate to recover a reconstructed signal based on the prediction residual signal 24''. As a result, the output of the combiner 56 produces the reconstructed signal, i.e., the image 12', as shown in FIG. 11.

Although not specifically described above, it is clear that the encoder 10 sets some coding parameters, including, for example, prediction modes and operational parameters, according to some optimization scheme, for example, in a manner that optimizes certain rate- and distortion-related criteria, i.e., coding cost. For example, the encoder 10 and the decoder 20 and corresponding modules 44, 58 support different prediction modes, such as intra-coded and inter-coded modes, respectively. The granularity with which the encoder and the decoder switch between these types of prediction modes corresponds to a subdivision of the image 12, 12' into coded segments or coded blocks, respectively. In units of these coded segments, for example, the image is sub-segmented into intra-coded and inter-coded blocks. The intra-coded blocks are predicted spatially based on the already coded/decoded neighborhood of the respective blocks, as outlined in more detail below. There are several intra-coding modes selected for each intra-coded segment, including directional or angular intra-coding modes according to which each segment is filled by extrapolating neighboring sample values along a particular direction, defined for each directional intra-coding mode, into each intra-coded segment. For example, the intra-coding modes include one or more other modes, such as a DC coding mode according to which the prediction for each intra-coded block is assigned a DC value to all samples in each intra-coded segment, and/or a planar intra-coding mode according to which the prediction for each block is approximated or determined to be a spatial distribution of sample values described by a two-dimensional linear function over the sample positions of each intra-coded block, with the planar drive slope and offset defined by the two-dimensional linear function based on neighboring samples. In comparison, for example, inter-coded blocks are predicted temporally. For inter-coded blocks, motion vectors are signaled in the data stream. The motion vector indicates a spatial displacement of a portion of a previously coded image of the video to which the image 12 belongs. The previously coded/decoded image is then extracted to obtain a prediction signal for each inter-coded block. This means that in addition to the residual signal coding constituted by the data stream 14, the entropy coded transform coefficient levels represent a quantized spectral domain prediction residual signal 24''. The data stream 14 has coded therein coding mode parameters to assign coding modes to the various blocks, and prediction parameters for some blocks, such as motion parameters for the inter-coded segments, and optionally other parameters, such as parameters to control and signal the sub-segments of the image 12 and the image 12' into segments, respectively. The decoder 20 uses these parameters to assign the same prediction modes to the segments, to subdivide the image in the same way as the encoder did, and to perform the same prediction resulting in the same prediction signal.

12 illustrates the relationship between the reconstructed signal, i.e. the reconstructed image 12', on the one hand, and the combination of the predicted residual signal 24''''' and the predicted signal 26 as a signal in the data stream 14, on the other hand. As already indicated above, the combination is additive. The predicted signal 26 is shown in FIG. 12 as a subdivision of the image area into intra-coded blocks, which are indicated diagrammatically with diagonal lines, and into inter-coded blocks, which are indicated diagrammatically without diagonal lines. The subsegment may be a regular subdivision of the image area into columns and rows of rectangular or non-rectangular blocks, or some subdivision such as a multi-tree subdivision of the image 12 from a tree root block into several leaf blocks of varying size. A mixture thereof is shown in FIG. 12. In FIG. 12, the image area is first subdivided into columns and rows of tree root blocks, which are then further subdivided according to a recursive multi-tree subdivision into one or more leaf blocks.

Also, for intra-coded blocks 80, the data stream 14 has an intra-coding mode coded therein, and assigns one of several supported intra-coding modes to each intra-coded block 80. For inter-coded blocks 82, the data stream 14 has one or more motion parameters coded therein. Generally speaking, inter-coded blocks 82 are not limited to being temporally coded. Instead, inter-coded blocks 82 may be blocks predicted from previously coded portions beyond the current picture 12 itself, such as previously coded pictures of the video to which the picture 12 belongs, or pictures of another representation, or even hierarchically lower layers in the case of an encoder and decoder that are scalable encoders and decoders, respectively.

The predicted residual signal 24'''' in Fig. 12 is also shown as a sub-segment of the image area into blocks 84. These blocks are called transform blocks to distinguish them from the coded blocks 80, 82. In effect, Fig. 12 shows that the encoder 10 and the decoder 20 use two different sub-divisions of the image 12 and the image 12' into blocks, respectively, one sub-division into coded blocks 80, 82 and another sub-division into transform blocks 84. Both sub-divisions are the same, i.e., each coded block 80, 82 simultaneously forms a transform block 84. However, Fig. 12 shows the case, for example, where the sub-division into transform blocks 84 forms an extension of the sub-division into coded blocks 80, 82. Thus, any boundary between two blocks 80, 82 overlaps the boundary between two blocks 84. Instead, each block 80, 82 corresponds either to one of the transform blocks or to a group of transform blocks 84. However, the subdivisions may also be determined or selected independently of each other, such that the transformation block 84 crosses the block boundary between the blocks 80 and 82 instead. As far as the subdivision into the transformation block 84 is concerned, similar statements apply as those made for the subdivision into the blocks 80 and 82, i.e., the block 84 is the result of a regular subdivision of the image area into blocks (with or without an arrangement into columns and rows), or the result of a recursive multi-tree subdivision of the image area, or a combination thereof, or any other kind of block. As an aside, it is noted that the blocks 80, 82, 84 are not limited to being quadratic, rectangular or any other shape.

FIG. 12 further illustrates that the combination of the prediction signal 26 and the prediction residual signal 24'''' directly results in the reconstructed signal 12'. However, it should be noted that one or more prediction signals 26 may be combined with the prediction residual signal 24'''' to result in the image 12' according to alternative embodiments.

12, the transform blocks 84 have the following meaning. The transformer 28 and the inverse transformer 54 perform their transforms in units of these transform blocks 84. For example, many encoders use some kind of DST or DCT for all transform blocks 84. Some encoders allow for omitting the transform for some of the transform blocks 84, so that the predicted residual signal is directly coded in the spatial domain. However, according to the embodiment described below, the encoder 10 and the decoder 20 are configured in such a way that they support several transforms. For example, the transforms supported by the encoder 10 and the decoder 20 include the following:

o DCT-II (or DCT-III), where DCT stands for Discrete Cosine Transform.
o DST-IV, where DST stands for discrete sine transform.
DCT-IV
DST-VII
Identity transformation (IT)

Of course, while the transformer 28 supports all of the previous transformation versions of these transformations, the decoder 20 or inverse transformer 54 supports the corresponding regression or inverse versions.

Inverse DCT-II (or inverse DCT-III)
○ Reverse DST-IV
Inverse DCT-IV
○ Reverse DST-VII
Identity transformation (IT)

The following description provides more details on which transforms are supported by the encoder 10 and the decoder 20. In any case, it should be noted that the set of supported transforms simply includes one transform, such as a spectral-to-spatial transform or a spatial-to-spectral transform.

As already outlined above, Figs. 10 to 12 show, by way of example, that the inventive concepts described further below are implemented to form concrete examples for an encoder and a decoder according to the present application. So far, the encoder and decoder of Figs. 10 and 11 respectively represent possible implementations of the encoder and decoder described here below. However, Figs. 10 and 11 are only examples. The encoder according to the embodiment of the present application performs block-based coding of the image 12, unlike the encoder of Fig. 10, etc., using the concepts outlined in more detail below. For example, it is a still image encoder, not a video encoder. It does not support inter prediction. Alternatively, the subdivision into blocks 80 is performed in a different way than illustrated in Fig. 12. Similarly, the decoder according to the embodiment of the present application performs block-based decoding of the image 12' from the data stream 14, using the coding concepts outlined further below. Unlike the encoder of Fig. 10, etc., it performs block-based coding of the image 12, unlike the decoder 20 of Fig. 11, for example, it is a still image decoder, not a video decoder. It does not support intra-prediction, or it sub-segments the image 12' into blocks in a different way than described for FIG. 12, and/or it derives prediction residuals from the data stream 14 in the spatial domain rather than the transform domain, for example.

Image partitioning by slices will now be briefly explained.

Starting with the H.263 standard, sequences of data representing adjacent blocks in a particular scan order are organized into groups called slices. In general, dependencies between the CTUs of different slices of an image, for example with respect to prediction and entropy coding, are prohibited. Thus, each slice in an image is reconstructed independently.

Figure 13 shows image segmentation by slices in raster scan order. The size of a slice is determined by the number of CTUs (coding tree units) and the size of each coded CTU belonging to the slice shown in Figure 13. Figure 13 contains 50 CTUs, e.g. CTU221, CTU224 and CTU251.

Image partitioning with tiles will now be briefly described with reference to FIG. 14. FIG. 14 includes 50 CTUs, e.g., CTU 223, CTU 227, and CTU 241.

Tiles are a concept introduced in HEVC, although the concept is quite similar to Flexible Macroblock Ordering (FMO) that was added to H.264/AVC. The tile concept allows to segment an image into several rectangular regions.

Thus, a tile is the result of segmenting the original image into a given number of columns and rows, with a specified height and width, respectively, as shown in Figure 14. As a result, tiles in a HEVC bitstream are required to have a common boundary that forms a regular grid.

In the following, a general video encoder according to an embodiment is described in FIG. 13. A general video decoder according to an embodiment is described in FIG. 2. And a general system according to an embodiment is described in FIG. 3.

Figure 1 shows a general video encoder 101 according to an embodiment.

The video encoder 101 is configured to encode a plurality of images of a video by generating an encoded video signal. Each of the plurality of images includes original image data.

The video encoder 101 includes a data encoder 110 configured to generate an encoded video signal including encoded image data. The data encoder is configured to encode a plurality of images of a video into the encoded image data.

Furthermore, the video encoder 101 includes an output interface 120 configured to output the encoded image data for each of the multiple images.

Figure 2 shows a typical video decoder 151 according to an embodiment.

The video decoder 151 is configured to decode an encoded video signal that includes encoded image data to reconstruct a plurality of images of the video.

The video decoder 151 includes an input interface 160 configured to receive an encoded video signal.

The video decoder further includes a data decoder 170 configured to reconstruct a plurality of images of the video by decoding the encoded image data.

Figure 3 shows a typical system according to an embodiment.

The system includes a video encoder 101 of FIG. 1 and a video decoder 151 of FIG. 2.

The video encoder 101 is configured to generate an encoded video signal. The video decoder 151 is configured to decode the encoded video signal to reconstruct images of the video.

The first aspect of the invention is claimed in claims 1 to 5, claims 45 to 49, and claims 89 to 93.

The second aspect of the invention is claimed in claims 6 to 13, claims 50 to 57, and claims 94 to 98.

The third aspect of the invention is claimed in claims 14 to 18, claims 58 to 62, and claims 99 to 103.

The fourth aspect of the invention is claimed in claims 19 to 22, claims 63 to 66, and claims 104 to 108.

The fifth aspect of the invention is claimed in claims 24 to 27, claims 68 to 71, and claims 109 to 113.

The sixth aspect of the invention is claimed in claims 28 to 32, claims 72 to 76, and claims 114 to 118.

The seventh aspect of the invention is claimed in claims 33 and 34, in claims 77 and 78, and in claims 119 to 123.

The eighth aspect of the invention is claimed in claims 35 to 38, claims 79 to 82, and claims 124 to 128.

The ninth aspect of the invention is claimed in claims 40 to 44, claims 84 to 88, and claims 129 to 133.

Claim 23, claim 39, claim 67, and claim 83, for example, claim specific examples used for different aspects of the invention.

Below, the complexities of handling for flexibly sized image partitions are described in detail according to an embodiment.

Tiles are defined in HEVC as a sub-segment structure for a picture. They are defined in the Picture Parameter Set (PPS), where their dimensions are given. Tiles, as just defined, have a size that is a multiple of a CTU (typically 64 samples). Only the last tiles of a picture, i.e. tiles at the right or bottom border, are allowed to have their CTUs at the right and bottom picture borders less than 64.

But such a restriction prevents a few use cases.

Merging MCTS bitstreams: Bitstreams that are merged together in a single bitstream need to be multiples of the size chosen for the CTU, so that the original picture boundaries are not picture boundaries in the merged bitstream.

Proper load balancing, where tiles adapt to content, not necessarily in multiples of a predefined size (CTU size).

Furthermore, current video coding standards developed by JVET-Versatile Video Coding allow larger values of CTUs (up to 128), which makes it even more important to allow tiles that are not multiples of a CTU. Although it is possible to define partial CTUs based on whether the requested functionality is necessary, this affects the coding efficiency for such cases. Nevertheless, it is desirable to allow the rightmost CTU and the bottom boundary CTUs of the tile to be less, rather than allowing the tile to contain the full number of CTUs of a determined size.

However, when such features are allowed, different aspects affect the decoding process.

Below, controlling the complexity of partial CTUs within an image is described.

In particular, the first aspect of this invention will now be described in detail.

Currently, all CTUs are the same size. The processor is optimal for decoding configurations of a predetermined size. Variable sizes allow, but impose increased complexity for the decoder. In particular, the number of CTUs can vary from image to image.

A first aspect of the current invention is that when such variable sized CTUs are allowed within each tile, a constraint needs to be met to keep the number of variable sized CTUs constant from image to image.

On the first side, restrictions always apply.

Figure 4 shows the affected CTUs by replacing them with partial CTUs that generate tile boundaries.

If the tile size is not a multiple of the predefined CTU size, variable sized CTUs smaller than the predefined CTU size are introduced. As shown in Figure 4, it is a bitstream matching requirement to have the same number of such variable sized CTUs in each image.

Furthermore, this property is signaled to the decoder using a flag in the parameter set indicating num_partial_ctus_constant_flag.

The second aspect of this invention will now be described in detail.

The second aspect distinguishes partial CTUs that generate tile boundaries from regular tile boundaries in their complexity impact. When the decoder checks the level limit for the maximum amount of decoded pels per time unit, it adds a complexity factor for partial CTUs. For example, if a variable sized CTU is not limited in size by a tile boundary, the partial CTU is counted at 1.5 times its actual size or the number of pels of a predefined CTU size. This means that the image size limit is affected by the presence of partial CTUs. For example, if all CTUs are the same size, the image limit for the decoder is 4096x2160 at 50 frames per second. But if all 5x4 tiles defined have partial CTUs at the right and bottom boundaries, i.e. 20 partial CTUs, the limit is 4056x2128 at the same frame rate.

Figure 5 shows luma samples per image compensation from a partial CTU. In particular, the complexity-adapted image area is shown in Figure 5.

Furthermore, a lower limit on the size of the variable sized CTU is imposed to account for granularity by storing the reference image motion vectors or displacement information with a granularity of, for example, 16x16 samples in the coding process, as opposed to arbitrarily allowing small variable CTUs with sizes between one sample and the pre-determined CTU size in any direction. Variable sized CTUs are not allowed to be smaller than a pre-determined minimum CTU size, which should be dictated or derived from the motion vector storage granularity, for example, so that no two CTUs are within the same motion vector storage unit.

The third aspect of this invention will now be described in detail.

In such another independent third aspect, the presence of partial CTUs is taken into account for tile complexity signaling. For example, HEVC includes a parameter min_spatial_segmentation_idc in the VUI that describes the complexity of the spatial segment that contains the tile. Currently, when applied to a tile, it implies that "no tile in the CVS (coded video sequence) contains more luma samples than (4 * PicSizeInSamplesY) / (min_spatial_segmentation_idc + 4)". Furthermore, as part of this invention, the complexity signaling is limited for tiles to contain no more than a given number of partial CTUs, subject to the maximum tile size and/or the relationship between the default CTU size and the partial CTU size. Instead, an additional parameter is signaled in the VUI that indicates the relative amount of partial CTUs allowed in a picture.

Below, the embodiment explains that CTU boundaries can vary from image to image.

In particular, the fourth aspect of this invention will now be described in detail.

If tiles have their right/bottom boundaries ending at a smaller granularity than the predefined CTU size and a subsequent tile starts with a CTU of the predefined size, alignment of CTUs across images is not achieved when the tile setup changes from image to image, which is detrimental for some predictive coding, such as using temporal motion vector prediction. Memory management becomes more difficult and it is desirable to limit the flexibility of misaligning CTU boundaries from image to image.

The fourth aspect should enforce that any tile boundary that generates a partial CTU can only be moved to a position that generates a partial CTU with the same split ratio in a different image in the same video sequence. This allows to have aligned CTUs in areas that were not covered by the range from the initial last CTU position and the modified last CTU position, as shown in Figure 6.

Figure 6 shows the CTU grid alignment mismatch.

The fifth aspect of this invention will now be described in detail.

In the fifth aspect, a constraint is applied for the rate at which a grid misalignment occurs between pictures with altering sub-tile boundaries, as shown in Figure 7. This constraint is either defined in the default method for a given profile/level, or indicated as a syntax element in the bitstream, e.g., num_misaligned_ctus_per_picture. Figure 7 shows the percentage of CTUs affected by grid misalignment.

Instead, the constraints described herein, in which the number of partial (border) CTUs remains constant in the same ratio, can be implemented as simple constraint directives, such as misalignment_constant_flag, or achieved in a less flexible way, where the constraints need to be fulfilled while the tile boundary directives remain flexible.

The sixth aspect of this invention will now be described in detail.

In the sixth aspect, misalignment of CTUs is completely avoided.Two sets of tile boundaries are defined: The first set of tile boundaries corresponds to conventional tile boundaries.
Entropy coding reset; Intra prediction constraint; Scan order conversion. And, furthermore, the first set of tiles allows generating partial CTUs at the right and bottom tile boundaries.

The second set of tile boundaries only generates partial CTUs at the right and/or bottom tile boundaries, i.e. to the left and/or above said tile boundaries, and does not imply an entropy coding reset or intra prediction or scan order conversion. Thus, the location of the partial CTUs within the image can remain static (and enforces) while the actual tile configuration is allowed to switch over time.

Figure 8 shows two sets of tile boundaries. Tile boundaries that change position in the CVS are not allowed to generate new partial tiles. Figure 8 shows an invention based on two images at different time instants, where a first set of tile boundaries changes position from time instant T-1 to time instant T, and a new second set of tile boundaries is introduced to maintain the position of the partial CTU at time instant T similar to time instant T-1, even if changes occur in scan order and entropy coding and intra prediction.

The seventh aspect of this invention will now be described in detail.

In another separate embodiment, the problem is addressed by introducing partial CTUs to reduce the impact of complexity on line buffer requirements.

Figure 9 shows a sequence of corresponding CTUs in the reference image after a partial CTU in the current image.

As shown in FIG. 9, a horizontal row of partial CTUs in one image leads to a misalignment of the CTU grid with respect to the reference image without a row of partial CTUs. A single row (the current CTU row) accesses the CUs (CU = coding unit) of two CTU rows for tools such as ATMVP (alternative time motion vector prediction). Since aligned CTU rows are often cached in a cost-intensive high-speed memory in a hardware implementation (referred to as a line buffer), it is not desirable to keep a large number of CTU rows in this cache. Therefore, in order to maintain the alignment of CTU rows across the image in the presence of partial CTUs generating tile boundaries, another constraint is that the horizontal tile boundaries generating partial CTUs above them are not allowed to change during the coded video sequence and are therefore static.

Below, predictor derivation from ordered CTUs according to an embodiment is described.

In particular, the eighth aspect of the invention will now be described in detail.

TMVP (Time Motion Vector Prediction) and ATMVP derivation currently depend on a predefined CTU size, i.e., they use a fixed CTU grid over the image.

To TMVP,
If yCb>>CtbLog2SizeY is equal to yColBr>>CtbLog2SizeY, then yColBr is less than pic_height_in_luma_samples and xColBr is less than pic_width_in_luma_samples.
(That is, if the right bottom CB of the reference image is in the same CTU row, it will be cached in the line buffer memory later.)

Then the right-bottom aligned predictor is taken. Otherwise (if the right-bottom CB of the reference image is NOT in the same CTU row, then it is NOT in the line buffer), the center aligned predictor is taken.

For ATMVP, the position (xColSb, yColSb) of the subblock aligned inside the ColPic is derived as follows:
xColSb = Clip3 (xCtb, Min (CurPicWidthInSamplesY - 1, xCtb + (1 << CtbLog2SizeY) + 3), xSb + (tempMv [0] >> 4))
yColSb = Clip3 (yCtb, Min (CurPicHeightInSamplesY - 1, yCtb + (1 << CtbLog2SizeY) - 1), ySb + (tempMv [1] >> 4))

That is, the coordinates of aligned subblocks are clipped in each dimension to lie inside columns of the same CTU. The same line buffer considerations apply.

Both TMVP and ATMVP use a predefined CTU size CtbLog2SizeY that divides all samples of the coded image into coded blocks for the derivation of aligned blocks and sub-blocks.

Due to the tile boundaries allowed to generate partial CTUs, the CTU grid is not constant and varies from image to image, generating overlap between rows of CTUs of varying numbers as shown in FIG. 9. In this part of the invention, the CTU grid of the current image is imposed on the frame of reference to organize buffer management and predictor access. That is, the dimensions or size of the current local partial CTU are applied to select the referenced area in the used (aligned) reference image (as opposed to a constant maximum CTU size across the image).

For TMVP, one embodiment of the invention is as follows:

yTileStart and xTileStart correspond to the top left sample position of a given tile that drives the current coding block, with yCbInTile and xCbInTile corresponding to the tile start and the sample position of the current coding block associated with the aligned block.

(Differences are highlighted in bold type):
yCbInTile=yCb-yTileStart
yColBrInTile = yColBr - yTileStart
CtbLog2HeightY(y) describes the column height of each CTU.
If yCbInTile>>CtbLog2HeightY(yCbInTile) is equal to yColBrInTile>>CtbLog2HeightY(yCbInTile), then yColBr is less than pic_height_in_luma_samples and xColBr is less than pic_width_in_luma_samples.

Then, the predictor aligned at the bottom right is taken. Otherwise, the predictor aligned at the center is taken.

For ATMVP, an embodiment of the type of coordinated clip operation is as follows.
The position (xColSb, yColSb) of the subblocks arranged inside the ColPic is derived as follows.
xColSb = Clip3 (xCtb, Min (CurPicWidthInSamplesY - 1, xCtb + (1 << CtbLog2WidthY (xCtb)) + 3), xSb + (tempMv [0] >> 4))
yColSb=Clip3(yCtb, Min(CurPicHeightInSamplesY-1, yCtb+(1<<CtbLog2HeightY(yCtb))-1),
ySb + (tempMv [1] >> 4)

The sub-image aspect coding tool restriction instructions are explained below.

In particular, the ninth aspect of this invention will now be described in detail.

In previous video coding standards, sub-image (eg slice or tile) aspect coding restrictions are distinguished as follows:
I_SLICE - Intra coding (usually random access points) does not use reference to other pictures.
P_SLICE - Predictive coding. Use one picture from one reference picture list for motion compensated prediction.
B_SLICE - Bi-predictive coding with two reference picture lists. Allows combining motion compensated prediction from pictures in both lists.

The terms slice or tile in this context are interchangeable; that is, a group of contiguous CTUs in the bitstream belong to a single coded image and are entropy coded together.

This difference allows two things:
- controlling the analysis and decoding process of components of each immediate sub-image from the analysis of components of the slice or tile header, and - profiling based on computationally complex types, for example by disallowing the use of such types such as B_SLICES.

The type indication can also be interpreted as a bag of restrictions associated with each type, e.g., the I_SLICE type, which is the most restrictive type, the P_SLICE type, which relaxes the restriction to include inter-prediction using one reference picture, and the B_SLICE type, which relaxes another restriction to include references from both temporally preceding and following pictures by allowing a bitstream order of pictures different from the presentation order.

The following problems arise:
- I_SLICE types also include "Current Picture Reference", which is a P_SLICE type such as a "motion" compensated or translation compensated block based on prediction using the current coded picture as a reference.
- P_SLICE is not used often anymore and is mostly replaced by B_SLICE in applications.
- Inter prediction goes beyond the use of two reference frames.

Current picture referencing, as seen above, imposes additional complexity on the decoding process compared to conventional intra prediction. Conventionally, intra prediction relies only on sample values in the current picture immediately adjacent to the currently predicted block, e.g. samples of the direct neighbourhood, or in so-called multi-reference line prediction, a small number (e.g. three) of lines of the direct neighbourhood samples, to generate sample values of the predicted block, e.g. through interpolation. Current picture referencing, on the other hand, relies on a mechanism only previously used in inter prediction between frames of a video sequence, where sample values of a reference block are copied to the position of the current block. The amount of samples accessed to generate a predictor in this technique is much higher than in regular intra prediction. For that reason, not only are the predictor samples derived from much fewer accessed samples compared, but the search on the encoder side to derive suitable motion vectors or substitution information is much higher. Furthermore, the reference picture needs to be updated along the decoding process of the current picture, to bring about the availability of a larger area, i.e. a reconstructed area, on top of the current picture for reference.

In one embodiment of the invention, instead of using the type distinctions above, the actual imposed tool limitations are signaled in the (tile) header.

1) A flag is used to indicate whether the use of a reference image other than the current image is used (intra/inter switch).

2) A flag is used to indicate the use of current picture referencing (inter prediction using the current coded picture). This is signaled dependent on the intra/inter selection.

3) A syntax element is signaled to indicate the number of reference picture lists used in the prediction process. In the case where only the current picture referencing is used, this syntax element is assumed to be equal to 1. In other cases, a value of 1 applies to what was previously a P_SLICE type, while a value of 2 applies to what was previously a B_SLICE type. Additional values are used, for example, to indicate multi-hypothesis prediction.
An example syntax is shown in Table 1.

current_picture_only_reference_flag equal to 1 indicates that only the current image is allowed to be used as a reference. This is similar to the traditional I_SLICE type.

current_picture_inter_reference_flag equal to 1 indicates that the current picture is available for inter prediction ("Reference current picture" or "Intra block copy" tool).

num_ref_pic_lists_active indicates the number of active reference picture lists.
If current_picture_only_reference_flag is equal to 1,
If current_picture_inter_reference_flag is equal to 0,
num_ref_pic_lists_active is active to 0 (and is excluded from the bitstream).
If current_picture_only_reference_flag is equal to 0,
num_ref_pic_lists_active equal to 1 indicates a tile type similar to P_SLICE,
num_ref_pic_lists_active equal to 2 indicates a tile type similar to B_SLICE,
A num_ref_pic_lists_active greater than 2 is used to indicate the use of more reference picture lists, eg, for multi-hypothesis inter prediction.

Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of the corresponding method. A block or device corresponds to a method step or a function of a method step. Analogously, aspects described in the context of a method step also represent a description of a corresponding block or item or function of a corresponding apparatus. Some or all of the method steps may be performed by (or using) a hardware apparatus, such as, for example, a microprocessor, a programmable computer, or an electronic circuit. In some embodiments, one or more of the most important method steps are performed by such an apparatus.

Depending on the specific implementation requirements, embodiments of the invention can be implemented in hardware or in software or at least partially in hardware or at least partially in software. Implementation can be performed using a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM, EPRROM, EEPROM, or flash memory, having electronically readable control signals stored thereon, in cooperation (or capable of cooperating) with a programmable computer system such that the respective methods are performed. The digital storage medium is thus computer readable.

Some embodiments according to the invention include a data carrier having electronically readable control signals that may cooperate with a programmable computer system to perform one of the methods described herein.

In general, embodiments of the invention may be implemented as a computer program product having program code. The program code is operated to perform one of the methods when the computer program product is run on a computer. The program code may, for example, be stored on a machine readable carrier.

Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

Thus, an embodiment of the inventive method is therefore a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

Another embodiment of the inventive method is therefore a data carrier (or digital storage medium or computer readable medium) comprising recorded thereon a computer program for performing one of the methods described herein. The data carrier or digital storage medium or recorded medium is generally tangible and/or non-transitory.

A further embodiment of the inventive method is therefore a data stream or sequence of signals representing a computer program for performing one of the methods described herein. The data stream or sequence of signals is adapted to be transferred via a data communication connection, for example via the Internet.

Another embodiment comprises a processing means, for example a computer or a programmable logic device, configured or adapted to perform one of the methods described herein.

Another embodiment includes a computer having installed thereon a computer program for performing one of the methods described herein.

Another embodiment according to the invention includes an apparatus or system configured to transfer (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. The receiver may be, for example, a computer, a mobile device, a memory device, or the like. The apparatus or system may include, for example, a file server for transferring the computer program to the receiver.

In some embodiments, a programmable logic device (e.g., a field programmable gate array) is used to perform some or all of the functions of the methods described herein. In some embodiments, the field programmable gate array cooperates with a microprocessor to perform one of the methods described herein. In general, the methods are preferably performed by any hardware apparatus.

The apparatus described herein may be implemented using a hardware apparatus, a computer, or a combination of a hardware apparatus and a computer.

The methods described herein may be performed using a hardware apparatus, or using a computer, or using a combination of a hardware apparatus and a computer.

The above described embodiments are merely illustrative of the principles of the invention. It is understood that modifications and variations of the arrangements and details described herein will be apparent to others skilled in the art. It is therefore the intention to be limited only by the scope of the impending patent claims and not by the specific details provided in the manner of description and illustration of the embodiments herein.

Claims (6)

1. A video encoder including at least one processor and a memory, the memory being configured to, when executed by the at least one processor, cause the video encoder to:
encoding a first flag for a segment of the current image into the data stream, said first flag being equal to 1 indicates that the blocks of said segment are only intra predicted, meaning that said blocks of said segment are predicted only on the basis of reconstructed samples of said current image;
determining, based on the first flag being equal to one, to encode a second flag into the data stream;
encoding the second flag into the data stream indicating whether a current block of the segment is coded using intra block copy (IBC), and if the current block is coded using IBC, the current block is predicted by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on the second flag indicating that the current block is coded using IBC;
encoding a prediction residual into the data stream based on the prediction of the current block;
A video encoder including instructions. 1. A video decoder including at least one processor and a memory, the memory, when executed by the at least one processor, causing the video decoder to:
decoding a first flag for a segment of a current image from a data stream, the segment being encoded in the data stream, the first flag being equal to 1 indicating that blocks of the segment are only intra predicted, meaning that blocks of the segment are predicted based only on reconstructed samples of the current image;
determining, based on the first flag being equal to one, to decode a second flag from the data stream;
decoding, from the data stream, the second flag indicating whether a current block of the segment is coded using intra block copy (IBC), and if the current block is coded using IBC, the current block is predicted by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on a second flag indicating that the current block is coded using IBC;
reconstructing the current block based on a prediction value of the current block and a prediction residual decoded from the data stream.
A video decoder including instructions. 1. A video encoding method comprising the steps of:
- encoding a first flag for a segment of the current image into a data stream, said first flag being equal to 1 indicates that the blocks of said segment are only intra predicted, meaning that said blocks of said segment are predicted based only on reconstructed samples of said current image;
determining, based on the first flag being equal to one, to encode a second flag into the data stream;
encoding the second flag into the data stream indicating whether a current block of the segment is coded using intra block copy (IBC), and if the current block is coded using IBC, the current block is predicted by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on the second flag indicating that the current block is coded using IBC;
encoding a prediction residual into the data stream based on the prediction of the current block;
13. A video encoding method comprising: 1. A video decoding method, comprising:
decoding a first flag for a segment of a current image from a data stream, the segment being encoded in the data stream, the first flag being equal to 1 indicating that blocks of the segment are only intra predicted, meaning that blocks of the segment are predicted based only on reconstructed samples of the current image;
determining to decode a second flag from the data stream based on the first flag being equal to one;
decoding, from the data stream, the second flag indicating whether a current block of the segment is coded using intra block copy (IBC), and if the current block is coded using IBC, predicting the current block by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on a second flag indicating that the current block is coded using IBC;
reconstructing the current block based on a prediction value of the current block and a prediction residual decoded from the data stream;
13. A video decoding method comprising: A non-transitory digital storage medium having a computer program stored thereon, the computer program performing steps when executed by a computer,
The steps include:
- coding a first flag for a segment of the current image into a data stream, said first flag being equal to 1 indicates that the blocks of said segment are only intra predicted, meaning that said blocks of said segment are predicted only on the basis of reconstructed samples of said current image;
determining, based on the first flag being equal to 1, to encode a second flag into the data stream;
encoding the second flag into the data stream indicating whether a current block of the segment is coded using Intra Block Copy (IBC), and if the current block is coded using IBC, the current block is predicted by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on the second flag indicating that the current block is coded using IBC;
encoding a prediction residual into the data stream based on the prediction of the current block;
(c) Non-transitory digital storage media, including A non-transitory digital storage medium having a computer program stored thereon, the computer program performing steps when executed by a computer,
The steps include:
- decoding a first flag for a segment of the current image from a data stream, said segment being coded in said data stream, said first flag being equal to 1 indicates that blocks of said segment are only intra predicted, meaning that blocks of said segment are predicted only on the basis of reconstructed samples of said current image;
determining to decode a second flag from the data stream based on the first flag being equal to 1;
decoding, from the data stream, the second flag indicating whether a current block of the segment is coded using intra block copy (IBC), and if the current block is coded using IBC, predicting the current block by copying samples from a reference block of reconstructed samples of the current image;
determining a prediction of the current block by copying samples from the reference block of reconstructed samples of the current image based on a second flag indicating that the current block is coded using IBC;
reconstructing the current block based on a prediction value of the current block and a prediction residual decoded from the data stream;
(c) Non-transitory digital storage media, including

Images