JP7832938B2 - Improved motion vector on the multipath decoder side - Google Patents

Description

This application claims priority to U.S. Patent Application No. 17/556,142, filed on December 20, 2021, and U.S. Provisional Application No. 63/129,221, titled "MULTI-PASS DECODER-SIDE MOTION VECTOR REFINEMENT," filed on December 22, 2020, the entire contents of these applications being incorporated herein by reference. U.S. Patent Application No. 17/556,142, filed on December 20, 2021, claims the benefit of U.S. Provisional Application No. 63/129,221, filed on December 22, 2020.

This disclosure relates to video coding and video decoding.

Digital video capabilities can be incorporated into a wide range of devices, including digital television, digital direct broadcast systems, wireless broadcast systems, personal digital assistants (PDAs), laptop or desktop computers, tablet computers, e-book readers, digital cameras, digital recording devices, digital media players, video gaming devices, video game consoles, cellular or satellite radio phones, so-called "smartphones," video teleconferencing devices, and video streaming devices. Digital video devices implement video coding techniques such as those described in standards defined by MPEG-2, MPEG-4, ITU-T H.263, ITU-T H.264/MPEG-4, Part 10, Advanced Video Coding (AVC), ITU-T H.265/High Efficiency Video Coding (HEVC), and extensions of such standards. By implementing such video coding techniques, video devices can more efficiently transmit, receive, encode, decode, and/or store digital video information.

Video coding techniques involve spatial (intra-picture) and/or temporal (inter-picture) predictions to reduce or eliminate redundancy inherent in video sequences. In block-based video coding, a video slice (e.g., a video picture, or a portion of a video picture) may be divided into video blocks, which are sometimes called coding tree units (CTUs), coding units (CUs), and/or coding nodes. Video blocks in an intra-coded (I) slice of a picture are coded using spatial predictions for reference samples in adjacent blocks within the same picture. Video blocks in an intercoded (P or B) slice of a picture may use spatial predictions for reference samples in adjacent blocks within the same picture or temporal predictions for reference samples in other reference pictures. Pictures are sometimes called frames, and reference pictures are sometimes called reference frames.

In general, this disclosure describes techniques for decoder-side motion vector derivation techniques. More specifically, this disclosure describes a multi-pass decoder-side motion vector improvement technique for use in video coding. In some draft video standards, the scope of motion vector improvement may be too narrow for all cases. The technique of this disclosure addresses this problem, which can result in more accurate motion prediction and therefore more accurate decoding and reproduction of encoded video data.

In one example, the method includes the steps of: applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector; and decoding the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of the block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of the block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

In another example, the device includes a memory configured to store video data and one or more processors implemented in circuitry and communicatively coupled to the memory, the one or more processors configured to apply a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector, wherein the multipath DMVR comprises a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

In another example, a non-temporary computer-readable medium stores instructions that, when executed, cause one or more processors to apply a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector. The multipath DMVR comprises: a first pass that is block-based and applied to the block of video data; a second pass that is sub-block-based and applied to at least one second pass sub-block of the block of video data, wherein the width of the second pass sub-block is less than or equal to the width of the block of video data and the height of the second pass sub-block is less than or equal to the height of the block of video data; and a third pass that is sub-block-based and applied to at least one third pass sub-block of the block of video data, wherein the width of the third pass sub-block is less than or equal to the width of the second pass sub-block and the height of the third pass sub-block is less than or equal to the height of the second pass sub-block.

In another example, the device includes means for applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector, and means for decoding the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of the block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of the block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

In one example, the method includes the steps of applying a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine the improved motion vectors, and coding the blocks based on the improved motion vectors.

In another example, the device includes memory configured to store video data and one or more processors implemented by circuitry and communicatively coupled to the memory, the one or more processors configured to perform any of the techniques of this disclosure.

In another example, the device includes at least one means for performing any of the techniques of this disclosure.

In another example, a computer-readable storage medium encodes instructions that, when executed, cause a programmable processor to perform one of the techniques of this disclosure.

Details of one or more examples are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims.

Block diagram shows an exemplary video encoding and decoding system capable of performing the techniques of this disclosure. This is a conceptual diagram illustrating an exemplary quadrow-binary tree (QTBT) structure. This is a conceptual diagram showing the corresponding coding tree unit (CTU). A block diagram illustrating a video encoder capable of performing the techniques of this disclosure. A block diagram showing an exemplary video decoder capable of performing the techniques of this disclosure. This is a conceptual diagram showing exemplary spatially adjacent MV candidates for merge mode. This is a conceptual diagram showing exemplary spatially adjacent MV candidates for AMVP mode. This is a conceptual diagram showing an example of a TMVP candidate. This is a conceptual diagram illustrating exemplary MV scaling. This is a conceptual diagram illustrating exemplary template matching in the search area around the initial MV. This is a conceptual diagram illustrating an example where MVD0 and MVD1 are proportional based on time distance. This is a conceptual diagram illustrating an example where MVD0 and MVD1 are mirror images of each other regardless of time and distance. This is a conceptual diagram showing an example of a 3x3 square search pattern within the search range [-8,8]. This is a conceptual diagram illustrating an example of decoder-side motion vector improvement. This is a conceptual diagram illustrating an example of an extended CU region used in BDOF. This is a conceptual diagram illustrating an exemplary 3-pass DMVR technique. This is a conceptual diagram illustrating an example of BDOF motion vector improvement. This flowchart shows an exemplary multi-pass DMVR technique in this disclosure. This flowchart shows an exemplary method for encoding a current block using the technique of the present disclosure. This flowchart shows an exemplary method for decrypting a current block using the technique of the present disclosure.

In some draft video standards, the scope of motion vector improvement may be too narrow for all cases. This can lead to erroneous motion prediction and therefore less accurate decoding. The technique of this disclosure addresses this problem and can result in more accurate motion prediction and therefore more accurate decoding and reproduction of encoded video data.

Figure 1 is a block diagram illustrating an exemplary video coding and decoding system 100 capable of performing the techniques of this disclosure. The techniques of this disclosure generally pertain to coding (encoding and/or decoding) video data. Generally, video data includes any data for processing video. Therefore, video data may include raw, unencoded video, encoded video, decoded (e.g., reconstructed) video, and video metadata such as signaling data.

As shown in Figure 1, system 100 includes, in this example, a source device 102 that provides encoded video data to be decoded and displayed by the destination device 116. Specifically, the source device 102 provides video data to the destination device 116 via a computer-readable medium 110. The source device 102 and the destination device 116 may comprise any of the wide range of devices, including desktop computers, notebook (i.e., laptop) computers, mobile devices, tablet computers, set-top boxes, telephone handsets such as smartphones, televisions, cameras, display devices, digital media players, video gaming consoles, video streaming devices, broadcast receiver devices, and so on. In some cases, the source device 102 and the destination device 116 may be compatible with wireless communication and may therefore be referred to as wireless communication devices.

In the example in Figure 1, the source device 102 includes a video source 104, memory 106, a video encoder 200, and an output interface 108. The destination device 116 includes an input interface 122, a video decoder 300, memory 120, and a display device 118. According to this disclosure, the video encoder 200 of the source device 102 and the video decoder 300 of the destination device 116 may be configured to apply techniques for decoder-side motion vector derivation. Thus, the source device 102 represents an example of a video encoding device, while the destination device 116 represents an example of a video decoding device. In other examples, the source and destination devices may include other components or configurations. For example, the source device 102 may receive video data from an external video source, such as an external camera. Similarly, the destination device 116 may interface with an external display device instead of including an integrated display device.

The system 100 shown in Figure 1 is merely an example. Generally, any digital video decoding device can perform techniques for decoder-side motion vector derivation. Source device 102 and destination device 116 are merely examples of coding devices, such that source device 102 generates coded video data for transmission to destination device 116. This disclosure refers to a device that performs coding (encoding and/or decoding) of data as a “coding” device. Therefore, video encoder 200 and video decoder 300 represent examples of coding devices, specifically, video encoder and video decoder, respectively. In some examples, source device 102 and destination device 116 may operate substantially symmetrically, such that each of source device 102 and destination device 116 includes video encoding and decoding components. Thus, system 100 may support one-way or two-way video transmission between source device 102 and destination device 116 for, for example, video streaming, video playback, video broadcasting, or video phone.

Generally, the video source 104 represents a source of video data (i.e., raw, unencoded video data), providing the video encoder 200 with a sequence of pictures (also called "frames") of video data, which the video encoder 200 then encodes. The video source 104 of source device 102 may include video capture devices such as a video camera, a video archive containing previously captured raw video, and/or a video feed interface for receiving video from a video content provider. As a further alternative, the video source 104 may generate computer graphics-based data as source video, or a combination of live video, archived video, and computer-generated video. In each case, the video encoder 200 encodes the captured, pre-captured, or computer-generated video data. The video encoder 200 may rearrange the pictures from the order in which they were received (sometimes called the "display order") to the coding order for encoding. The video encoder 200 may generate a bitstream containing the encoded video data. The source device 102 can then output the encoded video data to the computer-readable medium 110 via the output interface 108, for reception and/or retrieval, for example, through the input interface 122 of the destination device 116.

Memory 106 of source device 102 and memory 120 of destination device 116 represent general-purpose memory. In some examples, memories 106 and 120 may store raw video data, for example, raw video from video source 104 and raw decoded video data from video decoder 300. Additionally or alternatively, memories 106 and 120 may store, for example, software instructions executable by video encoder 200 and video decoder 300, respectively. Although memories 106 and 120 are shown separately from video encoder 200 and video decoder 300 in this example, it should be understood that video encoder 200 and video decoder 300 may also include internal memory for functionally similar or equivalent purposes. Furthermore, memories 106 and 120 may store encoded video data, for example, output from video encoder 200 and input to video decoder 300. In some examples, portions of memory 106, 120 may be allocated as one or more video buffers for storing, for example, raw, decoded, and/or encoded video data.

The computer-readable medium 110 may represent any type of medium or device capable of transferring encoded video data from the source device 102 to the destination device 116. For example, the computer-readable medium 110 may represent a communication medium that enables the source device 102 to directly transmit encoded video data to the destination device 116 in real time, for example, over a radio frequency network or a computer-based network. The output interface 108 may modulate the transmission signal containing the encoded video data, and the input interface 122 may demodulate the received transmission signal according to a communication standard such as a wireless communication protocol. The communication medium may include any wireless or wired communication medium, such as a radio frequency (RF) spectrum or one or more physical transmission lines. The communication medium may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The communication medium may include routers, switches, base stations, or any other equipment that may be useful in facilitating communication from the source device 102 to the destination device 116.

In some examples, the source device 102 may output encoded data to the storage device 112 via the output interface 108. Similarly, the destination device 116 may access encoded data from the storage device 112 via the input interface 122. The storage device 112 may include any of a variety of distributed or locally accessed data storage media, such as a hard drive, Blu-ray disc, DVD, CD-ROM, flash memory, volatile or non-volatile memory, or any other suitable digital storage medium for storing encoded video data.

In some examples, the source device 102 may output the encoded video data to a file server 114 or another intermediate storage device capable of storing the encoded video data generated by the source device 102. The destination device 116 may access the stored video data from the file server 114 via streaming or download.

The file server 114 can be any type of server device capable of storing encoded video data and transmitting that encoded video data to the destination device 116. The file server 114 may represent a web server (for example, a website), a server configured to provide file transfer protocol services (such as the File Transfer Protocol (FTP) or File Delivery over Unidirectional Transport (FLUTE) protocol), a Content Delivery Network (CDN) device, a Hypertext Transfer Protocol (HTTP) server, a Multimedia Broadcast Multicast Service (MBMS) or Enhanced MBMS (eMBMS) server, and/or a Network Attached Storage (NAS) device. The file server 114 may, additionally or alternatively, implement one or more HTTP streaming protocols, such as Dynamic Adaptive Streaming over HTTP (DASH), HTTP Live Streaming (HLS), Real Time Streaming Protocol (RTSP), or HTTP Dynamic Streaming.

The destination device 116 may access encoded video data from the file server 114 via any standard data connection, including an internet connection. This may include wireless channels (e.g., Wi-Fi connection), wired connections (e.g., digital subscriber line (DSL), cable modem, etc.), or a combination of both suitable for accessing encoded video data stored on the file server 114. The input interface 122 may be configured to operate according to one or more of the various protocols described above for retrieving or receiving media data from the file server 114, or other such protocols for retrieving media data.

The output interface 108 and input interface 122 may represent a wireless transmitter/receiver, modem, wired networking component (e.g., Ethernet card), wireless communication component operating according to any of the various IEEE 802.11 standards, or other physical components. In examples where the output interface 108 and input interface 122 include wireless components, the output interface 108 and input interface 122 may be configured to transfer data such as encoded video data according to cellular communication standards such as 4G, 4G-LTE (Long-Term Evolution), LTE Advanced, or 5G. In some examples where the output interface 108 includes a wireless transmitter, the output interface 108 and input interface 122 may be configured to transfer data such as encoded video data according to other wireless standards such as the IEEE 802.11 specification, the IEEE 802.15 specification (e.g., ZigBee®), or the Bluetooth® standard. In some examples, the source device 102 and/or destination device 116 may include their respective system-on-chip (SoC) devices. For example, the source device 102 may include an SoC device for performing functions related to the video encoder 200 and/or the output interface 108, and the destination device 116 may include an SoC device for performing functions related to the video decoder 300 and/or the input interface 122.

The techniques of this disclosure can be applied to video coding supporting any of a variety of multimedia applications, such as television broadcasting via radio waves, cable television transmission, satellite television transmission, internet streaming video transmission such as dynamic adaptive streaming over HTTP (DASH), digital video encoded on a data storage medium, decoding of digital video stored on a data storage medium, or other applications.

The input interface 122 of the destination device 116 receives an encoded video bitstream from a computer-readable medium 110 (e.g., a communication medium, storage device 112, file server 114, etc.). The encoded video bitstream may include signaling information defined by the video encoder 200, which is also used by the video decoder 300, such as syntax elements having values describing the characteristics and/or processing of video blocks or other coded units (e.g., slices, pictures, picture groups, sequences, etc.). The display device 118 displays the decoded picture of the decoded video data to the user. The display device 118 may represent any of various display devices, such as a liquid crystal display (LCD), plasma display, organic light-emitting diode (OLED) display, or another type of display device.

Although not shown in Figure 1, in some examples, the video encoder 200 and video decoder 300 may each be integrated with an audio encoder and/or audio decoder, and may include a suitable MUX-DEMUX unit or other hardware and/or software to handle multiplexed streams containing both audio and video in a common data stream. Where applicable, the MUX-DEMUX unit may comply with the ITU H.223 Multiplexer Protocol or other protocols such as the User Datagram Protocol (UDP).

The video encoder 200 and video decoder 300 may each be implemented as one or more suitable encoder and/or decoder circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, software, hardware, firmware, or any combination thereof. When the technique is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable medium and execute those instructions in hardware using one or more processors to perform the technique of this disclosure. Each of the video encoder 200 and video decoder 300 may be included in one or more encoders or decoders, any of which may be integrated as part of a composite encoder/decoder (codec) within each device. A device including the video encoder 200 and/or video decoder 300 may comprise an integrated circuit, a microprocessor, and/or a wireless communication device such as a mobile phone.

The video encoder 200 and video decoder 300 may operate in accordance with video coding standards such as ITU-T H.265, also known as High Efficiency Video Coding (HEVC), or their extensions, such as multiview and/or scalable video coding extensions. Alternatively, the video encoder 200 and video decoder 300 may operate in accordance with other proprietary or industry standards, such as ITU-T H.266, also known as Versatile Video Coding (VVC). The draft VVC standard is described in Bross et al., "Versatile Video Coding Editorial Refinements on Draft 10," ITU-T SG 16 WP 3 and the Joint Video Experts Team (JVET) of ISO/IEC JTC 1/SC 29/WG 11, 18th meeting held remotely, October 7-16, 2020, JVET-T2001-v1 (hereinafter "VVC Draft 10"). However, the techniques described herein are not limited to any specific coding standard.

Generally, the video encoder 200 and video decoder 300 may perform block-based coding of pictures. The term "block" generally refers to a structure containing data to be processed (e.g., to be coded, coded, or otherwise used in the coding and/or decoding process). For example, a block may contain a two-dimensional matrix of samples of luminance and/or chromaticity data. Generally, the video encoder 200 and video decoder 300 may code video data represented in YUV (e.g., Y, Cb, Cr) format. That is, rather than coding red, green, and blue (RGB) data for the samples of a picture, the video encoder 200 and video decoder 300 may code luminance and chromaticity components, and the chromaticity component may include both red and blue chromaticity components. In some examples, the video encoder 200 converts the received RGB-formatted data to a YUV representation before coding, and the video decoder 300 converts the YUV representation to RGB format. Alternatively, pre-processing and post-processing units (not shown) may perform these conversions.

This disclosure may generally refer to coding a picture (e.g., encoding and decoding) as including the process of encoding or decoding the data of a picture. Similarly, this disclosure may refer to coding a block of a picture as including the process of encoding or decoding data for a block, e.g., predictive and/or residual coding. An encoded video bitstream generally contains a set of values for syntax elements that represent the coding decision (e.g., coding mode) and the division of the picture into blocks. Therefore, references to coding a picture or a block should generally be understood as coding values for the syntax elements that form the picture or block.

HEVC defines various blocks, including coding units (CUs), prediction units (PUs), and transformation units (TUs). According to HEVC, a video coder (such as video encoder 200) divides coding tree units (CTUs) into CUs according to a quadtree structure. That is, the video coder divides the CTUs and CUs into four equal, non-overlapping squares, and each node in the quadtree has either zero or four child nodes. Nodes without child nodes are sometimes called "leaf nodes," and the CU of such a leaf node may contain one or more PUs and/or one or more TUs. The video coder may further divide the PUs and TUs. For example, in HEVC, the residual quadtree (RQT) represents a division of TUs. In HEVC, PUs represent intra-predicted data, while TUs represent residual data. Intra-predicted CUs contain intra-predicted information, such as intra-mode indications.

As another example, the video encoder 200 and video decoder 300 may be configured to operate according to VVC. According to VVC, the video coder (such as the video encoder 200) divides a picture into multiple coding tree units (CTUs). The video encoder 200 may divide the CTUs according to a tree structure such as a quadtree-binary tree (QTBT) structure or a multi-type tree (MTT) structure. The QTBT structure eliminates the concept of multiple division types, such as the separation of CUs, PUs, and TUs in HEVC. The QTBT structure includes two levels: a first level divided according to quadtree divisions, and a second level divided according to binary tree divisions. The root node of the QTBT structure corresponds to a CTU. The leaf nodes of the binary tree correspond to coding units (CUs).

In MTT partitioning structures, blocks can be partitioned using quadtree (QT) partitions, binary tree (BT) partitions, and one or more types of triple tree (TT) partitions (also called ternary tree (TT) partitions). A triple tree or ternary tree partition is a partition in which a block is divided into three subblocks. In some examples, a triple tree or ternary tree partition divides a block into three subblocks without dividing the original block through a center. The partition types in MTT (e.g., QT, BT, and TT) can be symmetrical or asymmetrical.

In some examples, the video encoder 200 and video decoder 300 may use a single QTBT or MTT structure to represent the luminance and chromaticity components, respectively. In other examples, the video encoder 200 and video decoder 300 may use two or more QTBT or MTT structures, such as one QTBT/MTT structure for the luminance component and another QTBT/MTT structure for both chromaticity components (or two QTBT/MTT structures for each chromaticity component).

The video encoder 200 and video decoder 300 may be configured to use a quadtree partition, QTBT partition, MTT partition, or other partition structure according to HEVC. For illustrative purposes, the description of the techniques of this disclosure is presented in relation to QTBT partitions. However, it should be understood that the techniques of this disclosure may also be applicable to video coders configured to use quadtree partitions or other types of partitions.

In some examples, a CTU includes a coding tree block (CTB) of a luma sample, two corresponding CTBs of a chroma sample for a picture with three sample arrays, or a CTB of a sample for a picture coded using three separate color planes and syntax structures used to code a monochrome picture or sample. A CTB can be an N×N block of samples for some value N, such that the division of components into the CTB is a partition. A component is a single sample from one array or one of three arrays (luma and two chromas) that make up a picture in a 4:2:0, 4:2:2, or 4:4:4 color format, or a single sample from an array or array that makes up a picture in a monochrome format. In some examples, a coding block is an M×N block of samples for some values M and N, such that the division of the CTB into the coding block is a partition.

Blocks (e.g., CTUs or CUs) can be grouped in various ways within a picture. For example, a brick might refer to a rectangular area of CTU rows within a specific tile in a picture. A tile can be a rectangular area of CTU within a specific tile column or tile row in a picture. A tile column refers to a rectangular area of CTU with a height equal to the picture's height and a width specified by a syntax element (e.g., in a picture parameter set). A tile row refers to a rectangular area of CTU with a height specified by a syntax element (e.g., in a picture parameter set) and a width equal to the picture's width.

In some examples, a tile may be divided into multiple bricks, each brick containing one or more CTU rows within the tile. A tile that is not divided into multiple bricks may also be called a brick. However, a brick that is a true subset of a tile may not be called a tile.

Bricks within a picture can also be arranged in a slice. A slice can be an integer number of picture bricks that can exclusively reside within a single Network Abstraction Layer (NAL) unit. In some examples, a slice may contain either several complete tiles or only a series of consecutive complete bricks within a single tile.

This disclosure may use "N×N" and "N to N" interchangeably to refer to the sample dimensions of a block (such as a CU or other video block) with respect to vertical and horizontal dimensions, for example, 16×16 samples or 16 to 16 samples. Generally, a 16×16 CU has 16 samples vertically (y=16) and 16 samples horizontally (x=16). Similarly, an N×N CU generally has N samples vertically and N samples horizontally, where N represents a non-negative integer. The samples within a CU can be arranged as rows and columns. Furthermore, a CU does not necessarily have to have the same number of samples horizontally as vertically. For example, a CU may have N×M samples, where M is not necessarily equal to N.

The video encoder 200 encodes video data for the predictive block (CU), representing prediction information and/or residual information, as well as other information. The prediction information indicates how the CU should be predicted in order to form a prediction block for the CU. The residual information generally represents the sample-by-sample difference between the CU samples before encoding and the prediction block.

To predict the CU (Control Unit), the video encoder 200 can generally form prediction blocks for the CU through inter-prediction or intra-prediction. Inter-prediction generally refers to predicting the CU from data of a previously coded picture, while intra-prediction generally refers to predicting the CU from previously coded data of the same picture. To perform inter-prediction, the video encoder 200 can generate prediction blocks using one or more motion vectors. The video encoder 200 can generally perform motion search to identify a reference block that closely matches the CU, for example, with respect to the difference between the CU and the reference block. The video encoder 200 can calculate a difference metric using absolute difference sum (SAD), squared difference sum (SSD), mean absolute difference (MAD), mean squared difference (MSD), or other such difference calculations to determine whether the reference block closely matches the current CU. In some examples, the video encoder 200 can predict the current CU using unidirectional or bidirectional prediction.

Some examples of VVC also offer an affine motion compensation mode, which can be considered an interpredictive mode. In affine motion compensation mode, the video encoder 200 may determine two or more motion vectors representing non-translational motion, such as zooming in or out, rotation, projection, or other irregular motion types.

To perform intra-prediction, the video encoder 200 may select an intra-prediction mode to generate prediction blocks. Several examples of VVCs offer 67 intra-prediction modes, including various directional modes, as well as planar and DC modes. Generally, the video encoder 200 selects an intra-prediction mode that describes neighboring samples to the current block (e.g., a block of CUs) from which samples of the current block should be predicted. Such samples can generally be located above, above, and to the left of, or to the left of, the current block, in the same picture as the current block, assuming the video encoder 200 codes CTUs and CUs in raster scanning order (left to right, top to bottom).

The video encoder 200 encodes data representing the prediction mode for the current block. For example, in interprediction mode, the video encoder 200 may encode data representing which of the various available interprediction modes is used, as well as motion information for the corresponding mode. For unidirectional or bidirectional interprediction, for example, the video encoder 200 may encode motion vectors using advanced motion vector prediction (AMVP) mode or merge mode. The video encoder 200 may also encode motion vectors for affine motion compensation mode using similar modes.

According to predictions such as intra-prediction or inter-prediction of a block, the video encoder 200 may calculate residual data for the block. Residual data, such as residual blocks, represents the sample-by-sample difference between the block and the predicted block for the block, formed using the corresponding prediction mode. The video encoder 200 may apply one or more transformations to the residual blocks to generate transformed data in the transformation region instead of the sample region. For example, the video encoder 200 may apply a discrete cosine transform (DCT), integer transform, wavelet transform, or a conceptually similar transform to the residual video data. In addition, the video encoder 200 may apply secondary transformations such as mode-dependent non-separable secondary transform (MDNSST), signal-dependent transform, or Carunen-Löwe transform (KLT) following the initial transformation. The video encoder 200 generates transformation coefficients following the application of one or more transformations.

As described above, following any transformation to generate the transformation coefficients, the video encoder 200 may perform quantization of the transformation coefficients. Quantization generally refers to the process of quantizing the transformation coefficients to reduce the amount of data used to represent them as much as possible, thereby achieving further compression. By performing the quantization process, the video encoder 200 may reduce the bit depth associated with some or all of the transformation coefficients. For example, the video encoder 200 may truncate an n-bit value to an m-bit value during quantization, where n is greater than m. In some examples, to perform quantization, the video encoder 200 may perform a bitwise right shift of the value to be quantized.

Following quantization, the video encoder 200 may scan the transformation coefficients and generate a one-dimensional vector from a two-dimensional matrix containing the quantized transformation coefficients. The scanning may be designed to place higher-energy (and therefore lower-frequency) transformation coefficients at the beginning of the vector and lower-energy (and therefore higher-frequency) transformation coefficients at the end. In some examples, the video encoder 200 may generate a serialized vector using a predetermined scanning order to scan the quantized transformation coefficients, and then entropy-encode the quantized transformation coefficients of the vector. In other examples, the video encoder 200 may perform adaptive scanning. After scanning the quantized transformation coefficients to form a one-dimensional vector, the video encoder 200 may entropy-encode the one-dimensional vector, for example, according to context-adaptive binary arithmetic coding (CABAC). The video encoder 200 may also entropy-encode values for syntax elements describing metadata related to the encoded video data for use by the video decoder 300 when decoding the video data.

To perform CABAC, the video encoder 200 may assign a context within a context model to the symbols to be transmitted. This context may relate, for example, to whether the adjacent values of a symbol are 0. Probability decisions may be based on the context assigned to the symbols.

The video encoder 200 may further generate syntax data for the video decoder 300, such as block-based syntax data, picture-based syntax data, and sequence-based syntax data, for example, in the form of a picture header, block header, slice header, or other syntax data such as a sequence parameter set (SPS), picture parameter set (PPS), or video parameter set (VPS). The video decoder 300 may similarly decode such syntax data to determine how to decode the corresponding video data.

In this way, the video encoder 200 can generate a bitstream containing encoded video data, for example, syntax elements describing the division of a picture into blocks (e.g., CUs) and prediction and/or residual information for those blocks. Finally, the video decoder 300 can receive the bitstream and decode the encoded video data.

Generally, the video decoder 300 performs the reverse process of the process performed by the video encoder 200 to decode the encoded video data of the bitstream. For example, the video decoder 300 may decode values for syntax elements of the bitstream using CABAC in a substantially similar, but reverse, manner to the CABAC encoding process of the video encoder 200. The syntax elements may define the CUs of the CTUs by defining partitioning information for partitioning the picture into CTUs, and the partitions of each CTU according to a corresponding partitioning structure such as a QTBT structure. The syntax elements may further define prediction and residual information for blocks of video data (e.g., CUs).

Residual information can be represented, for example, by quantized transformation coefficients. The video decoder 300 may reconstruct the residual block for the block by inverse quantizing and inverse transforming the quantized transformation coefficients of the block. The video decoder 300 uses the signaled prediction mode (intra-prediction or inter-prediction) and associated prediction information (e.g., motion information for inter-prediction) to form a predicted block for the block. The video decoder 300 may then combine the predicted block and the residual block (sample by sample) to reconstruct the original block. The video decoder 300 may perform additional processing, such as deblocking, to reduce visual artifacts along the block boundaries.

According to the technique of this disclosure, the method comprises the steps of: applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector; and decoding the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of the block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of the block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

According to the technique of this disclosure, the device includes a memory configured to store video data and one or more processors implemented by circuitry and communicatively coupled to the memory, the one or more processors configured to apply a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector, wherein the multipath DMVR comprises a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

According to the technique of this disclosure, a non-temporary computer-readable medium stores instructions that, when executed, cause one or more processors to apply a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector. The multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

According to the technique of this disclosure, the device includes means for applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector, and means for decoding the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of the block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of the block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

According to the technique of this disclosure, the method includes the steps of: applying a multipath decoder-side motion vector improvement (DMVR) to the motion vector for a block of video data to determine an improved motion vector; and coding the block based on the improved motion vector.

According to the techniques of this disclosure, the device includes a memory configured to store video data and one or more processors that are circuit-implemented and communicatively coupled to the memory, the one or more processors being configured to perform any of the techniques of this disclosure.

According to the techniques of this disclosure, the device includes at least one means for performing any of the techniques of this disclosure.

According to the techniques of this disclosure, a computer-readable storage medium encodes instructions that, when executed, cause a programmable processor to perform one of the techniques of this disclosure.

This disclosure may refer in general to "signaling" any information, such as syntax elements. The term "signaling" may generally refer to the communication of values for syntax elements and/or other data used to decode the encoded video data. That is, the video encoder 200 may signal values for syntax elements in the bitstream. Generally, signaling refers to generating values in the bitstream. As stated above, the source device 102 may transfer the bitstream to the destination device 116 substantially in real time or non-real time, which may occur, for example, when the destination device 116 stores the syntax elements in the storage device 112 for later retrieval.

Figures 2A and 2B are conceptual diagrams showing an exemplary quadtree-binary tree (QTBT) structure 130 and its corresponding coding tree unit (CTU) 132. Solid lines represent quadtree partitions, and dotted lines represent binary tree partitions. At each partition (i.e., non-leaf) node of the binary tree, one flag is signaled to indicate which partition type (i.e., horizontal or vertical) is used; in this example, 0 indicates a horizontal partition and 1 indicates a vertical partition. In the case of a quadtree partition, there is no need to indicate the partition type, as the quadtree node divides the block horizontally and vertically into four subblocks of equal size. Therefore, the video encoder 200 may encode syntax elements (such as partition information) for the domain tree level (i.e., solid lines) of the QTBT structure 130, and the video decoder 300 may decode syntax elements (such as partition information) for the predictor tree level (i.e., dashed lines) of the QTBT structure 130. The video encoder 200 may encode video data, such as prediction data and transformation data, for CUs represented by the terminal leaf nodes of the QTBT structure 130, and the video decoder 300 may decode it.

Generally, CTU132 in Figure 2B can be associated with parameters that define the size of the blocks corresponding to the nodes of the QTBT structure 130 at the first and second levels. These parameters may include the CTU size (representing the size of CTU132 in the sample), the minimum quadtree size (MinQTSize, representing the minimum allowed quadtree leaf node size), the maximum binary tree size (MaxBTSize, representing the maximum allowed binary tree root node size), the maximum binary tree depth (MaxBTDepth, representing the maximum allowed binary tree depth), and the minimum binary tree size (MinBTSize, representing the minimum allowed binary tree leaf node size).

The root node of a QTBT structure corresponding to a CTU may have four child nodes at the first level of the QTBT structure, each of which may be partitioned according to a quadtree partition. That is, a node at the first level may have a leaf node (no child nodes) or four child nodes. An example of QTBT structure 130 represents such a node, including a parent node and child nodes with solid lines for branching. If a node at the first level is not larger than the maximum allowable binary tree root node size (MaxBTSize), the node may be further partitioned by its respective binary tree. Binary tree partitioning of a single node may be repeated until the node resulting from the partition reaches the minimum allowable binary tree leaf node size (MinBTSize) or the maximum allowable binary tree depth (MaxBTDepth). An example of QTBT structure 130 represents a node with dashed lines for branching. Binary tree leaf nodes are called coding units (CUs), which are used for prediction (e.g., in-picture or between-picture predictions) and transformations without further partitioning. As discussed above, CU can also be called "video block" or "block."

In an example of a QTBT partitioned structure, the CTU size is set to 128×128 (a chroma sample and two corresponding 64×64 chroma samples), MinQTSize is set to 16×16, MaxBTSize is set to 64×64, MinBTSize (for both width and height) is set to 4, and MaxBTDepth is set to 4. To generate a quadtree leaf node, the quadtree partition is first applied to the CTU. The quadtree leaf node can have sizes ranging from 16×16 (i.e., MinQTSize) to 128×128 (i.e., CTU size). If the quadtree leaf node is 128×128, the size exceeds MaxBTSize (i.e., 64×64 in this example), so the quadtree leaf node is not further partitioned by a binary tree. Otherwise, the quadtree leaf node is further partitioned by a binary tree. Therefore, a quadtree leaf node is also the root node of a binary tree and has a binary tree depth of 0. When the binary tree depth reaches MaxBTDepth (4 in this example), further partitioning is not permitted. A binary tree node with a width equal to MinBTSize (4 in this example) suggests that further vertical partitioning (i.e., partitioning by width) is not permitted for that binary tree node. Similarly, a binary tree node with a height equal to MinBTSize suggests that further horizontal partitioning (i.e., partitioning by height) is not permitted for that binary tree node. As mentioned above, a binary tree leaf node is called a CU and is further processed according to prediction and transformation without further partitioning.

Figure 3 is a block diagram showing an exemplary video encoder 200 capable of performing the techniques of this disclosure. Figure 3 is provided for illustrative purposes and should not be considered a limitation of the techniques as broadly illustrated and described herein. For illustrative purposes, this disclosure describes a video encoder 200 using VVC (ITU-T H.266) and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be performed by video encoding devices configured according to other video coding standards.

In the example in Figure 3, the video encoder 200 includes a video data memory 230, a mode selection unit 202, a residual generation unit 204, a conversion processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse conversion processing unit 212, a reconstruction unit 214, a filter unit 216, a decoded picture buffer (DPB) 218, and an entropy coding unit 220. Any or all of the video data memory 230, mode selection unit 202, residual generation unit 204, conversion processing unit 206, quantization unit 208, inverse quantization unit 210, inverse conversion processing unit 212, reconstruction unit 214, filter unit 216, DPB 218, and entropy coding unit 220 may be implemented in one or more processors or processing circuits. For example, the units of the video encoder 200 may be implemented as one or more circuits or logic elements as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video encoder 200 may include additional or alternative processors or processing circuits to perform these and other functions.

The video data memory 230 can store video data to be encoded by the components of the video encoder 200. The video encoder 200 may receive video data to be stored in the video data memory 230, for example, from a video source 104 (Figure 1). The DPB 218 may act as a reference picture memory, storing reference video data for use by the video encoder 200 in predicting subsequent video data. The video data memory 230 and DPB 218 may be formed by any of various memory devices, such as dynamic random access memory (DRAM) including synchronous DRAM (SDRAM), magnetoresistive RAM (MRAM), resistive RAM (RRAM), or other types of memory devices. The video data memory 230 and DPB 218 may be provided by the same memory device or by separate memory devices. In various examples, the video data memory 230 may be on-chip with the other components of the video encoder 200, as shown, or off-chip relative to those components.

In this disclosure, references to the video data memory 230 should not be interpreted as being limited to memory inside the video encoder 200 unless otherwise stated, nor should they be interpreted as being limited to memory outside the video encoder 200 unless otherwise stated. Rather, references to the video data memory 230 should be understood as reference memory that stores video data received by the video encoder 200 for encoding (e.g., video data for the current block to be encoded). Memory 106 in Figure 1 may also temporarily store outputs from various units of the video encoder 200.

The various units in Figure 3 are shown to help understand the operations performed by the video encoder 200. Units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Fixed-function circuits refer to circuits that provide a specific function, with predefined operations. Programmable circuits refer to circuits that can be programmed to perform various tasks, offering flexibility in the operations they can perform. For example, a programmable circuit may execute software or firmware that operates it, defined by software or firmware instructions. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more units may be integrated circuits.

The video encoder 200 may include a programmable core formed from an arithmetic logic unit (ALU), an basic function unit (EFU), digital circuits, analog circuits, and/or programmable circuits. In an example where the operation of the video encoder 200 is performed using software executed by the programmable circuits, memory 106 (Figure 1) may store instructions (e.g., object code) of the software that the video encoder 200 receives and executes, or another memory (not shown) within the video encoder 200 may store such instructions.

The video data memory 230 is configured to store the received video data. The video encoder 200 can retrieve the video data picture from the video data memory 230 and provide the video data to the residual generation unit 204 and the mode selection unit 202. The video data in the video data memory 230 may be raw video data to be encoded.

The mode selection unit 202 includes a motion estimation unit 222, a motion compensation unit 224, and an intra-prediction unit 226. The mode selection unit 202 may include additional functional units for performing video prediction according to other prediction modes. For example, the mode selection unit 202 may include a palette unit, an intra-block copy unit (which may be part of the motion estimation unit 222 and/or the motion compensation unit 224), an affine unit, a linear model (LM) unit, and so on.

The mode selection unit 202 generally coordinates multiple coding paths to test combinations of coding parameters and the resulting rate distortion values. Coding parameters may include the division of the CTU to the CU, the prediction mode for the CU, the transformation type for the residual data of the CU, and the quantization parameters for the residual data of the CU. The mode selection unit 202 can ultimately select a combination of coding parameters that yields a better rate distortion value than other tested combinations.

The video encoder 200 divides the picture retrieved from the video data memory 230 into a series of CTUs, and may encapsulate one or more CTUs within a slice. The mode selection unit 202 may divide the picture's CTUs according to a tree structure, such as a QTBT structure or the HEVC quadtree structure described above. As described above, the video encoder 200 may form one or more CUs from dividing the CTUs according to a tree structure. Such CUs may also be commonly referred to as “video blocks” or “blocks.”

Generally, the mode selection unit 202 also controls its components (e.g., motion estimation unit 222, motion compensation unit 224, and intra-prediction unit 226) to generate a predicted block for the current block (e.g., the current CU, or in HEVC, the overlapping portion of PU and TU). For intra-prediction of the current block, the motion estimation unit 222 may perform a motion search to identify one or more well-matching reference blocks among one or more reference pictures (e.g., one or more previously coded pictures stored in DPB 218). Specifically, the motion estimation unit 222 may calculate a value representing how similar a potential reference block is to the current block, for example, according to the sum of absolute differences (SAD), sum of squared differences (SSD), mean absolute difference (MAD), mean squared difference (MSD), etc. The motion estimation unit 222 may generally perform these calculations using the sample-by-sample difference between the current block and the reference block under consideration. The motion estimation unit 222 may identify the reference block with the lowest value resulting from these calculations, indicating the reference block that best matches the current block.

The motion estimation unit 222 may form one or more motion vectors (MVs) that define the position of a reference block in a reference picture relative to the position of the current block in the current picture. The motion estimation unit 222 may then provide the motion vectors to the motion compensation unit 224. For example, in the case of unidirectional interpretation, the motion estimation unit 222 may provide a single motion vector, while in the case of bidirectional interpretation, the motion estimation unit 222 may provide two motion vectors. The motion compensation unit 224 may then generate prediction blocks using the motion vectors. For example, the motion compensation unit 224 may use the motion vectors to extract data for the reference blocks. As another example, if the motion vectors have non-integer sample precision, the motion compensation unit 224 may interpolate values for the prediction blocks according to one or more interpolation filters. Furthermore, in the case of bidirectional interpretation, the motion compensation unit 224 may extract data for the two reference blocks identified by each motion vector and combine the extracted data, for example, through a sample-by-sample average or weighted average.

As another example, in the case of intra-prediction or intra-prediction coding, the intra-prediction unit 226 may generate a prediction block from samples adjacent to the current block. For example, in directional mode, the intra-prediction unit 226 may generally mathematically combine the values of adjacent samples and populate these calculated values in a direction defined across the current block to generate a prediction block. As another example, in DC mode, the intra-prediction unit 226 may calculate the average of the samples adjacent to the current block and generate a prediction block that includes this obtained average for each sample in the prediction block.

The mode selection unit 202 provides the prediction block to the residual generation unit 204. The residual generation unit 204 receives a raw, unencoded version of the current block from the video data memory 230 and the prediction block from the mode selection unit 202. The residual generation unit 204 calculates the sample-by-sample difference between the current block and the prediction block. The resulting sample-by-sample difference defines the residual block relative to the current block. In some examples, the residual generation unit 204 may also determine the differences between sample values within the residual block to generate the residual block using residual differential pulse code modulation (RDPCM). In some examples, the residual generation unit 204 may be formed using one or more subtractor circuits that perform binary subtraction.

In an example where the mode selection unit 202 divides CUs into PUs, each PU may be associated with a luma prediction unit and a corresponding chroma prediction unit. The video encoder 200 and video decoder 300 may support PUs of various sizes. As shown above, the size of a CU may refer to the size of the luma coding block of the CU, and the size of a PU may refer to the size of the luma prediction unit of the PU. Assuming a particular CU size is 2N×2N, the video encoder 200 may support PU sizes of 2N×2N or N×N for intra-prediction, and 2N×2N, 2N×N, N×2N, N×N, or similar symmetric PU sizes for inter-prediction. The video encoder 200 and video decoder 300 may also support asymmetric divisions for PU sizes of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-prediction.

In cases where the mode selection unit 202 does not further subdivide the CUs into PUs, each CU may be associated with a lumacoding block and a corresponding chromacoding block. As shown above, the size of the CU may refer to the size of the lumacoding block within the CU. The video encoder 200 and video decoder 300 may support CU sizes of 2N×2N, 2N×N, or N×2N.

In some examples, such as intra-block copy mode coding, affine mode coding, and other video coding techniques including linear model (LM) mode coding, the mode selection unit 202 generates a predicted block for the current block being coded via the respective units associated with the coding technique. In some examples, such as palette mode coding, the mode selection unit 202 may not generate a predicted block; instead, it may generate syntax elements indicating a scheme for reconstructing the block based on the selected palette. In such modes, the mode selection unit 202 may provide these syntax elements to the entropy coding unit 220 to be coded.

As described above, the residual generation unit 204 receives video data for the current block and the corresponding predicted block. The residual generation unit 204 then generates residual blocks for the current block. To generate the residual blocks, the residual generation unit 204 calculates the sample-by-sample difference between the predicted block and the current block.

The transformation processing unit 206 applies one or more transformations to the residual block to generate a block of transformation coefficients (referred to herein as the "transformation coefficient block"). The transformation processing unit 206 may apply various transformations to the residual block to form the transformation coefficient block. For example, the transformation processing unit 206 may apply a discrete cosine transform (DCT), a direction transform, a Carunenlebe transform (KLT), or a conceptually similar transformation to the residual block. In some examples, the transformation processing unit 206 may perform multiple transformations on the residual block, such as linear and quadratic transformations, including a rotation transform. In some examples, the transformation processing unit 206 does not apply any transformations to the residual block.

The quantization unit 208 can quantize the transformation coefficients within the transformation coefficient block to generate a quantized transformation coefficient block. The quantization unit 208 can quantize the transformation coefficients of the transformation coefficient block according to the quantization parameter (QP) value associated with the current block. The video encoder 200 can adjust the degree of quantization applied to the transformation coefficient block associated with the current block (for example, via the mode selection unit 202) by adjusting the QP value associated with the CU. Quantization may result in information loss; therefore, the quantized transformation coefficients may be less precise than the original transformation coefficients generated by the transformation processing unit 206.

The inverse quantization unit 210 and the inverse transform processing unit 212 can reconstruct the residual block from the quantized transform coefficient block by applying inverse quantization and inverse transform, respectively. The reconstruction unit 214 can generate a reconstructed block corresponding to the current block (which may be with some distortion) based on the reconstructed residual block and prediction block generated by the mode selection unit 202. For example, the reconstruction unit 214 can generate a reconstructed block by adding a sample from the reconstructed residual block to a corresponding sample from the prediction block generated by the mode selection unit 202.

The filter unit 216 may perform one or more filtering operations on the reconstructed block. For example, the filter unit 216 may perform a deblocking operation to reduce blockingness artifacts along the edges of the CU. In some examples, the operation of the filter unit 216 may be skipped.

The video encoder 200 stores the reconstructed blocks in the DPB 218. For example, in an example where the filter unit 216 does not operate, the reconstruction unit 214 may store the reconstructed blocks in the DPB 218. In an example where the filter unit 216 operates, the filter unit 216 may store the filtered reconstructed blocks in the DPB 218. The motion estimation unit 222 and the motion compensation unit 224 may retrieve a reference picture from the DPB 218, formed from the reconstructed (and possibly filtered) blocks, to inter-predict blocks of the picture to be encoded later. In addition, the intra-prediction unit 226 may use the reconstructed blocks in the DPB 218 of the current picture to intra-predict other blocks in the current picture.

In general, the entropy coding unit 220 can entropy code syntax elements received from other functional components of the video encoder 200. For example, the entropy coding unit 220 can entropy code a quantized conversion coefficient block from the quantization unit 208. As another example, the entropy coding unit 220 can entropy code prediction syntax elements from the mode selection unit 202 (e.g., motion information for inter-prediction or intra-mode information for intra-prediction). The entropy coding unit 220 can perform one or more entropy coding operations on syntax elements, which are another example of video data, to generate entropy-coded data. For example, the entropy coding unit 220 may perform context-adaptive variable-length coding (CAVLC), CABAC, variable-length to variable-length (V2V) coding, syntax-based context-adaptive binary arithmetic coding (SBAC), probability interval partitioned entropy (PIPE) coding, exponential Golomb coding, or another type of entropy coding on the data. In some examples, the entropy coding unit 220 may operate in a bypass mode where syntax elements are not entropically coded.

The video encoder 200 may output a bitstream containing entropy-encoded syntax elements necessary for reconstructing slices or blocks of pictures. Specifically, the entropy encoding unit 220 may output the bitstream.

The behavior described above is explained in relation to blocks. Such explanations should be understood as behaviors for luma-coding blocks and/or chroma-coding blocks. As described above, in some examples, the luma-coding block and chroma-coding block are the luma and chroma components of the CU. In some examples, the luma-coding block and chroma-coding block are the luma and chroma components of the PU.

In some cases, actions performed for a lumacoding block do not need to be repeated for a chromacoding block. For example, actions to identify the motion vector (MV) and reference picture for a lumacoding block do not need to be repeated to identify the MV and reference picture for a chromablock. Rather, the MV for the lumacoding block may be scaled to determine the MV for the chromablock, and the reference picture may be the same. Another example is that the intra-prediction process may be the same for both lumacoding and chromacoding blocks.

Figure 4 is a block diagram showing an exemplary video decoder 300 capable of performing the techniques of this disclosure. Figure 4 is provided for illustrative purposes and is not intended to limit the techniques that are broadly illustrated and described in this disclosure. For illustrative purposes, this disclosure describes a video decoder 300 using VVC (ITU-T H.266) and HEVC (ITU-T H.265) techniques. However, the techniques of this disclosure may be performed by video coding devices configured according to other video coding standards.

In the example shown in Figure 4, the video decoder 300 includes a coding picture buffer (CPB) memory 320, an entropy decoding unit 302, a prediction processing unit 304, an inverse quantization unit 306, an inverse transformation processing unit 308, a reconstruction unit 310, a filter unit 312, and a decoding picture buffer (DPB) 314. Any or all of the CPB memory 320, entropy decoding unit 302, prediction processing unit 304, inverse quantization unit 306, inverse transformation processing unit 308, reconstruction unit 310, filter unit 312, and DPB 314 may be implemented in one or more processors or processing circuits. For example, the units of the video decoder 300 may be implemented as one or more circuits or logic elements as part of a hardware circuit, or as part of a processor, ASIC, or FPGA. Furthermore, the video decoder 300 may include additional or alternative processors or processing circuits to perform these and other functions.

The prediction processing unit 304 includes a motion compensation unit 316 and an intra-prediction unit 318. The prediction processing unit 304 may include additional units for performing predictions according to other prediction modes. For example, the prediction processing unit 304 may include a pallet unit, an intra-block copy unit (which may form part of the motion compensation unit 316), an affine unit, a linear model (LM) unit, etc. In other examples, the video decoder 300 may include more, fewer, or different functional components. The motion compensation unit 316 may include a multipath DMVR unit (MPDMVR) 317, which will be discussed in the following discussion of the motion compensation unit 316.

The CPB memory 320 may store video data, such as an encoded video bitstream, to be decoded by the components of the video decoder 300. Video data stored in the CPU memory 320 may be obtained, for example, from a computer-readable medium 110 (Figure 1). The CPU memory 320 may include a CPB that stores encoded video data (e.g., syntax elements) from the encoded video bitstream. The CPB memory 320 may also store video data other than syntax elements of the coded picture, such as temporary data representing outputs from various units of the video decoder 300. The DPB 314 generally stores the decoded picture, and the video decoder 300 may output and/or use this decoded picture as reference video data when decoding subsequent data or pictures from the encoded video bitstream. The CPB memory 320 and DPB 314 may be formed by any of various memory devices, such as DRAM, MRAM, RRAM, or other types of memory devices, including SDRAM. The CPU memory 320 and DPB 314 may be provided by the same memory device or by separate memory devices. In various examples, the CPB memory 320 may be on-chip with the other components of the video decoder 300, or it may be off-chip relative to those components.

As an addition or alternative, in some examples, the video decoder 300 may retrieve coded video data from memory 120 (Figure 1). That is, memory 120 may store data such as those discussed above, together with the CPB memory 320. Similarly, memory 120 may store instructions to be executed by the video decoder 300 when some or all of the functions of the video decoder 300 are implemented in software to be performed by the processing circuit of the video decoder 300.

The various units shown in Figure 4 are presented to help understand the operations performed by the video decoder 300. The units can be implemented as fixed-function circuits, programmable circuits, or a combination thereof. Similar to Figure 3, fixed-function circuits refer to circuits that provide a specific function, with predefined operations that can be performed. Programmable circuits refer to circuits that can be programmed to perform various tasks, offering flexibility in the operations they can perform. For example, a programmable circuit may execute software or firmware that operates it, defined by software or firmware instructions. Fixed-function circuits may execute software instructions (e.g., to receive or output parameters), but the type of operation performed by a fixed-function circuit is generally immutable. In some examples, one or more of the units may be different circuit blocks (fixed-function or programmable), and in some examples, one or more of the units may be integrated circuits.

The video decoder 300 may include a programmable core formed from an ALU, EFU, digital circuitry, analog circuitry, and/or programmable circuitry. In an example where the operation of the video decoder 300 is performed by software running on the programmable circuitry, on-chip or off-chip memory may store software instructions (e.g., object code) that the video decoder 300 receives and executes.

The entropy decoding unit 302 can receive video data encoded from the CPB and reconstruct the syntax elements by entropy decoding the video data. The prediction processing unit 304, inverse quantization unit 306, inverse transformation processing unit 308, reconstruction unit 310, and filter unit 312 can generate the decoded video data based on the syntax elements extracted from the bitstream.

Generally, the video decoder 300 reconstructs the picture block by block. The video decoder 300 can perform the reconstruction operation for each block individually (where the block currently being reconstructed, i.e., decoded, may be called the "current block").

The entropy decoding unit 302 can entropy decode the syntax elements that define the quantized transformation coefficients of the quantized transformation coefficient block, as well as transformation information such as quantization parameters (QP) and/or transformation mode indications. The inverse quantization unit 306 may use the QP associated with the quantized transformation coefficient block to determine the degree of quantization and, similarly, the degree of inverse quantization that the inverse quantization unit 306 should apply. The inverse quantization unit 306 may, for example, perform a bitwise left shift operation to inverse quantize the quantized transformation coefficients. Thus, the inverse quantization unit 306 can form a transformation coefficient block containing the transformation coefficients.

After the inverse quantization unit 306 forms a transformation coefficient block, the inverse transformation processing unit 308 may apply one or more inverse transformations to the transformation coefficient block to generate a residual block associated with the current block. For example, the inverse transformation processing unit 308 may apply an inverse DCT, an inverse integer transformation, an inverse Carunenlebe transformation (KLT), an inverse rotation transformation, an inverse direction transformation, or another inverse transformation to the transformation coefficient block.

Furthermore, the prediction processing unit 304 generates prediction blocks according to the prediction information syntax elements entropy-decoded by the entropy decoding unit 302. For example, if the prediction information syntax elements indicate that the current block is interpredicted, the motion compensation unit 316 may generate a prediction block. In this case, the prediction information syntax elements may indicate the reference picture in the DPB 314 from which the reference block should be extracted, as well as a motion vector that identifies the position of the reference block in the reference picture relative to the position of the current block in the current picture. The motion compensation unit 316 can generally perform interprediction processing in substantially the same manner as described with respect to the motion compensation unit 224 (Figure 3).

In some examples, the motion compensation unit 316 may include a multi-pass DMVR unit 317. The multi-pass DMVR unit 317 may apply multi-pass DMVR to the motion vectors for blocks of video data to determine improved motion vectors. The multi-pass DMVR may include a first pass, which is block-based and applied to blocks of video data. The multi-pass DMVR may include a second pass, which is sub-block-based and applied to at least one second-pass sub-block of the block of video data. The multi-pass DMVR may include a third pass, which is sub-block-based and applied to at least one third-pass sub-block of the block of video data. The width of the second-pass sub-block may be less than or equal to the width of the block of video data, and the height of the second-pass sub-block may be less than or equal to the height of the block of video data. The width of the third-pass sub-block may be less than or equal to the width of the second-pass sub-block, and the height of the third-pass sub-block may be less than or equal to the height of the second-pass sub-block. Further examples and explanations of multi-pass DMVR techniques are described later in this disclosure.

As another example, if the prediction information syntax element indicates that the current block is intra-predicted, the intra-prediction unit 318 may generate a predicted block according to the intra-prediction mode indicated by the prediction information syntax element. Again, the intra-prediction unit 318 may generally perform intra-prediction processing in substantially the same manner as described with respect to the intra-prediction unit 226 (Figure 3). The intra-prediction unit 318 may retrieve data from the DPB 314 for samples adjacent to the current block.

The reconstruction unit 310 can reconstruct the current block using the predicted block and the residual block. For example, the reconstruction unit 310 can reconstruct the current block by adding samples from the residual block to the corresponding samples from the predicted block.

The filter unit 312 may perform one or more filtering operations on the reconstructed block. For example, the filter unit 312 may perform a deblocking operation to reduce blockingness artifacts along the edges of the reconstructed block. In all examples, the filter unit 312 may not necessarily perform any operations.

The video decoder 300 may store the reconstructed blocks in the DPB 314. For example, in an example where the filter unit 312 does not operate, the reconstruction unit 310 may store the reconstructed blocks in the DPB 314. In an example where the filter unit 312 operates, the filter unit 312 may store the filtered reconstructed blocks in the DPB 314. As discussed above, the DPB 314 may provide the prediction processing unit 304 with reference information such as the current picture for intra-prediction and samples of previously decoded pictures for subsequent motion compensation. Furthermore, the video decoder 300 may output the decoded picture (e.g., decoded video) from the DPB 314 for later presentation on a display device such as the display device 118 in Figure 1.

Thus, the video decoder 300 represents an example of a video decoding device, comprising a memory configured to store video data and one or more processors implemented in circuitry and communicatively coupled to the memory, wherein the one or more processors are configured to apply a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector, the multipath DMVR comprising: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

The video decoder 300 also represents an example of a video decoding device, comprising a memory configured to store video data and one or more processing units implemented in the circuit, the processing units configured to apply multipath decoder-side motion vector improvement (DMVR) to the motion vectors for blocks of video data to determine improved motion vectors, and to decode the blocks based on the improved motion vectors.

This disclosure relates to decoder-side motion vector derivation techniques (e.g., template matching, bilateral matching, decoder-side MV improvements, bidirectional optical flow, etc.). The techniques described herein may be applied to any existing video codec such as HEVC (High Efficiency Video Coding), VVC (Versatile Video Coding), and Essential Video Coding (EVC), or may serve as efficient coding tools in any future video coding standard. This section first considers HEVC and JEM techniques, as well as ongoing work on Versatile Video Coding (VVC) related to this disclosure.

Video coding standards include ITU-T H.261, ISO/IEC MPEG-1 Visual, ITU-T H.262 or ISO/IEC MPEG-2 Visual, ITU-T H.263, ISO/IEC MPEG-4 Visual, and ITU-T H.264 (also known as ISO/IEC MPEG-4 AVC), including their Scalable Video Coding (SVC) and Multi-view Video Coding (MVC) extensions. The algorithmic description of Versatile Video Coding and Test Model 10 (VTM 10.0) may be referred to as JVET-T2002, available from https://jvet-experts.org/.

The CU structure and motion vector prediction in HEVC are discussed here. In HEVC, the largest coding unit in a slice is called a coding tree block (CTB) or coding tree unit (CTU). A CTB may contain a quadtree whose nodes are coding units.

(Technically, an 8x8 CTB size may be supported.) CTB sizes in HEVC main profiles can range from 16x16 to 64x64. Coding units (CUs) can range in size from the same size as the CTB to as small as 8x8. Each CU is coded using one mode, i.e., intermode or intramode. When a CU is intercoded, it may be further divided into two or four predictive units (PUs), or it may remain as a single PU if no further division is applied. When two PUs exist within a single CU, the two PUs may each be a rectangle half the size of the CU (half the size of the CU), or they may be two rectangles, one one being 1/4 the size of the CU and the other 3/4 the size of the CU.

When a CU is intercoded, each PU possesses a set of motion information, which is derived using a unique inter-prediction mode.

Here, motion vector prediction is discussed. The HEVC standard provides two interpretation modes for the PU: merge mode (skipping is considered a special case of merging) and advanced motion vector prediction (AMVP) mode.

In either AMVP mode or merge mode, a motion vector (MV) candidate list is maintained for multiple motion vector predictors. Currently, the MV, and the reference index in merge mode, are generated by selecting one candidate from the MV candidate list. For example, the video decoder 300 may maintain an MV candidate list.

The MV candidate list contains up to five candidates for merge mode and only two candidates for AMVP mode. A merge candidate may contain MVs corresponding to both a set of motion information, such as a reference picture list (lists 0 and 1) and its corresponding reference index. When a merge candidate is identified by its merge index, the reference picture used for the current block prediction, as well as the associated motion vector, are determined. On the other hand, in AMVP mode for each potential prediction direction from either list 0 or list 1, the AMVP candidate contains only MVs, so the video encoder 200 may explicitly signal the reference index along with the MV predictor (MVP) index to the MV candidate list. In AMVP mode, the predicted MV can be further refined.

The video decoder 300 can similarly derive candidates for both modes from the same spatial and temporally adjacent blocks.

Figures 5A and 5B are conceptual diagrams showing exemplary spatially adjacent MV candidates for merge mode and AMVP mode, respectively. Spatial MV candidates are derived from the adjacent blocks shown in Figures 5A and 5B for a given PU (PU0), but the method of generating candidates from the blocks differs between merge mode and AMVP mode.

In merge mode, up to four spatial MV candidates for PU0 500 can be derived in the order shown in Figure 5A, where the numbers increase as follows: left (0, A1), top (1, B1), top right (2, B0), bottom left (3, A0), and top left (4, B2). For example, video decoder 300 can derive up to four spatial MV candidates for PU0 500 using the order described above.

As shown in Figure 5B, in AMVP mode, the adjacent blocks of PU0 502 are divided into two groups: the left group consisting of blocks 0 and 1, and the upper group consisting of blocks 2, 3, and 4. For example, the video decoder 300 may divide adjacent blocks into the left and upper groups. For each group, potential candidates among adjacent blocks that reference the same reference picture as the reference picture indicated by the signaled reference index have the highest priority when selected to form the final candidate for the group. It is possible that not all adjacent blocks contain motion vectors pointing to the same reference picture. Therefore, if no such candidate can be found, the difference in time distance can be compensated for by scaling the first available candidate to form the final candidate.

Time-motion vector prediction in HEVC is discussed here. The video decoder 300 may add time-motion vector predictor (TMVP) candidates to the MV candidate list after any spatial motion vector candidates, if valid and available. The motion vector derivation process for TMVP candidates is the same for both merge mode and AMVP mode. However, the target reference index for TMVP candidates in merge mode can always be set to 0.

Figures 6A and 6B are conceptual diagrams illustrating exemplary TMVP candidates and MV scaling, respectively. The primary block location for TMVP candidate derivation is the lower-right block outside the same-position PU, shown in Figure 6A as block "T" 600, to compensate for biases to the upper and left blocks used to generate spatial adjacency candidates. However, if the block is currently located outside the row of the CTB (shown as block 602), or if motion information is unavailable, the block is replaced by the central block 604 of PU0 606.

The motion vector for the TMVP candidate is derived from the co-position PU of the co-position picture, shown at the slice level. The motion vector of the co-position PU is called the co-position MV.

Similar to the time-direct mode in AVC, in order to derive TMVP candidates, the same-position MV610 needs to be scaled to compensate for the time-distance difference, as shown in Figure 6B. For example, the video decoder 300 may scale the same-position MV610 to compensate for the time-distance difference.

Other aspects of motion prediction in HEVC are discussed here. Several aspects of merge mode and AMVP mode are worth mentioning, as follows:

Motion Vector Scaling: The MV value is proportional to the distance between pictures at presentation time. The MV relates two pictures: a reference picture and a picture containing the motion vector (e.g., a containing picture or a picture containing the block predicted using the motion vector). When an MV is used to predict another MV, the time distance between the containing picture and the reference picture is calculated based on the Picture Order Count (POC) value.

For a motion vector to be predicted, both the associated containing picture and the reference picture may differ. Therefore, a new distance (based on POC) is calculated. The MV is scaled based on these two POC distances. For example, the video decoder 300 may calculate the new distance based on the POC, or it may scale the MV based on the two POC distances. For a spatially adjacent candidate, the containing pictures of the two MVs may be the same, but the reference pictures may be different. In HEVC, MV scaling applies to both TMVP and AMVP for both spatially and temporally adjacent candidates.

Generation of artificial motion vector candidates: If the MV candidate list is incomplete, artificial MV candidates may be generated and inserted at the end of the list until the list contains all candidates (e.g., the list is filled).

In merge mode, there are two types of artificial MV candidates: synthetic candidates derived solely for B-slice, and zero candidates used only for AMVP when the first type does not provide sufficient artificial candidates.

For each pair of candidates already present in the candidate list and possessing the necessary motion information, a bidirectional composite MV candidate is derived by combining the MV of the first candidate referencing a picture in List 0 with the MV of the second candidate referencing a picture in List 1.

Pruning Process for Candidate Insertion: Candidates from different blocks may coincidentally be the same, which reduces the efficiency of the merge/AMVP candidate list. A pruning process may be applied to solve this problem. During the pruning process, the video decoder 300 compares a candidate to other candidates in the current candidate list to some extent to avoid inserting identical candidates. To reduce complexity, the pruning process may be applied to a limited number of candidates rather than comparing each possible candidate to all other existing candidates.

Template matching prediction is discussed here. Template matching (TM) prediction is a special merge mode based on the Frame-Rate Up Conversion (FRUC) technique. In this TM prediction mode, block motion information is not signaled but is derived on the decoder side by the video decoder 300. TM prediction is applied to both AMVP mode and normal merge mode. In AMVP mode, the selection of the MVP candidate is determined using basic template matching to select the candidate with the smallest difference between the current block template and the reference block template. In normal merge mode, the video encoder 200 signals a TM mode flag to indicate the use of TM, and then TM is applied to the merge candidate indicated by the merge index for MV improvement.

Figure 7 is a conceptual diagram illustrating exemplary template matching in the search area around the initial MV. As shown in Figure 7, template matching may be used to derive motion information for the current CU. Deriving motion information may involve finding the best match between template 700 in current picture 702 (the adjacent block above and/or to the left of the current CU) and block 704 in reference picture 706 (for example, the same size as the template). Using AMVP candidates selected based on the initial matching error, the candidate MVP is refined by template matching. Using merge candidates indicated by signaled merge indices, the merged MVs of candidates corresponding to reference picture list0 (L0) and reference picture list1 (L1) are refined independently by template matching, and then the less accurate MVs are further refined again using more accurate MVs as previous references. For example, video decoder 300 may receive and parse signaled merge indices and apply template matching to the merged MVs to refine them.

Cost function: When the motion vector points to a non-integer sample position, the video decoder 300 may use motion-compensated interpolation. To reduce complexity, bilinear interpolation is used for both template matching and generating the template on the reference picture, instead of the usual 8-tap discrete cosine transform-interpolation filter (DCT-IF) interpolation. The matching cost C for template matching can be calculated as follows:
C=SAD+w・(|MV _x -MV _x ^s |+|MV _y -MV _y ^s |)
Here, w is a weighting coefficient empirically set to 4, and MV and ^MVs represent the current MV under test and the initial MV (e.g., the MVP candidate in AMVP mode or the merged motion vector in merge mode), respectively. The absolute difference sum (SAD) can be used as the matching cost for template matching.

When motion correction (TM) is used, motion is improved by using only luma samples. The derived motion is used for both luma and chroma for motion compensation (MC) interpretation. After the motion value (MV) is determined, the final MC is performed using an 8-tap interpolation filter for luma and a 4-tap interpolation filter for chroma. For example, video decoder 300 may improve motion using only luma samples.

Search Method: MV refinement can be a pattern-based MV search using a template matching cost criterion. Two search patterns are supported for MV refinement: diamond search and cross search. For example, video decoder 300 may use either diamond search or cross search for MV refinement. The MV is directly searched using the diamond pattern with accuracy of 1/4 lumasample motion vector difference (MVD), then using the cross pattern with accuracy of 1/4 lumasample MVD, and subsequently refined to 1/8 lumasample MVD using the cross pattern. The search range for MV refinement can be set to equal (-8, +8) lumasamples around the initial MV.

Bilateral matching prediction is discussed here. Bilateral matching (also known as bilateral merging) (BM) prediction is another merge mode based on the FRUC technique. Once a decision is made to apply the BM mode to a given block, two initial MVs (MV0 and MV1) are derived by using signaled merge candidate indices to select merge candidates from a constructed merge list. The video decoder 300 can perform a bilateral matching search around MV0 and MV1 and derive the final MV0' and MV1' based on the minimum bilateral matching cost.

Figures 8A and 8B are conceptual diagrams illustrating examples where MVD0 and MVD1 are proportional based on time distance, and examples where MVD0 and MVD1 are mirror images regardless of time distance, respectively. The motion vector differences MVD0 (denoted by MV0'-MV0) and MVD1 (denoted by MV1'-MV1) pointing to two reference blocks can be proportional to the time distance (TD) between picture 804 and the two reference pictures 806 and 808, for example, TD0 800 and TD1 802. Figure 8A shows an example of MVD0 and MVD1, where TD1 802 is four times TD0 800.

However, there is an optional design in which MVD0 and MVD1 are mirror images of each other regardless of the time distance TD0 and TD1. Figure 8B shows an example of mirror images of MVD0 and MVD1, where TD1 812 is four times TD0 810.

Figure 9 is a conceptual diagram showing an example of a 3x3 square search pattern in the search range [-8,8]. Bilateral matching may involve performing a local search around the initial MV0 and MV1 to derive the final MV0' and MV1'. To apply the local search, the video decoder 300 may apply a 3x3 square search pattern, looping through the search range [-8,8]. In each iteration of the search, the bilateral matching costs of the eight surrounding MVs in the search pattern are calculated and compared to the bilateral matching cost of the central MV. The MV with the smallest bilateral matching cost becomes the new central MV in the next iteration of the search. The local search terminates when the current central MV has the smallest cost within the 3x3 square search pattern, or when the local search reaches a predetermined maximum number of iterations. For example, the video decoder 300 may perform bilateral matching as described herein. In the example in Figure 9, the initial MV900 is used, and a 3x3 search pattern 902 is explored around the initial MV900. In the first iteration, the MV with the lowest cost among the initial 8 MVs is MV904. In the second iteration, the video decoder 300 then repeats the search pattern 902 around MV904. In this example, the MV finally selected after N iterations is MV906.

Figure 10 is a conceptual diagram illustrating an exemplary decoder-side motion vector improvement. To improve the accuracy of the merge-mode MV, decoder-side motion vector improvement (DMVR) can be applied, as shown in VVD Draft 10. For example, video decoder 300 may apply DMVR. In the biprediction operation, the improved MV is searched around the initial MV in reference picture list0 (L0) and reference picture list1 (L1). The DMVR method calculates the strain between two candidate blocks in L0 and L1. For example, video decoder 300 may calculate the strain between two candidate blocks. As shown in Figure 10, the SAD between blocks 1000 and 1002 is calculated based on each MV candidate around the initial MV. For example, video decoder 300 may determine the SAD between blocks 1000 and 1002. The MV candidate with the lowest SAD becomes the improved MV and is used to generate the bipredicted signal.

The improved MV derived by the DMVR technique is used to generate interpredictive samples and is also used in time motion vector prediction for coding future pictures. The video decoder 300 can use the original MV in the deblocking process and in spatial motion vector prediction for coding future CUs.

The DMVR in VVC Draft 10 is a subblock-based merge mode that uses a predetermined maximum PU of 16x16 lumasamples. When the width and/or height of a CU is greater than 16 lumasamples, the CU may be further divided into subblocks with widths and/or heights equal to 16 lumasamples. For example, the video decoder 300 may further divide larger CUs into subblocks with widths and/or heights equal to 16 lumasamples.

An exemplary search scheme is discussed here. In DVMR, the search points and MV offsets around the initial MV follow the mirroring rule of the MV difference discussed above. In other words, any point identified by the video decoder 300 implementing the DMVR, denoted by a candidate MV pair (MV0, MV1), follows the following two equations:
MV0' = MV0 + MV_offset
MV1' = MV1 - MV_offset
Here, MV_offset represents the refinement offset between the initial MV and the refined MV in one of the reference pictures. The refinement search range is two integer samples from the initial MV. This search includes an integer sample offset search phase and a non-integer sample refinement phase.

For integer sample offset search, a full 25-point search may be applied. For example, video decoder 300 may perform a full 25-point search. The SAD of the initial MV pair is calculated first. If the SAD of the initial MV pair is less than the threshold, the integer sample stage of DMVR terminates. Otherwise, the SADs of the remaining 24 points are calculated and verified in raster scan order. The point with the minimum SAD is selected as the output of the integer sample offset search stage. To mitigate the disadvantage of uncertainty in DMVR refinement, the original MV may be preferred during the DMVR process. The SAD between reference blocks referenced by the initial MV candidates may be reduced by 1/4 of the SAD value.

Following integer sample search, there is a non-integer sample refinement. For example, the video decoder 300 may perform an integer sample search and then a non-integer sample refinement. To reduce computational complexity, the non-integer sample refinement may be derived by using a parametric error surface equation rather than an additional search using SAD comparison. The non-integer sample refinement is conditionally invoked based on the output of the integer sample search stage. If the integer sample search stage terminates with the center having the smallest SAD in either the first or second iterative search, the non-integer sample refinement is further applied.

In parametric error surface-based subpixel offset estimation, the central position cost and the costs at four adjacent positions from the center are used to fit a two-dimensional (2-D) parabolic error surface equation of the following form:
E(x,y)=A(xx _min ) ² +B(yy _min ) ² +C
Here, (x _min , y _min ) corresponds to the non-integer position with the minimum cost, A and B are constants, and C corresponds to the minimum cost value. By solving the above equation using the cost values of the five search points, the position of the minimum value (x _min , y _min ) is calculated as follows:
x _min =(E(-1,0)-E(1,0))/(2(E(-1,0)+E(1,0)-2E(0,0)))
y _min =(E(0,-1)-E(0,1))/(2((E(0,-1)+E(0,1)-2E(0,0)))

Since all cost values are integers and the smallest value is E(0,0), the values of x _min and y _min are automatically constrained between -8 and 8. This corresponds to a half-pel offset, using the 1/16th pel accuracy of the MV in VVC Draft 10. To obtain the accurate improved delta MV for subpixels, the calculated non-integer (x _min , y _min ) is added to the integer distance improved MV.

Bilinear interpolation and sample padding are discussed here. These techniques can be applied by the video decoder 300. In VVC Draft 10, the resolution of the MV is 1/16 luma samples. Samples at non-integer positions can be interpolated using an 8-tap interpolation filter. In DMVR, the search points surround the initial non-integer phase MV with an offset of integer samples. Therefore, samples at those non-integer positions need to be interpolated for DMVR search. To reduce computational complexity, a bilinear interpolation filter is used to generate non-integer samples for search in DMVR. Another effect is that by using a bilinear filter with a 2-sample search range, DVMR accesses fewer reference samples compared to a normal motion compensation process. After the improved MV is obtained using DMVR search, a normal 8-tap interpolation filter is applied to generate the final prediction. To avoid accessing more reference samples than a normal motion compensation process, samples not required for the interpolation process based on the original MV but required for the interpolation process based on the improved MV are padded from the available samples.

Exemplary activation conditions for DMVR are discussed here. For example, DMVR may be enabled if all of the following conditions are met: 1) CU-level merge mode is used with bi-prediction MV; 2) with respect to the current picture, one reference picture is in the past and another reference picture is in the future; 3) the distance from both reference pictures to the current picture (e.g., POC difference) is the same; 4) the CU has more than 64 luma samples; 5) both the height and width of the CU are greater than or equal to 8 luma samples; 6) the bi-prediction with CU weights (BCW) weight indices show equal weights; 7) weighted prediction (WP) is not enabled for the current block; and 8) the Combined inter-intra prediction (CIIP) mode is not used for the current block.

Bidirectional optical flow is discussed here. The video decoder 300 may use bidirectional optical flow (BDOF) to improve the bi-predicted signals of luma samples in the CU at the 4x4 subblock level. As its name suggests, the BDOF mode is based on the concept of optical flow, which assumes that the motion of the object is smooth. For each 4x4 subblock, motion improvement (v _x , v _y ) is calculated by minimizing the difference between the L0 predicted sample and the L1 predicted sample. The motion improvement is then used to adjust the bi-predicted sample values in the 4x4 subblock. The following steps are applied in the BDOF process.

First, the horizontal and vertical gradients of the two prediction signals,
and
k=0,1 is calculated by directly calculating the difference between two adjacent samples, for example,
Here, I ^(k) (i,j) is the sample value at coordinate (i,j) of the predicted signal in list k, where k=0,1 and shift1 is calculated based on the luma bit depth bitDepth, and shift1 is set to equal to 6.

Next, the autocorrelation and crosscorrelation of gradients _S1 , _S2 , _S3 , _S5 , and _S6 are calculated as follows:
S ₁ =Σ _(i,j)∈Ω │ψ _x (i,j)│, S ₃ =Σ _(i,j)∈Ω θ(i,j)・(-sign(ψ _x (i,j)))
S ₂ =Σ _(i,j)∈Ω ψ _x (i,j)・sign(ψ _y (i,j))
S ₅ =Σ _(i,j)∈Ω │ψ _y (i,j)│ S ₆ =Σ _(i,j)∈Ω θ(i,j)・(-sign(ψ _y (i,j)))

Here
θ(i,j)=(I ⁽⁰⁾ (i,j)≫shift2)-(I ⁽¹⁾ (i,j)≫shift2)
Therefore, Ω is a 6x6 window around a 4x4 subblock, the value of shift2 is set to equal to 4, and the value of shift3 is set to equal to 1.

The motion improvement (v _x , v _y ) is then derived using the cross-correlation and autocorrelation terms, as follows:
Here,
This is the floor function.

Based on motion improvements and gradients, the following adjustments are calculated for each sample in the 4x4 subblock.
Finally, the BDOF sample of CU is calculated by adjusting the biprediction sample as follows:
pred _BDOF (x,y)=(I ⁽⁰⁾ (x,y)+I ⁽¹⁾ (x,y)+b(x,y)+ο _offset )≫shift5

Here, shift5 is Max(3, 15-BitDepth), and the variable o _offset is set to equal (1<<(shift5-1)).

These values are selected so that the multiplier in the BDOF process does not exceed 15 bits, and the maximum bit width of the intermediate parameters in the BDOF process is kept within 32 bits.

Figure 11 is a conceptual diagram showing an exemplary extended CU region used in BDOF. To derive gradient values, it may be necessary to generate some predicted samples I ^(k) (i,j) in the list k(k=0,1) that are currently outside the CU boundary. As shown in Figure 11, BDOF uses one extended row/column around the boundary of CU1100. To control the computational complexity of generating out-of-bounds predicted samples, predicted samples within the extended area (e.g., outermost positions) are generated by directly taking reference samples at nearby integer positions without interpolation (using the floor() operation on the coordinates), and a standard 8-tap motion-compensated interpolation filter is used to generate predicted samples within CU1100 (e.g., diagonal or patterned positions within CU1100). These extended sample values may only be used in gradient calculations. If any sample values and gradient values outside the boundary of CU1100 are needed for the remaining steps in the BDOF process, they may be padded (e.g., iterated) from the nearest neighbor.

BDOF is used to improve the biprediction signal of a CU at the 4x4 subblock level (e.g., subblock 1102). For example, BDOF may be applied to a CU if it satisfies all of the following conditions: 1) the CU is coded using a "true" biprediction mode, e.g., one of two reference pictures is before the current picture in display order and the other is after the current picture in display order; 2) the CU is not coded using affine mode or ATMVP merge mode; 3) the CU has more than 64 luma samples; 4) both the height and width of the CU are greater than or equal to 8 luma samples; 5) the BCW weight indices show equal weights; 6) WP is not currently valid for the CU; and 7) CIIP mode is not currently used for the CU.

In VVC Draft 10, the DMVR is subblock-based and includes up to 16 × 16 luma samples. The improved MV for each subblock has a delta MV (Δhor, Δver) from the original MV. Δhor and Δver are the motion vector offsets in the horizontal and vertical directions, respectively. The range of values for Δhor and Δver is determined by the search range of the DMVR. In VVC Draft 10, the search range of the DMVR is [-2, 2]. Therefore, the improved motion vectors have an offset of up to ±2 pel from the original MV in both the horizontal and vertical directions.

The range of ±2 Pel values for Delta MV may be too small for some blocks. For blocks with the best Delta MV outside this range, the video decoder 300 cannot derive an optimally improved MV using a DMVR with such a range.

The range of the Delta MV value can be broadened by widening the DMVR search range. For example, the DMVR search range can be widened to [-8, 8]. Therefore, the improved motion vector will have an offset of up to ±8 Pels from the original MV in both the horizontal and vertical directions.

However, expanding the search range increases the complexity of the DMVR process. For example, when expanding to a fixed search range [-8,8], the video decoder 300 needs to perform more than 11 times more DMVR searches compared to a search range of [-2,2] for the coded block of the DMVR. In addition, even if the derived subsets of MVs are similar or identical, subsets of subblocks within the coded block of the DMVR may have similar improved MVs, and the subblock-based DMVR process includes MV improvements for each subblock. On the other hand, a sub-area of a subblock may have a different optimal improved MV than other sub-areas of the subblock. Since the DMVR of VVC Draft 10 is 16×16 luma-sample subblock-based, the video decoder 300 cannot derive different improved MVs in, for example, 8×8 or 4×4 sub-areas within a 16×16 subblock.

Techniques for improving the DMVR process are disclosed herein.

Example 1. In this example, the improved motion vectors of the subblocks within the W×H coding block are derived by a Multi-Pass DMVR (Multi-Pass Decoder Motion Vector Improvement) process. A given number N may represent the total number of passes in the Multi-Pass DMVR technique. The video decoder 300 can utilize these Multi-Pass DMVR techniques.

Figure 12 is a conceptual diagram illustrating an exemplary three-pass DMVR technique. In this example, a 32x16 coding block 1200 starts with an initial MV _org . The first pass can be block-based. Thus, the first pass can use the entire block 1200, such as currently a PU or CU. A video decoder 300 implementing the first pass 1202 can generate an improved MV, MV _pass1 . The second pass can be subblock-based. In this example, the video decoder 300 can divide block 1200 into two 16x16 subblocks, subblocks 1204A and 1204B. A video decoder 300 implementing the second pass can generate improved MVs for each of subblock 1204A (MV _(pass2,0) ) and subblock 1204B (MV _(pass2,1) ). In this example, the video decoder 300 may divide block 1200 into eight 8x8 subblocks, subblocks 1208A to 1208H. As shown, the video decoder 300 implementing the third pass may generate improved MVs for each of the subblocks 1208A to 1208G.

For example, the video decoder 300 may apply a multipath DMVR to the motion vector for a block of video data (e.g., block 1200) to determine an improved motion vector, and decode the block based on the improved motion vector. The multipath DMVR may include a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second-pass subblock of the block of video data, wherein the width of the second-pass subblock is less than or equal to the width of the first-pass block and the height of the second-pass subblock is less than or equal to the height of the first-pass block; and a third pass that is subblock-based and applied to at least one third-pass subblock, wherein the width of the third-pass subblock is less than or equal to the width of the second-pass subblock and the height of the third-pass subblock is less than or equal to the height of the second-pass subblock.

The multi-pass DMVR technique begins with the original motion vector MV _org of the W×H coding block. The coding block can be PU or CU. The first pass can be block-based. The first pass may derive an improved motion vector MV _pass1 for the entire W×H coding block. MV _pass1 can be saved and used as the initial motion vector for subsequent passes.

The second pass may be subblock-based, for example, based on one or more subblocks of a W×H coding block. A subblock in the second pass (second pass SB) may have a given maximum dimension sbW_1×sbH_1. A W×H coding block may be divided into K1 subblocks (second pass SB) such that K1≧1. Each second pass SB may have dimensions M1×N1 such that M1≦W and N1≦H. Each second pass SB may have an initial motion vector MV _pass1 (for example, MV derived from the first pass). The second pass may derive an improved motion vector MV _(pass2,i) for each second pass SB, where i represents the index of the second pass SB, and 0≦i≦K1-1. MV _(pass2,i) may be saved and used as the initial motion vector for subsequent passes.

The third pass may be subblock-based, for example, based on one or more subblocks of each subblock of the second pass. A subblock in the third pass (third pass SB) has a predetermined maximum dimension sbW_2 × sbH_2, where sbW_2 ≤ sbW_1 and sbH_2 ≤ sbH_1. Each i-th second pass SB in the second pass may be divided into K2 subblocks (third pass SBs), where K2 ≥ 1. The total number of third pass SBs in a W × H coding block may be K2 * K1. Each third pass SB may have dimension M2 × N2, where M2 ≤ sbW_1 and N2 ≤ sbH_1. Each third pass SB in the i-th second pass SB may have an initial motion vector MV _(pass2,i) (for example, the MV derived during the second pass). The third pass derives an improved motion vector MV _(pass3,j) for each third pass SB, where j represents the index of the third pass SB, and 0 ≤ j ≤ K2 * K1 - 1. MV _(pass3,j) can be saved and used as the initial motion vector for subsequent passes.

In some examples, the multi-pass DMVR technique extends to the pth pass. The video decoder 300 performing the MV refinement may derive an MV _(passP,i) for each subblock (pth pass SB) in the pth pass, where i represents the index of the pth pass SB in the W×H coding block. _{MV (passP,i)} may be saved and used to derive the predicted block for the current coding block. MV _(passP,i) represents the refined MV for the i-th subblock.

Example 2. As in Example 1, when both the p-th pass and the preceding pass (the (p-1)th pass) of the DMVR technique are subblock-based, the dimensions of the p-th pass subblock may be less than or equal to the dimensions of the subblock in the preceding pass.

As in Example 1, the range of values for the delta motion vector MV(Δhor, Δver) in the p-th path may be predetermined. For example, minDeltaHorPassP ≤ Δhor ≤ maxDeltaHorPassP and minDeltaVerPassP ≤ Δver ≤ maxDeltaVerPassP. When the p-th path is not the first path (for example, p > 1), the range of values for Δhor and Δver in the p-th path may be less than or equal to the range of values in the preceding path. For example, minDeltaHorPassP ≥ minDeltaHorPass(P-1), maxDeltaHorPassP ≤ maxDeltaHorPass(P-1), minDeltaVerPassP ≥ minDeltaVerPass(P-1), and maxDeltaVerPassP ≤ maxDeltaVerPass(P-1). Since the p-th pass can start from the improved motion vector of the preceding pass, the overall range of values for the delta (final improved) motion vector is extended compared to a single-pass DMVR.

As in Example 1, when the video decoder 300 decides to divide the current coding block into K subblocks, the subblocks can be in the raster scan order from the top left to the bottom right of the current coding block.

Example 3 - Skipping the p-th pass of the DMVR technique. As in Example 1, a given number N may represent the total number of passes in the multi-pass DMVR technique. A video decoder 300 implementing the multi-pass DMVR technique may skip one or more passes to derive the final improved MV. In other words, the video decoder 300 may derive the final improved motion vector by applying a subset of the multi-pass DMVR technique. Skipping the p-th pass of the DMVR technique may reduce the complexity of the video decoder 300.

The decision of whether to skip the p-th pass in the DMVR technique can be based on the results of the preceding passes in the DMVR technique. For example, if the preceding passes derive a relatively optimal improved motion vector, the p-th pass may be skipped.

For example, the video decoder 300 may apply a shortened multipath DMVR to the motion vector for a block. The video decoder 300 may decide to skip a given path of the multipath DMVR for the block, and based on this decision, may skip a given path of the multipath DMVR for the block. For example, the decision to skip a given path may be based on the result of the preceding path, such as when the improved MV of the preceding path is relatively optimal (e.g., further improvements may not result in a change to the MV in terms of MV (sub-per) resolution, or the cost of further improvements may outweigh the benefits of further improvements).

Example 4 - Subblock-Based First-Pass DMVR Technique. In some hardware designs, the maximum size for the motion compensation process may be constrained, and larger coding blocks may be divided into multiple subblocks for hardware processing. In some examples, the multi-pass DMVR technique may begin with a subblock size of min{P,W} × min{Q,H} for the first pass, where P and Q are predetermined integer values determined by hardware constraints.

As in Examples 1 and 3, the first pass of the DMVR technique can be block-based. When the multi-pass DMVR technique begins with a sub-block-based pass, the first pass of the DMVR technique may also be known as the sub-block-based first-pass DMVR technique or the skip first-pass DMVR technique. The video decoder 300 may apply the sub-block-based first-pass DMVR technique.

Example 5 - Skipping the p-th pass of the DMVR technique for a sub-area of a coding block. Assuming a W × H coding block, as in Examples 1 and 3, a given number N may represent the total number of passes of the multi-pass DMVR technique. The video decoder 300 may derive improved motion vectors for a sub-area of a coding block by applying N passes of the DMVR technique. The video decoder 300 may also derive improved motion vectors for different sub-areas of a coding block by applying M passes of the DMVR technique, where M < N. In other words, the video decoder 300 may derive a final improved motion vector for a given sub-area of a coding block by skipping one or more passes. A sub-area may contain one or more sub-blocks of a coding block.

For example, the video decoder 300 applies a shortened multipath DMVR to the motion vector of a block. The video decoder 300 may decide to skip a given subblock-based path of the multipath DMVR for a specific sub-area of a block (e.g., a different sub-area of the coding block mentioned in the paragraph above), which contains one or more sub-blocks, and may skip a given subblock-based path of the multipath DMVR for that specific sub-area based on this decision. For example, the decision to skip a given subblock-based path may be based on the result of the preceding path, such as when the improved MV of the preceding path is relatively optimal (e.g., further improvements may not result in a change to the MV in terms of MV (sub-per) accuracy, or the cost of further improvements may outweigh the benefits of further improvements).

Example 6 - Deriving the improved motion vector in the p-th pass of a DMVR. This example illustrates several decoder-side motion vector improvement techniques. In a multi-pass DMVR technique, the video decoder 300 may apply bilateral matching-based motion vector improvement, as discussed below, through at least one pass, and/or apply BDOF-based motion vector improvement through at least one pass. In other words, at least one pass of a multi-pass DMVR may include applying BDOF, and/or at least one pass of a multi-pass DMVR may include applying bilateral matching. In one example, the first pass includes applying bilateral matching, the second pass includes applying bilateral matching, and the third pass includes applying BDOF.

Figure 13 is a conceptual diagram illustrating an exemplary BDOF motion vector improvement. The derivation of an improved motion vector by bidirectional optical flow is described here. In this example, the video decoder 300 can derive an improved motion vector in the p-th pass DMVR technique by using bidirectional optical flow (BDOF). Improvements to BDOF MV may include the following:
Mv0'=Mv0+bioMv
Mv1'=Mv1-bioMv
Here, Mv0 and Mv1 represent the initial Mv of the p-th path in the current block/subblock within reference picture 0 1300 and reference picture 1 1302, respectively; Mv0' and Mv1' represent the BDOF-enhanced MV of the current block within reference picture 0 1300 and reference picture 1 1302, respectively; and bioMv is the BDOF delta MV.

In the BDOF MV improvement process, bioMv(Δhor,Δver) can be derived from the following steps.
1) As discussed above, from the prediction signals predSig0 and predSig1, the horizontal and vertical gradients,
and
Then, we derive k=0 and k=1.
2) As discussed above, the autocorrelation and crosscorrelation of the gradients, S1, S2, S3, S5, and S6 are derived from the derived horizontal and vertical gradients and the predicted signals predSig0 and predSig1.
3) The two parameters v _x and v _y are derived as follows:
Here, m is a predetermined value. For example, m = 3.
4) The delta MV bioMV(Δhor,Δver) is derived as follows.
Δhor=clip3(minDeltaHorPass3, maxDeltaHorPass3, ((v _x +2 ^n-1 )≫n))
Δver=clip3(minDeltaVerPass3, maxDeltaVerPass3, ((v _y +2 ^n-1 )≫n))
Here,
n is a predetermined value. For example, n=3.
minDeltaHorPass3 is a predetermined value. For example, minDeltaHorPass3 = -2.
maxDeltaHorPass3 is a predetermined value. For example, maxDeltaHorPass3 = 2.
minDeltaVerPass3 is a predetermined value. For example, minDeltaVerPass3 = -2.
maxDeltaVerPass3 is a predetermined value. For example, maxDeltaVerPass3 = 2.

A video decoder 300 that derives improved motion vectors by bilateral matching is described here. Bilateral matching involves searching around two initial motion vectors MV0 and MV1 in the p-th path within a given local search area in reference picture 0 and reference picture 1, respectively. The final MV0' and MV1' are derived based on the minimum bilateral matching cost.

The local search area for bilateral matching for a coding block has a horizontal search range, e.g., [sMinHor, sMaxHor], and a vertical search range, e.g., [sMinVer, sMaxVer]. The local search area for bilateral matching for a coding block may be (sMaxHor - sMinHor + 1) × (sMaxVer - sMinVer + 1).

As in Example 2, if there is a predetermined range of values for the delta motion vector MV(Δhor,Δver) in the p-th pass, the value of the search range can be determined by the range of values for the delta motion vector in the p-th pass DMVR technique, as follows:
sMinHor≧minDeltaHorPassP
sMaxHor≦maxDeltaHorPassP
sMinVer≧minDeltaVerPassP
sMaxVer≦maxDeltaVerPassP

Further decoder-side motion vector refinement methods are described here. The refined motion vectors may be derived by alternative decoder-side motion vector derivation techniques, such as template matching or decoder-side motion vector derivation (DMVD). A video decoder 300 implementing a p-pass multi-pass DMVR technique may use one of these motion vector refinement methods described in this disclosure. However, the details of the DMVR technique may differ from those described in this document, and may still remain within the scope of this disclosure.

Example 7 - Deriving the prediction signal in the p-th pass DMVR technique by applying an interpolation filter or by using the prediction signal from a preceding pass. As in Example 6, the motion vector improvement technique in the p-th pass starts with the initial motion vector in the p-th pass and the prediction signal in the reference picture. The prediction signal in the reference picture can be derived by applying an interpolation filter along with the initial motion vector information in the reference picture.

In this example, the video decoder 300 is
1) The predicted signal is derived using the p-th pass DMVR technique by applying an interpolation filter. The interpolation filter may be determined in the p-th pass by an MV improvement technique (e.g., bilateral matching, BDOF, etc.), and/or
2) By using the preceding prediction signal, the prediction signal is derived using the p-pass DMVR technique.

A video decoder 300 is described here that derives a predicted signal using the p-pass DMVR technique by applying an interpolation filter. In DMVR techniques for bilateral matching or improved motion vector derivation, some simplified interpolation filter may be used to generate motion compensation results for the search. For example, a bilinear interpolation filter may be used to generate non-integer samples for the search process in bilateral matching or DMVR.

In some examples, when applying BDOF-based techniques, as in Example 6, the input may be samples generated by motion compensation using the original (unsimplified) interpolation filter, in order to derive an improved motion vector in the p-th pass.

In other examples, when applying BDOF-based techniques, as in Example 6, the input may be samples generated by motion compensation using a simplified interpolation filter, such as a bilinear interpolation filter, in order to derive an improved motion vector in the p-th pass.

A video decoder 300 is described here that derives a prediction signal in the p-th pass DMVR technique by using the prediction signal of the preceding pass. For example, the video decoder 300 may decide whether to use the prediction signal of the preceding pass by checking the accuracy of the delta motion vector in the preceding pass. For example,
1) When the delta motion vector in the preceding path has integer Pell precision, the prediction signal in the p-th path of the DMVR technique may be derived by using the prediction signal of the preceding path, and 2) When the improved motion vector in the preceding path is identical to the initial motion vector in the preceding path, the prediction signal in the p-th path of the DMVR technique may be derived by using the prediction signal of the preceding path.

Example 8 - Example of 3-pass decoder-side motion improvement. In this example, the video decoder 300 uses a 3-pass decoder-side motion improvement technique. In this example, the process involves three passes as follows: 1) The first pass is block-based. The improved motion vector is derived by applying bilateral matching-based motion vector improvement. The delta motion value range is, for example, [-8,8] horizontally and, for example, [-8,8] vertically. 2) The second pass is sub-block-based. The improved motion vector is derived by applying bilateral matching-based motion vector improvement. The maximum sub-block dimension is, for example, 16 × 16 luma samples. For example, the sub-block of the second pass has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples. The delta motion value range is, for example, [-8,8] horizontally and, for example, [-8,8] vertically. 3) The third pass is sub-block-based. The improved motion vector is derived by applying a BDOF-based motion vector improvement. The maximum subblock dimension is, for example, 8 x 8 luma samples. For example, the subblock of the third pass has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples. The delta motion value range is, for example, [-2,2] in the horizontal direction and, for example, [-2,2] in the vertical direction.

For example, the delta motion range for at least one of the first or second path may be [-8,8] horizontally and [-8,8] vertically, and the delta motion range for the third path may be [-2,2] horizontally and [-2,2] vertically.

The aforementioned techniques can be applied by the video decoder 300 of the video coding system. The following is a detailed example of multipath DMVR. The video decoder 300 may implement the techniques described herein by all or a subset of the following steps in order to decode interpredicted blocks in a picture from a bitstream.
1) The position component (x,y) is derived as the top-left Luma position of the current block by decoding the syntax element in the bitstream.
2) The size of the current block is derived as width W and height H by decoding the syntax elements in the bitstream.
3) By decoding the elements in the bitstream, we determine that the current block is the block that can be interpreted.
4) By decoding the elements in the bitstream, the motion vector components (mvL0 and mvL1) and reference indices (refPicL0 and refPicL1) of the current block are derived.
5) A flag is inferred from decoding the elements in the bitstream, and the flag indicates whether a decoder-side motion vector derivation (e.g., DMVR, bilateral merge, template matching, etc.) is currently applied to the block. The flag inference method may be, but is not limited to, the same as the DMVR activation conditions discussed earlier in this disclosure. In another example, this flag may be explicitly signaled in the bitstream to avoid complex conditional checks by the video decoder 300.
6) (Pass 1) If the decision is not to apply DMVR (Bilateral Merge or Template Matching) to the current block, according to the value of the flag mentioned above, then the motion vectors mvL0 and mvL1 are set as the motion vectors for MV0pass1 and MV1pass1, respectively; otherwise (if the decision is to apply DMVR to the current block), then the following applies:
(a) Set the mvL0 and mvL1 of the current block as the initial motion vectors for the current block.
(b) Determine the variables sHor and sVer as follows:
sHor=maximum(maxDeltaHorPass1, W×sFactor)
sVer=maximum(maxDeltaVerPass1, H×sFactor)
Here,
maxDeltaHorPass1 is a predetermined variable (for example, 8).
maxDeltaVerPass1 is a predetermined variable (for example, 8).
sFactor is a given variable (for example, 0.5).
sHor is the search range of the DMVR in the horizontal direction [-sHor, sHor].
sVer is the vertical search range of the DMVR [-sVer, sVer].
(c) Derive the prediction signal predSig0 from reference picture 0 by using the derived mvL0 and refPicL0. The width of predSig0 is equal to W + 2 × sHor. The height of predSig0 is equal to H + 2 × sVer.
(d) Using the derived mvL1 and refPicL1, the prediction signal predSig1 is derived from reference picture 1. The width of predSig1 is equal to W + 2 × sHor. The height of predSig0 is equal to H + 2 × sVer.
(e) Set the variable minCostPass1 to the maximum cost value.
(f) Set the variable best delta MV(Δhor_best,Δver_best) to delta MV(0,0).
(g) Loop through each or a subset of DeltaMV(Δhor,Δver) within the current block's search range, where -sVer≦Δver≦sVer and -sHor≦Δhor≦sHor.
(i) Derive the bilateral matching cost bilCost in the current delta MV(Δhor,Δver).
(ii) If bilCost is less than minCostPass1,
(a) Set minCostPass1 to be equal to bilCost.
(b) Set the best delta MV(Δhor_best,Δver_best) to be equal to MV(Δhor,Δver).
(h) Derive the improved motion vector (mvL0+MV(Δhor_best,Δver_best)) as the motion vector for MV0 _pass1 .
(i) Derive the improved motion vector (mvL1-MV(Δhor_best,Δver_best)) as the motion vector for MV1 _pass1 .
7) (Path 2) The number of subblocks in the horizontal direction numSbX and the number of subblocks in the vertical direction numSbY, the width of the subblocks sbWidthPass2, and the height sbHeightPass2 are derived as follows:
numSbX=(W>thW)?(W/thW):1
numSbY=(H>thH)?(H/thH):1
sbWidthPass2=(W>thW)?thW:W
sbHeightPass2=(H>thH)?thH:H
Here, thW and thH are predetermined integer values that represent the width and height of the largest subblock for the second path, respectively (for example, thW = thH = 16).
(a) If the decision is not to apply DMVR (Bilateral Merge or Template Matching) to the current block, according to the value of the flag mentioned above, then the motion vectors MV0 _pass1 and MV1 _pass1 are set as the motion vectors MV0 _(pass2,i) and MV1 _(pass2,i) for each subblock, respectively; otherwise (if the decision is to apply DMVR to the current block), then the following applies:
(b) (Check whether to skip pass 2) Derive a variable costThPass2 equal to (thFactorPass2 × W × H), where thFactorPass2 is a predetermined value, for example thFactorPass2 = 1. If minCostPass1 is less than costThPass2, set MV0 _pass1 and MV1 _pass1 as the motion vectors MV0 _{(pass2, i)} and MV1 _{(pass2, i)} for each subblock, respectively; otherwise (if minCostPass1 is greater than or equal to costThPass2), the following applies:
(i) Set the position component (sbX,sbY)=(x,y) as the top-left corner position of the first subblock of the current block.
(ii) For each subblock, starting from the top left and going down to the bottom right,
(a) Set the variable i = (sbY/sbHeightPass2)*(W/sbWidthPass2)+(sbX/sbWidthPass2) as the current subblock index.
(b) Set MV0 _pass1 and MV1 _pass1 as the initial motion vectors for the current subblock.
(c) Determine the variables sHor and sVer as follows:
sHor=maximum(maxDeltaHorPass2, sbWidthPass2×sFactor)
sVer=maximum(maxDeltaVerPass2, sbHeightPass2×sFactor)
Here,
maxDeltaHorPass2 is a predetermined variable (for example, 8), and maxDeltaVerPass2 is a predetermined variable (for example, 8).
sFactor is a given variable (for example, 0.5).
sHor specifies the horizontal search range [-sHor,sHor] for path 2.
sVer specifies the vertical search range [-sVer,sVer] for path 2.
(d) Derive the prediction signal predSig0 from reference picture 0 by using the derived MV0 _pass1 and refPicL0. The width of predSig0 is equal to sbWidthPass2 + 2 × sHor. The height of predSig0 is equal to sbHeightPass2 + 2 × sVer.
(e) Derive the prediction signal predSig1 from reference picture 1 by using the derived MV1 _pass1 and refPicL1. The width of predSig1 is equal to sbWidthPass2 + 2 × sHor. The height of predSig0 is equal to sbHeightPass2 + 2 × sVer.
(f) Set the variable minCostPass2 to the maximum cost value.
(g) Set the variable best delta MV(Δhor_best,Δver_best) to delta MV(0,0).
(h) Loop through each or a subset of DeltaMV(Δhor,Δver) within the current subblock's search range, where -sVer≦Δver≦sVer and -sHor≦Δhor≦sHor.
(i) Derive the bilateral matching cost bilCost in the current delta MV(Δhor,Δver).
(ii) If bilCost is less than minCostPass2,
(a) Set minCostPass2 to be equal to bilCost.
(b) Set the best delta MV(Δhor_best,Δver_best) to be equal to MV(Δhor,Δver).
(i) Derive the improved motion vector (MV0 _pass1 + MV(Δhor_best, Δver_best)) as the motion vector for MV0 _{(pass2, i)} .
(j) Derive the improved motion vector (MV1 _pass1 -MV(Δhor_best,Δver_best)) as the motion vector of MV1 _(pass2,i) .
(k) Update the top-left luma position of the subblock as follows:
sbX=(sbX+sbWidthPass2)<W?sbX+sbWidthPass2:0
sbY=(sbX+sbWidthPass2)<W?sbY:sbY+sbHeightPass2
8) Infer a flag from decoding elements in the bitstream, which indicates whether a bidirectional optical flow is currently applied to the block. The method for inferring the flag is not limited, but could be the same as in the example above. In another example, this flag may be explicitly signaled in the bitstream to avoid complex conditional checks in the decoder.
9) (Path 3) When the decision is to apply BDOF to the current block according to the value of the flag mentioned above, the following is true:
(a) The number of subblocks in the horizontal direction numSbX and the number of subblocks in the vertical direction numSbY, the width of the subblock sbW, and the height sbH are derived as follows:
numSbX=(W>thW)?(W/thW):1
numSbY=(H>thH)?(H/thH):1
sbWidthPass3=(W>thW)?thW:W
sbHeightPass3=(H>thH)?thH:H
Here, thW and thH are predetermined integer values that represent the width and height of the largest subblock for the third path, respectively (for example, thW = thH = 8).
(b) Derive a variable costThPass3 that is equal to (thFactorPass3 × sbWidth × sbHeight), where thFactorPass3 is a predetermined value, for example, thFactorPass3 = 32.
(c) Set the position component (sbX,sbY)=(x,y) as the top-left corner position of the first subblock of the current block.
(d) For each subblock, starting from the top left and going down to the bottom right,
(i) Set the variable i = (sbY/sbHeightPass3)*(W/sbWidthPass3)+(sbX/sbWidthPass3) as the current subblock index of path 3.
(ii) Set the variable j = (sbY/sbHeightPass2)*(W/sbWidthPass2)+(sbX/sbWidthPass2) as the current subblock index of path 2.
(iii) Set MV0 _(pass2,j) and MV1 _(pass2,j) as the initial motion vectors for the current subblock.
(iv) Using the derived MV0 _(pass2,j) and refPicL0, the prediction signal predSig0 is derived from reference picture 0.
(v) Using the derived MV1 _(pass2,j) and refPicL1, the predicted signal predSig1 is derived from reference picture 1.
(vi) Derive the strain cost distance between predSig0 and predSig1 in the current subblock.
(vii) (Check whether to skip sub-area pass 3) If the strain cost distance is less than costThPass3, set MV0 _(pass2,j) and MV1 _(pass2,j) as the improved motion vectors MV0 _(pass3,i) and MV1 _(pass3,i) for the sub-block, respectively; otherwise (if the strain cost distance is greater than or equal to costThPass3), the following applies:
(a) As discussed above, from the prediction signals predSig0 and predSig1, the horizontal and vertical gradients,
and
Then, we derive k=0 and k=1.
(b) As discussed above, the autocorrelation and crosscorrelation of the gradients, S1, S2, S3, S5, and S6 are derived from the derived horizontal and vertical gradients and the predicted signals predSig0 and predSig1.
(c) The two parameters v _x and v _y are derived as follows:
Here, m is a predetermined value. For example, m = 3.
(d) The delta MV bioMV(Δhor,Δver) is derived as follows.
Δhor=clip3(minDeltaHorPass3, maxDeltaHorPass3, ((v _x +2 ^n-1 )≫n))
Δver=clip3(minDeltaVerPass3, maxDeltaVerPass3, ((v _y +2 ^n-1 )≫n))
Here,
n is a predetermined value. For example, n=3.
minDeltaHorPass3 is a predetermined value. For example, minDeltaHorPass3 = -2.
maxDeltaHorPass3 is a predetermined value. For example, maxDeltaHorPass3 = 2.
minDeltaVerPass3 is a predetermined value. For example, minDeltaVerPass3 = -2.
maxDeltaVerPass3 is a predetermined value. For example, maxDeltaVerPass3 = 2.
(e) Derive the improved motion vector (MV0 _(pass2,j) +bioMV(Δhor,Δver)) as the motion vector for MV0 _(pass3,i) .
(f) The improved motion vector (MV1 _(pass2,j) -bioMV(Δhor,Δver)) is derived as the motion vector of MV1 _(pass3,i) .
(viii) Update the top-left luma position of the subblock as follows:
sbX=(sbX+sbWidthPass3)<W?sbX+sbWidthPass3:0
sbY=(sbX+sbWidthPass3)<W?sbY:sbY+sbHeightPass3
10) Using the improved motion vectors MV0 _(pass3,i) and MV1 _(pass3,i) of each subblock for video decoding, derive the predicted block.

Example 9 - All passes in the multi-pass DMVR technique are skipped. When all passes in the multi-pass DMVR technique are skipped, the final improved motion vector MV _(passP,i) for each subblock in the last pass (pass P) is equal to the initial motion vector MV _Org .

For example, as in Example 8, the video decoder 300 may decide whether to apply a BDOF-based motion vector improvement (pass 3) to the current block, depending on the conditions under which a DMVR (e.g., bilateral merge or template matching) is applied to the current block. For example, in step 5 of Example 8 above, the video decoder 300 decides not to apply a DMVR to the current block, and all three passes of the multi-pass DMVR technique are skipped. The improved motion vectors MV0 _(pass3,i) and MV1 _(pass3,i) for each subblock in step 10 of Example 8 are equal to mvL0 and mvL1, respectively. For example, the video decoder 300 may decide not to apply a DMVR to the block. Based on the decision not to apply a DMVR to the block, the video decoder 300 may skip all passes of the multi-pass DMVR and decode the block based on the initial motion vector.

Figure 14 is a flowchart illustrating an exemplary multi-pass DMVR technique of the present disclosure. The video decoder 300 may apply multi-pass DMVR to the MV for blocks of video data to determine an improved MV (1400). For example, the video decoder 300 may apply multi-pass DMVR including a first pass that is block-based, a second pass that is sub-block-based, where the width of the second-pass subblock is less than or equal to the width of the first-pass block and the height of the second-pass subblock is less than or equal to the height of the first-pass block, and a third pass that is sub-block-based, where the width of the third-pass subblock is less than or equal to the width of the second-pass subblock and the height of the third-pass subblock is less than or equal to the height of the second-pass subblock.

The video decoder 300 may code blocks based on the improved MV (1402). For example, the video decoder 300 may predict blocks using the improved MV.

In some examples, at least one third-pass subblock of a block of video data is a subblock for at least one second-pass subblock of the block of video data. In some examples, the video decoder 300 may apply a first pass to derive at least one first improved motion vector for a block of video data and use at least one first improved motion vector in a second pass. For example, the video decoder 300 may use a first improved motion vector as an initial motion vector for a second pass. In some examples, the video decoder 300 may apply a second pass to derive at least one second improved motion vector for at least one second-pass subblock and use at least one second improved motion vector in a third pass. For example, the video decoder 300 may derive one or more second improved motion vectors for one or more subblocks of the second pass and use one or more second improved motion vectors as an initial motion vector for a third pass. In some examples, the video decoder 300 may apply a third pass to derive at least one third improved motion vector for at least one third pass subblock, thereby determining at least one improved motion vector as at least one third improved motion vector.

In some examples, at least one pass of a multipath DMVR includes applying BDOF or bilateral matching. In some examples, the first pass includes applying bilateral matching, the second pass includes applying bilateral matching, and the third pass includes applying BDOF.

In some examples, at least one second-pass subblock has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples. In some examples, at least one third-pass subblock has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples.

In some examples, the delta movement range for at least one of the first or second paths is [-8,8] horizontally and [-8,8] vertically, and the delta movement range for the third path is [-2,2] horizontally and [-2,2] vertically.

In some examples, the block of video data is the first block. In some examples, the video decoder 300 may apply a shortened multipath DMVR to the motion vector for the second block of video data. For example, the video decoder 300 may decide to skip a given path of the multipath DMVR for the second block, and based on that decision, may skip a given path of the multipath DMVR for the second block. In some examples, the video decoder 300 may decide to skip a given path based on the result of a preceding path.

In some examples, the video block is the first block. In some examples, the video decoder 300 may apply a shortened multipath DMVR to the motion vector for the second block of video data. For example, the video decoder 300 may decide to skip a given subblock-based path of the multipath DMVR for a specific sub-area of the second block of video data, where the specific sub-area comprises one or more sub-blocks of the second block. For example, the video decoder 300 may skip a given subblock-based path of the multipath DMVR for a specific sub-area of the second block based on its decision to skip that path. In some examples, the video decoder 300 may decide to skip a given path based on the result of a preceding path.

In some examples, the block is the first block of video data. In some examples, the video decoder 300 may decide not to apply the DMVR to the second block of video data. Based on the decision not to apply the DMVR to the second block, the video decoder 300 may skip all paths of the multipath DMVR for the second block and decode the second block based on the initial motion vector for the second block.

Figure 15 is a flowchart illustrating an exemplary method for encoding a current block using the technique of this disclosure. A current block may comprise a current CU. While the video encoder 200 (Figures 1 and 3) is described, it should be understood that other devices may be configured to perform a similar method to that shown in Figure 15.

In this example, the video encoder 200 first predicts the current block (350). For example, the video encoder 200 may form a predicted block for the current block. Next, the video encoder 200 may compute the residual block for the current block (352). To compute the residual block, the video encoder 200 may compute the difference between the original unencoded block and the predicted block for the current block. Next, the video encoder 200 may transform the residual block and quantize the transformation coefficients of the residual block (354). Then, the video encoder 200 may scan the quantized transformation coefficients of the residual block (356). During or following the scan, the video encoder 200 may entropy encode the transformation coefficients (358). For example, the video encoder 200 may encode the transformation coefficients using CAVLC or CABAC. Next, the video encoder 200 may output the entropy encoded data of the block (360).

Figure 16 is a flowchart illustrating an exemplary method for decoding the current block of video data using the technique of this disclosure. The current block may comprise the current CU. While the video decoder 300 (Figures 1 and 4) is described, it should be understood that other devices may be configured to perform a method similar to that shown in Figure 16.

The video decoder 300 may receive entropy-encoded data for the current block, such as entropy-encoded prediction information and entropy-encoded data of the transformation coefficients of the residual block corresponding to the current block (370). The video decoder 300 may entropy-decode the entropy-encoded data to determine the prediction information for the current block and reconstruct the transformation coefficients of the residual block (372). The video decoder 300 may predict the current block to compute the prediction block for the current block, for example, using an intra-prediction mode or an inter-prediction mode as indicated by the prediction information for the current block (374). As part of predicting the current block, the video decoder 300 may use any of the multi-pass DMVR techniques of this disclosure, including, but not limited to, the technique shown in Figure 14. The video decoder 300 may then inversely scan the reconstructed transformation coefficients to create a block of quantized transformation coefficients (376). The video decoder 300 may then inversely quantize the transformation coefficients and apply the inverse transform to the transformation coefficients to generate a residual block (378). The video decoder 300 can finally decode the current block by combining the predicted block and the residual block (380).

This disclosure includes the following non-exclusive provisions.

Clause 1A. A method for coding video data, comprising the steps of: applying a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine improved motion vectors; and coding the block based on the improved motion vectors.

Clause 2A. The method according to Clause 1A, wherein the total number of paths in a multipath DMVR is a predetermined integer.

Clause 3A. The method according to Clause 1A or Clause 2A, wherein the multipath DMVR comprises a first path that is block-based, a second path that is subblock-based, and a third path that is subblock-based.

Clause 4A. The method according to Clause 3A, wherein the step of applying the first path derives the first improved motion vector.

Clause 5A. The method according to Clause 4A, wherein an improved motion vector is used in a second path, the subblocks of the second path have a predetermined maximum width and a predetermined maximum height, and the step of applying the second path derives a second improved motion vector for at least one of each subblocks of the second path.

Clause 6A. The method according to Clause 5A, wherein each second improved motion vector is used in a third path, the subblocks of the third path have a predetermined maximum width and a predetermined maximum height, and the step of applying the third path derives a third improved motion vector for at least one of each subblocks of the third path.

Clause 7A. A method of any combination of Clauses 1A through 6A in which the multipath DMVR is iterative.

Clause 8A. A method according to any combination of Clauses 1A through 7A, wherein the subblocks of a path are less than or equal to the size of the blocks or subblocks of the preceding path.

Clause 9A. A method according to any combination of Clauses 1A through 8A, further comprising the steps of determining whether to skip a given path in a multipath DMVR, and skipping the given path in the multipath DMVR based on the decision to skip the given path.

Clause 10A. The method according to Clause 9A, wherein the step of determining whether to skip a given path comprises the step of determining that a given improved motion vector from a preceding path is optimal.

Clause 11A. A method according to any combination of Clauses 1A to 10A, further comprising the step of determining that a given subblock size is the smaller of (P, W) and the smaller of (Q, H), wherein P and Q are predetermined integers.

Clause 12A. The method described in Clause 11A, where P and Q are based on hardware constraints.

Clause 13A. A method of any combination of Clauses 1A to 12A, further comprising the steps of determining whether to skip a subblock path of a multipath DMVR for a particular subblock, and, based on the decision to skip the subblock path, skipping the subblock path of the multipath DMVR for the particular subblock.

Clause 14A. The method according to Clause 13A, wherein the step of determining whether to skip a subblock path comprises the step of determining that the improved motion vector of the subblock from the preceding path is optimal.

Clause 15A. A method according to any combination of Clauses 1A through 14A, wherein at least one path of a multipath DMVR comprises a step of applying bidirectional optical flow.

Clause 16A. A method according to any combination of Clauses 1A through 15A, wherein at least one path of a multipath DMVR comprises a step of applying bilateral matching.

Clause 17A. A method according to any combination of Clauses 1A through 16A, wherein at least one pass of a multipath DMVR comprises a step of applying template matching.

Clause 18A. A method according to any combination of Clauses 1A to 17A, wherein at least one pass of a multipass DMVR comprises a step of applying an interpolation filter.

Clause 19A. The method according to any combination of Clauses 1A to 18A, wherein at least one pass of a multipass DMVR comprises a step of applying a simplified interpolation filter.

Clause 20A. The method of any combination of Clauses 1A to 19A, further comprising the steps of determining whether to skip all paths of a multipath DMVT and coding a block based on an initial motion vector, based on the decision to skip all paths of the multipath DMVT.

Clause 21A. The method according to any one of Clauses 1A to 20A, wherein the coding step comprises a decryption step.

Clause 22A. The method according to any one of Clauses 1A to 21A, wherein the coding step comprises an encoding step.

Clause 23A. A device for coding video data, comprising one or more means for performing the method described in any of Clauses 1A to 22A.

Clause 24A. The device described in Clause 23A, wherein one or more means comprises one or more processors implemented in the circuit.

Clause 25A. A device as described in either Clause 23A or 24A, further comprising memory for storing video data.

Clause 26A. A device described in any combination of Clauses 23A through 25A, further comprising a display configured to display decoded video data.

Clause 27A. A device described in any combination of Clauses 23A through 26A, comprising one or more of the following: a camera, a computer, a mobile device, a broadcast receiver device, or a set-top box.

Clause 28A. A device described in any combination of Clauses 23A through 27A, wherein the device comprises a video decoder.

Clause 29A. A device described in any combination of Clauses 23A through 28A, wherein the device comprises a video encoder.

Clause 30A. A computer-readable storage medium storing instructions, wherein, when the instructions are executed, causes one or more processors to perform the method described in any of Clauses 1A to 22A.

Clause 1B. A method for decoding video data,
A method comprising the steps of: applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector; and decoding the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass which is block-based and applied to a block of video data; a second pass which is subblock-based and applied to at least one second pass subblock of the block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass which is subblock-based and applied to at least one third pass subblock of the block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

Clause 2B. The method according to Clause 1B, wherein at least one third-path subblock of a block of video data is a subblock with respect to at least one second-path subblock of a block of video data.

Clause 3B. The method according to Clause 1B or Clause 2B, wherein the step of applying the first pass derives at least one first improved motion vector for a block of video data, and at least one first improved motion vector is used in the second pass.

Clause 4B. The method according to Clause 3B, wherein the step of applying the second pass derives at least one second improved motion vector for at least one each second pass subblock, and at least one second improved motion vector is used in the third pass.

Clause 5B. The method according to Clause 4B, wherein the step of applying the third path derives at least one third improved motion vector for at least one each third path subblock, and at least one improved motion vector is determined to be at least one third improved motion vector.

Clause 6B. A method according to any combination of Clauses 1B to 5B, wherein at least one path of the multipath DMVR comprises a step of applying bidirectional optical flow (BDOF) or a step of applying bilateral matching.

Clause 7B. The method according to Clause 6B, comprising: a first pass applying bilateral matching; a second pass applying bilateral matching; and a third pass applying BDOF.

Clause 8B. The method according to any combination of Clauses 1B to 7B, wherein at least one second pass subblock has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples.

Clause 9B. The method according to any combination of Clauses 1B to 8B, wherein at least one third-path subblock has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples.

Clause 10B. The method according to any combination of Clauses 1B to 9B, wherein the delta motion range for at least one of the first or second paths is [-8,8] horizontally and [-8,8] vertically, and the delta motion range for the third path is [-2,2] horizontally and [-2,2] vertically.

Clause 11B. The method of any combination of Clauses 1B to 10B, wherein a block of video data is a first block, and the method further comprises the step of applying a shortened multipath DMVR to a motion vector for a second block of video data, wherein the applying step comprises the step of determining to skip a given path of the multipath DMVR for the second block, and the step of skipping a given path of the multipath DMVR for the second block based on the determination to skip a given path of the multipath DMVR for the second block.

Clause 12B. The method described in Clause 11B, wherein the step of deciding to skip a given path is based on the result of a preceding path.

Clause 13B. The method of any combination of Clauses 1B to 12B, wherein a block of video data is a first block, and the method further comprises the step of applying a shortened multipath DMVR to a motion vector for a second block of video data, wherein the applying step is a step of determining to skip a given subblock-based path of the multipath DMVR for a particular sub-area of the second block of video data, the particular sub-area comprising one or more sub-blocks of the second block; and the step of skipping a given subblock-based path of the multipath DMVR for a particular sub-area of the second block based on the determination to skip a given subblock-based path of the multipath DMVR for a particular sub-area of the second block.

Clause 14B. The method described in Clause 13B, wherein the step of deciding to skip a given subblock-based path is based on the result of a preceding path.

Clause 15B. The method according to any combination of Clauses 1B to 10B, wherein Block is a first block of video data, and the method further comprises the steps of: determining not to apply DMVR to a second block of video data; skipping all passes of the multipath DMVR for the second block based on the decision not to apply DMVR to the second block; and decoding the second block based on an initial motion vector for the second block.

Clause 16B. A device for decoding video data, comprising a memory configured to store video data, and one or more processors implemented by circuitry and communicatively coupled to the memory, wherein one or more processors are configured to apply a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector, and to decode the block based on the at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

Clause 17B. The device according to Clause 16B, wherein at least one third-path subblock of a block of video data is a subblock with respect to at least one second-path subblock of a block of video data.

Clause 18B. The device described in Clause 16B or Clause 17B, wherein one or more processors are configured to apply a first pass to derive at least one first improved motion vector for a block of video data and to use at least one first improved motion vector in a second pass.

Clause 19B. The device according to Clause 18B, wherein one or more processors are configured to apply a second pass to derive at least one second improved motion vector for at least one each second pass subblock, and to use at least one second improved motion vector in a third pass.

Clause 20B. The device according to Clause 19B, wherein one or more processors are configured to apply a third path to derive at least one third improved motion vector for at least one third path subblock, and to determine at least one improved motion vector as at least one third improved motion vector.

Clause 21B. A device according to any combination of Clauses 16B to 20B, wherein at least one path of the multipath DMVR comprises a step of applying bidirectional optical flow (BDOF) or a step of applying bilateral matching.

Clause 22B. The device described in Clause 21B, comprising a first pass applying bilateral matching, a second pass applying bilateral matching, and a third pass applying BDOF.

Clause 23B. A device according to any combination of Clauses 16B to 22B, wherein at least one second pass subblock has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples.

Clause 24B. A device according to any combination of Clauses 16B to 23B, wherein at least one third-pass subblock has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples.

Clause 25B. A device according to any combination of Clauses 16B to 24B, wherein the delta motion range for at least one of the first or second passes is [-8,8] horizontally and [-8,8] vertically, and the delta motion range for the third pass is [-2,2] horizontally and [-2,2] vertically.

Clause 26B. A device according to any combination of Clauses 16B to 25B, wherein a block of video data is a first block, and one or more processors are configured to apply a shortened multipath DMVR to the motion vectors for a second block of video data, and one or more processors decide to skip a given path of the multipath DMVR for the second block in order to apply the shortened multipath DMVR to the motion vectors for the second block, and are configured to skip a given path of the multipath DMVR for the second block based on the decision to skip a given path of the multipath DMVR for the second block.

Clause 27B. The device described in Clause 26B, in which one or more processors are configured to determine whether to skip a given path based on the results of a preceding path.

Clause 28B. A device according to any combination of Clauses 16B to 27B, wherein a block of video data is a first block, and one or more processors are configured to apply a shortened multipath DMVR to motion vectors for a second block of video data, and to determine, in order to apply the shortened multipath DMVR to motion vectors for the second block, one or more processors are configured to determine, and to perform the given subblock-based path of the multipath DMVR for a particular sub-area of the second block, wherein the particular sub-area comprises one or more sub-blocks of the second block, and based on the determination to perform the given subblock-based path of the multipath DMVR for the particular sub-area of the second block, the device is configured to perform the given subblock-based path of the multipath DMVR for the particular sub-area of the second block.

Clause 29B. A device as described in Clause 28B, in which one or more processors are configured to determine whether to skip a given subblock-based path based on the results of a preceding path.

Clause 30B. A device as described in any combination of Clauses 16B to 25B, wherein a block is a first block of video data, and one or more processors are further configured to decide not to apply a DMVR to a second block of video data, to skip all paths of the multipath DMVR for the second block based on the decision not to apply a DMVR to the second block, and to decode the second block based on the initial motion vector for the second block.

Clause 31B. A non-temporary computer-readable storage medium storing instructions, wherein, when an instruction is executed, it causes one or more processors to apply a multipath decoder-side motion vector improvement (DMVR) to the motion vectors for a block of video data to determine at least one improved motion vector, and to decode the block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

Clause 33B. A device for coding video data, comprising means for applying a multipath decoder-side motion vector improvement (DMVR) to a motion vector for a block of video data to determine at least one improved motion vector, and means for decoding a block based on at least one improved motion vector, wherein the multipath DMVR comprises: a first pass that is block-based and applied to a block of video data; a second pass that is subblock-based and applied to at least one second pass subblock of a block of video data, wherein the width of the second pass subblock is less than or equal to the width of the block of video data and the height of the second pass subblock is less than or equal to the height of the block of video data; and a third pass that is subblock-based and applied to at least one third pass subblock of a block of video data, wherein the width of the third pass subblock is less than or equal to the width of the second pass subblock and the height of the third pass subblock is less than or equal to the height of the second pass subblock.

It should be noted that, depending on the example, some of the actions or events among the techniques described herein may be executed in a different order, added, merged, or completely excluded (for example, not all described actions or events are necessarily required for the practice of the technique). Furthermore, in some examples, the actions or events may be executed not sequentially, but concurrently, for example, through multithreading, interrupt handling, or across multiple processors.

In one or more examples, the described functions may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a computer-readable medium, transmitted through a computer-readable medium, or executed by a hardware-based processing unit. The computer-readable medium may include computer-readable storage media corresponding to tangible media such as data storage media, or communication media including any medium that facilitates the transfer of computer programs from one location to another according to a communication protocol, for example. Thus, the computer-readable medium may generally correspond to (1) non-transient tangible computer-readable storage media, or (2) communication media such as signals or carrier waves. The data storage medium may be any available medium that can be accessed by one or more computers or one or more processors to retrieve instructions, codes, and/or data structures for implementing the techniques described herein. A computer program product may include computer-readable media.

As an example, and not an limitation, such computer-readable storage media may include RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory, or any other media that can be used to store desired program code in the form of instructions or data structures and can be accessed by a computer. Any connection is also appropriately called computer-readable media. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. However, it should be understood that computer-readable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but instead refer to non-temporary tangible storage media. The terms "disk" and "disc" as used herein include compact discs (CDs), laser discs, optical discs, digital multi-purpose discs (DVDs), floppy disks, and Blu-ray discs, where a disk typically reproduces data magnetically, and a disc reproduces data optically using a laser. Combinations of the above should also be included within the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one or more DSPs, general-purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuits. Therefore, the terms “processor” and “processing circuit” as used herein may refer to any of the above-described structures or any other structure suitable for implementing the techniques described herein. In addition, in some embodiments, the functions described herein may be provided within dedicated hardware and/or software modules configured for encoding and decoding, or incorporated into a composite codec. Furthermore, the techniques may be fully implemented with one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide variety of devices or apparatus, including wireless handsets, integrated circuits (ICs), or sets of ICs (e.g., chipsets). Various components, modules, or units have been described in this disclosure to highlight the functional aspects of devices configured to perform the disclosed techniques, but these do not necessarily require implementation by different hardware units. Rather, as described above, the various units may be combined in a codec hardware unit, or provided by a set of interoperable hardware units, including one or more processors as described above, along with appropriate software and/or firmware.

Various examples were described. These and other examples fall within the scope of the following claims.

102 Source Device
104 Video Sources
106 memory
108 Output Interfaces
110 Computer-readable media
112 Storage Devices
114 File Server
116 Destination Devices
118 Display Devices
120 memory
122 Input Interfaces
130 QTBT structure
132 Coding Tree Units
200 video encoders
202 Mode Selection Unit
204 Residual Generation Unit
206 Conversion Processing Unit
208 Quantization Units
210 Inverse Quantization Unit
212 Inverse Transform Processing Unit
214 Reconstruction Unit
216 Filter Unit
218 Decoded picture buffer
220 Entropy Coding Units
222 Motion Estimation Unit
224 Motion Compensation Unit
226 Intra Prediction Units
230 video data memory
300 video decoders
302 Entropy Decoding Unit
304 Predictive Processing Unit
306 Inverse Quantization Unit
308 Inverse Transform Processing Unit
310 Reconstruction Unit
312 Filter Unit
314 DPB
316 Motion Compensation Unit
317 MPDMVR
318 Intra Prediction Units
320 CPB memory
500 PU0
502 PU0
600 blocks
602 blocks
610 Same position MV
700 templates
702 Current Picture
706 Reference Picture
800 TD0
802 TD1
804 Current Picture
806 Reference Picture
808 Reference Picture
810 TD0
812 TD1
900 Initial MV
902 3x3 search pattern
904 MV
906 The last selected MV
1000 blocks
1002 blocks
1100 CU
1102 Subblock
1200 coding blocks
1202 First Pass
1204A-1204B Subblock
1208A-1208H Subblock

Claims (15)

A method for decoding video data,
The steps include applying a multipath decoder-side motion vector improvement (DMVR) to the motion vector for a block of video data in order to determine at least one improved motion vector,
The step of decoding the block based on the at least one improved motion vector is included.
The aforementioned multipath DMVR
A first path which is block-based and applied to the blocks of the video data,
A second path that is subblock-based and applied to at least one second path subblock of the block of the video data, wherein the width of the second path subblock is less than or equal to the width of the block of the video data, and the height of the second path subblock is less than or equal to the height of the block of the video data,
A method comprising: a third path which is subblock-based and applied to at least one third path subblock of the block of video data, wherein the width of the third path subblock is less than or equal to the width of the second path subblock, the height of the third path subblock is less than or equal to the height of the second path subblock, and the at least one third path subblock of the block of video data is a subblock with respect to the at least one second path subblock of the block of video data. The step of applying the first pass derives at least one first improved motion vector for the block of video data, and the at least one first improved motion vector is used in the second pass.
The step of applying the second pass derives at least one second improved motion vector for each of at least one second pass subblocks, and the at least one second improved motion vector is used in the third pass.
The method according to claim 1, wherein the step of applying the third pass derives at least one third improved motion vector for at least one each third pass subblock, and the at least one improved motion vector is determined to be the at least one third improved motion vector. At least one path of the multipath DMVR includes applying bidirectional optical flow (BDOF) or bilateral matching,
The method according to claim 1, wherein the first pass comprises applying bilateral matching, the second pass comprises applying bilateral matching, and the third pass comprises applying BDOF. The at least one second pass subblock has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples, and/or
The method according to claim 1, wherein the at least one third pass subblock has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples. The method according to claim 1, wherein the delta motion range for at least one of the first or second paths is [-8,8] horizontally and [-8,8] vertically, and the delta motion range for the third path is [-2,2] horizontally and [-2,2] vertically. The block of the video data is a first block, and the method further comprises the step of applying a shortened multipath DMVR to a motion vector for a second block of the video data, the applying step is
A step of deciding to skip a given path of the multipath DMVR for the second block based on the result of the preceding path,
The process includes the step of skipping the given path of the multipath DMVR for the second block based on the decision to skip the given path of the multipath DMVR for the second block,
or
The block of the video data is a first block, and the method further comprises the step of applying a shortened multipath DMVR to a motion vector for a second block of the video data, the applying step is
A step of determining, based on the results of a preceding pass, to fly a given subblock-based path of the multipath DMVR for a specific sub-area of the second block of the video data, wherein the specific sub-area comprises one or more sub-blocks of the second block;
The method according to claim 1, further comprising the step of flying the given subblock-based path of the multipath DMVR for the particular sub-area of the second block, based on the decision to fly the given subblock-based path of the multipath DMVR for the particular sub-area of the second block. The block is the first block of the video data, and the method further
The steps include deciding not to apply DMVR to the second block of the aforementioned video data,
Based on the decision not to apply the DMVR to the second block, the steps include skipping all paths of the multipath DMVR for the second block,
The method according to claim 1, further comprising the step of decoding the second block based on an initial motion vector for the second block. A device for decoding video data,
A memory configured to store the aforementioned video data,
The circuit comprises one or more processors that are implemented in a circuit and are communicably coupled to the memory, and the one or more processors
To determine at least one improved motion vector, a multipath decoder-side motion vector improvement (DMVR) is applied to the motion vector for the block of video data,
The block is configured to decode based on at least one improved motion vector,
The aforementioned multipath DMVR
A first path which is block-based and applied to the blocks of the video data,
A second path that is subblock-based and applied to at least one second path subblock of the block of the video data, wherein the width of the second path subblock is less than or equal to the width of the block of the video data, and the height of the second path subblock is less than or equal to the height of the block of the video data,
A device comprising: a third path which is subblock-based and applied to at least one third path subblock of the block of video data, wherein the width of the third path subblock is less than or equal to the width of the second path subblock, the height of the third path subblock is less than or equal to the height of the second path subblock, and the at least one third path subblock of the block of video data is a subblock with respect to the at least one second path subblock of the block of video data. The one or more processors are configured to apply the first pass in order to derive at least one first improved motion vector for the block of video data and to use the at least one first improved motion vector in the second pass.
The one or more processors are configured to apply the second pass in order to derive at least one second improved motion vector for each of at least one second pass subblocks, and to use the at least one second improved motion vector in the third pass.
The device according to claim 8, wherein one or more processors are configured to apply the third pass to derive at least one third improved motion vector for each of at least one third pass subblocks, and to determine the at least one improved motion vector as the at least one third improved motion vector. The device according to claim 8, wherein at least one path of the multipath DMVR includes applying bidirectional optical flow (BDOF), or at least one path of the multipath DMVR includes applying bilateral matching, wherein the first path includes applying bilateral matching, the second path includes applying bilateral matching, and the third path includes applying BDOF. The at least one second pass subblock has a predetermined maximum width of 16 luma samples and a predetermined maximum height of 16 luma samples, and/or
The device according to claim 8, wherein the at least one third pass subblock has a predetermined maximum width of 8 luma samples and a predetermined maximum height of 8 luma samples. The device according to claim 8, wherein the delta motion range for at least one of the first or second passes is [-8,8] horizontally and [-8,8] vertically, and the delta motion range for the third pass is [-2,2] horizontally and [-2,2] vertically. The block of the video data is a first block, and one or more processors are configured to apply a shortened multipath DMVR to the motion vector for a second block of the video data, and in order to apply the shortened multipath DMVR to the motion vector for the second block, the one or more processors
Based on the results of the preceding pass, it is decided to skip a given path of the multipath DMVR for the second block.
Based on the decision to skip the given path of the multipath DMVR for the second block, the multipath DMVR is configured to skip the given path for the second block.
or
The block of the video data is a first block, and one or more processors are configured to apply a shortened multipath DMVR to the motion vector for a second block of the video data, and in order to apply the shortened multipath DMVR to the motion vector for the second block, the one or more processors
Determining, based on the results of a preceding pass, to perform a given subblock-based pass of the multipath DMVR for a specific sub-area of the second block of the video data, wherein the specific sub-area comprises one or more sub-blocks of the second block,
The device according to claim 8, configured to perform the given subblock-based path of the multipath DMVR for the particular sub-area of the second block, based on the decision to perform the given subblock-based path of the multipath DMVR for the particular sub-area of the second block. The block is the first block of the video data, and the one or more processors further
It was decided not to apply DMVR to the second block of the aforementioned video data.
Based on the decision not to apply the DMVR to the second block, all paths of the multipath DMVR for the second block are skipped.
The device according to claim 8, configured to decode the second block based on an initial motion vector for the second block. A computer program, when executed, causes one or more processors to perform the method described in any one of claims 1 to 7.