JP7840973B2 - Machine learning-based flow decision for video coding - Google Patents

Description

[0001] This disclosure generally relates to image and video coding, including encoding (or compression) and decoding (decompression) of images and/or videos. For example, aspects of this disclosure relate to techniques for determining flow information of rumor and chroma components of one or more image frames or pictures (e.g., video frames/pictures).

[0002] Many devices and systems enable video data to be processed and output for consumption. Digital video data contains large amounts of data to meet the demands of consumers and video providers. For example, consumers of video data desire high-quality video, including high fidelity, resolution, and frame rate. As a result, the large amounts of video data required to meet these demands burden communication networks and devices that process and store the video data.

[0003] Video coding techniques can be used to compress video data. The goal of video coding is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As video services are constantly evolving and more readily available, coding techniques with better coding efficiency are needed.

[0004] Systems and techniques for coding (e.g., encoding and/or decoding) image and/or video content using one or more machine learning systems are described. According to at least one example, a method for processing video data is provided. This method includes: obtaining input video data containing at least one luminance component of the current frame using a machine learning system; and using the at least one luminance component of the current frame, determining motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using a machine learning system.

[0005] In another example, a device for processing video data is provided, comprising at least one memory (for example, configured to store data such as virtual content data, one or more images) and one or more processors coupled to the at least one memory (for example, implemented in circuitry). The one or more processors are configured to use a machine learning system to acquire input video data including at least one luminance component of the current frame, and to use the machine learning system to determine, and can determine, the motion information of at least one luminance component of the current frame and the motion information of one or more chrominance components of the current frame.

[0006] In another example, a non-transitory computer-readable medium is provided that, when executed by one or more processors, stores instructions causing one or more processors to: use a machine learning system to acquire input video data including at least one luminance component of the current frame; and use the machine learning system to determine, using at least one luminance component of the current frame, the motion information of at least one luminance component of the current frame and the motion information of one or more chrominance components of the current frame.

[0007] In another example, a device for processing video data is provided. This device includes means for acquiring input video data including at least one luminance component of the current frame, and means for determining motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, using at least one luminance component of the current frame.

[0008] In some embodiments, one or more of the above methods, apparatus, and computer-readable media further comprises: determining a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame by a machine learning system using motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame; and determining one or more inter-frame predictions of the current frame using the warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame.

[0009] In some embodiments, one or more interframe predictions are determined at least partially by applying an interpolation operation using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame.

[0010] In some embodiments, the interpolation operation includes a trilinear interpolation operation.

[0011] In some embodiments, the warping parameters of at least one luminance component of the current frame and the warping parameters of one or more chrominance components of the current frame include space-scale flow (SSF) warping parameters.

[0012] In some embodiments, the SSF warping parameters include a learned scale-flow vector.

[0013] In some embodiments, to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame using at least one luminance component of the current frame, one or more of the above methods, apparatus, and computer-readable media further include: determining motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one reconstructed luma component of the previous frame; and determining motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.

[0014] In some embodiments, motion information for one or more chrominance components of the current frame is determined using a convolutional layer of a machine learning system.

[0015] In some embodiments, one or more of the methods, apparatus, and computer-readable media described above further comprises sampling the determined motion information for at least one luminance component of the current frame in order to determine the motion information for one or more chrominance components of the current frame.

[0016] In some embodiments, the current frame includes a video frame.

[0017] In some embodiments, one or more chrominance components include at least one chrominance-blue component and at least one chrominance-red component.

[0018] In some embodiments, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is the YUV 4:2:0 format.

[0019] In some embodiments, the apparatus described herein may comprise or be part of a mobile device (e.g., a mobile phone or so-called “smartphone”, a tablet computer, or other type of mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device in a vehicle), or other device. In some embodiments, the apparatus includes at least one camera for capturing one or more images or video frames. For example, the apparatus may include a camera (e.g., an RGB camera) or multiple cameras for capturing one or more videos containing one or more images and/or video frames. In some embodiments, the apparatus includes a display for displaying one or more images, videos, notifications, or other displayable data. In some embodiments, the apparatus includes a transmitter configured to transmit one or more video frames and/or syntax data to at least one device via a transmission medium. In some embodiments, the processor includes a neural processing unit (NPU), a central processing unit (CPU), a graphics processing unit (GPU), or other processing devices or components.

[0020] This summary does not identify the principal or essential features of the claimed subject matter, nor is it to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the entire specification of this patent, any or all of the drawings, and the appropriate portions of each claim.

[0021] The above, along with other features and embodiments, will become more apparent with reference to the following specification, claims, and accompanying drawings.

[0022] Exemplary embodiments of this application are described in detail below with reference to the following figures.

[0023] A diagram showing an exemplary implementation of a system-on-a-chip (SOC). [0024] A diagram showing an example of a fully connected neural network. [0025] A diagram showing an example of a locally connected neural network. [0026] A diagram showing an example of a convolutional neural network. [0027] A detailed example of a deep convolutional network (DCN) designed to recognize visual features from an image. [0028] A block diagram showing a deep convolutional network (DCN). [0029] A diagram illustrating an example of a system that includes a device capable of performing image and/or video coding (encoding and decoding) using a neural network-based system, by several examples. [0030] A figure illustrating an example of an end-to-end neural network-based image and video coding system for inputs having a red-green-blue (RGB) format, by several examples. [0031] A figure showing an example of a spatial-scale flow (SSF) neural network architecture configured to process one or more luminance-chrominance (YUV) input formats (e.g., 4:2:0 YUV input format), which may be part of an end-to-end neural network-based image and video coding system, by several examples. [0032] A diagram illustrating an example of a machine learning-based flow engine that operates with rumor input, using several examples. [0033] A figure showing an example of subsampling of lumar motion information to obtain chroma motion information, using several examples. [0034] A figure illustrating an example of a machine learning-based architecture with YUV (e.g., YUV4:2:0) residuals, by several examples. [0035] A diagram illustrating the exemplary operation of a 1x1 convolutional layer by several examples. [0036] A figure illustrating an example of a machine learning-based architecture (e.g., an end-to-end neural network-based image and video coding system) that directly interacts with YUV inputs (Y, U, and V), such as YUV4:2:0 inputs, by several examples. [0037] A figure illustrating another example of a machine learning-based architecture (e.g., an end-to-end neural network-based image and video coding system) that directly interacts with YUV inputs (Y, U, and V), such as YUV4:2:0 inputs, by several examples. [0038] A flowchart illustrating an example of a process for processing video data, using several examples. [0039] A diagram showing an exemplary computing device architecture of an exemplary computing device capable of implementing the various techniques described herein.

[0040] Several aspects and embodiments of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects and embodiments may be applied independently, and some of them may be applied in combination. Specific details are provided below for illustrative purposes to provide a complete understanding of the embodiments of this application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not limiting.

[0041] The subsequent description provides only exemplary embodiments and does not limit the scope, applicability, or configuration of the present disclosure. Rather, the subsequent description of exemplary embodiments will provide a description that enables the implementation of exemplary embodiments for those skilled in the art. It should be understood that various modifications can be made to the function and configuration of the elements without departing from the spirit and scope of this application, as described in the appended claims.

[0042] Digital video data can contain large amounts of data, especially as the demand for high-quality video data continues to grow. For example, consumers of video data typically desire increasingly higher quality video with high fidelity, resolution, frame rate, etc. However, the large amounts of video data required to meet such demands can place a significant burden on communication networks, as well as on devices that process and store the video data.

[0043] Various techniques can be used to code video data. Video coding may be performed according to a specific video coding standard. Exemplary video coding standards include High Efficiency Video Coding (HEVC), Advanced Video Coding (AVC), Moving Picture Expert Group (MPEG) coding, and Multipurpose Video Coding (VVC). Video coding often uses predictive methods such as inter-prediction or intra-prediction, which utilize redundant configurations present in video images or sequences. A common goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0044] Image and/or video coding can be performed using machine learning (ML) based systems. Generally, ML is a subset of artificial intelligence (AI). ML systems can include algorithms and statistical models that computer systems can use to perform various tasks by relying on patterns and inferences without using explicit instructions. An example of an ML system is a neural network (also called an artificial neural network), which may include an interconnected group of artificial neurons (e.g., neuron models). Neural networks may be used in a variety of applications and/or devices, including, among others, image and/or video coding, image analysis and/or computer vision applications, Internet Protocol (IP) cameras, Internet of Things (IoT) devices, autonomous vehicles, and service robots.

[0045] Individual nodes within a neural network can emulate biological neurons by acquiring input data and performing simple operations on the data. The results of the simple operations performed on the input data are selectively passed to other neurons. Weight values are associated with each vector and node in the network, and these values constrain how the input data relates to the output data. For example, the input data of each node may be multiplied by the corresponding weight value, and the product may be summed. The sum of the products may be adjusted by an arbitrary bias, and an activation function is applied to the result, yielding the node's output signal or "output activation" (sometimes called an activation map or feature map). The weight values may initially be determined by an iterative flow of training data through the network (for example, the weight values are established during the training phase, when the network learns how to identify those classes by typical input data characteristics of those classes).

[0046] In particular, there are different types of neural networks, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), generative adversarial networks (GANs), and multilayer perceptron (MLP) neural networks. For example, a convolutional neural network (CNN) is a type of feedforward artificial neural network. A convolutional neural network may contain a set of artificial neurons that collectively tile the input space, each having a receptive field (e.g., a spatially localized region of the input space). RNNs operate on the principle of saving the output of a layer and feeding this output back into the input to help predict the outcome of the layer. A GAN is a form of generative neural network that can learn patterns in input data so that the neural network model can generate a new synthetic output that may reasonably be from the original dataset. A GAN can include two neural networks working together, one which is a generative neural network that generates the synthetic output, and the other which is a discriminative neural network that evaluates the output for reliability. In an MLP neural network, data may be fed into the input layer, and one or more hidden layers may provide a level of abstraction to the data. Predictions may then be made on the output layer based on the abstracted data.

[0047] In a hierarchical neural network architecture (called a deep neural network when multiple hidden layers exist), the output of the first layer of artificial neurons becomes the input to the second layer of artificial neurons, the output of the second layer of artificial neurons becomes the input to the third layer of artificial neurons, and so on. A CNN may be trained, for example, to recognize a hierarchy of features. Computation in a CNN architecture may be distributed across a set of processing nodes that may constitute one or more computation chains. These multilayer architectures may be trained one layer at a time and may be fine-tuned using backpropagation.

[0048] In many cases, deep learning-based systems are designed as a combination of an autoencoder subnetwork (encoder subnetwork) and a second subnetwork (decoder subnetwork) that is involved in learning a probabilistic model on a quantization latent used for entropy coding (sometimes also called a hyperplier network). In some cases, other subnetworks of the decoder may be present. Such a deep learning-based system architecture can be seen as a combination of a transformation-plus-quantization module (or encoder subnetwork) and an entropy modeling subnetwork module.

[0049] Most existing deep learning-based architectures for video compression are designed to operate with non-subsampled input formats such as RGB, YUV4:4:4, or other non-subsampled input formats. However, video coding standards such as HEVC and VVC are designed to support the YUV4:2:0 color format within their respective main profiles. Deep learning-based architectures designed to operate with non-subsampled input formats need to be modified to support the 4:2:0 YUV format.

[0050] Systems, apparatus, processes (also called methods), and computer-readable media (collectively referred to as “Systems and Techniques”) are described herein as providing ML-based systems (e.g., deep learning-based systems) that can use one color component of one or more frames (e.g., video frames) to estimate information about the color components of a frame and other color components. In some embodiments, ML-based systems may be designed to process input data having a luminance-chrominance (YUV) input format. In such embodiments, the ML-based system can use the luma component of both the current frame and a previously reconstructed frame (e.g., reconstructed by the ML-based system) to estimate motion information (e.g., flow information such as optical flow information) about both the luma component and one or more chroma components. In some cases, after learning the motion information of the luma component, a convolutional layer with downsampling can be used to learn the motion information (e.g., flow information) of one or more chroma components. In some cases, motion information for one or more chroma components can be obtained by directly subsampling the motion information for lumen components (e.g., without using convolutional layers). Such techniques can be performed for all components of a frame. Using such techniques, ML-based systems can determine chroma motion information (e.g., flow information) without requiring chroma information coded as part of latent data or a bitstream (e.g., reducing the need to transmit side information along with chroma information).

[0051] As described above, ML-based systems may be designed to process input data having a YUV input format. The YUV format includes a luminance channel (Y) and a pair of chrominance channels (U and V). The U channel may be called the chrominance (or chroma)-blue channel, and the U channel may be called the chrominance (or chroma)-red channel. In some cases, the luminance (Y) channel or component may also be called the luma channel or component. In some cases, the chrominance (U and V) channels or components may also be called the chroma channel or component. The YUV input format may include, among other things, YUV4:2:0, YUV4:4:4, and YUV4:2:2. In some cases, the systems and techniques described herein may be designed to process other input formats such as the Y-chroma blue (Cb)-chroma red (Cr) (YCbCr) format, the red-green-blue (RGB) format, and/or other formats. The ML-based systems described herein can encode and/or decode video data containing standalone frames (also called images) and/or multiple frames.

[0052] Further details and additional aspects of this disclosure will be described with reference to the figures.

[0053] Figure 1 shows an exemplary implementation of a system-on-a-chip (SOC) 100, which may include a central processing unit (CPU) 102 or a multi-core CPU configured to perform one or more of the functions described herein. Among some of the information, parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with computation devices (e.g., weighted neural networks), delays, frequency bin information, and task information may be stored in memory blocks associated with the neural processing unit (NPU) 108, memory blocks associated with the CPU 102, memory blocks associated with the graphics processing unit (GPU) 104, memory blocks associated with the digital signal processor (DSP) 106, memory block 118, and/or distributed across multiple blocks. Instructions executed in the CPU 102 may be loaded from program memory associated with the CPU 102 or from memory block 118.

[0054] The SOC 100 may also include additional processing blocks adapted to specific functions, such as a GPU 104 and a DSP 106, a connectivity block 110 which may include fifth-generation (5G) connectivity, fourth-generation Long-Term Evolution (4G LTE®) connectivity, Wi-Fi® connectivity, USB connectivity, Bluetooth® connectivity, etc., and a multimedia processor 112 which can detect and recognize gestures, for example. In one implementation, the NPU is implemented within the CPU 102, DSP 106, and/or GPU 104. The SOC 100 may also include a navigation module 120 which may include a sensor processor 114, an image signal processor (ISP) 116, and/or a global positioning system.

[0055] The SOC 100 may be based on an ARM instruction set. In one aspect of this disclosure, instructions loaded into the CPU 102 may include code for retrieving the stored multiplication result in a lookup table (LUT) corresponding to the multiplication product of the input value and the filter weights. Instructions loaded into the CPU 102 may also include code for disabling the multiplier during the multiplication operation of the multiplication product when a lookup table hit is detected for the multiplication product. Furthermore, instructions loaded into the CPU 102 may include code for storing the calculated multiplication product of the input value and the filter weights when a lookup table miss is detected for the multiplication product.

[0056] The SOC 100 and/or its components may be configured to perform video compression and/or decompression (also known as video coding and/or decoding, collectively referred to as video coding) using machine learning techniques in accordance with the embodiments of the disclosure described herein. By using a deep learning architecture to perform video compression and/or decompression, embodiments of the disclosure can increase the efficiency of video compression and/or decompression on a device. For example, a device using the described video coding techniques can compress video more efficiently using machine learning-based techniques and transmit the compressed video to another device, which can then decompress the compressed video more efficiently using the machine learning-based techniques described herein.

[0057] As described above, a neural network is an example of a machine learning system and may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processing is performed by the hidden nodes of one or more hidden layers, and the output is produced via the output nodes of the output layer. Deep learning networks typically include multiple hidden layers. Each layer of a neural network may include a feature map or activation map that may contain artificial neurons (or nodes). Feature maps may include filters, kernels, etc. Nodes may include one or more weights used to indicate the importance of one or more nodes in the layer. In some cases, a deep learning network may have a series of many hidden layers, with early layers used to determine simple, low-level characteristics of the input, and later layers building a hierarchy of more complex and abstract characteristics.

[0058] Deep learning architectures can learn feature hierarchies. For example, when visual data is presented, a first layer can be learned to recognize relatively simple features, such as edges in the input stream. In another example, when auditory data is presented, a first layer can be learned to recognize spectral powers of specific frequencies. A second layer, which takes the output of the first layer as input, can be learned to recognize combinations of features, such as simple shapes in the case of visual data, or combinations of sounds in the case of auditory data. For example, a higher layer can be learned to represent complex shapes in visual data, or words in auditory data. A higher layer can be learned to recognize common visual objects or speech phrases.

[0059] Deep learning architectures can perform particularly well when applied to problems with natural hierarchical structures. For example, the classification of motorized vehicles can benefit from initial learning to recognize wheels, windshields, and other features. These features may be combined in different ways in higher layers to recognize cars, trucks, and airplanes.

[0060] Neural networks can be designed using various connection patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating with neurons in higher layers. As described above, a hierarchical representation can be constructed in the consecutive layers of a feedforward network. Neural networks can also have recurrent or (also called top-down) feedback connections. In recurrent connections, the output from a neuron in a given layer may be communicated to another neuron in the same layer. Recurrent architectures can be useful when recognizing patterns across two or more chunks of input data delivered sequentially to the neural network. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. Networks with many feedback connections can be useful when recognizing a high-level concept can help distinguish specific low-level features of the input.

[0061] The connections between layers of a neural network can be fully connected or locally connected. Figure 2A shows an example of a fully connected neural network 202. In the fully connected neural network 202, neurons in the first layer can communicate their outputs to any neuron in the second layer, so that each neuron in the second layer receives input from any neuron in the first layer. Figure 2B shows an example of a locally connected neural network 204. In the locally connected neural network 204, neurons in the first layer may be connected to a limited number of neurons in the second layer. More generally, the locally connected layers of the locally connected neural network 204 may be configured such that each neuron in the layer has the same or similar connection pattern, but may be configured with connection strengths that may have different values (e.g., 210, 212, 214, and 216). The connection patterns of local connections can create spatially distinct receptive fields in the upper layers, as higher-layer neurons within a given region receive inputs that have been trained to be tuned to the properties of a limited portion of the total input to the network.

[0062] An example of a locally connected neural network is a convolutional neural network. Figure 2C shows an example of a convolutional neural network 206. The convolutional neural network 206 may be configured such that the connection strength associated with the input for each neuron in the second layer is shared (e.g., 208). The convolutional neural network may be suitable for problems where the spatial location of the input is meaningful. The convolutional neural network 206 may be used to perform one or more embodiments of video compression and/or decompression according to embodiments of this disclosure.

[0063] One type of convolutional neural network is a deep convolutional network (DCN). Figure 2D shows a detailed example of a DCN 200 designed to recognize visual features from an image 226 input from an image capture device 230, such as an in-vehicle camera. The DCN 200 in this example may be trained to identify traffic signs and the numbers provided on them. Naturally, the DCN 200 may be trained for other tasks, such as identifying lane markings or traffic signals.

[0064] The DCN 200 may be trained using supervised learning. During training, the DCN 200 may be presented with images, such as image 226 of a speed limit sign, and then a forward pass may be calculated to generate output 222. The DCN 200 may include a feature extraction section and a classification section. Upon receiving image 226, the convolutional layer 232 may apply a convolutional kernel (not shown) to image 226 to generate a first set 218 of feature maps. As an example, the convolutional kernel for the convolutional layer 232 may be a 5x5 kernel that generates a 28x28 feature map. In this example, four different feature maps are generated in the first set 218 of feature maps, so four different convolutional kernels were applied to image 226 in the convolutional layer 232. Convolutional kernels are sometimes called filters or convolutional filters.

[0065] The first set of feature maps 218 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 220. The max pooling layer reduces the size of the first set of feature maps 218. That is, the size of the second set of feature maps 220, such as 14x14, is smaller than the size of the first set of feature maps 218, such as 28x28. The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 220 may be further convolved through one or more subsequent convolutional layers (not shown) to generate one or more subsequent sets of feature maps (not shown).

[0066] In the example of Figure 2D, a second set of feature maps 220 is convolved to generate a first feature vector 224. Furthermore, the first feature vector 224 is further convolved to generate a second feature vector 228. Each feature in the second feature vector 228 may contain a number corresponding to possible features of image 226, such as "label", "60", and "100". A softmax function (not shown) can convert the numbers in the second feature vector 228 into probabilities. Thus, the output 222 of DCN 200 is the probability that image 226 contains one or more features.

[0067] In this example, the probabilities in output 222 for “label” and “60” are higher than the probabilities for other outputs 222 such as “30”, “40”, “50”, “70”, “80”, “90”, and “100”. Before training, the outputs 222 generated by DCN200 may be inaccurate. Therefore, an error may be calculated between output 222 and the target output. The target output is the ground truth of image 226 (e.g., “label” and “60”). The weights of DCN200 may then be adjusted so that the output 222 of DCN200 is more closely matched to the target output.

[0068] To adjust the weights, the learning algorithm can calculate a gradient vector for the weights. The gradient can indicate the amount by which the error increases or decreases when the weights are adjusted. In the top layer, the gradient can directly correspond to the weight values connecting the activated neurons in the second-to-last layer to the neurons in the output layer. In lower layers, the gradient may depend on the weight values and the calculated error gradients in the upper layers. The weights may then be adjusted to reduce the error. This method of adjusting weights is sometimes called "backpropagation" because it involves a "backward path" through the neural network.

[0069] In practice, the error gradient of the weights may be calculated over a small number of examples so that the calculated gradient approximates the true error gradient. This approximation method is sometimes called stochastic gradient descent. Stochastic gradient descent may be repeated until the achievable error rate for the entire system no longer decreases, or until the error rate reaches a target level. After training, the DCN may be presented with a new image, and a forward pass through the network may yield an output 222 that can be considered an inference or prediction of the DCN.

[0070] Deep belief networks (DBNs) are probabilistic models with multiple layers of hidden nodes. DBNs can be used to extract hierarchical representations of training datasets. DBNs can be obtained by stacking layers of restricted Boltzmann machines (RBMs). RBMs are a type of artificial neural network that can learn probability distributions across a set of inputs. RBMs are often used in unsupervised learning because they can learn probability distributions in the absence of information about the class to which each input should be classified. Using hybrid unsupervised and supervised paradigms, the lower RBMs of a DBN can be trained unsupervised and can function as feature extractors, while the upper RBMs can be trained supervised (on a joint distribution of inputs from previous layers and target classes) and can function as classifiers.

[0071] A deep convolutional network (DCN) is a network of convolutional networks composed of additional pooling and normalization layers. DCNs achieve state-of-the-art performance for many tasks. DCNs can be trained using supervised learning, where both the input and output targets are known for many samples, and the network weights are modified using gradient descent.

[0072] A DCN can be a feedforward network. In addition, as described above, the connections from neurons in the first layer of a DCN to groups of neurons in the next higher layer are shared across the neurons in the first layer. The feedforward and shared connections of a DCN can be leveraged for high-speed processing. The computational burden of a DCN can be significantly less than that of a similarly sized neural network with, for example, recurrent or feedback connections.

[0073] The processing of each layer of a convolutional network may be considered as a spatially invariant template or basis projection. If the input is initially decomposed into multiple channels, such as the red, green, and blue channels of a color image, the convolutional network trained on that input may be considered three-dimensional, having two spatial dimensions along the image axes and a third dimension for capturing color information. The output of the convolutional connections may be considered to form a feature map in subsequent layers, where each element of the feature map (e.g., 220) receives input from various neurons in the previous layer (e.g., feature map 218) and from each of the multiple channels. The values in the feature map may be further processed using nonlinearities such as rectification and max(0,x). Values from adjacent neurons may be further pooled, which corresponds to downsampling and can provide further local invariance and dimensionality reduction.

[0074] Figure 3 is a block diagram showing an example of a deep convolutional network 350. The deep convolutional network 350 may include several different types of layers based on connectivity and weight sharing. As shown in Figure 3, the deep convolutional network 350 includes convolutional blocks 354A and 354B. Each of the convolutional blocks 354A and 354B may consist of a convolutional layer (CONV) 356, a normalization layer (LNorm) 358, and a maximum pooling layer (MAX POOL) 360.

[0075] The convolutional layer 356 may include one or more convolutional filters, which may be applied to the input data 352 to generate feature maps. Although only two convolutional blocks 354A and 354B are shown, this disclosure is not limited in that way, and instead, any number of convolutional blocks (e.g., blocks 354A and 354B) may be included in the deep convolutional network 350, depending on design preferences. A normalization layer 358 can normalize the output of the convolutional filters. For example, the normalization layer 358 can provide whitening or lateral suppression. A maximum pooling layer 360 can provide downsampling aggregation across space for local invariance and dimensionality reduction.

[0076] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 102 or GPU 104 of the SOC 100 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 116 of the SOC 100. In addition, the deep convolutional network 350 can access other processing blocks that may reside on the SOC 100, such as the sensor processor 114 and navigation module 120, which are dedicated to sensors and navigation, respectively.

[0077] The deep convolutional network 350 may also include one or more fully connected layers, such as layer 362A (labeled "FC1") and layer 362B (labeled "FC2"). The deep convolutional network 350 may further include a logistic regression (LR) layer 364. Between each layer 356, 358, 360, 362A, 362B, and 364 of the deep convolutional network 350 are weights (not shown) that need to be updated. Each output of a layer (e.g., 356, 358, 360, 362A, 362B, 364) can serve as input to one of the subsequent layers (e.g., 356, 358, 360, 362A, 362B, 364) in the deep convolutional network 350 to learn a hierarchical feature representation from the input data 352 (e.g., image, audio, video, sensor data, and/or other input data) supplied in the first convolutional block 354A. The output of the deep convolutional network 350 is a classification score 366 for the input data 352. The classification score 366 can be a set of probabilities, where each probability is the probability that the input data contains a feature from the set of features.

[0078] As described above, digital video data can contain large amounts of data, which can place a considerable burden on communication networks and devices that process and store video data. For example, recording uncompressed video content generally results in large file sizes that increase significantly as the resolution of the video content being recorded increases. In one exemplary example, uncompressed 16-bit per channel video recorded at 1080p/24 (for example, a resolution of 1920 pixels wide and 1080 pixels high captured at 24 frames per second) may occupy 12.4 megabytes per frame or 297.6 megabytes per second. Uncompressed 16-bit per channel video recorded at 4K resolution at 24 frames per second may occupy 49.8 megabytes per frame or 1195.2 megabytes per second.

[0079] Network bandwidth is another constraint that can be problematic for large video files. For example, video content is often delivered over wireless networks (e.g., LTE, LTE Advanced, New Wireless (NR), Wi-Fi®, Bluetooth, or other wireless networks) and can constitute a significant portion of a consumer's internet traffic. Despite advancements in the amount of available bandwidth on wireless networks, it may still be desirable to reduce the amount of bandwidth used to deliver video content over these networks.

[0080] Uncompressed video content can result in large files that may require considerable memory for physical storage and considerable bandwidth for transmission; therefore, video coding techniques may be used to compress and then decompress such video content.

[0081] To reduce the size of video content, and therefore the amount of storage required to store it, and the amount of bandwidth required to deliver it, various video coding techniques can be employed, particularly according to specific video coding standards such as HEVC, AVC, MPEG, and VVC. Video coding often uses prediction methods such as inter-prediction or intra-prediction, which utilize redundant configurations present in video images or sequences. A common goal of video coding techniques is to compress video data into a format that uses a lower bitrate while avoiding or minimizing degradation of video quality. As the demand for video services increases and new video services become available, coding techniques with better coding efficiency, performance, and rate control are needed.

[0082] Generally, encoding devices encode video data according to a video coding standard to produce an encoded video bitstream. In some examples, an encoded video bitstream (or “video bitstream” or “bitstream”) is a series of one or more encoded video sequences. An encoding device can produce an encoded representation of a picture by dividing each picture into multiple slices. A slice is independent of other slices so that the information within the slice is encoded without dependencies on data from other slices within the same picture. A slice includes one or more slice segments containing independent slice segments and one or more dependent slice segments that, if present, depend on previous slice segments. In HEVC, a slice is divided into coding tree blocks (CTBs) of lumane samples and chromane samples. One or more CTBs of lumane samples and chromane samples, along with the syntax for the samples, are called coding tree units (CTUs). CTUs are sometimes called “tree blocks” or “maximum coding units” (LCUs). The CTU is the basic processing unit for HEVC coding. A CTU can be divided into multiple coding units (CUs) of varying sizes. Each CU contains sample sequences of lumens and chromens, called coding blocks (CBs).

[0083] The lumana CB and chroma CB may be further divided into prediction blocks (PBs). A PB is a block of samples of a lumana or chroma component that uses the same motion parameters for inter-prediction or intra-block copy (IBC) prediction (when available or enabled for use). The lumana PB and one or more chroma PBs, along with the associated syntax, form a prediction unit (PU). For inter-prediction, a set of motion parameters (e.g., one or more motion vectors, reference indices, etc.) is signaled in the bitstream for each PU and used for inter-prediction of the lumana PB and one or more chroma PBs. The motion parameters are sometimes called motion information. The CB may also be divided into one or more transformation blocks (TBs). A TB represents a square block of samples of a color component to which a residual transformation (e.g., possibly the same two-dimensional transformation) is applied to code the prediction residual signal. A conversion unit (TU) represents the TB of the lumens and chroma samples and their corresponding syntax elements. The conversion coding is described in more detail below.

[0084] According to the HEVC standard, the conversion may be performed using a TU. The TU may be sized based on the size of the PU within a given CU. The TU may be the same size as the PU or smaller. In some examples, the residual samples corresponding to the CU may be subdivided into smaller units using a quadtree structure known as a residual quadtree (RQT). The leaf nodes of the RQT may correspond to the TU. The pixel difference values associated with the TU may be converted to generate conversion coefficients. The conversion coefficients may then be quantized by the encoding device.

[0085] Once the picture of the video data is segmented into CUs, the encoding device uses a prediction mode to predict each PU. The prediction unit or prediction block is then subtracted from the original video data to obtain the residual (described below). For each CU, the prediction mode may be signaled in the bitstream using syntax data. The prediction mode may include intra-prediction (or intra-picture prediction) or inter-prediction (or inter-picture prediction). Intra-prediction takes advantage of the correlation between spatially adjacent samples within a picture. For example, using intra-prediction, each PU is predicted from adjacent image data within the same picture using, for example, a DC prediction to find the mean value about the PU, a plane prediction to fit a plane to the PU, a direction prediction to extrapolate from adjacent data, or any other appropriate type of prediction. Inter-prediction uses the temporal correlation between pictures to derive motion-compensated predictions about blocks of image samples. For example, using inter-picture prediction, each PU is predicted using motion-compensated predictions from image data in one or more reference pictures (before or after the current picture in output order). The decision of whether to encode a picture area using inter-picture prediction or intra-picture prediction may be made, for example, at the CU level.

[0086] After performing predictions using intra-prediction and/or inter-prediction, the coding device can perform transformations and quantization. For example, after prediction, the coding device can calculate residual values corresponding to the PU. The residual values may comprise pixel difference values between the current block (PU) of the pixels being coded and the prediction block used to predict the current block (e.g., the predicted version of the current block). For example, after generating a prediction block (e.g., issuing an inter-prediction or intra-prediction), the coding device can generate a residual block by subtracting the prediction block generated by the prediction unit from the current block. The residual block contains a set of pixel difference values that quantify the difference between the pixel values of the current block and the pixel values of the prediction block. In some examples, the residual block may be represented in a two-dimensional block format (e.g., a two-dimensional matrix or array of pixel values). In such examples, the residual block is a two-dimensional representation of the pixel values.

[0087] Any residual data that may remain after the prediction has been performed is transformed using block transforms, which may be based on discrete cosine transforms, discrete sine transforms, integer transforms, wavelet transforms, other suitable transform functions, or any combination thereof. In some cases, one or more block transforms (e.g., of sizes 32x32, 16x16, 8x8, 4x4, or other suitable sizes) may be applied to the residual data in each CU. In some embodiments, TUs may be used for the transform and quantization processes implemented by the coding device. A given CU having one or more PUs may also include one or more TUs. As will be described in more detail below, the residual values may be transformed into transform coefficients using block transforms, and then quantized and scanned using TUs to generate serialized transform coefficients for entropy coding.

[0088] The encoding device can perform quantization of the conversion coefficients. Quantization provides further compression by quantizing the conversion coefficients to reduce the amount of data used to represent the coefficients. For example, quantization can reduce the bit depth associated with some or all of the coefficients. In one example, a coefficient with an n-bit value may be truncated to an m-bit value during quantization, where n is greater than m.

[0089] Once quantization is performed, the coded video bitstream includes quantization transformation coefficients, prediction information (e.g., prediction mode, motion vector, block vector, etc.), segmentation information, and any other appropriate data such as other syntax data. Different elements of the coded video bitstream may then be entropy coded by the coding device. In some examples, the coding device can utilize a predefined scan order to scan the quantization transformation coefficients to generate a serialized vector that can be entropy coded. In some examples, the coding device can perform adaptive scanning. After scanning the quantization transformation coefficients to form a vector (e.g., a one-dimensional vector), the coding device can entropy coded the vector. For example, the coding device can use context-adaptive variable-length coding, context-adaptive binary arithmetic coding, syntax-based context-adaptive binary arithmetic coding, probability-interval segmentation entropy coding, or another appropriate entropy coding technique.

[0090] The encoding device can store the encoded video bitstream and/or send the encoded video bitstream data over a communication link to a receiving device which may include a decoding device. The encoding device can decode the encoded video bitstream data by entropy decoding (e.g., using an entropy decoder) and extracting elements of one or more encoded video sequences that make up the encoded video data. The decoding device can then rescale the encoded video bitstream data and perform an inverse transform on the encoded video bitstream data. The residual data is then passed to the prediction stage of the decoding device. The decoding device then predicts blocks of pixels (e.g., PU) using intra-prediction, inter-prediction, IBC, and/or other types of prediction. In some examples, the predictions are added to the output of the inverse transform (residual data). The decoding device can output the decoded video to a video destination device which may include a display or other output device for displaying the decoded video data to a consumer of the content.

[0091] Video coding systems and techniques, as defined by various video coding standards (e.g., the HEVC video coding technique described above), may be able to retain a large portion of the information in the raw video content and may be defined a priori based on concepts of signal processing and information theory. However, in some cases, machine learning (ML)-based image and/or video systems can offer advantages over non-ML-based image and video coding systems, such as deep learning-based end-to-end video coding (DLEC) systems. As described above, many deep learning-based systems are designed as a combination of an autoencoder subnetwork (encoder subnetwork) and a second subnetwork involved in learning a probabilistic model on a quantization latent used for entropy coding. Such architectures can be seen as a combination of a transform, quantization module (encoder subnetwork) and an entropy modeling subnetwork module.

[0092] Figure 4 depicts a system 400 including a device 402 configured to perform video encoding and decoding using a deep learning-based system 410. Device 402 is coupled to a camera 407 and a storage medium 414 (e.g., a data storage device). In some implementations, the camera 407 is configured to provide image data 408 (e.g., a video data stream) to a processor 404 for encoding by the deep learning-based system 410. In some implementations, device 402 may be coupled to and/or include multiple cameras (e.g., a dual-camera system, three cameras, or any other number of cameras). In some cases, device 402 may be coupled to a microphone and/or other input devices (e.g., touch input devices such as a keyboard, mouse, touchscreen and/or touchpad, and/or other input devices). In some examples, the camera 407, storage medium 414, microphone, and/or other input devices may be part of device 402.

[0093] Device 402 is also coupled to a second device 490 via a transmission medium 418, such as one or more wireless networks, one or more wired networks, or a combination thereof. For example, the transmission medium 418 may include channels provided by a wireless network, a wired network, or a combination of wired and wireless networks. The transmission medium 418 may form part of a packet-based network, such as a local area network, a wide area network, or a global network such as the Internet. The transmission medium 418 may include routers, switches, base stations, or any other equipment that may be useful to facilitate communication from a source device to a receiving device. The wireless network may include any wireless interface or combination of wireless interfaces, and may include any suitable wireless network (e.g., the Internet or other wide area networks, packet-based networks, Wi-Fi, radio frequency (RF), UWB, Wi-Fi-Direct, cellular, Long-Term Evolution (LTE), WiMax®, etc.). A wired network may include any wired interface (e.g., fiber optic, Ethernet®, power line Ethernet, Ethernet over coaxial cable, digital signaling line (DSL), etc.). Wired and/or wireless networks may be implemented using various devices such as base stations, routers, access points, bridges, gateways, and switches. Encoded video bitstream data may be modulated according to communication standards such as wireless communication protocols and transmitted to a receiving device.

[0094] Device 402 includes one or more processors 404 (referred to herein as “processors”) coupled to memory 406, a first interface (“I/F1”) 412, and a second interface (“I/F2”) 416. The processors 404 are configured to receive image data 408 from the camera 407, from memory 406, and/or from the storage medium 414. The processors 404 are coupled to the storage medium 414 via the first interface 412 (e.g., via a memory bus) and to the transmission medium 418 via the second interface 416 (e.g., a network interface device, a wireless transceiver and antenna, one or more other network interface devices, or a combination thereof).

[0095] The processor 404 includes a deep learning-based system 410. The deep learning-based system 410 includes an encoder section 462 and a decoder section 466. In some implementations, the deep learning-based system 410 may include one or more autoencoders. The encoder section 462 is configured to receive input data 470 and process the input data 470 to generate output data 474 based at least partially on the input data 470.

[0096] In some implementations, the encoder portion 462 of the deep learning-based system 410 is configured to perform lossy compression of the input data 470 to produce output data 474, so that the output data 474 has fewer bits than the input data 470. The encoder portion 462 may be trained to compress the input data 470 (e.g., image or video frame) without using motion compensation based on any previous representation (e.g., one or more previously restored frames). For example, the encoder portion 462 may compress a video frame using only the video data from that video frame and without using any data from previously restored frames. The video frame processed by the encoder portion 462 may be referred to herein as an intra-prediction frame (I-frame). In some examples, I-frames may be generated using legacy video coding techniques (e.g., HEVC, VVC, MPEG-4, or according to other video coding standards). In such examples, the processor 404 may include, or be coupled with, a video coding device (e.g., an encoding device) configured to perform block-based intra-prediction, such as those described above with respect to the HEVC standard. In such examples, the deep learning-based system 410 may be excluded from the processor 404.

[0097] In some implementations, the encoder portion 462 of the deep learning-based system 410 may be trained to compress input data 470 (e.g., video frames) using motion compensation based on previous representations (e.g., one or more previously reconstructed frames). For example, the encoder portion 462 may compress a video frame using video data from that video frame and data from previously reconstructed frames. The video frame processed by the encoder portion 462 may be referred to herein as an intra-predicted frame (P-frame). Motion compensation may be used to determine the data of the current frame by describing how pixels from previously reconstructed frames move to new positions in the current frame along with residual information.

[0098] As illustrated, the encoder portion 462 of the deep learning-based system 410 may include a neural network 463 and a quantizer 464. The neural network 463 may include one or more convolutional neural networks (CNNs), one or more fully connected neural networks, one or more gated recurrent units (GRUs), one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures that generate intermediate data 472. The intermediate data 472 is the input to the quantizer 464. Examples of components that may be included in the encoder portion 462 are shown in Figures 6 to 10.

[0099] The quantizer 464 is configured to perform quantization and, optionally, entropy coding of intermediate data 472 to generate output data 474. The output data 474 may include quantized (and optionally entropy-coded) data. The quantization operation performed by the quantizer 464 can result in the generation of a quantized code (or data representing a quantized code generated by a deep learning-based system 410) from the intermediate data 472. The quantized code (or data representing a quantized code) may also be called a laten code or laten (denoted as z). The entropy model applied to the laten may be called a “prior” in this specification. In some examples, the quantization and/or entropy coding operation may be performed using existing quantization and entropy coding operations performed when encoding and/or decoding video data according to existing video coding standards. In some examples, the quantization and/or entropy coding operation may be performed by a deep learning-based system 410. In one exemplary example, a deep learning-based system 410 may be trained using supervised training, where residual data is used as input during training, and quantization codes and entropy codes are used as known outputs (labels).

[0100] The decoder portion 466 of the deep learning-based system 410 is configured to receive output data 474 (for example, directly from the quantizer 464 and/or from the storage medium 414). The decoder portion 466 can process the output data 474 to generate a representation 476 of the input data 470 based at least partially on the output data 474. In some examples, the decoder portion 466 of the deep learning-based system 410 may include a neural network 468 that may include one or more CNNs, one or more fully connected neural networks, one or more GRUs, one or more long short-term memory (LSTM) networks, one or more ConvRNNs, one or more ConvGRUs, one or more ConvLSTMs, one or more GANs, any combination thereof, and/or other types of neural network architectures. Examples of components that may be included in the decoder portion 466 are shown in Figures 6 to 10.

[0101] The processor 404 is configured to send output data 474 to at least one of the transmission medium 418 or the storage medium 414. For example, the output data 474 may be stored in the storage medium 414 for later retrieval and decoding (or restoration) by the decoder portion 466 to generate a representation 476 of the input data 470 as restoration data. The restoration data may be used for various purposes, such as for playback of the video data encoded/compressed to generate the output data 474. In some implementations, the output data 474 may be decoded in another decoder device matching the decoder portion 466 (for example, in device 402, in a second device 490, or in another device) to generate a representation 476 of the input data 470 as restoration data. For example, the second device 490 may contain a decoder matching (or substantially matching) the decoder portion 466, and the output data 474 may be sent to the second device 490 via the transmission medium 418. The second device 490 can process the output data 474 to generate a representation 476 of the input data 470 as reconstructed data.

[0102] The components of System 400 may include, and/or be implemented using, one or more programmable electronic circuits (e.g., microprocessors, graphics processing units (GPUs), digital signal processors (DSPs), central processing units (CPUs), and/or other suitable electronic circuits) to perform the various operations described herein, and may also include, and/or be implemented using, computer software, firmware, or any combination thereof.

[0103] Although the system 400 is shown to include several components, those skilled in the art will understand that the system 400 may include more or fewer components than those shown in Figure 4. For example, the system 400 may also include, or be part of, a computing device including input devices and output devices (not shown). In some implementations, System 400 may also include, or be part of, a computing device including one or more memory devices (e.g., one or more random access memory (RAM) components, read-only memory (ROM) components, cache memory components, buffer components, database components, and/or other memory devices), one or more processing devices (e.g., one or more CPUs, GPUs, and/or other processing devices) communicating with and/or electrically connected to one or more memory devices, one or more wireless interfaces for performing wireless communication (e.g., including one or more transceivers and baseband processors per wireless interface), one or more wired interfaces for performing communication via one or more hardware connections (e.g., serial interfaces such as Universal Serial Bus (USB) inputs, Lightning connectors, and/or other wired interfaces), and/or other components not shown in Figure 4.

[0104] In some implementations, the system 400 may be locally implemented by and/or included in a computing device. For example, the computing device may include a mobile device, a personal computer, a tablet computer, a virtual reality (VR) device (e.g., a head-mounted display (HMD) or other VR device), an augmented reality (AR) device (e.g., an HMD, AR glasses, or other AR device), a wearable device, a server (e.g., in a Software as a Service (SaaS) system or other server-based system), a television, and/or any other computing device having the resource capacity to perform the techniques described herein.

[0105] In one example, a deep learning-based system 410 may be incorporated into a portable electronic device that includes a memory 406 coupled to a processor 404 and configured to store instructions executable by the processor 404, and an antenna and a wireless transceiver coupled to the processor 404 and operable to transmit output data 474 to a remote device.

[0106] As described above, deep learning-based systems are typically designed to handle non-subsample input formats such as RGB or YUV4:4:4. Examples of image and video coding schemes targeting RGB input include J. Balle, D. Minnen, S. Singh, S. J. Hwang, N. Johnson, "Variational image compression with a scale hyperprior", ICLR, 2018 (referred to as "J. Balle Paper"), and D. Minnen, J. Balle, G. Toderici, “Joint Autoregressive and Hierarchical Priors for Learned Image Compression,” CVPR, 2018, is included herein by reference in its entirety for all purposes.

[0107] Figure 5 shows an example of a deep learning-based system 500. The g _a subnetwork and g _s subnetwork in the deep learning-based system in Figure 5 correspond to an encoder subnetwork (e.g., encoder portion 462) and a decoder subnetwork (e.g., decoder portion 466), respectively. The g _a subnetwork and g _s subnetwork in Figure 5 are designed for 3-channel RGB input, and all three R input channels, G input channels, and B input channels pass through and are processed by the same neural network layers (convolutional layers and generalized neural activity normalization (GDN) layers). The neural network layers may include a convolutional layer (including convolutional layer 510) that performs convolution operations and a GDN and/or inverse GDN (IGDN) nonlinear layer that implements local partition normalization. Local neural activity normalization is a type of transformation shown to be particularly suitable for image density modeling and compression. Deep learning-based systems (such as those shown in Figure 5) target input channels with similar statistical properties, such as RGB data (where the statistical properties of different R, G, and B channels are similar).

[0108] While many deep learning-based systems are designed to handle RGB inputs, most image and video coding systems use the YUV input format (e.g., the YUV4:2:0 input format, often). In the YUV format, the chrominance (U and V) channels can be subsampled relative to the luminance (Y) channel. Subsampling has minimal impact on visual quality (e.g., no significant or noticeable impact on visual quality). Subsampling formats include the YUV4:2:0 format, the YUV4:2:2 format, and/or other YUV formats. The correlation between channels is reduced in the YUV format, which may not be the case in other color formats (e.g., the RGB format). Furthermore, the statistics of the luminance (Y) channel and the chrominance (U and V) channels are different. For example, the U and V channels have less variability compared to the luminance channel, while in the RGB format, for example, the statistical properties of the R, G, and B channels are quite similar. A video codec is designed according to the characteristics of the input data (for example, a codec can encode and/or decode data according to the input format of the data). For example, if the chrominance channel of a frame is subsampled (for example, the chrominance channel has half the resolution compared to the luminance channel), when the codec predicts blocks of the frame for motion compensation, the luminance blocks will be twice as large in both width and height compared to the chrominance blocks. In another example, the codec can, among other things, determine how many pixels will be encoded or decoded for chrominance and luminance.

[0109] To support the YUV format (e.g., the YUV4:2:0 format), the deep learning-based architecture must be modified. For example, replacing RGB input data (which most deep learning-based systems are designed to process, as described above) with YUV4:4:4 input data (where all channels have the same dimensions) will degrade the performance of the deep learning-based system processing the input data due to the different statistical properties of the luminance (Y) channel and the chrominance (U and V) channels. As described above, the chrominance (U and V) channels are subsampled in some YUV formats, such as YUV4:2:0. For example, for content with the YUV4:2:0 format, the resolution of the U and V channels is half that of the Y channel (the U and V channels are one-quarter the size of the Y channel due to their half width and height). Such subsampling can make the input data incompatible with the input of the deep learning-based system. The input data is the information that the deep learning-based system is attempting to encode and/or decode (for example, a YUV frame containing three channels: a luminance (Y) channel and chrominance (U and V) channels).

[0110] In some end-to-end video coding deep learning-based systems, autoencoders are used to code intraframes, motion vectors (e.g., dense optical flow), and residuals of motion-compensated frames relative to the original frame. For example, a flow autoencoder can be used to learn to code optical flow and scale space together, while a residual autoencoder codes the residuals between warp-predicted frames and the original frame, all within the RGB domain.

[0111] As stated above, this specification describes systems and techniques that provide ML-based systems (including, for example, one or more deep learning-based architectures) that efficiently support one or more YUV formats (e.g., YUV4:2:0 format). Deep learning-based architectures can encode and/or decode video data containing standalone frames (or images) and/or multiple frames. For example, an ML-based system can take the rumor components of the current frame and the rumor components of previously restored frames as input, which can be restored by previous instances of the ML-based system. The ML-based system can process the rumor components of the current and previous frames to estimate the motion information (e.g., flow information such as optical flow information) of the rumor components of the current frame. Then, using the rumor components of the current frame, the ML-based system can determine (e.g., estimate) the motion estimate (e.g., flow information such as optical flow information) of one or more chroma components of the current frame. Such techniques can be performed for all components of a frame. Further details are described below.

[0112] Figure 6 shows an example of a neural network architecture for a deep learning-based system 600 configured to perform video coding. The neural network architecture in Figure 6 includes an intra-prediction engine 602 and an inter-prediction engine 610. The intra-prediction engine 602 and the inter-prediction engine 610 may include autoencoders (e.g., variational autoencoders (VAEs)) as shown in Figure 6, but may include other types of neural network architectures in other implementations. As shown, the intra-prediction engine 602 processes the pixel information of the input frame 604 to obtain a latent representation of the input frame 604 (

It generates (shown as follows). For each pixel of the input frame 604, the luma component (

(shown as) and two chromatic components ( )

and

(As shown) and includes. The latent representation can also be called a bitstream, which contains the number of bits that are the coded version of the input frame 604. Latent representation

Based on (or a latent representation/bitstream received from another device), the decoder subnetwork of the intra-prediction engine 602 reconstructs the frame 606 (where the "hat" on the component indicates the reconstructed value).

,

,

This can generate (shown as shown), which is the restored version of input frame 604.

[0113] The interpretation engine 610 includes a flow engine 618, a residual engine 620, and a warping engine 622. As shown in the figure, the flow engine 618 calculates the lumen component of the current frame 614 (at time t) (

(shown as) and the restored luma component of the previous frame 615 (at the previous time t-1) (

The input is obtained as (as shown). Ruma component

and luma components

Using this, the flow engine 618 generates the rumor component of the current frame 614.

The latent representation of motion information (for example, flow information) (

This generates (as shown). The motion information may include optical flow information (e.g., multiple motion or displacement vectors, possibly per pixel or per sample scale component), which indicates the motion of the pixels in the current frame 614 (at time t) relative to the previous frame 615 (at time t-1). Latent representation.

This can also be called a bitstream, and is the rumor component of the current frame 614.

It can include a number of bits representing the coding version. The flow engine 618 uses the lumen component of the current frame 614, rather than the chroma component.

To process this, use latent representation.

The bitstream is smaller in size compared to using all components of the current frame 614 to determine motion information.

[0114] Luma component

Latent expression of

(Or, using a latent representation or bitstream received from another device representing the rumor components of the frame), the flow engine 618 calculates the rumor components of the current frame 614.

The motion information (shown as f ^L ) is determined, and the chroma component of the current frame 614 is also determined.

,

The motion information (shown as f ^C ) for the chroma component is determined. Details of determining or estimating the motion information for the chroma component based on the determined motion information for the chroma component are described below with reference to Figures 7A and 7B.

[0115] The warping engine 622 is the current frame 614 (at time t).

and chroma components

,

The system is configured to perform warping using motion information ( ^fL and ^fC ) determined for the current frame 614. For example, the warping engine 622 is configured to perform warping using motion information (fL and fC) determined for the current frame 614.

and chroma components

,

The pixels of the current frame 614 (at time t) can be warped by an amount indicated by the motion information (f ^L and f ^C ). In some embodiments, the warping engine 622 can perform spatial scale flow (SSF) warping. For example, SSF warping can apply trilinear interpolation to generate interframe predictions from a learned scale flow vector, where the predictor can be formulated as follows:

[0116] The above trilinear interpolation is a lunar interpolation.

and chroma components

Based on the motion information (f ^L and f ^C ), one or more warping parameters can be determined and performed component by component (for example, for each lumen component and each individual U and V chroma component). For example, the warping parameters represent the horizontal component (x direction) of the motion vector or displacement vector.

This represents the vertical component (y-direction) of the motion vector or displacement vector.

This may include s (called the scale field), which represents a steppedly smoothed version of the reconstructed frame, combined with spatial motion/displacement information (v _x and v _y ).

[0117] The output from the warping engine 622 (after warping has been performed by the warping engine 622) includes the predictions shown in Figure 6 as _PY , _PU , and _PV , where _PY is the rumor component

In response to the prediction, _PU is the chroma component

In response to the prediction, _PV is the chroma component of the current frame 614.

This corresponds to the prediction.

[0118] Next, the deep learning-based system 600 processes the corresponding rumor for the current frame 614.

and chroma

,

The residual signal can be obtained by subtracting the predicted _PY , _PU , and _PV from the components, which includes the residual signal _rY of the lumens component and the chromens component.

The residual signal _rU and the chromatic component

The residual signal r _V is included. The residual engine 620 provides a latent representation of the residual (

It is possible to generate (as shown below). Latent representation of residuals

Using (or a latent representation of the residual received from another device), the residual engine 620 generates the restored residual signal of the luma component.

Chroma components

The restored residual signal

, and chroma components

The restored residual signal

It can generate the reconstructed residuals of the current frame, including the predicted _PY , _PU , and _PV . The deep learning-based system 600 then generates the reconstructed residuals.

,

,

In addition, the restored frame 616 can be generated.

[0119] Figure 7A shows the luma component of the current frame (at time t).

and the restored luma component of the previous frame (at time t-1).

This figure shows an example of a flow engine 718 operating in a manner collectively represented as a luma component 722. As mentioned above, in some cases, the flow engine 718 may be implemented as an autoencoder (VAE _flow ). In some cases, as shown in Figure 7A, a combined deep learning-based architecture can be designed, in which the flow engine 718 operates in the current frame

And the previously restored frame

Both lumern components are used to estimate lumern motion information (e.g., SSFf ^L ) and chromern motion information (e.g., SSFf ^C ). For example, as described herein, chromern motion information (e.g., f ^C ) may be derived based on lumern motion information (e.g., f ^L ).

[0120] As shown in Figure 7A, the rumor component of the current frame

To determine the motion information ( ^fL ), the rumor component of the current frame is used.

, and the restored luma component of the previous frame.

This is processed by several convolutional and activation layers (collectively shown as forward pass 723). The notation "↓2" and "↑2" in Figure 7A refers to stride values, where ↓2 refers to stride 2 (in the case of downsampling indicated by "↓") and ↑2 also refers to stride 2 (in the case of upsampling indicated by "↑"). For example, convolutional layer 724 applies a 5x5 convolutional filter in the horizontal and vertical dimensions with a stride value of 2 to process the input luma components.

and

This is downsampled by 1/4. The output obtained as a result of convolutional layer 724 is the rumor component of the current frame.

This is an array of N feature values (corresponding to N channels) representing the rumor motion information (f ^L ). The notation "2/N" represents two input channels and N output channels. Next, the nonlinear layer following the convolutional layer 724 can process the feature values output by the convolutional layer 724. Each successive convolutional and nonlinear layer can process the features output by the previous layer until the final convolutional layer 725 of the forward pass 723 outputs features to the bottleneck portion 726 of the flow engine 718.

[0121] The output of the forward pass 723 is processed by the bottleneck portion 726 of the flow engine 718 to obtain the rumor component of the current frame.

A bitstream or latent representing the lumens motion information (f ^L ) is generated. The bottleneck portion 726 may include a quantization engine and an entropy coding engine in the forward pass 723, as well as an entropy decoding engine and an inverse quantization engine on the backward pass 728 of the flow engine 718. For example, the quantization engine can perform quantization on the features output by the final convolutional layer 725 of the forward pass 723 to produce a quantized output. The entropy coding engine can entropy code the quantized output from the quantization engine to produce a bitstream. In some cases, the entropy coding engine can perform entropy coding using pliers generated by the hyperplier network. The neural network system can output the bitstream to a server device or system for storage, to send to another device, and/or output the bitstream in other ways.

[0122] The backward path 728 may, in some cases, be a decoder subnetwork of the neural network system of the flow engine 718, or a decoder subnetwork of the neural network system of another flow engine (in another device). The entropy decoding engine of the flow engine 718 can entropy decode the bitstream output by the entropy coding engine of the bottleneck 726 (or the entropy coding engine of the bottleneck of another flow engine) and output the entropy-decoded data to the inverse quantization engine of the backward path 728. The entropy decoding engine can perform entropy decoding using pliers generated by the hyperplier network. The dequantization engine can dequantize the data.

[0123] Next, the convolutional and deactivation layers of the backward pass 728 process the dequantized data from the bottleneck 726 to obtain the lumen components of the current frame.

Motion information 729( ^fL ) can be generated. Motion information 729( ^fL ) is the rumor component of the current frame.

Motion vectors such as the motion vector of each sample (for example, having a magnitude in the horizontal or x direction and a magnitude in the vertical or y direction) may be included. In some cases, motion information 729( ^fL ) may further include a scale component. For example, as shown in Figure 7A for illustrative purposes, motion information 729 is

Ingredients and

It contains the component and the ^SL component. As described above,

,

The ^SL components are used by the warping engine 622 in equation (1) to warp the pixels of the current frame 614 (at time t) to generate predicted _PY , _PU , and _PV .

[0124] Current frame's lunar components

After learning the motion information 729 (f ^L ), the flow engine 718 can determine or predict the motion information 731 (f ^C ) of the chroma component of the current frame. For example, the flow engine 718 can learn the motion information 731 (f ^C ) of the chroma component

The motion information 729( ^fL ) can be subsampled. The motion information 731( ^fC ) of the chroma component may include motion vectors (for example, having a magnitude in the horizontal or x direction and a magnitude in the vertical or y direction), such as the motion vector of each sample of the chroma component in the current frame. In some cases, the motion information 731( ^fC ) may further include a scale component. For example, as shown in Figure 7A for illustrative purposes, the motion information 731( ^fC ) of the chroma component in the current frame is:

Ingredients and

It includes the component and the S ^C component. Similar to the motion information 729 (f ^L ) of the chroma component, the chroma motion information 731 (f ^C )

,

The S ^C components are used by the warping engine 622 in equation (1) to warp the pixels of the current frame 614 (at time t) to generate predicted P _Y , _PU , and _PV .

[0125] In some embodiments, the convolutional layer 730 with downsampling is used to obtain the lumen component of the current frame.

Based on the motion information 729 (f ^L ), the motion information 731 (f ^C ) of the chroma component of the current frame can be trained (for example, using unsupervised learning or training). In one exemplary example, a training set that can be used to train the flow engine 718 may include lumar and chroma motion information (as ground truth). The lumar motion information may be input to the neural network of the flow engine 718, and the resulting chroma motion information output from the flow engine 718 may be minimized using the chroma motion information of the ground truth with a loss function (e.g., L1 or sum of absolute differences, L2 Normal or sum of squared differences, or other loss function).

[0126] The convolutional layer 730 is shown in Figure 7A as |3/3|5×5conv↓2|. The notation "3/3" indicates that there are three input channels that result in three output channels. As described above, the notations "↓2" and "↑2" refer to stride values, where ↓2 refers to a downsampling stride of 2 (indicated by "↓") and ↑2 refers to an upsampling stride of 2 (indicated by "↑"). For example, the convolutional layer 730 applies a 5×5 convolutional filter in the horizontal and vertical dimensions with a stride value of 2 to create the lumen component.

The motion information 729( ^fL ) is downsampled by a quarter (for example, in the YUV4:2:0 format). In some examples, the convolutional layer 730 may be trained to downsample by other factors in other formats (for example, the YUV4:2:2 format). The output resulting from the convolutional layer 724 is a 3x3 array of feature values (corresponding to the three channels), which is the luma component.

This is a downsampled version of motion information 729( ^fL ).

[0127] In other embodiments (not shown in Figure 7A), the motion information 731(f ^C ) of the chroma component of the current frame is the luma component

The motion information 729( ^fL ) can be obtained by directly subsampling it. For example, the flow engine 718 can determine the chroma flow without processing the luma flow using the convolutional layer 730. In one exemplary example, instead of the convolutional layer 730, the flow engine 718 may include a subsampler (which may be separated from the neural network of the flow engine 718) that can directly subsample the luma motion information 729( ^fL ) to obtain the chroma motion information 731( ^fC ).

[0128] Figure 7B shows an example of a subsampling engine 735 for subsampling lumar motion information determined for the current frame (for example, using the flow engine 718 in Figure 7A) to obtain chroma motion information for the current frame. For illustrative purposes, a simplified example is provided with each channel N (N=2) of lumar motion information 732 having a 4x4 (4 rows and 4 columns) resolution with a total of 16 flow motion or displacement vectors. The subsampling engine 735 subsamples or downsamples the lumar motion information 732 to generate or obtain chroma motion information 738, which is a subsampled/downsampled version of the lumar motion information 732.

[0129] The exemplary example in Figure 7B shows chroma motion information 738, which is one-quarter the size of chroma motion information 732. For example, as mentioned above, for content with the YUV 4:2:0 format, the resolution of the U and V channels is half the resolution of the Y channel (the size of the U and V channels is one-quarter of the Y channel because the width and height are halved). The subsampling engine 735 may be trained or otherwise configured to handle formats other than the 4:2:0 format, in which case subsampling may include generating chroma information with a different resolution than that shown in Figure 7A.

[0130] In some embodiments, as described above, the subsampling engine 735 may include the convolutional layer 730 in Figure 7A, which can be trained (for example, using unsupervised learning or training) to determine chroma motion information 738 from luma motion information 732. In other embodiments, the subsampling engine 735 may include a subsampler that directly subsamples luma motion information 732 to obtain chroma motion information 738.

[0131] The convolution or transformation layers of the forward pass 723 and backward pass 728, as well as the number of channels in the bottleneck (M) (shown as N in Figure 7A), can be set to any appropriate value. In one exemplary example, the number of channels N may be selected as N = 192 and M = 128. Sequential smoothed versions of the restored frames (associated with scale fields s) can be obtained using filtering or smoothing operators. In one example, Gaussian blur filters of different widths can be used. In another example, a Gaussian pyramid with sequential filtering and interpolation can be used to generate smoothed versions of the restored frames. Furthermore, an arbitrary number of scales S can be used. In one example, the scale S can be set to S = 3, and the scale level is

This can be selected as follows: Here, _σ₀ can represent the Gaussian filter width.

[0132] On the other hand, the nonlinear activation layer in Figure 7A is shown as PReLU for illustrative purposes, but other types of nonlinear activation layers can be used, such as a generalized division-normalization (GDN) layer, or a combination of a PReLU layer and a GDN layer.

[0133] In some examples, to effectively support one or more YUV formats (e.g., YUV4:2:0), the intra-prediction engine 602 and residual engine 620 in Figure 6 may be designed based on the general neural network architectures shown in Figures 8A, 9, or 10. For example, the architectures shown in Figures 8A, 9, and 10 may be configured to process input data having the YUV4:2:0 format. In some examples, neural network architectures similar to those shown in Figures 8A, 9, or 10 may be used to encode and/or decode other types of YUV content (e.g., content having the YUV4:4:4 format, YUV4:2:2 format, etc.) and/or other input formats. In some cases, each architecture shown in Figures 8A, 9, and 10 includes a residual autoencoder operating with YUV (e.g., 4:2:0) residuals.

[0134] Figure 8A shows an example of a front-end neural network system 800 that can be configured to directly link 4:2:0 input (Y, U, and V) data. As shown in Figure 8A, in the encoder subnetwork (also called the forward path) of the neural network system, the branched lumen and chroma channels (lumen Y channel 802 and U and V chroma channels 804) are combined using a 1x1 convolutional layer 806, and then a nonlinear layer 808 (also called a nonlinear operator) is applied. Similar operations are performed in the decoder subnetwork (also called the backward path) of the neural network system, but in the reverse order. For example, as shown in Figure 8A, an inverse nonlinear layer 809 (also called an inverse nonlinear operator) is applied, the Y and U, V channels are separated using a 1x1 convolutional layer 813, and the separate Y and U, V channels are processed using their respective inverse nonlinear layers 815, 816 and convolutional layers 817, 818.

[0135] The first two neural network layers in the encoder subnetwork of the neural network system 800 in Figure 8A include a first convolutional layer 811 (indicated as Nconv|3×3|↓1), a second convolutional layer 810 (indicated as Nconv|5×5|↓2), a first nonlinear layer 814, and a second nonlinear layer 812. The last two neural network layers in the decoder subnetwork of the front-end neural network architecture in Figure 8A include a first inverse nonlinear layer 816, a second inverse nonlinear layer 815, a first convolutional layer 818 (indicated as 2conv|3×3|↑1) for generating the restored chrominance (U and V) components of the frame, and a second convolutional layer 817 (indicated as 1conv|5×5|↑2) for generating the restored luminance (Y) component of the frame. The notation "Nconv" refers to the number of output channels (N) of a given convolutional layer (corresponding to the number of output features), where N is the value defining the number of output channels. The notations 3x3 and 5x5 indicate the size of each convolutional kernel (e.g., a 3x3 kernel and a 5x5 kernel). The notations "↓1" and "↓2" refer to stride values, where ↓1 refers to a stride of 1 (for downsampling indicated by "↓") and ↓2 refers to a stride of 2 (for downsampling). The notations "↑1" and "↑2" refer to stride values, where ↑1 refers to a stride of 1 (for upsampling indicated by "↑") and ↑2 refers to a stride of 2 (for upsampling).

[0136] For example, the convolutional layer 810 downsamples the input lumen channel 802 by a factor of four by applying a 5x5 convolutional filter in the horizontal and vertical dimensions with a stride value of 2. The resulting output from the convolutional layer 810 is an array of N feature values (corresponding to N channels). The convolutional layer 811 processes the input chroma (U and V) channel 804 by applying a 3x3 convolutional filter in the horizontal and vertical dimensions with a stride value of 1. The resulting output from the convolutional layer 811 is an array of N feature values (corresponding to N channels). The array of feature values output by the convolutional layer 810 has the same dimensions as the array of feature values output by the convolutional layer 811. Then, the nonlinear layer 812 can process the feature values output by the convolutional layer 810, and the nonlinear layer 814 can process the feature values output by the convolutional layer 811.

[0137] Next, the 1x1 convolutional layer 806 can process the feature values output by the nonlinear layers 812 and 814. The 1x1 convolutional layer 806 can generate a linear combination of the features associated with the lumern channel 802 and the chroma channel 804. The linear combination operation acts as a cross-channel mixture per value of the Y and UV components, resulting in predictions of cross-components (e.g., cross-luminance and chrominance components) that improve coding performance. Each 1x1 convolutional filter in the 1x1 convolutional layer 806 can include its respective scaling factor applied to the corresponding Nth channel of the lumern channel 802 and the corresponding Nth channel of the chroma channel 804.

[0138] Figure 8B shows an exemplary operation of a 1x1 convolutional layer 838. As described above, N represents the number of output channels. As shown in Figure 8B, 2N channels are provided as input to the 1x1 convolutional layer 838, including N channel chroma (combined U and V) outputs 832 and N channel lumer (Y) outputs 834. In the example in Figure 8B, the value of N is equal to 2, indicating two channels of value for the N channel chroma outputs 832 and two channels of value for the N channel lumer outputs 834. Referring to Figure 8A, the N channel chroma outputs 832 may be outputs from the nonlinear layer 814, and the N channel lumer outputs 834 may be outputs from the nonlinear layer 812.

[0139] The 1x1 convolutional layer 838 processes 2N channels, performs a feature-specific linear combination of the 2N channels, and then outputs N channel sets of features or coefficients. The 1x1 convolutional layer 838 includes two 1x1 convolutional filters (based on N=2). The first 1x1 convolutional filter is denoted by the value _S1 , and the second 1x1 convolutional filter is denoted by the value _S2 . The value _S1 represents the first scaling factor, and the value _S2 represents the second scaling factor. In one exemplary example, the value _S1 is equal to 3 and the value _S2 is equal to 4. Each of the 1x1 convolutional filters in the 1x1 convolutional layer 838 has a stride value of 1, indicating that the scaling factors _S1 and _S2 are applied to the respective values in the UV output 832 and the Y output 834.

[0140] For example, the scaling factor _S1 of the first 1x1 convolutional filter is applied to each value in the first channel (C1) of the UV output 832 and each value in the first channel (C1) of the Y output 834. When each value in the first channel (C1) of the UV output 832 and each value in the first channel (C1) of the Y output 834 are scaled by the scaling factor _S1 of the first 1x1 convolutional filter, the scaled values are combined into the first channel (C1) of the output value 839. The scaling factor _S2 of the second 1x1 convolutional filter is applied to each value in the second channel (C2) of the UV output 832 and each value in the second channel (C2) of the Y output 834. After each value of the second channel (C2) of the UV output 832 and each value of the second channel (C2) of the Y output 834 are scaled by the scaling factor _S2 of the second 1x1 convolutional filter, the scaled values are combined into the second channel (C2) of the output value 839. As a result, the four Y and UV channels (two Y channels and two combined UV channels) are mixed and combined into two output channels C1 and C2.

[0141] Returning to Figure 8A, the output of the 1x1 convolutional layer 806 is processed by additional nonlinear and additional convolutional layers of the encoder subnetwork. The bottleneck 820 may include a quantization engine and an entropy coding engine on the encoder subnetwork (or forward path), as well as an entropy decoding engine and an inverse quantization engine (or backward path) on the decoder subnetwork. The quantization engine can perform quantization on the features output by the last neural network layer 819 of the encoder subnetwork to generate a quantized output. The entropy coding engine can entropy code the quantized output from the quantization engine to generate a bitstream. In some cases, the entropy coding engine can use pliers generated by the hyperplier network to perform entropy coding. The neural network system may output a bitstream for storage, transmission to another device, server device, or server system, and/or otherwise output the bitstream.

[0142] A decoder subnetwork of a neural network system or a decoder subnetwork of another neural network system (in another device) can decode a bitstream. The entropy decoding engine of the (decoder subnetwork) bottleneck 820 can entropy decode the bitstream and output the entropy-decoded data to the inverse quantization engine of the decoder subnetwork. The entropy decoding engine can use pliers generated by the hyperplier network to perform entropy decoding. The inverse quantization engine can inverse quantize the data. The inverse quantized data can be processed by several convolutional layers and several inverse nonlinear layers of the decoder subnetwork.

[0143] After being processed by several convolutional and nonlinear layers, the 1x1 convolutional layer 813 can process the data output by the final inverse nonlinear layer 809. The 1x1 convolutional layer 813 may include 2N convolutional filters that can split the data into Y channel features and combined UV channel features. For example, each of the N channels output by the inverse nonlinear layer 809 may be processed using 2N 1x1 convolutions (which result in scaling) of the 1x1 convolutional layer 813. For each scaling factor n _i corresponding to the output channels (from a total of 2N output channels) applied to the N input channels, the decoder subnetwork can perform a sum over the N input channels, resulting in 2N outputs. In one exemplary example, with a scaling factor n _i , the decoder subnetwork can apply the scaling factor n _i to the N input channels, sum the results, thereby resulting in one output channel. The decoder subnetwork can perform this operation for 2N different scaling factors (for example, scaling factor _n1 , scaling factor _n2 , and up to scaling factor _n2N ).

[0144] The Y channel features output by the 1x1 convolutional layer 813 can be processed by the inverse nonlinear layer 815. The combined UV channel features output by the 1x1 convolutional layer 813 can be processed by the inverse nonlinear layer 816. The convolutional layer 817 can process the Y channel features and output a sample of the reconstructed Y channel or reconstructed frame per pixel (e.g., luminance sample or pixel), indicated as the reconstructed Y component 824. The convolutional layer 818 can process the combined UV channel features and output a sample of the reconstructed U channel or reconstructed frame per pixel (e.g., chrominance blue sample or pixel) and a sample of the reconstructed V channel or reconstructed frame per pixel (e.g., chrominance red sample or pixel), indicated as the reconstructed U and V components 825.

[0145] In some examples, different variations of the architecture of Figure 8A with different nonlinear operators can be used as the intra-prediction engine 602 and the residual engine 620. For example, Figures 9 and 10 show the front-end architecture of Figure 8A configured to process data having a YUV format (e.g., YUV4:2:0 input data with Y, U, and V components). In the neural network system 900 of Figure 9, on the encoder side, the branched lumen and chroma channels are combined using a 1x1 convolutional layer (similar to that in Figure 8A), and then the GDN nonlinear operator is applied. In the neural network system 1000 of Figure 10, on the encoder side, the branched lumen and chroma channels are combined using a 1x1 convolutional layer (similar to that in Figure 8A), and then the PReLU nonlinear operator is applied. In one example, both VAE _res and VAE _intra can use the variations shown in Figure 9. In another example, both VAE _res and VAE _intra can use the modification shown in Figure 10. In another example, VAE _res can use the modification shown in Figure 9, and VAE _intra can use the modification shown in Figure 10. In another example, VAE _intra can use the modification shown in Figure 9, and VAE _res can use the modification shown in Figure 10.

[0146] Figure 11 is a flowchart showing an example of a process 1100 for processing video data. In block 1102, process 1100 includes acquiring input video data by a machine learning system. The input video data is the lumen component of the current frame (for example, the current frame (at time t) in Figure 7A).

It includes at least one luminance component of the ) . In some cases, the input video data includes at least one luminance component of a previously restored frame (for example, the restored luminance component of the previous frame (at time t-1) in Figure 7A).

It includes at least one restored luminance component, which can be called a chrominance component. In some embodiments, the current frame includes a video frame. In some cases, one or more chrominance components include at least one chrominance blue component and at least one chrominance red component. In some embodiments, the current frame has a luminance-chrominance (YUV) format. In some cases, the YUV format is a YUV4:2:0 format.

[0147] In block 1104, the process includes using a machine learning system to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame, using at least one luminance component of the current frame. In some embodiments, process 1100 may include determining motion information for at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored lumen component of the previous frame. In some cases, process 1100 may further include determining motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame. In some cases, the motion information for one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. For example, referring to Figure 7A as an illustrative example, the flow engine 718 uses the current frame

And the previously restored frame

Using both of the luma components, the current frame

Luminance information (e.g., SSFf ^L ) and chroma motion information (e.g., SSFf ^C ) can be estimated. As described above, chroma motion information (e.g., f ^C ) 731 can be derived using the convolutional layer 730 based on luma motion information (e.g., f ^L ) 729. In some cases, motion information for one or more chrominance components of the current frame is determined at least partially by sampling motion information determined for at least one luminance component of the current frame.

[0148] In some embodiments, process 1100 includes determining warping parameters for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame by a machine learning system using motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame. In some embodiments, the warping parameters for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame include spatial scale flow (SSF) warping parameters. In some cases, the spatial scale flow (SSF) warping parameters include learned scale flow vectors. Referring to Figure 6 as an illustrative example, the warping parameters represent the horizontal component (in the x-direction) of the motion or displacement vector.

This represents the vertical component (in the y-direction) of the motion or displacement vector.

This can include s (called the scale field), which represents a steppedly smoothed version of the reconstructed frame combined with spatial motion/displacement information (v _x and v _y ).

[0149] Process 1100 may further include determining one or more interframe predictions of the current frame (e.g., predictors _PY , _PU , and _PV in Figure 6) using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame. In some cases, one or more interframe predictions are determined at least partially by applying an interpolation operation using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame. In one exemplary example, the interpolation operation includes a trilinear interpolation operation.

[0150] In some examples, the processes described herein may be executed by a computing device or apparatus, such as a computing device having the computing device architecture 1200 shown in Figure 11. In one example, one or more processes may be executed by a computing device having the computing device architecture 1200, which implements any one or more of the neural network architectures shown in Figure 6 and/or Figures 7A, 7B, 8A, 9, and/or 10. In some examples, the computing device may include, or be part of, a mobile device (e.g., a mobile phone, a tablet computing device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a television, a vehicle (or a computing device in a vehicle), a robotic device, and/or any other computing device having the resource capacity to perform the processes described herein.

[0151] In some cases, a computing device or apparatus may include various components such as one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, one or more transmitters, receivers, or combined transmitter-receivers (for example, called transceivers), one or more cameras, one or more sensors, and/or other components configured to perform the steps of the process described herein. In some examples, a computing device may include a display, a network interface configured to communicate and/or receive data, any combination thereof, and/or other components. The network interface may be configured to communicate and/or receive Internet Protocol (IP) based data or other types of data.

[0152] Components of a computing device may be implemented within a circuit. For example, a component may include, and/or be implemented using, one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), a neural processing unit (NPU), and/or other suitable electronic circuits) to perform the various operations described herein, and/or may include, and/or be implemented using, computer software, firmware, or any combination thereof.

[0153] The processes described herein may be represented as logical flowcharts, where their operations represent a sequence of operations that can be performed in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored in one or more computer-readable storage media that, when executed by one or more processors, performs the enumerated operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not to be interpreted as limiting, and any number of described operations may be combined in any order and/or in parallel to implement a process.

[0154] Furthermore, the processes described herein may be executed under the control of one or more computer systems consisting of executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed collectively by hardware or in combination thereof on one or more processors. As stated above, the code may be stored in a computer-readable or machine-readable storage medium in the form of a computer program comprising multiple instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-temporary.

[0155] Figure 12 shows an exemplary computing device architecture 1200 of an exemplary computing device that can implement various techniques described herein. In some examples, the computing device may include a mobile device, a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a video server, a vehicle (or a computing device in a vehicle), or other device. For example, the computing device architecture 1200 can implement the system shown in Figure 6. The components of the computing device architecture 1200 are shown communicating with each other using a connection 1205 such as a bus. The exemplary computing device architecture 1200 includes a processing unit (CPU or processor) 1210 and a computing device connection 1205 that connects various computing device components to the processor 1210, including a computing device memory 1215 such as read-only memory (ROM) 1220 and random access memory (RAM) 1225.

[0156] The computing device architecture 1200 may include a cache of high-speed memory that is directly connected to the processor 1210, very close to the processor 1210, or integrated as part of the processor 1210. The computing device architecture 1200 may copy data from memory 1215 and/or storage device 1230 to the cache 1212 for rapid access by the processor 1210. In this way, the cache can provide performance improvements that avoid delays in the processor 1210 while waiting for data. These and other modules may be able to control the processor 1210 to perform various actions, or may be configured to control the processor 1210. Other computing device memories 1215 may also be available for use. Memory 1215 may include several different types of memory with different performance characteristics. The processor 1210 may include any general-purpose processor, hardware or software services such as services 1232, 2234, and 31236 stored in the storage device 1230 and configured to control the processor 1210, as well as a dedicated processor in which software instructions are incorporated into the processor design. The processor 1210 may be a self-contained system including multiple cores or processors, a bus, a memory controller, a cache, etc. The multi-core processor may be symmetrical or asymmetrical.

[0157] To enable user interaction with the computing device architecture 1200, the input device 1245 can represent any number of input mechanisms, such as a microphone for voice, a touch screen for gesture or graphical input, a keyboard, a mouse, motion input, or voice. The output device 1235 can also be one or more of several output mechanisms known to those skilled in the art, such as a display, projector, television, or speaker device. In some cases, a multimodal computing device can allow the user to provide multiple types of inputs for communication with the computing device architecture 1200. The communication interface 1240 can generally control and manage user inputs and computing device outputs. There are no restrictions on operating on any particular hardware configuration, and therefore the basic features described herein may be readily substituted with improved hardware or firmware configurations as they are developed.

[0158] The storage device 1230 is a non-volatile memory and may be a hard disk or other type of computer-readable medium capable of storing computer-accessible data, such as a magnetic cassette, flash memory card, solid-state memory device, digital versatile disk, cartridge, random access memory (RAM) 1225, read-only memory (ROM) 1220, and hybrids thereof. The storage device 1230 may include services 1232, 1234, and 1236 for controlling the processor 1210. Other hardware or software modules are possible. The storage device 1230 may be connected to the computing device connection 1205. In one embodiment, a hardware module performing a particular function may include software components stored on a computer-readable medium, along with the necessary hardware components such as the processor 1210, the connection 1205, and the output device 1235, in order to perform that function.

[0159] Aspects of this disclosure are applicable to any suitable electronic device (such as a security system, smartphone, tablet, laptop computer, vehicle, drone, or other device) that includes or is coupled with one or more active depth sensing systems. While the following describes a device having or being coupled with one optical projector, aspects of this disclosure are applicable to devices having any number of optical projectors and are therefore not limited to any particular device.

[0160] The term “device” is not limited to one or a specific number of physical objects (such as one smartphone, one controller, or one processing system). As used herein, a device may be any electronic device having one or more parts that can implement at least some parts of this disclosure. The following descriptions and examples use the term “device” to describe various aspects of this disclosure, but the term “device” is not limited to a specific configuration, type, or number of objects. Furthermore, the term “system” is not limited to multiple components or a specific embodiment. For example, a system may be mounted on one or more printed circuit boards or other substrates and may have movable or static components. The following descriptions and examples use the term “system” to describe various aspects of this disclosure, but the term “system” is not limited to a specific configuration, type, or number of objects.

[0161] Specific details are provided in the above description in order to provide a complete understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that embodiments can be practiced without these specific details. For clarity of the description, in some cases the technology may be presented as including individual functional blocks, which include devices, device components, steps or routines in a method embodied in software, or combinations of hardware and software. Further components other than those shown in the figures and/or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in the form of block diagrams in order to avoid obscuring the embodiments with unnecessary details. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details in order to avoid obscuring the embodiments.

[0162] Individual embodiments may be described above as processes or methods depicted as flowcharts, flow diagrams, data flow diagrams, structural diagrams, or block diagrams. While flowcharts may describe operations as sequential processes, many operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process terminates when its operations are complete, but it may have additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function's return to the calling function or main function.

[0163] The processes and methods described above may be implemented using computer-executable instructions that are stored or otherwise available from a computer-readable medium. Such instructions may include instructions and data that cause, for example, a general-purpose computer, a dedicated computer, or a processing device to execute or otherwise configure a function or group of functions to execute. The portion of computer resources used may be accessible via a network. Computer-executable instructions may include, for example, binaries, intermediate format instructions such as assembly language, firmware, source code, etc.

[0164] The term “computer-readable media” includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and/or data. Computer-readable media may include non-temporary media on which data may be stored and which do not contain carrier waves and/or temporary electronic signals that propagate wirelessly or via wired connections. Examples of non-temporary media may include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as flash memory, memory or memory devices, magnetic disks or optical disks, flash memory, USB devices with non-volatile memory, network-attached storage devices, compact discs (CDs) or digital multipurpose discs (DVDs), and any appropriate combination thereof. Computer-readable media may store thereon code and/or machine-executable instructions that can represent any combination of procedures, functions, subprograms, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments may be coupled to other code segments or hardware circuits by passing and/or receiving information, data, arguments, parameters, or memory content. Information, arguments, parameters, data, etc., may be passed, transferred, or transmitted via any appropriate means, including memory sharing, message passing, token passing, and network transmission.

[0165] In some embodiments, computer-readable storage devices, media, and memory may include cables or wireless signals containing bitstreams, etc. However, as stated, non-transient computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0166] Devices implementing the processes and methods described herein may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the required tasks may be stored in computer-readable or machine-readable media. The processor can then perform the required tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mount devices, and standalone devices. The functions described herein may also be embodied within peripheral devices or add-in cards. Such functions may also, as a further example, be implemented on a circuit board between different chips or different processes running within a single device.

[0167] Instructions, a medium for transmitting such instructions, computing resources for executing them, and other structures for supporting such computing resources are exemplary means for providing the functions described herein.

[0168] While the above description refers to aspects of this application with reference to specific embodiments, those skilled in the art will recognize that this application is not limited thereto. Therefore, although exemplary embodiments of this application are described in detail herein, it should be understood that, except as limited by the prior art, the concepts of the present invention may be embodied and employed in various ways, and the appended claims should be construed to include such variations. The various features and aspects of the above-described examples of application may be used individually or together. Furthermore, embodiments may be used in any number of environments and applications other than those described herein without departing from the broader spirit and scope of this specification. Therefore, this specification and the drawings should be considered illustrative, not restrictive. For illustrative purposes, the methods are described in a specific order. It should be understood that in alternative embodiments, the methods may be performed in a different order than described.

[0169] Those skilled in the art will understand that the symbols or terms less than ("<") and greater than (">") used herein may be replaced, without departing from the scope of this specification, with the symbols less than or equal to ("≦") and greater than or equal to ("≧"), respectively.

[0170] Where a component is described as “configured” to perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform that operation, by programming programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform that operation, or by any combination thereof.

[0171] The phrase “combined” refers to any component that is physically connected to another component, either directly or indirectly, and/or any component that communicates with another component, either directly or indirectly (for example, connected to another component via a wired or wireless connection, and/or other appropriate communication interface).

[0172] Claim language or other language listing “at least one of” a set and/or “one or more” of a set indicates that one or more members of a set (any combination of) satisfy the claim. For example, claim language listing “at least one of A and B” or “at least one of A or B” means A, B, or A and B. In another example, claim language listing “at least one of A, B, and C” or “at least one of A, B, or C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and/or “one or more” of a set does not limit the set to the items listed within it. For example, claim wording that lists "at least one of A and B" or "at least one of A or B" can mean A, B, or A and B, and may also include items not listed within the set of A and B.

[0173] Various exemplary logic blocks, modules, circuits, and algorithmic steps described in relation to embodiments disclosed herein may be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly demonstrate this hardware- and software compatibility, various exemplary components, blocks, modules, circuits, and steps are described above in general terms of their function. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functions in various ways for each specific application, but such implementation decisions should not be construed as resulting in a departure from the scope of this application.

[0174] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication device handsets, or integrated circuit devices having multiple applications, including applications in wireless communication device handsets and other devices. Any feature described as a module or component may be implemented together in an integrated logic device, or separately as individual but interoperable logic devices. If implemented in software, the techniques may be at least partially implemented by a computer-readable data storage medium having program code that, when executed, includes instructions to perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. Computer-readable media may include random access memory (RAM), such as synchronous dynamic random access memory (SDRAM), read-only memory (ROM), non-volatile random access memory (NVRAM), electrically erasable programmable read-only memory (EEPROM®), FLASH® memory, and magnetic or optical data storage media. The technique may, as an addition or alternative, be at least partially implemented by computer-readable communication media, such as propagating signals or radio waves, that carry or communicate program code in the form of instructions or data structures, and can be accessed, read, and/or executed by a computer.

[0175] The program code may be executed by a processor which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field-programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described herein. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors working with a DSP core, or any other such configuration. Thus, the term “processor” as used herein may refer to any of the above structures, any combination thereof, or any other structure or device suitable for implementing the techniques described herein.

[0176] Illustrative examples of the present disclosure include:

[0177] Embodiment 1: A method for processing video data, comprising: acquiring input video data containing at least one luminance component of the current frame using a machine learning system; and using the machine learning system to determine motion information for at least one luminance component of the current frame and motion information for one or more chrominance components of the current frame, using at least one luminance component of the current frame.

[0178] Embodiment 2: The method according to Embodiment 1, further comprising: determining the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame by a machine learning system using motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame; and determining one or more interframe predictions of the current frame using the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame.

[0179] Embodiment 3: The method according to Embodiment 2, wherein one or more interframe predictions are determined at least partially by applying an interpolation operation using a warping parameter for at least one luminance component of the current frame and one or more warping parameters for one or more chrominance components of the current frame.

[0180] Embodiment 4: The method according to Embodiment 3, wherein the interpolation operation includes trilinear interpolation.

[0181] Embodiment 5: The method according to any one of Embodiments 2 to 4, wherein the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame include a spatial scale flow (SSF) warping parameter.

[0182] Embodiment 6: The method according to Embodiment 5, wherein the SSF warping parameters include a learned scale flow vector.

[0183] Embodiment 7: The method according to any one of Embodiments 1 to 6, wherein determining the motion information of at least one luminance component of the current frame and the motion information of one or more chrominance components of the current frame using at least one luminance component of the current frame comprises: determining the motion information of at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored lumen component of the previous frame; and determining the motion information of one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.

[0184] Embodiment 8: The method according to Embodiment 7, wherein motion information of one or more chrominance components of the current frame is determined using a convolutional layer of a machine learning system.

[0185] Embodiment 9: The method according to Embodiment 7, wherein the motion information of one or more chrominance components of the current frame is determined at least partially by sampling motion information determined for at least one luminance component of the current frame.

[0186] Embodiment 10: The method according to any one of Embodiments 1 to 9, wherein the current frame includes a video frame.

[0187] Embodiment 11: The method according to any one of Embodiments 1 to 10, wherein one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.

[0188] Embodiment 12: The method according to any one of Embodiments 1 to 11, wherein the current frame has a luminance-chrominance (YUV) format.

[0189] Embodiment 13: The method according to Embodiment 12, wherein the YUV format is the YUV4:2:0 format.

[0190] Embodiment 14: An apparatus for processing video data, comprising at least one memory and one or more processors coupled to the at least one memory, wherein the one or more processors are configured to use a machine learning system to acquire input video data including at least one luminance component of the current frame, and to use the machine learning system to determine motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame.

[0191] Embodiment 15: The apparatus according to Embodiment 14, wherein one or more processors are configured to use a machine learning system to determine the warping parameters of at least one luminance component and one or more chrominance components of the current frame, based on motion information of at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, and to determine one or more interframe predictions of the current frame using the warping parameters of at least one luminance component and one or more chrominance components of the current frame.

[0192] Embodiment 16: The apparatus according to Embodiment 15, wherein one or more interframe predictions are determined at least partially by applying an interpolation operation using a warping parameter of at least one luminance component of the current frame and one or more warping parameters of one or more chrominance components of the current frame.

[0193] Embodiment 17: The apparatus according to Embodiment 16, wherein the interpolation operation includes trilinear interpolation.

[0194] Embodiment 18: The apparatus according to any one of Embodiments 15 to 17, wherein the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame include a spatial scale flow (SSF) warping parameter.

[0195] Embodiment 19: The apparatus according to Embodiment 18, wherein the SSF warping parameters include a learned scale flow vector.

[0196] Embodiment 20: An apparatus according to any one of embodiments 14 to 19, wherein one or more processors are configured to determine the motion information of at least one luminance component of the current frame and the motion information of one or more chrominance components of the current frame, using at least one luminance component of the current frame, by determining the motion information of at least one luminance component of the current frame based on at least one luminance component of the current frame and at least one restored lumen component of the previous frame, and then determining the motion information of one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.

[0197] Embodiment 21: The apparatus according to Embodiment 20, wherein motion information of one or more chrominance components of the current frame is determined using a convolutional layer of a machine learning system.

[0198] Embodiment 22: The apparatus according to Embodiment 20, wherein one or more processors are configured to sample motion information determined for at least one luminance component of the current frame in order to determine motion information for one or more chrominance components of the current frame.

[0199] Embodiment 23: The apparatus according to any one of Embodiments 14 to 22, wherein the current frame includes a video frame.

[0200] Embodiment 24: The apparatus according to any one of Embodiments 14 to 23, wherein one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.

[0201] Embodiment 25: The apparatus according to embodiments 14 to 24, wherein the current frame has a luminance-chrominance (YUV) format.

[0202] Embodiment 26: The apparatus according to Embodiment 25, wherein the YUV format is the YUV4:2:0 format.

[0203] Embodiment 27: The apparatus according to any one of embodiments 14 to 26, further comprising at least one camera configured to capture one or more frames.

[0204] Embodiment 28: The apparatus according to any one of embodiments 14 to 27, further comprising at least one display configured to display one or more frames.

[0205] Embodiment 29: The apparatus according to any one of Embodiments 14 to 28, including a mobile device.

[0206] Embodiment 30: A computer-readable storage medium that, when executed, stores instructions causing one or more processors to perform any of the operations described in Embodiments 1 to 29.

[0207] Embodiment 31: An apparatus comprising means for performing any of the operations described in Embodiments 1 to 29.
The invention described in the original claims of this application is listed below.
[C1]
A method for processing video data,
The machine learning system acquires input video data containing at least one luminance component of the current frame,
A method comprising: using the machine learning system to determine motion information of the at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, using the at least one luminance component of the current frame.
[C2]
The machine learning system determines the warping parameter of the at least one luminance component of the current frame and the warping parameter of the one or more chrominance components of the current frame, using the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame.
The method according to C1, further comprising determining one or more interframe predictions of the current frame using the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame.
[C3]
The method according to C2, wherein the one or more interframe predictions are at least partially determined by applying an interpolation operation using the warping parameter of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame.
[C4]
The interpolation operation is the method described in C3, which includes a trilinear interpolation operation.
[C5]
The method according to C2, wherein the warping parameter of the at least one luminance component of the current frame and the warping parameter of the one or more chrominance components of the current frame include a spatial scale flow (SSF) warping parameter.
[C6]
The method of C5 wherein the SSF warping parameters include a learned scale flow vector.
[C7]
Using the at least one luminance component of the current frame, the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame are determined.
Based on the at least one luminance component of the current frame and the at least one restored lumens component of the previous frame, the motion information of the at least one luminance component of the current frame is determined.
The method according to C1, comprising determining the motion information of one or more chrominance components of the current frame using the motion information determined for the at least one luminance component of the current frame.
[C8]
The method according to C7, wherein the motion information of one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.
[C9]
The method according to C7, wherein the motion information of one or more chrominance components of the current frame is determined at least partially by sampling the motion information determined for at least one luminance component of the current frame.
[C10]
The method according to C1, wherein the current frame includes a video frame.
[C11]
The method according to C1, wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.
[C12]
The present frame is the method according to C1, having a luminance-chrominance (YUV) format.
[C13]
The method according to C12, wherein the YUV format is the YUV4:2:0 format.
[C14]
A device for processing video data,
At least one memory,
The system comprises one or more processors coupled to at least one memory, and the one or more processors are
Using a machine learning system, we obtain input video data that includes at least one luminance component of the current frame.
An apparatus configured to use the machine learning system to determine motion information of the at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame, using the at least one luminance component of the current frame.
[C15]
The one or more processors described above are
Based on the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame, the machine learning system is used to determine the warping parameter of the at least one luminance component of the current frame and the warping parameter of the one or more chrominance components of the current frame.
The apparatus according to C14, configured to determine one or more interframe predictions of the current frame using the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame.
[C16]
The apparatus according to C15, wherein the one or more interframe predictions are at least partially determined by applying an interpolation operation using the warping parameter of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame.
[C17]
The apparatus according to C16, wherein the interpolation operation includes a trilinear interpolation operation.
[C18]
The apparatus according to C15, wherein the warping parameter of the at least one luminance component of the current frame and the warping parameters of the one or more chrominance components of the current frame include a spatial scale flow (SSF) warping parameter.
[C19]
The apparatus according to C18, wherein the SSF warping parameters include a learned scale flow vector.
[C20]
Using the at least one luminance component of the current frame, the one or more processors determine the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame.
Based on the at least one luminance component of the current frame and the at least one restored lumens component of the previous frame, the motion information of the at least one luminance component of the current frame is determined.
The apparatus according to C14, configured to determine the motion information of one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.
[C21]
The apparatus according to C20, wherein the motion information of one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system.
[C22]
The apparatus according to C20, wherein, in order to determine the motion information of one or more chrominance components of the current frame, one or more processors are configured to sample the motion information determined for at least one luminance component of the current frame.
[C23]
The apparatus described in C14, wherein the current frame includes a video frame.
[C24]
The apparatus according to C14, wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component.
[C25]
The present frame is the apparatus according to C14, having a luminance-chrominance (YUV) format.
[C26]
The apparatus described in C25, wherein the YUV format is the YUV4:2:0 format.
[C27]
The apparatus according to C14, further comprising at least one camera configured to capture one or more frames.
[C28]
The apparatus according to C14, further comprising at least one display configured to display one or more frames.
[C29]
The apparatus described in C14, including a mobile device.
[C30]
When executed by one or more processors, the one or more processors will
Using a machine learning system, acquire input video data containing at least one luminance component of the current frame,
A non-temporary computer-readable medium storing instructions for using the machine learning system to determine, using the at least one luminance component of the current frame, motion information of the at least one luminance component of the current frame and motion information of one or more chrominance components of the current frame.

Claims (15)

A method for processing video data,
A machine learning system obtains a latent representation of at least one luminance component of the current frame, and the latent representation of the at least one luminance component of the current frame is based on the at least one luminance component of the current frame and the at least one restored luminance component of the previous frame.
The machine learning system determines the motion information of the at least one luminance component of the current frame using the latent representation of the at least one luminance component of the current frame.
A method comprising: determining motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame using the machine learning system. The machine learning system determines the warping parameter of the at least one luminance component of the current frame and the warping parameter of the one or more chrominance components of the current frame, using the motion information of the at least one luminance component of the current frame and the motion information of the one or more chrominance components of the current frame.
The method according to claim 1, further comprising determining one or more interframe predictions for the current frame using the warping parameter of at least one luminance component of the current frame and the warping parameter of one or more chrominance components of the current frame. The method according to claim 2, wherein the one or more interframe predictions are at least partially determined by applying an interpolation operation using the warping parameter of the at least one luminance component of the current frame and the one or more warping parameters of the one or more chrominance components of the current frame. The method according to claim 3, wherein the interpolation operation includes a trilinear interpolation operation. The method according to claim 2, wherein the warping parameter of the at least one luminance component of the current frame and the warping parameters of the one or more chrominance components of the current frame include a spatial-scale flow (SSF) warping parameter. The method according to claim 5, wherein the SSF warping parameters include a learned scale flow vector. The method according to claim 1, wherein the motion information of the one or more chrominance components of the current frame is determined using a convolutional layer of the machine learning system. The method according to claim 1, wherein the motion information of one or more chrominance components of the current frame is at least partially determined by sampling the motion information determined for at least one luminance component of the current frame. The method according to claim 1, wherein the current frame includes a video frame. The method according to claim 1, wherein the one or more chrominance components include at least one chrominance blue component and at least one chrominance red component. The present frame has a luminance-chrominance (YUV) format, according to claim 1. The method according to claim 11, wherein the YUV format is the YUV4:2:0 format. A device for processing video data,
At least one memory,
The system comprises one or more processors coupled to at least one memory, and the one or more processors are
Using a machine learning system, obtain a latent representation of at least one luminance component of the current frame, and the latent representation of the at least one luminance component of the current frame is based on the at least one luminance component of the current frame and the at least one restored luminance component of the previous frame.
Using the machine learning system, the motion information of the at least one luminance component of the current frame is determined using the latent representation of the at least one luminance component of the current frame.
An apparatus configured to determine motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame using the machine learning system. It further comprises at least one camera configured to capture one or more frames, or
The system further comprises at least one display configured to display one or more frames, or
The apparatus according to claim 13, including a mobile device. When executed by one or more processors, the one or more processors will
Using a machine learning system, obtain a latent representation of at least one luminance component of the current frame, and the latent representation of the at least one luminance component of the current frame is based on the at least one luminance component of the current frame and the at least one restored luminance component of the previous frame.
Using the machine learning system, the motion information of the at least one luminance component of the current frame is determined using the latent representation of the at least one luminance component of the current frame.
A non-temporary computer-readable medium storing instructions to perform the following actions: to use the machine learning system to determine motion information for one or more chrominance components of the current frame using the motion information determined for at least one luminance component of the current frame.