JP7703392B2 - Apparatus and program for generating super-resolution auxiliary data - Google Patents

Description

The present invention relates to a super-resolution auxiliary information generating device and program.

In inter-frame prediction in video coding methods such as H.265/HEVC (High Efficiency Video Coding) and H.266/VVC (Versatile Video Coding), motion information such as motion vectors and reference picture indexes are coded and transmitted in units of prediction units (PUs) and used for decoding. In addition, a skip mode in which motion vectors and transform coefficients are not transmitted can be specified in a coding unit (CU), and in the case of the skip mode, information such as motion vectors is in merge mode, and optimal candidate indexes are coded and transmitted from the motion information of prediction units adjacent in the spatial and temporal directions and used for decoding (see, for example, Non-Patent Document 1).

On the other hand, when displaying high definition broadcasts or moving images with resolutions less than 4K in high quality on 4K televisions and 4K monitors, super-resolution technology is widely used. For example, Patent Document 1 discloses a technology for generating a super-resolution image by aligning multiple frame images. Patent Document 2 discloses a technology for generating a super-resolution image by aligning different frequency component images of a single frame image.

Patent No. 6462372 Patent No. 6817784

Edited by Eiichiro Okubo, "Impress Standard Textbook Series H.265/HEVC Textbook", Impress Japan Co., Ltd., October 21, 2013

Conventionally, when a super-resolution processed image is generated using a decoded image obtained by decoding an encoded image, the decoding process information used when generating the decoded image is not utilized. By utilizing the decoding process information, it is expected that the decoded image can be subjected to super-resolution processing with high image quality.

In view of the above circumstances, the object of the present invention is to provide a super-resolution auxiliary information generating device and program capable of generating auxiliary information useful for super-resolution processing of a decoded image from decoding processing information.

To solve the above problem, a super-resolution auxiliary information generating device according to one embodiment includes an information acquiring unit that is used when decoding an encoded image to generate a decoded image and that acquires decoding process information set for each prediction unit, which is a unit of prediction processing when generating and decoding the encoded image, and an auxiliary information generating unit that estimates the degree of blurring of the decoded image or the alignment accuracy when performing super-resolution processing on the decoded image for each prediction unit according to the decoding process information, and generates auxiliary information indicating the estimation result.

Furthermore, in one embodiment, the information acquisition unit may acquire information indicating the size of the prediction unit as the decoding process information, and the auxiliary information generation unit may estimate that the larger the size, the higher the accuracy of alignment in the decoded image corresponding to the prediction unit.

Furthermore, in one embodiment, the information acquisition unit acquires, as the decoding process information, information indicating a value of a quantization parameter used when generating and decoding the encoded image, and the auxiliary information generation unit may estimate that the larger the value of the quantization parameter, the lower the alignment accuracy in the decoded image corresponding to the prediction unit.

Furthermore, in one embodiment, the information acquisition unit acquires, as the decoding process information, information indicating the magnitude of a motion vector used when generating and decoding the encoded image, and the auxiliary information generation unit may estimate that the greater the magnitude of the motion vector, the greater the degree of blurring in the decoded image corresponding to the prediction unit.

Furthermore, in one embodiment, the information acquisition unit may acquire, as the decoding process information, information indicating a value of a quantization parameter used when generating the encoded image, and the auxiliary information generation unit may estimate that the greater the value of the quantization parameter, the greater the degree of blurring in the decoded image corresponding to the prediction unit.

In addition, to solve the above problem, a program according to one embodiment causes a computer to function as the above super-resolution auxiliary information generating device.

According to the present invention, it is possible to generate auxiliary information useful for super-resolution processing of a decoded image from the decoding processing information.

1 is a block diagram showing an example of the configuration of a super-resolution processing system including a super-resolution auxiliary data generating device according to an embodiment. 1 is a block diagram showing an example of the configuration of a super-resolution auxiliary data generating device according to an embodiment. 10 is a flowchart illustrating an example of the operation of a super-resolution auxiliary data generating device according to an embodiment.

One embodiment is described in detail below with reference to the drawings.

FIG. 1 is a diagram showing an example of the configuration of a super-resolution processing system 1 including a super-resolution auxiliary information generating device 10 according to one embodiment. The super-resolution processing system 1 includes an encoding device 20, a decoding device 30, the super-resolution auxiliary information generating device 10, and a super-resolution device 40.

The encoding device 20 performs an encoding process on a moving image (input image) using a known method such as H.265/HEVC or H.266/VVC to generate an encoded image, which it transmits to the decoding device 30. The encoding device 20 also transmits encoding process information (motion vector information, various parameters, etc.) used during the encoding process to the decoding device 30.

The decoding device 30 performs a decoding process on the coded image received from the coding device 20 using the coding process information received from the coding device 20 to generate a decoded image, which it transmits to the super-resolution device 40. The decoding device 30 also transmits, from the processing information received from the coding device 20, motion vector information, size information, and QP (Quantization Parameter) information as decoding process information to the super-resolution auxiliary information generating device 10. The motion vector information, size information, and QP information are obtained for each prediction unit. Details of this information are described in, for example, Non-Patent Document 1.

Motion vector information is information that indicates the magnitude of the motion vector used when performing inter-picture prediction (motion compensation prediction) in the encoding device 20 and the decoding device 30. In other words, the motion vector information is information that indicates the magnitude of the motion vector used when generating and decoding the encoded image.

The size information is information indicating the size of the prediction unit, which is the unit of prediction processing when performing inter-screen prediction and intra-screen prediction in the encoding device 20 and the decoding device 30. In other words, the size information is information indicating the size of the prediction unit, which is the unit of prediction processing when generating and decoding an encoded image. Note that when merge encoding is performed, the size of the prediction unit is the size after merging.

The QP information is information that indicates the value of the quantization parameter used when performing the quantization process in the encoding device 20 and the inverse quantization process in the decoding device 30. In other words, the QP information is information that indicates the value of the quantization parameter used when generating and decoding the encoded image.

The super-resolution auxiliary information generating device 10 uses the decoding process information from the decoding device 30 to generate super-resolution auxiliary information to be used in super-resolution processing of the decoded image, and transmits it to the super-resolution device 40.

The super-resolution device 40 performs super-resolution processing on the decoded image received from the decoding device 30 using the super-resolution auxiliary information received from the super-resolution auxiliary information generating device 10 to generate a super-resolution image. The super-resolution device 40 can use the super-resolution auxiliary information to control the alignment accuracy judgment threshold of the registration super-resolution processing, the degree of emphasis of spatial high frequency components, and the like.

Figure 2 is a block diagram showing an example of the configuration of a super-resolution auxiliary information generating device 10 according to an embodiment. The super-resolution auxiliary information generating device 10 includes an information acquisition unit 11 and an auxiliary information generating unit 12. Each functional unit shown in Figure 2 constitutes a control and arithmetic circuit (controller). The control and arithmetic circuit may be constituted by dedicated hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field-Programmable Gate Array), may be constituted by a processor, or may be constituted by including both.

The information acquisition unit 11 is used when decoding an encoded image to generate a decoded image, and acquires decoding process information (motion vector information, size information, and QP information) set for each prediction unit, and outputs the decoding process information to the auxiliary information generation unit 12.

The auxiliary information generating unit 12 estimates the degree of blur of the decoded image or the alignment accuracy when performing super-resolution processing on the decoded image for each prediction unit according to the decoding processing information input from the information acquiring unit 11, and generates auxiliary information indicating the estimation result. The auxiliary information generating unit 12 includes an alignment accuracy estimating unit 121 and a blur degree estimating unit 122. Note that the information acquiring unit 11 does not need to output the motion vector information to the alignment accuracy estimating unit 121, and does not need to output the size information to the blur degree estimating unit 122.

The alignment accuracy estimation unit 121 inputs size information indicating the size of the prediction unit from the information acquisition unit 11. It is considered that the larger the size of the prediction unit, the more likely it is that the entire moving object contained in the prediction unit moves uniformly (or in parallel) between pictures or slices, or moves very little, making it easier to estimate the accuracy of motion. Therefore, the alignment accuracy estimation unit 121 estimates that the larger the size of the prediction unit, the higher the alignment accuracy in the decoded image corresponding to the prediction unit.

The alignment accuracy estimation unit 121 inputs QP information indicating the value of the quantization parameter from the information acquisition unit 11. It is considered that the larger the value of the quantization parameter, the more likely it is that the decoded image corresponding to the prediction unit is degraded compared to the original image. Therefore, the alignment accuracy estimation unit 121 estimates that the larger the value of the quantization parameter, the lower the alignment accuracy in the decoded image corresponding to the prediction unit.

When the alignment accuracy estimation unit 121 receives motion vector information indicating the magnitude of the motion vector from the information acquisition unit 11, the alignment accuracy estimation unit 121 may change the estimation amount according to the magnitude of the motion vector. However, in this embodiment, since the magnitude of the motion vector has little effect on the alignment accuracy, the alignment accuracy estimation unit 121 does not perform estimation according to the magnitude of the motion vector (does not change the estimation amount).

For example, the correspondence between the size of the prediction unit and the value of the quantization parameter and the alignment accuracy estimate is specified in advance, and the alignment accuracy estimation unit 121 determines the alignment accuracy estimate that indicates the estimation result based on the correspondence.

Table 1 is an example of a table showing the correspondence between the size of a prediction unit and the alignment accuracy estimate. Here, the default value of the alignment accuracy estimate is 1.0. If the shape of the prediction unit is not square, the size can be determined by the total number of pixels.

Table 2 is an example of a table showing the correspondence between the quantization parameter value and the alignment accuracy estimate. Here, the default value of the alignment accuracy estimate is 1.0.

For example, if the size of the prediction unit is 32 x 32 and the value of the quantization parameter is 28, the alignment accuracy estimation unit 121 determines the alignment accuracy estimate for the size of the prediction unit to be 1.5 using Table 1, and determines the alignment accuracy estimate for the value of the quantization parameter to be 1.5 using Table 2. Then, 1.5 is multiplied by 1.5 to finally determine the alignment accuracy estimate to be 2.25. In this way, the alignment accuracy estimation unit 121 generates an alignment accuracy estimate for each prediction unit of the decoded image based on the decoding process information.

The blur degree estimation unit 122 inputs motion vector information indicating the magnitude of a motion vector from the information acquisition unit 11. It is considered that the larger the magnitude of the motion vector (the average value of motion vectors if there are multiple motion vectors for the prediction unit), the greater the possibility that motion blur of a moving object included in the prediction unit will be large. Therefore, the blur degree estimation unit 122 estimates that the larger the magnitude of the motion vector, the higher the degree of blur in the decoded image corresponding to the prediction unit.

The blur degree estimation unit 122 inputs QP information indicating the value of the quantization parameter from the information acquisition unit 11. It is considered that the larger the value of the quantization parameter, the more likely it is that the decoded image corresponding to the prediction unit is degraded compared to the original image. Therefore, the blur degree estimation unit 122 estimates that the larger the value of the quantization parameter, the higher the degree of blur in the decoded image corresponding to the prediction unit.

When the blurriness estimation unit 122 receives size information indicating the size of the prediction unit from the information acquisition unit 11, the blurriness estimation unit 122 may change the estimated value depending on the size of the prediction unit. However, in this embodiment, since the size of the prediction unit has little effect on the blurriness, the blurriness estimation unit 122 does not perform estimation depending on the size of the prediction unit (does not change the estimated amount).

For example, the correspondence between the magnitude of the motion vector and the value of the quantization parameter and the blur degree estimate is defined in advance, and the blur degree estimation unit 122 determines the blur degree estimate that indicates the estimation result based on the correspondence.

Table 3 is an example of a table showing the correspondence between the magnitude of a motion vector and the blur level estimate. Here, the default value of the blur level estimate is 1.0.

Table 4 is an example of a table showing the correspondence between the quantization parameter value and the blur level estimate. Here, the default value of the blur level estimate is 1.0.

For example, if the magnitude of the motion vector is 43 and the value of the quantization parameter is 38, the blur degree estimation unit 122 determines the blur degree estimate for the magnitude of the motion vector to be 1.5 using Table 3, and determines the blur degree estimate for the value of the quantization parameter to be 1.2 using Table 4. Then, 1.5 and 1.2 are multiplied together to finally determine the blur degree estimate to be 1.8. In this way, the blur degree estimation unit 122 generates a blur degree estimate for each prediction unit of the decoded image based on the decoding process information.

When performing hierarchical alignment within a single frame, for example, as described in Patent Document 2, the super-resolution device 40 divides the super-resolved image into division blocks of a predetermined size, and for each division block, determines a similar block in the low-frequency component image that has the highest similarity (correlation) with the division block by block matching, and generates alignment information indicating the position of the similar block. Here, if the difference value between the division block and the similar block exceeds a threshold, the alignment information may be discarded, and the alignment accuracy can be improved by controlling the threshold using the alignment accuracy estimate input from the super-resolution auxiliary information generating device 10. In addition, the super-resolution device 40 can improve the degree of blurring of the decoded image by adjusting the degree of emphasis of the spatial high-frequency components, such as emphasizing the spatial high-frequency components of the decoded image as the blurring degree estimate becomes larger, using the blurring degree estimate input from the super-resolution auxiliary information generating device 10.

Next, the operation of the super-resolution auxiliary data generating device 10 will be described with reference to FIG. 3. FIG. 3 is a flowchart showing an example of the operation of the super-resolution auxiliary data generating device 10. The super-resolution auxiliary data generating device 10 may perform the processes of steps S102 and S103 in parallel, or may perform the process of step S102 after performing the process of step S103.

In step S101, the information acquisition unit 11 acquires the decoding process information (motion vector information, size information, and QP information) used when decoding the encoded image.

In step S102, the alignment accuracy estimation unit 121 estimates that the larger the size of the prediction unit indicated by the size information, the higher the alignment accuracy in the decoded image corresponding to the prediction unit. Also, the alignment accuracy estimation unit 121 estimates that the larger the value of the quantization parameter indicated by the QP information, the lower the alignment accuracy in the decoded image corresponding to the prediction unit.

In step S103, the blur degree estimation unit 122 estimates that the larger the motion vector indicated by the motion vector information, the higher the degree of blur in the decoded image corresponding to the prediction unit. Also, the blur degree estimation unit 122 estimates that the larger the quantization parameter indicated by the QP information, the higher the degree of blur in the decoded image corresponding to the prediction unit.

As described above, the super-resolution auxiliary information generating device 10 acquires the decoding process information set for each prediction unit, and estimates the degree of blur of the decoded image or the alignment accuracy when super-resolution processing the decoded image for each prediction unit according to the decoding process information, and generates super-resolution auxiliary information useful for super-resolution processing of the decoded image. By using the super-resolution auxiliary information, the super-resolution device 40 can improve the alignment accuracy and improve the degree of blur of the decoded image, making it possible to super-resolution process the decoded image with high image quality.

<Program>
A computer capable of executing program instructions can also be used to function as the above-mentioned super-resolution auxiliary data generating apparatus 10. Here, the computer may be a general-purpose computer, a special-purpose computer, a workstation, a personal computer (PC), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor may be a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), or the like, and may be composed of multiple processors of the same or different types. The processor reads out a program from the storage unit and executes it to control each of the above components and perform various arithmetic processing. Note that at least a part of the processing content may be realized by hardware. The input unit is an input interface that receives a user's input operation and acquires information based on the user's operation, and is a pointing device, keyboard, mouse, or the like. The output unit is an output interface that outputs information, and is a display, speaker, or the like. The communication interface is an interface for communicating with an external device.

The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

The above-mentioned embodiment has been described as a representative example, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or alterations are possible without departing from the scope of the claims. For example, it is possible to integrate multiple configuration blocks shown in the configuration diagram of the embodiment, or to divide one configuration block.

REFERENCE SIGNS LIST 1 super-resolution processing system 10 super-resolution auxiliary information generating device 11 information acquiring unit 12 auxiliary information generating unit 20 encoding device 30 decoding device 40 super-resolution device 121 alignment accuracy estimating unit 122 blur degree estimating unit

Claims (6)

an information acquisition unit that is used when decoding an encoded image to generate a decoded image, and that acquires decoding process information set for each prediction unit that is a unit of prediction process when generating and decoding the encoded image;
a support information generating unit that estimates a degree of blurring of the decoded image or a position alignment accuracy when performing a super-resolution process on the decoded image for each of the prediction units according to the decoding process information, and generates support information indicating a result of the estimation;
A super-resolution auxiliary data generating device comprising: the information acquisition unit acquires, as the decoding process information, information indicating a size of the prediction unit;
The super-resolution auxiliary information generating device according to claim 1 , wherein the auxiliary information generating section estimates that the larger the size is, the higher the accuracy of alignment in the decoded image corresponding to the prediction unit is. the information acquisition unit acquires, as the decoding process information, information indicating a value of a quantization parameter used when generating and decoding the encoded image;
The super-resolution auxiliary information generating device according to claim 1 , wherein the auxiliary information generating section estimates that the greater the value of the quantization parameter, the lower the accuracy of alignment in the decoded image corresponding to the prediction unit. the information acquisition unit acquires, as the decoding process information, information indicating a magnitude of a motion vector used when generating and decoding the encoded image;
The super-resolution auxiliary data generating device according to claim 1 , wherein the auxiliary data generating section estimates that the greater the magnitude of the motion vector, the greater the degree of blurring in the decoded image corresponding to the prediction unit. the information acquisition unit acquires, as the decoding process information, information indicating a value of a quantization parameter used when generating the encoded image;
The super-resolution auxiliary information generating device according to claim 1 , wherein the auxiliary information generating section estimates that the greater the value of the quantization parameter, the greater the degree of blurring in the decoded image corresponding to the prediction unit. A program for causing a computer to function as a super-resolution auxiliary information generating device according to any one of claims 1 to 5.

Images