JP7736899B2 - Decoding device and program - Google Patents

Description

The present invention relates to a decoding device and a program.

Video coding methods such as H.265/HEVC (High Efficiency Video Coding) are configured to generate a residual signal by switching between two types of prediction: inter-prediction, which utilizes temporal correlation between frames, and intra-prediction, which utilizes spatial correlation within a frame. Then, orthogonal transform processing, loop filter processing, and entropy coding processing are performed, and the resulting stream is output.

HEVC provides a total of 35 different intra prediction modes, including planar prediction, DC prediction, and directional prediction, and is configured to perform intra prediction using adjacent decoded reference pixels according to the mode determined by the encoder.

Here, with intra prediction, for CUs (Coding Units) that do not have adjacent decoded reference pixels, such as the CU located at the top left of a frame, reference pixels to be used when generating a predicted image are created by filling in a specified value ("512" for 10-bit video images).

In addition, in conventional HEVC, the encoding process is performed in raster scan order starting from the top left, so there are cases where the reference pixel has not yet been decoded. In such cases, the predicted image is generated using a value obtained by zero-order extrapolation of the closest decoded reference pixel.

A CU is divided into multiple blocks (TUs: Transform Units) and orthogonal transform processing is performed on them. In HEVC, the intra prediction mode, which indicates the type of intra prediction to be applied, is common within a CU, and predictions are performed for multiple TUs using the same intra prediction mode.

As shown in Figure 7, with intra prediction, due to the encoding process using raster scan order, the reference pixel N located in the lower left or upper right of CU#X often has not yet been decoded. In such cases, performing directional prediction from the direction in which the undecoded reference pixel exists reduces prediction accuracy, resulting in reduced encoding efficiency.

These problems will be explained in detail below using Figures 8(a) to 8(d). Figure 8 shows an example of intra prediction in conventional HEVC.

In this example, as shown in Figure 8(a), similar to the example in Figure 7, all reference pixels for CU#A1 (the CU in the upper left of the frame) have already been decoded. Similarly, as shown in Figure 8(c), all reference pixels for CU#A3 (the CU in the lower left of the frame) have already been decoded.

In contrast, as shown in Figure 8(b), reference pixels W1 to W3 located in CU#A1 have already been decoded, but reference pixels B1 to B4 located in CU#A3 have not been decoded, and therefore cannot be used as reference pixels when generating a predicted image for CU#A2 (the CU in the upper right corner of the frame).

For this reason, conventional HEVC specifies that the value of reference pixel W1, located at the bottom of decoded reference pixels W1 to W3 in CU#A1, is copied to undecoded reference pixels B1 to B4 located in the same column in CU#A3, as shown in Figure 8(b).

Similarly, in conventional HEVC, as shown in Figure 8(d), the value of reference pixel B1, which is located at the bottom of decoded reference pixels B1 to B3 located in CU#A3, is specified to be copied to undecoded reference pixel P3, which is located in the same column in the CU below CU#A3.

As a result, when directional prediction is performed from the bottom left to the top right, as in the example of Figure 8, many of the generated predicted images are composed of undecoded reference pixels filled with copies, resulting in reduced prediction accuracy and reduced coding efficiency.

To address this issue, a technique is known for improving prediction accuracy in intra prediction, where TU division is performed, by allowing flexibility in the coding order for multiple TUs within a CU, such as raster scan order (e.g., Z-type), U-type, X-type, etc. (see Non-Patent Document 1).

Mochizuki et al., "An adaptive intra prediction method based on mean coordinates," Information Processing Society of Japan Research Report, vol. 2012-AVM-77, No. 12

However, the technology defined in the above-mentioned Non-Patent Document 1 requires the transmission of a flag indicating the encoding process order to be used on a CU-by-CU basis, which increases the amount of information to be transmitted. In addition, in order to select the best encoding process order from all the possible encoding process orders, the encoding device must try all possible combinations, which increases the calculation time on the encoding device side.

The present invention has been made to solve the above-mentioned problems, and aims to provide an encoding device, a decoding device, and a program that can improve prediction accuracy and encoding efficiency without increasing the amount of information transmitted by the encoding device or the calculation time on the encoding device side.

A first feature of the present invention is a decoding device configured to decode current blocks obtained by dividing an original image in units of frames constituting a video sequence, the decoding device comprising: a first prediction unit that generates a first predicted image by a first prediction process determined based on a prediction mode; a synthesis region determination unit that is configured to determine a synthesis region corresponding to the current block; a second prediction unit that generates a second predicted image by a second prediction process different from the first prediction process; and a predicted image synthesis unit that generates a predicted image included in the synthesis region by taking a weighted average of the first predicted image and the second predicted image, the predicted image synthesis unit determining a weight to use for the weighted average based on whether blocks adjacent to the current block have been decoded. An encoding device according to one embodiment is configured to divide an original image in units of frames constituting a video sequence into current blocks for encoding, the encoding device comprising: a synthesis region determination unit that is configured to determine a synthesis region in which a predicted image is generated from neighboring pixels within the current block that have not been decoded based on an intra prediction mode; and an intra prediction unit that is configured to change a method for generating a predicted image for each region based on whether the region is included in the synthesis region.

A decoding device according to one embodiment is configured to divide frame-by-frame original images constituting a moving image into blocks to be coded and decode them. The decoding device includes: a synthesis region determination unit configured to determine a synthesis region in which a predicted image is generated from adjacent pixels within the block to be coded that have not yet been decoded, based on an intra prediction mode; and an intra prediction unit configured to change the method for generating the predicted image for each region, based on whether or not it is included in the synthesis region.

The program according to one embodiment is a program for causing a computer to function as the encoding device described above.

The program according to one embodiment is a program for causing a computer to function as the above-described decoding device.

The present invention provides an encoding device, a decoding device, and a program that can improve prediction accuracy and encoding efficiency without increasing the amount of information transmitted by the encoding device or the calculation time on the encoding device side.

FIG. 1 is a functional block diagram of an encoding device 1 according to the first embodiment. FIG. 2 is a diagram showing an example of how a predicted image is generated in the first embodiment. FIG. 3 is a diagram showing an example of how a predicted image is generated in the first embodiment. FIG. 4 is a functional block diagram of the decoding device 3 according to the first embodiment. FIG. 5 is a flowchart showing the operations of the encoding device 1 and the decoding device 3 according to the first embodiment. FIG. 6 is a diagram showing an example of how a predicted image is generated in the second embodiment. FIG. 7 is a diagram showing an example of how a predicted image is generated in conventional HEVC. FIG. 8 is a diagram showing an example of how a predicted image is generated in conventional HEVC.

(First embodiment)
An encoding device 1 and a decoding device 3 according to a first embodiment of the present invention will be described below with reference to Figures 1 to 5. Here, the encoding device 1 and the decoding device 3 according to this embodiment are configured to support intra prediction in HEVC.

The encoding device 1 according to this embodiment is configured to divide original images, each consisting of a frame of a moving image, into blocks to be encoded and encode them. Note that in this embodiment, it is assumed that all adjacent pixels to the left and above the CU to be encoded have already been decoded.

The encoding device 1 according to this embodiment may also be configured to be able to divide a CU into multiple TUs. Therefore, in this embodiment, the block to be encoded may be either a CU or a TU. Below, this embodiment will be described taking as an example a case where a CU is used as the block to be encoded.

Furthermore, in this embodiment, an example is described in which the raster scan order (Z-type as shown in Figures 7 and 8) used in conventional HEVC is used as the coding order and decoding order of CUs, but other coding orders and decoding orders such as U-type and X-type may also be used as such coding order and decoding order.

As shown in FIG. 1, the encoding device 1 according to this embodiment includes an intra-prediction mode determination unit 11, a synthesis region determination unit 12, an intra-prediction unit 13, a residual signal generation unit 14, an orthogonal transform/quantization unit 15, an inverse orthogonal transform/inverse quantization unit 16, a local decoded image generation unit 17, a memory unit 18, and an entropy encoding unit 19.

The intra prediction mode determination unit 11 is configured to determine the optimal intra prediction mode to apply to the CU.

The synthesis area determination unit 12 is configured to determine a synthesis area X in which a predicted image is generated from adjacent pixels within the CU that have not yet been decoded, based on the intra prediction mode determined by the intra prediction mode determination unit 11.

Specifically, as shown in FIG. 2, when the direction of the intra prediction mode determined by the intra prediction mode determination unit 11 is from the bottom left to the top right (i.e., when directional prediction is performed from the bottom left to the top right), the synthesis area determination unit 12 may be configured to determine synthesis area X in which a predicted image is generated from adjacent pixels B1 to B4 that have not yet been decoded within CU#A2.

The present invention is also applicable when the direction of the intra prediction mode determined by the intra prediction mode determination unit 11 is from the top right to the bottom left (i.e., when directional prediction is performed from the top right to the bottom left). However, the following description will be given using an example where the direction of the intra prediction mode determined by the intra prediction mode determination unit 11 is from the bottom left to the top right (i.e., when directional prediction is performed from the bottom left to the top right).

Note that in the figures in this specification, the arrow indicating the direction of the intra prediction mode (prediction direction) points from the pixel targeted for intra prediction to the reference pixel, as described in the HEVC standard (the same applies below).

The intra prediction unit 13 is configured to change the method for generating a predicted image for each region based on whether or not it is included in the synthesis region X determined by the synthesis region determination unit 12.

Specifically, as shown in FIG. 3(a), in CU#A2, in an area Y that is not included in synthesis area X (i.e., an area where reference pixels W1 to W3 have already been decoded), the intra prediction unit 13 is configured to generate a predicted image based on the decoded reference pixels W1 to W3 using the intra prediction mode determined by the intra prediction mode determination unit 11.

On the other hand, as shown in Figure 3(b), the intra prediction unit 13 is configured to generate a predicted image using a predefined prediction method in the area of CU#A2 that is included in synthesis area X (i.e., the area where reference pixels B2 to B4 have not yet been decoded).

Here, examples of predefined prediction methods include intra-prediction methods such as DC prediction and planar prediction, as well as prediction methods that generate predicted images by averaging the values of adjacent decoded reference pixels.

Note that if the encoding device 1 and decoding device 3 have previously determined the prediction method, there is no need to transmit a new flag indicating the prediction method from encoding device 1 to decoding device 3, regardless of the prediction method used.

Furthermore, since the synthesis area determined by the synthesis area determination unit 12 can be uniquely determined based on the direction of the intra prediction mode and whether or not adjacent pixels have already been decoded, there is no need to transmit a flag indicating which area is the new synthesis area from the encoding device 1 to the decoding device 3.

For example, as shown in FIG. 3(b), the intra prediction unit 13 may be configured to generate a predicted image by using planar prediction to synthesize pixels W3, D1, D2, and D4 at the position of pixel X1 in an area included in synthesis area X in CU#A2. Here, the value of pixel B4 is copied into pixel D4, and the value of pixel W1 is copied into pixel B4.

The residual signal generation unit 14 is configured to generate a residual signal based on the difference between the predicted image generated by the intra prediction unit 13 and the original image.

The orthogonal transform and quantization unit 15 is configured to perform orthogonal transform processing and quantization processing on the residual signal generated by the residual signal generation unit 14, and generate quantized transform coefficients.

The inverse orthogonal transform and inverse quantization unit 16 is configured to again perform inverse quantization and inverse orthogonal transform processing on the quantized transform coefficients generated by the orthogonal transform and quantization unit 15, thereby generating a quantized residual signal.

The local decoded image generation unit 17 is configured to generate a local decoded image by adding the predicted image generated by the intra prediction unit 13 to the quantized residual signal generated by the inverse orthogonal transform and inverse quantization unit 16.

The memory unit 18 is configured to store the locally decoded image generated by the locally decoded image generation unit 17 so that it can be used as a reference image.

The entropy coding unit 19 is configured to perform entropy coding processing on flag information including the intra-prediction mode determined by the intra-prediction mode determination unit 11 and quantized transform coefficients, and output the result as a stream.

The decoding device 3 according to this embodiment is configured to divide original images, each consisting of a frame, into CUs and decode them. The decoding device 3 according to this embodiment is also configured to be able to divide a CU into multiple TUs, similar to the encoding device 1 according to this embodiment.

As shown in FIG. 4, the decoding device 3 according to this embodiment includes an entropy decoding unit 31, a synthesis region determination unit 32, an intra prediction unit 33, an inverse quantization and inverse transform unit 34, a local decoded image generation unit 35, and a memory unit 36.

The entropy decoding unit 31 is configured to decode transform coefficients, flag information, etc. from the stream output from the encoding device 1. Here, the transform coefficients are quantized transform coefficients obtained as signals encoded by the encoding device 1 by dividing an original image in frame units into CUs. Furthermore, the flag information includes accompanying information such as the prediction mode.

The synthesis area determination unit 32 is configured to determine a synthesis area X in which a predicted image is generated from adjacent pixels within the CU that have not yet been decoded, based on the intra prediction mode output by the entropy decoding unit 31.

Specifically, as shown in FIG. 2, when the direction of the intra prediction mode output by the entropy decoding unit 31 is from the bottom left to the top right (i.e., when directional prediction is performed from the bottom left to the top right), the synthesis area determination unit 32 may be configured to determine synthesis area X in which a predicted image is generated from adjacent pixels B1 to B4 that have not yet been decoded within CU#A2.

The intra prediction unit 33 may be configured to generate a predicted image using the synthesis area X determined by the synthesis area determination unit 32 and the intra prediction mode output by the entropy decoding unit 31.

Specifically, like the intra prediction unit 13, the intra prediction unit 33 is configured to change the method for generating a predicted image for each region based on whether or not the region is included in the synthesis region X determined by the synthesis region determination unit 12.

For example, as shown in FIG. 3(a), in CU#A2, in an area Y that is not included in synthesis area X (i.e., an area where reference pixels W1 to W3 have already been decoded), the intra prediction unit 33 is configured to generate a predicted image based on the decoded reference pixels W1 to W3 using the intra prediction mode output by the entropy decoding unit 31.

On the other hand, as shown in Figure 3(b), the intra prediction unit 33 is configured to generate a predicted image using the above-mentioned predefined prediction method in the area of CU#A2 that is included in synthesis area X (i.e., the area where reference pixels B2 to B4 have not yet been decoded).

The inverse quantization and inverse transform unit 34 is configured to generate a residual signal by performing inverse quantization processing and inverse transform processing (e.g., inverse orthogonal transform processing) on the quantized transform coefficients output by the entropy decoding unit 31.

The local decoded image generation unit 35 is configured to generate a local decoded image by adding the predicted image generated by the intra prediction unit 33 and the residual signal generated by the inverse quantization and inverse transform unit 34.

The memory unit 36 is configured to store the locally decoded images generated by the locally decoded image generation unit 35 so that they can be used as reference images for intra-prediction and inter-prediction.

Figure 5 shows a flowchart illustrating an example of the operation of generating a predicted image by the encoding device 1 and decoding device 3 according to this embodiment.

First, with reference to Figure 5, an example of the operation of generating a predicted image by the encoding device 1 according to this embodiment will be described.

As shown in FIG. 5, in step S101, the encoding device 1 determines the optimal intra prediction mode to apply to the CU.

In step S102, the encoding device 1 determines a synthesis area X in which a predicted image is generated from adjacent pixels within the CU that have not yet been decoded, based on the intra prediction mode determined in step S101.

In step S103, the encoding device 1 determines whether the area in the CU for which a predicted image is to be generated is included in the synthesis area. If the answer is "No," the operation proceeds to step S104; if the answer is "Yes," the operation proceeds to step S105.

In step S104, the encoding device 1 generates a predicted image based on the decoded reference pixels using the intra prediction mode determined in step S101.

In step S105, the encoding device 1 generates a predicted image using the above-mentioned predefined prediction method.

Secondly, with reference to Figure 5, an example of the operation of generating a predicted image by the decoding device 3 according to this embodiment will be described.

As shown in FIG. 5, in step S101, the decoding device 3 determines the intra prediction mode based on information obtained by the entropy decoding process.

In step S102, the decoding device 3 determines a synthesis area X in which a predicted image is generated from adjacent pixels within the CU that have not yet been decoded, based on the intra prediction mode determined in step S101.

In step S103, the decoding device 3 determines whether the area in the CU for which a predicted image is to be generated is included in the synthesis area. If the answer is "No," the operation proceeds to step S104; if the answer is "Yes," the operation proceeds to step S105.

In step S104, the decoding device 3 generates a predicted image based on the decoded reference pixels using the intra prediction mode determined in step S101.

In step S105, the decoding device 3 generates a predicted image using the above-mentioned predefined prediction method.

With the encoding device 1 and decoding device 3 according to this embodiment, when an undecoded pixel is included in the reference destination of the direction (prediction direction) of the intra prediction mode, a new predicted image is generated by combining a predicted image generated using reference pixels interpolated by copying adjacent decoded reference pixels with a predicted image generated using a predefined prediction method, thereby preventing a decrease in prediction accuracy.

Second Embodiment
Hereinafter, with reference to FIG. 6, the encoding device 1 and the decoding device 3 according to the second embodiment of the present invention will be described, focusing on the differences from the encoding device 1 and the decoding device 3 according to the first embodiment described above.

In the encoding device 1 according to this embodiment, the intra prediction unit 13 is configured to apply a smoothing filter to the predicted image of a boundary region between a region included in synthesis region X and a region Y not included in synthesis region X when a predetermined condition is satisfied in the boundary region.

Similarly, in the decoding device 3 according to this embodiment, the intra prediction unit 33 is configured to apply a smoothing filter to the predicted image of a boundary region between a region included in synthesis region X and a region Y not included in synthesis region X when a predetermined condition is satisfied in the boundary region.

For example, in the example of Figure 6, such a boundary area corresponds to the area where pixels X1, X2, X3, and X4 in the area included in synthesis area X and pixels Y1, Y2, and Y3 in area Y not included in synthesis area X are located.

Another predetermined condition is that the discontinuity in the boundary region is higher than a predetermined threshold. Here, discontinuity can be evaluated, for example, by expressing it as a linear combination of horizontal, vertical, or diagonal differences.

In other words, the intra prediction unit 13 and the intra prediction unit 33 are configured to apply a smoothing filter to the predicted image of the boundary region when the discontinuity evaluation value in the boundary region is higher than a predetermined threshold.

For example, in the example of Figure 6, the intra prediction unit 13 and the intra prediction unit 33 are configured to apply a smoothing filter to the predicted image of the area where pixels X1, X2, X3, and X4 in the area included in synthesis area X and pixels Y1, Y2, and Y3 in area Y not included in synthesis area X are located when the evaluation value of discontinuity in such a boundary area is higher than a predetermined threshold.

In the example of Figure 6, the intra prediction unit 13 and the intra prediction unit 33 are configured to also apply a smoothing filter to the predicted image of the area in which pixels X5, X6, and X7 are located within the area included in synthesis area X.

Here, by sharing the above-mentioned predetermined threshold between encoding device 1 and decoding device 3, there is no need to transmit a new flag from encoding device 1 to decoding device 3.

The encoding device 1 and decoding device 3 according to this embodiment can reduce discontinuity in boundary areas caused by changing the method of generating predicted images for each area based on whether or not it is included in synthesis area X, thereby improving encoding performance.

(Third embodiment)
Below, with reference to Figure 6, the encoding device 1 and decoding device 3 according to the third embodiment of the present invention will be described, focusing on the differences from the encoding device 1 and decoding device 3 according to the first and second embodiments described above.

In the encoding device 1 and decoding device 3 according to this embodiment, the intra prediction unit 13 and intra prediction unit 33 are configured to generate a predicted image in the area included in synthesis area X by taking a weighted average of a predicted image generated using a conventional HEVC prediction method (a predicted image generated based on undecoded reference pixels filled in by copying) and a predicted image generated in synthesis area X (a predicted image generated using the above-mentioned predefined prediction method).

In the encoding device 1 and decoding device 3 according to this embodiment, the intra prediction unit 13 and the intra prediction unit 33 may be configured to generate a predicted image in an area included in synthesis area X by taking a weighted average of a predicted image generated using a conventional HEVC prediction method (a predicted image generated based on undecoded reference pixels filled in by copying) and a predicted image generated in synthesis area X (a predicted image generated using the above-mentioned predefined prediction method) only when the above-mentioned predetermined conditions are met.

In the encoding device 1 and decoding device 3 according to this embodiment, the intra prediction unit 13 and the intra prediction unit 33 may be configured to calculate weighting coefficients used when calculating such weighted averages.

The intra prediction unit 13 and the intra prediction unit 33 may be configured to use predetermined values as weighting factors, or may be configured to calculate weighting factors using the reliability of decoded reference pixels when generating a predicted image.

In the example of Figure 3(b), the reliability of the decoded reference pixels W1 to W4 is high, while the reliability of the reference pixels B1 to B4 generated by copying is low. Furthermore, when comparing the reliability of reference pixel B1 with that of reference pixel B4, the reliability of reference pixel B4, which is closer to the source of the copy, is higher.

Furthermore, the intra prediction unit 13 and the intra prediction unit 33 may be configured to calculate such weighting coefficients according to the distance from the boundary area between the area included in synthesis area X and area Y not included in synthesis area X.

For example, the intra prediction unit 13 and the intra prediction unit 33 may calculate a weighting factor to be stronger near such boundary areas for predicted images generated using a conventional HEVC prediction method (predicted images generated based on undecoded reference pixels filled in by copying), and may calculate a weighting factor to be stronger in areas away from such boundary areas for predicted images generated in synthesis area X (predicted images generated using the above-mentioned predefined prediction method).

(Other embodiments)
As described above, the present invention has been described by the above-mentioned embodiment, but the descriptions and drawings forming part of the disclosure of such embodiment should not be understood to limit the present invention. From such disclosure, various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art.

Although not specifically mentioned in the above embodiment, a program may be provided that causes a computer to execute the processes performed by the encoding device 1 and decoding device 3 described above. Such a program may also be recorded on a computer-readable medium. Using a computer-readable medium, such a program can be installed on a computer. Here, the computer-readable medium on which such a program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a CD-ROM, DVD-ROM, or other recording medium.

Alternatively, a chip may be provided that is configured with a memory that stores a program for implementing at least some of the functions of the encoding device 1 and decoding device 3 described above, and a processor that executes the program stored in the memory.

1...Encoding device 11...Intra prediction mode determination unit 12...Synthesis region determination unit 13...Intra prediction unit 14...Residual signal generation unit 15...Orthogonal transform/quantization unit 16...Inverse quantization unit/inverse orthogonal transform unit 17...Local decoded image generation unit 18...Memory unit 19...Entropy encoding unit 3...Decoding device 31...Entropy decoding unit 32...Synthesis region determination unit 33...Intra prediction unit 34...Inverse quantization/inverse transform unit 35...Local decoded image generation unit 36...Memory unit

Claims (2)

1. A decoding device configured to decode encoding target blocks obtained by dividing an original image in units of frames that constitute a moving image, comprising:
a first prediction unit that generates a first predicted image by a first prediction process that is determined based on a prediction mode designated by flag information transmitted from an encoding side;
a synthesis area determination unit configured to determine a synthesis area in the encoding target block;
a second prediction unit that generates a second predicted image by a second prediction process that is different from the first prediction process and is determined without transmission of new flag information other than the flag information transmitted from an encoding side;
a predicted image synthesis unit that generates a predicted image included in the synthesis area by taking a weighted average of the first predicted image and the second predicted image,
The decoding device is characterized in that the predicted image synthesis unit determines the weight to be used for the weighted average for each pixel depending on the position from a boundary area between the first predicted image and the second predicted image. A program for causing a computer to function as the decoding device described in claim 1.