JP7812966B2 - Image encoding device, image decoding device, and program - Google Patents

Description

The present invention relates to an image encoding device, an image decoding device, and a program.

In a conventional image coding device that encodes target images in blocks obtained by dividing a current image in frames, a method is known in which the target image is predicted using multiple reference images to generate a predicted image, the prediction residual indicating the difference between the target image and the predicted image is subjected to orthogonal transform processing to calculate transform coefficients, and the transform coefficients are quantized and entropy coded to output coded data.

Also, like an image encoding device, an image decoding device predicts a target image using multiple reference images to generate a predicted image. The image decoding device decodes encoded data to obtain transform coefficients and dequantizes them, performs inverse orthogonal transform processing on the dequantized transform coefficients to calculate prediction residuals, and decodes the target image by combining the predicted image and the prediction residuals.

HEVC specifies two types of orthogonal transform that can be applied to transform processes (orthogonal transform processes and inverse orthogonal transform processes): DCT-2 and DST-7 (see Non-Patent Document 1). Specifically, HEVC determines which of the two types of orthogonal transform to apply based on the block size of the target image and the intra-prediction mode applied to the target image.

Recommendation ITU-T H. 265, (12/2016), “High efficiency video coding”, International Telecommunication Union

However, simply determining the type of orthogonal transform based on the block size of the target image and the intra-prediction mode applied to the target image does not allow for the optimal type of orthogonal transform to be applied in accordance with the energy distribution of the prediction residual. For example, even when DCT-2 would actually be more efficient at concentrating the energy of the prediction residual, DST-7 may be selected as the orthogonal transform to be applied, resulting in a decrease in coding efficiency.

KLT is an orthogonal transform method that can more efficiently concentrate the energy of prediction residuals. However, since the image decoding device requires information for the inverse process of KLT performed by the image encoding device, the amount of information to be transmitted increases, resulting in a problem of reduced encoding efficiency.

Therefore, an object of the present invention is to provide an image encoding device, an image decoding device, and a program that can improve encoding efficiency.

An image coding device according to a first feature encodes a target image in blocks obtained by dividing a current image in frames, and includes: a prediction unit that predicts the target image using multiple reference images to generate a predicted image; an evaluation unit that generates map information indicating an error distribution in the predicted image by evaluating the similarity between the multiple reference images on a pixel-by-pixel basis; a subtraction unit that calculates a prediction residual indicating a difference between the target block and the predicted image on a pixel-by-pixel basis; a determination unit that determines an orthogonal transform to apply to the prediction residual based on the map information; and a transformation unit that performs orthogonal transform processing on the prediction residual using the determined orthogonal transform. An image coding device according to another feature encodes a target image in blocks obtained by dividing a current image in frames, and includes: a prediction unit that predicts the target image using multiple reference images to generate a predicted image; and an evaluation unit that calculates a sum of absolute differences indicating the similarity between the multiple reference images in units of regions smaller than the blocks and consisting of multiple pixels; and controls the encoding process based on the sum of absolute differences calculated by the evaluation unit for each region.

An image decoding device according to a second feature is an image decoding device that decodes a target image in blocks obtained by dividing a current image in frames, and includes: a decoding unit that obtains transform coefficients by decoding encoded data; a prediction unit that predicts the target image using multiple reference images to generate a predicted image; an evaluation unit that generates map information indicating an error distribution in the predicted image by evaluating the similarity between the multiple reference images on a pixel-by-pixel basis; a determination unit that determines an inverse orthogonal transform to apply to the transform coefficients based on the map information; and an inverse transform unit that performs inverse orthogonal transform processing on the transform coefficients using the determined inverse orthogonal transform. An image decoding device according to another feature is an image decoding device that decodes a target image in blocks obtained by dividing a current image in frames, and includes: a prediction unit that predicts the target image using multiple reference images to generate a predicted image; and an evaluation unit that calculates a sum of absolute differences indicating the similarity between the multiple reference images in units of regions smaller than the blocks and consisting of multiple pixels; and controls the decoding process based on the sum of absolute differences calculated by the evaluation unit for each region.

The program relating to the third feature is characterized in that it causes a computer to function as the image encoding device relating to the first feature.

The program relating to the fourth feature is characterized in that it causes a computer to function as the image decoding device relating to the second feature.

The present invention provides an image encoding device, an image decoding device, and a program that can improve encoding efficiency.

FIG. 1 is a diagram showing a configuration of an image encoding device according to a first embodiment. FIG. 10 is a diagram showing an example of inter prediction according to the first to third embodiments. FIG. 2 is a diagram illustrating an example of a configuration of an evaluation unit according to the first to third embodiments. FIG. 4 is a diagram illustrating an operation of an adaptive transformation generation unit according to the first embodiment. FIG. 1 is a diagram showing the configuration of an image decoding device according to a first embodiment. FIG. 10 is a diagram showing the configuration of an image encoding device according to a second embodiment. FIG. 10 is a diagram showing the configuration of an image decoding device according to a second embodiment. FIG. 11 is a diagram showing the configuration of an image encoding device according to a third embodiment. FIG. 11 is a diagram illustrating an operation of a feature amount evaluation unit according to the third embodiment. FIG. 11 is a diagram showing the configuration of an image decoding device according to a third embodiment.

An image encoding device and an image decoding device according to an embodiment will be described with reference to the drawings. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals.

First Embodiment
An image encoding device and an image decoding device according to the first embodiment will be described below. The image encoding device and the image decoding device according to the first embodiment encode and decode moving images, such as MPEG.

(Image encoding device)
Fig. 1 is a diagram showing the configuration of an image coding device 1 according to the first embodiment. As shown in Fig. 1, the image coding device 1 includes a block division unit 100, a subtraction unit 110, a transformation and quantization unit 120, an entropy coding unit 130, an inverse quantization and inverse transform unit 140, a synthesis unit 150, a memory 160, a prediction unit 170, an evaluation unit 180, and a determination unit 190.

The block division unit 100 divides the input image, which is made up of frames (or pictures) that make up a moving image, into small block-shaped regions and outputs the blocks obtained by division to the subtraction unit 110. The size of the blocks is, for example, 32 x 32 pixels, 16 x 16 pixels, 8 x 8 pixels, or 4 x 4 pixels. The shape of the blocks is not limited to squares and may be rectangular. Blocks are the units used for encoding by the image encoding device 1 and for decoding by the image decoding device 2.

The subtraction unit 110 calculates a prediction residual that indicates the pixel-by-pixel difference between the block input from the block division unit 100 and the predicted image (predicted block) obtained by predicting that block by the prediction unit 170. Specifically, the subtraction unit 110 calculates the prediction residual by subtracting each pixel value of the predicted image from each pixel value of the block, and outputs the calculated prediction residual to the transformation/quantization unit 120.

The transform/quantization unit 120 performs orthogonal transform processing and quantization processing on a block-by-block basis. The transform/quantization unit 120 includes a transform unit 121 and a quantization unit 122.

The transform unit 121 performs orthogonal transform processing on the prediction residuals input from the subtraction unit 110 to calculate transform coefficients, and outputs the calculated transform coefficients to the quantization unit 122. Examples of orthogonal transform include discrete cosine transform (DCT), discrete sine transform (DST), and Karhunen-Loeve transform (KLT). In the first embodiment, the transform unit 121 performs orthogonal transform processing using KLT.

The quantization unit 122 quantizes the transform coefficients input from the transform unit 121 using a quantization parameter (Qp) and a quantization matrix, and outputs the quantized transform coefficients to the entropy coding unit 130 and the inverse quantization/inverse transform unit 140. The quantization parameter (Qp) is a parameter that is commonly applied to each transform coefficient in a block and determines the coarseness of quantization. The quantization matrix is a matrix whose elements are the quantization values used when quantizing each transform coefficient.

The entropy coding unit 130 performs entropy coding on the transform coefficients input from the quantization unit 122, compresses the data, generates coded data (bitstream), and outputs the coded data to the outside of the image coding device 1. Entropy coding can use Huffman coding or CABAC (Context-based Adaptive Binary Arithmetic Coding), among others. Note that the entropy coding unit 130 also receives control information related to prediction from the prediction unit 170 and performs entropy coding on the input control information.

The inverse quantization and inverse transform unit 140 performs inverse quantization and inverse orthogonal transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 140 includes an inverse quantization unit 141 and an inverse transform unit 142.

The inverse quantization unit 141 performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 122. Specifically, the inverse quantization unit 141 restores the transform coefficients by inverse quantizing the transform coefficients input from the quantization unit 122 using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 142.

The inverse transform unit 142 performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 121. For example, if the transform unit 121 performs a discrete cosine transform, the inverse transform unit 142 performs an inverse discrete cosine transform. The inverse transform unit 142 performs inverse orthogonal transform processing on the transform coefficients input from the inverse quantization unit 141 to restore the prediction residual, and outputs the restored prediction residual, which is the restored prediction residual, to the synthesis unit 150.

The synthesis unit 150 synthesizes, on a pixel-by-pixel basis, the reconstructed prediction residual input from the inverse transform unit 142 with the predicted image input from the prediction unit 170. The synthesis unit 150 adds each pixel value of the reconstructed prediction residual to each pixel value of the predicted image to reconstruct (decode) a block, and outputs the decoded image in block units to the memory 160. Such a decoded image is sometimes referred to as a reconstructed image.

Memory 160 stores the decoded image input from synthesis unit 150. Memory 160 stores the decoded image on a frame-by-frame basis. Memory 160 outputs the stored decoded image to prediction unit 170. Note that a loop filter may be provided between synthesis unit 150 and memory 160.

The prediction unit 170 performs prediction on a block-by-block basis. The prediction unit 170 includes an intra prediction unit 171, an inter prediction unit 172, and a switching unit 173.

The intra prediction unit 171 generates an intra prediction image by referencing decoded pixel values surrounding the block to be predicted from among the decoded images stored in the memory 160, and outputs the generated intra prediction image to the switching unit 173. The intra prediction unit 171 also selects the optimal intra prediction mode to apply to the current block from among multiple intra prediction modes, and performs intra prediction using the selected intra prediction mode. The intra prediction unit 171 outputs control information related to the selected intra prediction mode to the entropy coding unit 130. Note that intra prediction modes include planar prediction, DC prediction, and directional prediction.

The inter prediction unit 172 uses the decoded image stored in the memory 160 as a reference image to calculate a motion vector using a technique such as block matching, predicts the block to be predicted, and generates an inter prediction image, which it outputs to the switching unit 173. The inter prediction unit 172 selects the most appropriate inter prediction method from inter prediction using multiple reference images (typically, bi-prediction) and inter prediction using a single reference image (unidirectional prediction), and performs inter prediction using the selected inter prediction method. The inter prediction unit 172 outputs control information related to the inter prediction (such as information on the inter prediction method and motion vectors) to the entropy coding unit 130. When multiple reference images are used to generate the inter prediction image, the inter prediction unit 172 outputs the multiple reference images to the evaluation unit 180.

Note that prediction using multiple reference images is typically performed using bi-prediction in inter prediction, but is not limited to this. The prediction unit 170 may also perform prediction using intra block copying using multiple reference images. In intra block copying, a reference image in the same frame as the current frame is used to predict a block in the current frame. When performing prediction using intra block copying using multiple reference images, the prediction unit 170 outputs the multiple reference images to the evaluation unit 180.

The switching unit 173 switches between the intra-predicted image input from the intra-prediction unit 171 and the inter-predicted image input from the inter-prediction unit 172, and outputs either of the predicted images to the subtraction unit 110 and the synthesis unit 150.

The evaluation unit 180 evaluates the similarity between the multiple reference images input from the prediction unit 170 on a pixel-by-pixel basis, thereby generating map information indicating the distribution of errors in the predicted image generated using the multiple reference images, and outputs the generated map information to the determination unit 190. Details of the evaluation unit 180 will be described later.

Based on the map information input from the evaluation unit 180, the determination unit 190 determines the orthogonal transform to apply to the prediction residual corresponding to the predicted image whose prediction accuracy has been evaluated by the evaluation unit 180, and outputs the determined orthogonal transform to the transformation unit 121 and the inverse transformation unit 142. In the first embodiment, the determination unit 190 includes an adaptive transformation generation unit 191 that generates a vertical orthogonal transformation and a horizontal orthogonal transformation based on the map information. The transformation unit 121 performs orthogonal transformation processing in accordance with the orthogonal transformation input from the determination unit 190. The inverse transformation unit 142 performs orthogonal transformation processing in accordance with the orthogonal transformation input from the determination unit 190. Details of the adaptive transformation generation unit 191 will be described later.

(An example of inter prediction)
2A and 2B are diagrams showing an example of inter prediction, in which Fig. 2A shows bi-prediction as an example of inter prediction, and Fig. 2B shows an example of a predicted image generated by bi-prediction.

As shown in Figure 2(a), bi-prediction refers to frames temporally before and after the target frame (current frame). In the example of Figure 2(a), prediction of a block in the image of frame t is performed by referring to frames t-1 and t+1. Motion estimation detects locations (blocks) similar to the target image block within the reference frames t-1 and t+1 within a search range set by the system.

The detected location is the reference image. Information indicating the relative position of the reference image with respect to the target image block is represented by the arrow in the figure and is called a motion vector. The motion vector information is encoded by the image encoding device 1 using entropy coding together with the frame information of the reference image. Meanwhile, the image decoding device detects the reference image based on the motion vector information generated by the image encoding device 1.

As shown in Figures 2(a) and 2(b), reference images 1 and 2 detected by motion detection are similar partial images that are aligned with the target image block within the reference frame, and are therefore similar to the target image block (image to be encoded). In the example of Figure 2(b), the target image block includes a star pattern and a partial circle pattern. Reference image 1 includes a star pattern and a full circle pattern. Reference image 2 includes a star pattern, but does not include a circle pattern.

A predicted image is generated from these reference images 1 and 2. Note that prediction processing generally generates a predicted image with the characteristics of each reference image by averaging reference images 1 and 2, which have different characteristics but are partially similar. However, predicted images may also be generated using more advanced processing, such as signal enhancement processing using low-pass filters or high-pass filters. Here, reference image 1 contains a circular pattern, while reference image 2 does not. Therefore, when reference images 1 and 2 are averaged to generate a predicted image, the signal of the circular pattern in the predicted image is halved compared to reference image 1.

The difference between the predicted image obtained from reference images 1 and 2 and the target image block (image to be encoded) is the prediction residual. In the prediction residual shown in Figure 2(b), a large difference occurs only in the misaligned parts of the star image and the circle image (shaded area), but for other parts, predictions are made accurately and the difference is small (no difference occurs in the example of Figure 2(b)).

The areas where no differences occur (non-edge parts of the star pattern and background parts) are areas where the similarity between Reference Image 1 and Reference Image 2 is high, and where highly accurate predictions have been made. On the other hand, the areas where large differences occur are areas unique to each reference image, i.e., areas where the similarity between Reference Image 1 and Reference Image 2 is significantly low. Therefore, it can be seen that areas where the similarity between Reference Image 1 and Reference Image 2 is significantly low have low prediction accuracy, resulting in large differences (residuals).

When prediction residuals, which contain a mixture of large and small differences, are orthogonally transformed and quantization causes degradation of the transform coefficients, this degradation of the transform coefficients propagates throughout the image (block) through inverse quantization and inverse orthogonal transform. When the prediction residuals (restored prediction residuals) restored by inverse quantization and inverse orthogonal transform are combined with a predicted image to reconstruct the target image block, the degradation of image quality propagates to areas where high-precision predictions were made, such as the non-edge and background parts of the star pattern shown in Figure 2(b).

(Evaluation Department)
Fig. 3 is a diagram showing an example of the configuration of the evaluation unit 180. As shown in Fig. 3, the evaluation unit 180 includes a difference calculation unit (subtraction unit) 180a, a normalization unit 180b, and an adjustment unit 180c.

The difference calculation unit 180a calculates the difference value (absolute value of the difference) between reference image 1 and reference image 2 on a pixel-by-pixel basis, and outputs the calculated difference value to the normalization unit 180b. Such difference value is an example of a value indicating similarity. It can be said that the smaller the difference value, the higher the similarity, and the larger the difference value, the lower the similarity. The difference calculation unit 180a may calculate the difference value after performing a filter process on each reference image. The difference calculation unit 180a may also calculate a statistical quantity such as squared error and use this statistical quantity as the similarity.

The normalization unit 180b normalizes the difference value input from the difference calculation unit 180a by the largest difference value within the block (i.e., the maximum difference value within the block) and outputs the normalized value. The smaller the difference value, the higher the similarity and the higher the prediction accuracy. On the other hand, the larger the difference value, the lower the similarity and the lower the prediction accuracy (the larger the prediction error).

The normalization unit 180b normalizes the difference value of each pixel input from the difference calculation unit 180a by the difference value of the pixel with the largest difference value in the block (i.e., the maximum difference value in the block), and outputs the normalized difference value. This normalized difference value can be used as an estimate representing the magnitude of the prediction error.

The adjustment unit 180c adjusts the normalized difference value input from the normalization unit 180b based on a quantization parameter (Qp) that determines the coarseness of quantization, and outputs the adjusted normalized difference value. Since the greater the coarseness of quantization, the greater the degree of degradation of the restored prediction residual, the adjustment unit 180c adjusts the normalized difference value (weight) based on the quantization parameter (Qp).

The estimated value Rij of the prediction error at each pixel position (ij) output by the evaluation unit 180 can be expressed, for example, as in the following equation (1):

Rij = (abs(Xij-Yij)/maxD × Scale(Qp)) ・・・(1)

In equation (1), Xij is the pixel value of pixel ij in reference image 1, Yij is the pixel value of pixel ij in reference image 2, and abs is a function that obtains the absolute value. The difference calculation unit 180a outputs abs(Xij - Yij).

In addition, in equation (1), maxD is the maximum value of the difference value abs(Xij - Yij) within the block. To calculate maxD, it is necessary to calculate the difference value for all pixels within the block, but to omit this process, the maximum value of an adjacent block that has already been encoded may be used instead. Alternatively, maxD may be calculated from the quantization parameter (Qp) using a table that defines the correspondence between the quantization parameter (Qp) and maxD. Alternatively, a fixed value defined in advance in the specifications may be used as maxD. The normalization unit 180b outputs abs(Xij - Yij)/maxD.

In addition, in equation (1), Scale (Qp) is a coefficient multiplied according to the quantization parameter (Qp). Scale (Qp) is designed to approach 1.0 when Qp is large and to approach 0 when Qp is small, with the degree of this being adjusted by the system. Alternatively, a fixed value defined in advance in the specifications may be used as Scale (Qp). Furthermore, to simplify processing, Scale (Qp) may be a fixed value designed according to the system, such as 1.0.

The adjustment unit 180c outputs abs(Xij-Yij)/maxD×Scale(Qp) as the error estimate Rij. This Rij may also be weighted and adjusted using a sensitivity function designed for the system. For example, the sensitivity may be adjusted by setting abs(Xij-Yij)/maxD×Scale(Qp)=Rij and Rij=Clip(Rij,1.0,0.0), or by adding an offset such as Rij=Clip(Rij+offset,1.0,0.0). Note that Clip(x,max,min) indicates clipping at max if x exceeds max, and at min if x is below min.

The error estimate Rij for each pixel position calculated in this way is a value in the range from 0 to 1.0. Basically, the error estimate Rij approaches 1.0 when the difference value of pixel position ij between the reference images is large (i.e., the prediction accuracy is low), and approaches 0 when the difference value of pixel position ij between the reference images is small (i.e., the prediction accuracy is high). The evaluation unit 180 outputs two-dimensional map information (hereinafter referred to as the "error map") consisting of the error estimate Rij for each pixel position ij within the block.

(Adaptive transformation generator)
4 is a diagram showing the operation of the adaptive transform generation unit 191 according to the first embodiment. The adaptive transform generation unit 191 uses the error map input from the evaluation unit 180 to generate a vertical adaptive orthogonal transform to be applied to the prediction residual in the vertical direction and a horizontal adaptive orthogonal transform to be applied to the prediction residual in the horizontal direction by principal component analysis.

Specifically, the adaptive transform generation unit 191 generates a covariance matrix by treating the error map as a set of column vectors, and calculates the eigenvectors of the generated covariance matrix. The adaptive transform generation unit 191 outputs the obtained eigenvectors as a vertical adaptive orthogonal transform. Furthermore, the adaptive transform generation unit 191 generates a covariance matrix by treating the matrix obtained by applying the generated vertical adaptive orthogonal transform in the vertical direction as a set of row vectors, and calculates its eigenvectors. The adaptive transform generation unit 191 outputs the obtained eigenvectors as a horizontal adaptive orthogonal transform.

As shown in FIG. 4A, when the error map has width w and height h, the adaptive transformation generation unit 191 regards the error map as w h × 1 column vectors and calculates a covariance matrix Λ _h . The adaptive transformation generation unit 191 calculates eigenvectors by diagonalizing the obtained covariance matrix. Here, the covariance matrix is diagonalized by iterative calculations using, for example, the Jacobi method. The adaptive transformation generation unit 191 outputs an h × h matrix as a vertical adaptive orthogonal transform by combining the obtained eigenvectors e ₀ to e _h transforms.

4(b), the adaptive transformation generator 191 regards the error map as h 1×w row vectors and calculates a covariance matrix Λ _h . The adaptive transformation generator 191 calculates eigenvectors by diagonalizing the obtained covariance matrix. The adaptive transformation generator 191 combines the obtained eigenvectors to output a w×w matrix as a horizontal adaptive orthogonal transform.

Video coding methods such as HEVC (see Non-Patent Document 1) and the latest video coding technology (JEM) currently being considered by international standardization organizations use integer-precision transform coefficients and bit shifting to perform transform processing quickly and efficiently. The adaptive transform generation unit 191 according to this embodiment may also approximate the obtained eigenvectors to integer coefficients, and the amount of bit shifting may be specified in advance in the image coding device and image decoding device so as not to expand the dynamic range of the transform coefficients.

As shown in FIG. 1, the transform unit 121 calculates transform coefficients by performing orthogonal transform processing in the vertical and horizontal directions on the prediction residuals generated by the subtraction unit 110 using the vertical adaptive orthogonal transform and horizontal adaptive orthogonal transform input from the adaptive transform generation unit 191, and outputs the calculated transform coefficients to the quantization unit 122.

In addition, the inverse transform unit 142 uses the vertical adaptive orthogonal transform and horizontal adaptive orthogonal transform input from the adaptive transform generation unit 191 to perform inverse orthogonal transform processing on the transform coefficients input from the inverse quantization unit 141, corresponding to the orthogonal transform processing performed by the transform unit 121.

(Image decoding device)
Fig. 5 is a diagram showing the configuration of an image decoding device 2 according to the first embodiment. As shown in Fig. 5, the image decoding device 2 includes an entropy code decoding unit 200, an inverse quantization and inverse transform unit 210, a synthesis unit 220, a memory 230, a prediction unit 240, an evaluation unit 250, and a determination unit 260.

The entropy code decoding unit 200 decodes the coded data generated by the image coding device 1 and outputs the quantized transform coefficients to the inverse quantization and inverse transform unit 210. The entropy code decoding unit 200 also acquires control information related to prediction (intra prediction and inter prediction) and outputs the acquired control information to the prediction unit 240.

The inverse quantization and inverse transform unit 210 performs inverse quantization and inverse orthogonal transform processing on a block-by-block basis. The inverse quantization and inverse transform unit 210 includes an inverse quantization unit 211 and an inverse transform unit 212.

The inverse quantization unit 211 performs inverse quantization processing corresponding to the quantization processing performed by the quantization unit 122 of the image encoding device 1. The inverse quantization unit 211 restores the transform coefficients by inverse quantizing the quantized transform coefficients input from the entropy encoding/decoding unit 200 using a quantization parameter (Qp) and a quantization matrix, and outputs the restored transform coefficients to the inverse transform unit 212.

The inverse transform unit 212 performs inverse orthogonal transform processing corresponding to the orthogonal transform processing performed by the transform unit 121 of the image encoding device 1. The inverse transform unit 212 performs inverse orthogonal transform processing on the transform coefficients input from the inverse quantization unit 211 to restore the prediction residual, and outputs the restored prediction residual (restored prediction residual) to the synthesis unit 220.

The synthesis unit 220 reconstructs (decodes) the original block by synthesizing the prediction residual input from the inverse transform unit 212 and the predicted image input from the prediction unit 240 on a pixel-by-pixel basis, and outputs the decoded image on a block-by-block basis to the memory 230.

The memory 230 stores the decoded image input from the synthesis unit 220. The memory 230 stores the decoded image in frame units. The memory 230 outputs the decoded image in frame units to the outside of the image decoding device 2. Note that a loop filter may be provided between the synthesis unit 220 and the memory 230.

The prediction unit 240 performs prediction on a block-by-block basis. The prediction unit 240 includes an intra prediction unit 241, an inter prediction unit 242, and a switching unit 243.

The intra prediction unit 241 references the decoded image stored in the memory 230 and performs intra prediction in accordance with the control information input from the entropy coding/decoding unit 200 to generate an intra prediction image, and outputs the generated intra prediction image to the switching unit 243.

The inter prediction unit 242 performs inter prediction, which predicts the block to be predicted, using the decoded image stored in the memory 230 as a reference image. The inter prediction unit 242 generates an inter prediction image by performing inter prediction in accordance with the control information (inter prediction method, motion vector information, etc.) input from the entropy code decoding unit 200, and outputs the generated inter prediction image to the switching unit 243. When multiple reference images are used to generate the inter prediction image, the inter prediction unit 242 outputs the multiple reference images to the evaluation unit 250.

Note that prediction using multiple reference images is typically performed using bi-prediction in inter prediction, but is not limited to this. The prediction unit 240 may also perform prediction using intra block copying using multiple reference images. When performing prediction using intra block copying using multiple reference images, the prediction unit 240 outputs the multiple reference images to the evaluation unit 250.

The switching unit 243 switches between the intra-predicted image input from the intra-prediction unit 241 and the inter-predicted image input from the inter-prediction unit 242, and outputs one of the predicted images to the synthesis unit 220.

The evaluation unit 250 operates in the same manner as the evaluation unit 180 (see Figure 3) of the image encoding device 1. The evaluation unit 250 evaluates the similarity between the multiple reference images input from the prediction unit 240 on a pixel-by-pixel basis, thereby generating an error map that indicates the distribution of errors in the predicted image generated using the multiple reference images, and outputs the generated error map to the determination unit 260.

The determination unit 260 determines the inverse orthogonal transform to apply to the prediction residual corresponding to the predicted image whose prediction accuracy has been evaluated by the evaluation unit 250, based on the error map input from the evaluation unit 250, and outputs the determined inverse orthogonal transform to the inverse transform unit 212. The determination unit 260 includes an adaptive transform generation unit 261 that generates a vertical orthogonal transform and a horizontal orthogonal transform based on the error map. The adaptive transform generation unit 261 performs the same operation as the adaptive transform generation unit 191 of the image encoding device 1 (see Figure 4). The inverse transform unit 212 performs inverse orthogonal transform processing in accordance with the inverse orthogonal transform input from the determination unit 260.

(Summary of the first embodiment)
An image encoding device 1 according to the first embodiment encodes a target image in units of blocks obtained by dividing a current image in units of frames. The image encoding device 1 includes a prediction unit 170 that predicts the target image using multiple reference images to generate a predicted image, an evaluation unit 180 that generates an error map indicating the distribution of errors in the predicted image by evaluating the similarity between the multiple reference images on a pixel-by-pixel basis, a subtraction unit 110 that calculates a prediction residual indicating the difference between the target block and the predicted image on a pixel-by-pixel basis, a determination unit 190 that determines an orthogonal transform to apply to the prediction residual based on the error map, and a transformation unit 121 that performs orthogonal transform processing on the prediction residual using the determined orthogonal transform. The determination unit 190 includes an adaptive transform generation unit 191 that generates an orthogonal transform based on the map information.

The image decoding device 2 according to the first embodiment also decodes target images in units of blocks obtained by dividing a current image in units of frames. The image decoding device 2 includes an entropy code decoding unit 200 that obtains transform coefficients by decoding encoded data, a prediction unit 240 that predicts the target image using multiple reference images to generate a predicted image, an evaluation unit 250 that generates an error map indicating the distribution of errors in the predicted image by evaluating the similarity between the multiple reference images on a pixel-by-pixel basis, a determination unit 260 that determines the inverse orthogonal transform to apply to the transform coefficients based on the error map, and an inverse transform unit 212 that performs inverse orthogonal transform processing on the transform coefficients using the determined inverse orthogonal transform. The determination unit 260 includes an adaptive transform generation unit 261 that generates an inverse orthogonal transform based on the map information.

In this way, according to the first embodiment, by generating an orthogonal transform based on an error map that indicates the distribution of errors in a predicted image, it is possible to apply an optimal orthogonal transform that corresponds to the energy distribution in the prediction residual.

Furthermore, because the image encoding device 1 and the image decoding device 2 can each generate an orthogonal transform based on an error map, the image decoding device 2 does not need information for the inverse process of the orthogonal transform (KLT) performed by the image encoding device 1. This prevents an increase in the amount of information to be transmitted.

Therefore, the image encoding device 1 and image decoding device 2 according to the first embodiment make it possible to apply adaptive orthogonal transform that efficiently concentrates the energy of prediction residuals, thereby improving encoding efficiency.

Second Embodiment
The image encoding device 1 and image decoding device 2 according to the second embodiment will be described, focusing on the differences from the first embodiment.

(Image encoding device)
Fig. 6 is a diagram showing the configuration of an image encoding device 1 according to the second embodiment. As shown in Fig. 6, the image encoding device 1 according to the second embodiment differs from the first embodiment in the configuration of a decision unit 190. As in the first embodiment, the decision unit 190 includes an adaptive transform generation unit 191 that generates an orthogonal transform by principal component analysis of an error map (see Fig. 4). In the second embodiment, the decision unit 190 further includes a candidate selection unit (first selection unit) 192 and an orthogonal transform selection unit (second selection unit) 193.

The adaptive transform generation unit 191 outputs the adaptive orthogonal transforms (vertical adaptive orthogonal transform and horizontal adaptive orthogonal transform) generated based on the error map to the candidate selection unit 192.

The candidate selection unit 192 selects one or more orthogonal transform candidates from among multiple predefined types of orthogonal transforms in descending order of correlation with the adaptive orthogonal transform input from the adaptive transform generation unit 191, and outputs the selected one or more orthogonal transform candidates to the orthogonal transform selection unit 263. The multiple predefined types of orthogonal transforms are shared by the image encoding device 1 and the image decoding device 2. In the second embodiment, it is assumed that DCT-2, DST-7, DCT-8, DST-1, and DCT-5 are predefined as the multiple types of orthogonal transforms.

Specifically, the candidate selection unit 192 evaluates the correlation between the horizontal adaptive orthogonal transform input from the adaptive transform generation unit 191 and each orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5). Based on the results of the correlation evaluation, the candidate selection unit 192 then outputs one or more orthogonal transforms from among multiple types of orthogonal transforms in descending order of correlation as adaptive orthogonal transform candidates to the orthogonal transform selection unit 193.

The number of adaptive orthogonal transform candidates output by the candidate selection unit 192 is specified in advance to be the same for the image encoding device 1 and the image decoding device 2. The number of adaptive orthogonal transform candidates output by the candidate selection unit 192 may also be variable depending on the block size and color components (luminance component, chrominance component) of the image block to be encoded, etc.

Furthermore, the candidate selection unit 192 may select adaptive orthogonal transform candidates separately for the horizontal and vertical directions. In such a case, the candidate selection unit 192 further evaluates the correlation between the vertical adaptive orthogonal transform input from the adaptive transform generation unit 191 and each orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5), and outputs adaptive vertical orthogonal transform candidates in addition to adaptive horizontal orthogonal transform candidates to the orthogonal transform selection unit 193.

The orthogonal transform selection unit 193 selects an orthogonal transform to apply to the prediction residual from one or more orthogonal transform candidates input from the candidate selection unit 192, and outputs the selected orthogonal transform to the transform unit 121 and the inverse transform unit 142. The orthogonal transform selection unit 193 also outputs an index indicating the type of selected orthogonal transform to the entropy coding unit 130.

For example, the orthogonal transform selection unit 193 calculates the coding efficiency when each orthogonal transform candidate is applied by simulation, and selects the optimal orthogonal transform based on the results of the simulation. Note that the orthogonal transform selection unit 193 may be configured to select the same type of orthogonal transform in the horizontal and vertical directions, or may be configured to select different types of orthogonal transform in the horizontal and vertical directions.

The entropy coding unit 130 entropy codes the adaptive orthogonal transform index input from the orthogonal transform selection unit 193. When selecting different types of orthogonal transform in the horizontal and vertical directions, the entropy coding unit 130 entropy codes different adaptive orthogonal transform indices in the horizontal and vertical directions. However, when the adaptive orthogonal transform candidates are composed of a single type of orthogonal transform, the orthogonal transform can be uniquely identified in the image decoding device 2, and therefore such indices do not need to be entropy coded.

(Image decoding device)
Fig. 7 is a diagram showing the configuration of an image decoding device 2 according to the second embodiment. As shown in Fig. 7, the image decoding device 2 according to the second embodiment differs from the first embodiment in the configuration of the decision unit 260. As in the first embodiment, the decision unit 260 includes an adaptive transform generation unit 261 that generates an orthogonal transform by principal component analysis of an error map (see Fig. 4). In the second embodiment, the decision unit 260 further includes a candidate selection unit (first selection unit) 262 and an orthogonal transform selection unit (second selection unit) 263.

The adaptive transform generation unit 261 outputs the adaptive orthogonal transforms (vertical adaptive orthogonal transform and horizontal adaptive orthogonal transform) generated based on the error map to the candidate selection unit 262.

The candidate selection unit 262 selects one or more orthogonal transform candidates from among multiple predefined types of orthogonal transforms in descending order of correlation with the adaptive orthogonal transform input from the adaptive transform generation unit 261, and outputs the selected one or more orthogonal transform candidates to the orthogonal transform selection unit 263. As described above, DCT-2, DST-7, DCT-8, DST-1, and DCT-5 are predefined as multiple types of orthogonal transforms.

Specifically, the candidate selection unit 262 evaluates the correlation between the horizontal adaptive orthogonal transform input from the adaptive transform generation unit 261 and each orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5). Based on the results of the correlation evaluation, the candidate selection unit 262 then outputs one or more orthogonal transforms from among the multiple types of orthogonal transforms in descending order of correlation as adaptive orthogonal transform candidates to the orthogonal transform selection unit 263.

As described above, the number of adaptive orthogonal transform candidates output by the candidate selection unit 262 is predetermined to be the same for the image encoding device 1 and the image decoding device 2. In addition, the number of adaptive orthogonal transform candidates output by the candidate selection unit 262 may be variable depending on the block size and color components (luminance component, chrominance component) of the image block to be decoded, etc.

Furthermore, the candidate selection unit 262 may select adaptive orthogonal transform candidates separately for the horizontal and vertical directions. In such a case, the candidate selection unit 262 further evaluates the correlation between the vertical adaptive orthogonal transform input from the adaptive transform generation unit 261 and each orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5), and outputs adaptive vertical orthogonal transform candidates in addition to the adaptive horizontal orthogonal transform candidates to the orthogonal transform selection unit 263.

On the other hand, the entropy code decoding unit 200 decodes an index indicating the orthogonal transform selected by the image encoding device 1 from one or more orthogonal transform candidates, and outputs the index to the orthogonal transform selection unit 263. When different types of orthogonal transform are selected in the horizontal and vertical directions, the entropy code decoding unit 200 decodes different adaptive orthogonal transform indices for the horizontal and vertical directions.

The orthogonal transform selection unit 263 selects an inverse orthogonal transform to apply to the transform coefficients from one or more orthogonal transform candidates input from the candidate selection unit 262 based on the index input from the entropy coding/decoding unit 200, and outputs the selected inverse orthogonal transform to the inverse transform unit 212. Note that the orthogonal transform selection unit 263 may be configured to select the same type of orthogonal transform in the horizontal and vertical directions, or may be configured to select different types of orthogonal transform in the horizontal and vertical directions.

(Summary of the second embodiment)
In the image coding device 1 according to the second embodiment, the determination unit 190 includes an adaptive transform generation unit 191 that generates an orthogonal transform by principal component analysis of an error map, a candidate selection unit 192 that selects one or more orthogonal transform candidates from among a plurality of predefined types of orthogonal transforms in descending order of correlation with the generated orthogonal transform, and an orthogonal transform selection unit 193 that selects an orthogonal transform to apply to a prediction residual from among the one or more orthogonal transform candidates. The entropy coding unit 130 codes an index indicating the orthogonal transform selected by the orthogonal transform selection unit 193 from among the one or more orthogonal transform candidates.

In the image decoding device 2 according to the second embodiment, the determination unit 260 includes an adaptive transform generation unit 261 that generates an orthogonal transform by principal component analysis of an error map, a candidate selection unit 262 that selects one or more orthogonal transform candidates from among multiple predefined types of orthogonal transforms in descending order of correlation with the generated orthogonal transform, and an orthogonal transform selection unit 263 that selects an inverse orthogonal transform to apply to the transform coefficients from among the one or more orthogonal transform candidates. The entropy code decoding unit 200 decodes an index indicating the orthogonal transform selected by the image encoding device 1 from among the one or more orthogonal transform candidates. The orthogonal transform selection unit 263 selects an inverse orthogonal transform to apply to the transform coefficients from among the one or more orthogonal transform candidates based on the index.

In this way, according to the second embodiment, from among multiple predefined types of orthogonal transforms, orthogonal transform candidates highly correlated with the orthogonal transform (KLT) generated by principal component analysis of the error map are selected, and from these orthogonal transform candidates, the orthogonal transform to be applied to the transform processing is selected. The multiple predefined types of orthogonal transforms (DCT-2, DST-7, DCT-8, DST-1, DCT-5) require less computational processing than KLT.

Therefore, according to the second embodiment, it is possible to apply an orthogonal transform that efficiently concentrates the energy of the prediction residual, improving coding efficiency, while reducing the amount of computation required for the transform process compared to the first embodiment.

Third Embodiment
The image encoding device 1 and image decoding device 2 according to the third embodiment will be described, focusing mainly on the differences from the first and second embodiments.

In the second embodiment described above, an orthogonal transform candidate highly correlated with the orthogonal transform (KLT) generated by principal component analysis of the error map was selected from among multiple predefined types of orthogonal transform. In contrast, in the third embodiment, an orthogonal transform candidate is selected from among multiple predefined types of orthogonal transform by evaluating the feature quantities of the error map.

(Image encoding device)
8 is a diagram showing the configuration of an image encoding device 1 according to the third embodiment. As shown in Fig. 8, the image encoding device 1 according to the third embodiment differs from the second embodiment in that the determination unit 190 includes a feature amount evaluation unit 191a. The feature amount evaluation unit 191a evaluates the feature amounts of the error map input from the evaluation unit 180 and outputs the evaluation result to the candidate selection unit 192.

9 is a diagram showing the operation of the feature amount evaluation unit 191a. As shown in FIG. 9, in order to evaluate the energy distribution of the error map, the feature amount evaluation unit 191a divides the error map into four regions horizontally and four regions vertically, and calculates the sum of error estimates Exy (E ₀₀ to E ₃₃ ) for each divided region. The feature amount evaluation unit 191a outputs the evaluated energy distributions E ₀₀ to E ₃₃ to the candidate selection unit 192.

The candidate selection unit 192 selects an adaptive orthogonal transform candidate from among several predefined types of orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5) based on the following conditions, and outputs the selected adaptive orthogonal transform candidate to the orthogonal transform selection unit 193.

(a) The candidate selection unit 192 selects the horizontal direction as follows:

When , DCT-2 and DST-1 are selected as horizontal adaptive orthogonal transform candidates,

When , DCT-2 and DST-7 are selected as horizontal adaptive orthogonal transform candidates,

select DCT-2 and DCT-5 as horizontal adaptive orthogonal transform candidates when
If neither of these applies, DCT-8 and DST-7 are selected as horizontally adaptive orthogonal transform candidates.

(b) The candidate selection unit 192 determines the vertical direction as follows:

When , DCT-2 and DST-1 are selected as vertical adaptive orthogonal transform candidates,

When , DCT-2 and DST-7 are selected as vertical adaptive orthogonal transform candidates,

select DCT-2 and DCT-5 as vertical adaptive orthogonal transform candidates when
If neither of these applies, DCT-8 and DST-7 are selected as candidates for vertical adaptive orthogonal transform.

Here, the number of adaptive orthogonal transform candidates output by the candidate selection unit 192 is two in each of the horizontal and vertical directions, but the number of adaptive orthogonal transform candidates output by the candidate selection unit 192 may be variable depending on the block size and color components (luminance component, chrominance component) of the image block to be encoded, etc.

As in the second embodiment, the orthogonal transform selection unit 193 selects an orthogonal transform to apply to the prediction residual from the orthogonal transform candidates input from the candidate selection unit 192, and outputs the selected orthogonal transform to the transform unit 121 and the inverse transform unit 142. The orthogonal transform selection unit 193 also outputs an index indicating the type of selected orthogonal transform to the entropy coding unit 130. The entropy coding unit 130 entropy codes the adaptive orthogonal transform index input from the orthogonal transform selection unit 193.

(Image decoding device)
Fig. 10 is a diagram showing the configuration of an image decoding device 2 according to the third embodiment. As shown in Fig. 10, the image decoding device 2 according to the third embodiment differs from the second embodiment in that the determination unit 260 includes a feature amount evaluation unit 261a. The feature amount evaluation unit 261a performs the same operation as the feature amount evaluation unit 191a of the image encoding device 1 (see Fig. 9).

In order to evaluate the energy distribution of the error map, the feature amount evaluation unit 261a divides the error map into four regions horizontally and vertically, and calculates the sum of error estimates Exy (E ₀₀ to E ₃₃ ) for each divided region. The feature amount evaluation unit 261a outputs the evaluated energy distributions E ₀₀ to E ₃₃ to the candidate selection unit 262.

The candidate selection unit 262 selects an adaptive orthogonal transform candidate from among multiple predefined types of orthogonal transform (DCT-2, DST-7, DCT-8, DST-1, DCT-5) based on the same conditions as the candidate selection unit 192 of the image encoding device 1, and outputs the selected adaptive orthogonal transform candidate to the orthogonal transform selection unit 263.

Meanwhile, the entropy code decoding unit 200 decodes an index indicating the orthogonal transform selected by the image encoding device 1 from among the orthogonal transform candidates, and outputs the index to the orthogonal transform selection unit 263. The orthogonal transform selection unit 263 selects an inverse orthogonal transform to apply to the transform coefficients from among the orthogonal transform candidates input from the candidate selection unit 262, based on the index input from the entropy code decoding unit 200, and outputs the selected inverse orthogonal transform to the inverse transform unit 212.

(Summary of the third embodiment)
In the image coding device 1 according to the third embodiment, the determination unit 190 includes a feature evaluation unit 191 a that evaluates feature quantities of the error map, a candidate selection unit 192 that selects one or more orthogonal transform candidates from among a plurality of predefined types of orthogonal transform based on the evaluated feature quantities, and an orthogonal transform selection unit 193 that selects an orthogonal transform to be applied to the prediction residual from among the one or more orthogonal transform candidates. The entropy coding unit 130 codes an index indicating the orthogonal transform selected by the orthogonal transform selection unit 193 from among the one or more orthogonal transform candidates.

In the image decoding device 2 according to the third embodiment, the determination unit 260 includes a feature evaluation unit 261a that evaluates features of the error map, a candidate selection unit 262 that selects one or more orthogonal transform candidates from multiple predefined types of orthogonal transform based on the evaluated features, and an orthogonal transform selection unit 263 that selects an inverse orthogonal transform to apply to the transform coefficients from the one or more orthogonal transform candidates. The entropy code decoding unit 200 decodes an index indicating the orthogonal transform selected by the image encoding device 1 from the one or more orthogonal transform candidates. The orthogonal transform selection unit 263 selects an inverse orthogonal transform to apply to the transform coefficients from the one or more orthogonal transform candidates based on the index.

In this way, according to the third embodiment, the feature quantities of the error map can be evaluated through simple calculations, thereby reducing the amount of calculations required compared to the second embodiment, in which an orthogonal transform (KLT) is generated through principal component analysis of the error map.

Therefore, according to the third embodiment, it is possible to apply an orthogonal transform that efficiently concentrates the energy of the prediction residual, improving coding efficiency, while reducing the amount of computation required for analyzing the error map compared to the second embodiment.

<Other embodiments>
In the first to third embodiments described above, an example has been described in which a one-dimensional orthogonal transform is used to perform separate transform processing in the vertical direction and the vertical direction. However, instead of the one-dimensional orthogonal transform, a two-dimensional orthogonal transform may be used to perform the transform processing in the vertical direction and the vertical direction together.

The image encoding device 1 may also be provided by a program that causes a computer to execute each process performed by the image decoding device 2, and a program that causes a computer to execute each process performed by the image decoding device 2. The program may also be recorded on a computer-readable medium. Using a computer-readable medium makes it possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. There are no particular limitations on the non-transitory recording medium, and it may be, for example, a CD-ROM, DVD-ROM, or other recording medium.

Furthermore, the circuits that perform each process performed by the image encoding device 1 may be integrated, and the image encoding device 1 may be configured as a semiconductor integrated circuit (chip set, SoC). Similarly, the circuits that perform each process performed by the image decoding device 2 may be integrated, and the image decoding device 2 may be configured as a semiconductor integrated circuit (chip set, SoC).

The above describes the embodiments in detail with reference to the drawings, but the specific configuration is not limited to that described above, and various design changes can be made without departing from the spirit of the invention.

1: Image encoding device 2: Image decoding device 100: Block division unit 110: Subtraction unit 120: Transformation and quantization unit 121: Transformation unit 122: Quantization unit 130: Entropy encoding unit 140: Inverse quantization and inverse transformation unit 141: Inverse quantization unit 142: Inverse transformation unit 150: Synthesis unit 160: Memory 170: Prediction unit 171: Intra prediction unit 172: Inter prediction unit 173: Switching unit 180: Evaluation unit 180a: Difference calculation unit 180b: Normalization unit 180c: Adjustment unit 190: Determination unit 191: Adaptive transformation generation unit 191a: Feature evaluation unit 192: Candidate selection unit 193: Orthogonal transformation selection unit 200 : Entropy encoding/decoding unit 210 : Inverse quantization/inverse transform unit 211 : Inverse quantization unit 212 : Inverse transform unit 220 : Combining unit 230 : Memory 240 : Prediction unit 241 : Intra prediction unit 242 : Inter prediction unit 243 : Switching unit 250 : Evaluation unit 260 : Decision unit 261 : Adaptive transform generation unit 261a : Feature evaluation unit 262 : Candidate selection unit 263 : Orthogonal transform selection unit

Claims (4)

An image encoding device that encodes a target image in units of blocks obtained by dividing a current image in units of frames, comprising:
a block dividing unit that divides the current image into blocks;
a prediction unit that predicts the target image by inter prediction including bi-prediction using a plurality of reference images to generate a predicted image;
an evaluation unit that calculates a value indicating a similarity between the plurality of reference images in units of regions smaller than the block and consisting of a plurality of pixels only when the prediction unit performs the bi-prediction using a reference image that is temporally earlier and a reference image that is temporally later than the target image,
an image encoding device, characterized in that the evaluation unit controls encoding processing based on the value calculated for each region. An image decoding device that decodes a target image in units of blocks obtained by dividing a current image in units of frames, comprising:
a prediction unit that predicts the target image by inter prediction including bi-prediction using a plurality of reference images to generate a predicted image;
an evaluation unit that calculates a value indicating a similarity between the plurality of reference images in units of regions smaller than the block and consisting of a plurality of pixels only when the prediction unit performs the bi-prediction using a reference image that is temporally earlier and a reference image that is temporally later than the target image,
The image decoding device controls a decoding process based on the value calculated by the evaluation unit for each region. A program that causes a computer to function as the image encoding device described in claim 1. A program that causes a computer to function as the image decoding device described in claim 2.