JP7686426B2 - Encoding device and program - Google Patents

Description

The present invention relates to an encoding device and a program.

It is known that coding methods such as H.265/HEVC (High Efficiency Video Coding) and H.266/VVC (Versatile Video Coding) use pixel correlation to code moving images (see, for example, Non-Patent Document 1). In inter-frame prediction, prediction is performed by referring to reference frames before and after the frame to be coded (frame to be predicted). In general, when a unit to be predicted and a decoded unit of the reference frame are the same object, pixel correlation between the same objects is considered to be high, and therefore inter-frame prediction can be performed with high accuracy.

Edited by Eiichiro Okubo, "Impress Standard Textbook Series H.265/HEVC Textbook", Impress Japan Co., Ltd., October 21, 2013

However, with conventional coding techniques, when sharpness differs between frames, inter-frame prediction is difficult, and coding efficiency is likely to decrease.

In view of the above circumstances, the object of the present invention is to provide an encoding device and program capable of improving the encoding efficiency of video images in which the sharpness differs between frames.

An encoding device according to one embodiment is an encoding device that encodes an original image, and includes a frame memory that stores a frame obtained by locally decoding the original image, an image processing unit that performs image processing on a reference frame obtained from the frame memory to generate one or more filter-processed frames with changed frequency components, and an inter prediction unit that generates an inter prediction image for a frame to be encoded by referring to the one or more filter-processed frames.
The image processing unit includes a multi-resolution decomposition unit that performs multi-resolution decomposition on the reference frame to generate an n1 hierarchical decomposition image with a decomposition rank of n1 and an n2 hierarchical decomposition image with a decomposition rank of n2, which is greater than n1; a registration unit that determines, for each divided block of a lowest frequency component image of the n1 hierarchical decomposition image, a similar block with the highest similarity in the lowest frequency component image of the n2 hierarchical decomposition image and generates registration information indicating a position of the similar block; a high frequency component allocation unit that identifies a position of the similar block according to the registration information, identifies a block at the same spatial phase position as the similar block in each frequency component image of the n2 hierarchical decomposition image as an allocation block, and generates a high frequency component allocated image in which the allocation block is allocated to the same spatial phase position as the divided block in the high frequency components of the n1 hierarchical decomposition image; and a sharpening reconstruction unit that performs multi-resolution reconstruction on the high frequency component allocated image to generate the filtered frame in which the high frequency components of the reference frame are sharpened.
The high frequency component allocation unit classifies the n2 hierarchical decomposition image into high frequency component images and low frequency component images within each frequency component image of n3 hierarchical levels, where the decomposition rank is one less than n2, identifies a block within the low frequency component image that has the same spatial phase position as the similar block as a comparison block, and allocates the allocation block when the difference between the pixel value of the allocation block and the pixel value of the comparison block within each frequency component image of the n3 hierarchical levels exceeds a first allocation threshold.
Alternatively, the high frequency component allocation unit classifies the n2 hierarchical decomposition image into high frequency component images and low frequency component images within each frequency component image of n3 hierarchical layers, where the decomposition rank is one less than n2, identifies a block in the low frequency component image that has the same spatial phase position as the similar block as a comparison block, and when the difference between the pixel values of the similar block and the comparison block exceeds a second allocation threshold, allocates the allocation block in the same frequency component image as the comparison block in the n3 hierarchical layer.

Furthermore, in one embodiment, the registration unit may generate the registration information only when the difference between the pixel values of the divided block and the pixel values of the similar block exceeds a registration threshold.

Furthermore, in one embodiment, the device may further include a geometric transformation unit that performs a geometric transformation on the reference frame and generates one or more reference frames after the geometric transformation, and the image processing unit may perform image processing on the reference frames after the geometric transformation.

Furthermore, in one embodiment, the reference frames may further include one or more frames located temporally before the frame to be encoded, or one or more frames located temporally after the frame to be encoded.

In one embodiment, the program causes a computer to function as the encoding device.

The present invention makes it possible to improve the efficiency of image coding.

1 is a block diagram showing an example of the configuration of an encoding device according to an embodiment. FIG. 2 is a block diagram showing an example of the configuration of an image processing unit in an encoding device according to an embodiment. FIG. 1 is a block diagram showing an example of the configuration of a sharpened image generating unit in an encoding device according to an embodiment. FIG. 10A to 10C are diagrams illustrating an example of a registration process in an encoding device according to an embodiment. 11A and 11B are diagrams illustrating an example of an allocation process in an encoding device according to an embodiment. 11 is a diagram showing a first example of an allocation accuracy determination in an encoding device according to an embodiment. FIG. FIG. 11 is a diagram showing a second example of allocation accuracy determination in an encoding device according to an embodiment. 1 is a block diagram showing an example of the configuration of a blurred image generating unit in an encoding device according to an embodiment. FIG. 10A to 10C are diagrams illustrating an example of high-frequency component suppression processing in an encoding device according to an embodiment. FIG. 13 is a block diagram showing a modified example of an image processing unit in the encoding device according to an embodiment.

One embodiment is described in detail below with reference to the drawings.

Fig. 1 is a block diagram showing an example of the configuration of an encoding device according to an embodiment. The encoding device 1 shown in Fig. 1 includes an image processing unit 10, a block division unit 11, a subtraction unit 12, a transformation unit 13, a visual activity calculation unit 14, a QP determination unit 15, a quantization unit 16, a dequantization unit 17, an inverse transformation unit 18, an addition unit 19, an intra-screen prediction unit 20, a frame memory 21, an inter-screen prediction unit 22, a switching unit 23, and an entropy encoding unit 24.

The block division unit 11 divides the input image (original image) into divided blocks, which are then generated as coding units (CUs), and outputs the CUs to the subtraction unit 12, the visual activity calculation unit 14, and the inter-screen prediction unit 22.

The subtraction unit 12 subtracts each pixel value of the prediction unit input from the intra-screen prediction unit 20 or the inter-screen prediction unit 22 described below from each pixel value of the coding unit input from the block division unit 11 to generate a residual signal indicating the difference between the coding unit, the prediction unit, and the pixel value, and outputs the residual signal to the conversion unit 13.

The transform unit 13 further divides the residual signal input from the subtraction unit 12 into transform units (TUs), performs a transform process such as an orthogonal transform for each TU, calculates transform coefficients, and outputs them to the quantization unit 16.

The visual activity calculation unit 14 calculates the visual activity of the coding unit input from the block division unit 11 and outputs it to the QP determination unit 15. If the coding unit contains many high frequency components, the visual activity is large, and if the coding unit does not contain many high frequency components, the visual activity is small.

The QP determination unit 15 determines a quantization parameter (QP value) based on the visual activity input from the visual activity calculation unit 14 and outputs it to the quantization unit 16.

The quantization unit 16 generates quantization coefficients by dividing the transformation coefficients input from the transformation unit 13 by the quantization step corresponding to the QP value input from the QP determination unit 15 and quantizing the coefficients, and outputs the quantization coefficients to the inverse quantization unit 17 and the entropy coding unit 24.

The inverse quantization unit 17 restores the transform coefficients by multiplying the quantization coefficients input from the quantization unit 16 by the quantization step, and outputs the restored transform coefficients to the inverse transform unit 18.

The inverse transform unit 18 performs an inverse transform process (a process of undoing the transform performed by the transform unit 13) on the transform coefficients input from the inverse quantization unit 17 to restore the residual signal, and outputs it to the addition unit 19. For example, if the transform unit 13 performs a discrete cosine transform, the inverse transform unit 18 performs an inverse discrete cosine transform.

The adder 19 adds the residual signal input from the inverse transformer 18 to the prediction unit input from the switching unit 23 to generate a decoded unit, and outputs it to the frame memory 21 and the intra-screen prediction unit 20.

The inverse quantization unit 17, the inverse transform unit 18, and the addition unit 19 constitute a local decoding unit, which performs local decoding processing. That is, the local decoding unit multiplies the quantization coefficients by the quantization step to restore the transform coefficients, performs inverse transform processing on the transform coefficients to restore the residual signal, and adds the residual signal to the prediction unit to generate a decoded unit.

The intra-screen prediction unit 20 references the decoded units input from the adder unit 19, performs intra-screen prediction (intra-prediction) for each prediction unit (PU: Prediction Unit), generates an intra-screen predicted image, and outputs it to the switching unit 23.

The frame memory 21 stores the decoded units input from the adder 19 on a frame-by-frame basis. In other words, the frame memory 21 stores frames that have been locally decoded after quantizing the original image.

The image processing unit 10 acquires a reference frame from the frame memory 21, performs image processing on the reference frame, generates one or more filter processing frames in which the frequency components are changed, and outputs the generated frames to the inter prediction unit 22. The image processing unit 10 may also perform a process of blurring or sharpening the high frequency components of the reference frame to generate a filter processing frame. Details of the image processing unit 10 will be described later.

The inter prediction unit 22 divides the coding unit input from the block division unit 11 into prediction units. The inter prediction unit 22 then references one or more filter processing frames input from the image processing unit 10 in addition to the reference frame to generate a motion vector for each prediction unit by a technique such as block matching. The inter prediction unit 22 may adopt a motion vector with the highest coding efficiency by RD optimization or the like. The inter prediction unit 22 performs motion compensation prediction based on the motion vector for each prediction unit to generate an inter prediction image for the frame to be coded, and outputs it to the switching unit 23. The coding device 1 is different in that, while conventional coding devices perform inter prediction by referring to the reference frame, the coding device 1 performs inter prediction by referring to the filter processing frame in addition to the reference frame.

The switching unit 23 switches between the intra-screen predicted image input from the intra-screen prediction unit 20 and the inter-screen predicted image input from the inter-screen prediction unit 22 to obtain a predicted image, and outputs the predicted image to the subtraction unit 12 and the addition unit 19.

The entropy coding unit 24 performs entropy coding on the quantization coefficients input from the quantization unit 16, the prediction mode information input from the intra-screen prediction unit 20, the motion vector information input from the inter-screen prediction unit 22, and the like, compresses the data, generates a bit stream, and outputs it to the outside of the coding device 1. For the entropy coding, any entropy coding method can be used, such as zeroth-order exponential Golomb coding or context-based adaptive binary arithmetic coding (CABAC).

<Image Processing Unit>
Next, a description will be given of the processing of the image processing unit 10. Fig. 2 is a block diagram showing an example of the configuration of the image processing unit 10. The image processing unit 10 shown in Fig. 2 includes a multi-resolution decomposition unit 30, a sharpened image generation unit 40, and a blurred image generation unit 50.

The image processing unit 10 inputs (acquires) a reference frame locally decoded by the local decoding unit (the inverse quantization unit 17, the inverse transformation unit 18, and the addition unit 19) from the frame memory 21. The reference frame is a frame that is referenced when performing inter-prediction processing in the inter-prediction unit 22. The reference frame is specified in encoding methods such as H.265/HEVC and H.266/VVC. If the frame to be encoded is a P-picture (Predictive Picture), there is one reference frame, and if the frame to be encoded is a B-picture (Bidirectionally Predictive Picture), there are two reference frames. The image processing unit 10 may further input (acquire) a previous frame and/or a subsequent frame of the frame to be encoded from the frame memory 21. The previous frame means one or more frames located chronologically before the frame to be encoded, and the subsequent frame means one or more frames located chronologically after the frame to be encoded. In the embodiment described below, the image processing unit 10 is described as inputting only the reference frame.

The multi-resolution decomposition unit 30 performs multi-resolution decomposition on the reference frame input from the frame memory 21 to generate an n1-layer decomposition image with a decomposition level of n1, and an n2-layer decomposition image with a decomposition level of n2, which is greater than n1. The multi-resolution decomposition unit 30 then outputs the n1-layer decomposition image and the n2-layer decomposition image to the sharpened image generation unit 40, and outputs the n1-layer decomposition image to the blurred image generation unit 50.

The multi-resolution decomposition unit 30 may perform multi-resolution decomposition by wavelet packet decomposition. By performing wavelet packet decomposition, the phase information of each frequency component image can be treated as being of the same size. In this embodiment, the multi-resolution decomposition unit 30 performs wavelet packet decomposition with decomposition levels n1=1 and n2=2. The multi-resolution decomposition unit 30 may obtain information on the decomposition levels n1 and n2 from the outside.

The sharpened image generation unit 40 inputs the n1-layer decomposition image and the n2-layer decomposition image from the multi-resolution decomposition unit 30. The sharpened image generation unit 40 then assigns the high-frequency band of the n2-layer decomposition image to the high-frequency band of the n1-layer decomposition image, and then performs multi-resolution reconstruction to generate a filter processing frame (sharpened reference frame) in which the reference frame has been sharpened. The sharpened image generation unit 40 outputs the generated sharpened reference frame to the inter-screen prediction unit 22.

The blurred image generation unit 50 inputs the n1-layer decomposition image from the multi-resolution decomposition unit 30. The blurred image generation unit 50 then suppresses the high-frequency band of the n1-layer decomposition image, and performs multi-resolution reconstruction to generate a filtered frame (blurred reference frame) in which the reference frame is blurred. The blurred image generation unit 50 outputs the generated blurred reference frame to the inter-screen prediction unit 22.

<<Sharpened image generation unit>>
Next, a detailed description will be given of the sharpened image generating unit 40. Fig. 3 is a block diagram showing an example of the configuration of the sharpened image generating unit 40. The sharpened image generating unit 40 shown in Fig. 3 includes a registration unit 41, a high frequency component allocation unit 42, and a sharpening reconstruction unit 43.

The registration unit 41 divides the lowest frequency component image of the n1-hierarchical decomposition image into division blocks of a predetermined size (e.g., 8 x 8 pixels). Then, for each division block in the lowest frequency component image of the n1-hierarchical decomposition image, a similar block that has the highest similarity (correlation) to the division block in the lowest frequency component image of the n2-hierarchical decomposition image is determined by block matching. The registration unit 41 then generates registration information indicating the position of the similar block and outputs it to the high frequency component allocation unit 42.

A specific example of the registration process by the registration unit 41 will be described with reference to FIG. 4. In this example, the multi-resolution decomposition unit 30 performs wavelet packet decomposition at decomposition level 1 on the first reference frame to generate a one-layer decomposition image F1, and performs wavelet packet decomposition at decomposition level 2 on the second reference frame to generate a two-layer decomposition image F2. The first reference frame and the second reference frame may be the same frame. By performing multi-resolution decomposition on the second reference frame at decomposition level n2=2, four frequency component images (LL, HL, LH, HH) are generated as one-layer divided images. In addition, 16 frequency component images (LLLL, LLLH, LLHL, LLHH, HLLL, HLLH, HLHL, HLHH, LHLL, LHLH, LHHL, LHHH, HHLL, HHLH, HHHL, HHHH) are generated as the two-layer decomposition image F2.

The registration unit 41 registers (aligns) each divided block B of the lowest frequency component image LL of the one-level decomposition image F1 with the lowest frequency component image LLLL of the two-level decomposition image F2, and determines the similar block S in the lowest frequency component image LLLL that has the highest similarity (correlation) with the divided block B.

The registration unit 41 may determine the accuracy of the registration and generate registration information only if the accuracy is high.

The following describes the process performed by the registration unit 41 when determining the accuracy. For example, the registration unit 41 determines that the accuracy of the registration is high and generates registration information only when the difference between the pixel values of the divided block B and the pixel values of the similar block S exceeds a registration threshold. In other words, when the difference is equal to or less than the registration threshold, the registration unit 41 determines that the accuracy of the registration is low and does not generate registration information. The registration unit 41 may calculate the sum of absolute difference (SAD) or sum of squared difference (SSD) as the difference in pixel values.

The high frequency component allocation unit 42 identifies the position of the similar block S according to the registration information input from the registration unit 41. The high frequency component allocation unit 42 then identifies, within each frequency component image of the n2 hierarchical decomposition image F2, a block at the same spatial phase position as the similar block S as an allocation block A. The high frequency component allocation unit 42 then generates a high frequency component allocation image in which the allocation block A is allocated to the same spatial phase position as the divided block B within the high frequency components of the n1 hierarchical decomposition image F1. The high frequency component allocation unit 42 then outputs the high frequency component allocation image to the sharpening reconstruction unit 43.

The allocation process by the high frequency component allocation unit 42 will be described in detail with reference to FIG. 5. Here, as in FIG. 4, n1=1 and n2=2. The high frequency component allocation unit 42 identifies the position of the similar block S in the lowest frequency component image LLLL of the two-layer decomposition image F2 according to the registration information. In FIG. 5, this alignment is indicated by the dashed arrow.

The high frequency component allocation unit 42 identifies, as allocation block A, a block at the same spatial phase position as the similar block S within the high frequency component images (LLHL, LLLH, LLHH, HLHL, HLLH, HLHH, LHHL, LHLH, LHHH, HHHL, HHLH, HHHH) in each frequency component image of the decomposition rank n2-1=1 in the two-layer decomposition image F2. For ease of explanation, FIG. 5 shows allocation block A only for the high frequency component images LLHL, HLHL, LHHL, and HHHL. Then, the high frequency component allocation unit 42 allocates allocation block A in the horizontal low frequency/vertical high frequency component images (LLHL, HLHL, LHHL, HHHL) in each frequency component image of the decomposition rank 1 in the two-layer decomposition image F2 to the same spatial phase position as the divided block B in the horizontal low frequency/vertical high frequency component image (HL) of the one-layer decomposition image F1. In FIG. 5, this allocation is indicated by four solid arrows.

Similarly, the high frequency component allocation unit 42 allocates allocation block A in the horizontal high frequency and vertical low frequency component images (LLLH, HLLH, LHLH, HHLH) in each frequency component image of decomposition rank 1 in the two-layer decomposition image F2 to the same spatial phase position as divided block B in the horizontal high frequency and vertical low frequency component image (LH) of the one-layer decomposition image F1. Similarly, the high frequency component allocation unit 42 allocates allocation block A in the horizontal high frequency and vertical high frequency component images (LLHH, HLHH, LHHH, HHHH) in each frequency component image of decomposition rank 1 in the two-layer decomposition image F2 to the same spatial phase position as divided block B in the horizontal high frequency and vertical high frequency component image (HH) of the one-layer decomposition image F1.

The high frequency component allocation unit 42 may determine the accuracy of the allocation and perform the allocation only if the accuracy is high.

The high frequency component allocation unit 42 performs the process of determining the accuracy. The high frequency component allocation unit 42 classifies the n2-layer decomposition image into a high frequency component image and a low frequency component image in a layer where the decomposition level is one less than n2, identifies a block in the low frequency component image that has the same spatial phase position as the similar block as a comparison block, and determines that the accuracy of the allocation is high and allocates the allocation block only when the difference between the pixel value of the allocation block in the high frequency component image and the comparison block in the low frequency component image exceeds the first allocation threshold. In other words, when the difference exceeds the first allocation threshold, the accuracy of the allocation is determined to be low and the allocation block is not allocated. This comparison may be performed for each decomposition image in a layer where the decomposition level is one less than n2. The high frequency component allocation unit 42 may calculate the SAD or SSD as the difference in pixel values.

With reference to FIG. 6, a first example of the accuracy judgment of the allocation will be described. Here, as in FIG. 4 and FIG. 5, n1=1 and n2=2. The high frequency component allocation unit 42 classifies the two-layer decomposition image F2 into high frequency component images (in FIG. 6, the background is displayed dark) and low frequency component images (in FIG. 6, the background is displayed light) in each frequency component image of n3 layers, where the decomposition rank is 1 less than n2, and specifies a block in each low frequency component image at the same spatial phase position as the similar block S as a comparison block C. Then, the high frequency component allocation unit 42 calculates the difference between the pixel value of the allocation block A and the pixel value of the comparison block C in each frequency component image (HL, LH, HH) of the n3 layers, and allocates the allocation block A when the difference exceeds the first allocation threshold T1. Note that since there is no comparison block C in the frequency component image LL, this accuracy judgment is not performed.

With reference to FIG. 7, a second example of the accuracy judgment of the allocation will be described. Here, as in FIG. 4 to FIG. 6, n1=1 and n2=2. The high frequency component allocation unit 42 identifies the comparison block C in the same manner as in the first example. Then, the high frequency component allocation unit 42 obtains the difference between the pixel value of the similar block S and the pixel value of the comparison block C, and if the difference exceeds the second allocation threshold T2, allocates the allocation block A in the same frequency component image as the comparison block C in the n3 layer. For example, in the first layer decomposition image HL, if the difference exceeds the second allocation threshold T2, three allocation blocks A in the first layer decomposition image HL are allocated. Note that since there is no comparison block C in the frequency component image LL, this accuracy judgment is not performed. In addition, the high frequency component allocation unit 42 may combine the first and second examples of the accuracy judgment and allocate the allocation block A when both conditions are satisfied.

The sharpening reconstruction unit 43 performs multi-resolution reconstruction on the high frequency component allocated image input from the high frequency component allocation unit 42 to generate a filter processing frame (sharpened reference frame) in which the high frequency components of the reference frame are sharpened. For example, when the n1 hierarchical decomposition image is an image generated by wavelet packet decomposition of decomposition rank n1 on the reference frame, the sharpening reconstruction unit 43 performs wavelet packet reconstruction of decomposition rank n1 on the high frequency component allocated image.

<<Blurred image generation unit>>
Next, a description will be given of the details of the blurred image generating unit 50. Fig. 8 is a block diagram showing an example of the configuration of the blurred image generating unit 50. The blurred image generating unit 50 shown in Fig. 8 includes a high frequency component suppression unit 51 and a blur reconstruction unit 52.

The high-frequency component suppression unit 51 generates a high-frequency component suppressed image by suppressing the high-frequency components of the n1-layer decomposition image input from the multi-resolution decomposition unit 30, and outputs the image to the blur reconstruction unit 52.

Figure 9 is a diagram showing an example of high frequency component suppression processing by the high frequency component suppression unit 51. The high frequency component suppression unit 51 multiplies the pixel values of the high frequency component image (HL, LH, HH) of the n1 hierarchical decomposition image by the blur coefficient γ to generate a high frequency component suppressed image. The blur coefficient γ is a value smaller than 1, for example, γ = 0.9. The high frequency component suppression unit 51 may obtain information on the blur coefficient γ from an external source.

The blur reconstruction unit 52 performs multi-resolution reconstruction on the high-frequency component suppressed image input from the high-frequency component suppression unit 51 to generate a filter processing frame (blurred reference frame) in which the high-frequency components of the reference frame are blurred. For example, if the n1-hierarchical decomposition image is an image generated by wavelet packet decomposition of decomposition rank n1 on the reference frame, the blur reconstruction unit 52 performs wavelet packet reconstruction of decomposition rank n1 on the high-frequency component suppressed image.

<<Modifications of the sharpened image generating unit>>
Next, a modified example of the sharpened image generating unit 40 will be described. The sharpened image generating unit 40 may increase high-frequency components by multiplying coefficients in the same manner as the blurred image generating unit 50. That is, in the modified example, the sharpened image generating unit 40 multiplies the pixel values of the high-frequency component images (HL, LH, HH) of the n1-layer decomposition image by the sharpening coefficient γ' to generate an image with increased high-frequency components. The sharpening coefficient γ' is a value greater than 1, and for example, γ'=1.1. The sharpened image generating unit 40 may obtain information on the sharpening coefficient γ' from an external source. Then, the sharpened image generating unit 40 performs multi-resolution reconstruction on the image with increased high-frequency components to generate a filter processing frame (sharpened reference frame) in which the high-frequency components of the reference frame are sharpened.

<Modifications of Image Processing Section>
Next, a modified example of the image processing unit 10 will be described. Fig. 10 is a block diagram showing a configuration example of an image processing unit 10a which is a modified example of the image processing unit 10. The image processing unit 10a includes a geometric transformation unit 60, a multi-resolution decomposition unit 30, a sharpened image generation unit 40, and a blurred image generation unit 50. The image processing unit 10a differs from the image processing unit 10 in that it further includes a geometric transformation unit 60.

The geometric transformation unit 60 inputs a reference frame from the frame memory 21, performs a geometric transformation on the reference frame, and generates one or more reference frames after the geometric transformation. The geometric transformation unit 60 then outputs one or more reference frames after the geometric transformation (a group of geometrically transformed images) to the multi-resolution decomposition unit 30. For example, the geometric transformation unit 60 performs an affine transformation as the geometric transformation. The geometric transformation parameters are enlargement/reduction a = {0.9, 1.0, 1.1}, rotation θ = {1/16π, 0, -1/16π}, and no skew. When this affine transformation is performed, a group of nine geometrically transformed images is generated for each reference frame.

The multi-resolution decomposition unit 30 performs multi-resolution decomposition on the reference frame after the geometric transformation. The other processes are the same as those of the image processing unit 10, so the explanation is omitted.

<Program>
A computer capable of executing program instructions can be used to function as the above-mentioned encoding device 1. Here, the computer may be a general-purpose computer, a special-purpose computer, a workstation, a personal computer (PC), an electronic notepad, etc. The program instructions may be program code, code segments, etc. for performing the necessary tasks.

The computer includes a processor, a storage unit, an input unit, an output unit, and a communication interface. The processor is specifically a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), a SoC (System on a Chip), etc., and may be composed of multiple processors of the same or different types. The processor reads and executes a program from the storage unit to control each of the above components and perform various calculations.

The program may be recorded on a computer-readable recording medium. Using such a recording medium, the program can be installed on a computer. Here, the recording medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, and may be, for example, a CD-ROM, a DVD-ROM, or a USB (Universal Serial Bus) memory. In addition, the program may be in a form that is downloaded from an external device via a network.

As described above, in the present invention, image processing is performed on a reference frame obtained from the frame memory 21 to generate one or more filter processed frames with changed frequency components, and an inter-prediction image for the frame to be encoded is generated by referring to the filter processed frame. As a result, the number of frames referenced in inter-prediction increases compared to conventional encoding methods, which makes it possible to improve encoding accuracy and encoding efficiency.

It is desirable that the filter processing frame be a sharpened reference frame in which the high frequency components of the reference frame are blurred, and/or a blurred reference frame in which the high frequency components of the reference frame are blurred. Even when encoding an image with low correlation between frames, the use of a sharpened reference frame or a blurred reference frame is expected to improve the correlation with the frame to be encoded.

In addition, the number of frames to be filtered can be increased by performing a geometric transformation on the reference frame before filtering, or by performing filtering on the previous or next frame of the frame to be coded. In this case, the number of frames to be referenced in inter prediction is further increased, making it possible to further improve coding efficiency.

The above-mentioned embodiment has been described as a representative example, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-mentioned embodiment, and various modifications or alterations are possible without departing from the scope of the claims. For example, it is possible to integrate multiple configuration blocks shown in the configuration diagram of the embodiment, or to divide one configuration block.

REFERENCE SIGNS LIST 1 Encoding device 10, 10a Image processing unit 11 Block division unit 12 Subtraction unit 13 Transformation unit 14 Visual activity calculation unit 15 QP determination unit 16 Quantization unit 17 Inverse quantization unit 18 Inverse transformation unit 19 Addition unit 20 Intra-frame prediction unit 21 Frame memory 22 Inter-frame prediction unit 23 Switching unit 24 Entropy encoding unit 30 Multi-resolution decomposition unit 40 Sharpened image generation unit 41 Registration unit 42 High-frequency component allocation unit 43 Sharpening reconstruction unit 50 Blurred image generation unit 51 High-frequency component suppression unit 52 Blur reconstruction unit 60 Geometric transformation unit

Claims (6)

An encoding device for encoding an original image, comprising:
a frame memory for storing a frame obtained by locally decoding the original image;
an image processing unit that performs image processing on the reference frame acquired from the frame memory to generate one or more filter processed frames in which frequency components are changed;
an inter prediction unit that generates an inter prediction image for a frame to be coded by referring to the one or more filter processed frames;
Equipped with
The image processing unit includes:
a multi-resolution decomposition unit that performs multi-resolution decomposition on the reference frame to generate an n1-layer decomposition image having a decomposition rank of n1 and an n2-layer decomposition image having a decomposition rank of n2 greater than n1;
a registration unit that determines a similar block having the highest similarity in the lowest frequency component image of the n2 hierarchical decomposition image for each divided block of the lowest frequency component image of the n1 hierarchical decomposition image, and generates registration information indicating a position of the similar block;
a high frequency component allocation unit that identifies a position of the similar block according to the registration information, identifies a block having the same spatial phase position as the similar block in each frequency component image of the n2 hierarchical decomposition image as an allocation block, and generates a high frequency component allocation image in which the allocation block is allocated to the same spatial phase position as the divided block in the high frequency component of the n1 hierarchical decomposition image;
a sharpening reconstruction unit that performs multi-resolution reconstruction on the high-frequency component allocated image to generate the filtered frame in which the high-frequency components of the reference frame are sharpened;
Equipped with
the high frequency component allocation unit classifies the n2 hierarchical decomposition image into high frequency component images and low frequency component images within each frequency component image of n3 hierarchical levels, the decomposition rank of which is one less than n2, identifies a block within the low frequency component image that has the same spatial phase position as the similar block as a comparison block, and allocates the allocation block when a difference between pixel values of the allocation block and pixel values of the comparison block within each frequency component image of the n3 hierarchical levels exceeds a first allocation threshold . An encoding device for encoding an original image, comprising:
a frame memory for storing a frame obtained by locally decoding the original image;
an image processing unit that performs image processing on the reference frame acquired from the frame memory to generate one or more filter processed frames in which frequency components are changed;
an inter prediction unit that generates an inter prediction image for a frame to be coded by referring to the one or more filter processed frames;
Equipped with
The image processing unit includes:
a multi-resolution decomposition unit that performs multi-resolution decomposition on the reference frame to generate an n1-layer decomposition image having a decomposition rank of n1 and an n2-layer decomposition image having a decomposition rank of n2 greater than n1;
a registration unit that determines a similar block having the highest similarity in the lowest frequency component image of the n2 hierarchical decomposition image for each divided block of the lowest frequency component image of the n1 hierarchical decomposition image, and generates registration information indicating a position of the similar block;
a high frequency component allocation unit that identifies a position of the similar block according to the registration information, identifies a block having the same spatial phase position as the similar block in each frequency component image of the n2 hierarchical decomposition image as an allocation block, and generates a high frequency component allocation image in which the allocation block is allocated to the same spatial phase position as the divided block in the high frequency component of the n1 hierarchical decomposition image;
a sharpening reconstruction unit that performs multi-resolution reconstruction on the high-frequency component allocated image to generate the filtered frame in which the high-frequency components of the reference frame are sharpened;
Equipped with
the high frequency component allocation unit classifies the n2 hierarchical decomposition image into high frequency component images and low frequency component images within each frequency component image of n3 hierarchical layers, the decomposition rank of which is one less than n2, identifies a block in the low frequency component image that has the same spatial phase position as the similar block as a comparison block, and when a difference between pixel values of the similar block and the comparison block exceeds a second allocation threshold, allocates the allocation block in the same frequency component image as the comparison block in the n3 hierarchical layers. The encoding device according to claim 1 , wherein the registration unit generates the registration information when a difference between pixel values of the divided block and pixel values of the similar block exceeds a registration threshold. A geometric transformation unit that performs a geometric transformation on the reference frame to generate one or more reference frames after the geometric transformation,
The image processing unit performs image processing on the reference frame after the geometric transformation.
An encoding device according to any one of claims 1 to 3 . The encoding device according to claim 1 , wherein the reference frames further include one or more frames located temporally before the encoding target frame, or one or more frames located temporally after the encoding target frame. A program for causing a computer to function as the encoding device according to any one of claims 1 to 5 .

Images