JP7324899B2 - Image decoding device, image decoding method and program - Google Patents

Description

The present invention relates to an image decoding device, an image decoding method and a program.

Conventionally, a technique of applying refinement processing to a block that satisfies an applicable condition and is composed only of information that can be obtained on the decoding device side, that is, a technique called DMVR (Decoder-side motion vector refinement: decoding side motion vector refinement) is known (see, for example, Non-Patent Document 1).

Versatile Video Coding (Draft 5), JVET-N1001

However, for example, in the technology disclosed in Non-Patent Document 1, refinement processing is always executed for blocks that satisfy the above-mentioned applicable conditions. On the contrary, there is a problem that the encoding efficiency may decrease.

Therefore, the present invention has been made in view of the above problems, and is an image decoding method that can prevent a decrease in coding efficiency by not performing refinement processing on blocks with low correlation in the temporal direction. An object is to provide an apparatus, an image decoding method, and a program.

A first feature of the present invention is an image decoding device comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; A motion vector is searched for using the value as an initial value, and if the search cost at the initial search point is greater than a predetermined threshold or if the search cost at the initial search point is equal to or greater than the threshold, the decoded and a refiner configured to determine the motion vector as a final motion vector.

A second feature of the present invention is an image decoding device comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; A motion vector is searched for using the value as an initial value, and if the minimum search cost among search points is greater than a predetermined threshold or if the minimum search cost is equal to or greater than the threshold, the decoded motion vector as the final motion vector.

A third feature of the present invention is an image decoding device comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; A motion vector is searched for using the value as an initial value, and the difference between the search cost at the initial search point and the minimum search cost among the search points is smaller than a predetermined threshold, or the difference is equal to or less than the threshold. and a refiner configured to determine the decoded motion vector as the final motion vector if .

A fourth feature of the present invention is an image decoding device comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; A refining unit configured to search for a motion vector using a value as an initial value; and a prediction signal generation unit configured to generate a prediction signal based on the motion vector output from the refining unit. and the prediction signal generating unit, when an index value representing the degree of similarity between the block on the first reference frame side and the block on the second reference frame side is larger than a predetermined threshold value, or when the index value is the threshold value The gist is that it is configured to determine not to execute BDOF (Bi-Directional Optical Flow) processing in the above case.

A fifth feature of the present invention is an image decoding device comprising: a motion vector decoding unit configured to decode a motion vector from encoded data; a refinement unit configured to search for a motion vector using the value as an initial value, wherein the refinement unit includes the norm of the difference vector between the initial search point and the search point in the search cost. The gist is that it is configured.

A sixth feature of the present invention is a step of decoding a motion vector from encoded data, a step of searching for a motion vector using the value of the decoded motion vector as an initial value, and a search cost at an initial search point being determining the decoded motion vector as a final motion vector if it is greater than a predetermined threshold or if the search cost at the initial search point is equal to or greater than the threshold. .

A seventh feature of the present invention is a program for use in an image decoding device, comprising: a step of decoding a motion vector from encoded data; and converting the decoded motion vector to the final motion The gist is to execute a step of determining as a vector.

According to the present invention, it is possible to provide an image decoding device, an image decoding method, and a program that can prevent deterioration of coding efficiency by not performing refinement processing on blocks with low correlation in the temporal direction. can.

It is a figure showing an example of composition of image processing system 10 concerning one embodiment. 1 is a diagram showing an example of functional blocks of an image encoding device 100 according to an embodiment; FIG. 3 is a diagram showing an example of functional blocks of an inter prediction unit 111 of the image encoding device 100 according to one embodiment; FIG. 3 is a flowchart showing an example of a processing procedure of a refining unit 111C of an inter prediction unit 111 of the image encoding device 100 according to one embodiment; 4 is a flow chart showing an example of a processing procedure of a prediction signal generation unit 111D of an inter prediction unit 111 of the video decoding device 30 according to one embodiment. 2 is a diagram showing an example of functional blocks of an image decoding device 200 according to one embodiment; FIG. 3 is a diagram showing an example of functional blocks of an inter prediction unit 241 of the image decoding device 200 according to one embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings. It should be noted that constituent elements in the following embodiments can be appropriately replaced with existing constituent elements and the like, and various variations including combinations with other existing constituent elements are possible. Therefore, the following description of the embodiments is not intended to limit the scope of the invention described in the claims.

(First embodiment)
An image processing system 10 according to a first embodiment of the present invention will be described below with reference to FIGS. 1 to 7. FIG. FIG. 1 is a diagram showing an image processing system 10 according to an embodiment according to this embodiment.

As shown in FIG. 1 , the image processing system 10 has an image encoding device 100 and an image decoding device 200 .

The image encoding device 100 is configured to generate encoded data by encoding an input image signal. The image decoding device 200 is configured to generate an output image signal by decoding encoded data.

Here, such encoded data may be transmitted from the image encoding device 100 to the image decoding device 200 via a transmission line. Also, the encoded data may be stored in a storage medium and then provided from the image encoding device 100 to the image decoding device 200 .

(Image encoding device 100)
The image coding apparatus 100 according to this embodiment will be described below with reference to FIG. FIG. 2 is a diagram showing an example of functional blocks of the image encoding device 100 according to this embodiment.

As shown in FIG. 2, the image coding apparatus 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform/quantization unit 131, an inverse transform/inverse quantization It has a section 132 , an encoding section 140 , an in-loop filtering section 150 and a frame buffer 160 .

The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 111 identifies a reference block included in the reference frame by comparing a frame to be encoded (hereinafter referred to as a target frame) and a reference frame stored in the frame buffer 160, and identifies a reference block included in the reference frame. It is configured to determine a motion vector for the reference block.

Also, the inter prediction unit 111 is configured to generate, for each prediction block, a prediction signal included in the prediction block based on the reference block and motion vector. The inter prediction section 111 is configured to output the prediction signal to the subtractor 121 and the adder 122 . Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 112 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. Also, the intra prediction unit 112 is configured to output the prediction signal to the subtractor 121 and the adder 122 .

Here, the reference block is a block that is referred to for a prediction target block (hereinafter referred to as target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform/quantization section 131 . Here, the subtractor 121 is configured to generate a prediction residual signal that is a difference between a prediction signal generated by intra prediction or inter prediction and an input image signal.

The adder 122 adds the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 132 to generate a pre-filtering decoded signal. It is configured to output to the loop filter processing unit 150 .

Here, the unfiltered decoded signal constitutes a reference block used in intra prediction section 112 .

The transform/quantization unit 131 is configured to perform transform processing on the prediction residual signal and to obtain coefficient level values. Further, the transform/quantization unit 131 may be configured to quantize the coefficient level values.

Here, transform processing is processing for transforming a prediction residual signal into a frequency component signal. In such transformation processing, a basis pattern (transformation matrix) corresponding to discrete cosine transform (DCT) may be used, and a basis pattern (transformation matrix) corresponding to discrete sine transform (DST) may be used. may be used.

The inverse transform/inverse quantization unit 132 is configured to perform inverse transform processing on the coefficient level values output from the transform/quantization unit 131 . Here, the inverse transform/inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transform processing and inverse quantization are performed in the reverse order of the transform processing and quantization performed by the transform/quantization unit 131 .

The encoding unit 140 is configured to encode the coefficient level values output from the transform/quantization unit 131 and output encoded data.

Here, for example, the coding is entropy coding that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

In addition to the coefficient level values, the encoding unit 140 is also configured to encode control data used in the decoding process.

Here, the control data may include size data such as a coding block (CU: Coding Unit) size, a prediction block (PU: Prediction Unit) size, a transform block (TU: Transform Unit) size, and the like.

The in-loop filtering section 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122 and output the filtered decoded signal to the frame buffer 160 .

Here, for example, the filter processing is deblocking filter processing that reduces distortion that occurs at boundaries of blocks (encoding blocks, prediction blocks, or transform blocks).

The frame buffer 160 is configured to accumulate reference frames used by the inter prediction section 111 .

Here, the filtered decoded signal constitutes a reference frame used in inter prediction section 111 .

(Inter prediction unit 111)
The inter prediction unit 111 of the image coding apparatus 100 according to this embodiment will be described below with reference to FIG. FIG. 3 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding device 100 according to this embodiment.

As shown in FIG. 3, the inter prediction unit 111 has a motion vector search unit 111A, a motion vector encoding unit 111B, a refinement unit 111C, and a prediction signal generation unit 111D.

The inter prediction unit 111 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on motion vectors.

The motion vector search unit 111A is configured to identify a reference block included in the reference frame by comparing the target frame and the reference frame, and search for a motion vector for the identified reference block.

Also, the search described above is performed for a plurality of reference frame candidates, and the reference frame and motion vector used for prediction in the prediction block are determined. A maximum of two reference frames and motion vectors can be used for one block. The case of using only one set of reference frames and motion vectors for one block is called uni-prediction, and the case of using two sets of reference frames and motion vectors is called bi-prediction. Hereinafter, the first set will be called L0, and the second set will be called L1.

Furthermore, the motion vector search unit 111A is configured to determine the encoding method of the reference frame and motion vector. Coding methods include a merge mode, which will be described later, in addition to a normal method of transmitting reference frame information and motion vector information.

It should be noted that the motion vector search method, the reference frame determination method, and the reference frame and motion vector encoding method can employ known methods, and thus details thereof will be omitted.

The motion vector encoding unit 111B is configured to encode the reference frame and motion vector information determined by the motion vector search unit 111A using the encoding method similarly determined by the motion vector search unit 111A.

When the encoding method for the block is the merge mode, the image encoding device 100 first creates a merge list for the block. Here, the merge list is a list listing a plurality of combinations of reference frames and motion vectors.

Each combination is assigned an index, and the image encoding device 100 encodes only the index and transmits it to the image decoding device 200 instead of encoding the reference frame and motion vector information individually. The image encoding device 100 side and the image decoding device 200 side share the merge list creation method, so that the image decoding device 200 side decodes the reference frame and motion vector information only from the information related to the index. can do.

As for the method of creating the merge list, since it is possible to adopt a known method, the details thereof will be omitted.

For encoding motion vector information, first, a predicted motion vector is generated as a predicted value of a motion vector to be encoded, and a differential motion vector, which is a difference value between the predicted motion vector and a motion vector to be actually encoded, is generated. Encode the vector.

The refinement unit 111C is configured to perform a refinement process (for example, DMVR) for correcting the motion vector encoded by the motion vector encoder 111B.

Specifically, the refining unit 111C sets a search range based on the reference position specified by the motion vector encoded by the motion vector encoding unit 111B, and selects a modification having the lowest predetermined cost from the search range. A refinement process is configured to identify the reference position and modify the motion vector based on the modified reference position.

FIG. 4 is a flow chart showing an example of a processing procedure of the refining unit 111C.

As shown in FIG. 4, in step S41, the refiner 111C determines whether a predetermined condition for applying the refinement process is satisfied. If all such predetermined conditions are satisfied, the process proceeds to step S42. On the other hand, if any one of the predetermined conditions is not satisfied, the processing procedure proceeds to step S48 and the refinement processing ends.

Here, the predetermined condition includes a condition that the block is a block for bi-prediction. Further, the predetermined condition may include a condition that motion vectors are encoded in merge mode.

In step S42, the refining unit 111C generates a search image based on the motion vector encoded by the motion vector encoding unit 111B and the reference frame information.

Here, if the motion vector points to a non-integer pixel position, the refining unit 111C applies a filter to the pixel values of the reference frame and interpolates the pixels at the non-integer pixel position. At this time, the refining unit 111C can reduce the amount of calculation by using an interpolation filter with a smaller number of taps than an interpolation filter used in the prediction signal generation unit 111D, which will be described later. For example, the refinement unit 111C can interpolate pixel values at non-integer pixel positions by bilinear interpolation.

In step S43, the refining unit 111C calculates the search cost at the initial position using the search image generated in step S42.

Here, the initial position is the position indicated by the motion vector encoded by the motion vector encoding unit 111B. Further, the search cost is an index value of the degree of similarity between the reference block on the L0 side (first reference frame side) and the reference block on the L1 side (second reference frame side) indicated by the motion vector. A sum of absolute errors or a sum of squared errors between values can be used.

In step S44, the refining unit 111C receives the search cost at the initial position calculated in step S43 as input and determines whether or not a termination condition for terminating the refining process for the block is satisfied. If all such termination conditions are satisfied, the process proceeds to step S45. On the other hand, if any one of these end conditions is not satisfied, the process proceeds to step S48 to end the refinement process.

Here, the termination condition (termination condition) in step S44 is, for example, that the search cost at the initial position is smaller than a predetermined first threshold value (or a first threshold) can be included.
Further, the termination condition (termination condition) in step S44 includes, for example, that the search cost at the initial position is greater than a predetermined second threshold value (or that the search cost at the initial position is greater than a predetermined threshold value). 1 threshold or more) can be included.

In step S45, the refining unit 111C performs a search with integer pixel accuracy using the search image generated in step S42.

Here, the integer pixel precision means searching only for points that are at integer pixel intervals based on the motion vector encoded by the motion vector encoding unit 111B.

The refining unit 111C determines the motion vector after correction at the integer-pixel interval position through the search in step S45. Here, a known method can be used as a method for such search.

For example, the refining unit 111C can also perform a search by a method of searching only for points where the differential motion vectors on the L0 side and the L1 side are combinations in which only the signs are inverted.

Here, the search point with the lowest search cost at each search position is used as the motion vector after correction at the integer-pixel interval position. As the search cost, as described above, an index such as the sum of absolute differences or the sum of squared errors can be used. At this time, the refining unit 111C adds the Lp norm of the difference vector between the motion vector after modification and the motion vector at the initial position to the index for evaluating the similarity between blocks such as the sum of absolute differences. may be used as the search cost.

Specifically, the refining unit 111C may use, for example, the sum of the sum of absolute differences and the L1 norm of the difference vector as the search cost. Further, when adding the L1 norm to the sum of absolute differences, the refining unit 111C may perform the addition after weighting with a predetermined ratio. For example, the refining unit 111C may add a value obtained by doubling the L1 norm to the sum of absolute differences. At this time, if the magnification is a power of 2, equivalent processing can be realized by bit shifting.
It should be noted that, as a result of the search in step S45, there is a possibility that the motion vector has the same value as before the search.

In step S46, the refining unit 111C uses the search cost corresponding to the corrected motion vector at the integer-pixel interval position determined in step S45, and the end condition for terminating the refining process for the block is satisfied. determine whether or not If all such end conditions are satisfied, the process proceeds to step S47. On the other hand, if any one of these end conditions is not satisfied, the process proceeds to step S48 to end the refinement process.

Here, the termination condition (termination condition) in step S46 includes, for example, that the above-described search cost is greater than a predetermined third threshold (or that the above-described search cost is greater than or equal to a predetermined third threshold). condition can be included. At this time, the third threshold may be set to the same value as the above-described second threshold.

Further, if the refining unit 111C determines that the termination condition is satisfied, the refining unit 111C discards the search result in step S45, and the same motion vector as when the refining process is not performed (that is, the motion vector The motion vector encoded by the encoding unit 111B) may be used as the final motion vector of the block.

The termination condition (termination condition) in step S46 includes, for example, the difference value between the search cost at the initial position calculated in step S43 and the search cost corresponding to the corrected motion vector calculated in step S45. A condition that the difference value is smaller than a predetermined fourth threshold value (or that the difference value is equal to or less than a predetermined fourth threshold value) may be included. At this time, the fourth threshold may be set to the same value as the first threshold.

Further, if the refining unit 111C determines that the end condition is satisfied, the refining unit 111C discards the search result in step S45, and adds the same motion vector as when the refining process is not performed to the block. It may be used as the final motion vector.

In step S47, the refining unit 111C searches for a motion vector in non-integer pixel precision using the motion vector after modification in integer pixel precision determined in step S43 as an initial value. Here, a known method can be used as a motion vector search method.

The refining unit 111C can also use the result of step S43 as an input and determine a motion vector with non-integer pixel precision using a parametric model such as parabolic fitting without actually performing a search.

In step S47, the refining unit 111C determines the motion vector after modification with non-integer pixel precision, and then proceeds to step S48 to end the refining process. Here, for the sake of convenience, the expression "corrected motion vector with non-integer pixel precision" is used, but the result of the search in step S47 may result in the same value as the motion vector with integer pixel precision obtained in step S45. There is also

For the sake of convenience, step S43 and step S45 have been described above as separate steps, but both processes may be executed within the same step. For example, the processing procedure moves to step S45 immediately after step S42, and the refinement unit 111C calculates both the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel interval position in step S45. can be calculated. After that, the process proceeds to step S46, and the refining unit 111C considers at least one of the conditions described as step S44 and the conditions described as step S46 to determine whether the termination condition is satisfied. A determination can be made as to whether there is

Further, for example, in step S48, the refining unit 111C may use the search cost at the initial position and the search cost corresponding to the corrected motion vector at the pixel interval position to determine whether to discard the search result. can.

For example, if the search cost at the initial position is greater than the second threshold (or is greater than or equal to the second threshold), the refinement unit 111C discards the search result and does not perform the refinement process. The same motion vector as (that is, the motion vector encoded by the motion vector encoding unit 111B) may be used as the final motion vector of the block.

Also, for example, if the search cost corresponding to the modified motion vector at the integer-pixel interval position is greater than the third threshold (or is greater than or equal to the third threshold), the refining unit 111C discards the search result. Then, the same motion vector as when refinement processing is not performed may be used as the final motion vector for the block.

Further, for example, when the difference value between the search cost at the initial position and the search cost corresponding to the corrected motion vector at the integer-pixel interval position is smaller than the fourth threshold (or is equal to or less than the fourth threshold). , the refining unit 111C may discard the search result and use the same motion vector as when the refining process is not performed as the final motion vector of the block.

Although the configuration including all steps S41 to S48 has been described above, steps S44 and S46 do not necessarily have to be included in the configuration.

The refining unit 111C may divide a block larger than a predetermined threshold into smaller sub-blocks and perform refinement processing on each sub-block. For example, the refining unit 111C sets the execution unit of the refining process to 16×16 pixels, and if the horizontal or vertical size of the block is larger than 16 pixels, it is divided into 16 pixels or less. can do. At this time, the motion vectors of the block encoded by the motion vector encoding unit 111B are used for all the sub-blocks in the same block as the reference motion vectors for the refinement process.

When performing processing for each subblock, the refinement unit 111C may perform all the procedures in FIG. 4 for each subblock. Further, the refining unit 111C may process only part of the processing in FIG. 4 for each subblock. Specifically, the refining unit 111C may process steps S41 and S42 of FIG. 4 for each block, and may process steps S43 to S48 for each sub-block.

The predicted signal generator 111D is configured to generate a predicted signal based on the modified motion vector output from the refiner 111C.

Here, as will be described later, the prediction signal generation unit 111D performs BDOF (Bi-Directional Optical Flow) processing for each block based on information (for example, search cost) calculated in the course of the refinement processing described above. It is configured to determine whether or not to perform.

Specifically, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector encoded by the motion vector encoding unit 111B when the motion vector is not corrected. On the other hand, the prediction signal generation unit 111D is configured to generate a prediction signal based on the motion vector corrected by the refinement unit 111C when the motion vector is corrected.

FIG. 5 is a flow chart showing an example of a processing procedure of the prediction signal generator 111D. Here, when refinement processing is performed in units of subblocks by the refinement unit 111C, processing in the prediction signal generation unit 111D is also performed in units of subblocks. In that case, the word "block" in the following description can be appropriately read as "sub-block".

Also, if the block size is larger than a predetermined threshold value for a block that has not been refined by the refinement unit 111C, it may be divided into small sub-blocks and the refinement process may be performed for each sub-block. good. For example, similarly to the refining unit 111C, if the execution unit of the prediction signal generation process is set to 16×16 pixels and the size of the block in the horizontal direction or the vertical direction is larger than 16 pixels, it will be 16 pixels or less. can be divided as follows: Also in this case, the word "block" in the following description can be appropriately read as "sub-block".

As shown in FIG. 5, in step S51, the prediction signal generator 111D generates a prediction signal.

Specifically, the prediction signal generation unit 111D receives the motion vector encoded by the motion vector encoding unit 111B or the motion vector encoded by the refinement unit 111C, and the position indicated by the motion vector is a non-integer. For pixel locations, a filter is applied to the pixel values of the reference frame to interpolate pixels at non-integer pixel locations. Here, as a specific filter, a horizontal/vertical separable filter with a maximum of 8 taps disclosed in Non-Patent Document 1 can be applied.

When the block is a block to be bi-predicted, the prediction signal generation unit 111D generates a prediction signal based on the first (hereinafter referred to as L0) reference frame and motion vector and the second (hereinafter referred to as L1) generates both the reference frame and the motion vector prediction signal.

In step S52, the prediction signal generation unit 111D checks whether or not the conditions for applying the BDOF process, which will be described later, are satisfied.

As such applicable conditions, the conditions described in Non-Patent Document 1 can be applied. The application condition includes at least the condition that the block in question is a block for bi-prediction. Also, the application condition may include a condition that the motion vector of the block is not encoded in Symmetric MVD mode, as described in Non-Patent Document 1.

If the applicable condition is not satisfied, the process proceeds to step S55 and ends. At this time, the prediction signal generator 111D outputs the prediction signal generated in step S51 as the final prediction signal.

On the other hand, if all the applicable conditions are satisfied, the processing procedure moves to step S53. In step S53, this processing procedure determines whether or not the BDOF process of step S54 is actually executed for blocks that satisfy the application condition.

In the following, the case where the sum of absolute differences is used as the search cost will be described as an example, but other indices can also be used as the search cost. For example, an index value for determining the similarity between image signals, such as the sum of absolute differences or the sum of squared errors between signals after local mean values have been removed, can be used as the search cost. .

For example, the predicted signal generation unit 111D calculates the sum of the absolute difference between the predicted signal of L0 and the predicted signal of L1, and if the calculated value is smaller than a predetermined threshold (or if the calculated value is If it is equal to or less than a predetermined threshold), it is determined that the BDOF process is not performed.

Further, for example, the predicted signal generation unit 111D calculates the sum of the absolute difference between the predicted signal of L0 and the predicted signal of L1, and if the calculated value is larger than a predetermined threshold value (or if the calculated value is equal to or greater than a predetermined threshold), it is also possible to make a determination such as not performing the BDOF process.

Here, the prediction signal generating unit 111D can also use the result of the refinement processing to determine whether or not to apply the BDOF processing to the blocks for which the refinement processing has been executed by the refinement unit 111C.

The prediction signal generation unit 111D uses the search cost (for example, the sum of the absolute differences between the pixel values of the reference block on the L0 side and the pixel values of the reference block on the L1 side) calculated in the process of the refinement process described above. , whether or not to apply the BDOF process.

For example, in the search with integer pixel accuracy in step S45, the prediction signal generation unit 111D determines that the sum of absolute differences at the search point at which the above-described search cost (sum of absolute differences) is the lowest is greater than a predetermined fifth threshold. is smaller (or less than or equal to a predetermined fifth threshold), it can be determined not to apply the BDOF processing. At this time, the fifth threshold may be set to the same value as the first threshold.

For example, in the search with integer pixel accuracy in step S45, the prediction signal generation unit 111D determines that the sum of absolute differences at the search point at which the above-described search cost (sum of absolute differences) is the lowest is greater than a predetermined sixth threshold. is large (or is equal to or greater than a predetermined sixth threshold), it can be determined that the BDOF process is not applied. At this time, the sixth threshold may be set to the same value as the second threshold or the third threshold.

For example, when the search cost at the initial position calculated in step S43 is smaller than a predetermined fifth threshold (or equal to or less than a predetermined fifth threshold), the prediction signal generation unit 111D performs BDOF processing. can be determined not to apply. At this time, the fifth threshold may be set to the same value as the first threshold.

For example, when the search cost at the initial position calculated in step S43 is greater than a predetermined sixth threshold (or is equal to or greater than a predetermined sixth threshold), the prediction signal generator 111D performs BDOF processing. can be determined not to apply. At this time, the sixth threshold may be set to the same value as the second threshold or the third threshold.

For example, the prediction signal generation unit 111D determines that the difference value between the search cost at the initial position calculated in step S43 and the minimum search cost in the search with integer pixel accuracy in step S45 is higher than a predetermined seventh threshold. When it is small (or when it is equal to or less than a predetermined seventh threshold), it can be determined not to apply the BDOF processing. At this time, the seventh threshold may be set to the same value as the first threshold or the fourth threshold.

In addition, the prediction signal generation unit 111D may make a determination using a method based on the result of the refinement processing for blocks that have undergone refinement processing, and a method based on the sum of absolute differences for other blocks.

Further, as described above, the predicted signal generation unit 111D does not newly calculate the sum of absolute differences between the predicted signal on the L0 side and the predicted signal on the L1 side. It is also possible to adopt a configuration in which the propriety of BDOF processing is determined using only information. In this case, in step S53, the prediction signal generation unit 111D determines to always apply the BDOF processing to blocks for which refinement processing has not been performed.

According to such a configuration, in this case, since there is no need to perform the process of calculating the sum of absolute differences in the prediction signal generation unit 111D, it is possible to reduce the amount of processing and the processing delay from the viewpoint of hardware implementation.

In addition, according to such a configuration, from the viewpoint of software implementation, the result of the refinement process is used to prevent the execution of the BDOF process in blocks where the effect of the BDOF process is estimated to be low. The processing time for the entire image can be shortened while maintaining the

Further, the determination process itself using the result of the above-described refinement process is executed inside the refinement unit 111C, and information indicating the result is transmitted to the prediction signal generation unit 111D, whereby the prediction signal generation unit 111D , BDOF processing can also be determined.

For example, as described above, the values of motion vectors and search costs before and after refinement processing are determined. A flag is prepared so that it becomes "0" when the conversion process is not applied, and the prediction signal generation unit 111D can refer to the value of the flag to determine whether the BDOF process is appropriate.

Also, here, for the sake of convenience, step S52 and step S53 have been described as different steps, but determinations in step S52 and step S53 can be performed at the same time.

For blocks determined not to apply the BDOF process in the determination as described above, the process proceeds to step S55. For other blocks, the process proceeds to step S54.

In step S54, the prediction signal generator 111D executes BDOF processing. Since the BDOF process itself can use a known technique, the detailed description is omitted. After the BDOF process is performed, the process proceeds to step S55 and ends.

The above-mentioned first to seventh thresholds are defined to vary according to the block size (the product of block height and width) or the number of pixels used for calculating search costs such as SAD in the block. may

Each of the first to seventh thresholds described above may be defined so as to vary in value according to the quantization parameter (QP) of the block. For example, if the cost value tends to be large when the quantization parameter is large, the threshold can be defined to be large as the quantization parameter becomes large.

(Image decoding device 200)
The image decoding device 200 according to this embodiment will be described below with reference to FIG. FIG. 6 is a diagram showing an example of functional blocks of the image decoding device 200 according to this embodiment.

As shown in FIG. 6, the image decoding device 200 includes a decoding unit 210, an inverse transform/inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, and an in-loop filtering unit. 250 and a frame buffer 260 .

The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and decode the coefficient level values.

Here, for example, the decoding is entropy decoding in a procedure opposite to the entropy encoding performed by the encoding unit 140 .

Further, the decoding unit 210 may be configured to acquire the control data by decoding the encoded data.

Note that, as described above, the control data may include size data such as the encoding block size, prediction block size, transform block size, and the like.

The inverse transform/inverse quantization unit 220 is configured to perform inverse transform processing on the coefficient level values output from the decoding unit 210 . Here, the inverse transform/inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transform processing and inverse quantization are performed in the reverse order of the transform processing and quantization performed by the transform/quantization unit 131 .

The adder 230 adds the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 220 to generate a pre-filtering decoded signal. It is configured to output to the filter processing unit 250 .

Here, the unfiltered decoded signal constitutes a reference block used in intra prediction section 242 .

Like the inter prediction section 111, the inter prediction section 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from encoded data and a reference signal included in a reference frame. The inter prediction section 241 is configured to output a prediction signal to the adder 230 .

The intra prediction unit 242, like the intra prediction unit 112, is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction section 242 is configured to output the prediction signal to the adder 230 .

Similar to in-loop filtering section 150, in-loop filtering section 250 performs filtering on the unfiltered decoded signal output from adder 230, and outputs the filtered decoded signal to frame buffer 260. is configured to

Here, for example, the filter processing is deblocking filter processing that reduces distortion occurring at boundaries of blocks (encoding blocks, prediction blocks, transform blocks, or sub-blocks obtained by dividing them).

The frame buffer 260, like the frame buffer 160, is configured to accumulate reference frames used by the inter prediction section 241. FIG.

Here, the decoded signal after filtering constitutes a reference frame used in the inter prediction section 241 .

(Inter prediction unit 241)
The inter prediction unit 241 according to this embodiment will be described below with reference to FIG. FIG. 7 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

As shown in FIG. 7, the inter prediction unit 241 has a motion vector decoding unit 241B, a refinement unit 241C, and a prediction signal generation unit 241D.

The inter prediction unit 241 is an example of a prediction unit configured to generate a prediction signal included in a prediction block based on motion vectors.

The motion vector decoding unit 241B is configured to acquire motion vectors by decoding control data received from the image encoding device 100 .

The refinement unit 241C is configured to perform a refinement process for correcting motion vectors, similar to the refinement unit 111C.

The prediction signal generator 241D is configured to generate a prediction signal based on the motion vector, like the prediction signal generator 111D.

According to the image encoding device 100 and the image decoding device 200 according to the present embodiment, if the search cost is greater than the predetermined threshold value (or if the search cost is equal to or greater than the predetermined threshold value), If there is, it can be configured to discard the search result. Here, the refinement processing can improve the encoding efficiency when the pixel value correlation between the block, the reference block on the L0 side, and the reference block on the L1 side is high. With the above configuration, refinement processing is performed on blocks with low correlation in the time direction without transmitting additional information (information on whether to apply refinement processing) from the image encoding device 100 side. Therefore, it is possible to prevent deterioration of coding efficiency.

According to the image encoding device 100 and the image decoding device 200 according to the present embodiment, in the refining units 111C and 241C, the difference value between the search cost at the initial position and the search cost corresponding to the post-search point is A configuration can be adopted in which the search result is discarded when it is smaller than a predetermined threshold value (or when it is equal to or less than a predetermined threshold value). This makes it possible to avoid unnecessarily correcting the motion vector when the search cost is slightly smaller than the initial position by chance due to the influence of noise, etc., thereby preventing the deterioration of the coding efficiency. can be done.

According to the image encoding device 100 and the image decoding device 200 according to this embodiment, the refinement units 111C and 241C can be configured to include the norm of the difference vector between the initial position and the search point in the search cost. In this case, if the index value for evaluating the degree of similarity, such as the sum of absolute error values, is approximately the same at each search point, regularization using the norm of the difference vector will result in unnecessary correction of the motion vector. It is possible to prevent it from becoming large, and it is possible to prevent the deterioration of the coding efficiency.

According to the image encoding device 100 and the image decoding device 200 according to the present embodiment, the reference block on the L0 side and the reference block on the L1 side are used to determine whether or not to execute the BDOF processing in the prediction signal generation units 111D and 241D. (or the similarity is greater than or equal to a predetermined threshold). The BDOF process, like the refinement process, is also effective when the block in question, the reference block on the L0 side, and the reference block on the L1 side have a high correlation. Therefore, with the above configuration, BDOF processing is performed on blocks with low correlation in the time direction without transmitting additional information (information on whether to apply BDOF processing) from the image encoding device 100 side. Therefore, it is possible to prevent deterioration of coding efficiency. Also, the result of the refinement process described above can be used for the determination described above. This makes it possible to reduce the processing for calculating the above-described absolute value difference.

Also, the image encoding device 100 and the image decoding device 200 described above may be implemented as a program that causes a computer to execute each function (each process).

In each of the above embodiments, the application of the present invention to the image encoding device 100 and the image decoding device 200 has been described as an example, but the present invention is not limited to such examples. The same can be applied to an image encoding/decoding system having the functions of the encoding device 100 and the image decoding device 200 .

According to the present invention, it is possible to prevent deterioration of coding efficiency by not performing refinement processing on blocks with low temporal correlation.

10... Image processing system 100... Image encoding devices 111, 241... Inter prediction unit 111A... Motion vector search unit 111B... Motion vector encoding units 111C, 241C... Refinement units 111D, 241D... Prediction signal generation units 112, 242... Intra prediction unit 121 Subtractors 122, 230 Adder 131 Transformation/quantization units 132, 220 Inverse transformation/inverse quantization unit 140 Encoding units 150, 250 In-loop filtering units 160, 260 Frame Buffer 200... Image decoding device 210... Decoding unit 241B... Motion vector decoding unit

Claims (3)

An image decoding device,
a motion vector decoder configured to decode motion vectors from encoded data;
and a refinement section,
the refiner determines whether a predetermined condition for applying the refinement process is satisfied;
The predetermined conditions include a condition that the target block is a block that performs bi-prediction, and a condition that the motion vector is coded in merge mode,
When the refinement unit determines that all the predetermined conditions are satisfied,
searching for a motion vector using the value of the motion vector decoded by the motion vector decoding unit as an initial value;
if the search cost at the initial search point is less than a predetermined first threshold, terminate the motion vector search;
If the search cost at the initial search point is equal to or greater than the first threshold, search is performed at integer pixel intervals;
when the difference value between the search cost at the initial search point and the minimum search cost in the search at the integer pixel interval position is smaller than a predetermined third threshold, or when the difference value is equal to or less than the third threshold , an image decoding apparatus configured to discard a search result at the integer-pixel interval position. decoding motion vectors from the encoded data;
determining whether predetermined conditions for applying the refinement process are satisfied, and if it is determined that all of the predetermined conditions are satisfied,
a step of searching for a motion vector using the value of the decoded motion vector as an initial value;
terminating the motion vector search if the search cost at the initial search point is less than a predetermined first threshold;
If the search cost at the initial search point is equal to or greater than the first threshold, searching at integer-pixel intervals;
When a difference value between the search cost at the initial search point and the minimum search cost in the search at the integer pixel interval position is smaller than a predetermined third threshold, or when the difference value is equal to or less than the threshold, and discarding the search result at the integer pixel interval position,
The image decoding method, wherein the predetermined conditions include a condition that the target block is a block for bi-prediction and a condition that the motion vector is coded in merge mode. A program for use in an image decoding device, the computer comprising:
decoding motion vectors from the encoded data;
determining whether predetermined conditions for applying the refinement process are satisfied, and if it is determined that all of the predetermined conditions are satisfied,
a step of searching for a motion vector using the value of the decoded motion vector as an initial value;
terminating the motion vector search if the search cost at the initial search point is less than a predetermined first threshold;
If the search cost at the initial search point is equal to or greater than the first threshold, searching at integer-pixel intervals;
when the difference between the search cost at the initial search point and the minimum search cost in the search at the integer-pixel interval position is smaller than a predetermined threshold, or when the difference is equal to or less than the threshold, the integer pixel and discarding the search results at the interval positions;
The program, wherein the predetermined conditions include a condition that the target block is a block that performs bi-prediction, and a condition that the motion vector is coded in merge mode.

Images