JP7593876B2 - Image encoding device, image decoding device, image processing system and program - Google Patents

Description

The present invention relates to an image encoding device, an image decoding device, an image processing system, and a program.

Non-Patent Document 1 discloses a technique for decoding images.

ITU-T Recommendation H.266 Versatile video coding ITU-T Recommendation H.274 Versatile supplemental enhancement information messages for coded video bitstreams

However, the technology disclosed in Non-Patent Document 1 has the problem that spatial scalability coding cannot be performed in the Main Profile.

The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image encoding device, an image decoding device, an image processing system, and a program that can realize pseudo-spatial scalability using RPR, a tool that can be used in the Main Profile.

The first feature of the present invention is an image coding device comprising an image coding unit, the image coding unit being configured to generate a low-resolution image by performing a downsampling process on a high-resolution image, and configured to perform inter-prediction by referring to only the low-resolution image, and configured to perform inter-prediction for the high-resolution image by referring to the low-resolution image at the same time and using RPR (Reference Picture Resampling) processing.

The second feature of the present invention is an image decoding device including an image decoding unit, which is configured to decode a high-resolution image and a low-resolution image of the same time as different frames, and to perform inter prediction using RPR (Reference Picture Resampling) processing when the resolution of the frame to be decoded differs from the resolution of the reference frame.

The third feature of the present invention is an image processing system including an image encoding device and an image decoding device, the image encoding device including an image encoding unit, the image encoding unit configured to generate a low-resolution image by performing a downsampling process on a high-resolution image, and configured to perform inter-prediction by referring to only the low-resolution image, and configured to perform inter-prediction by referring to the low-resolution image of the same time for the high-resolution image and using RPR (Reference Picture Resampling) processing, the image decoding device including an image decoding unit, the image decoding unit configured to decode the high-resolution image and the low-resolution image of the same time as different frames, and to perform inter-prediction by using RPR (Reference Picture Resampling) processing when the resolution of the frame to be decoded differs from the resolution of the reference frame.

The fourth feature of the present invention is a program for causing a computer to function as an image coding device, the image coding device including an image coding unit configured to generate a low-resolution image by performing downsampling processing on a high-resolution image, and configured to perform inter-prediction by referring to only the low-resolution image, and configured to perform inter-prediction for the high-resolution image by referring to the low-resolution image at the same time and using RPR (Reference Picture Resampling) processing.

The fifth feature of the present invention is a program for causing a computer to function as an image decoding device, the image decoding device including an image decoding unit configured to decode a high-resolution image and a low-resolution image of the same time as different frames, and to perform inter prediction using RPR (Reference Picture Resampling) processing when the resolution of the frame to be decoded differs from the resolution of the reference frame.

According to the present invention, it is possible to provide an image encoding device, an image decoding device, an image processing system, and a program that can realize pseudo-spatial scalability using RPR, a tool that can be used in the Main Profile.

FIG. 1 is a diagram showing an example of a configuration of an image processing system 1 according to an embodiment. 1 is a diagram illustrating an example of functional blocks of an image encoding device 10 according to an embodiment. FIG. 3 is a diagram showing an example of functional blocks of the image encoding unit 100 of the image encoding device 10 according to an embodiment. FIG. 4 is a flowchart showing an example of processing of the image encoding device 10 according to an embodiment. FIG. 5 is a diagram showing an example of a pre-processed image input to the image encoding unit 100 when adaptive downsampling is performed for each frame by the processing in steps S401 to S405 shown in FIG. FIG. 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding unit 100 of the image encoding device 10 according to an embodiment. FIG. 7 is a diagram showing an example of functional blocks of an image decoding device 20 according to an embodiment. FIG. 8 is a diagram showing an example of functional blocks of the image decoding unit 200 according to an embodiment. FIG. 9 is a diagram illustrating an example of functional blocks of the inter prediction unit 241 according to an embodiment. FIG. 10 is a diagram for explaining the second modification. FIG. 11 is a diagram for explaining the second modification. FIG. 12 is a diagram for explaining the third modification. FIG. 13 is a diagram for explaining the third modification. FIG. 14 is a diagram for explaining the third modification. FIG. 15 is a diagram for explaining the third modification. FIG. 16 is a diagram for explaining the fourth modification. FIG. 17 is a diagram for explaining the fourth modification.

The following describes embodiments of the present invention with reference to the drawings. Note that the components in the following embodiments can be replaced with existing components as appropriate, and various variations, including combinations with other existing components, are possible. Therefore, the description of the following embodiments does not limit the content of the invention described in the claims.

First Embodiment
An image processing system 1 according to a first embodiment of the present invention will be described below with reference to Figures 1 to 12. Figure 1 is a diagram showing an image processing system 1 according to the first embodiment of the present invention.

As shown in FIG. 1, the image processing system 1 has an image encoding device 10 and an image decoding device 20.

The image encoding device 10 is configured to generate encoded data by encoding an input image signal. The image decoding device 20 is configured to generate an output image signal by decoding the encoded data.

Here, the encoded data may be transmitted from the image encoding device 10 to the image decoding device 20 via a transmission path. Also, the encoded data may be stored in a storage medium and then provided from the image encoding device 10 to the image decoding device 20.

(Image encoding device 10)
The image encoding device 10 according to this embodiment will be described below with reference to Fig. 2. Fig. 2 is a diagram showing an example of functional blocks of the image encoding device 10 according to this embodiment.

As shown in FIG. 2, the image encoding device 10 has a preprocessing unit 300, an analysis unit 400, a control unit 500, a UI unit 600, and an image encoding unit 100.

The pre-processing unit 300 is configured to receive an input image signal and control information output from the control unit 500, perform pre-processing, and output a pre-processed image signal and control data.

The analysis unit 400 is configured to receive an input image signal and output analysis information to the control unit 500.

The control unit 500 is configured to receive the analysis information output from the analysis unit 400 and the setting information output from the UI unit 600, and to output control information to the preprocessing unit 300 and the image encoding unit 100.

The UI unit 600 is configured to output setting information specified by the user via the GUI or CUI to the control unit 500.

The image encoding unit 100 is configured to receive the preprocessed image signal and control data output from the preprocessing unit 300 and the control information output from the control unit 500, and to output a bitstream.

(Image Encoding Unit 100)
The image encoding unit 100 of the image encoding device 10 according to this embodiment will be described below with reference to Fig. 3. Fig. 3 is a diagram showing an example of functional blocks of the image encoding unit 100 according to this embodiment.

As shown in FIG. 3, the image encoding unit 100 includes an inter prediction unit 111, an intra prediction unit 112, a subtractor 121, an adder 122, a transform/quantization unit 131, an inverse transform/inverse quantization unit 132, an encoding unit 140, an in-loop filter processing unit 150, and a frame buffer 160.

The inter prediction unit 111 is configured to generate a prediction signal by inter prediction (inter-frame prediction), similar to the inter prediction unit 241 described later.

Specifically, the inter prediction unit 111 is configured to identify a reference block included in a reference frame by comparing a frame to be coded (hereinafter, the target frame) with a reference frame stored in the frame buffer 160, and to determine a motion vector for the identified reference block.

Furthermore, the inter prediction unit 111 is configured to generate a prediction signal included in the prediction block for each prediction block based on the reference block and the motion vector. The inter prediction unit 111 is configured to output the prediction signal to the subtractor 121 and the adder 122. Here, the reference frame is a frame different from the target frame.

The intra prediction unit 112 is configured to generate a prediction signal by intra prediction (intra-frame prediction), similar to the intra prediction unit 242 described below.

Specifically, the intra prediction unit 112 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 112 is also configured to output the prediction signal to the subtractor 121 and the adder 122.

Here, the reference block is a block that is referenced for the block to be predicted (hereinafter, the target block). For example, the reference block is a block adjacent to the target block.

The subtractor 121 is configured to subtract the prediction signal from the input image signal and output the prediction residual signal to the transform/quantization unit 131. Here, the subtractor 121 is configured to generate a prediction residual signal that is the difference between the prediction signal generated by intra prediction or inter prediction and the input image signal.

The adder 122 is configured to add the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 132 to generate a pre-filter decoded signal, and output the pre-filter decoded signal to the intra prediction unit 112 and the in-loop filter processing unit 150.

Here, the unfiltered decoded signal constitutes a reference block used by the intra prediction unit 112.

The transform/quantization unit 131 is configured to perform a transform process on the prediction residual signal and to obtain coefficient level values. Furthermore, the transform/quantization unit 131 may be configured to quantize the coefficient level values.

The conversion process is a process of converting a prediction residual signal into a frequency component signal. In such a conversion process, a basis pattern (transformation matrix) corresponding to a discrete cosine transform (DCT) may be used, or a basis pattern (transformation matrix) corresponding to a discrete sine transform (DST) may be used.

The inverse transform and inverse quantization unit 132 is configured to perform inverse transform processing of the coefficient level values output from the transform and quantization unit 131, similar to the inverse transform and inverse quantization unit 220. Here, the inverse transform and inverse quantization unit 132 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

Here, the inverse transformation process and inverse quantization are performed in the reverse order to the transformation process and quantization performed by the transformation/quantization unit 131.

The encoding unit 140 is configured to encode the coefficient level values output from the transform/quantization unit 131 and output the encoded data.

Here, for example, the coding is entropy coding, which assigns codes of different lengths based on the probability of occurrence of coefficient level values.

The encoding unit 140 is also configured to encode control data used in the decoding process in addition to the coefficient level values.

In addition to the control data input from the preprocessing unit 300, the control data may also include size data such as the coding block size, prediction block size, and transformation block size.

Like the in-loop filter processing unit 250, the in-loop filter processing unit 150 is configured to perform filtering on the unfiltered decoded signal output from the adder 122, and to output the filtered decoded signal to the frame buffer 160.

The frame buffer 160 is configured to store reference frames used by the inter prediction unit 111.

Below, an example of the processing of the image encoding device 10 according to this embodiment will be described with reference to FIG. 4. FIG. 4 is a flowchart showing an example of the processing of the image encoding device 10.

As shown in FIG. 4, in step S401, the analysis unit 400 analyzes the input image signal, estimates the encoding difficulty of the input image signal, and outputs the result as analysis information.

For example, the analysis unit 400 may calculate the encoding difficulty from the variance of pixel values (luminance signal values or color difference signal values) of the input image signal. For example, the analysis unit 400 may estimate that the encoding difficulty is high when the variance value is greater than a predetermined threshold, and may estimate that the encoding difficulty is low when the variance value is equal to or less than the threshold.

For example, the analysis unit 400 may calculate the encoding difficulty from the difference between pixel values and a reference frame. Specifically, the analysis unit 400 may estimate the encoding difficulty using the sum of absolute values of the difference values, the sum of squares of the difference values, or a difference value per pixel obtained by dividing each value by the number of pixels. For example, the analysis unit 400 may estimate that the encoding difficulty is high when the difference value is greater than a predetermined threshold, and may estimate that the encoding difficulty is low when the difference value is equal to or less than the threshold.

For example, the analysis unit 400 may estimate the encoding difficulty using a machine learning-based technique such as a convolutional neural network (CNN) that is configured and trained to receive an input image signal and output an estimated value of the encoding difficulty.

Here, the analysis unit 400 may use a method other than the above as long as it is configured to receive an input image signal and output an encoding difficulty level.

In the above, we have explained an example in which the encoding difficulty level is divided into two levels, "high" and "low," but the encoding difficulty level may be divided into three or more levels.

In the above, an example has been described in which the analysis unit 400 outputs the encoding difficulty, but the analysis unit 400 may output information for determining the encoding difficulty (e.g., the variance value described above), and the encoding difficulty itself may be determined within the control unit 500.

Furthermore, as described below, if downsampling processing is performed on all input image signals according to user settings from the UI unit 600, this step S401 may be omitted.

After performing the above processing, the procedure proceeds to step S402.

In step S402, the control unit 500 determines whether or not it is necessary to perform downsampling processing.
For example, the control unit 500 may determine whether or not to perform downsampling processing based on the analysis information output from the analysis unit 400. For example, the control unit 500 may determine that it is necessary to perform downsampling processing when it is determined that the encoding difficulty is high.

As described above, the control unit 500 may determine the encoding difficulty internally and then decide whether or not to perform downsampling processing.

As described above, when the encoding difficulty level is defined in three or more stages, the control unit 500 may determine that it is not necessary to perform downsampling processing when the encoding difficulty level is the lowest, and may determine that it is necessary to perform downsampling processing in other cases, and may then determine the method of performing downsampling processing so that the resolution after downsampling processing decreases as the encoding difficulty level increases. In this specification, resolution refers to the size of one frame or one entire field of an image (the value obtained by multiplying the number of pixels in the horizontal direction by the number of pixels in the vertical direction).

For example, the control unit 500 may also determine whether or not to perform downsampling processing based on the resolution of the input image (the image included in the input image signal).

For example, the control unit 500 may determine that it is necessary to perform downsampling processing if the resolution of the input image is greater than a predetermined resolution, and may determine that it is not necessary to perform downsampling processing if the resolution of the input image is equal to or less than the predetermined resolution.

For example, the control unit 500 may determine that downsampling processing needs to be performed when the resolution of the input image signal is 8K resolution, and may determine that downsampling processing does not need to be performed when the resolution of the input image signal is less than 8K resolution.

The control unit 500 may also predefine the resolution of the image after downsampling processing, and if the resolution of the input image is greater than the resolution of the image, determine that downsampling processing needs to be performed and decide to perform downsampling processing so that the resolution of the image after downsampling processing becomes the same.

For example, the control unit 500 may determine whether or not to perform downsampling processing based on control information input from the UI unit 600. For example, when the control information sets the downsampling processing to be always performed, the control unit 500 may determine that the downsampling processing must be performed.

The control unit 500 may also determine whether or not downsampling processing needs to be performed by combining the determination methods described above.

For example, the control unit 500 may obtain information for controlling the On/Off of the downsampling process from the UI unit 600, and when the downsampling process is On and the resolution of the input image is greater than a predetermined resolution, the control unit 500 may determine whether or not to perform the downsampling process based on the encoding difficulty, and otherwise determine that it is not necessary to perform downsampling.

The number of encoding difficulty levels and the specified resolution used in the above process may be specified by the user via the UI unit 600.

After performing the above processing, the procedure proceeds to step S403.

In step S403, the procedure branches based on the result of step S402. If it is determined in step S402 that downsampling processing needs to be performed, the procedure proceeds to step S404. If it is determined in step S402 that downsampling processing does not need to be performed, the procedure proceeds to step S406.

In step S404, the pre-processing unit 300 performs downsampling processing on the input image signal. Note that if step S404 is not performed (if it is determined that downsampling processing is not necessary), the pre-processing unit 300 may output the input image signal directly to the encoding unit 100.

The pre-processing unit 300 may perform the downsampling process using a one-dimensional or two-dimensional downsampling filter.

The pre-processing unit 300 may also perform downsampling processing using a machine learning-based technique such as CNN that learns downsampling processing and upsampling processing as a pair.

After the downsampling process is completed, the procedure proceeds to step S405.

In step S405, the pre-processing unit 300 determines the type of upsampling process to be performed by the image decoding device 20 and the parameters to be used for the upsampling process based on the input image and the downsampled image, and outputs the determined type of upsampling process and the parameters to be used for the upsampling process as control data.

The pre-processing unit 300 may prepare a variety of types of upsampling processing in advance, such as a one-dimensional convolution filter, a two-dimensional convolution filter, a machine learning-based method such as CNN, and an upsampling processing identical to the RPR described below, and output an index value indicating which of these to use as control data.

In addition, the preprocessing unit 300 may have multiple candidates for the same type of filter (e.g., a one-dimensional convolution filter) with variations in filter coefficients, etc.

Furthermore, the preprocessing unit 300 may determine filter coefficients and network parameters (weights, etc.) for machine learning-based methods such as CNN, and output them as control data.

The pre-processing unit 300 may select, as a method for determining the type and parameters of the upsampling process, a method that minimizes the error (sum of absolute error, sum of squared error, etc.) between the input image and the image after the upsampling process. These control data may be encoded as a user-defined SEI message in the image encoding unit 100.

In addition, the pre-processing unit 300 may be configured not to specify the upsampling process and not to output it as control data.

The pre-processing unit 300 may switch whether or not to output control data related to the upsampling process based on control information input from the UI unit 600, for example.

After performing the above process, the procedure proceeds to step S406.

In step S406, the image encoding unit 100 performs encoding processing and outputs a bit stream. Here, an example of the encoding processing in the image encoding unit 100 will be described with reference to Figures 5 and 6.

Figure 5 shows an example of a preprocessed image input to the image encoding unit 100 when adaptive downsampling is performed for each frame by the processing in steps S401 to S405.

In Figure 5, Fr0, Fr2, and Fr5 indicate frames for which downsampling processing was not performed, and Fr1, Fr3, and Fr4 indicate frames for which downsampling processing was performed.

When the image resolution differs for each frame in this way, the inter prediction unit 111 uses a technique called RPR (Reference Picture Resampling) described in Non-Patent Document 1. The processing of the inter prediction unit 111 when using RPR will be described with reference to FIG. 6.

Figure 6 is a diagram showing an example of functional blocks of the inter prediction unit 111 of the image encoding unit 100 of the image encoding device 10 according to this embodiment.

As shown in FIG. 6, the inter prediction unit 111 includes a resolution conversion unit 1111, a parameter determination unit 1112, and a predicted image generation unit 1113.

The resolution conversion unit 1111 is configured to convert the resolution of the reference frame used for inter prediction to be the same as the resolution of the frame to be encoded when the resolution of the frame to be encoded differs from the resolution of the reference frame used for inter prediction.

If the resolution of the reference frame is greater than the resolution of the frame to be encoded, downsampling is performed, and if the resolution of the reference frame is less than the resolution of the frame to be encoded, upsampling is performed. The specific processing content is defined in Non-Patent Document 1, so a detailed explanation will be omitted.

The parameter determination unit 1112 is configured to determine parameters related to inter prediction. Such parameters include, for example, a motion vector and an index value indicating which reference frame to use.

The predicted image generating unit 1113 is configured to generate a predicted image using the parameters determined by the parameter determining unit 1112.

(Image Decoding Device 20)
The image decoding device 20 according to this embodiment will be described below with reference to Fig. 7. Fig. 7 is a diagram showing an example of functional blocks of the image decoding device 20 according to this embodiment.

As shown in FIG. 7, the image decoding device 20 includes an extraction unit 700, an image decoding unit 200, a post-processing unit 800, a control unit 900, and a UI unit 1000.

The extraction unit 700 is configured to receive as input the bitstream and control information output from the control unit 900, extract data to be input to the image decoding unit 200 from the bitstream based on the control information, and output the extracted data as a bitstream after extraction.

The image decoding unit 200 is configured to perform a decoding process using the extracted bitstream described above as input, and to output a decoded image signal and control data.

The post-processing unit 800 is configured to receive the decoded image signal, the control data, and the control information output from the control unit 900 as input, and to output the post-processed decoded signal.

The control unit 900 is configured to receive the decoded image signal and control data output from the image decoding unit 200 and the setting information output from the UI unit 1000, and to output control information.

(Image Decoding Unit 200)
The image decoding unit 200 of the image decoding device 20 according to this embodiment will be described below with reference to Fig. 8. Fig. 8 is a diagram showing an example of functional blocks of the image decoding unit 200 according to this embodiment.

As shown in FIG. 8, the image decoding unit 200 includes a decoding unit 210, an inverse transform/inverse quantization unit 220, an adder 230, an inter prediction unit 241, an intra prediction unit 242, an in-loop filter processing unit 250, and a frame buffer 260.

The decoding unit 210 is configured to decode the encoded data generated by the image encoding device 100 and to decode the coefficient level values.

Here, for example, the decoding is of entropy coded data that assigns codes of different lengths based on the probability of occurrence of coefficient level values.

The decoding unit 210 may be configured to obtain the control data by decoding the encoded data.

Here, the control data may include size data such as coding block (CU: Coding Unit) size, prediction block (PU: Prediction Unit) size, and transform block (TU: Transform Unit) size.

The inverse transform/inverse quantization unit 220 is configured to perform inverse transform processing of the coefficient level values output from the decoding unit 210. Here, the inverse transform/inverse quantization unit 220 may be configured to perform inverse quantization of the coefficient level values prior to the inverse transform processing.

The adder 230 is configured to add the prediction signal to the prediction residual signal output from the inverse transform/inverse quantization unit 220 to generate a pre-filter decoded signal, and output the pre-filter decoded signal to the intra prediction unit 242 and the in-loop filter processing unit 250.

Here, the unfiltered decoded signal constitutes a reference block used by the intra prediction unit 242.

The inter prediction unit 241 is configured to generate a prediction signal by inter prediction (inter-frame prediction).

Specifically, the inter prediction unit 241 is configured to generate a prediction signal for each prediction block based on a motion vector decoded from the encoded data and a reference signal included in a reference frame. The inter prediction unit 241 is configured to output the prediction signal to the adder 230.

The intra prediction unit 242 is configured to generate a prediction signal by intra prediction (intra-frame prediction).

Specifically, the intra prediction unit 242 is configured to identify a reference block included in the target frame and generate a prediction signal for each prediction block based on the identified reference block. The intra prediction unit 242 is configured to output the prediction signal to the adder 230.

The in-loop filter processing unit 250 is configured to perform filtering on the unfiltered decoded signal output from the adder 230, and to output the filtered decoded signal to the frame buffer 260.

Here, the in-loop filter process may be composed of multiple filter processes. For example, the filter process may be a deblocking filter process that reduces distortion occurring at the boundary between blocks (encoding blocks, prediction blocks, transformation blocks, or subblocks obtained by dividing them), or an adaptive loop filter process that switches filters based on filter coefficients, filter selection information, local properties of the image pattern, and the like transmitted from the image encoding device 100.

The frame buffer 260 is configured to store reference frames used by the inter prediction unit 241.

Here, the filtered decoded signal constitutes a reference frame used by the inter prediction unit 241.

(Inter prediction unit 241)
An example of the process of the inter prediction unit 241 of the image decoding unit 200 of the image decoding device 20 according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of functional blocks of the inter prediction unit 241 according to this embodiment.

As shown in FIG. 9, the inter prediction unit 241 includes a parameter decoding unit 2411, a resolution conversion unit 2412, and a predicted image generation unit 2413.

The parameter decoding unit 2411 is configured to decode parameters related to inter prediction from the bitstream. Here, the parameters include, for example, a motion vector and an index value indicating which reference frame is to be used.

The resolution conversion unit 2412 is configured to identify a reference frame based on the above-mentioned parameters, and convert the resolution of the reference frame when the resolution of the reference frame differs from the resolution of the frame to be decoded. As with the resolution conversion unit 1111, the resolution conversion unit 2412 can use the method defined in Non-Patent Document 1 as the resolution conversion method.

The predicted image generation unit 2413 is configured to generate a predicted image, similar to the predicted image generation unit 1113.

(Post-processing section 800)
In this embodiment, the post-processing unit 800 is configured to perform upsampling processing as necessary, and generate and output a post-processed decoded image.

Here, the post-processing unit 800 may, for example, determine that upsampling processing is necessary and perform upsampling processing if the resolution of the decoded image output from the image decoding unit 200 is smaller than the resolution of the image of the sequence to be decoded.

The image resolution of the sequence to be decoded may be included in the control data decoded by the image decoding unit 200. The image resolution of the sequence to be decoded may be input as control information from the UI unit 1000 via the control unit 900.

When it is determined that the upsampling process needs to be performed, the post-processing unit 800 performs the upsampling process. Here, the post-processing unit 800 may select a method for performing the upsampling process from among a plurality of methods for performing the upsampling process.

The post-processing unit 800 may have various types of upsampling processing prepared in advance, such as a one-dimensional convolution filter, a two-dimensional convolution filter, a machine learning-based method such as CNN, and an upsampling processing similar to the RPR described above.

In addition, the post-processing unit 800 may have multiple candidates for the same type of filter (e.g., a one-dimensional convolution filter) with variations in filter coefficients, etc.

In addition, the pre-processing unit 300 may determine the filter coefficients and network parameters (weights, etc.) of a machine learning-based method such as CNN, and the post-processing unit 800 may perform upsampling processing based on values (parameters determined by the pre-processing unit 300) input as control data via the bit stream.

The pre-processing unit 300 may determine which of multiple upsampling processes to use, and the post-processing unit 800 may determine which upsampling process to use based on a value input as control data via the bit stream (a value indicating the upsampling process determined by the pre-processing unit 300).

The UI unit 1000 may output setting information based on the user's settings, and the post-processing unit 800 may determine which of multiple upsampling processes to use based on the setting information.

The post-processing unit 800 may determine filter coefficients in the upsampling process and network parameters (weights, etc.) for machine learning-based methods such as CNN, based on the setting information output from the UI unit 1000 via the control unit 900.

(Variation 1)
Hereinafter, a first modification of the present invention will be described, focusing on differences from the first embodiment described above. In the first modification, a post-processing unit 800 is configured to perform post-filtering using the decoded image and control data output from the image decoding unit 200 as input.

In this modified example 1, the analysis unit 400 analyzes the area of the screen content (such as a game screen or a remote desktop screen) in the input image signal, and outputs area information of the screen content (such as the number of areas, the position and size of each area, etc.) to the control unit 500.

Here, the analysis unit 400 may output information indicating only whether the entire image is screen content or not.

The control unit 500 is configured to output the area information of the screen content output from the analysis unit 400 to the image encoding unit 100.

Alternatively, the control unit 500 is configured to output the area specified by the UI unit 600 to the image encoding unit 100 as screen content area information.

The image encoding unit 100 is configured to encode the above-mentioned screen content area information as part of the control data and generate a bitstream. At this time, the screen content area information may be encoded as a user-defined SEI message.

On the other hand, the image decoding unit 200 is configured to decode the screen content area information from the extracted bitstream described above and output it to the post-processing unit 800.

The post-processing unit 800 is configured to determine whether or not to perform post-filtering on each area of the decoded image signal based on the above-mentioned screen content area information, and to actually perform the post-filtering.

Here, the post-processing unit 800 may control the screen content area information so as not to perform post-filter processing on areas that are defined as screen content.

In the above example, a case has been described in which the image encoding device 10 transmits screen content area information to the image decoding device 20 and controls post-filter processing in the image decoding device 20 based on the screen content area information.

As another example, the image encoding device 10 may transmit information about the post-filter application area (or non-application area) and control whether or not to apply the post-filter according to the application area (or non-application area) information.

(Variation 2)
Hereinafter, the second modified example of the present invention will be described with reference to FIGS. 10 and 11, focusing on the differences from the first embodiment and the first modified example.

In this modified example 2, the image encoding device 10 and the image decoding device 20 are configured to switch between multi-channel and single-channel configurations.

Figure 10 shows an example of switching from a single-channel configuration to a multi-channel configuration, and Figure 11 shows an example of switching from a multi-channel configuration to a single-channel configuration.

Figure 10 shows an example in which, when switching from single-channel to multi-channel format, the input image signal is downsampled to reduce transmission capacity in order to broadcast content on a sub-channel other than the content broadcast during single-channel format.

In the example of FIG. 10, switching from single-channel configuration to multi-channel configuration is performed between Fr2 and Fr3. In the above-mentioned downsampling process, the UI unit 600 outputs control information indicating a switch to multi-channel configuration via the control unit 500, and the pre-processing unit 300 performs downsampling processing based on the control information.

In the example of FIG. 11, switching from multi-channel to single-channel configuration is performed between Fr2 and Fr3. In this case, as in the above example, the pre-processing unit 300 performs downsampling processing up to Fr2, but does not perform downsampling processing from Fr3 onwards.

The inter prediction unit 111 of the image encoding device 10 is configured to perform resolution conversion using RPR as described above when the resolution of the frame to be encoded and the resolution of the reference frame differ due to switching between multi-channel and single-channel organization.

Similarly, the inter prediction unit 241 of the image decoding device 20 is configured to perform resolution conversion using RPR as described above when the resolution of the frame to be decoded differs from the resolution of the reference frame.

Finally, the post-processing unit 800 is configured to perform an upsampling process. Here, the content of the upsampling process is the same as in the above example.

That is, in this modified example 2, the image encoding device 10 includes a pre-processing unit 300 configured to output a pre-processed input image obtained by performing a downsampling process on the input image signal in multi-channel formation, and an image encoding unit 100 configured to encode the input image signals before and after switching between single-channel formation and multi-channel formation as the same bit stream by using an RPR that converts the resolution of the reference frame when the resolution of the reference frame differs from the resolution of the pre-processed input image.

In addition, in this modified example 2, the image decoding device 20 includes an image decoding unit 200 configured to decode the bit stream and output a decoded image, and a post-processing unit 800 configured to perform an upsampling process on the decoded image downsampled in the image encoding device 10 in the case of multi-channel composition.

Here, the image decoding unit 200 may include an inter prediction unit 241 configured to perform an RPR process to convert the resolution of the reference frame when the resolution of the frame to be decoded differs from the resolution of the reference frame.

The above-mentioned upsampling process may be a one-dimensional or two-dimensional convolution filter, a method using a CNN, the same method as the RPR process, or may be performed based on control data decoded from the bitstream.

(Variation 3)
12 to 15, a third modification of the present invention will be described below, focusing on differences from the first embodiment and the first and second modifications. Specifically, as the third modification, an example of encoding and decoding processes in the case where the input image is an interlaced image will be described.

In this third variant, when the input image is an interlaced image, the even field becomes the Top field, the odd field becomes the Bottom field, and the encoding process is performed so that the Top field and the Bottom field immediately following it are paired, as shown in FIG. 12.

In the example of Figure 12, the field number starts from 0, i.e., Fi0.

Here, the analysis unit 400 may be configured to determine whether the target input image is an interlaced image and output the determination result to the control unit 500.

For example, the analysis unit 400 may define in advance possible resolutions and information on whether the resolutions correspond to an interlaced image or a progressive image, and determine whether the target input image is an interlaced image or a progressive image based on the resolution of the input image and this information.

Alternatively, the user may specify whether the target input image is an interlaced image via the UI unit 600. Here, the information (interlaced/progressive) specified by the user via the UI unit 600 is output to the control unit 500.

The control unit 500 is configured to receive the information output from the analysis unit 400 or the UI unit 600 as input, and when the target input image is an interlaced image, to set control data so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following it are paired, as shown in FIG. 12, and to output the control data to the image encoding unit 100.

The image encoding unit 100 is configured to encode such control data and output it as a bit stream.

Here, the image encoding unit 100 may encode such control data as a Frame-field information SEI message defined in Non-Patent Document 2.

Figure 13 shows an example of the setting values of each syntax element when such control data is encoded as a Frame-field information SEI message.

As another example, when the input image is an interlaced image, the image encoding unit 100 may synthesize two interlaced (field) images to generate one frame image and encode it, so that the even lines become the top field and the odd lines become the bottom field, as shown in Figure 14.

Here, in the example of FIG. 14, the first line is line 0. The process of generating a frame image from a field image may be performed, for example, by the preprocessing unit 300.

The control unit 500 may input to the pre-processing unit 300 control information on whether or not to generate a frame image from a field image based on the setting information input from the UI unit 600. FIG. 15 shows an example of the setting of the Frame-field information SEI message at this time.

In addition, when the input image is an interlaced image, the image encoding unit 100 may encode the control data by setting the value of sps_field_flag described in Non-Patent Document 1 to 1.

The above describes the processing in the image encoding device 10 according to the third modification. Below, we will explain the processing in the image decoding device 20.

The image decoding unit 200 is configured to decode the control data from the bitstream and obtain the above-mentioned sps_field_flag value and Frame-field information SEI message value.

Here, the image decoding unit 200 may be configured to output an error when the value of sps_field_flag is 1 and the value of sps_field_flag and the setting value of the Frame-field information SEI message are different from the setting values of FIG. 13 or FIG. 15.

In addition, even if the value of sps_field_flag is 1 and the value of sps_field_flag and the setting value of the Frame-field information SEI message are different from the setting values of FIG. 13 or FIG. 15, the image decoding unit 200 may be configured to perform a decoding operation for the setting values of FIG. 13 when the decoded image signal is a field image, and to perform a decoding operation for the setting values of FIG. 15 when the decoded image signal is a frame image.

Here, the image decoding unit 200 determines whether the decoded image signal is a field image based on the resolution of the decoded image signal contained in the control data or the setting information output from the UI unit 1000.

In other words, in this modified example 3, the image encoding device 10 includes an image encoding unit 100 that is configured to perform encoding processing when the input image is an interlaced image such that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

Here, when the input image is an interlaced image, the image encoding unit 100 may be configured to set the value of the Frame-field information SEI message so that the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired, and to encode the values as control data to generate a bitstream.

The image encoding device 10 may also include a UI unit 600 configured to set whether the input image is an interlaced image, and a control unit 500 configured to output control information that controls the encoding process based on the setting information output from the UI unit 600.

Here, the image encoding unit 100 may be configured to determine whether or not the input image is an interlaced image based on such control information.

In addition, in this modified example 3, the image decoding device 20 includes an image decoding unit 200 that is configured to perform a decoding process such that, when the value of sps_field_flag is 1 in the control data decoded from the bit stream, the even field becomes the Top field, the odd field becomes the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

Here, even if the image decoding unit 200 determines based on the control data decoded from the bit stream that the setting values of the Frame-field information SEI message are not set so that the even field is the Top field, the odd field is the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired, the image decoding unit 200 may be configured to perform a decoding process so that the even field is the Top field, the odd field is the Bottom field, and the Top field and the Bottom field immediately following the Top field are paired.

(Variation 4)
16 and 17, a fourth modification of the present invention will be described below, focusing on differences from the first embodiment and modifications 1 to 3. Specifically, as the fourth modification, the encoding process and the decoding process will be described for the case where spatial scalability is pseudo-realized using only the encoding technique defined in the H.266 Main profile specified in Non-Patent Document 1, using the above-mentioned RPR.

First, the process in the image encoding device 10 will be described. Specifically, in the fourth modification, the case where an input image is input as shown in FIG. 16 will be described as an example.

In this fourth modification, as shown in FIG. 17, images (Fr0' to Fr3') are generated by performing downsampling processing on each input frame (Fr0 to Fr3). Such downsampling processing may be performed, for example, by the pre-processing unit 300.

The UI unit 600 may be configured to output setting information indicating whether to implement the pseudo spatial scalability in this variant example 4 to the control unit 500, and the control unit 500 may be configured to output such setting information to the pre-processing unit 300 as control information.

The pre-processing unit 300 may perform downsampling processing based on such control information.

That is, the pre-processing unit 300 is configured to generate low-resolution images (Fr0' to Fr3') from input images (Fr0 to Fr3), as shown in FIG. 17.

The image encoding unit 100 is configured to perform encoding processing using as input low-resolution images (Fr0' to Fr3') generated from the input images (Fr0 to Fr3).

Here, the arrows pointing to each Fr/Fr' in Figure 17 show examples of reference relationships used in the inter prediction unit 111.

For example, in FIG. 17, Fr3' indicates that Fr0' is used as the reference frame. Similarly, in FIG. 17, Fr1 indicates that Fr0, Fr3, and Fr1' are used as the reference frames.

Here, as shown in FIG. 17, the low-resolution images (Fr0' to Fr3') use only the low-resolution images (Fr0' to Fr3') as reference frames.

On the other hand, the high-resolution images (Fr0 to Fr3) can use both high-resolution and low-resolution images as reference frames.

The high-resolution image uses as a reference frame the low-resolution image (e.g., Fr0' in the case of Fr0) obtained by performing downsampling processing on the target frame.

Note that the reference frame relationships shown in FIG. 17 are merely an example. As long as the following constraints are met: "For low-resolution images (Fr0' to Fr3'), only low-resolution images (Fr0' to Fr3') are used as reference frames" and "For high-resolution images, the low-resolution image obtained by performing a downsampling process on the target frame (e.g., in the case of Fr0, Fr0') is used as a reference frame," the inter prediction unit 111 can use reference relationships other than those shown in FIG. 17.

When the resolution of the reference frame and the resolution of the frame to be encoded differ, the image encoding unit 100 uses RPR as in the above example.

At this time, the image encoding unit 100 may set the value of TemporalId defined in Non-Patent Document 1 as control data to be as shown in Figure 17, and generate a bitstream.

That is, the image encoding unit 100 may encode the low-resolution image frame so that TemporalId = 0 and the high-resolution image frame so that TemporalId = 1.

Note that the setting value of TemporalId is merely an example, and it is sufficient that the TemporalId of the high-resolution image is set to be larger than the TemporalId of the low-resolution image.

In the above, an example has been described in which the high-resolution image uses the input image itself, but the image encoding unit 100 may also encode the error image between the input image and an image obtained by upsampling a low-resolution image as the high-resolution image.

Secondly, we will explain the processing in the image decoding device 20.

The extraction unit 700 is configured to extract only the data that is actually to be decoded from the bitstream.

For example, the value of the TemporalID described above may be used for such extraction. That is, the extraction unit 700 may be configured to obtain the maximum value of the TemporalId to be decoded from the control unit 900, and output only the bitstream of frames having a TemporalId equal to or less than the maximum value to the image decoding unit 200.

For example, if a low-resolution image frame is coded so that TemporalId=0 and a high-resolution image frame is coded so that TemporalId=1, the extraction unit 700 will extract a bitstream related only to the low-resolution image if the above-mentioned maximum value is set to 0, and will extract a bitstream including both the low-resolution image and the high-resolution image if the maximum value is set to 1.

Here, the extraction unit 700 may determine the maximum value based on setting information from the UI unit 1000. In addition, the maximum value may be set in advance for each image decoding device 20.

When high-resolution images are also decoded, the inter prediction unit 241 uses RPR if the resolution of the reference frame differs from the resolution of the frame to be decoded.

When the high-resolution image is also decoded, the post-processing unit 800 is configured to discard the decoded image of the low-resolution image and output only the decoded image of the high-resolution image.

In addition, when an error image between an upsampled low-resolution image and the input image is used as the high-resolution image, the post-processing unit 800 upsamples the low-resolution image and adds it to a decoded image of the high-resolution error image to generate a final decoded image of the high-resolution image.

Then, as described above, the post-processing unit 800 discards the low-resolution image and outputs only the decoded image of the high-resolution image.

The upsampling method may be performed based on a method or parameters specified by the image encoding device 10, as in the above example, or may be performed using a predetermined upsampling method.

When the image encoding device 10 specifies an upsampling method, the image encoding device 10 may specify that the image encoding device 10 and the image decoding device 20 perform the same upsampling processing.

If the high-resolution image is not decoded, the post-processing unit 800 may be configured to output the decoded image of the low-resolution image output from the image decoding unit 200 as is.

That is, in the image encoding device 10 according to this modified example 4, the image encoding unit 100 is configured to generate a low-resolution image by performing a downsampling process on a high-resolution image, and is configured to perform inter-prediction by referring to only the low-resolution image, and is configured to perform inter-prediction for the high-resolution image by referring to the low-resolution image at the same time and using RPR processing.

Here, the image encoding unit 100 may be configured to set the TemporalID of the high-resolution image to a value greater than the TemporalId of the low-resolution image and perform encoding.

In addition, in the image decoding device 20 according to this modified example 4, the image decoding unit 200 is configured to decode a high-resolution image and a low-resolution image of the same time as different frames, and to perform inter prediction using RPR processing when the resolution of the frame to be decoded differs from the resolution of the reference frame.

Here, the image decoding device 20 may include a control unit 900 configured to control whether to decode only the low-resolution image or to decode both the low-resolution image and the high-resolution image based on the TemporalID.

Furthermore, the image decoding device 20 may include a post-processing unit 800 configured to, when both a low-resolution image and a high-resolution image are decoded, discard the low-resolution image and output only the high-resolution image.

Furthermore, the above-mentioned image encoding device 10 and image decoding device 20 may be realized as a program that causes a computer to execute each function (each process).

In the above embodiments and modified examples, the present invention has been described using examples in which the present invention is applied to the image encoding device 10 and the image decoding device 20. However, the present invention is not limited to such examples, and can be similarly applied to an image encoding/decoding system having the functions of the image encoding device 10 and the image decoding device 20.

In addition, according to this embodiment, for example, it is possible to realize an improvement in the overall service quality in video communication, which makes it possible to contribute to Goal 9 of the Sustainable Development Goals (SDGs) led by the United Nations, which is to "build resilient infrastructure, promote sustainable industrialization and foster innovation."

1...Image processing system 10...Image encoding device 100...Image encoding unit 111...Inter prediction unit 1111...Resolution conversion unit 1112...Parameter determination unit 1113...Prediction image generation unit 112...Intra prediction unit 121...Subtractor 122...Adder 131...Transformation and quantization unit 132...Inverse transformation and inverse quantization unit 140...Encoding unit 300...Preprocessing unit 400...Analysis unit 500...Control unit 600...UI unit 20...Image decoding device 200...Image decoding unit 210...Decoding unit 220...Inverse transformation and inverse quantization unit 230...Adder 241...Inter prediction unit 2411...Parameter decoding unit 2412...Resolution conversion unit 2413...Prediction image generation unit 242...Intra prediction unit 250...In-loop filter processing unit 260...Frame buffer 700...Extraction unit 800...Post-processing unit 900...Control unit 1000...UI unit

Claims (4)

An image encoding device, comprising:
An image encoding unit is provided,
The image encoding unit includes:
A low-resolution image is generated by performing a downsampling process on the high-resolution image,
configured to perform inter prediction by referring only to the low resolution image,
11. An image coding device, comprising: an image coding apparatus configured to perform inter prediction for a high-resolution image by referring to the low-resolution image at the same time and using a Reference Picture Resampling (RPR) process. The image encoding device according to claim 1, characterized in that the image encoding unit is configured to set the TemporalID of the high-resolution image to a value greater than the TemporalId of the low-resolution image and perform encoding. An image processing system including an image encoding device and an image decoding device,
The image encoding device comprises:
An image encoding unit is provided,
The image encoding unit includes:
A low-resolution image is generated by performing a downsampling process on the high-resolution image,
configured to perform inter prediction by referring only to the low resolution image,
The high-resolution image is configured to refer to the low-resolution image at the same time and to perform inter prediction using a reference picture resampling (RPR) process,
The image decoding device comprises:
An image decoding unit is provided,
The image decoding unit
Decoding the high-resolution image and the low-resolution image at the same time as different frames;
1. An image processing system characterized in that, when a resolution of a frame to be decoded differs from a resolution of a reference frame, the image processing system is configured to perform inter prediction using RPR (Reference Picture Resampling) processing. A program for causing a computer to function as an image encoding device, comprising:
The image encoding device comprises:
An image encoding unit is provided,
The image encoding unit includes:
A low-resolution image is generated by performing a downsampling process on the high-resolution image,
configured to perform inter prediction by referring only to the low resolution image,
A program configured to perform inter prediction for the high-resolution image by referring to the low-resolution image at the same time and using Reference Picture Resampling (RPR) processing.

Images