JP6915473B2 - Coding devices and programs, decoding devices and programs, and image processing systems - Google Patents

Description

The present invention relates to coding devices and programs, decoding devices and programs, and image processing systems, and is applicable to, for example, a system for compressing video.

In recent years, surveillance cameras have become widespread, and higher resolution, higher frame rates, and multiple viewpoints are also desired. However, higher resolution, higher frame rate, and higher multi-viewpoint cause a significant increase in the code amount of the moving image, resulting in an increase in communication cost and storage cost.

Therefore, in order to alleviate this problem, a method has been proposed in which a region of interest is detected from a moving image and a large number of bits are allocated to the region of interest. The area of interest is, for example, a face area. In the following, the region of interest will be referred to as "ROI", which is an abbreviation for Region of Interest. Further, a method of allocating a large number of bits to a region of interest is called "ROI coding".

In Patent Document 1, the parameters given to the encoder are controlled for each block by utilizing the function of the encoder in which the parameters for controlling the code amount, the image, and the quality are given for each block, which is a region other than ROI. We are proposing a system configuration that reduces the amount of information in the area of interest less than the amount of information in the ROI. Hereinafter, in this specification, the non-attention area will be referred to as a “background area”. Further, the background area is not necessarily in a stationary state.

If the amount of information is reduced by using the function provided by the encoder in this way, the amount of code can be reduced efficiently and surely. H. 264 / MPEG-4 AVC (Advanced Video Coding: hereinafter also referred to as "AVC") and H.A. Since major standards such as 265 / MPEG-H HEVC (High Efficiency Video Coding: hereinafter also referred to as "HEVC") have a function of controlling quality for each region, it is not possible to use this function. It can be said that this is a typical method of allocating a large number of bits to ROI.

On the other hand, for example, when it is unavoidable to use an encoder that does not have a function of giving video quality control parameters for each block due to various reasons such as cost and compatibility, the above-mentioned method cannot be adopted. Even in such a case, the technique described in Patent Document 2 has been proposed as a technique capable of controlling the image quality for each region without depending on the encoder.

In Patent Document 2, in order to reduce the amount of information in the background region without depending on the encoder, the background region is preprocessed by a low-pass filter to remove information on high-frequency components, thereby suppressing the amount of information in the background region. It is a suggestion to do.

FIG. 15 is a diagram showing a configuration example when a conventional image coding device represented by Patent Document 2 is incorporated as a part of an image processing system.

In FIG. 15, the image processing system Z is input to the image encoding device X1 that compresses the pixels belonging to the background region in the input input image with a number of bits smaller than the ROI information and outputs it as a bit stream. It is composed of an image decoding device X2 that decodes the bit stream and outputs an output image.

The image encoding device X1 applies the ROI setting unit 402 that adds ROI to the input image and outputs the ROI-applied image, and performs preprocessing for reducing the amount of information in the background area on the ROI-applied image, and performs the pre-encoding image. It has a preprocessing unit U1 that outputs as, and an encoder unit 406 that compresses an unencoded image using an image coding method such as AVC and outputs a bit stream. Here, the pretreatment includes, for example, a process of suppressing a high frequency component by convolving a Gaussian filter.

The ROI setting unit 402 is a function of giving an ROI to an input image, but is not specified as a means for specifying the ROI. For example, as a means for identifying ROI, the position and size of ROI can be detected from the input image by applying a face detection algorithm, a person detection algorithm, a license plate detection algorithm, a vehicle body detection algorithm, or the like to the input image. A method of specifying with is conceivable. Further, as the ROI auxiliary information, for example, a method of specifying by the position and coordinates of the ROI manually input may be considered, or the ROI may be specified by the position and coordinates of the ROI input via a user interface (not shown). Further, as the ROI auxiliary information, for example, a method of specifying the ROI by using an infrared camera image or a depth sensor image corresponding to the input image may be used.

The ROI setting unit 402 may output information including the coordinates and size of the specified ROI as ROI information. The output ROI information may be multiplexed in the bitstream in the form of an additional information packet, for example, or may be embedded in the bitstream by embedding the information in the image. In FIG. 15, for example, the encoder unit 406 plays a role of inserting ROI information into the bit stream by the method described above.

The image decoding device X2 has a decoder unit 407 that decodes the input bitstream data by a method corresponding to the encoder unit 406 and outputs it as a decoded image (output image).

Note that FIG. 15 shows an example in which the image coding device X1 reduces the amount of information in the background region from the amount of information in the ROI by preprocessing, but the region in which the encoder includes the encoder unit 406 of the image coding device X1. By allocating a large number of bits to the ROI by utilizing the function of controlling the quality for each, it becomes a configuration example incorporating the video coding system represented by Patent Document 1.

In this case, the preprocessing unit U1 may be considered to be in charge of the function of controlling the quality parameter.

Japanese Unexamined Patent Publication No. 2009-49979 Japanese Unexamined Patent Publication No. 04-219089

J. Ascenso, C.I. Brites, and F. Pereira. Motion compensated refined for low complexity pixel based distributed video coding. In Proceedings. IEEE Conference on Advanced Video and Signal Based Surveillance, 2005. , Pages 593-598. IEEE, 2005 Yuri Vatis, Sven Klom, and John Ostermann. Enhanced Reconstruction of the Quantified Transfer Coefficients for WYNER-ZIV Coding. In Multimedia and Expo, 2007 IEEE International Convention on, number Figure 2, pages 172-175. IEEE, ju1 2007. Denis Kubasov, Janth Nayak and Christine Gui11emot. Optima1 Reconstruction in Winer-Ziv Video Coding with Mu1tip1e Side Information. In 2007 IEEE 9th Workshop on Mu1timedia Signa1 Processing, pages183-186, IEEE, 2007.

As described above, it is possible to allocate a large number of bits to the ROI by using the preprocessing unit U1.

However, in principle, deterioration of the image quality of the entire background area is unavoidable.

Therefore, a coding device and a program, a decoding device and a program, and an image processing system capable of alleviating the deterioration of the image quality of the entire background area while maintaining the feature of ROI coding that the transmission cost and the storage cost of the video can be reduced are desired. It is rare.

According to the first aspect of the present invention, in a coding device that encodes an input image, (1) a region of interest setting unit that assigns a region of interest to the input image and outputs the image as an image of the region of interest, and (2) the region of interest. Among the added images, a preprocessing unit that applies preprocessing for reducing the amount of information in the background area to the area of interest added image and outputs it as a pre-encoded image, and (3) a predetermined image coding method. It has a coding unit that compresses the pre-encoded image and outputs the compressed coded data, and the pre-processing unit (4) indicates whether the input image is a key picture or a non-key picture. The pre-processed picture type control unit that determines the pre-processed picture type and outputs the determined pre-processed picture type, and (5) if the pre-processed picture type is a key picture, the attention area imparting image is not displayed. In the case of a key picture, the dynamic range compression unit that outputs a filtered image that refers to the key picture to the image with the region of interest as the pre-encoded image, and (6) for the dynamic range compression unit. (7) The dynamic range compression unit outputs the attention region-imparted image determined to be a key picture as a preprocessing storage picture to the preprocessing buffer. The key picture information is extracted from the preprocessing buffer as a preprocessing reference picture in order to filter the image with the attention area that has been accumulated and determined to be a non-key picture using the key picture information. It is a feature.

In the second compound unit of the present invention, (1) a region of interest was added to the input image, and preprocessing was performed to reduce the amount of information in the background region of the input image to which the region of interest was assigned. A decoding unit that decodes the coded data that has been encoded with respect to the unencoded image and outputs the decoded image, and (2) a post-processed picture that indicates whether the decoded image is a key picture or a non-key picture. A post-processing picture type control unit that determines the type and outputs the determined post-processing picture type, and (3) if the post-processing picture type is a key picture, the decoded image is a non-key picture. In some cases, it has a dynamic range expansion unit that outputs a filtered image that refers to a key picture to the decoded image as an output image, and (4) a post-processing buffer that is a buffer for the dynamic range expansion unit. (5) The dynamic range extension unit outputs the decoded image determined to be a key picture as a post-processing storage picture, accumulates it in the post-processing buffer, and determines that the decrypted image is a non-key picture. In order to filter the image using the key picture information, the key picture information is extracted from the post-processing buffer as a post-processing reference picture.

The third image processing system of the present invention is a coding device that encodes an input image to generate coded data, and a decoding device that decodes the coded data generated by the coding device to generate a decoded image. In the image processing system including the above, the coding device of the first invention is applied as the coding device, and the decoding device of the second invention is applied as the decoding device.

The fourth coding program of the present invention sets a region of interest in which a computer mounted on a coding device that encodes an input image (1) assigns a region of interest to the input image and outputs the image as a region of interest. (2) A preprocessing unit that applies preprocessing for reducing the amount of information in the background area of the attention area imparted image to the attention area imparted image and outputs the image as a pre-encoded image. 3) Using a predetermined image coding method, the pre-encoded image is compressed and functioned as a coding unit that outputs the compressed coded data. In the pre-processing unit, (4) the input image is the key. A pre-processed picture type control unit that determines a pre-processed picture type indicating whether it is a picture or a non-key picture and outputs the determined pre-processed picture type, and (5) when the pre-processed picture type is a key picture. A dynamic range compression unit that outputs a filtered image that refers to a key picture to the attention area imparted image when the attention area imparted image is a non-key picture as a pre-encoded image, and ( 6) It has a preprocessing buffer which is a buffer for the dynamic range compression unit , and (7) the dynamic range compression unit outputs the attention area imparting image determined to be a key picture as a storage picture for preprocessing. The key picture information is used as a preprocessing reference picture in order to filter the attention region-imparted image which is accumulated in the preprocessing buffer and is determined to be a non-key picture by using the key picture information. It is characterized by extracting from the preprocessing buffer.

In the fifth compound program of the present invention, the computer is subjected to (1) preprocessing for reducing the amount of information in the background area of the input image to which the attention area is given to the input image. A decoding unit that decodes the coded data subjected to the coding process and outputs the decoded image with respect to the performed unencoded image, and (2) indicates whether the decoded image is a key picture or a non-key picture. A post-processing picture type control unit that determines the post-processing picture type and outputs the determined post-processing picture type, and (3) if the post-processing picture type is a key picture, the decoded image is non-keyed. In the case of a picture, the decoded image is filtered to refer to the key picture, and the image is output as an output image as a dynamic range expansion unit and (4) as a post-processing buffer which is a buffer for the dynamic range expansion unit. (5) The dynamic range extension unit is determined to be a non-key picture by outputting the decoded image determined to be a key picture as a post-processing storage picture and accumulating it in the post-processing buffer. In order to filter the decoded image using the key picture information, the key picture information is extracted from the post-processing buffer as a post-processing reference picture.

In the sixth aspect of the present invention, in a coding device that encodes an input image, (1) a region of interest setting unit that assigns a region of interest to the input image and outputs the image as an image of the region of interest, and (2) the region of interest. Among the added images, a preprocessing unit that applies preprocessing for reducing the amount of information in the background area to the area of interest added image and outputs it as a pre-encoded image, and (3) a predetermined image coding method. It has a coding unit that compresses the pre-encoded image and outputs the compressed coded data, and the pre-processing unit (4) indicates whether the input image is a key picture or a non-key picture. The pre-processed picture type control unit that determines the pre-processed picture type and outputs the determined pre-processed picture type, and (5) if the pre-processed picture type is a key picture, the attention area imparting image is not displayed. In the case of a key picture, it is provided with a dynamic range compression unit that outputs a filtered image that refers to a virtual key picture pixel value with respect to the attention area imparted image as the pre-encoded image. It is a feature.

In the seventh compound unit of the present invention, (1) a region of interest was added to the input image, and preprocessing was performed to reduce the amount of information in the background region of the input image to which the region of interest was assigned. A decoding unit that decodes the coded data that has been encoded with respect to the unencoded image and outputs the decoded image, and (2) a post-processed picture that indicates whether the decoded image is a key picture or a non-key picture. A post-processing picture type control unit that determines the type and outputs the determined post-processing picture type, and (3) if the post-processing picture type is a key picture, the decoded image is a non-key picture. In the case, a dynamic range extension unit that outputs a filtered image that refers to the pixel value of a virtual key picture to the decoded image as a pre-reconstruction image, and (4) the post-processing picture type are key pictures. In the case of (5) It has a post-processing buffer which is a buffer for the reconstruction unit, and (6) the reconstruction unit outputs the pre-reconstruction image determined to be a key picture as a post-processing storage picture. Therefore, in order to reconstruct the pre-reconstruction image that has been accumulated in the post-processing buffer and is determined to be a non-key picture by using the key picture information, the key picture information is used as a post-processing reference picture. It is characterized by extracting from the post-processing buffer.

The eighth image processing system of the present invention is a coding device that encodes an input image to generate coded data, and a decoding device that decodes the coded data generated by the coding device to generate a decoded image. In the image processing system including the above, the coding device of the sixth invention is applied as the coding device, and the decoding device of the seventh invention is applied as the decoding device.

The ninth coding program of the present invention sets a region of interest in which a computer mounted on a coding device that encodes an input image (1) assigns a region of interest to the input image and outputs the image as a region of interest. (2) A preprocessing unit that applies preprocessing for reducing the amount of information in the background area of the attention area imparted image to the attention area imparted image and outputs it as a pre-encoded image. 3) Using a predetermined image coding method, the pre-encoded image is compressed and functioned as a coding unit that outputs the compressed coded data. In the pre-processing unit, (4) the input image is the key. A pre-processed picture type control unit that determines a pre-processed picture type indicating whether it is a picture or a non-key picture and outputs the determined pre-processed picture type, and (5) when the pre-processed picture type is a key picture. When the attention area-imparted image is a non-key picture, a dynamic image obtained by filtering the attention area-imparted image by referring to the pixel value of a virtual key picture is output as the pre-encoded image. It is characterized by having a range compression unit.

In the tenth decoding program of the present invention, the computer is subjected to (1) preprocessing for reducing the amount of information in the background area of the input image to which the attention area is given to the input image. A decoding unit that decodes the coded data subjected to the coding process and outputs the decoded image with respect to the unencoded image, and (2) after showing whether the decoded image is a key picture or a non-key picture. A post-processing picture type control unit that determines the processing picture type and outputs the determined post-processing picture type, and (3) if the post-processing picture type is a key picture, the decoded image is a non-key picture. If this is the case, the dynamic range extension unit that outputs a filtered image that refers to the pixel value of the virtual key picture to the decoded image as a pre-reconstruction image, and (4) the post-processed picture type are If it is a key picture, the image before reconstruction is output. If it is a non-key picture, the image before reconstruction using the key picture is output as an output image. And (5) function as a post-processing buffer which is a buffer for the reconstruction unit, and (6) the reconstruction unit outputs the pre-reconstruction image determined to be a key picture as a storage picture for post-processing. In order to reconstruct the pre-reconstruction image that is accumulated in the post-processing buffer and is determined to be a non-key picture by using the key picture information, the key picture information is used as a post-processing reference picture. It is characterized by extracting from the post-processing buffer.

According to the eleventh invention, in a coding device that encodes an input image, (1) a region of interest setting unit that assigns a region of interest to the input image and outputs the image as an image of the region of interest, and (2) the region of interest. Among the added images, a preprocessing unit that applies preprocessing for reducing the amount of information in the background area to the area of interest added image and outputs it as a pre-encoded image, and (3) a predetermined image coding method. The pre-encoded image is compressed based on the coding quality parameter that induces coarse quantization in the background region and fine quantization in the region of interest, and the compressed encoded data is output. The preprocessing unit has an encoding unit, and the preprocessing unit (4) determines a preprocessing picture type indicating whether the input image is a key picture or a non-key picture, and outputs the determined preprocessing picture type. The picture type control unit and (5) a virtual key picture for the attention area imparted image when the preprocessed picture type is a key picture, and for the attention area imparted image when the preprocessed picture type is a non-key picture. A dynamic range compression unit that outputs a filtered image that refers to the pixel value of It is characterized by having a quality parameter control unit that outputs the pre-encoded image after assigning the coding quality parameter based on the area of interest information indicating the size.

The twelfth image processing system of the present invention is a coding device that encodes an input image to generate coded data, and a decoding device that decodes the coded data generated by the coding device to generate a decoded image. The image processing system including the above is characterized in that the eleventh coding device of the present invention is applied as the coding device and the seventh decoding device of the present invention is applied as the decoding device.

According to the present invention, it is possible to alleviate the deterioration of the image quality of the entire background region while maintaining the feature of ROI coding that the transmission cost and the storage cost of the video can be reduced.

It is a block diagram which shows the structure of the image processing system which concerns on 1st Embodiment. It is a flowchart which shows the operation of the image processing system which concerns on 1st Embodiment. It is a figure which shows the correction example of the specified ROI which concerns on 1st Embodiment. It is a flowchart (the 1) which shows an example of the filtering performed on the non-key picture which concerns on 1st Embodiment. It is a flowchart (No. 2) which shows an example of filtering performed on the non-key picture which concerns on 1st Embodiment. It is a flowchart (No. 3) which shows an example of filtering performed on the non-key picture which concerns on 1st Embodiment. It is a block diagram which shows the structure of the image processing system which concerns on 2nd Embodiment. It is a flowchart which shows the operation of the image processing system which concerns on 2nd Embodiment. It is a flowchart (the 1) which shows the reconstruction process of the reconstruction part which concerns on 2nd Embodiment. It is a flowchart (the 2) which shows the reconstruction process of the reconstruction part which concerns on 2nd Embodiment. It is a flowchart (the 3) which shows the reconstruction process of the reconstruction part which concerns on 2nd Embodiment. It is a figure which shows an example of the look-up table which concerns on 2nd Embodiment. It is a block diagram which shows the structure of the image processing system which concerns on 3rd Embodiment. It is a flowchart which shows the operation of the image processing system which concerns on 3rd Embodiment. It is a block diagram which shows the configuration example when the conventional image coding apparatus represented by Patent Document 2 is incorporated as a part of an image processing system.

(A) First Embodiment Hereinafter, the coding device and program, the decoding device and program according to the present invention, and the first embodiment of the image processing system will be described in detail with reference to the drawings. Hereinafter, an example in which the coding device of the present invention is applied to an image coding device and the decoding device of the present invention is applied to an image decoding device will be described.

(A-1) Configuration of First Embodiment FIG. 1 is a block diagram showing a configuration of an image processing system 1 according to the first embodiment. In FIG. 1, the solid line arrow indicates the flow of essential processing (and information) in this embodiment, while the broken line arrow does not necessarily indicate the essential processing (and information) in order to achieve the effect of this embodiment. ) Is not. Hereinafter, the processing (and information) related to the broken line arrow will be supplementarily described as appropriate.

The image processing system 1 encodes an input image and outputs a stream (bit string), and decodes a stream (bit stream) encoded by the image coding device 10 and outputs a decoded image. It has an image decoding device 20.

The medium for inputting the stream output from the image coding device 10 to the image decoding device 20 is not limited. For example, the stream output from the image coding device 10 may be transmitted to the image decoding device 20 by communication (for example, the Internet or the like), or the data of the stream output from the image coding device 10 may be used as data. It may be recorded on a recording medium (for example, a medium such as a DVD or an HDD) and input to the image decoding device 20 offline.

In this embodiment, the image coding device 10 will be described as performing coding for each input image, but by causing the image coding device 10 to process a plurality of input images in succession, the moving image can be coded. It may be applied. Similarly, the image decoding device 20 may be applied to the decoding process of a moving image by continuously decoding a plurality of encoded data streams.

Next, the internal configuration of the image coding device 10 will be described. FIG. 1 is also a block diagram showing a functional configuration of the image coding device 10 according to the first embodiment. The image coding device may be configured by hardware (for example, a dedicated semiconductor chip or the like), or may be partially or wholly configured by software.

The image coding device 10 has an ROI setting unit 402, a preprocessing unit 200, and an encoder unit 406. The functions of the ROI setting unit 402 and the encoder unit 406 have the same functions as those described with reference to FIG.

The pre-processing unit 200 determines and outputs a "pre-processing picture type" indicating either a "key picture" or a "non-key picture" for an input image, and a pre-processing picture type control unit 101 and a pre-processing picture type. refers to the, the ROI-given image when a key picture, the image subjected to filtering that references the key picture to ROI-given image when a non-key picture, DRC output as coded previous picture (Dynamic It has a Range Compression) filter unit 103 and a preprocessing buffer 104 which is a buffer for the DRC filter unit 103.

Further, the DRC filter unit 103 outputs the ROI-added image determined to be a key picture as a preprocessing storage picture, accumulates it in the preprocessing buffer 104, and stores the ROI-added image determined to be a non-key picture as a key. The key picture is also taken out from the preprocessing buffer 104 as a preprocessing reference picture in order to filter using the picture information.

Various methods can be applied to how the pre-processing picture type control unit 101 determines the picture type. A specific method example will be described later, but for example, there is also a method of determining based on an input image or an ROI-imparted image. Therefore, in FIG. Shown.

The ROI setting unit 402 can also add operating conditions in addition to the functions described with reference to FIG. A specific method example will be described later, but the operating conditions are determined according to, for example, the pre-processed picture type (input from the pre-processed picture type control unit 101 as shown by the dotted line in FIG. 1).

In addition to the functions described in FIG. 15 above, the encoder unit 406 is embedded in an additional information packet or an image in the same manner as in the case of transmitting ROI information, so that the encoder unit 406 is a preprocessed picture type (as shown by a dotted line in FIG. 1). , Input from the pre-processed picture type control unit 101) may be inserted into the bit stream. Further, the picture type determination method needs to be shared with the post-processing picture type control unit 108, which will be described later, by manual setting or transmission, but the picture type determination method can be converted into a bitstream by embedding in an additional information packet or an image. You may insert it.

Next, the internal configuration of the image decoding device 20 will be described. FIG. 1 is also a block diagram showing a functional configuration of the image decoding device 20 according to the first embodiment. The image coding device may be configured by hardware (for example, a dedicated semiconductor chip or the like), or may be partially or wholly configured by software.

The image decoding device 20 has a decoder unit 407 and a post-processing unit 300 for inputting the decoded image and outputting the output image.

The post-processing unit 300 refers to the post-processing picture type control unit 108 that determines and outputs the post-processing picture type and the post-processing picture type, and if it indicates that it is a key picture, the decoded image is non-keyed. An IDRC (Inverse Dynamic Range Compression) filter unit 109 and a buffer for the IDRC filter unit 109 that output a filtered image that refers to a key picture to the decoded image as an output image when it indicates that it is a picture. It has a post-processing buffer 110 which is.

Further, the IDRC filter unit 109 outputs the decoded image determined to be a key picture as a post-processing storage picture, stores the decoded image in the post-processing buffer 110, and outputs the decoded image determined to be a non-key picture to the key picture. The key picture is also taken out from the post-processing buffer 110 as a post-processing reference picture in order to filter using the information.

Various methods can be applied to how the post-processing picture type control unit 108 determines the picture type. A specific method example will be described later, but for example, since there is also a method of determining based on the information embedded in the bit stream or the decoded image, in FIG. 1, there is also an example in which the picture type auxiliary information is input to the post-processing picture type control unit 108. It is shown by the dotted line. The picture type auxiliary information is, for example, the picture type itself, the decoded image, the method of determining the picture type, and the like.

(A-2) Operation of First Embodiment Next, the operation of the image processing system 1 according to the first embodiment will be described in detail with reference to the drawings. FIG. 2 is a flowchart showing the operation of the image processing system according to the first embodiment. Hereinafter, the operations of the image coding device 10 and the image decoding device 20 will be described separately.

(A-2-1) Operation of Image Coding Device 10 FIG. 2 (A) is a flowchart showing the operation of the image coding device according to the first embodiment.

First, the pre-processed picture type control unit 101 determines the picture type of the input image and outputs it as the pre-processed picture type (S11). There are many possible methods for determining the picture type. For example, the following methods (1) to (3) can be considered.

(1) Method of periodically determining based on a predetermined key picture interval For example, when the key picture interval is set to 2, the preprocessing picture type control unit 101 sets a key picture and a non-key picture for each image. Switch. Specifically, when the T-th sheet is a key picture, the (T + 1) th sheet is a non-key picture, and the (T + 2) sheet is a key picture again. Similarly, when the key picture interval is set to 3 and the Tth sheet is a key picture, the preprocessing picture type control unit 101 sets the (T + 1) and (T + 2) sheets as non-key pictures. T + 3) Make the third sheet a key picture again.

(2) Method of determining according to the property of the input image The preprocessing picture type control unit 101 analyzes the property of the image, for example, whether or not there is movement, by using an existing computer vision method, for example, of the input image. When it is determined that there is no moving area inside, the input image is used as a key picture, and the other input images are used as non-key pictures.

More specifically, for example, the motion vector is estimated for each pixel or for each block, which is a set of one or more pixels, or for the entire image, using the input image to be processed and the input image input immediately before it. However, when there is no motion vector having a size larger than a predetermined threshold, or when the average motion vector of one or more motion vectors is calculated and the size of the average motion vector is smaller than the predetermined threshold. , The input image to be processed is used as a key picture.

In addition, the picture type determination results for the past N input images are traced back, and only when there is no input image determined to be a key picture among them, the input image determined to have no movement in the entire image is used as a key picture. It is also effective to make other input images non-key pictures.

(3) Method of determining according to the presence or absence of ROI For example, the preprocessing picture type control unit 101 determines that the ROI is a key picture when the ROI is not present in the image to which the ROI is attached, and determines that the ROI is a non-key picture when the ROI is present. do. This can be said to be a judgment under the assumption that there will be no movement in the entire image when the ROI does not exist (in other words, it can be said to be a special case of the method (2) described above).

Similar to the method (2) described above, the picture type determination results for the past N input images are traced back, and only when there is no input image determined to be a key picture in the input image, the ROI does not exist. It is also effective to use as a key picture and other input images as non-key pictures. Further, a method of combining the above-mentioned methods (1) to (3) can be considered.

The determined picture type may be transmitted to the image decoding device 20 side via the encoder unit 406. Further, the method of determining the picture type may be transmitted to the image decoding device 20 side via the encoder unit 406, or may be shared with the post-processing picture type control unit 108 by manual setting.

Next, the ROI setting unit 402 assigns the ROI to the input image and outputs the ROI-added image to the DRC filter unit 103 (S12).

Here, the ROI setting unit 402 may give the ROI to all the input images, or may give the ROI only when, for example, the preprocessed picture type indicates that it is a non-key picture. .. By limiting the pictures to which the ROI is given to only non-key pictures, the effect of reducing the amount of calculation can be obtained. This is because, in this embodiment, filtering is applied to the background area of the non-key picture, and therefore, if limited to this embodiment, the reason for specifying the ROI is only the non-key picture. Since a certain amount of calculation is required to identify the ROI and add information, the amount of calculation can be reduced by stopping the operation of the unnecessary ROI setting unit 402.

Further, the specified ROI may be, for example, one that has undergone any of the following corrections (1) to (3). (1) If even one pixel belonging to a block having a predetermined size is included in the ROI, all the pixels included in the block are considered to belong to the ROI. (2) If more than half of the pixels belonging to a block of a predetermined size are included in the ROI, all the pixels included in the block are considered to belong to the ROI. (3) Only when all the pixels belonging to a block having a predetermined size are included in the ROI, the pixels included in the block are considered to belong to the ROI.

FIG. 3 is a diagram showing a correction example of the specified ROI according to the first embodiment. FIG. 3 is an image of the case of correction by the method (2) in the above-mentioned method example. In FIG. 3, the size of each block (B1 to B3) is 4 pixels in width and 4 pixels in height. For example, since the block B3 has 9 pixels included in the ROI (that is, more than half of the pixels), the pixels included in the block belong to the ROI. In many image coding methods that can be adopted in the encoder unit 406, the boundary between the ROI and the background area is also image-coded in order to reduce the amount of information by predicting or converting each block of a specific size. It is desirable that it matches the boundaries of the blocks adopted by the method. This correction is a process for matching the boundary between the ROI and the background area with the boundary of the block adopted by the image coding method.

Next, the DRC filter unit 103 refers to the pre-processed picture type, and if the ROI-imparted image is a key picture, outputs the ROI-imparted image as the pre-processing reference picture and the pre-encoded image (S13). On the other hand, if the ROI-assigned image is a non-key picture, the DRC filter unit 103 takes out the pre-processing reference picture from the pre-processing buffer 104, filters the pixels classified in the background area of the ROI-assigned image, and adds a code. Output as an image before conversion.

FIG. 4A is a flowchart showing an example of filtering performed by the DRC filter unit 103 on the non-key picture.

In FIG. 4, p _i is the i-th pixel value of the ROI-imparted image, and r _i is the i-th pixel value of the key picture. Also, R _e is 0.0 to 1.0 parameters corresponding to the compression rate of the dynamic range by a factor for manipulating the reduction of information amount. p ₀ is the i-th pixel value of the unencoded image, and x ₁ , x ₂ , and x ₃ are intermediate values.

For example, the image processing shown in the flowchart of FIG. 4A is applied to the pixels belonging to the background region. In this method, for the pixels belonging to the ROI, and _{p 0} without processing the _{p i.} The dynamic range of the pixel value _{p i} in the background area, by compressing around the pixel values _{r i} key picture and reduce the amount of information (S101 to S103). In step S103, an example of rounding when compressing the dynamic range is shown, but since the effect of the present embodiment is not obtained only when rounding is performed, rounding or rounding other than rounding is performed. There is a certain effect even in the form.

The process of step S101 to S103, after compressing the dynamic range, adds the intermediate values _{x 3} 128 compressed (S104). After compressing the dynamic range, the center value is moved to 128 in order to surely reduce the amount of information in the non-key picture. If the maintaining the central value remains r _i, if the non-key picture itself will have the information of similar pixel value distribution key picture, resulting in information amount of the non-key picture is hardly reduced Because there is. Specifically, in the encoder unit 406, when it is not possible to specify whether to encode as an intra picture or as an inter picture, when the non-key picture is encoded as an intra picture, the amount of information is hardly reduced. The result is. This is because, as described above, the non-key picture itself has the same pixel value distribution as the key picture, that is, the entropy (compression limit) is almost unchanged. Incidentally, the center value when a value other than r _i, but transferred to 128 a central value for equally securing the dynamic range of the difference value in the positive and negative in this example, also a central value as a value other than 128 There is a certain effect.

In the encoder unit 406, when it can be specified that the unencoded image determined to be a key picture is encoded as an intra-picture and the unencoded image determined to be a non-key picture is encoded as an inter-picture, the center value. even then maintained at r _i, the effect of reducing the amount of codes for the difference between pictures is reduced is obtained. Therefore, the encoder unit 406 controls the unencoded image determined to be a key picture to be encoded as an intra-picture, and the unencoded image determined to be a non-key picture to be encoded as an inter-picture. for example, as one embodiment of the present invention to maintain the center value as shown in FIG. 5 (in step S301) remains r _i.

In this embodiment (process in step S102), an operation of halving the dynamic range with respect to _{the difference value x 1} between the pixel value of the ROI-imparted image and the pixel value of the key picture _{, specifically x 2} = x _1. ÷ 2 While running, this is because the dynamic range of x ₁ is twice the dynamic range of p ₁ by subtraction. If the dynamic range is not aligned to the same size, p ₀ may underflow or overflow, and even if clipping processing is added as a countermeasure against underflow and overflow, deterioration of image quality may occur. _{Therefore, x 2} = x ₁ ÷ 2 is executed in order to make the dynamic range equal to the same size. However, this process also has the side effect of reducing the accuracy of the difference value, so when inputting video that is known to have small fluctuations between pictures, it is better not to align the dynamic range to the same magnification for higher quality. Can be coded. Therefore, one of the examples is a configuration in which the dynamic range is not aligned to the same magnification. In this case, instead, p ₀ may be clipped in the domain (for example, 0 to 255) at the end.

Further, when filtering non-key pictures, if there is no retrievable key picture in the preprocessing buffer 104, a predetermined fixed center value (128 as an example in FIG. 6) as shown in FIG. _May be used as the pixel value ri of the virtual key picture. Further, in the example of FIG. 6, the center value at the time of output is set to 128, but as described above, the center value may be set to 128 at the time of output, or a pixel value other than 128 may be used as the center value to have a certain effect. do. In the example of FIG. 6, since the dynamic range is compressed around 128 as described above (S501, S502) and the center value at the time of output is also 128 (S503), the above-mentioned underflow and overflow may occur. do not have. Therefore, FIG. 6 shows an example in which the above-mentioned dynamic range is not adjusted to the same magnification. Of course, as a derivative of the example of FIG. 6, an example of aligning the dynamic range to the same magnification can be considered.

Note that "128", which is often described in this specification, assumes that the pixel value is an 8-bit integer value, and may be appropriately changed according to the expression range of the pixel value. For example, when the image has a pixel value of 0 to M, that is, when the domain of the pixel value is 0 to M, M / 2 may be used instead of 128.

Furthermore, it is simplest that the key picture used for filtering is a key picture located immediately before the non-key picture to be filtered among the images arranged in chronological order, that is, the images arranged in the order of time stamps. As a method, it may be a key picture located immediately before or after the nearest key picture, that is, a time stamp closest to the time stamp of the non-key picture.

Then, the encoder unit 406 compresses the unencoded image by using an image coding method such as AVC, and outputs a bit stream (S14).

(A-2-2) Operation of Image Decoding Device 20 FIG. 2 (B) is a flowchart showing the operation of the image decoding device 20 according to the first embodiment.

First, the decoder unit 407 receives a bit stream from the image coding device 10, decodes it using an image coding method such as AVC, and outputs the decoded image (S21).

The post-processing picture type control unit 108 determines the picture type of the decoded image and outputs it as the post-processing picture type (S22). Various methods can be applied to the method for determining the picture type, and examples thereof include the following methods.

(1) Method of periodically determining based on a predetermined key picture interval For example, when the key picture interval is set to 2, the key picture and the non-key picture are switched for each image. Specifically, when the T-th sheet is a key picture, the (T + 1) th sheet is a non-key picture, and the (T + 2) sheet is a key picture again. Similarly, when the key picture interval is set to 3, when the Tth sheet is a key picture, the (T + 1) and (T + 2) sheets are set as non-key pictures, and the (T + 3) sheet is set as a key picture again. do.

(2) A method in which a decoded image is treated as picture type auxiliary information and determined according to the properties of the decoded image. The properties of the decoded image, such as whether or not there is movement, are analyzed using existing computer vision methods, for example, decoding. When it is determined that there is no moving area in the image, the decoded image is used as a key picture, and the other decoded images are used as non-key pictures. More specifically, for example, using the decoded image to be processed and the decoded image input immediately before it, the motion vector is estimated for each pixel or for each block which is a set of one or more pixels, or for the entire image. If there is no motion vector with a magnitude greater than or equal to a predetermined threshold, or if the average motion vector of one or more motion vectors is calculated and the magnitude of the average motion vector is smaller than the predetermined threshold. The decoded image to be processed is used as a key picture.

The picture type determination results for the past N decoded images are traced back, and only when there is no decoded image determined to be a key picture among them, the decoded image determined to have no movement in the entire image is used as a key picture. It is also effective to make the other decoded images non-key pictures.

(3) A method of treating the decoded image as picture type auxiliary information and determining it according to the presence or absence of ROI For example, when the decoded image does not have ROI, it is determined as a key picture, and when ROI is present, it is determined as a non-key picture. do. It can be said that the judgment is based on the assumption that there will be no movement in the entire image when the ROI does not exist. It can be said that it is a special case of (2) mentioned above.

Further, as in (2) above, the decoded image having no ROI is obtained only when the picture type determination result for the past N decoded images is traced back and the decoded image determined to be the key picture does not exist among them. It is also effective to use 1 as a key picture and other decoded images as non-key pictures.

(4) A method of treating the determination of the preprocessing picture type control unit 101 as picture type auxiliary information and determining according to the information As described above, the picture type determined by the preprocessing picture type control unit 101 is an additional information packet or an additional information packet. It may be inserted into the bitstream by embedding it in the image. If information about the picture type is embedded in the bitstream, the picture type is determined based on the information.

Further, the post-processing picture type control unit 108 may adopt a method of combining the above-mentioned methods (1) to (4). The method for determining the determination may be determined in advance or may be obtained from the picture type auxiliary information.

The IDRC filter unit 109 refers to the post-processing picture type, outputs the decoded image as a post-processing reference picture and an output image if the decoded image is a key picture, and post-processes if the decoded image is a non-key picture. The reference picture for post-processing is taken out from the buffer 110, the pixels classified in the background area of the decoded image are filtered, and the image is output as an output image.

A specific method example of filtering performed by the IDRC filter unit 109 on a non-key picture is shown in FIG. 4 (B) (as a modification, FIGS. 5 (B) and 6 (B)).

In FIG. 4 (B), _{q i} is q-th pixel value of the decoded image, r _'i is the i-th pixel value of the key picture, _{R d} is the reciprocal of the compression rate _{R e} of the dynamic range _R d = 1 / _Re and q ₀ are the i-th pixel values of the output image, and y ₁ , y ₂ , and y ₃ are intermediate values. R _e is assumed to be shared by the DRC filter unit 103 and the IDRC filter unit 109 by manual configuration or transmission.

The image processing shown in this flowchart is applied to the pixels belonging to the background area. In this method, for pixels belonging to ROI, q _i is set to q ₀ without processing. The dynamic range of the pixel values q _i of the background area, by expanding around the pixel values r _'i for key picture, perform decompression of the dynamic range. As described in the paragraph explaining the operation of the DRC filter, this example shows an example of rounding when compressing the dynamic range, but the effect of the present invention is effective only when rounding is performed. Since it does not appear, there is a certain effect even in the form of rounding down or rounding up other than rounding.

Since the functions of the IDRC filter unit 109 (including the processes of FIGS. 4B to 6B) correspond to the inverse calculation of the functions of the DRC filter unit 103, detailed description thereof will be omitted (S23).

Note that the center value of the pixel values of the uncoded image when a value other than r _i, in the present embodiment are transferred to 128 the center value to equally ensure the dynamic range of the difference value in the positive and negative, As explained in the section of operation of the DRC filter unit 103, a certain effect exists even if the center value is set to a value other than 128. In this case, the IDRC filter unit 109 extends the dynamic range around the center value used by the DRC filter unit 103 as the center value of the unencoded image. Specifically, in the operation of the IDRC filter unit 109 shown in FIG. 4 (B) and FIG. 6 (B), the instead of subtracting the _{q i} from 128 initially, to subtract the center value used in the DRC filter unit 103 do. It is assumed that the center value is shared by the DRC filter unit 103 and the IDRC filter unit 109 by manual setting or transmission.

Further, when the DRC filter unit 103 does not align the dynamic range to the same magnification, the IDRC filter unit 109 also _{executes y 3} = y ₂ × 2, which is the inverse operation of the operation for aligning the dynamic range to the same magnification (process in step S203). ) Is unnecessary. It is assumed that the DRC filter unit 103 and the IDRC filter unit 109 share the presence or absence of the calculation for aligning the dynamic range to the same magnification by manual setting or transmission.

If there is no retrievable key picture in the post-processing buffer 110 when filtering non-key pictures, for example, the pixel value 128 of the virtual key picture is used as _{r'i as shown in FIG.} The same is true. It is assumed that the pixel value of the virtual key picture is shared by the DRC filter unit 103 and the IDRC filter unit 109 by manual setting or transmission.

(A-3) Effect of First Embodiment According to the first embodiment, the following effects can be obtained.

In the image processing system 1, the input image is divided into a key picture and a non-key picture, the key picture is encoded without processing, and the non-key picture is encoded after compressing the dynamic range using the key picture. The video transmission cost and storage cost can be reduced without modifying the encoder specifications and implementation, and the dynamic range is extended by using key pictures, so the change is small. The image quality is also maintained for the area. In other words, the image processing system 1 of the first embodiment can achieve both high image quality and reduction of the code amount of the entire image while guaranteeing the quality of ROI.

(B) Second Embodiment Hereinafter, the coding device and the program, the decoding device and the program according to the present invention, and the second embodiment of the image processing system will be described in detail with reference to the drawings. Hereinafter, an example in which the coding device of the present invention is applied to an image coding device and the decoding device of the present invention is applied to an image decoding device will be described.

(B-1) Configuration of Second Embodiment FIG. 7 is a block diagram showing a configuration of an image processing system 1A according to the second embodiment. In FIG. 7, the solid line arrow indicates the flow of essential processing (and information) in this embodiment, while the broken line arrow does not necessarily indicate the essential processing (and information) in order to achieve the effect of this embodiment. Information) is not. Hereinafter, the processing (and information) related to the broken line arrow will be supplementarily described as appropriate.

The image processing system 1A is different from the first embodiment in that the image coding device 10 and the image decoding device 20 of FIG. 1 are replaced with the image coding device 10A and the image decoding device 20B. Hereinafter, the second embodiment will be described focusing on the differences from the first embodiment.

In the image coding apparatus 10A of the second embodiment, the preprocessing unit 200A refers to the preprocessing picture type control unit 101 and the preprocessing picture type, and assigns an ROI when indicating that the image is a key picture. When the image indicates that it is a non-key picture, the DRC filter unit 103 outputs an image obtained by filtering the ROI-assigned image by referring to the pixel value of the virtual key picture as a pre-encoded image. have.

In the image decoding device 20A of the second embodiment, the post-processing unit 300A refers to the post-processing picture type control unit 108 and the post-processing picture type, and if it indicates that it is a key picture, the decoded image is displayed. , IDRC filter unit 109 that outputs a filtered image that refers to the pixel value of the virtual key picture to the decoded image as a pre-reconstruction image when it indicates that it is a non-key picture, and post-processing. Reconstruction processing using the key picture for the pre-reconstruction image when it refers to the picture type and indicates that it is a key picture, and for the pre-reconstruction image when it indicates that it is a non-key picture. It has a reconstruction unit 211 that outputs the image to which the image has been applied as an output image, and a post-processing buffer 110 that is a buffer for the reconstruction unit 211.

The reconstruction unit 211 stores the pre-reconstruction image determined to be a key picture in the post-processing buffer 110 by outputting it as a post-processing storage picture, and uses the pre-reconstruction image determined to be a non-key picture as a key. It also has a function of extracting a key picture from the post-processing buffer 110 as a post-processing reference picture in order to perform reconstruction processing using the picture information.

Although the details will be described later, since the reconstruction unit 211 can also perform reconstruction according to the quality parameter (for example, quantization parameter) used by the encoder unit 406, the quality parameter is input to the reconstruction unit 211 in FIG. An example of this is also shown by the dotted line.

(B-2) Operation of the Second Embodiment FIG. 8 is a flowchart showing the operation of the image processing system according to the second embodiment. Of FIG. 8 and FIG. 2 described above, the processes having the same reference numerals are the same, and therefore, only the processes specific to the image processing system 1A of the second embodiment will be described below.

(B-2-1) Operation of Image Coding Device 10A FIG. 8 (A) is a flowchart showing the operation of the image coding device 10A according to the second embodiment.

After step S12 described above, the DRC filter unit 103 refers to the preprocessed picture type, and if the ROI-imparted image is a key picture, outputs the ROI-imparted image as a pre-encoded image, and the ROI-imparted image is a non-key picture. If this is the case, the pixels classified in the background region of the ROI-imparted image are filtered using the pixel values of the virtual key picture and output as an unencoded image (S31).

(B-2-2) Operation of Image Decoding Device 20A FIG. 8 (B) is a flowchart showing the operation of the image decoding device 20A according to the second embodiment.

After step S22 described above, the IDRC filter unit 109 refers to the post-processed picture type, outputs the decoded image as a pre-reconstruction image if the decoded image is a key picture, and outputs the decoded image as a non-key picture if the decoded image is a non-key picture. , The pixels classified in the background area of the decoded image are filtered using the pixel values of the virtual key picture and output as the image before reconstruction (S41).

Here, some of the filters used by the DRC filter unit 103 and the IDRC filter unit 109 do not use the key picture. Therefore, the dynamic range is compressed or decompressed using the algorithm shown in FIG. 6 above. Specifically, for example, 128 is used as the pixel value of a virtual key picture to compress or expand the dynamic range. The detailed description of FIG. 6 is as described in the first embodiment.

In the second embodiment, since the DRC filter unit 103 does not perform filtering based on the key picture, the dynamic range of the background region is only compressed, that is, the amplitude of the pixel value signal is suppressed. So, it is recognizable by humans as an image. Therefore, in the second embodiment, a certain effect can be obtained even if the post-processing unit 300A does not exist. Similarly, the IDRC filter unit 109 can be used, and a certain effect can be obtained even when the reconstruction unit 211 does not exist. In this case, it becomes possible to watch the video in which the suppressed amplitude is restored with limited accuracy.

Next, the reconstruction unit 211 refers to the post-processing picture type, and if the pre-reconstruction image is a key picture, outputs the pre-reconstruction image as a post-processing reference picture and an output image, and the pre-reconstruction image is If it is a non-key picture, the post-processing reference picture is taken out from the post-processing buffer 110, the pixels classified in the background area of the pre-reconstruction image are reconstructed, and the image is output as an output image (S42).

In the following, specific algorithm examples for performing reconstruction in the reconstruction unit 211 will be described with reference to FIGS. 9 to 11.

FIG. 9 is a flowchart (No. 1) showing a reconstruction process of the reconstruction unit according to the second embodiment.

In FIG. 9, s _i is the i-th pixel value of the key picture, L _i is the lower limit of the presence section of the i-th pixel value in the output image, U _i is the upper limit of the presence section of the i-th pixel value of the output image The value, z ₀ , indicates the i-th pixel value of the output image.

Reconstruction unit 211, _{s i} is a judgment whether _{L i} is smaller than (S701), _{s i} is when _{L i} smaller than the _{z 0} and _{L i} (S702). Furthermore, the reconstruction unit 211, if _{s i} is greater than _{L i} is _{s i} is determined if greater than _{U i} (S703). Reconstruction unit 211, _{s i} is the _{z 0} to greater than _{U i} and _{U i} (S704), the _{z 0} and _{s i} are otherwise (S705).

In other words, the process of step S701~S705, if the pixel value _{s i} key picture is included in the present section is adopted as the pixel value _{z 0} of the output image _{s i, if s i} is outside the existence interval is an approach to adopt the boundary values of the side where the s _i as z _0.

L _i and U _i, using the i-th pixel value v _i of the reconstructed previous image, obtained by the following (1) and (2) below.

In the above equations (1) and (2), w represents the size of the existing section of the output image. Note that w / 2 is basically rounded down, but there is a certain effect whether it is rounded up or rounded up. w is obtained by, for example, the following equation (3).

In the above equation (3), W _Q is a quantization step generated in the encoder unit 406, and is 1.0 when the encoder unit 406 is a lossless encoder. As described above, R _d is the reciprocal _{of the compressibility R e of the dynamic range R d} = 1 / _Re , and can be interpreted as the size of the existence section generated by the DRC filter unit 103. Therefore, in this embodiment, it occurs in the entire coding. The size w of the existing section is obtained by multiplying _{R d} by W _Q. It is assumed that R _d is shared by the IDRC filter unit 109 and the reconstruction unit 211 by manual setting or transmission.

If R _d is an even number, it may be subtracted by 1 when calculating _{U i.} That is, the following equation (4) may be used.

The reason for subtracting 1 is basically assumed to be rounding when compressing the dynamic range in FIGS. 4 to 6 described above (as described above, this is only when rounding is performed in the DRC filter and IDRC filter. Since the effect of the invention is not obtained, there is a certain effect even in the form of rounding down or rounding up other than rounding), and rounding has the property that a positive bias can occur. For example, the number of integers that becomes 10 when rounded to the 1st place is 5 to 14, but the number less than 10 is 5 to 9, while the number greater than 10 is only 4 from 11 to 14. Considering this effect, the v _i large number of candidates than in determining the presence interval, v _i that thought to reduce one smaller number of candidates than is the previously described (4).

When the encoder unit 406 compresses with AVC, W _Q uses the quantization parameter qP (an example of the quality parameter in FIG. 7 is the quantization parameter qP), which is a parameter given to the video encoder, and uses, for example, the following. It is obtained by the equations (5) and (6).

In the above equation (6), the function L is, for example, the look-up table T shown in FIG. (6) is a calculation formula of the quantization step _{W s} to DC components defined in AVC. Since W _s is a quantization step defined by the transform domain, the transform domain quantization step W _s is converted into the pixel domain quantization step W _{Q by dividing W s by 4.} As long as the video encoder quantizes in the frequency domain, it is not possible to obtain the exact existence interval in the pixel domain. Therefore, w must be regarded as an approximate existence interval.

_{The method of obtaining w and W Q} mentioned here is just an example, and for example, _{a method of predetermining w and W Q} according to the quality parameter in the form of a look-up table or an arbitrary function may be used.

FIG. 10 is a flowchart (No. 2) showing a reconstruction process of the reconstruction unit according to the second embodiment.

Whereas the method of FIG. 9 determines the pixel value of the output image based only on the color component of interest (the pixel value of the output image is to be obtained from this), the method of FIG. 10 pays attention to other colors (in the present embodiment, attention is paid). This is a method of determining the pixel value of an output image using the color components (a total of three colors including the color components). In FIG. 10, S _i is the i-th pixel value of the key picture of the color component of interest, and s _i ^c1 and s _i ^c2 are the i-th pixel values of the key picture of the other color component. For example, when the color space is YUV (luminance component (Y) and color difference component (U, V)) which are common in moving image coding, and s _i is the pixel value of the key picture of the Y component, s _i ^c1, _s ^{i c2} is the pixel value of the key picture of the U and V components, respectively, the _{s i} is assumed to be the pixel values of the key picture of the U _component, ^s i _{c1, s} ^{i c2} respectively Y component and V the pixel value of the components of the key picture, the s _i is assumed to be the pixel values of the key picture of the V component, s i ^_c1, s _i ^c2 is the pixel value of the key picture of each Y component and the U component.

In video coding, the vertical and horizontal resolutions of the color difference components (U component and V component) or both are often compressed at half the resolution of the luminance component (Y component). In this case, if _{s i} is a pixel value of the Y _component, ^s i _{c1, s} ^{i c2} is the pixel value of the U and V components of a position enclosing the _{s i} spatially, _{s i} is a pixel of the U component If the value, s i ^_c1, s _i ^c2 is s _i is any pixel value in the space which encloses spatially (e.g., the V component of the pixel values spatially same position of the upper left of the Y component, respectively a pixel value), if s _i is a pixel value of the V component, s i ^_c1, s _i ^c2 is any pixel values in the space s _i is contained spatially (e.g., the upper left of the Y component, respectively The pixel value of the U component at the same position spatially as the pixel value).

L _i is the lower limit of the presence section of the i-th pixel value in the output image, _{U i} is the upper limit of the presence section of the i-th pixel value of the output ^{_{image, L i c1, L i c2}} , U i c1, U _i ^c2 is the lower limit and the upper limit of the existence interval of the pixel value of the output image of another color obtained by the following equations (7) to (10).

In the above formulas, v i c1, v i c2, on the other color components of the reconstructed previous image corresponding to s i c1, s i c2, a pixel value generated by the IDRC filter unit 109. The meaning of w is the same as the method shown in FIG. 9 above.

Further, in FIG. 10, z ₀ indicates the i-th pixel value of the output image. Note that w / 2 is basically rounded down, but there is a certain effect whether it is rounded up or rounded up.

Reconstruction unit 211, s i is less L i or U i, s i c1 is less L i c1 or U i c1, s i c2 is a judgment of whether L i c2 or U i c2 below (S801 ), if it meets the conditions, the z 0 and s i (S802). The processing (S701 to 705) when the conditions are not met is the same as in the case of FIG.

In FIG. 10, when the pixel value of the key picture is within the existence section of the color component other than the color component of interest that is spatially located at the same position or in the vicinity in addition to the color component of interest, the key picture is displayed. a s _i be adopted as z ₀ to believe that there is a high correct possibility. On the other hand, if s _i is outside of the present section, the boundary values of the side where the s _i z ₀ to be adopted. s _i Despite is within the existence period, if s i ^_c1, s _i ^c2 is outside the present section, because you can not believe key picture, corresponding to an intermediate value of the present interval before reconstruction image a _{v i} corresponding to the i-th pixel value is set to _{z 0.}

If R _d is an even number, it may be subtracted by 1 when calculating _{U i} , U _i ^c1 , and U _i ^{c 2.} That is, the following equations (11) to (13) may be used.

The reason for subtracting 1 is basically assumed to be rounding when compressing the dynamic range in the processes of FIGS. 4 to 6 (as described above, this is only for rounding in the DRC filter and IDRC filter. Since the effect of the invention is not obtained, there is a certain effect even in the form of rounding down or rounding up other than rounding), and rounding has the property that a positive bias can occur. For example, the number of integers that becomes 10 when rounded to the 1st place is 5 to 14, but the number less than 10 is 5 to 9, while the number greater than 10 is only 4 from 11 to 14. Considering this effect, the v _i large number of candidates than in determining the presence interval, v those thought to reduce one smaller number of candidates than _i is above (11) - (13) Is.

FIG. 11 is a flowchart (No. 3) showing a reconstruction process of the reconstruction unit according to the second embodiment.

The method of FIG. 11 is a derivative method of the method of FIG. Reconstruction unit 211, if not meet the condition of step S801 described above, s i is less than L i, s i c1 is less than L i c1, s i c2 is a judgment of whether less than L i c2 (S901), if it meets the conditions, the z 0 and L i (S902). On the other hand, when the condition of step S901 is not satisfied, the reconstruction unit 211 determines whether or not s i is U i or more, s i c 1 is U i c 1 or more, and s i c 2 is U i c 2 or more. was carried out (S903), if it meets the conditions, the z 0 and U i (S904), if it does not meet the conditions, the z 0 and v i (S905).

That is, the method of FIG. 11 is when the pixel value of the key picture is within the existing section even for a color component other than the color component of interest that is spatially located at the same position or a nearby position in addition to the color component of interest. , key picture is to adopt a s _i as z ₀ to believe that there is a high correct possibility. In the case where s _i is outside the present _interval, ^s i _{c1, s} ^{i c2} also if it is outside of the present section on the same side, employing the boundary value of the side where the _{s i} as _{z 0.} In other cases, because you can not believe key picture, the v _i corresponding to i-th pixel value of the corresponding to the intermediate value before reconstruction images present section and z _0.

The above-mentioned methods of FIGS. 9 to 11 are examples in which the threshold value is determined by the scale of the pixel value of the key picture (that is, the scale of the input image original without compressing the dynamic range), but conversely, of the decoded image. The threshold value may be determined by the scale of the pixel value in the background area of the non-key picture (that is, the scale in the state where the dynamic range is compressed). When determining the threshold by the scale of the pixel values in the background area of the decoded image, the dynamic range of the pixel values in the background area of the key picture is the same as the dynamic range of the pixel values in the background area of the non-key picture of the decoded image. After compression, it is sufficient to compare whether the pixel value of the background area of the key picture is within the existing section. In this case, w may _{be obtained by w = W Q} , that is, the existence section may be obtained by assuming _{that R d = 1.0.} Regardless of which scale is compared, the difference is only in the scale to be compared, there is no semantic difference, and the same effect is exhibited.

In any of the methods of FIGS. 9 to 11, the existing interval of the pixel value of the output image is obtained from the pixel value of the image before reconstruction, the parameter of the DRC filter unit 103, and the parameter of the encoder unit 406, and the key picture is obtained. This is a common method in the sense that the pixel value of the output image that seems to be appropriate is estimated from the existing section by using the pixel value of.

Further, as a method of probabilistically determining the pixel value based on the existing section and the side information, for example, Non-Patent Documents 1 to 3 have been proposed, but the pixels of the output image as shown in these documents. It may be used as a method for determining the value. Each of the methods of FIGS. 12 to 14 is the most basic method of probabilistically determining the pixel value based on the existence section and the side information.

Further, in the present embodiment has a side information s _i and the pixel values of the key picture, as shown in Non-Patent Document 1 Non-Patent Document 3, side information s _i is defined as the pixel value of the key picture Not at all. Therefore, the pixel value of the key picture may be, for example, the pixel value of the output image output from the image decoding device 20A immediately before, or the image before reconstruction and the reference image (output from the key picture or the image decoding device 20A immediately before). The motion vector may be estimated for each pixel or for each block which is a set of one or more pixels using the output image), and may be used as the pixel value of the predicted image for which motion compensation is predicted using the motion vector. In this case, the post-processing buffer 110 is assumed to store not only the key picture but also the image information necessary for generating the side information.

(B-3) Effect of Second Embodiment According to the second embodiment, the following effects can be obtained.

The image processing system 1A divides the input image into a key picture and a non-key picture, encodes the key picture without processing, and encodes the non-key picture after compressing the dynamic range. It is possible to reduce the video transmission cost and storage cost without modifying the system, and by extending the dynamic range and reconstructing using key pictures, the image quality is high for areas with small changes. Is also maintained. In other words, the image processing system 1A can achieve both high image quality and reduction of the code amount of the entire image while guaranteeing the quality of ROI.

Also, unlike the first embodiment, the image coding apparatus 10A of the second embodiment compresses the dynamic range without buffering (that is, referring to) the key picture, and thus the image. The memory consumption of the coding device is reduced.

Further, in the second embodiment, the background region has only the dynamic range compressed, that is, the amplitude of the pixel value signal is suppressed, and is recognizable by humans as an image. Therefore, there is an effect that a meaningful image can be viewed even if the post-processing unit does not exist. Similarly, the IDRC filter unit can be used, and there is an effect that a meaningful image can be viewed even when the reconstruction unit does not exist. In this case, it becomes possible to watch the video in which the suppressed amplitude is restored with limited accuracy. Therefore, for example, it is possible to reproduce a meaningful image even if the post-processing unit or the reconstruction unit cannot be used due to the problem of the amount of calculation. As described above, the second embodiment also has an effect that the amount of calculation on the decoding side can be made scalable.

(C) Third Embodiment Hereinafter, the coding device and program, the decoding device and program according to the present invention, and the third embodiment of the image processing system will be described in detail with reference to the drawings. Hereinafter, an example in which the coding device of the present invention is applied to an image coding device and the decoding device of the present invention is applied to an image decoding device will be described.

(C-1) Configuration of Third Embodiment FIG. 13 is a block diagram showing a configuration of an image processing system 1B according to a third embodiment. In FIG. 13, the solid line arrow indicates the flow of essential processing (and information) in this embodiment, while the broken line arrow does not necessarily indicate the essential processing (and information) in order to achieve the effect of this embodiment. Information) is not. Hereinafter, the processing (and information) related to the broken line arrow will be supplementarily described as appropriate.

In the image processing system 1B, the image coding device 10 (10A) and the image decoding device 20 (20A) of FIG. 1 (FIG. 7) are replaced with the image coding device 10B and the image decoding device 20B. It is different from the second embodiment. Hereinafter, the third embodiment will be described focusing on the differences from the first and second embodiments.

In the image coding apparatus 10B of the third embodiment, the preprocessing unit 200B refers to the preprocessing picture type control unit 101 and the preprocessing picture type, and assigns an ROI when indicating that the image is a key picture. When the image indicates that it is a non-key picture, the DRC filter unit 103 that outputs a filtered image that refers to the pixel value of the virtual key picture to the ROI-assigned image as an internal image, and the front When the processed picture type points to a non-key picture, quality parameter control for assigning a quality parameter for coding based on ROI information (position and size of ROI) to the internal image and outputting it as an image before coding. It has a part 305 and.

The encoder unit 406 compresses the pre-encoded image based on the coding quality parameters given to the pre-encoded image by using an image coding method such as AVC, and outputs a bit stream.

In the image decoding device 20B, the reconstruction unit 211 refers to the post-processing picture type, and when it indicates that it is a key picture, it indicates that the image before reconstruction is a non-key picture. Outputs the image obtained by performing the reconstruction process on the image before reconstruction as an output image, and when the quality parameter is used in the process of the reconstruction process, the quality parameter of the corresponding background area is used.

(C-2) Operation of the Third Embodiment FIG. 14 is a flowchart showing the operation of the image processing system according to the third embodiment. Of FIGS. 14 and 2 (and FIG. 8), the processes having the same reference numerals are the same, and therefore, only the processes specific to the image processing system 1B of the third embodiment will be described below.

(C-2-1) Operation of Image Coding Device 10B FIG. 14 (A) is a flowchart showing the operation of the image coding device 10B according to the third embodiment.

After step S12 described above, the DRC filter unit 103 refers to the preprocessed picture type, and if the ROI-imparted image is a key picture, outputs the ROI-imparted image as an internal image, and the ROI-imparted image is a non-key picture. For example, the pixels classified in the background area of the ROI-imparted image are filtered using the pixel values of the virtual key picture and output as an internal image (S51).

The quality parameter control unit 305 refers to the preprocessed picture type, and if the internal image is a key picture, assigns a uniform coding quality parameter to the entire image of the internal image and outputs it as a pre-encoding image. If the internal image is a non-key picture, a non-uniform coding quality parameter is added based on the ROI information given to the internal image and output as a pre-encoding image (S52). Specifically, a quality parameter for coding that induces coarse quantization (lower image quality) is given to the background region and fine quantization (higher image quality) is given to the ROI.

The uniform coding quality parameter described in this embodiment does not necessarily mean that the quantization parameter is the same in all regions. In the field of moving image coding, there is also a method of allocating quantization parameters on the screen according to the features of an image, and such a method is also regarded as a kind of uniform quality parameter for coding. The difference between the uniform coding quality parameter and the non-uniform coding quality parameter is whether or not the quality parameter is differentiated by region based on the ROI information.

The encoder unit 406 compresses the unencoded image by using an image coding method such as AVC, and outputs a bit stream (S53). As the quality parameter at the time of coding, the one given to the uncoded image is used.

(C-2-2) Operation of Image Decoding Device 20B After step S41 described above, the reconstruction unit 211 refers to the post-processing picture type, and if the image before reconstruction is a key picture, the image before reconstruction is displayed. If the post-processing reference picture and the output image are output and the pre-reconstruction image is a non-key picture, the post-processing reference picture is taken out from the post-processing buffer 110 and classified into the background area of the pre-reconstruction image. The pixels are reconstructed and output as an output image (S61).

As described in the second embodiment, the reconstruction unit 211 may perform reconstruction based on the quality parameter. Specifically, there is an operation example in which w is determined using a quality parameter. When performing reconstruction using quality parameters, the quantization step generated in the encoder unit 406 is based on the quality parameters (values assigned by the quality parameter control unit 305) assigned to each background area to be reconstructed. Or, find and use an approximate value thereof.

In this embodiment, the preprocessing unit 200 (200A) also compresses the dynamic range using the DRC filter unit 103 as in the first embodiment and the second embodiment. Then, since the quality parameter control unit 305 makes a difference between the quality of the ROI and the background area, _{it is constant even when Re} = 1.0, that is, the configuration in which the ROI-assigned image is input to the quality parameter control unit 305 as an internal image. The effect of is obtained. In this case, since R _d = 1.0, a certain effect can be obtained even in a configuration in which the decoded image is input to the reconstruction unit 211 as the image before reconstruction.

(C-3) Effect of Third Embodiment According to the third embodiment, the following effects can be obtained.

The image processing system 1B divides the image into key pictures and non-key pictures, the key pictures are unprocessed and the entire image is encoded with uniform quality parameters, and the non-key pictures have non-uniform quality while compressing the dynamic range. By assigning parameters and then encoding, it is possible to reduce the transmission cost and storage cost of video, and by reconstructing using key pictures, the image quality is good for areas with small changes. Is also maintained. In other words, the image processing system 1B can achieve both high image quality and reduction of the code amount of the entire image while guaranteeing the quality of ROI. The effect that a meaningful image can be viewed even if the post-processing unit and the reconstruction unit cannot be used is the same as that of the second embodiment.

(D) Other Embodiments Although various modified embodiments have been mentioned in the above-mentioned first to third embodiments, the present invention can also be applied to the following modified embodiments.

(D-1) In the first to third embodiments described above, in order to simplify the explanation, an example in which the entire image is divided into two types, ROI and background area, is shown. There may also be a level of importance. For example, a pedestrian detection algorithm and a face detection algorithm are used to set the ROI, the ROI detected by the face detection algorithm is a face area, and the pedestrian area detected by the pedestrian detection algorithm is a relatively important background area. The area that is neither a face nor a pedestrian may be a relatively insignificant background area. In this case, it allocates a different value to the quality parameter for the compression rate R _e and encoding the dynamic range according to the importance of the background region, for example, a weak filtering (relatively high for a relatively important background area It is also possible to perform a configuration in which the image quality is filtered) and the background area, which is relatively insignificant, is strongly filtered (filtered in a relatively low image quality). Allocation may be performed based on, for example, a look-up table of the importance of the background area and the filter strength that have been set or created in advance. Further, although the above shows an example in which filtering is not applied to the ROI for the sake of simplicity, the ROI may be configured to be filtered weaker than the background area.

(D-2) In the first to third embodiments described above, for example, when the face region is set to ROI, the background region is deteriorated to reduce the amount of data, but the ROI is often used as in the face region. The area selected as may be degraded. In this case, by definition of the term in the present specification, the non-face area is the ROI and the face area is the background area. By setting the non-face area as ROI, for example, the effect of protecting privacy can be obtained.

(D-3) In the first to third embodiments described above, it is described that data is transferred between functional blocks in image units, but the meaning of the pixel value signal data is clarified. For this reason, data may be passed in pixel units for mounting and the like.

(D-4) In the first to third embodiments described above, the ROI is mainly described as a limited area of a part of the image space, but the entire image is the ROI or the entire image. There may be an image in which is the background area.

(D-5) In the first to third embodiments described above, each function in the image coding device 10 (10A, 10B) and each function in the image decoding device 20 (20A, 20B) are a single device. Although it is described as being implemented in, these are one of the implementation examples, and even if each function is implemented in another device, each signal is a configuration diagram (FIGS. 1, FIG. 7. The effect of the present invention can be obtained if input / output is performed as shown in FIG. 13). That is, even if the image coding device 10 (10A, 10B) and the image decoding device 20 (20A, 20B) are composed of one or more devices and their respective functions are distributed and implemented, the effect of the present invention is still effective. exist.

(D-6), particularly in the second and third embodiments, although all are described as using a compression rate R _e for the same dynamic range color component, in all embodiments compression rate of the dynamic range for each color component R _e may be different ones.

(D-7) In the second and third embodiments described above, the image coding devices 10A and 10B have shown an example having the ROI setting unit 402, but as a modification, the image coding devices 10A and 10B have It is not necessary to have the ROI setting unit 402. In this case, the coding devices 10A and 10B of the modified example operate in the same manner as when the ROI setting unit 402 determines that the input image does not have an interest region (ROI). That is, the ROI-imparted image shown in FIGS. 7 and 13 is actually an input image in which the ROI does not exist (the input image is directly input to the DRC filter unit 103 in the subsequent stage). The processes described in the second and third embodiments described above can be applied as they are to the coding process of the modification and the decoding process of the image decoding devices 20A and 20B.

1, 1A, 1B ... Image processing system, 10, 10A, 10B ... Image encoder, 20, 20A, 20B ... Image decoding device, 101 ... Preprocessing Picture type control unit, 103 ... DRC filter unit, 104 ... Preprocessing Buffer, 108 ... Post-processing picture type control unit, 109 ... IDRC filter unit, 110 ... Post-processing buffer, 200, 200A, 200B ... Pre-processing unit, 211 ... Reconstruction unit, 300, 300A, 300B ... Post-processing unit, 305 ... Quality parameter control unit, 402 ... ROI setting unit, 406 ... Encoder unit, 407 ... Decoder unit.

Claims (25)

In a coding device that encodes an input image,
An area of interest setting unit that assigns an area of interest to the input image and outputs it as an image with an area of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
It has a coding unit that compresses the uncoded image using a predetermined image coding method and outputs the compressed coded data.
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is obtained, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the key picture. A dynamic range compression unit that outputs as a pre-converted image,
It has a preprocessing buffer which is a buffer for the dynamic range compression unit.
The dynamic range compression unit outputs the attention area imparting image determined to be a key picture as a preprocessing storage picture, accumulates it in the preprocessing buffer, and imparts the attention area determined to be a non-key picture. A coding device for extracting key picture information from the preprocessing buffer as a preprocessing reference picture in order to filter an image using the key picture information. A region of interest is assigned to the input image, and among the input images to which the region of interest is assigned, the pre-processed image for reducing the amount of information in the background region is encoded. A decoding unit that decodes the coded data and outputs the decoded image,
A post-processing picture type control unit that determines a post-processing picture type indicating whether the decoded image is a key picture or a non-key picture and outputs the determined post-processing picture type.
Dynamic range the post picture type, when a key picture is for outputting the decoded image, the image in the case of a non-key picture which has been subjected to filtering to reference the key picture in the decoded picture, an output image With the extension part
It has a post-processing buffer which is a buffer for the dynamic range extension part, and has.
The dynamic range extension unit outputs the decoded image determined to be a key picture as a post-processing storage picture, accumulates the decoded image in the post-processing buffer, and outputs the decoded image determined to be a non-key picture to the key picture. A compound device for extracting key picture information from the post-processing buffer as a post-processing reference picture in order to filter using the information of. In an image processing system including a coding device that encodes an input image to generate coded data and a decoding device that decodes the coded data generated by the coding device to generate a decoded image, the coding is performed. An image processing system characterized in that the coding apparatus according to claim 1 is applied as an apparatus, and the decoding apparatus according to claim 2 is applied as the decoding apparatus. The dynamic range compression unit sets the dynamic range of the pixel values in the background region to 0, which is a coefficient for manipulating the reduction amount of the amount of information centered on the pixel values of the preprocessing reference picture, and corresponds to the compression rate of the dynamic range. using real _{R e} of .0～1.0, to compress and reduce the amount of information,
The dynamic range extension unit the dynamic range of the pixel values of the background area, centered on the pixel values of the post-processing reference pictures, using the inverse of the R _e, in claim 3, characterized in that the extension The image processing system described. The coding unit controls to encode the pre-encoded image determined to be a key picture as an intra-picture and to encode the un-encoded image determined to be a non-key picture as an inter-picture. The image processing system according to claim 3 or 4. The dynamic range compression unit compresses the dynamic range of the pixel values in the background area and then shifts the center value to a predetermined pixel value, or sets the definition range of the pixel values in the background area to 0 to M. After compressing the dynamic range of the pixel values in the background area, the center value is moved to M / 2.
The dynamic range extension unit, according to claim wherein the dynamic range compression section mainly the central value used as a central value of the coded previous image, characterized by extending the dynamic range of the pixel values of the background area The image processing system according to 4 or 5. The dynamic range compression unit performs an operation to halve the dynamic range of the pixel value of the background region with respect to the difference value between the pixel value of the image to which the attention region is given and the pixel value of the reference picture for preprocessing.
The image processing system according to any one of claims 4 to 6, wherein the dynamic range extension unit performs an inverse operation of an operation of halving the dynamic range of pixel values in the background region. When the dynamic range compression unit performs filtering for referring to the key picture on the image to which the region of interest is given, if there is no extractable reference picture for preprocessing in the preprocessing buffer, the dynamic range compression unit is fixed in advance. Use the center value of as the pixel value of the virtual key picture,
When the decoded image is filtered to refer to the key picture, the dynamic range extension unit sets the fixed center value if the post-processing reference picture that can be extracted does not exist in the post-processing buffer. The image processing system according to any one of claims 4 to 7, wherein the image processing system is used as a pixel value of a virtual key picture. A computer mounted on a coding device that encodes an input image
An area of interest setting unit that assigns an area of interest to the input image and outputs it as an image with an area of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
Using a predetermined image coding method, the pre-encoded image is compressed and made to function as a coding unit that outputs the compressed coded data.
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is obtained, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the key picture. A dynamic range compression unit that outputs as a pre-converted image,
It has a preprocessing buffer which is a buffer for the dynamic range compression unit.
The dynamic range compression unit outputs the attention area imparting image determined to be a key picture as a preprocessing storage picture, accumulates it in the preprocessing buffer, and imparts the attention area determined to be a non-key picture. A coding program characterized by extracting key picture information from the preprocessing buffer as a preprocessing reference picture in order to filter an image using the key picture information. Computer,
A region of interest is assigned to the input image, and among the input images to which the region of interest is assigned, the pre-processed image for reducing the amount of information in the background region is encoded. A decoding unit that decodes the coded data and outputs the decoded image,
A post-processing picture type control unit that determines a post-processing picture type indicating whether the decoded image is a key picture or a non-key picture and outputs the determined post-processing picture type.
Dynamic range the post picture type, when a key picture is for outputting the decoded image, the image in the case of a non-key picture which has been subjected to filtering to reference the key picture in the decoded picture, an output image With the extension part
It is made to function as a post-processing buffer which is a buffer for the dynamic range extension part.
The dynamic range extension unit outputs the decoded image determined to be a key picture as a post-processing storage picture, accumulates the decoded image in the post-processing buffer, and outputs the decoded image determined to be a non-key picture to the key picture. A compound program characterized by extracting key picture information from the post-processing buffer as a post-processing reference picture in order to filter using the information of. In a coding device that encodes an input image,
An area of interest setting unit that assigns an area of interest to the input image and outputs it as an image with an area of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
It has a coding unit that compresses the uncoded image using a predetermined image coding method and outputs the compressed coded data.
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is filtered, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the pixel value of the virtual key picture. A coding device including a dynamic range compression unit that outputs the image as an image before coding. A region of interest is assigned to the input image, and among the input images to which the region of interest is assigned, the pre-processed image for reducing the amount of information in the background region is encoded. A decoding unit that decodes the coded data and outputs the decoded image,
A post-processing picture type control unit that determines a post-processing picture type indicating whether the decoded image is a key picture or a non-key picture and outputs the determined post-processing picture type.
If the post-processed picture type is a key picture, the decoded image is regenerated, and if the post-processed picture type is a non-key picture, the decoded image is filtered to refer to the pixel value of the virtual key picture. The dynamic range extension part that outputs as a pre-construction image,
When the post-processed picture type is a key picture, the image before reconstruction is output, and when the post-processed picture type is a non-key picture, the image before reconstruction using the key picture is output. The reconstruction part that outputs as an image and
It has a post-processing buffer which is a buffer for the reconstruction unit, and has a post-processing buffer.
The reconstruction unit outputs the pre-reconstruction image determined to be a key picture as a post-processing storage picture, accumulates it in the post-processing buffer, and stores the pre-reconstruction image determined to be a non-key picture. Is extracted from the post-processing buffer as a post-processing reference picture in order to reconstruct the key picture information using the key picture information. In an image processing system including a coding device that encodes an input image to generate coded data and a decoding device that decodes the coded data generated by the coding device to generate a decoded image, the coding is performed. An image processing system comprising applying the coding apparatus according to claim 11 as an apparatus and applying the decoding apparatus according to claim 12 as the decoding apparatus. The reconstruction unit obtains the existence section of the pixel value of the output image from the pixel value of the pre-reconstruction image and the size of an arbitrary existence section, and when the side information is included in the existence section, the side The feature is that the information is applied as the pixel value of the output image, and when the side information is outside the existing section, the boundary value on the side where the side information exists is applied as the pixel value of the output image. The image processing system according to claim 13. The reconstruction unit obtains the existence section of the pixel value of the output image from the pixel value of the pre-reconstruction image and the size of an arbitrary existence section, and in addition to the color component of interest, the spatially the same position or When the side information is within the existing section even for a color component other than the color component of interest located at a nearby position, the side information is applied as a pixel value of the output image, and the color component of interest of the side information is said. When it is outside the existing section, the boundary value on the side where the side information exists is applied as the pixel value of the output image, and the color component of interest of the side information is contained within the existing section. Nevertheless, when a color component other than the color component of interest is outside the existing section, the pixel value of the pre-reconstruction image, which is an intermediate value of the existing section, is applied as the pixel value of the output image. 13. The image processing system according to claim 13. The reconstruction unit obtains the existence section of the pixel value of the output image from the pixel value of the pre-reconstruction image and the size of an arbitrary existence section, and in addition to the color component of interest, spatially at the same position or in the vicinity. When the side information is within the existing section even for a color component other than the color component of interest located at the position, the side information is applied as a pixel value of the output image, and the color component of interest of the side information is said. If the side information is outside the existing section on the same side even if the color component other than the color component of interest is outside the existing section, the boundary value of the side with the side information of the color component of interest is set as described above. The image according to claim 13, wherein the image is applied as a pixel value of the output image, and in other cases, the pixel value of the pre-reconstruction image corresponding to the intermediate value of the existing section is applied as the pixel value of the output image. Processing system. The side information includes a reference picture postprocessing, or is a pixel value of the prediction image before Kifuku No. outputted the output image from the device, or the generated on the basis of the reconstructed previous image and the output image immediately before The image processing system according to any one of claims 14 to 16. The reconstructing unit is set in advance, or the quantization step generated in the coding unit or its approximate value obtained by calculation from the quality parameters used by the coding unit, and the dynamic range compression unit . The image processing system according to any one of claims 14 to 17, wherein the size of the existing section is multiplied by the reciprocal of the compression rate of the dynamic range of the pixel value of the background region to be used. Does the reconstructing unit compress the dynamic range of the side information at the same compression rate as the dynamic range of the pixel value in the background region of the decoded image, and then whether the compressed side information exists in the existing section? The image processing system according to any one of claims 14 to 18, characterized in that A computer mounted on a coding device that encodes an input image
An area of interest setting unit that assigns an area of interest to the input image and outputs it as an image with an area of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
Using a predetermined image coding method, the pre-encoded image is compressed and made to function as a coding unit that outputs the compressed coded data.
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is filtered, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the pixel value of the virtual key picture. A coding program characterized by having a dynamic range compression unit that outputs the image as an image before coding. Computer,
A region of interest is assigned to the input image, and among the input images to which the region of interest is assigned, the pre-processed image for reducing the amount of information in the background region is encoded. A decoding unit that decodes the coded data and outputs the decoded image,
A post-processing picture type control unit that determines a post-processing picture type indicating whether the decoded image is a key picture or a non-key picture and outputs the determined post-processing picture type.
If the post-processed picture type is a key picture, the decoded image is regenerated, and if the post-processed picture type is a non-key picture, the decoded image is filtered to refer to the pixel value of the virtual key picture. The dynamic range extension part that outputs as a pre-construction image,
When the post-processed picture type is a key picture, the image before reconstruction is output, and when the post-processed picture type is a non-key picture, the image before reconstruction using the key picture is output. The reconstruction part that outputs as an image and
It is made to function as a post-processing buffer which is a buffer for the reconstructing part.
The reconstructing unit outputs the pre-reconstruction image determined to be a key picture as a post-processing storage picture, accumulates it in the post-processing buffer, and stores the pre-reconstruction image determined to be a non-key picture. Is extracted from the post-processing buffer as a post-processing reference picture in order to reconstruct the key picture information using the key picture information. In a coding device that encodes an input image,
A region of interest setting unit that assigns a region of interest to the input image and outputs the image as an image of the region of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
Using a predetermined image coding method, the pre-encoded image is compressed and compressed based on a coding quality parameter that induces coarse quantization in the background region and fine quantization in the region of interest. It has a coding unit that outputs the coded data.
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is filtered, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the pixel value of the virtual key picture. When the dynamic range compression unit that outputs the image as an internal image and the preprocessed picture type are non-key pictures, the coding is performed based on the area of interest information indicating the position and size of the area of interest in the internal image. A coding apparatus having a quality parameter control unit that outputs a pre-encoded image after being given a quality parameter for use. In an image processing system including a coding device that encodes an input image to generate coded data and a decoding device that decodes the coded data generated by the coding device to generate a decoded image, the coding is performed. An image processing system comprising applying the coding apparatus according to claim 22 as an apparatus and applying the decoding apparatus according to claim 12 as the decoding apparatus. The reconstruction unit, for each background region of the reconstructed previous image, the use of seeking quantization step or an approximate value thereof occurs at basis the encoding unit in the encoding for the quality parameters that are allocated The image processing system according to claim 23. A computer mounted on a coding device that encodes an input image
A region of interest setting unit that assigns a region of interest to the input image and outputs the image as an image of the region of interest.
A preprocessing unit that applies preprocessing to reduce the amount of information in the background region of the attention region imparted image to the attention region imparted image and outputs it as a pre-encoded image.
Using a predetermined image coding method, the pre-encoded image is compressed and compressed based on a coding quality parameter that induces coarse quantization in the background region and fine quantization in the region of interest. It functions as a coding unit that outputs the coded data
The pretreatment unit
A pre-processed picture type control unit that determines a pre-processed picture type indicating whether the input image is a key picture or a non-key picture and outputs the determined pre-processed picture type.
When the preprocessed picture type is a key picture, the attention area-imparted image is filtered, and when the preprocessed picture type is a non-key picture, the attention area-imparted image is filtered to refer to the pixel value of the virtual key picture. When the dynamic range compression unit that outputs the image as an internal image and the preprocessed picture type are non-key pictures, the coding is performed based on the area of interest information indicating the position and size of the area of interest in the internal image. A coding program characterized by having a quality parameter control unit that outputs a pre-coded image after assigning quality parameters for use.

Images