JP4360985B2 - Moving picture coding apparatus, moving picture coding method, and moving picture coding program - Google Patents

Description

  The present invention relates to a moving image encoding apparatus and a moving image encoding method, and more particularly to a moving image encoding apparatus and a moving image encoding method that compress image data for each frame.

  In recent years, the reproduction of moving images by a computer has been rapidly spreading, and its usage form is wide from a small one such as a mobile phone to a large one such as a plasma TV. .

  The moving image data is generally composed of image data of each frame constituting the moving image, but if all the image data of the number of frames are stored as they are, the amount of data becomes considerably large. For example, in the case of a moving image having a playback time of 10 minutes at 30 frames per second, it is necessary to store 18000 pieces of image data.

  Therefore, many methods for compressing and storing image data have been studied. A typical example is a moving picture compression method standardized by the Moving Pictures Experts Group (MPEG), and MPEG1, MPEG2, MPEG4 and the like are mainly spread.

  The moving picture compression by MPEG is independent of the preceding and following frame image data, the I frame for encoding only the frame image data, the P frame for encoding using motion prediction from the previous frame image data, It is composed of B frames that are encoded using motion prediction from both frame image data.

  The problem here is the weight of processing when decoding the compressed code data. Since I frames are encoded with only image data without motion prediction, it is very unlikely that processing will be in time. However, P frames and B frames are decoded based on motion prediction information. There is a possibility that the calculation amount for decoding increases and the processing becomes heavy. If the processing becomes heavy and it takes a lot of calculation time, the decoding process will not be in time, and the decoded image cannot be displayed. This state is called frame skip. In this case, the image decoded immediately before is continuously displayed, and the image decoded and displayed after the frame skip becomes an image discontinuous in time.

  When a frame skip occurs in the P frame and the B frame, the next frame is decoded based on the motion prediction information based on the contents of the previous frame. In this case, the frame skip state is maintained until an I frame that does not require information on the previous frame appears.

In order to solve such a problem, whether or not to perform encoding based on a comparison result between the set target code amount and the predicted code amount is predicted when encoding by data compression is performed Furthermore, a method for determining whether to perform overall encoding or partial encoding has been proposed (for example, Patent Document 1). In addition, a method of generating a plurality of pieces of code data having different code amounts and selecting an optimum one has been proposed (for example, Patent Document 2).
JP 2002-101417 A JP 2004-128941 A

  However, the amount of calculation for decoding is not necessarily proportional to the amount of code at the time of encoding. Even if the amount of code is large, the processing required for decoding may be light, or even if the amount of code is small In some cases, the processing required for this is heavy. In addition, the method of generating and selecting a plurality of code data has a problem that extra calculation is required and the encoding efficiency is deteriorated.

  The present invention has been made to solve such a problem, and is a moving image encoding apparatus, a moving image encoding method, and a moving image that minimize the problem of discontinuous image display due to frame skipping during decoding. An object is to provide an image encoding program.

Moving picture coding apparatus in the present invention, a code data that can be decoded and reproduced by the moving image reproducing apparatus comprising a video coding device that generates a plurality of frames picture included in the moving picture data input and a single image encoder for generating a first code data from one frame image data among the image data, and the previous frame picture image data of the frame image data of the said one frame image data temporary A prediction encoding unit that generates second code data based on the difference information, a load estimation unit that estimates a processing amount when the moving image reproduction device performs decoding of the second code data, based on prior Ki推 constant has been processed quantity, the encoding of the next frame picture image data of the one frame image data to determine whether to perform either of the single image encoder and the prediction encoding unit moving with a control unit, the An image encoding device. With such a configuration, it is possible to minimize the problem of discontinuous image display during frame skipping.

The predictive encoding unit includes a local decoder for decoding at least the second sign-data, the load estimation unit, based on the amount of processing according to the decoding process by the local decoder, the video The processing amount when the image reproducing apparatus performs decoding may be estimated. With such a configuration, decoding performed at the time of encoding can be used for load determination, and encoding time can be reduced.

The control unit may pre Ki推 based on the comparison result of the constant has been processed amount with a predetermined value, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit it may be characterized by determining any on whether performing the. By doing so, it is possible to cope with a change in the moving image reproduction apparatus.

Moving picture coding method in the present invention, a code data that can be decoded and reproduced by the moving image reproducing apparatus a moving picture coding method that generates, by a single image encoding unit, the moving image data input from one frame image data among the plurality of frame image picture data included to generate a first code data, the predictive coding unit, wherein one of the one frame image data and the one frame image data before based of the difference information between the frame picture image data to generate second encoded data, the decoding of the second code data to estimate such processing amount when the moving image reproduction device performs said estimation result of, based on the estimated throughput, the coding of the next frame picture image data of the one frame image data to determine whether to perform either of the single image encoder and the prediction encoding unit How to encode video It is. By doing so, it is possible to minimize the problem of discontinuous image display during frame skipping.

The generation of the second code data by predictive coding unit, the previous frame image data the one frame image and image data obtained by decoding by the local decoder the code data corresponding to the The amount of processing that is performed by generating code data based on the difference information with respect to the image data and that is required when the moving image reproduction device performs decoding is the amount of processing that is required for the decoding processing by the local decoding unit It is good also as performing by estimating based on. By doing in this way, decoding performed at the time of encoding can be used for load determination, and encoding time can be shortened.

Before based on a comparison result between the processing amount and the predetermined value of the decoding process by km over local decoding unit, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit it may be characterized that you decide whether to perform either. By doing so, it is possible to cope with a change in the moving image reproduction apparatus.

Video encoding program of the present invention, a code data that can be decoded and reproduced by the moving image reproducing apparatus a moving picture encoding program for causing a moving picture coding method that generates the computer, a single image encoding the unit, to generate a first encoded data from one frame image data among the plurality of frame image picture data included in the moving image data input, the prediction encoding unit, and the one-frame image data on the basis of the difference information between the previous frame image picture data of one frame image data, when the second generates a code data, the decoding of the second code data is the moving image reproducing apparatus performs estimating such process amount, the result of the estimation, based on the estimated throughput, the coding of the next frame picture image data of the one frame image data, said single image encoder and pre A video encoding program for causing a video coding for determining whether to perform any of the predictive coding unit to the computer. By doing so, it is possible to minimize the problem of discontinuous image display during frame skipping.

The generation of the second code data by predictive coding unit, the previous frame image data the one frame image and image data obtained by decoding by the local decoder the code data corresponding to the The amount of processing that is performed by generating code data based on the difference information with respect to the image data and that is required when the moving image reproduction device performs decoding is the amount of processing that is required for the decoding processing by the local decoding unit It is good also as performing by estimating based on. By doing in this way, decoding performed at the time of encoding can be used for load determination, and encoding time can be shortened.

Before based on a comparison result between the processing amount and the predetermined value of the decoding process by km over local decoding unit, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit it may be characterized that you decide whether to perform either. By doing so, it is possible to cope with a change in the moving image reproduction apparatus.

  According to the present invention, it is possible to provide a moving image encoding apparatus, a moving image encoding method, and a moving image encoding program that minimize the problem of discontinuous image display due to frame skipping during decoding.

Embodiment 1 of the Invention
FIG. 1 is a diagram showing a configuration of a moving picture coding apparatus according to an embodiment of the present invention. The moving image encoding apparatus 1 includes an encoding processing unit 10, a load estimation unit 11, a control unit 12, an input image storage unit 13, and an output image storage unit 14.

  The encoding processing unit 10 performs code data creation processing for converting image data input from the input image storage unit 13 into compressed code data. The encoding processing unit 10 includes a single image encoding unit 100, a predictive encoding unit 101, and a local decoding unit 102. The single image encoding unit 100 encodes information of only the image of the frame. A frame that is encoded by the single image encoding unit 100 is referred to as an I frame. The predictive encoding unit 101 detects difference information between the image of the frame and another frame, and encodes the motion information to reduce the amount of encoded data. A frame that is encoded by the predictive encoding unit 101 is referred to as a P frame. The local decoding unit 102 decodes the code data encoded by the single image encoding unit 100 and the predictive encoding unit 101. The image data decoded by the local decoding unit 102 is used as target image data whose difference information is detected by the predictive encoding unit 101. The encoding processing unit 10 performs encoding processing by a dedicated processor, or a CPU (Central Processing Unit) performs this processing by software using a program.

  The load estimator 11 estimates a processing amount when decoding the code data encoded by the encoding processor 10. It is preferable to estimate the processing amount by actually decoding and measuring the processing amount at that time, and estimating the processing amount based on the measurement result. However, the processing amount may be estimated from the data structure of the encoded code data.

  The control unit 12 determines whether a frame to be encoded is an I frame or a P frame. As a determination method, when the amount of change is large, such as screen switching, or when the estimation result of the processing amount by the load estimation unit 11 is equal to or larger than a predetermined value, the code data of the next frame is changed to I frame code data. There is. The method of forcing the code data of the next frame into code data of the I frame is to send a compulsory signal to the encoding processing unit 10, and when the encoding processing unit 10 receives the compulsory signal, the next frame is changed to an I frame. There is a method of acquiring I-frame code data on the control unit 12 side and directly writing the data.

  The input image storage unit 13 is a memory for storing moving image data to be encoded, and includes a RAM (Random Access Memory) or the like. The moving image data at this time includes those shot by a video camera or the like, or created by CG (Computer Graphics), but the content is not particularly limited.

  The output image storage unit 14 is a memory for storing encoded data, and is configured by a RAM or the like in the same manner as the input image storage unit 13. Alternatively, a configuration in which data is directly written to an HDD (Hard Disk Drive), a DVD (Digital Versatile Disc), or the like may be used.

  Next, the flow of processing in the embodiment of the present invention will be described using the flowchart shown in FIG. It is assumed that moving image data to be encoded is stored in the input image storage unit 13 in advance. Here, the frames constituting the moving image are only the I frame and the P frame.

  First, the encoding processing unit 10 inputs image data for one frame from the input image storage unit 13 (S101). Subsequently, the control unit 12 determines whether the input image data is encoded with the I frame or the P frame (S102). The determination method at this time is not particularly defined in the MPEG standard and is not limited, but is usually an I frame when the amount of change is large due to a screen change or the like, and a P frame in other cases, and at equal intervals. Encoding with I frames is performed.

  When it is determined that the encoding with the I frame is performed (S103), the single image encoding unit 100 performs the encoding only with the frame image information (S104). When encoding by P frame is performed, the prediction encoding unit 101 detects difference information between image information of another frame and image information of the frame, and encodes the detected difference information to reduce the amount of data. Reduce (S105).

  Encoding with P frames often imposes a load on the processor during decoding due to the amount of difference calculation. If the load becomes too heavy at the time of decoding, the calculation takes time, and the image display of the frame at that time is not in time and frame skip occurs. Therefore, in the present invention, the load estimation unit 11 estimates the processing amount at the time of decoding immediately after encoding or immediately after encoding (S106). The processing amount estimation method will be described in detail later.

  Here, the encoding method will be briefly described. In both the case of encoding by I frame and the case of encoding by P frame, the input data is subjected to an operation called DCT (Discrete Cosine Transform) and converted to DCT coefficients. Furthermore, the amount of information is reduced by approximating and omitting a part of the numerical values of the converted DCT coefficients. This approximation process is called quantization. For example, by dividing each DCT coefficient by 8 and rounding after the decimal point, it is possible to reduce the information amount of each DCT coefficient by 3 bits. By performing entropy coding in which short codes are assigned to values having a high appearance frequency for the DCT coefficients quantized in this way, coding with a small amount of data becomes possible. In the case of the I frame, this series of processing is performed on the image data constituting one screen, and in the case of the P frame, the difference from the image information of the previous frame is extracted, and this series of processing is performed on the difference information. Do.

  When the encoding with the I frame or the P frame is completed, the encoding processing unit 10 outputs the code data to the output image storage unit 14 (S107). When the output of the code data is completed, the process proceeds to encoding of the next frame image (S108). Here, when the output code data is P frame code data, the next code frame is determined according to the measured amount of decoding processing. Specifically, when the processing amount is larger than a predetermined value (S109), the control unit 12 outputs an I frame signal to the encoding processing unit 10. When the I-frame signal is input from the control unit 12, the encoding processing unit 10 determines to perform encoding with the I frame without determining whether the next frame is to be an I frame or a P frame, and Data is input (S110) and encoding is performed. By doing this, even if the frame skip occurs because the calculation is not in time during decoding, the next frame is the I frame, so only one frame skip is required, minimizing the problem of discontinuous image display during video playback. It becomes possible to suppress.

  In this way, by repeating the encoding of the image data for each frame, finally, the code data of the image data of all the frames is stored in the output image storage unit. Encoding of moving image data is completed by adding necessary information such as header information to these frame image data. When the moving image data created in this way is decoded and reproduced as a moving image, even when the decoding is performed by a decoding device having a processing capability lower than that of the encoded device, the discontinuous image at the time of reproducing the moving image is used. It is possible to minimize display defects.

  Next, a processing amount measurement method at the time of decoding will be described. In moving picture coding by MPEG, decoding called local decoding is performed when encoding is performed. Decoding by local decoding is performed by the local decoding unit 102. The local decoding is to perform decoding of actually encoded data in order to obtain a comparison target image when encoding for detecting a motion such as a P frame is performed. An error caused by an approximation process at the time of encoding occurs in an image obtained by decoding the input image before encoding and the encoded image. Since the image displayed at the time of reproduction is an image obtained by decoding the encoded image, the encoded data is decoded without using the image before encoding as a comparison target image, and difference information is obtained. A comparison target image for detection is obtained.

  As a decoding method, the encoding method may be reversed. That is, entropy decoding of entropy-encoded data is performed, and the data is inversely quantized. However, since it is approximated data, it cannot be completely restored. Decoding is completed by performing inverse DCT operation on the DCT coefficients obtained by inverse quantization. The data decoded at this time is image data of one screen in the case of an I frame, and is difference data in the case of a P frame. Whether the frame is an I frame or a P frame can be known from the header information.

  Decoding performed during playback is performed in principle in the same manner as decoding performed during local decoding. However, when decoding is performed during playback, correction may be added in consideration of display effects such as anti-aliasing. The amount of decoding processing performed at the time of reproduction and the amount of decoding performed at the time of local decoding does not always match completely. Even when the decoding performed at the time of reproduction and the decoding performed at the time of local decoding are performed by the same algorithm, the processing amounts do not completely coincide with each other if the respective processing capabilities are different. However, even if they do not completely match, it has been confirmed that the correlation of the processing amount is quite high, and it is possible to estimate the decoding processing amount performed during reproduction from the decoding processing amount performed during local decoding. It is.

  FIG. 4 is a graph obtained by measuring and comparing the processing amounts of decoding performed at the time of reproduction and decoding performed at the time of local decoding in moving image data prepared as samples. For the measurement, encoding and decoding were performed using a dedicated simulator, and the number of execution cycles of local decoding processing for each frame during encoding and the number of execution cycles of decoding processing during decoding were measured. The measured value was determined by measuring the number of execution cycles of the inverse DCT process, the inverse zigzag scan process, the inverse quantization process, and the result data memory writing process. Further, the scale value of the graph is changed between decoding performed at the time of reproduction and decoding performed at the time of local decoding so that the correlation can be easily confirmed. Also from the result shown in this graph, it can be confirmed that the correlation between the decoding amount performed at the time of reproduction and the decoding processing amount performed at the time of local decoding is quite high.

  Therefore, the processing weight for actual decoding is estimated by measuring the processing amount of local decoding. The amount of processing is measured by counting the number of times local decode instructions are actually executed. At this time, if the instruction processing system of the moving image reproducing apparatus is different from the instruction processing system of the moving image encoding apparatus, the measurement of the amount of processing can be improved by performing the measurement in consideration thereof. For example, when the moving image reproducing apparatus can perform parallel processing of two lines, the number of instruction executions at a place where parallel processing can be performed is counted as half. Since local decoding is a process that is performed for encoding, using the processing amount of local decoding makes processing more efficient than when performing only the processing weight at the time of decoding. Can be estimated.

  Based on the processing amount estimated at this time, it is determined whether or not the next frame is to be an I frame. If the amount of processing is large, there is a risk that processing will not be in time for decoding and frame skipping may occur, so the next frame is forcibly set to I frame. Since the reference value of the processing amount at this time varies depending on the processing capability on the decoding side, it can be arbitrarily set.

  In this way, the processing amount for encoding is estimated using local decoding performed at the time of code data generation, and it is determined whether the next frame is an I frame or a P frame based on the estimated processing amount. As a result, it is possible to reduce the problem of discontinuous image display due to frame skip when decoding encoded data and reproducing a moving image. In addition, local decoding is a process necessary for encoding processing. The amount of processing performed at the time of local decoding and the processing performed at the time of encoding difference information between image data obtained by local decoding and input image data. Since there is no significant difference in the processing amount, it is possible to efficiently perform encoding without generating unnecessary waiting time.

Embodiment 2 of the Invention
The present invention can also be applied to moving image transmission by long-distance wireless communication such as a TV phone system. FIG. 3 is a diagram illustrating a configuration of a moving image transmission system using long-distance wireless communication. The moving image transmission system here includes a transmission side terminal 2, a reception side terminal 3, and a base station 4. The transmission side terminal 2 is a side that transmits moving images, and includes moving image encoding means and wireless communication means. The details of the moving image encoding means are the same as those of the first embodiment of the invention, and the description thereof is omitted here.

  When the moving picture coding means provided in the transmission side terminal 2 performs moving picture coding, the processing amount serving as a reference for forcing the next frame to be an I frame is the processing amount at the time of decoding of the receiving side terminal 3 The reference value is set so that it is determined in consideration of In this way, when decoding is performed by the receiving side terminal 3 and a moving image is played back, even if a frame skip occurs, the next frame is an I frame, so that the problem of discontinuous image display is minimized. It becomes possible to suppress to.

  The moving image data encoded by the transmission side terminal 2 is transmitted to the reception side terminal 3 via the base station 4 by wireless communication means. The receiving side terminal 3 also includes wireless communication means, and receives the moving image data transmitted by the transmitting side terminal 2. The received moving image data is decoded by decoding means provided in the receiving side terminal 3, and the moving image can be reproduced. The wireless communication means at this time may be wireless communication using the Internet or wireless communication using a mobile phone network, and is not particularly limited.

Other Embodiments of the Invention
In the above-described example, encoding including only an I frame and a P frame is performed. However, encoding may be performed by adding a B frame thereto.
In the above example, the encoding side and the decoding side are separate devices. However, the same device may be used for encoding and decoding.

It is a block diagram which shows the structure of the moving image encoder in this invention. It is a flowchart which shows the flow of a process of the moving image encoder in this invention. It is a figure which shows the structure of the moving image transmission system by long-distance wireless communication in this invention. It is the graph which compared the processing amount of the decoding performed at the time of reproduction | regeneration performed at the time of reproduction | regeneration in this invention at the time of local decoding.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving image encoder 2 Transmission side terminal 3 Reception side terminal 4 Base station 10 Encoding process part 100 Single image encoding part 101 Predictive encoding part 102 Local decoding part 11 Load estimation part 12 Control part 13 Input image memory | storage part 14 Output image storage

Claims (10)

The code data that can be decoded and reproduced by the moving image reproducing apparatus comprising a video coding device that generates,
And a single image encoder for generating a first code data from one frame image data among the plurality of frame image picture data included in the input moving image data,
On the basis of the difference information between the one frame image data and the one of the previous frame image picture data of the frame image data, and the prediction encoding unit for generating a second code data,
A load estimator for estimating a processing amount required when the moving image reproducing apparatus performs decoding of the second code data;
Based on prior Ki推 constant has been processed quantity, the encoding of the next frame picture image data of the one frame image data to determine whether to perform either of the single image encoder and the prediction encoding unit A control unit ;
A video encoding device comprising: The predictive encoding unit includes a local decoder for decoding at least the second sign-data,
The moving image according to claim 1, wherein the load estimation unit estimates a processing amount when the moving image reproduction apparatus performs decoding based on a processing amount applied to a decoding process by the local decoding unit. Image encoding device. Wherein, prior Ki推 based on the comparison result of the constant has been processed amount with a predetermined value, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit moving picture encoding apparatus according to claim 1 or 2, wherein the determining one in or perform the. 4. The moving image code according to claim 1, wherein the first code data is data encoded by an I frame, and the second code data is data encoded by a P frame. Device. The code data that can be decoded and reproduced by the moving image reproducing apparatus a moving picture coding method that generates,
By a single image encoding unit to generate a first encoded data from one frame image data among the plurality of frame image picture data included in the moving image data input,
The predictive encoding unit, on the basis of the difference information between the one frame image data and the one of the previous frame image picture data of the frame image data to generate second encoded data,
Estimating the amount of processing required when the moving image reproduction device performs decoding of the second code data,
Results of the estimation or based on the estimated throughput, the coding of the next frame picture image data of the one frame image data, performed by any of the single image encoder and the prediction encoding unit A moving picture encoding method to be determined. The generation of the second code data by predictive coding unit, the previous frame image data the one frame image and image data obtained by decoding by the local decoder the code data corresponding to the Is performed by generating code data based on difference information with image data,
6. The moving image according to claim 5 , wherein estimation of a processing amount when the moving image reproducing apparatus performs decoding is performed by estimating based on a processing amount required for a decoding process by the local decoding unit. Image coding method. Before based on a comparison result between the processing amount and the predetermined value of the decoding process by km over local decoding unit, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit video encoding method of claim 6, wherein that you decide whether to perform either. The code data that can be decoded and reproduced by the moving image reproducing apparatus a moving picture encoding program for causing a moving picture coding method that generates the computer,
By a single image encoding unit to generate a first encoded data from one frame image data among the plurality of frame image picture data included in the moving image data input,
The predictive encoding unit, on the basis of the difference information between the one frame image data and the one of the previous frame image picture data of the frame image data to generate second encoded data,
Estimating the amount of processing required when the moving image reproduction device performs decoding of the second code data,
Results of the estimation or based on the estimated throughput, the coding of the next frame picture image data of the one frame image data, performed by any of the single image encoder and the prediction encoding unit A moving picture coding program for causing a computer to perform moving picture coding to be determined. The generation of the second code data by predictive coding unit, the previous frame image data the one frame image and image data obtained by decoding by the local decoder the code data corresponding to the Is performed by generating code data based on difference information with image data,
9. The moving image according to claim 8 , wherein the amount of processing required when the moving image reproduction device performs decoding is estimated by estimating based on the amount of processing required for the decoding processing by the local decoding unit. Image encoding program. Before based on a comparison result between the processing amount and the predetermined value of the decoding process by km over local decoding unit, the encoding of the next frame picture image data, said single image encoder and the prediction encoding unit claim 9, wherein the moving picture encoding program characterized that you decide whether to perform either.

Images