JP4127799B2 - Decoding and playback device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a decoding / playback apparatus that inputs and plays back encoded information, and more particularly to correction processing for errors in playback time.
[0002]
[Prior art]
In a multimedia information processing system that encodes, transmits, decodes and reproduces a multimedia program composed of a moving image and sound using the MPEG method, the moving image and sound are encoded for each unit called a frame. Is added to the playback time (Presentation Time Stamp, hereinafter referred to as PTS) and transmitted to the decoding / playback apparatus. A frame for a moving image is a still image that is continuous in time, and a frame for audio is audio that is divided at regular time intervals.
[0003]
In order to reproduce the encoded moving image and sound in synchronization, the encoded frame is decoded, and both the decoded frame and the PTS added to the encoded frame are stored in the memory. To remember. Then, the frame stored in the memory is reproduced when the time indicated by the reference synchronization signal (System Time Clock, hereinafter referred to as STC) of the decoding / reproducing apparatus coincides with the time of the PTS.
[0004]
Here, the STC is always calibrated so as to indicate the time indicated by a program time reference value (Program Clock Reference, hereinafter referred to as PCR) added to the encoded multimedia program.
[0005]
By the way, for example, in a system that transmits and reproduces encoded multimedia information in real time using a mobile communication terminal device, an error may occur in the PTS during transmission of the encoded information. In this case, the error is not input to the decoding / playback apparatus without being corrected by the transmission apparatus.
[0006]
This is because a system for transmitting and reproducing encoded multimedia information in real time using a mobile communication terminal device has a low transmission bit rate, and thus is used for robust error correction used in digital television broadcasting systems. Redundant bits cannot be added, and because the UDP (User Datagram Protocol) protocol that does not perform retransmission is used, the retransmission function of the TCP (Transmission Control Protocol) protocol used on the Internet may be used. This is because it cannot be done.
[0007]
Here, since the encoded moving image has a larger amount of information than the encoded audio, the PTS error is more likely to be mixed into the encoded moving image information than the encoded audio information.
[0008]
Therefore, in a moving image decoding / reproducing method for a system that transmits and reproduces encoded multimedia information in real time using a mobile communication terminal device, the moving image is audio even if an error is included in the PTS. It is necessary to correct the error in order to play back in synchronization with. As a correction method, time information different from PTS is added to an encoded image frame, and PTS error detection and correction is performed using the time information (for example, Patent Document 1). reference.).
[0009]
[Patent Document 1]
JP 2002-135777 A (page 2-3, FIG. 2)
[0010]
[Problems to be solved by the invention]
However, in the conventional method described above, when an error is included in the PTS added to a plurality of consecutive encoded frames, the error is included in the PTS added to the subsequent encoded frame. There is a problem that a large amount of calculation is required to determine whether or not it has been.
[0011]
The present invention has been made to solve the above-described problems, and an object of the present invention is to provide a decoding / playback apparatus that can determine that an error is not included in a PTS with a small amount of calculation.
[0012]
[Means for Solving the Problems]
In order to achieve the above object, the decoding / reproducing apparatus of the present invention reproduces at least one frame signal composed of an image signal encoded in frame units and time information set in frame units, and an image signal. In a decoding / playback apparatus that receives a packet including playback time information to be used when performing playback, and plays back an image signal based on the playback time information, playback time information included in past packets determined to have no error And time information storage means for storing time information, detection means for detecting a reproduction time information reset signal, and when the detection means does not detect a reproduction time information reset signal, the reproduction time information included in the received packet, The reproduction time information stored in the time information storage means is compared with the reproduction time information included in the received packet. When indicating a time before the stored reproduction time information, an error determination means for determining an error in the reproduction time information included in the received packet, and when this error determination means determines that there is an error in the reproduction time information, A difference value between the time information stored in the time information storage unit and the time information included in the received packet is obtained, and the difference value is added to the reproduction time information stored in the time information storage unit to reproduce the reproduction time. Reproduction time estimation means for estimating the reproduction time information, and when the detection means detects a reproduction time information reset signal or when the error determination means does not detect an error in the reproduction time information, it is based on the reproduction time information included in the received packet. When the error determination unit detects an error in the reproduction time information, the reproduction time estimated by the reproduction time estimation unit is reproduced. Characterized by comprising a reproduction control means for reproducing the image signal based on the information.
[0013]
In addition, the decoding / reproducing apparatus of the present invention uses at least one frame signal composed of an image signal encoded in units of frames and time information set in units of frames and a reproduction used when reproducing the image signals. In a decoding / reproducing apparatus that receives a packet including time information and reproduces an image signal based on the reproduction time information, the reproduction time information and time information included in past packets determined to have no error are stored. Time information storage means, detection means for detecting a reproduction time information reset signal, and when the detection means does not detect a reproduction time information reset signal, the reproduction time information contained in the received packet and the time information storage means Stored in the time information storage means and the time information included in the received packet A second difference value with respect to the received time information, and if the difference between the first difference value and the second difference value exceeds a predetermined value, it is determined as an error in the reproduction time information included in the received packet When the determination unit and the error determination unit determine that there is an error in the reproduction time information, a reproduction time for estimating the reproduction time by adding the second difference value to the reproduction time information stored in the time information storage unit When the estimation unit and the detection unit detect a reproduction time information reset signal or when the error determination unit does not detect an error in the reproduction time information, the image signal is reproduced based on the reproduction time information included in the received packet. And a reproduction control means for reproducing an image signal based on the reproduction time information estimated by the reproduction time estimation means when the error determination means detects an error in the reproduction time information. It is characterized in.
[0014]
According to the present invention, when the encoding scheme information is correctly input and when a time reset instruction is input to the means for measuring the playback time of the decoded information, it is played back with a small amount of calculation. It can be determined that there is no error in the time.
[0015]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of a decoding / playback apparatus according to the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing a configuration of a mobile communication terminal apparatus to which a decoding / playback apparatus according to an embodiment of the present invention is applied.
[0016]
The mobile communication terminal device includes a control device 1 that controls the entire device, an antenna 2a, a communication device 2b, a transmission / reception device 3, an input device 4 that inputs an operation instruction from a user, a moving image and audio. The encoding / decoding device 5 that performs encoding and decoding / reproduction of a multimedia program comprising: a camera 6a, a display device 6b, a microphone 6c, and a speaker 6d.
[0017]
The communication device 2b outputs the high frequency signal received by the antenna 2a to the transmission / reception device 3, and transmits the high frequency signal output from the transmission / reception device 3 from the antenna 2a.
[0018]
Next, the operation of the transmission / reception device 3 will be described. The transmission / reception device 3 amplifies, frequency-converts and demodulates the high-frequency signal from the communication device 2b, sends the encoded multimedia program signal obtained thereby to the encoding / decoding device 5, and transmits the control signal to the control device. Send to 1. The transmission / reception device 3 further modulates, converts and amplifies the encoded multimedia program signal output from the encoding / decoding device 5 and the control signal output from the control device 1 to obtain a high-frequency signal. And send it to the communication device 2b.
[0019]
FIG. 2 is a block diagram showing the configuration of the encoding / decoding device 5. The encoding / decoding device 5 includes a demultiplexing unit 11, a moving image encoding unit 12, a moving image decoding unit 13, an image buffer 14, an image reproduction control unit 15, a time management unit 21, and an STC unit. 22, an audio encoding unit 23, an audio decoding unit 24, an audio buffer 25, and an audio reproduction control unit 26.
[0020]
The operation of the mobile communication terminal apparatus configured as described above according to the embodiment of the present invention will be described.
[0021]
First, the operation of the demultiplexing unit 11 will be described. The demultiplexing unit 11 multiplexes the encoded image frame signal output from the moving image encoding unit 12 and the encoded audio frame signal output from the audio encoding unit 23, and obtains the obtained encoding The multimedia program signal thus transmitted is sent to the transmission / reception device 3.
[0022]
That is, the demultiplexing unit 11 creates a PES (Packetized Elementary Stream) packet by combining a plurality of encoded image frame signals output from the moving image encoding unit 12 with respect to the moving image input from the camera 6a. To do.
[0023]
The time at which each image frame called B_TS is reproduced is added to the frame header information in all the encoded image frame signals output from the moving image encoding unit 12, and a part of the encoding is performed. In the image frame signal, information called CI (Configuration Information) (information indicating the coding format of the moving image), for example, the coding profile, the size of the image, the size of the required image buffer 14, etc. Are added to the PES packet. B_TS is a time defined independently for moving images and unrelated to audio.
[0024]
The demultiplexing unit 11 further adds a PTS that is the time to reproduce the image frame signal to the encoded image frame signal at the head of the PES packet to create a PES packet. Note that PTS may be added to other encoded image frame signals.
[0025]
FIG. 3A shows a situation in which the first encoded image frame signal 31a to the mth encoded image frame signal 31b constitute one PES packet (image).
[0026]
Here, B_TS is added to frame header information in all encoded image frame signals. That is, B_TS (31c) is added to the first encoded image frame signal 31a, and B_TS (31d) is added to the m-th encoded image frame signal 31b in the same manner. ing.
[0027]
Further, CI (31e) and PTS (31f) are further added to the first encoded image frame signal 31a.
[0028]
The demultiplexing unit 11 further creates a PES packet by collecting a plurality of encoded audio frame signals output from the audio encoding unit 23 with respect to the audio input from the microphone 6c. Also, PTS, which is the time to reproduce the audio frame, is added to the encoded audio frame signal at the head of the PES packet. Note that PTS may be added to other encoded audio frame signals.
[0029]
FIG. 3 (b) shows a situation where the first encoded audio frame signal 32a to the nth encoded audio frame signal 32b constitute one PES packet (audio). Here, PTS (32c) is added to the first encoded audio frame signal 32a.
[0030]
Subsequently, the demultiplexing unit 11 divides these image PES packets and audio PES packets into fixed-length TS (Transport Stream) packets for each packet, and further adds a PCR to each TS packet to transmit / receive apparatus 3. Send to. FIG. 3C shows a situation in which one PES packet is divided from the first TS packet 33a to the pth TS packet 33b, and a PCR is added to each TS packet. That is, PCR (33c) is added to the first TS packet 33a, and PCR (33d) is added to the p-th TS packet 33b.
[0031]
The demultiplexing unit 11 separates the encoded multimedia program signal output from the transmission / reception device 3 into an encoded image frame signal, an encoded audio frame signal, and a PCR. Then, the encoded image frame signal is sent to the moving picture decoding unit 13, the encoded audio frame signal is sent to the audio decoding unit 24, and the PCR is sent to the time management unit 21.
[0032]
That is, the multimedia program signal output from the transmission / reception device 3 is composed of a plurality of TS packets having the structure of FIG. 3 (c), and receives a signal to which PCRs 33c,. Of these, PCRs 33c,..., 33d are separated and sent to the time management unit 21.
[0033]
Further, the demultiplexing unit 11 separates the TS packet and integrates it into the PES packet, and if the PES packet is a packet made up of the encoded image frame signal shown in FIG. The encoded image frame signal including PTS, CI, and B_TS is sent to the moving image decoding unit 13.
[0034]
If the PES packet is a packet composed of the encoded audio frame signal shown in FIG. 3B, the PTS included in the PES packet and the encoded audio frame signal are sent to the audio decoding unit 24.
[0035]
Next, the operation of the moving image encoding unit 12 will be described. The moving image encoding unit 12 first creates an image frame signal, that is, a temporally continuous still image, from the moving image signal output from the camera 6a.
[0036]
Subsequently, the obtained image frame signal is subjected to discrete cosine transform, and the obtained discrete cosine transform coefficient is quantized and encoded. Subsequently, a time B_TS for reproducing it is added to the frame header information in the obtained encoded image frame signal, and CI is added to some of the encoded image frame signals to multiplex Send to separation unit 11.
[0037]
Next, the operation of the moving picture decoding unit 13 will be described. FIG. 4 shows a flowchart of the operation of the video decoding unit 13. The moving picture decoding unit 13 starts its operation (step 41a), and receives the encoded picture frame signal output from the demultiplexing unit 11. At this time, the PTS, CI, and B_TS added to the encoded image frame signal are also received simultaneously (step 41b).
[0038]
Subsequently, the received encoded image frame signal is decoded according to the encoding method indicated by the CI. That is, the encoded image frame signal is decoded, and the decoded signal is inversely quantized and then subjected to inverse discrete cosine transform to obtain a decoded image frame signal, that is, a still image. The data is stored in the buffer 14 (step 41c).
[0039]
Subsequently, the moving image decoding unit 13 adds a reproduction time PTS to the obtained decoded image frame signal. Therefore, first, it is checked whether or not PTS is added to the received encoded image frame signal (step 41d). If added, it is checked whether an error is mixed in the added PTS.
[0040]
That is, first, if a CI is added to the received encoded image frame signal, it is added to the encoded image frame signal received in the past and stored in the moving image decoding unit 13. It is checked whether it is the same as the current CI (step 41e). If they are the same, the CI is correct, so it is determined that there is no error in the PTS added to the received encoded image frame signal. This is because the CI and the PTS are transmitted as the same or adjacent TS packets, so that when one is correctly transmitted, the other can be regarded as correctly transmitted.
[0041]
Subsequently, the PTS and the image frame signal decoded in step 41c are stored in the image buffer 14 (step 41f).
[0042]
If it is determined in step 41e that the correct CI has not been added, the moving picture decoding unit 13 checks whether or not an STC reset signal has been received from the time management unit 21 (step 41g). If the STC reset signal is received, it is determined that there is no error in the PTS, and the PTS and the image frame signal decoded in step 41c are stored in the image buffer 14 (step 41f).
[0043]
This is because when an STC reset signal is received from the time management unit 21, it means that a certain program ends and the first encoded image frame of another program is sent from the demultiplexing unit 11. Since the continuity of the PTS added to the received encoded image frame signal with the past PTS is lost and error detection based on the continuity described later becomes impossible, the added PTS is regarded as correct. This is because it is appropriate.
[0044]
If the STC reset signal has not been received in step 41g, the video decoding unit 13 adds the PTS added to the received encoded image frame signal and the encoded image frame signal received in the past. And the error-free PTS stored in the moving image decoding unit 13 is compared to verify whether there is an error in the PTS added to the received encoded image frame signal (step 41h). ).
[0045]
For example, two criteria are used as the verification criteria of step 41h. As a first criterion, when it is agreed that the image frame signals are sent in the order of reproduction, the PTS added to the received encoded image frame signal is compared with the past error-free PTS. If so, the added PTS is determined to be an error.
[0046]
Further, as a second reference, the difference between the PTS added to the received encoded image frame signal and the PTS without error in the past, the B_TS added to the received encoded image frame signal, and the moving image The difference between B_TS added to the encoded image frame signal added with error-free PTS stored in the image decoding unit 13 is compared, and the difference between the two differences is a predetermined number. If it is above, PTS judges that it is an error.
[0047]
If none of these criteria is met in step 41h, the video decoding unit 13 determines that there is no error in the PTS, and the image frame signal decoded in step P41 and the image frame signal decoded in step 41c. Is stored in the image buffer 14 (step 41f). At this time, the PTS and B_TS are stored in the video decoding unit 13 as an error-free PTS and B_TS added to the encoded image frame signal to which the PTS is added.
[0048]
On the other hand, when it is determined in step 41d that no PTS is added to the received encoded image frame signal, and in step 41h, there is an error in the PTS added to the received encoded image frame signal. If it is determined that there is, the video decoding unit 13 estimates the PTS corresponding to the received encoded image frame signal, and the image buffer 14 uses the PTS and the image frame signal decoded in step 41c. (Step 41i).
[0049]
Here, the estimated PTS value for the received encoded image frame signal is added to the encoded image frame signal to which B_TS added to the image frame signal and the error-free PTS are added. The difference from B_TS is obtained and added to the past error-free PTS.
[0050]
After the PTS of the image frame signal is stored in the image buffer 14 in step 41f or step 41i, the moving image decoding unit 13 returns to step 41b (step 41j), and processes the next image frame.
[0051]
Next, the operation of the image reproduction control unit 15 will be described. The image reproduction control unit 15 reads the time indicated by the STC unit 22 and searches whether the PTS at the same time as the read time is stored in the image buffer 14. If it is stored, the decoded image frame signal stored in the image buffer 14 together with the PTS, that is, the still image is read out and displayed on the display device 6b.
[0052]
Next, the operation of the time management unit 21 will be described. The time management unit 21 receives the PCR output from the demultiplexing unit 11 and calibrates the STC unit 22 so that the STC unit 22 indicates the time indicated by the PCR. When the time management unit 21 detects an STC reset, it sends a signal indicating the STC reset to the moving picture decoding unit 13 and the voice decoding unit 24.
[0053]
Here, the detection of the STC reset in the time management unit 21 is because the time indicated by the PCR changes stepwise before and after a certain time point with a difference of a predetermined value or more. Alternatively, the PCR may include an identifier indicating whether or not the program has been changed, and the identifier may indicate that the program has been changed.
[0054]
Next, the operation of the speech encoding unit 23 will be described. The voice encoding unit 23 first creates a voice frame signal, that is, a voice signal divided at regular time intervals from the voice signal output from the microphone 6c.
[0055]
Subsequently, the audio encoding unit 23 performs time-frequency conversion on each obtained audio frame signal by an AAC (Advanced Audio Coding) method. Subsequently, the obtained MDCT (Modified Discrete Cosine Transform) coefficient is quantized and encoded. The addition of PTS is as already described. Subsequently, the obtained encoded audio frame signal is sent to the demultiplexing unit 11.
[0056]
Next, the operation of the speech decoding unit 24 will be described. The speech decoding unit 24 first receives the encoded speech frame signal output from the demultiplexing unit 11. At that time, the PTS added thereto is also received. Subsequently, the received encoded audio frame signal is decoded, and the decoded signal is inversely quantized and subjected to inverse time frequency conversion, thereby decoding the decoded audio frame signal, that is, an audio signal having a certain time width. And the signal is stored in the audio buffer 25.
[0057]
Subsequently, the audio decoding unit 24 adds the time PTS to be reproduced to the decoded audio frame signal, similarly to the moving image. If a PTS is added to the received encoded audio frame signal, the PTS and the decoded audio frame signal are stored in the audio buffer 25.
[0058]
If no PTS is added to the audio frame signal, a predetermined estimated value is added to the PTS added to the previously encoded audio frame signal to obtain the PTS. That is, the number of frames k (k = m−1) between the encoded speech frame (frame m) currently received from the past encoded speech frame (frame 1) is obtained, and 1 is added to the number of frames k. The estimated value is calculated by adding the number (k × t) obtained by multiplying the time width t of the speech included in one speech frame. The PTS thus obtained and the decoded audio frame signal are stored in the audio buffer 25.
[0059]
Next, the operation of the audio reproduction control unit 26 will be described. The audio reproduction control unit 26 reads the time indicated by the STC unit 22 and searches whether the PTS at the same time as the read time is stored in the audio buffer 25. If it is stored, the decoded audio frame signal stored in the audio buffer 25 together with the PTS, that is, the audio signal having a predetermined time width is sent to the speaker 6d for sound generation.
[0060]
In the present embodiment, error detection of the PTS added to the encoded speech frame is not performed, but it may be performed. Further, although the moving image is one moving image, the moving image may be defined as a collection of a plurality of moving images having an arbitrary shape. In that case, the present invention can be applied to decoding and reproduction performed for each moving image of any arbitrary shape.
[0061]
Furthermore, although the encoding of the image frame is based on the discrete cosine transform method, it is not limited to this, and the encoding of the audio frame is based on the AAC method, but is not limited thereto. The present invention is not limited to the above configuration, and various modifications are possible.
[0062]
Moreover, although the above description has been made taking an example in which the present invention is applied to a mobile communication terminal device, the present invention can naturally be applied to a receiving device of a broadcasting system.
[0063]
【The invention's effect】
As described above, according to the present invention, when a CI (Configuration Information) that is added to an encoded frame and represents the format of the encoding is correctly received, and the reference synchronization signal of the decoding / playback apparatus When STC (System Time Clock) is reset, it is determined that the PTS received with a small amount of calculation is correct by determining that the received frame playback time and PTS (Presentation Time Stamp) are correct. Can do.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a mobile communication terminal apparatus according to an embodiment of the present invention.
FIG. 2 is a block diagram showing an encoding / decoding device according to an embodiment of the present invention.
FIG. 3 is a diagram showing a data structure of an encoded multimedia program according to an embodiment of the present invention.
FIG. 4 is a flowchart showing an operation of a video decoding unit according to the embodiment of the present invention.
[Explanation of symbols]
5 Coding / Decoding Device 11 Demultiplexing Unit 13 Video Decoding Unit 15 Image Reproduction Control Unit 21 Time Management Unit 22 STC Unit 24 Audio Decoding Unit 26 Audio Reproduction Control Unit

Claims (3)

Receiving a packet including at least one frame signal composed of an image signal encoded in units of frames and time information set in units of frames, and reproduction time information used when reproducing the image signals; In a decoding / playback apparatus for playing back an image signal based on the playback time information,
Time information storage means for storing reproduction time information and time information included in past packets determined to have no errors;
Detecting means for detecting a reproduction time information reset signal;
When this detection means does not detect the reproduction time information reset signal, the reproduction time information included in the received packet is compared with the reproduction time information stored in the time information storage means, and the reproduction included in the received packet is compared. An error determination unit that determines that the reproduction time information included in the received packet is an error when the time information indicates a time earlier than the reproduction time information stored in the time information storage unit;
When the error determination means determines that there is an error in the reproduction time information, a difference value between the time information stored in the time information storage means and the time information included in the received packet is obtained, and the difference value is calculated as the time information. Reproduction time estimation means for estimating the reproduction time in addition to the reproduction time information stored in the information storage means;
When the detection means detects a reproduction time information reset signal or when the error determination means does not detect an error in the reproduction time information, an image signal is reproduced based on the reproduction time information included in the received packet, and the error A reproduction control unit that reproduces an image signal based on the reproduction time information estimated by the reproduction time estimation unit when the determination unit detects an error in the reproduction time information;
The decoding reproduction | regeneration apparatus characterized by the above-mentioned. Receiving a packet including at least one frame signal composed of an image signal encoded in units of frames and time information set in units of frames, and reproduction time information used when reproducing the image signals; In a decoding / playback apparatus for playing back an image signal based on the playback time information,
Time information storage means for storing reproduction time information and time information included in past packets determined to have no errors;
Detecting means for detecting a reproduction time information reset signal;
When the detection means does not detect the reproduction time information reset signal, the first difference value between the reproduction time information included in the received packet and the reproduction time information stored in the time information storage means, and the received packet When the second difference value between the time information included in the time information and the time information stored in the time information storage means is obtained, and the difference between the first difference value and the second difference value exceeds a predetermined value Determining means for determining an error in the reproduction time information included in the received packet;
When the error determination means determines that there is an error in the reproduction time information, reproduction time estimation means for estimating the reproduction time by adding the second difference value to the reproduction time information stored in the time information storage means;
When the detection means detects a reproduction time information reset signal or when the error determination means does not detect an error in the reproduction time information, an image signal is reproduced based on the reproduction time information included in the received packet, and the error A reproduction control unit that reproduces an image signal based on the reproduction time information estimated by the reproduction time estimation unit when the determination unit detects an error in the reproduction time information;
The decoding reproduction | regeneration apparatus characterized by the above-mentioned. The reproduction time information reset signal is an identifier indicating whether or not a program has been changed, and the detection means determines that the reproduction time information reset signal has been detected when the identifier indicates a program change. Item 3. A decoding reproduction device according to Item 1 or 2.

Images