JP4153754B2 - Image transmission apparatus, image transmission method, and server moving image supply method - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image transmission apparatus that transmits an encoded moving image from a transmitting side via a communication channel, decodes the received moving image code, and outputs a displayable moving image on the receiving side. The present invention relates to an image transmission apparatus that makes it difficult to cause time interruption when a moving image is displayed on the receiving side by reducing the amount of transmission code when the quality of a communication channel is temporarily deteriorated.
[0002]
[Prior art]
Conventionally, as shown in, for example, Japanese Patent Application Laid-Open No. 11-164297, even when the quality of a communication channel is temporarily deteriorated, a time interruption when displaying a moving image by reducing the transmission code amount is prevented. Wireless image communication apparatuses that are less likely to occur are known. In this apparatus, the line quality detection unit detects the line quality, and changes the hierarchization processing method of the moving image to be transmitted in accordance with this, and the image coding unit converts the processed image by changing the hierarchization processing method. By encoding, the bit rate of a moving image code to be transmitted is controlled. As the hierarchization process, the image size is changed or the contrast of the image is changed.
[0003]
In this way, in order to make it difficult for playback to be interrupted on the receiving side when the quality of the communication channel is deteriorated, it is effective to control the code generation rate to be lower than the transmission rate of the communication channel on the transmitting side. In a wireless communication path, the quality of the communication path may deteriorate so rapidly that it cannot be controlled, or it may deteriorate so that it cannot be transmitted temporarily. In such a case, the receiving side has a buffer, the received data is temporarily accumulated in the buffer and delayed, and when the arrival of data is delayed, the data in the buffer is reproduced to prevent interruption Is common. When the reproducible time is reduced with the data in the buffer, control is performed to recover the reduced amount. In other words, if the code generation rate is reduced according to the transmission rate of the communication channel, the number of frames of moving images that can be transmitted within the transmission rate of the communication channel increases, and the time that can be played back with the data in the buffer is also restored did it.
[0004]
Conventionally, a transmission control apparatus for recovering the data amount itself stored in the buffer is known as disclosed in Japanese Patent Laid-Open No. 10-313294. This is a transmission control device that changes the retransmission request method in accordance with the buffer amount of data in the delay difference absorption buffer provided in the receiver, and if the quality of the communication path deteriorates, the delay difference absorption buffer Utilizing the fact that the buffer amount of existing data is reduced, and not performing retransmission control when it is less than the predetermined transition value, recovering the buffer amount while allowing errors, and interrupting real-time data Make it difficult to occur.
[0005]
[Problems to be solved by the invention]
However, the above-described conventional wireless image communication apparatus requires a moving image encoding apparatus that changes a moving image hierarchization processing method to be transmitted to the transmission side. Since this device has a heavy processing load, a content distribution server that supplies moving image content to many clients requires an expensive device with high-speed processing capability, or a client that can be connected at once. There is a problem that the number must be limited to a small number.
[0006]
Further, in the conventional transmission control apparatus described above, since the received data includes an error by not performing retransmission control, the reproduction image quality deteriorates, and finally reproduction becomes impossible temporarily. In other words, the reproduced information is temporarily interrupted instead of the real-time data not being delayed. In real-time data, it is known that interruption or reduction in image frame rate significantly lowers subjective evaluation rather than quality degradation, so conventional transmission control devices have the most important reproduction information due to interruption. There is a problem of lowering the subjective evaluation.
[0007]
[Means for Solving the Problems]
In order to achieve the above object, the present invention provides a server that is a moving image supply unit that supplies a hierarchically encoded moving image code, and a base station that is a moving image transmission unit that transmits the supplied moving image code And a terminal device that is a moving image receiving unit that receives the transmitted moving image code, wherein the base station temporarily stores the moving image code supplied from the server And the server notifies the server of the amount of data, and the server provides two threshold values of different upper and lower limit values related to the supply of moving picture codes for each layer , and is notified from the base station There is provided an image transmission apparatus that compares a data amount of a buffer with each of the threshold values, and controls the supply of a moving image code of each layer based on the comparison result .
[0008]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, embodiments of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram of an image transmission apparatus showing a preferred embodiment of the present invention.
[0009]
In FIG. 1, 1 is a moving image supply unit, 2 is a moving image transmitting unit, 3 is a wireless communication channel, 4 is a moving image receiving unit, 5 is a first buffer, 6 is a moving image decoding unit, and 7 is a second moving image. Reference numeral 14 denotes an output unit (image display and audio output), and 8 denotes a communication path (connection path).
[0010]
In FIG. 1, the moving image supply unit 1 and the moving image transmission unit 2 may be connected as an integrated device, or may be installed separately through a reliable network. For example, both may be connected through an IP network using the well-known TCP as a transport layer protocol. In this embodiment, for the sake of explanation, each component is a base station that connects the moving image supply unit 1 to the content server, the moving image transmission unit 2 to the IP network, and services data communication to the wireless communication path 3. The moving picture receiving unit 4, the first buffer 5, the moving picture decoding unit 6 and the output unit 14 are provided in the terminal device. However, this is an example, and the present invention is not limited to these devices. The wireless communication path 3 may also be a wired communication path depending on the system.
[0011]
Next, each component will be described.
The moving image supply unit 1 stores a hierarchically encoded moving image code in an internal hard disk drive (not shown), reads out the stored hierarchically encoded moving image code, and transmits a moving image code Supplying to part 2.
[0012]
The quality of the wireless communication path 3 may be temporarily deteriorated, and during that time, the transmission speed decreases or is interrupted. However, a reliable wireless communication path 3 is formed between the moving image transmitting unit 2 and the moving image receiving unit 4 by detecting a transmission error and performing retransmission.
[0013]
The moving image code received by the moving image receiving unit 4 is temporarily stored in the first buffer 5. Even if the code to be received is interrupted due to transmission delay or communication channel quality deterioration, the code stored in the first buffer 5 is decoded in advance, thereby preventing the interruption of moving image reproduction. It is. The first buffer 5 stores moving image codes for a preset time according to the characteristics of the wireless communication path 3, and the moving image decoding unit 6 starts decoding after the accumulation is completed. However, in the present invention, the moving image code is hierarchically encoded, and the time for preventing interruption may be different for each layer. In the present embodiment, the base layer in this hierarchical encoding is the most important and the storage time is set long so that it is difficult to break. Further, in the present invention, upper layers other than the base layer are allowed to be frequently interrupted as communication quality deteriorates, so that the moving image supply unit 1 responds to communication quality deterioration (change in wireless communication path). Actively stop (or stop) supply. For this reason, the base layer code may be stored in the first buffer 5 for a long time, and the upper layer code may be stored for at least one frame. This is effective in preventing interruption as the time for storing the base layer code is longer. On the contrary, if the storage time is longer, the time from the start of transmission to image display becomes longer. However, there is an advantage that this problem can be alleviated by minimizing the accumulation of upper layer codes that are allowed to be interrupted, and that the storage capacity of the first buffer 5 can be reduced.
[0014]
In order to realize such accumulation, when the moving image supply unit 1 starts supplying moving image codes, the moving image supply unit 1 supplies only the codes of the base layer with priority for a predetermined time, and thereafter, the codes of the upper layer And a base layer code which is advanced in time.
[0015]
The moving image decoding unit 6 starts decoding the code stored in the first buffer 5 after the code for a predetermined time is accumulated in the first buffer 5. In the decoding, after decoding the base layer, if the upper layer code is accumulated, the decoding is performed and a high-quality moving image is output, but if the upper layer code is not present in the first buffer 5, Since the supply is stopped or the reception is delayed, a low-quality moving image decoded only in the base layer is output. The image size of the image decoded only in the base layer is the same size as the frame 31 of the original image (see the description of FIG. 2 described later), and the image quality is obtained by removing the high frequency component.
[0016]
The moving image output from the moving image decoding unit 6 is displayed and auditioned in the output unit 14.
The moving image transmitting unit 2 transmits the moving image code to the moving image receiving unit 4 through the wireless communication path 3. The moving image transmission unit 2 includes a second buffer 7 that temporarily stores a moving image code waiting for transmission when the quality of the wireless communication path 3 is deteriorated. The moving image receiving unit 4 performs error detection on the received data, and if there is no error, transmits an acknowledgment (ACK) notification to the moving image transmitting unit 2 if an error is detected. The moving image transmission unit 2 stores the moving image code in the second buffer 7 in preparation for retransmission until ACK is returned. When the quality degradation of the wireless communication path 3 continues for a long time, the moving image supply unit 1 continues to supply the moving image code, so that the data amount of the moving image code stored in the second buffer 7 increases. . This amount of data is notified from the moving image transmission unit 2 to the moving image supply unit 1 via the communication path (connection path) 8. The moving image supply unit 1 controls the supply of each layer of the moving image code based on the notified data amount. The data amount is an example indicating the buffer status, and the notified information may be information indicating the buffer status such as the buffer storage status, and may be, for example, the remaining capacity of the buffer.
[0017]
Next, although hierarchical coding is a well-known technique, hierarchical coding used in this embodiment will be described with reference to FIG. The original image before encoding consists of a plurality of frames, one of which is a frame 31. The frame 31 is decomposed into an upper layer frame 32 mainly containing high frequency components and a base layer frame 33 which is reduced to 1/2 in both vertical and horizontal directions by removing high frequency components. This decomposition can reproduce the original frame 31 with almost no distortion at the time of reconstruction, and such a transformation is known as a wavelet transform, for example. Since the frames 33 in the base layer have a strong inter-frame correlation, the motion compensation inter-frame difference is encoded and transmitted. On the other hand, since the inter-frame correlation between the upper-layer frames 32 is weak, the frames are encoded and transmitted. Note that FIG. 2 shows an example of decomposition into two layers, but it is also possible to decompose into more layers. Also, in a moving image in which one frame is composed of two fields, the same hierarchization can be performed by using the field as a unit.
[0018]
Next, a control operation for supplying each layer of moving image codes will be described with reference to FIG.
FIG. 3 is a diagram for explaining how the moving image supply unit 1 inspects the data amount 20. In FIG. 3, the storage capacity of the second buffer 7 is indicated by a rectangle 34, in which the amount of data 20 to be notified is shown. An arrow 21 represents an upper limit value of the data amount 20 related to the upper layer, and an arrow 22 represents a lower limit value of the data amount 20 related to the upper layer. As the data amount 20 increases, the upper limit value 21 is reached. The moving image supply unit 1 compares the notified data amount 20 with the upper limit value 21 of the upper layer. When the moving image supply unit 1 detects that the upper limit value 21 is exceeded, the moving image supply unit 1 discards the upper layer code and sends it to the moving image transmission unit 2. Stop supplying. The upper layer is intra-frame coded, and even if there is an error or omission, the effect remains within one frame. Therefore, even if the upper layer is not reproduced, the influence on the subjective evaluation is relatively small.
[0019]
After that, when the quality of the wireless communication path 3 is recovered and the data waiting for transmission in the second buffer 7 is transmitted, the data amount 20 notified to the moving image supply unit 1 decreases. The moving image supply unit 1 compares the notified data amount 20 with the lower limit value 22 of the upper layer. When the moving image supply unit 1 detects that the lower limit value 22 or less is detected, supply of the upper layer is started. And will both supply. In the present invention, in order to prevent the accumulated amount of the first buffer 5 of the terminal apparatus from being lowered when the quality of the wireless communication path 3 is deteriorated, the code amount to be transmitted is actively reduced to only the minimum necessary base layer. The supply of the upper layer to the moving image transmission unit 2 is stopped.
[0020]
FIG. 3 also shows an upper limit value 23 and a lower limit value 24 for the data amount 20 related to the base hierarchy. As described above, the upper limit value and the lower limit value are set to predetermined values for each layer. The base layer is inter-frame encoded, and if the supply is stopped, the subjective evaluation of the reproduction image quality is greatly affected. Therefore, both the upper limit value and the lower limit value are set close to the maximum storage capacity of the second buffer 7. . When the data amount 20 reaches or exceeds the upper limit value 23 related to the base layer, the moving image supply unit 1 stops supplying the base layer, and when the data amount 20 reaches the lower limit value 24 or less, the supply of the base layer is resumed. While the supply is stopped, the base layer may be discarded or may be delayed without being discarded.
[0021]
The moving image transmission unit 2 is, for example, a base station device, and provides a highly reliable communication service with no error in the wireless communication path 3. However, the moving image transmission unit 2 actively supplies data or transmits data to be a moving image. It does not check whether it is an image code, or whether the hierarchy is an upper hierarchy or a base hierarchy. In the present invention, such a special process depending on the content can be handled by the moving image supply unit 1, and the moving image transmission unit 2 can handle a general-purpose and light-load process as much as possible.
[0022]
Next, the operation of the present invention will be described with reference to FIGS. FIG. 4 shows the code accumulation state of the first buffer 5 on the left side of the figure and the code accumulation state of the second buffer 7 on the right side of the figure in five states from state S1 to state S5. It is. Further, FIG. 5 shows how the moving image supply unit 1 inspects the data amount 20 in five states from state S1 to state S5.
[0023]
In FIG. 4, rectangles 41a to 41c indicate codes for one frame of the upper layer, and rectangles 42a to 42h indicate codes for one frame of the base layer (the length of the rectangle indicates the code amount). The average value of the code amount of one frame in the upper layer corresponds to three times the layer of one frame in the base layer, and is illustrated here for convenience. The wireless communication path 3 transmits the codes for one frame of both the base layer and the upper layer within a unit time when the quality is in a normal state.
[0024]
The state S1 in FIG. 4 is a state when the quality of the wireless communication path 3 is in the normal state, and the first buffer 5 stores 1 frame 41a of the upper layer and 8 frames of the base layer. The second buffer 7 stores codes of one frame 41b in the upper layer and one frame 42b in the base layer. In the state S1 in FIG. 5, the data amount 20a at this time is shown to be between the upper limit value 21 and the lower limit value 22 regarding the upper hierarchy.
[0025]
Next, a state S2 in FIG. 4 shows a state in which the quality of the wireless communication path 3 is deteriorated and the transmission is interrupted and the time for reproducing 4 frames, that is, 4 unit time has elapsed from the state S1. In the first buffer 5, after the upper layer frame 41a is reproduced, the next upper layer frame has not arrived, and the upper layer reproduction is interrupted for one unit time out of four unit times. Since four frames of the base layer are reproduced, the accumulation amount is reduced to four frames. The base layer frame 42a that is fifth in state S1 is first in state S2. In the second buffer 7 in the state S2, the upper layer frames 41b and 42b existing in the state S1 are not transmitted, and a new upper layer frame 41c and four base layer frames are accumulated. The reason why only one frame of the upper layer is accumulated even though 4 unit time has elapsed is that the moving image supply unit 1 detects an increase in the data amount 20b of the second buffer 7 and This is because the supply of the frame code is stopped. The state of this determination is shown in a state S2 in FIG. It is shown that the data amount 20b at this time exceeds the upper limit value 21 related to the upper hierarchy and is less than the upper limit value 23 related to the base hierarchy. In other words, the upper layer supply was stopped halfway, but the base layer was supplied with a code for one frame per unit time as usual.
[0026]
A state S3 in FIG. 4 shows a state in which the quality of the wireless communication path 3 is recovered and two unit times have elapsed from the state S2. During two unit times, the wireless communication path 3 first transmits the upper layer frame 41b and the base layer frame 42b, and then transmits the upper layer frame 41c and another base layer frame. In the first buffer 5, the upper layer frame 41 c is stored, but this is not reproduced in time because it is not in time for the display time, and is ignored. Similarly, the frame 41b has not been reproduced, but has already been removed from the first buffer 5. The number of base frame accumulation frames in the first buffer 5 is four, which is the same as in the state S2. The base layer frame 42c that was third in the state S2 comes to the top in the state S3. In the second buffer 7 in the state S3, the upper layer frame has not yet arrived, and the base layer frame has reached two units. The base layer frame 42d that was third in state 2 comes to the top in state S3. In the state S3 in FIG. 5, the data amount 20c is lower than that in the state S2, but the lower limit value 22 regarding the upper hierarchy is still exceeded. For this reason, the supply of the upper hierarchy is still stopped.
[0027]
A state S4 in FIG. 4 represents a state in which one unit time has elapsed from the state S3 while the quality of the wireless communication path 3 is recovered. If the quality of the wireless communication channel is normal, it has the ability to transmit one frame of the upper layer and one frame of the base layer one by one. However, the code of one frame of the upper layer is now three times the code of one frame of the base layer. Since it was assumed, codes for four frames of the base layer are transmitted in one unit time. The second base layer frame 42e in the first buffer 5 in the state S3 has come to the head in the state S4, and the accumulated base layer code has increased to 7 frames. This is because there is no code for the upper layer frame to be transmitted in the second buffer 7 because the supply for the upper layer is stopped, and instead, the code for the frame of the base layer is transmitted. In the second buffer 7, the base layer frame 42f, which was fifth in the state S3, comes to the top. The state S4 in FIG. 5 indicates that the data amount 20d is smaller than that in the state S3, but still shows that the lower limit value 22 regarding the upper layer is still exceeded, so that the supply of the upper layer continues to be stopped. .
[0028]
A state S5 in FIG. 4 represents a state in which one unit time has elapsed from the state S4 while the quality of the wireless communication path 3 is recovered. The second base layer frame 42g in the first buffer 5 in the state S4 has come to the head in the state S5, and the codes of the two base layer frames present in the second buffer 7 in the state S4 are the first ones. 1 buffer 5 has arrived. The transmission capacity of the wireless communication path 3 is higher than this, and it is assumed that four frames of codes can be transmitted in the case of a frame in the base layer, but that number of codes is stored in the second buffer 7 in the state S4. Since there is not, only two sheets are transmitted. In the state S5, the first buffer 5 stores eight frames of base layer frames. In the state S5 in FIG. 5, it is indicated that the data amount 20e is equal to or lower than the lower limit value 22 regarding the upper layer, and when the moving image supply unit 1 detects this, the supply of the upper layer that has been stopped is started. In addition, when 1 unit time passes from state S5, it becomes the same as state S1. Thereafter, the supplied upper layer frame is supplied based on the absolute time to be transmitted, or a frame at a certain time before the base layer frame supplied at the same time is supplied.
[0029]
【The invention's effect】
According to the image transmission apparatus of the present invention, the following effects can be obtained. When reducing the amount of code to be transmitted on the transmission side, the video encoding device is only required to perform processing with a light load of stopping and discarding the upper layer from among the already hierarchically encoded codes. The image supply apparatus can be realized by an inexpensive apparatus that is unnecessary and has a low processing capability, and can provide moving image transmission services to a large number of clients.
[0030]
In addition to the above effect, in a network where the quality of the communication path such as a wireless network changes abruptly, if the quality of the communication path temporarily deteriorates, the moving image code stored in the first buffer 5 Since the reproduction is continued using the base layer code, it is possible to make it difficult to interrupt the reproduction of the moving image, and it is not necessary to allow an error when transmitting the communication path 3. It is possible to prevent interruption of reproduction due to decoding.
[0031]
In addition, the image transmission unit 2 only needs to notify the image supply unit 1 of only the amount of data 20 accumulated in the second buffer 7 regardless of whether the image is hierarchically encoded. It can be produced inexpensively with a simple configuration and can be used for a wide range of applications without depending on the image encoding method.
[Brief description of the drawings]
FIG. 1 is a configuration diagram of an image transmission apparatus according to a preferred embodiment of the present invention. FIG. 2 is a diagram illustrating a hierarchical encoding method. FIG. 3 is a diagram illustrating how a moving image supply unit checks the amount of data. FIG. 4 is a diagram showing the code accumulation status of the first buffer and the code accumulation status of the second buffer. FIG. 5 is a diagram showing how the moving image supply unit checks the data amount. ]
1: moving image supply unit 2: moving image transmission unit 3: wireless communication path 4: moving image reception unit 5: first buffer 6: moving image decoding unit 7: second buffer 8: communication path (connection path)
14: Output unit (image display and audio output)
20, 20a to 20b: Data amount of moving picture code in second buffer 7 waiting for transmission 21: Upper limit value of data amount 20 relating to upper layer 22: Lower limit value of data amount 20 relating to upper layer 23: Base layer Upper limit value 24 of data amount 20 related to: Lower limit value of data amount 20 related to base layer 31: Frame 32 of original image: Frame 33 of upper layer 33: Frame 34 of base layer: Storage capacity 41a to 41c of second buffer 7: Codes 42a to 42g for one frame in the upper layer: codes for one frame in the base layer

Claims (4)

A server that is a moving image supply unit that supplies a hierarchically encoded moving image code, a base station that is a moving image transmission unit that transmits the supplied moving image code, and the transmitted moving image code In an image transmission device comprising a terminal device that is a moving image receiving unit,
The base station includes a buffer for temporarily storing the video code supplied from the server, and notifies the server of the data amount of the buffer,
The server provides two threshold values of different upper and lower limits regarding the supply of moving image codes for each layer, compares the data amount of the buffer notified from the base station with each threshold value, An image transmission apparatus that controls supply of a moving image code of each layer based on a result . 2. The image transmission apparatus according to claim 1 , wherein the server discards the video code of the upper layer when detecting that the data amount of the buffer is equal to or greater than an upper limit value of a threshold of the upper layer. . In an image transmission method for transmitting a hierarchically encoded video code,
Supplying a hierarchically encoded video code, temporarily storing the supplied video code, notifying the temporarily stored video code amount,
Two threshold values of different upper and lower limit values relating to the supply of moving image codes are provided for each layer, the notified moving image code amount is compared with each threshold value, and each layer is based on the comparison result. An image transmission method for controlling supply of the moving image code. In the moving image supply method of the server, which is a moving image supply unit that supplies the hierarchically encoded moving image code to the base station,
Supplying the moving picture code to the base station, receiving notification of the amount of data in a buffer that temporarily stores the moving picture code at the base station, and different upper limit values relating to the supply of the moving picture code for each layer , and Two threshold values of a lower limit value are provided, the notified data amount of the buffer is compared with each threshold value, and the supply of the moving image code for each layer is controlled based on the comparison result. A server moving image supply method.

Images