JP7499583B2 - Video distribution device, video transmission device, video distribution system, video distribution method, and video distribution program - Google Patents

Description

The present invention relates to a device, system, method, and program for video distribution.

Conventionally, adaptive streaming technology has been used as a video distribution method, which adaptively switches between multiple video streams with different video qualities according to the throughput. For example, standards such as MPEG-DASH (see Non-Patent Document 1) enable large-scale distribution by configuring a CDN (Content Delivery Network).

The video formats used in MPEG-DASH, whether they are MPEG-2 TS, H.264, or H.265, have a GOP (Group of Pictures) structure. A GOP is a group of frames that includes I frames, which can be decoded independently, and P frames, which are encoded and decoded using information from previous frames. Since P frames represent the difference between the previous I frame or P frame, the amount of data is small, and including P frames improves coding efficiency rather than making the entire video stream out of I frames.

As shown in Figure 10, the GOP interval (the interval between I frames) is usually fixed in a video stream, and switchable video streams are coded in sync. Video playback always starts from an I frame, and switching between multiple video streams is only possible at an I frame. Note that, as shown in Figure 11, switching is possible even if the source is a P frame, so long as the destination is an I frame.

MPEG-DASH, [online], [Retrieved February 26, 2020], Internet <https://mpeg.chiariglione.org/standards/mpeg-dash/implementation-guidelines/text-isoiec-dtr-23009-3-2nd-edition-dash> AVC-Intra, [online], [Retrieved February 26, 2020], Internet <ftp://ftp.panasonic.com/pub/Panasonic/Drivers/PBTS/papers/WP_AVC-Intra.pdf> Dynamic GOP, [online], [Retrieved February 26, 2020], Internet <https://www.axis.com/ja-jp/learning/web-articles/reducing-the-bit-rate-with-axis-zipstream/dynamic-gop>

However, in conventional methods, the playback start point of a video is limited to an I frame and depends on the GOP interval (for example, about 1 to 10 seconds), so when real-time delivery is required to instantly deliver video to a receiving terminal on a frame-by-frame basis, the response to a playback start request is delayed. In addition, the switching point of a video is also limited to an I frame and depends on the GOP interval, so it takes time from selecting the video quality to switching.

In view of this, for example, Non-Patent Document 2 proposes a method of making all video streams I-frames. However, this method requires a high transmission rate in order to maintain video quality.
Also, Non-Patent Document 3 proposes a method for dynamically adjusting the GOP interval. However, this method requires an encoder for each receiving terminal, which increases costs.

The present invention aims to provide a video distribution device, a video sending device, a video distribution system, a video distribution method, and a video distribution program that are capable of low-latency real-time distribution to multiple receiving terminals.

The video distribution device according to the present invention includes a data receiving unit that receives a first video stream consisting of only I frames and a second video stream consisting of P frames based on the same video data, a distribution stream generating unit that generates a distribution stream using frames of the second video stream following the I frames of the first video stream in response to a playback request from a video receiving terminal, and a data sending unit that sends the distribution stream to the video receiving terminal.

I frames may be inserted into the second video stream at predetermined intervals.

The data receiving unit may receive a plurality of stream groups each having a different video quality from the first video stream and the second video stream as a stream group, and the delivery stream generating unit may reconstruct the delivery stream using one of the plurality of stream groups in response to a request to switch the video quality.

The video distribution device may include a throughput measurement unit that measures the throughput for the video receiving terminal, and the distribution stream generation unit may select one of the plurality of stream groups based on the measured throughput.

The delivery stream generation unit may compare the bit rate when each of the plurality of stream groups is used with the measured throughput, and select the stream group with the highest bit rate within a range that does not exceed the throughput.

The video transmission device according to the present invention includes a first video stream encoder that generates a first video stream consisting of only I frames based on video data, a second video stream encoder that generates a second video stream consisting of P frames based on the video data in synchronization with the first video stream, and a data transmission unit that transmits the first video stream and the second video stream to a video distribution device that generates a distribution stream using frames of the second video stream following the I frames of the first video stream in response to a playback request from a video receiving terminal.

The video distribution system according to the present invention includes a first video stream encoder that generates a first video stream consisting of only I frames based on video data, a second video stream encoder that generates a second video stream consisting of P frames based on the video data in synchronization with the first video stream, a distribution stream generation unit that generates a distribution stream using frames of the second video stream following the I frames of the first video stream in response to a playback request from a video receiving terminal, and a data sending unit that sends the distribution stream to the video receiving terminal.

The video distribution method according to the present invention includes a data receiving step of receiving a first video stream composed only of I frames and a second video stream composed of P frames based on the same video data, a distribution stream generating step of generating a distribution stream using frames of the second video stream following the I frames of the first video stream in response to a playback request from a video receiving terminal, and a data sending step of sending the distribution stream to the video receiving terminal, performed by a computer.

The video distribution program according to the present invention is for causing a computer to function as the video distribution device.

The present invention makes it easy to achieve low-latency real-time delivery to multiple receiving terminals.

1 is a diagram showing a configuration of a video distribution system according to an embodiment. 2 is a diagram showing a functional configuration of a moving image transmission device according to an embodiment. FIG. FIG. 2 is a diagram illustrating a functional configuration of a video distribution device according to an embodiment. FIG. 2 is a diagram illustrating a functional configuration of a video receiving terminal according to an embodiment. FIG. 1 is a first diagram illustrating a flow of video distribution in an embodiment. FIG. 2 is a second diagram illustrating the flow of video distribution in the embodiment. 10 is a flowchart showing a GOP configuration distribution process in the embodiment. 11 is a flowchart showing a distribution stream switching process in the embodiment. 11 is a diagram illustrating a distribution stream selection algorithm according to an embodiment. FIG. FIG. 1 is a first diagram illustrating a conventional video stream switching method. FIG. 2 is a second diagram illustrating a conventional video stream switching method.

An example of an embodiment of the present invention will now be described.
FIG. 1 is a diagram showing the configuration of a video distribution system 1 according to the present embodiment.
The moving image distribution system 1 includes a moving image sending device 10, a moving image distribution device 20, and a moving image receiving terminal 30.

The moving image sending device 10 generates two types of video streams in synchronization with an audio stream based on the moving image data, and provides them to the moving image distribution device 20.
The moving image distribution device 20 reconstructs the GOPs for each moving image receiving terminal 30 in response to a request, generates a distribution stream, and distributes the stream to the moving image receiving terminal 30 .

FIG. 2 is a diagram showing the functional configuration of the moving image transmission device 10 in this embodiment.
The video transmission device 10 is an information processing device (computer) equipped with a control unit (e.g., a CPU) and a memory unit, as well as various input/output interfaces, and the following functional units are realized by the control unit executing software stored in the memory unit.

The video transmission device 10 comprises a video/audio separation unit 11, an encoder group 12 consisting of an I stream encoder 121 (first video stream encoder) and a P stream encoder 122 (second video stream encoder), an audio encoder 13, and a data transmission unit 14.
Here, the moving image transmission device 10 may be provided with a plurality of encoder groups 12 in order to adaptively switch the video quality by adaptive streaming.

The video/audio separation unit 11 separates the input video data into video data and audio data, and provides them to the encoder group 12 and the audio encoder 13, respectively.

The I stream encoder 121 generates an All I stream (first video stream) consisting of only I frames based on the input video data.

The P stream encoder 122 generates an All P stream (second video stream) including P frames based on the same video data as the I stream encoder 121, in synchronization with the first video stream.
Here, I frames may be inserted into the second video stream at predetermined intervals to suppress accumulation of drift errors. In this embodiment, for the sake of simplicity, the second video stream is described as an All P frame consisting of only P frames except for the first frame.

The I stream encoder 121 and the P stream encoder 122 constitute the encoder group 12. The encoder group 12 provides a set of the All I stream generated by the I stream encoder 121 and the All P stream generated by the P stream encoder to the data sending unit 14 as one stream group.

The audio encoder 13 generates an audio stream from the audio data input from the video/audio separator 11 .
Here, all stream groups and audio streams are synchronized and time-stamped.
In addition, existing standards (for example, MPEG-2, H.264, H.265, etc.) may be applied to the encoding specifications including data structures such as timestamps.

The data sending unit 14 receives stream groups and audio streams from all encoder groups 12 and audio encoders 13 and sends them to the video distribution device 20.

FIG. 3 is a diagram showing the functional configuration of the video distribution device 20 in this embodiment.
The video distribution device 20 is an information processing device (computer) that includes a control unit (e.g., a CPU) and a memory unit, as well as various input/output interfaces. The control unit executes software stored in the memory unit to realize the following functional units:

The video distribution device 20 includes a data receiving unit 21, a distribution stream generating unit 22, a throughput measuring unit 23, and a data sending unit 24.
Here, the video distribution device 20 is provided with a plurality of pairs of a distribution stream generation unit 22 and a throughput measurement unit 23 corresponding to each of the video receiving terminals 30 connected thereto, in accordance with the number of the video receiving terminals 30 connected thereto.

The data receiving unit 21 receives all stream groups consisting of All I streams and All P streams with different video qualities, and an audio stream, from the video transmission device 10.

In response to a video playback request from the video receiving terminal 30, the distribution stream generation unit 22 generates a distribution stream using each frame from the All I frame to the I frame with the latest timestamp, followed by the All P frame, through a GOP configuration distribution process described below.
Furthermore, the delivery stream generating unit 22 reconstructs the delivery stream using one of the multiple stream groups in response to a request to switch the video quality.

The throughput measurement unit 23 works in cooperation with the video receiving terminals 30 to measure the throughput for each video receiving terminal 30. Note that the throughput measurement method may be an existing method such as iperf.

Then, the delivery stream generating unit 22 selects one of the video qualities, i.e., one of the multiple stream groups, for each video receiving terminal 30 based on the measured throughput through the delivery stream switching process described below, and reconstructs the delivery stream.
For example, the distribution stream generation unit 22 compares the bit rate expected when using each of the multiple stream groups with the throughput measured by the throughput measurement unit 23, and selects the stream group with the highest bit rate within the range that does not exceed the measured throughput.

The data sending unit 24 sends the distribution stream generated by the distribution stream generating unit 22 to the video receiving terminal 30.

FIG. 4 is a diagram showing the functional configuration of the video receiving terminal 30 in this embodiment.
The video receiving terminal 30 is an information processing device (computer) that includes a control unit (e.g., a CPU) and a memory unit, as well as various input/output interfaces. The control unit executes software stored in the memory unit to realize the following functional units:

The video receiving terminal 30 includes a data receiving unit 31, a decoding unit 32, and a video presenting unit 33.

The data receiving unit 31 receives a distribution stream and an audio stream from the moving image distribution device 20 .
The decoding unit 32 decodes the received distribution stream and audio stream in accordance with the encoding specifications used by the moving image transmission device 10 , and provides the decoded stream and audio stream to the moving image presenting unit 33 .
The video presenting unit 33 outputs the decoded distribution stream and audio stream to a display, a speaker, etc., and presents them to the user.

FIG. 5 is a first diagram illustrating a flow of video distribution in this embodiment.
In this example, the video distribution device 20 receives a stream group consisting of an All I stream and an All P stream from the video transmission device 10, and starts video distribution to three video receiving terminals 30A to 30C at different times.
The numbers in each video stream correspond to time stamps, and frames with the same number indicate synchronized frames at the same time.

First, in response to a playback start request from the video receiving terminal 30A, the video distribution device 20 distributes an I frame with the latest timestamp "1" from the All I stream, and thereafter continues to distribute P frames from the All P stream (distribution stream A).

Next, in response to a playback start request from video receiving terminal 30B, video distribution device 20 distributes an I frame with the latest timestamp "3", and thereafter continues to distribute P frames (distribution stream B).
Similarly, in response to a playback start request from video receiving terminal 30C, video distribution device 20 distributes an I frame with the latest timestamp "6", and thereafter continues to distribute P frames (distribution stream C).

In this way, the video distribution device 20 instantly configures a GOP and sends out a distribution stream in response to a playback start request from the video receiving terminal 30, regardless of the timing.

FIG. 6 is a second diagram illustrating the flow of video distribution in this embodiment.
In this example, the video distribution device 20 receives a stream group (Hi) having relatively high video quality and a stream group (Low) having relatively low video quality from the video sending device 10, and distributes videos to three video receiving terminals 30A to 30C while adaptively switching the video quality.

First, in response to a playback start request from the video receiving terminal 30A, the video distribution device 20 distributes a high-quality I frame with the latest timestamp "1" at the time of the request from the All I stream Hi, and thereafter continues to distribute high-quality P frames from the All P stream Hi.
Next, in response to a request to switch video quality, a low-quality I frame with the latest timestamp "4" at the time of the request is distributed from the All I stream Low, and thereafter, low-quality P frames are continuously distributed from the All P stream Low.
Furthermore, in response to a request to switch video quality, a high-quality I frame with the latest timestamp "7" at the time of the request is distributed from All I stream Hi, and thereafter, high-quality P frames are continuously distributed from All P stream Hi.

Similarly, in response to a playback start request from video receiving terminal 30B, video distribution device 20 distributes a high-quality P frame following a high-quality I frame with the latest timestamp "3", and then in response to a video quality switching request, distributes a low-quality P frame following a low-quality I frame with the latest timestamp "6".

In addition, in response to a playback start request from the video receiving terminal 30C, the video distribution device 20 distributes a high-quality P frame following the high-quality I frame with the latest timestamp "6", and then, in response to a video quality switching request, distributes a low-quality I frame with the latest timestamp "9".

In this way, the video distribution device 20 instantly reconstructs the GOP and sends out the distribution stream in response to a request to switch video quality, regardless of the timing.

FIG. 7 is a flowchart showing the GOP configuration delivery process in this embodiment.
This process is executed by the video distribution device 20 in response to a request to start distribution or a request to switch the video quality.

In step S1, the distribution stream generation unit 22 receives a distribution start request from the video receiving terminal 30, or a video quality switching request based on the measured throughput.

In step S2, the distribution stream generation unit 22 extracts the I frame with the latest timestamp from the All I stream and distributes it to the video receiving terminal 30 via the data transmission unit 24.

In step S3, the delivery stream generation unit 22 determines whether or not to end the video delivery. For example, it is determined that the video delivery is to end when the delivery stream ends or when a delivery end request is received. If the determination is YES, the process ends, and if the determination is NO, the process proceeds to step S4.

In step S4, the distribution stream generation unit 22 sequentially receives I frames and P frames from the video transmission device 10 via the data receiving unit 21.

In step S5, the distribution stream generating unit 22 distributes the latest received P frame to the video receiving terminal 30 via the data sending unit 24.
The process then returns to step S3.

FIG. 8 is a flowchart showing the distribution stream switching process in this embodiment.
This process is executed in the video distribution device 20 when a video quality switching function (adaptive streaming) is used.

In step S11, the delivery stream generation unit 22 determines whether or not to enable the delivery stream switching function. For example, an enable/disable setting instruction may be input from the video receiving terminal 30 or directly to the video delivery device 20. If the determination is YES, the process proceeds to step S12, and if the determination is NO, the process ends.

In step S12, the throughput measurement unit 23 measures the throughput for each video receiving terminal 30 and notifies the distribution stream generation unit 22.

In step S13, the delivery stream generation unit 22 selects which stream group to use to deliver the delivery stream based on the measured throughput.

In step S14, the delivery stream generation unit 22 determines whether the selection result in step S13 is the same as the current result. If the determination is YES, the process returns to step S11, and if the determination is NO, the process proceeds to step S15.

In step S15, the delivery stream generation unit 22 executes the GOP configuration delivery process in response to a request to switch to the selected delivery stream. Then, the process returns to step S11.

FIG. 9 is a diagram for explaining a delivery stream selection algorithm in this embodiment.
Assuming that a GOP is reconstructed using each of the multiple stream groups that are options, the distribution stream generation unit 22 calculates the expected bit rate of the distribution stream immediately after this reconstruction, i.e., for a certain period of time after the insertion of an I frame.

For example, the expected bit rate _bg is calculated by the following formula.
_bg = (b _i + b _p × (f-1)) / f
Here, b _i is the target bit rate [bps] of the All I stream, b _p is the target bit rate [bps] of the All P stream, f is the frame rate [fps], and b _g is the average bit rate for one second including the I frame.

After calculating _bg for all stream groups, the delivery stream generating unit 22 compares each _bg with the measured throughput, and selects the stream group with the largest _bg within a range that does not exceed the measured throughput.
For example, if the estimated bit rates are calculated as A>B>C, and A>measured throughput>B, the distribution stream corresponding to the estimated bit rate B is selected.

According to this embodiment, the video distribution system 1 generates a first video stream consisting of only I frames and a second video stream consisting of P frames based on the same video data, and distributes frames of the second video stream following the I frames of the first video stream in response to a playback request from the video receiving terminal 30.
This allows the video distribution system 1 to instantly reconstruct a GOP starting from an I frame when a request is received. Therefore, the video distribution system 1 can perform video distribution that can be played immediately in response to a playback start request from a plurality of receiving terminals without increasing the transmission rate or providing an encoder for each video receiving terminal 30, and thus easily realizes low-latency real-time distribution.

In addition, I-frames may be inserted into the second video stream at a predetermined interval, which allows the video distribution system 1 to suppress deterioration of image quality due to accumulation of drift errors. In this case, the interval can be set to a longer interval (e.g., 10 seconds) than the normal GOP interval (e.g., 1 second), thereby suppressing the transmission rate.

The video distribution system 1 generates multiple pairs of a first video stream and a second video stream as multiple stream groups having different video qualities, and reconstructs the distribution stream using one of the multiple stream groups in response to a request to switch the video quality.
This allows the video distribution system 1 to instantly reconstruct a GOP starting from an I frame using the corresponding stream group when a request to switch video quality is made. Therefore, the video distribution system 1 can instantly switch video quality in response to a request.

Furthermore, by measuring the throughput for the video receiving terminal 30, the video distribution system 1 can appropriately select one of the multiple stream groups based on the measured throughput.
This makes it possible to realize adaptive streaming with improved responsiveness based on throughput.

At this time, the video distribution system 1, for example, compares the estimated bitrate when using each of the multiple stream groups with the measured throughput, and selects the stream group with the highest estimated bitrate within the range that does not exceed the measured throughput.
This enables the video distribution system 1 to easily select an appropriate distribution stream based on clear selection criteria and distribute it in real time.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most favorable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

In the above embodiment, the video distribution device 20 has been described as a single device such as a server, but this is not limited thereto. For example, a CDN may be configured such that multiple video distribution devices 20 are deployed. In addition, each functional unit of the video distribution device 20 may be distributed across multiple devices, or the video distribution device 20 may also have the functions of the video transmission device 10.

In this embodiment, the configuration and operation of the video distribution system 1 have been mainly described, but the present invention is not limited to this, and may be configured as a method or program for distributing videos, including each component.

Furthermore, the functions of the video distribution system 1 may be realized by recording a program for realizing the functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.

The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.

Furthermore, "computer-readable recording medium" may include a medium that dynamically holds a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, or a medium that holds a program for a fixed period of time, such as a volatile memory inside a computer system that serves as a server or client in such a case. Furthermore, the above program may be one that realizes part of the functions described above, or may be one that can realize the functions described above in combination with a program already recorded in the computer system.

REFERENCE SIGNS LIST 1 Video distribution system 10 Video transmission device 11 Video/audio separation unit 12 Encoder group 13 Audio encoder 14 Data transmission unit 20 Video distribution device 21 Data reception unit 22 Distribution stream generation unit 23 Throughput measurement unit 24 Data transmission unit 30 Video receiving terminal 31 Data reception unit 32 Decoding unit 33 Video presentation unit 121 I stream encoder (first video stream encoder)
122 P stream encoder (second video stream encoder)

Claims (8)

a data receiving unit that receives a plurality of stream groups having different video qualities, the stream groups being a set of a first video stream consisting of only I frames and a second video stream consisting of P frames, based on the same video data;
a delivery stream generating unit that generates a delivery stream using a frame of the second video stream following an I frame of the first video stream in response to a playback request from a video receiving terminal;
a data sending unit that sends the distribution stream to the video receiving terminal ;
The video distribution device , wherein the distribution stream generation unit reconstructs the distribution stream using one of the plurality of stream groups in response to a request to switch video quality . The video distribution device according to claim 1, wherein I frames are inserted into the second video stream at predetermined intervals. A throughput measurement unit for measuring a throughput for the video receiving terminal,
The video distribution device according to claim 1 , wherein the distribution stream generation unit selects one of the plurality of stream groups based on the measured throughput. The video distribution device according to claim 3 , wherein the distribution stream generation unit compares the bit rate when each of the plurality of stream groups is used with the measured throughput, and selects the stream group having the highest bit rate within a range not exceeding the throughput. a first video stream encoder for generating a first video stream including only I frames based on the video data;
a second video stream encoder that generates a second video stream including P frames in synchronization with the first video stream based on the video data ;
the plurality of encoder groups generate stream groups having different video qualities from each other, with a pair of the first video stream and the second video stream generated by one encoder group as a stream group;
The video transmission device further includes a data transmission unit that transmits the stream group generated by the multiple encoder groups to a video distribution device for generating a distribution stream using frames of the second video stream following an I frame of the first video stream in response to a playback request from a video receiving terminal. a plurality of encoder groups each including a first video stream encoder for generating a first video stream composed only of I frames based on video data , and a second video stream encoder for generating a second video stream composed including P frames based on the video data in synchronization with the first video stream;
a delivery stream generating unit that generates a delivery stream using a frame of the second video stream following an I frame of the first video stream in response to a playback request from a video receiving terminal;
a data sending unit that sends the distribution stream to the video receiving terminal ;
the plurality of encoder groups generate stream groups having different video qualities from each other, with a pair of the first video stream and the second video stream generated by one encoder group as a stream group;
The video distribution system , wherein the distribution stream generation unit reconstructs the distribution stream using one of the stream groups generated by the plurality of encoder groups in response to a request to switch video quality . a data receiving step of receiving a plurality of stream groups having different video qualities, the stream groups being a set of a first video stream composed only of I frames and a second video stream composed including P frames, based on the same video data;
a delivery stream generating step of generating a delivery stream using a frame of the second video stream subsequent to an I frame of the first video stream in response to a playback request from a video receiving terminal;
a data transmission step of transmitting the distribution stream to the video receiving terminal,
The video distribution method, wherein the delivery stream generating step reconstructs the delivery stream using one of the plurality of stream groups in response to a request to switch video quality . A video distribution program for causing a computer to function as the video distribution device according to any one of claims 1 to 4 .

Images