JP7641766B2 - Content generating device, receiving device, and program - Google Patents

Description

The present invention relates to a content generating device, a receiving device, and a program.

The mainstream streaming video distribution on the Internet these days is a method of streaming video using a general-purpose web server over the HTTP protocol. One example of such a streaming distribution method (adaptive streaming) using the HTTP protocol is the international standard MPEG-DASH (ISO/IEC23009-1).

In MPEG-DASH, a web server is provided with initialization segments, media segments, and a manifest file that describes their URLs and the attributes of the video content. Media segments are segments in which a stream of video content encoded at one or more qualities (parameters such as screen size and bit rate) is divided into files of several to several tens of seconds each. Initialization segments contain metadata required for presenting the segments, such as generation parameters related to encoding and encryption. A receiving device selects the appropriate quality from the manifest file, taking into account the screen size of the receiving device and the state of the network bandwidth of the transmission path, receives the segments one after the other, and connects them together into a single video content for playback (for example, see Non-Patent Document 1).

In addition to the quality switching mentioned above, other possible methods for providing video streaming services include multi-view video and multilingual audio distribution, and broadcast-like program distribution according to schedule. MPEG-DASH defines a method for writing manifests that will enable such services.

Figure 7 shows a conceptual diagram of a manifest file in the MPD (Media Presentation Description) format for MPEG-DASH. The MPD format has a hierarchical structure in the order of Period, AdaptationSet, and Representation. A Period indicates a section into which content is divided in the time direction, and the example in Figure 7 shows that Periods, each of which has a media duration of 300 seconds, are linked together. A Period is made up of one or more AdaptationSets.

An AdaptationSet indicates information about media components (video, audio, text, etc.) and can contain multiple components with different expressions in the same media format (video from different angles, audio in different languages, etc.). In the example in Figure 7, there are two videos (mimeType=video/mp4) and one audio (mimeType=audio/mp4). Information about each media component included in an AdaptationSet is indicated by a Representation.

Information about segments (Initialization Segment, Media Segment) which are the actual data that make up a media element is described in the Representation. Initialization Segment indicates an initialization segment, and Media Segment indicates a media segment. In the example in Figure 7, the file name of the media segment has a sequence number ($Number$) that is incremented for each segment length (duration = 5s).

By using such a manifest file, it is possible to sequentially receive the segments (initialization segments and media segments) that form the media component (AdaptationSet) selected in the Period, and by reselecting the media component, it is possible to switch to receiving a segment that forms a different media component and display a different video.

Non-Patent Document 2 also discloses a method for realizing adaptive streaming by preparing a common initialization segment that contains the generation parameters of all Representations included in one AdaptationSet, and simply inserting the common initialization segment at the beginning of the AdaptationSet, without having to insert an initialization segment every time a Representation is switched.

Patent document 1 also discloses a method for suppressing changes in image quality before and after switching streams by generating a delivery stream by replacing part of the stream after switching with a stream generated using pictures before and after switching to a stream with a different bit rate as reference images.

JP 2015-19329 A

"Next-generation video distribution technology "MPEG-DASH" technical overview and trends in standardization and related technologies", Journal of the Institute of Image Information and Television Engineers, Vol.67, No.2, 2013, pp.109-115 “Digital Video Broadcasting (DVB); MPEG-DASH Profile for Transport of ISO BMFF Based DVB Services over IP Based Networks - DVB Document A168”, [online], October 2019, [retrieved February 22, 2021], Internet <URL:https://dvb.org/wp-content/uploads/2019/12/a168_dvb_mpeg-dash_oct_2019.pdf>

Figure 8 shows the concept of a buffer in a receiving device. When switching to a stream (representation) with different generation parameters, the receiving device inserts an initialization segment of the destination representation immediately before inserting a media segment belonging to the destination representation into the buffer, as shown in Figure 8. This allows the generation parameters of the subsequent media segment to be properly notified to the decoder downstream of the buffer, thereby realizing seamless adaptive streaming. Note that in Figure 8, each media segment of each quality level is composed of three blocks, with one block corresponding to one representation.

However, in the MPEG-DASH standard, as shown in Figure 7, only one initialization segment's information could be described in one Representation. Therefore, if the generation parameters of the initialization segment described in a Representation were changed while the receiving device was playing that Representation, the receiving device was unable to recognize the contents and timing of the change.

Even if the technology described in Non-Patent Document 2 is used, the receiving device cannot recognize changes to generation parameters during playback of the Representation. Furthermore, the technology described in Patent Document 1 is not necessarily applicable to changes in generation parameters such as changes in resolution or encryption key.

In light of these circumstances, the object of the present invention is to provide a content generation device, a receiving device, and a program that enable stable continuous streaming by allowing the receiving device to recognize changes in generation parameters during playback of streaming content.

In order to solve the above problem, a content generation device according to one embodiment is a content generation device that generates segments of video content and a manifest file, and includes: a media segment generation unit that generates media segments according to generation parameters used to generate the segments of video content; an initialization segment generation unit that generates new initialization segments including information required to decode the media segments in accordance with changes to the generation parameters; a manifest generation unit that generates a manifest file including segment correspondence information indicating the corresponding initialization segment for each of the media segments; and a content transmission unit that transmits the media segments, the initialization segments, and the manifest file.

Furthermore, in one embodiment, the segment correspondence information may be information for identifying a file name of a media segment and a file name of an initialization segment that corresponds to the media segment.

Furthermore, in one embodiment, the manifest generation unit may generate a manifest file in which information on multiple initialization segments is described within one representation by adding the segment correspondence relationship information to a manifest file conforming to the MPD (Media Presentation Description) format.

In order to solve the above problem, a receiving device according to one embodiment is a receiving device that receives an initialization segment, a media segment, and a manifest file of video content, and includes a manifest acquisition unit that acquires a manifest file in which segment correspondence information indicating a corresponding initialization segment for each media segment is described, and generates a segment URL list that associates the playback time of the media segment with the URL of the initialization segment and the URL of the media segment, a buffer control unit that extracts the URL of the media segment and the URL of the initialization segment corresponding to the media segment based on the playback time that is updated as the media segment is played and the segment URL list, and a segment acquisition unit that acquires the media segment based on the URL of the media segment, and acquires the initialization segment based on the URL of the initialization segment.

To solve the above problem, a program according to one embodiment causes a computer to function as the above content generation device.

To solve the above problem, a program according to one embodiment causes a computer to function as the receiving device.

According to the present invention, a receiving device can recognize changes in generation parameters during playback of streaming content and play the content continuously. This makes it possible to dynamically change the generation parameters of a stream being distributed in accordance with fluctuations in network bandwidth and the operational requirements of the service provider.

1 is a diagram showing an outline of a content distribution system including a receiving device and a content generating device according to an embodiment of the present invention. 1 is a block diagram showing an example of the configuration of a content generating device according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of a manifest file in MPD format generated by a content generating device according to an embodiment of the present invention. 4 is a flowchart illustrating a content generating method of the content generating device according to an embodiment of the present invention. 1 is a block diagram showing an example of the configuration of a receiving device according to an embodiment of the present invention; 4 is a flowchart showing a segment acquisition control method of a receiving device according to an embodiment of the present invention. FIG. 1 is a diagram illustrating the concept of a conventional manifest file. FIG. 1 is a diagram illustrating the concept of a buffer in a conventional receiving device.

One embodiment of the present invention will be described in detail below with reference to the drawings. In the embodiment described below, as an example, the manifest file is in MPD format of MPEG-DASH (ISO/IEC23009-1).

Figure 1 shows an overview of a content distribution system including a receiving device and a content generating device according to one embodiment of the present invention. The content distribution system is a system that distributes video content composed of multiple segments and a manifest file on a segment-by-segment basis. The receiving device 1 and distribution server 3 are connected via the Internet. The content generating device 2 and distribution server 3 may be connected by a dedicated local network or via the Internet.

The receiving device 1 receives video content segments (initialization segments and media segments) and a manifest file from the distribution server 3, and performs streaming playback.

The content generation device 2 generates video content segments and a manifest file and supplies them to the distribution server 3.

The distribution server 3 is a server that distributes the manifest file and segments of the URL specified by the receiving device 1 to the receiving device 1, and can be, for example, a general web server.

(Content generation device)
Next, details of the content generator 2 will be described. Fig. 2 is a block diagram showing an example of the configuration of the content generator 2. The content generator 2 shown in Fig. 2 includes a material video output unit 21, a generation parameter change unit 22, a media segment generator 23, an initialization segment generator 24, a manifest generator 25, a content transmitter 26, and a communication I/F 27.

The raw video output unit 21 outputs the content to be generated to the media segment generation unit 23 from raw video input from an external source or from pre-stored raw video.

The generation parameter change unit 22 sets or changes the generation parameters, and outputs the generation parameters to the media segment generation unit 23 and the initialization segment generation unit 24. The generation parameters are parameters used to generate segments of video content, and include media encoding parameters (resolution, bit rate, etc.) and encryption parameters for content protection. The generation parameters may be set or changed by the user via the input interface. In addition, the generation parameter change unit 22 outputs a parameter change trigger indicating the timing at which the generation parameters are changed to the initialization segment generation unit 24.

The media segment generation unit 23 generates a media segment according to the generation parameters. The media segment generation unit 23 includes an encoding unit 231 and a segmentation unit 232.

The encoding unit 231 encodes the raw video signal input from the raw video output unit 21 using an existing encoding method or encrypts it using an existing encryption method according to the generation parameters input from the generation parameter change unit 22. Existing encoding methods include, for example, H.264|MPEG-4 AVC (Advanced Video Coding) and H.265|MPEG-H HEVC (High Efficiency Video Coding). Existing encryption methods include, for example, the AES (Advanced Encryption Standard) method. The encoding unit 231 then outputs the encoded stream to the segmentation unit 232. Note that audio can also be processed in the same way as video.

The segmentation unit 232 segments the encoded stream input from the encoding unit 231, for example in ISO-Base Media File Format (ISO-BMFF) format, to generate media segments containing media information. The segmentation unit 232 then outputs the generated media segments to the manifest generation unit 25.

The initialization segment generation unit 24 inputs the generation parameters and parameter change trigger from the generation parameter change unit 22. The initialization segment generation unit 24 then generates a new initialization segment including information required to decode the media segment in accordance with the change in the generation parameters. For example, the initialization segment generation unit 24 segments in ISO-BMFF (ISO-Base Media File Format) format and generates an initialization segment including the generation parameters. The initialization segment generation unit 24 then outputs the generated initialization segment to the manifest generation unit 25.

The manifest generation unit 25 inputs the media segments from the segmentation unit 232 and the initialization segments from the initialization segment generation unit 24. The manifest generation unit 25 then generates a manifest file including segment correspondence relationship information. The segment correspondence relationship information is information indicating the corresponding initialization segment for each media segment. The manifest generation unit 25 adds the segment correspondence relationship information to a manifest file that complies with the conventional MPD format of MPEG-DASH, for example. The manifest generation unit 25 then outputs the initialization segments, media segments, and manifest file to the content transmission unit 26. The manifest file may be generated by a method of updating the manifest file sequentially each time a segment is input, or may be generated after all segments have been input.

The content sending unit 26 sends the initialization segment, media segment, and manifest file input from the manifest generating unit 25 to the distribution server 3 via the communication I/F 27.

Here, a specific example of content generated by the content generation device 2 will be described with reference to FIG. 3. FIG. 3 shows an example (excerpt) of a manifest file in MPD format to which segment correspondence information has been added. Here, the segment correspondence information is information for identifying the file name of a media segment and the file name of an initialization segment that corresponds to the media segment.

Figure 3 shows a manifest file where the content duration is 32 seconds, the segment length is 2 seconds, and the generation parameters are changed every 8 seconds from the start of encoding. In this example, there is one Period with content duration information (duration) of 32 seconds, the Period has one AdaptationSet whose media format is video (mimeType="video/mp4"), and the AdaptationSet has one Representation.

Here, we explain how to identify a media segment that contains a certain playback time and the file name of the initialization segment that corresponds to that media segment.

The start value on the second line of Figure 3 indicates that the start time of the Period is 0. The timescale value on the fourth line of Figure 3 indicates that the timescale (number of ticks per second) of the AdaptationSet is 1000.

The value of media on line 13 of Figure 3 indicates that the base value of the file name of the media segment is s1_$Time$.mp4, and the file name will be the value obtained by replacing $Time$ with the timestamp value of the media segment. The value of initialization indicates that the base value of the file name of the initialization segment is s1_init_$Index$.mp4, and the file name will be the value obtained by replacing $Index$ with the identifier of the initialization segment.

In addition, the SegmentTimeline shown on lines 5 to 10 of Figure 3 indicates the relationship between the playback start time of each segment, the segment length, and the identifier of the corresponding initialization segment. In S, t indicates the playback start time, d indicates the segment length, i indicates the identifier of the initialization segment, and r indicates the number of times that the same segments with i and d continue. In other words, S on line 6 indicates that the start time is 0 (t = 0), the segment length is 2 seconds (d/timescale), the initialization segment index is 0, and the same i and d continue for three segments. Therefore, the file names of the media segments with start times of 0 seconds, 2 seconds, 4 seconds, and 6 seconds are s1_0.mp4, s1_2000.mp4, s1_4000.mp4, and s1_6000.mp4, and the file name of the initialization segment corresponding to these is s1_init_0.mp4. Also, from the seventh line in Figure 5, it can be seen that the media segments with start times of 8, 10, 12, and 14 seconds are s1_8000.mp4, s1_10000.mp4, s1_12000.mp4, and s1_14000.mp4, and that the initialization segment corresponding to these is s1_init_1.mp4. In this way, it is possible to identify the file name of a media segment that includes a certain playback time, and the file name of the initialization segment that corresponds to that media segment.

In the example of FIG. 3, lines 5 to 10 and line 13 are segment correspondence relationship information. Here, i is an identifier that is not defined in conventional standards, and $Index$ is a variable that is not defined in conventional standards. In this way, the manifest generation unit 25 can generate a manifest file in which information on multiple initialization segments is described in one Representation by adding segment correspondence relationship information to a manifest file that complies with the MPD format. In the example of FIG. 3, information on four initialization segments with i=0 to 3 is described for the Representation with id=1.

(How content is generated)
Next, a content generating method according to the present invention will be described with reference to the flow chart of FIG.

In step S001, the generation parameter change unit 22 checks whether there are any initial settings for the generation parameters if encoding has not yet started, and checks whether there are any changes to the generation parameters if encoding has started. If there are any initial settings or changes to the generation parameters (step S001-YES), the process proceeds to step S002, and if there are no initial settings or changes to the generation parameters (step S001-NO), the process proceeds to step S004.

In step S002, the generation parameter change unit 22 sets the set or changed generation parameters as new generation parameters.

In step S003, the initialization segment generation unit 24 generates an initialization segment using the newly generated parameters.

In step S004, the encoding unit 231 continues encoding using the previously set parameters.

In step S005, the segmentation unit 232 generates media segments segmented at predetermined intervals.

In step S006, the manifest generation unit 25 associates the initialization segment for each generation parameter with the media segment generated by each generation parameter.

In step S007, the manifest generation unit 25 updates the manifest file. Then, the content transmission unit 26 and the communication I/F 27 transmit the initialization segment, media segment, and manifest file.

In step S008, it is confirmed whether the encoding has reached the end of the content or whether an instruction to end the encoding has been given. If the encoding has reached the end of the content or an instruction to end the encoding has been given (step S008-YES), the processing ends. If the encoding has not reached the end of the content and there has been no instruction to end the encoding (step S008-NO), the processing continues.

A computer can be suitably used to function as the above-mentioned content generation device 2, and such a computer can be realized by storing a program describing the processing contents for realizing each function of the content generation device 2 in the memory unit of the computer, and reading and executing this program by the processor of the computer. Note that this program can be recorded on a computer-readable recording medium.

(Receiving device)
Next, details of the receiving device 1 will be described. Fig. 5 is a block diagram showing an example of the configuration of the receiving device 1. The receiving device 1 shown in Fig. 5 includes a communication I/F 10, a manifest acquisition unit 11, a segment list storage unit 12, a segment acquisition unit 13, a buffer 14, a buffer control unit 15, a playback control unit 16, a playback control user I/F 17, a playback unit 18, and a display unit 19.

The manifest acquisition unit 11 receives a manifest file of a desired program from the distribution server 3 via the communication I/F 10. The manifest acquisition unit 11 then analyzes the manifest file in which the segment correspondence relationship information is described, generates a segment URL list that associates the playback times of media segments with the URLs of initialization segments and media segments, and stores the segment URL list in the segment list storage unit 12.

The segment acquisition unit 13 inputs the URL of the segment to be acquired from the buffer control unit 15, and sends a request including the URL of the input segment to the distribution server 3 via the communication I/F 10. The segment acquisition unit 13 then receives a segment from the distribution server 3 via the communication I/F 10 as a response to the request, and outputs it to the buffer control unit 15.

The buffer 14 inputs segments from the buffer control unit 15, and stores the media segments in association with the initialization segment, and the playback start time and playback duration (segment length) of the media segment. The buffer 14 then outputs information on the segment playback times and playback durations of all currently stored media segments to the buffer control unit 15. The buffer 14 also sequentially outputs segments with playback times requested by the playback unit 18 to the playback unit 18.

The playback control unit 16 outputs the current playback state (playing, stopped) and the current playback time information input from the playback unit 18 to the playback control user I/F 17. In addition, when the playback control unit 16 inputs a playback start or stop command from the playback control user I/F 17, it instructs the playback unit 18 to start or stop playback.

The playback control user I/F 17 is a function that controls playback through user operations. It may include buttons such as playback start/stop buttons and buttons that allow a seek operation to a desired playback time, and may graphically display the current playback time based on the start and end times of the content and the playback time information input from the playback control unit 16. For example, the start time of the content may be displayed as a horizontal bar at the bottom of the playback screen with the left end at the left and the end time at the right end at the right, and a knob-shaped operation part may be placed so as to overlap with it. By moving the knob-shaped operation part from the left end to the right end as playback progresses, it is possible to grasp at a glance which part of the entire playback time is currently being played. Also, a seek operation to a desired playback time may be performed by changing the knob-shaped operation part.

When the playback unit 18 receives a playback command including a playback start time from the playback control unit 16, it sequentially inputs media segments from the buffer 14 based on the playback start time, decodes the media segments using parameters indicated by the initialization segment corresponding to the media segments, and outputs the decoded media segments to the display unit 19. The playback unit 18 also outputs playback time information, which is information on the playback time that is updated as the media segments are played, to the playback control unit 16. The playback unit 18 also stops playback when it receives a stop command from the playback control unit 16.

The display unit 19 displays the decoded media data input from the playback unit 18 according to the format of the media.

The buffer control unit 15 determines (extracts) the URL of the segment to be acquired from the playback time information and segment URL list input from the playback control unit 16, and outputs it to the segment acquisition unit 13. The buffer control unit 15 also outputs the segment input from the segment acquisition unit 13 to the buffer 14.

(Segment Acquisition Control Method)
Next, a segment acquisition control method of the receiving device 1 will be described with reference to the flowchart in Fig. 6. The flowchart shown in Fig. 6 starts from a state in which the manifest file has already been acquired from the distribution server 3 by the manifest acquisition unit 11 and a segment URL list has already been created. Also, it is assumed that the buffer 14 has a range from the start time to the end time of the Period described in the MPD, and stores the segments in association with information on the playback start time and the playback duration time. Also, the range in which the content segments are inserted in the buffer 14 is referred to as the buffered segment range.

In step S101, when any playback start is executed in the playback control user I/F 17, the playback control unit 16 outputs the playback start time to the buffer control unit 15 as the current playback time.

In step S102, the buffer control unit 15 determines whether the current playback time is within the buffered segment range. If it is within the buffered segment range (step S102-YES), the process proceeds to step S103. If it is not within the buffered segment range (step S102-NO), the process proceeds to step S104.

In step S103, the buffer control unit 15 extracts from the segment URL list the URL of the media segment that follows the buffered segment range, and the URL of the initialization segment that corresponds to that media segment.

In step S104, the buffer control unit 15 extracts from the segment URL list the URL of the media segment that includes the current start time, and the URL of the initialization segment that corresponds to that media segment.

In step S105, the buffer control unit 15 determines whether the URL of the extracted initialization segment is different from the URL of the previously acquired media segment. If they are different (step S105-YES), the process proceeds to step S106; if they are the same (step S105-NO), the process proceeds to step S108.

In step S106, the segment acquisition unit 13 sends a segment request including the URL of the extracted initialization segment to the distribution server 3, and receives the initialization segment corresponding to the request.

In step S107, the buffer control unit 15 inserts the received initialization segment into the buffer 14.

In step S108, the segment acquisition unit 13 sends a segment request including the URL of the media segment extracted in step S103 or step S104 to the distribution server 3, and receives the media segment corresponding to the request.

In step S109, the buffer control unit 15 inserts the received media segment into the buffer 14.

In step S110, it is confirmed whether the end of the segment URL list has been reached or the stop button has been pressed. If the end of the segment URL list has been reached or the stop button has been pressed (step S110-YES), the process ends. If the end of the segment URL list has not been reached and the stop button has not been pressed (step S110-NO), the process proceeds to step S1111.

In step S111, the playback unit 18 continues playback, updates the current playback time, and returns to step S102.

A computer can be suitably used to function as the receiving device 1 described above, and such a computer can be realized by storing a program describing the processing contents for realizing each function of the receiving device 1 in the memory unit of the computer, and having the processor of the computer read and execute this program. Note that this program can be recorded on a computer-readable recording medium.

As described above, the content generating device 2 or the program for functioning as the content generating device 2 generates a new initialization segment in accordance with the change in the generation parameters, and generates a manifest file including the segment correspondence relationship information. Then, the receiving device 1 or the program for functioning as the receiving device 1 can recognize the change in the generation parameters during playback of the streaming content by acquiring the manifest file in which the segment correspondence relationship information is described, and can continue stable streaming.

The above-described embodiments have been described as representative examples, but it will be apparent to those skilled in the art that many modifications and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be interpreted as being limited by the above-described embodiments, and various modifications and changes are possible without departing from the scope of the claims. For example, it is possible to combine multiple component blocks or processing steps described in the embodiments into one, or to divide one into multiple steps.

Reference Signs List 1 Receiving device 2 Content generating device 3 Distribution server 10 Communication I/F
REFERENCE SIGNS LIST 11: Manifest acquisition unit 12: Segment list storage unit 13: Segment acquisition unit 14: Buffer 15: Buffer control unit 16: Playback control unit 17: Playback control user I/F
18 Playback unit 19 Display unit 21 Material video output unit 22 Generation parameter change unit 23 Media segment generation unit 24 Initialization segment generation unit 25 Manifest generation unit 26 Content transmission unit 27 Communication I/F
231 Encoding section 232 Segmentation section

Claims (6)

A content generation device that generates video content segments and a manifest file, comprising:
a media segment generator for generating media segments in accordance with generation parameters used to generate segments of the video content;
an initialization segment generating unit that generates a new initialization segment including information necessary for decoding the media segment in accordance with the change in the generation parameter;
a manifest generation unit that generates a manifest file including segment correspondence relationship information indicating the corresponding initialization segment for each of the media segments;
a content transmitter that transmits the media segments, the initialization segment, and the manifest file;
A content generating device comprising: The content generating device according to claim 1, wherein the segment correspondence information is information for identifying a file name of a media segment and a file name of an initialization segment corresponding to the media segment. The content generating device according to claim 1 or 2, wherein the manifest generating unit generates a manifest file in which information on multiple initialization segments is described in one representation by adding the segment correspondence relationship information to a manifest file conforming to the MPD (Media Presentation Description) format. A receiving device for receiving an initialization segment, a media segment, and a manifest file of video content, the receiving device comprising:
a manifest acquisition unit that acquires a manifest file in which segment correspondence information indicating a corresponding initialization segment for each media segment is described, and generates a segment URL list in which the playback times of the media segments are associated with the URLs of the initialization segments and the media segments;
a buffer control unit that extracts a URL of the media segment and a URL of the initialization segment corresponding to the media segment based on a playback time that is updated with playback of the media segment and the segment URL list;
a segment acquisition unit that acquires the media segment based on the URL of the media segment and acquires the initialization segment based on the URL of the initialization segment;
A receiving device comprising: A program for causing a computer to function as a content generating device according to any one of claims 1 to 3. A program for causing a computer to function as the receiving device according to claim 4.

Images