JP7444872B2 - Methods, apparatus, and computer programs for encapsulating media data into media files - Google Patents

Description

The present invention relates to a method and apparatus for encapsulating and transmitting media data.

The International Organization for Standardization Base Media File Format (ISOBMFF, ISO/IEC 14496-12) is an encoded, timed media data format for either local storage or transmission over a network or another bitstream delivery mechanism. A well-known, flexible and extensible format for describing bitstreams. An example of an extension is ISO/IEC 14496-15, which describes encapsulation tools for various NAL (Network Abstraction Layer) unit-based video encoding formats. Examples of such encoding formats are AVC (Advanced Video Coding), SVC (Scalable Video Coding), HEVC (High Efficiency Video Coding), and L-HEVC (Layered HEVC). Another example of file format extension is ISO/IEC23008-12, which describes a tool for encapsulating still images and still image sequences, such as HEVC still images. Also, another example of file format extension is ISO/IEC 23090-2, which defines the Omnidirectional Media Application Format (OMAF). The ISO base media file format is object oriented. It consists of building blocks called boxes (or data structures characterized by four-letter codes) that define the parameters of an encoded timed media data bitstream, such as timing and structure parameters.

In file formats, the entire presentation is called a movie. Movies are described by a movie box (four-letter code "moov") at the top level of a media or presentation file. This movie box represents an initialization information container that includes a set of various boxes that describe the presentation. It is logically divided into tracks represented by track boxes (four letter code "trak"). Each track (uniquely identified by a track identifier (track_ID)) represents a timed sequence of media data (eg, frames of video) belonging to a presentation. Within each track, each unit of timed data is called a sample, which is a frame of video, audio, or timed metadata. Samples are implicitly numbered sequentially. Actual sample data is stored in a box called a media data box (four-letter code "mdat") located at the same level as the movie box. The sample description is stored in the sample table box in the metadata portion of the file. Movies can be organized temporally such that a movie box containing information for the entire presentation is followed by a list of couple movie fragments and media data boxes. Within one movie fragment (box with four-letter code "moof"), there is a set of zero or more track fragments (box with four-letter code "traf") for one movie fragment. A track fragment includes zero or more track run boxes ("truns"), each recording consecutive samples of the track fragment.

An ISOBMFF file may include a plurality of bitstreams of encoded timed media data and subparts of a bitstream of encoded timed media data forming multiple tracks. If the subpart corresponds to one or a contiguous spatial portion of the video source taken over time (e.g. at least one rectangular region (also called a "tile") taken over time) ), the corresponding multiple tracks may be called subpicture tracks.ISOBMFF and its extensions provide several grouping mechanisms for grouping tracks, static items, or samples.Groups are , typically share common semantics and/or properties.

The inventors have noticed several problems in describing and signaling information about the media data being transmitted, particularly for multiple tracks when one track references another.

An example is signaling from a client of a data entity referenced in another track requesting specific parsing, which incurs overhead and is complex.

Another example is the signaling of data entities that are copied from a group of tracks to form a composite track, especially when the composite track depends on the type of relationship between these groups of tracks.

Existing solutions are complex or poorly defined, and do not fully comply with existing mechanisms for synthesizing tracks from groups of tracks.

The present invention was conceived to address one or more of the concerns mentioned above.

In this context, a solution is provided for streaming media content (eg omnidirectional media content) over an IP network, such as the Internet, using eg the http protocol.

According to a first aspect of the invention, a method is provided for encapsulating media data into a media file, the method comprising:
including in the media file a first track that includes a plurality of media samples, each media sample including a set of one or more NAL units;
including a second track in the media file that includes an extractor, the extractor being a structure that references a data entity within a media sample included in the first track; ,
including in the extractor a copy mode attribute that identifies the referenced data entity within the one media sample for one or more NAL units contained within the one media sample;
including.

In particular, said copy mode attribute is set to one of a list of one or more modes.

In one embodiment, the list of modes is:
a first mode in which the data entity includes a set of all NAL units included in the sample;
a second mode in which the data entity includes one NAL unit of the set of NAL units;
a third mode in which the data entity comprises a payload of one NAL unit of the set of NAL units;
a fourth mode in which the data entity includes one NAL unit at a specified position within the set of NAL units;
a fifth mode in which the data entity includes a payload of one NAL unit at a specified position within the set of NAL units;
including one or more of.

In one embodiment, in the second or third mode, the one NAL unit is the first NAL unit in the sample.

In one embodiment, in the second or third mode, the one NAL unit is the last NAL unit in the sample.

According to a second aspect of the invention, a method of generating a media file is provided, the method comprising:
encoding media content into media data;
encapsulating the media data into a plurality of tracks according to the encapsulating method described above;
generating at least one media file including the encapsulated media data;
including.

According to a third aspect of the invention, a method of transmitting a media file is provided, the method comprising:
generating a media file by a server device according to the method described above;
transmitting the generated media file to a client device;
including.

According to a fourth aspect of the invention, there is provided a method of parsing a media file to generate media data, the method comprising:
retrieving a first track from the media file that includes an extractor, the extractor being a structure that references a data entity within a media sample included in a second track; and,
retrieving the second track from the media file that includes a plurality of media samples, each media sample including a set of one or more NAL units;
retrieving one data entity identified by a copy mode attribute included in the extractor from one media sample of the second track;
including;
The copy mode attribute identifies the referenced data entity within the one media sample for one or more NAL units included within the one media sample.

Other aspects of the invention relate to computer devices and corresponding computer programs for encapsulating media data and parsing media files.

Further advantages of the invention will become apparent to those skilled in the art from consideration of the drawings and detailed description. It is intended that any additional advantages be incorporated herein.

Embodiments of the invention are described below by way of example with reference to the following drawings.

1 illustrates an example system including an encapsulation/decapsulation module adapted to implement embodiments of the present invention. FIG. FIG. 2 is a diagram illustrating an example structure of a NAL (Network Abstraction Layer) unit. 1 is a diagram illustrating an example structure of a video media sample based on the ISO base media file format; FIG. FIG. 2 is a diagram showing an example of the structure of an extractor and aggregator based on ISO/IEC14496-15. FIG. 3 is a diagram illustrating encapsulation processing according to an embodiment of the invention. FIG. 3 is a diagram illustrating decapsulation processing according to an embodiment of the present invention. FIG. 3 illustrates an example of encapsulation of media data with data extraction from track groups in accordance with an embodiment of the present invention. FIG. 1 is an illustration of an example system that captures, processes, encapsulates, transmits, and renders independently encoded bitstreams from a server to a client. FIG. 2 is a diagram illustrating a process of encoding 2D video into multiple tracks. FIG. 3 is a diagram illustrating an example of sub-picture track encapsulation including multiple track groups to describe 2D spatial relationships. FIG. 3 illustrates an example of explicit reconstruction from a set or group of tracks. FIG. 3 illustrates an example of explicit reconstruction from a set or group of tracks. 1 is a schematic block diagram of a computing device for implementing one or more embodiments of the invention; FIG.

FIG. 1 depicts exemplary systems 191 and 195 adapted to implement embodiments of the present invention. System 191 includes an encapsulation module 150 connected to communication network 199. System 195 includes a decapsulation module 100 connected to communication network 199.

According to one embodiment, system 191 is for processing content (eg, video and/or audio content) for streaming or storage. System 191 obtains/receives content including original sequence images 151 and encodes the sequence images into media data (i.e., bitstream) using a media encoder (e.g., a video encoder) and encapsulation module 150. encapsulates media data into a media file 101 using Encapsulation module 150 includes at least one of a writer and a packager that encapsulates media data. A media encoder may be implemented within or separate from encapsulation module 150 to encode received content. Therefore, the encapsulation module 150 can be dedicated only to encapsulating already encoded content (media data).

According to one embodiment, system 195 is for processing media data for display/output to a user. The system 195 obtains/receives a media file 101 via a communication network 199, decapsulates the media file 101 using a decapsulation module 100 to obtain media data, and uses a media decoder to convert the media data into audio. and/or decoding into video content (signal). Decapsulation module 100 includes at least one of a parser and a player. The media decoder may be implemented within the decapsulation module 100 or may be provided separately from the decapsulation module 100 to decode media data (bitstream).

Media file 101 may be communicated to a parser or player of module 100 in several ways, but may be, for example, previously generated by a writer or packager of encapsulation module 150 and stored on a storage device within communications network 199 (e.g., on a server or The content may be stored as data in a cloud storage until a user requests the content encoded therein from the storage device. When content is requested, data is communicated/streamed from the storage device to the decapsulation module 100.

The system 191 also provides/streams content information for content stored on the storage device (e.g., content title and other meta/storage location data for identifying, selecting, and requesting content) to the user. The content providing device may also be included. The content providing device may also be adapted to receive and process user requests for content to be delivered/streamed from the storage device to the user terminal.

Alternatively, encapsulation module 150 may generate media file 101 and communicate/stream it directly to decapsulation module 100 when a user requests content. Thereafter, the decapsulation module 100 receives the media file 101 and performs decapsulation and decoding of the media data according to embodiments of the present invention to obtain/generate a video signal 109 and/or an audio signal, which is Used by the user terminal to provide the requested content to the user.

A user accesses audio/video content (signals) via a user interface of a user terminal that makes up the module 100 or has means for communicating with the module 100. Such user terminal may be a computer, mobile phone, tablet, or other type of device capable of providing/displaying content to a user.

According to one implementation, media file 101 boxes media data (e.g., encoded audio or video) according to the ISO Base Media File Format (ISOBMFF, ISO/IEC 14496-12 and ISO/IEC 14496-15 standards). encapsulate in. Media file 101 may correspond to one media file (indicated by the file type box "ftyp") or one or more segment files (indicated by the segment type box "styp"). According to ISOBMFF, a media file 101 includes two types of boxes: a "media data box (mdat)" that contains media data, and a "metadata box" that contains metadata that defines the placement and timing of media data. It is said that it is okay to stay.

A video encoder encodes video content using a video standard and produces an encoded video bitstream (media data). The video encoding/decoding (codec) standard is ITU-T H. 261 (ISO/IEC MPEG-1 Visual), ITU-T H. 262 (ISO/IEC MPEG-2 Visual), ITU-T H. ITU-T H.263 (ISO/IEC MPEG-4 Visual), including Scalable Video Coding (SVC) and Multi-View Video Coding (MVC) extensions. ITU-T H.264 (ISO/IEC MPEG-4 AVC), including scalable (SHVC) and multi-view (MV-HEVC) extensions. 265 (HEVC).

Many embodiments described herein describe examples that use the HEVC standard or its extensions. However, the techniques and systems described herein are compatible with other already available encoding standards such as AVC and the ITU-T H. It is also applicable to encoding standards that are not yet available or developed, such as H.266 (VVC).

Figure 2a shows H. 264/AVC and HEVC/H. 2 shows an example structure of a NAL (Network Abstraction Layer) unit 200 used in a video codec such as H.265.

A NAL unit includes a NAL unit header 201 and a NAL unit payload 202. The NAL unit header 201 is of fixed length and provides general information of the NAL unit. For example, in HEVC, the NAL unit header 201 indicates each NAL unit's type, layer identifier, and temporal sublayer identifier. There are mainly two types of NAL units 200: video coding layer NAL units (VCLNAL) and non-VCLNAL units. A VCLNAL unit typically includes encoded slice segments 205 in its payload. Non-VCLNAL units typically include parameter sets (ie, configuration information) or supplemental enhancement information messages.

The encoded slice segment 205 is encoded such that a slice_segment_header or "slice header" 206 is followed by a slice_segment_data or "slice data" 207 in the HEVC bitstream. A slice segment includes an integer number of consecutive (in raster scan order) encoding tree units (ie, blocks within a picture). Slices are not necessarily rectangular in shape (and are therefore less suitable than tiles for representing spatial subparts). Video compression formats define an access unit as a set of consecutive NAL units in decoding order that correspond to an encoded picture.

FIG. 2b shows an example structure of a media (video) sample 210 based on the ISO base media file format.

A media sample is an audio/video data unit (eg, an audio frame or a video frame) that has one time. According to ISO/IEC 14496-15, a sample is a set of one or more NAL units 212 that correspond to an access unit or a part of an access unit. Each NAL unit 212 is preceded by a NAL unit length field 211. For example, for a single layer video, the samples correspond to encoded pictures. In the case of layered video, the samples may correspond to part of an access unit, for example a NAL unit for the base layer.

The sample size (in bytes) is listed in the sample size box "stsz" or "stsz2". Given the sample size and the length of the NAL units, the ISOBMFF parser (eg, module 100) can determine the number of NAL units in the sample. ISO/IEC 14496-15 defines certain NAL units, extractors, and aggregators that are ISOBMFF structures embedded within the media data ("mdat" box). These are also called "in-stream structures."

FIG. 3 shows an example of an extractor and aggregator structure according to ISO/IEC 14496-15.

A first (media) track 300, corresponding to a media stream (e.g. a compressed video bitstream), is composed of media (e.g. video) samples 301, each media sample as illustrated in FIG. 2b. , including a set of one or more NAL units. The number of NAL units within a sample may vary from one to another. A second track 320, e.g. referred to as a reconstruction, synthesis, reference or extraction track, uses a NAL unit to reference data from another track, here the first track 300, as illustrated by arrow 310. 323 and an extractor NAL unit 322 . Samples 321 from current track 320 may then be reconstructed by extracting data from track 300 and concatenating with data 323 from current track 320.

Note that some samples 321 of the reconstruction track 320 may contain only extractors or only data. Also, the number of extractors or NAL units may be different for each sample 321. The extractor 322 is a structure for efficiently extracting NAL units from tracks other than the track containing the extractor. The extractor NAL unit is identified by a specific NAL unit type value (the specific value may depend on the codec in use so as not to conflict with type values assigned to VLC and non-VLC NAL units). ).

ISO/IEC14496-15 defines extractors that support various compression formats such as SVC, MVC, and HEVC. In HEVC, the extractor introduces a specific constructor to reconstruct samples from data in the referenced track (sample constructor) or data provided directly within the constructor (inline constructor). Existing extractors and constructors specify the extraction or copying of data as a copy of a number of bytes from a given byte position.

The present invention proposes to specify data extraction and copying in extractors and constructors not by byte ranges, but rather by syntactic structures. One advantage is that by copying syntax structures, byte alignment of the bitstream can be ensured. Another advantage is that the extractor description size can be reduced. This is valuable because extractors are defined at the sample level and repeated over time. Another advantage is the ability to extract from alternate or group tracks. The latter is not possible if the extraction operation is expressed in byte ranges. This is because byte positions from one track to another may not correspond to the same syntactic structure. Yet another advantage is that the writer or packager can calculate the length in bytes of the NAL unit length field of the NAL unit in order to calculate and provide the number of bits in the data offset and data length fields of the extractor or sample constructor. It's something you don't need to worry about. Existing extractors or sample constructors may require the writer or packager to reformat the extracted data to fit the length field size of the destination track.

Figure 4a illustrates an encapsulation process according to an embodiment of the invention. In embodiments, this processing is performed by the writer or packager of encapsulation module 150 shown in FIG. 1 to encapsulate media data.

Step 400 initializes the encapsulation module so that the media data (bitstream) can be properly read. This initialization can be performed by the user via the user interface or by the application. Initialization includes identifying the syntactic structure of the bitstream (referred to for generality as data entities) and setting encapsulation parameters. Encapsulation settings include, for example, whether the media file 101 is generated as one media file or multiple media segments; whether the media file includes one track or multiple tracks of a video stream; This may include determining settings for dividing the video track into parts, views, or layers, etc.

If multiple tracks are included, the encapsulation module may set references between tracks or define groups of tracks during step 400. Thus, a track that is constructed with reference to one or more other tracks contains track references to those one or more tracks. Track references may be of different types to describe the type of relationship or dependency between the referencing track and the referenced track. The type of track reference can be encoded using a four letter code. For example, according to ISO/IEC 14496-15, the type code "scal" specifies a track that contains an extractor that references another track from which data is extracted.

Once the encapsulation module is initialized, the bitstream is read NAL unit by NAL unit in step 401. The first NAL unit corresponding to the parameter set may be embedded in the decoder configuration record structure depending on the initialization (in-band parameter set or out-of-band parameter set) at step 400. These parameter sets can be inspected by the writer or packager to learn details about the bitstream splitting. For example, it can be determined whether the HEVC bitstream is a tiled HEVC bitstream by checking the SEI (Supplementary Enhancement Information) message of the temporal motion constraint tile set or the presence of the tile in the picture parameter set. Once a NAL unit is read in step 401, the writer checks in step 402 whether it corresponds to a new sample. This can be done, for example, by decoding the picture order count or by checking whether the slice corresponding to the NAL unit is the first slice of the picture. If yes, the previous sample is determined in step 403 by setting the parameters of the sample description (size, position within the media data, properties within some sample groups...). Next, in step 404, it is checked whether the current NAL unit should be included in the media portion of the track, referenced from another track, or partially modified. This is determined from the track dependencies or relationships set in the initialization step 400. If the NAL unit is not referenced, the length of the NAL unit is first inserted into the media data "mdat" box, followed by the header and payload of the NAL unit (step 405). The size of the current sample is then incremented by the number of bytes in these three structures, and the writer or packager checks the next NAL unit from the video bitstream in step 406. If this is not the last NAL unit, processing repeats to step 401 until all NAL units have been processed.

If the NAL unit is included by reference in the media file (test 404 is true), the writer or packager includes an extractor and sets the attributes of the extractor in accordance with an embodiment of the invention in step 407. In particular, this process adds a NAL unit length field with the size of the extractor structure in bytes in the "mdat" box and creates the extractor's NAL unit. The extractor includes a copy mode attribute "copy_mode" in the media sample that identifies a referenced data entity for one or more NAL units included in the media sample. The copy mode attribute may take on different values depending on how the writer or packager was initialized at step 400.

In one embodiment, the initialization performed in step 400 results in the copy mode attribute being set to "sample" mode if the reconstruction track is used to composite video samples from different tracks. In this mode, the data entity contains the set of all NAL units included in the sample.

In one embodiment, the initialization or bitstream inspection performed in step 400 is performed when the video bitstream is tiled (e.g., HEVC with a motion constrained tileset) and the user or application directs the repositioning of the tiles. If so, the copy mode attribute is set to "NALU" mode. In this mode, the data entity contains one given NAL unit included in the sample.

In one implementation, if as a result of the initialization performed in step 400, the tile relocation requires modification of the tile position in the bitstream, the writer determines whether the tile position is encoded in the slice header or not. The rewrite instructions can be provided using an inline constructor+extractor with the copy mode attribute set to "slice_data" mode or "NALU_payload" mode, depending on whether it is encoded in the NAL unit header. In these modes, the data entity includes the payload or slice data of one given NAL unit included in the sample. The advantage of these modes is that they avoid copying unnecessary data that would otherwise be overwritten by inline constructor instructions.

When an extractor is written to the media data part ("mdat" box), the sample description is updated (sample size, current NALU index within the sample, etc.). The writer or packager then checks the next NAL unit at step 406. Upon reaching the last NAL unit, the writer finalizes the media file at step 408, for example by writing the size of the last sample, index table, user data, or any metadata on the media.

If the initialization step 400 indicates encapsulation into a segment, an additional test is performed (not shown) to check whether the segment duration has been reached before starting a new sample. Please note that. When the segment's duration is reached, the segment is finalized and ready for use by the player or transmitted via the distribution network. If the segment duration has not been reached, the writer or packager repeats the samples and NAL units.

FIG. 4b illustrates a decapsulation process according to an embodiment of the invention. In one embodiment, this processing is performed by a parser or player of the decapsulation module 100 illustrated in FIG. 1 to decapsulate the media data.

In step 410, the player first receives a media file 101 (either as a single file or as a continuous segment). This file may be stored in the parser or player's memory, or read from a network socket.

First, in step 411 initialization data, typically a "moov" box and its subboxes, are parsed and the parameters/settings of the media file (number of tracks, track relationships and dependencies, sample type, duration, position) are parsed. and size).

From the set of tracks determined in step 411, the player or parser selects one or more tracks to render in step 412. Then, reconstruction begins by parsing the media data box sample by sample. In the case of interdependent tracks, parsing of media data is done according to the track references from the referenced track to the referencing track using the sample position information, e.g. from the samples available in the sample table box to the chunk box " Reference is made to the chunk offset box "stsc" and the chunk offset box "stco".

The parser or player repeats the sample until it reaches the end of the file (test 413 - Fail). In the case of segments, once one segment is completely read, the parser reads the next segment sample by sample.

For a given sample, it reads data from the position specified in the chunk offset box and adds the cumulative size of previous samples parsed in that chunk. From this position, the parser finds the NAL unit length field. The parser then reads the number of bytes specified in the NAL unit length field to obtain the payload of the NAL unit. If the NAL unit corresponds to an extractor (test 415), the parser reads the copy mode attribute of the extractor in step 416. If the NAL unit is not an extractor, the parser appends bytes corresponding to the NAL unit payload to the reconstructed bitstream (media data) provided to the media decoder for decoding in step 417. do. After step 417, the process iterates through the next NAL unit (proceeding to step 414) until the current sample size is reached.

Depending on the value of the copy mode attribute, the parser extracts samples, NAL units, NAL unit payloads or slice data from the referenced track. Indeed, according to embodiments of the invention, the copy mode attribute indicates the syntax structure or data entity to copy (instead of a byte range). For example, the copy mode may indicate copying samples or access units, NAL units, NAL unit payloads, or slice data.

In the following, an example is provided to explain the proposed new extractor according to an embodiment of the invention. The new extractor implements the copy mode attribute as described above in steps 407 and 415 of the encapsulation/decapsulation process of Figures 4a and 4b.

The implementation of the copy mode attribute is applicable both to extractors without constructors, such as SVC, MVC extractors, and to extractors with constructors, such as HEVC or L-HEVC extractors. For extractors with constructors, a new type of constructor can be defined (identified by "constructor_type") as follows:

class aligned(8) Extractor () {
NALUnitHeader();
do {
unsigned int(8) constructor_type;
if( constructor_type == 0 )
SampleConstructor();
else if( constructor_type == 2 )
InlineConstructor();
else if ( constructor_type == 3 )
SampleConstructorFromStructure();
} while( !EndOfNALUnit() )
}

The name of the new constructor "SampleConstructorFromStructure" is provided as an example. Furthermore, the reserved value "3" of "constructor_type" is also given as an example. Rather than indicating ("SampleConstructor") or providing ("InlineConstructor") a byte range, the new constructor indicates a copy mode that depends on syntactic structure. Reserved names and reserved values for "constructor_type" can be used. A new constructor is defined as follows. In a preferred embodiment, the new constructor draws from temporally aligned samples of the referenced track. This eliminates the need to use the extractor's NAL unit to repeat the 8 bits for "sample_offset" with each sample.

class aligned(8) SampleConstructorFromStructure () {
unsigned int(8) ref_index;
unsigned int(2) copy_mode;
if (copy_mode != 0) {
unsigned int(1) nalu_idc_field_size;
unsigned int(5) reserved;
unsigned int((nalu_idc_field_size+1)*8) nalu_idc;
} else {
unsigned int(6) reserved;
}
}

The new constructor parameters, fields, and attributes have the following meanings:
- "ref_index" specifies the index of the track reference of type "scal" used to find the Track_ID to extract data from. Note that modifications to ISOBMFF allow track references to refer to tracks or track groups. If the ref_index resolves to a track_group_id, the parser or player selects the most appropriate track within the corresponding track group according to the track grouping_type. The default behavior is to select the first track of the file with the specified track_group_id.
- "copy_mode" specifies the copy operation (ie copy mode) to be performed when resolving the extractor. For example, one or more of the following values are selected:
- "0" indicates "sample" mode, ie, copies from the first byte of the sample to the end of the sample. The first byte of the sample can be determined from the Sample to Chunk Box and Chunk Offset Box.
- "1" indicates "NALU" mode, ie copying from the first byte of the i-th NAL unit to the last byte of the same NAL unit (i corresponds to the nalu_idc field); The length of the bytes to copy corresponding to the NAL unit is determined from the length field 211 preceding the NAL unit 212 of the media data box "mdat".
- "2" indicates "NALU_payload" mode, ie copying from the first byte of the payload of the i-th NAL unit to the last byte of the same NAL unit.
- "3" indicates "slice_data" mode, ie copying from the first byte of the i-th slice_data to the last byte of this same slice (i corresponds to the nalu_idc field). The parser or reader determines the position of the first byte to be copied, for example, by reading the ISOBMFF structure dedicated to describing the slice header length. This may be indicated by a dedicated sample group that provides the slice header length for each sample, or an in-stream structure that provides the slice header length for each NAL unit.
- "nalu_idc_field_size": This field indicates the number of bytes minus 1 byte used for encoding the nalu_idc parameter.
- "nalu_idc": When copy_mode is NALU-based (copy_mode !=0), this field indicates the index (1-based index) of the NALU from which to extract. The value 0 is reserved.

Note that the last two copy modes ("NALU_payload" and "slice_data" modes) are valid if any header rewriting (NAL unit header or slice header) is done. In such cases, it is necessary to extract only the payload and combine it with the rewritten header. Header rewriting can be handled by an inline constructor.

If the extractor references an aggregator, the offset of the first byte to copy is the NAL unit length field value plus the NAL unit length field size plus the additional_bytes field for an aggregator of nalu_idc - 1 NAL units. It is calculated as follows. Aggregators are interesting as a copy mode sample, but we recommend avoiding them when extracting at the NAL unit or slice level. If copy_mode is present with set to NAL unit or slice-based copy, the aggregator shall be ignored and the i-th NAL unit indicated by the nalu_idc parameter shall be processed.

In all of the above copy modes, the data to be copied for one or more NAL units included in the sample, whether the entire NAL unit or a portion of the NAL unit (payload/slice) is being copied. Entities are easily determined.

From the above definition, the existing "sample constructor" requires 4 to 10 bytes depending on the length of the NAL unit ("lengthSizeMinusOne" field in the decoder configuration information), whereas the new constructor requires It turns out that it takes 2 to 4 bytes. Setting the copy mode to "sample" mode reduces the size of the constructor by at least half.

The above copy modes are not exhaustive, and other copy modes that save more bytes are also envisioned. For example, one implementation may set the copy mode attribute to "First_NALU" mode, signaling the parser to copy the first NAL unit of the current sample. In this mode, the nalu_idc parameter is not required. This mode can save 1 or 2 bytes of memory. According to another implementation, the copy mode may be targeted to a range of NAL units. For example, copy_mode is set to "NALU_range" mode, and the start NAL unit index and end NAL unit index are specified as parameters.

Considering reserved bits, a maximum of 128 copy modes can be defined. In order for the parser to clearly identify the copy mode set by the media file writer or packager, one reserved value is prepared for each copy mode. If flexibility is needed in the extraction, especially for extraction from time-displaced samples, the new constructor can embed a sample_offset parameter in addition to the copy_mode parameter or attribute.

In another embodiment, a new extractor with a reserved NAL unit type is defined to distinguish new extractors operating in syntax structure-based copy mode from existing byte-based extractors. For example, a new extractor called "StructureExtractor" is defined as follows.

class aligned(8) StructureExtractor () {
NALUnitHeader();
unsigned int(8) ref_index;
signed int(8) sample_offset;
unsigned int(2) copy_mode;
if (copy_mode !=0 ) { // 0: sample copy mode
unsigned int(1) nalu_idc_field_size;
unsigned int(5) reserved;
unsigned int((nalu_idc_field_size+1)*8) nalu_idc;
} else {
unsigned int(6) reserved;
}
}

It has the same semantics as the new constructor disclosed above. The main difference is that here we have a specific "NALUnitHeader". "NALUnitHeader" is a NAL unit header corresponding to the video coding format in use, but is already reserved for a VCL of the video coding format in use, a NAL unit for a non-VCL, or an existing extractor or aggregator. has a reserved value that is not

In another embodiment, the "sample_offset" parameter is not present in the new extractor structure because the new extractor always operates on samples in time. The two new extractors (time-aligned or non-time-aligned) can be distinguished by different specific values of nalu_type (in the NAL unit header).

In another embodiment, one of the reserved bits is used as a flag to indicate the presence or absence of a sample offset. This allows for chronological or non-temporal extraction with one extractor or constructor. If the flag is set, sample_offset is present and has the same semantics as existing extractors or constructors, allowing non-time-aligned extraction. If the flag is not set, sample_offset does not exist and only extraction along time is possible. An example is shown below. Similar embodiments can also be applied to extractors that provide copy_mode.

class aligned(8) SampleConstructorFromTrackGroup () {
unsigned int(8) ref_index; // an identifier for a group of tracks
unsigned int(2) copy_mode; // sample, NALU, NALU payload, slice data
unsigned int(1) time_aligned_extraction;
if (time_aligned_extraction == 1) {
signed int(8) sample_offset;
}
if (copy_mode != 0) {
unsigned int(1) nalu_idc_field_size;
unsigned int(4) reserved;
unsigned int((nalu_idc_field_size + 1) * 8) nalu_idc;
} else {
unsigned int(5) reserved;
}
}

According to a preferred embodiment, the newly defined extractor or constructor may be advantageously used for media files that encapsulate multiple tracks, such as groups, alternatives or equivalents, as described below. I can do it.

An ISOBMFF file can encapsulate multiple encoded bitstreams or subparts of bitstreams resulting in multiple tracks. For example, an ISOBMFF file contains multiple tracks, each representing a video sequence of a given quality, resolution, or bitrate, encapsulated in the same media file to provide choices to the player. ISOBMFF and its extensions provide several grouping mechanisms for grouping tracks, static items, or samples. Groups usually have common semantics and characteristics. ISOBMFF also defines tools to describe relationships between tracks.

For example, ISOBMFF consists of an entity group mechanism and a track group mechanism. The entity group mechanism can be used to indicate that tracks and/or static items are grouped according to an indicated grouping type or semantic. A track grouping mechanism may be used to indicate that tracks are grouped according to an indicated grouping type or semantics. For track groups, as a first tool, ISOBMFF provides a box labeled "trgr" located at the track level (i.e. within the "trak" box of the ISOBMFF box hierarchy), where each group has specific characteristics. Describes a group of tracks that share or have specific relationships between tracks within the group. A track group includes at least an identifier (“track_group_id”) and a grouping type (“track_group_type”). All tracks with a "TrackGroupTypeBox" with the same track_group_type and track_group_id values are part of the same track group. In the Omnidirectional Media Application Format (OMAF, ISO/IEC23091-2), to indicate that a track in this type of track group is an alternative to be used as a source for a "scal" or "sabt" track reference. We are currently considering a new group type, ``alte.'' It also extends the semantics of the track reference type to allow references to both tracks or track groups within a "tref" box. This last point can affect the resolution of extractors by media parsers or players.

The second tool for defining groups of tracks, and in particular groups of alternate tracks, is the "alternate_group" parameter in the track header. This parameter specifies an integer identifier that specifies a group or collection of tracks. Tracks with the same value for the "alternate_group" parameter contain alternative data to each other. Along with the "alternate_group" identifier, to distinguish tracks within an alternate track group, media files have attributes (e.g. bitrate, frame rate, codec, language, packet size, etc.) that can be used as differentiation criteria. can be included.

The third tool is EntityGrouping, which groups entities: items and/or tracks. In the case of a track group, "EntityToGroup" includes an identifier and "grouping_type". In particular, ISO/IEC 14496-12 declares grouping_type "altr" to group alternatives together.

Another tool for describing relationships between tracks is the track reference mechanism. This means that a track reference box "tref" declared at the track level contains one or more track reference type boxes, each providing the type of the reference and an identifier (track_ID) of the one or more tracks being referenced. ing. This mechanism is used, for example, in layered video, scalable video ("sbas"), and tiled video ("tbas") to indicate that a set of tracks share the same "base track." Alternate or equivalent tracks may be signaled via the track reference mechanism with a specific track reference type instead of using the track group mechanism. For example, the "eqiv" track reference type indicates that the referencing track and the referenced track(s) can be used transparently for extraction, i.e. they represent the same visual content and the corresponding bitstream can be combined (they have the same parameter set). There may be other signaling indicating that sets of tracks are equivalent. Such signaling allows new extractors or constructors from track groups to be used once a set or group of tracks can be uniquely identified.

FIG. 5 illustrates an example of encapsulation of media data with extraction of data from track groups, according to an embodiment of the invention.

The extractor track 500 (track_id=100) has a track reference 501 of type "scal" to two track groups 520 and 530 (with track_group_id=11 and 22, respectively). These two track groups may have track_group_type set to "alte" to indicate that the tracks are substitutes for each other, or to select only one track from the track group. may have other grouping types with semantics that allow the decision to always be made. In this example, each of these track groups declares two alternative tracks 521, 522 and 531, 532 (tracks with track_ID=111, 112 and tracks with track_ID=221, 222, respectively). For example, tracks within a track group describe the same video but are encoded at different quality levels. In this example, a sample at time 502t of extractor track 500 is a sample at time t from one track of first track group 520 and a sample at time t from one track of second track group 530. It consists of a concatenation of. For each track group 520 or 530, writer 150 may cause sample 502 extractor 503 to extract a sample from any one of the alternative tracks within track group 520, and sample 502 extractor 504 may extract a sample from any one of the alternative tracks within track group 520. 530 indicates that samples may be extracted from any one of the alternative tracks within 530. The copy_mode of extractors 503 and 504 is set to a value corresponding to the "sample" copy mode.

Resolving such an extractor from a track group requires an additional step by the parser or player-like 100 (represented at 550). When identifying the extractor NAL unit in step 415 (FIG. 4b), the parser must check whether ref_index corresponds to track_ID or track_group_id. This can be determined by the type of extractor or constructor, by the brand of the file, if the constructor or extractor from track group takes only track_group_id as ref_index. If the constructor or extractor from track groups accepts both track_ID and track_group_id, the parser maintains a list of IDs indicating that they are associated with the track or track group. From this list, the parser determines whether the extractor or constructor's ref_index from track group directly resolves to track_ID or track_group_id. If the ref_index resolves the track_group_id, it is up to the parser or player to select the most appropriate track within the corresponding track group, depending on the track grouping_type. The default behavior is to select the first track of the file with the specified track_group_id. You can also see the parameters that distinguish between alternative tracks in the track selection box and other track descriptions, and select the most appropriate track within a track group depending on the application and context (bandwidth, processing power). be.

To avoid decoding artifacts, the parser may re-evaluate the selection of tracks of the reference track group only on random access samples (eg, intra-frames in the case of video). The parser may determine the location of the random access samples by examining the "rap" sample group of the alternate track. In the preferred embodiment, the bitstream corresponding to the alternate track has its random access points aligned so that the parser only needs to see one track sample group to determine when to re-evaluate track selection for extraction. You can know.

The consistency of random access points of alternate tracks may be signaled by a particular "track_group_type", track reference type, or any signaling that describes track equivalence. A new constructor or extractor that references a track group has one parameter that indicates whether the random access points of the tracks in the track group are aligned (e.g., encoded with 1 bit by reusing reserved bits). It is also possible to embed a flag). Once aligned, the parser knows the samples from which it can re-evaluate the selection of tracks within the track group. If this new parameter is set to false, it means that the random access points are not aligned between the tracks in the track group. Then, before reevaluating the selection of a single track when resolving an extractor or constructor referencing a track group, the parser first ensures that at least one track in the referenced track group You must check that you have a random access point for the current sample (the one containing the extractor).

In other embodiments, the above checks may be avoided. In this alternative embodiment, the extractor or constructor that references the track group has an optional parameter combined with a flag or parameter that indicates alignment or misalignment of random access points between tracks. If the flag is set to false, a new optional parameter is present, providing a list of tracks in the referenced track group with random access points for the current sample (the one containing the extractor) . The list of tracks is described as a list of track_IDs. From this list, the parser determines, for a given sample, a list of possible candidate tracks within the track group to be extracted.

According to an alternative embodiment, the extractor is defined as follows:

class aligned(8) StructureExtractor () {
NALUnitHeader();
unsigned int(8) ref_index;
signed int(8) sample_offset;
unsigned int(1) aligned_RAPs
if (aligned_RAPs == 0) {
unsigned int (8) switchable_tracks;
}
unsigned int(2) copy_mode;
if (copy_mode !=0 ) { // 0: sample copy mode
unsigned int(1) nalu_idc_field_size;
unsigned int(4) reserved;
unsigned int((nalu_idc_field_size+1)*8) nalu_idc;
} else {
unsigned int(5) reserved;
}
}

Similar to the previous embodiment, "StructureExtractor" has two versions: a time-aligned version in which "sample_offset" is not present (as in the structure above), and a non-time-aligned version in which "sample_offset" is present. be. Each version has a specific NAL unit type so that the parser can easily identify the type of extractor. A parameter called "aligned_RAPs" (the name is an example) corresponds to a flag indicating whether the tracks in the referenced track group (via ref_index) have aligned random access points. The parameter called "switchable_tracks" (the name is an example) is a parameter that provides a list of switchable tracks for the extractor solution. This contains the index of the track within the track group that provides a random access point for a given sample. The writer or packager 150 populates this list in step 407 in response to the initialization information obtained in step 400, such as instructions on how to group the tracks listed in the "trgr" box.

Note that this parameter can also be an array of track_IDs, but for the sake of exposition, instead of a list track_ID (32 bits each), we use the index of the track within the track group (8 bits each). ). This is a 1-based index. The index is determined as follows (both on the writer side and on the parser side): For each track group, an ordered list of track_IDs is stored in memory. The order is in descending order of the track_ID value. For example, if tracks with track_ID 1, 12, 23 are part of the same track group, index 1 would provide track_ID=1, index 2 would provide track_ID=12, and so on. ．． ．． This also works if track equivalence is signaled via a specific track reference type. The index is also based on an ordered list of track_IDs referenced by a particular track reference type. There are scenarios where placing random access points (such as intra-decoding refresh (IDR) frames) at different times on alternate tracks can increase switching points for dynamic content adaptation and reduce switching latency.

Of course, the same parameters can be applied to a new constructor that references the track group (here, the chronological version of the constructor, i.e. the version that does not include the sample_offset parameter):

class aligned(8) SampleConstructorFromStructure () {
unsigned int(8) ref_index;
unsigned int(2) copy_mode;
unsigned int(1) aligned_RAPs
if (aligned_RAPs == 0) {
unsigned int (8) switchable_tracks;
}
if (copy_mode != 0) {
unsigned int(1) nalu_idc_field_size;
unsigned int(4) reserved;
unsigned int((nalu_idc_field_size+1)*8) nalu_idc;
} else {
unsigned int(5) reserved;
}
}

The new parameters ``aligned_RAPs'' and optional ``switchable_tracks'' have the same meaning as used in StructureExtractor above. Although the name "SampleConstructorFromStructure" above is provided as an example, it could also be "SampleConstructorFromTrackGroup" or any name corresponding to a unique and reserved single constructor_type value.

For both extractors or constructors that have a copy mode (referring to a generic case or track group), alternative embodiments provide another way to indicate the index of the NAL unit to extract (if copy_mode is a "sample" copy mode) (indicates a different mode). This alternative embodiment results in a new definition of an extractor or constructor as follows.

For HEVC or L-HEVC extractors, extend the HEVC and L-HEVC extractors (or any compression format that reuses the concept of a constructor within the extractor) and regenerate from a set or group of tracks as follows: We propose a new type of constructor (here called, for example, SampleConstructorFromTrackGroup), which has the possibility to construct.

class aligned(8) Extractor () {
NALUnitHeader();
do {
unsigned int(8) constructor_type;
if( constructor_type == 0 )
SampleConstructor();
else if( constructor_type == 2 )
InlineConstructor();
else if ( constructor_type == 3
SampleConstructorFromTrackGroup ();
} while( !EndOfNALUnit() )
}

The extractor constructor_type semantics have been updated as follows:
"constructor_type" specifies the subsequent constructor. SampleConstructor, InlineConstructor, and SampleConstructorFromTrackGroup correspond to constructor_types of 0, 2, and 3, respectively. Other constructor_type values are reserved.

An example when constructor_type is 3 is shown below (however, other unused values can also be used). :

class aligned(8) SampleConstructorFromTrackGroup () {
unsigned int(8) ref_index;
unsigned int(2) copy_mode;
unsigned int(6) reserved;
if (copy_mode != 0) {
unsigned int(8) nalu_idc; // 1-based index
if (nalu_idc == 0){
unsigned int(8) escape_nalu_idc; // To address from 255 to 511
}
}
}

It has the following semantics:
- ref_index specifies the index of the track reference of type "scal" used to find the track_group_id for extracting data. It is the role of the parser or player to select the most appropriate track from the corresponding track group according to track grouping_type. The default behavior may be to select the first track of the file with the specified track_group_id.
- The attribute "copy_mode" specifies the copy operation to be performed when resolving the extractor, and in particular specifies which syntax structures or data entities to copy (rather than which byte ranges to copy).
- Set copy_mode to 0: Sample-based copy mode, ie, copy bytes from the first byte of the sample to the end of the sample. The first byte of the sample can be determined from the Chunk From Sample and Chunk Offset boxes.
- Set copy_mode to 1: NAL unit-based copy mode, ie, copy from the first byte of the i-th NAL unit to the last byte of the same NAL unit. The offset of the first byte to copy is calculated by adding the value of the NAL unit length field, the size of the NAL unit length field, and the value of the additional_bytes field for an aggregator of nalu_idc-1 NAL units. The length of the bytes to copy corresponding to the NAL unit is determined from the length field 211 preceding the NAL unit 212 of the media data box "mdat".
- Set copy_mode to 2: NALU payload-based copy mode, ie copy from the first byte of the i-th NALU payload to the last byte of this same NALU, where i corresponds to the nalu_idc field.
- Set copy_mode to 3: Perform slice_data based copy mode, ie copy from the first byte of the i-th slice_data to the last byte of this same slice (i corresponds to the nalu_idc field). The parser or reader determines the location of the first byte to be copied by, for example, reading the ISOBMFF structure that describes the length of the slice header.
- The nalu_idc attribute specifies the index of the NAL unit (starting from 1-based index, ie value 1) when copy_mode is NAL unit-based or slice-based (copy_mode !=0). The value 0 is reserved to indicate the presence of an escape nalu_idc for extracting beyond the 255th NALU (if up to 600 slice segments are included per picture, such as in HEVC level 6). for rare cases such as).
-escape_nalu_idc: indicates the offset to start extraction beyond the 255th NALU. If it does not exist, it is assumed to be 0. If present, the value 0 is reserved and can be used as an optional 8-bit separate escape value (not shown) to the address from the 512th NAL unit. A NAL unit is represented by nalu_idc+escape_nalu_idc.

The mode based on the NALU payload and slice data is effective when rewriting the header. In such cases, only the payload is extracted and combined with the rewritten header. Use an inline constructor for rewriting. The sample_offset parameter does not appear in the new SampleConstructorFromTrackGroup, assuming that the reconstruction is done from samples in time from one track in the referenced track group. An alternative embodiment that provides more flexibility in extraction includes keeping the sample_offset parameter in this new SampleConstructorFromTrackGroup. If present, sample_offset has the same meaning as in ISO/IEC 14496-15. The writer or packager 150 may use a particular constructor in the encapsulation step 407, particularly if the initialization step 400 indicates that an alternative encoded bitstream or representation of the original sequence 151 exists. Such an extractor can be used. A new extractor is created so that this new constructor is interoperable between the file/segment encapsulation means 150 (e.g. an mp4 writer) and the file/segment decapsulation means 100 (e.g. an mp4 reader). It is proposed to include a new section in ISO/IEC 14496-15 that defines: This SampleConstructorFromTrackGroup can also embed additional parameters, such as a parameter or flag indicating whether the random access points are aligned across tracks in the track group, and if this parameter or flag is set to false. has an optional parameter that provides a list of tracks that can be switched.

SVC, 3D-AVC, MVC extractors support copy modes of syntactic structures or data entities rather than byte ranges, and can also be extended to extract from track groups. A particular NAL unit type indicates that the extractor is based on syntactic structure extraction (i.e. includes an attribute or parameter indicating copy_mode), can refer to a group or set of alternative or equivalent tracks, and has a NAL unit index representation of Reserved to indicate the use of NAL unit escape mode for (here described in non-time aligned extraction mode, as indicated by the presence of the "sample_offset" parameter):

class aligned(8) StructureExtractor () {
NALUnitHeader();
unsigned int(8) ref_index;
signed int(8) sample_offset;
unsigned int(2) copy_mode;
unsigned int(6) reserved;
if (copy_mode !=0 ) { // 0: sample copy mode
unsigned int(8) nalu_idc; // a 1-based index;
if (nalu_idc == 0)
unsigned int(8) escape_nalu_idc; // to address from 256 to 511
}
}

The semantics are the same as described for SampleConstructorFromTrackGroup. This new extractor can also embed additional parameters such as a parameter or flag indicating whether the random access points are aligned across tracks in a track group, and this parameter or flag is set to false. In some cases, it has an optional parameter that provides a list of tracks that can be switched.

FIG. 6 illustrates an example system 60 implementing the encapsulation and transmission method. System 60 can stream media content (eg, 2D images). System 60 includes a server device 601 and a client device 670, and media content is transmitted from server device 601 to client device 670. As shown, media content is a video sequence captured by a camera system 600 and delivered to a client device 670 for display on a 2D screen 675 (TV, tablet, smartphone, head-mounted display...), for example by a user. 6011.

The images 6011 forming the video sequence are in a preferred embodiment divided into spatial parts 6013 by the dividing means 6012 so as to be independently encoded by the encoding means 640. Independently encoded means that one spatial part does not use data from another spatial part as a reference for differential or predictive encoding. For example, if the encoding means 640 is based on the HEVC (High Efficiency Video Coding) compression format, the spatial part 6013 can be encoded as an independent tile. In an alternative embodiment, spatial part 6013 may be encoded as a motion-constrained tile. The encoding means provides as many bitstreams as spatial parts, or one bitstream with N independent sub-bitstreams (e.g. when HEVC is used to encode independent tiles). ). Each provided bitstream or sub-bitstream is then encapsulated into a plurality of sub-picture tracks 6014 by a file/segment encapsulation means 650 (corresponding to encapsulation module 150). A sub-picture track is a track that embeds data for a sub-portion of a picture or image, typically a spatial part or a rectangular region. A subpicture track may be associated with other subpicture tracks or tracks that describe the full picture from which the subpicture was extracted.

For example, a sub-picture track can be a tile track. It can also be represented as an AVC track, HEVC track, HEVC tile track, or any compressed video bitstream encapsulated as a sequence of samples. Subpicture tracks from the same video source can be grouped using a track group mechanism. For example, OMAF is considering a track_group_type of "2dcc" to express a two-dimensional relationship between video tracks.

The packaging or encapsulation format used in 650 can follow, for example, the ISO base media file format and ISO/IEC 14496-15 defined by the MPEG standards body. The resulting file or segment file may be a single mp4 file or mp4 segment. During encapsulation, an audio stream can be added to the video bitstream. A metadata track is also added that provides descriptive information (metadata) about the video sequence and added audio streams.

The encapsulated file or segment file is delivered via a distribution means 660, e.g. over an IP network such as the Internet using the http (hypertext transfer protocol) protocol, or via a removable device such as a disk or a USB key. It is delivered to client device 670 on digital media. By way of example, the distribution means 660 implements adaptive streaming over HTTP, such as the MPEG standards committee's DASH (Dynamic Adaptive Streaming over HTTP) ("ISO/IEC 23009-1, HTTP Dynamic Adaptive Streaming (DASH), Part 1: Media Presentation Description and Segment Format”). The distribution means may include a streaming server 661 and a streaming client 662. The media presentation description may provide a description and URL of a media segment that corresponds to a track that encapsulates a video sequence containing full images, or may provide a description and URL of a media segment that corresponds only to a subpicture track. Alternatively, a description and URL of the media segment that corresponds to both may be provided. The media presentation description may provide alternative groups of sub-picture tracks, each group allowing a different level of reconstruction of the scene captured by camera 600. The alternative may be a different division (coarse grid or fine grid associated with the division means 6012), for example in terms of resolution, quality or bit rate.

Upon receipt by streaming client 662, the encapsulated media file or media segment is parsed by file/segment decapsulation means 671 (corresponding to decapsulation module 100) to extract one or more data streams. Ru. The extracted data stream is decoded by decoding means 672. For ISOBMFF files or segments received by the file/segment decapsulator 671, parsing is typically handled by an mp4 reader or mp4 parser. A parser may extract the encapsulated video bitstream and/or video sub-bitstream from the descriptive metadata.

Optionally, the decoded images or sub-images of the video sequence provided by the decoding means 672 are then combined by the rendering means 674 into a resultant image for video rendering. The rendered video is displayed on a display means 675, such as a screen (user device).

Note that video rendering depends on several parameters, including display size and client processing power. The rendering may be configured to display only a subset of the parsed and decoded sub-picture tracks. This may be controlled by the rendering means 674 or directly by content selection by the streaming client 662.

Transmitting and rendering multiple images of a VHD (Very High Definition) video stream has been observed to result in a very high bit rate and high resolution media data stream. Therefore, considering the overall system, access to media data needs to be optimized in order to avoid wasting bandwidth and comply with the processing capabilities of the client player.

Specifically, the media data stream can be used to display images on a dedicated display, such as an array of projectors. It can also be used to display specific regions of interest within the captured video 6011.

FIG. 7 shows the process of encapsulating 2D video (by means 650) into multiple tracks. At step 700, the server determines whether to encapsulate the encoded input bitstream(s) as a single track or multiple tracks. If single track encapsulation is on (test 700 is "yes"), the video is encapsulated as a single track (step 710), optionally including a NAL that indicates which NAL unit corresponds to which region. With unit mapping. If multiple tracks are to be generated (test 700 is "no"), e.g. if the splitting is performed by means 6012 of FIG. can be added. A composite or reconstructed track makes it possible to provide an entry point or "main" or "default" track for a parser or player. For example, a composite track or a reconstructed track has a valid flag value set in the track header, which is used in the movie and optionally as a preview. Tracks referenced from a composite track may not have these flag values set (other than the track_enable flag value) to hide these tracks from selection by the client, player, or user. If no composite or reconstruction track exists, then in step 730 the encoded media file and each bitstream or sub-bitstream is encapsulated into its own track.

The optional step may be configured to reduce the number of tracks by aggregating bitstreams or sub-bitstreams to form a larger region than the original partitioned one. If the encapsulation provides a composite track or a reconstruction track (test 720 is "yes"), two options are possible for the sample reconstruction rule: implicit or explicit reconstruction representation in the media file.

In the case of implicit reconstruction (test 740 is "true", branch is "yes"), in step 741 the synthesis or reconstruction track is a tile-based track (e.g. , track with "hvt1" sample entry). Each sub-picture track is then encapsulated as a tile track in step 743 according to this tile base track as specified in ISO/IEC 14496-15. Note that in addition to the "trif" descriptor for tile tracks, each tile track can also be declared as part of the same track group for 2D spatial relationship description.

If the composite or reconstruction track is provided as a track with an extractor for explicit reconstruction (test 740 "false", branch "no"), then in step 742 additional tracks are added to the media file. Created in This track primarily contains an extractor according to an embodiment of the invention, ie allowing copying of syntax structures (data entities) like samples, NAL units or slices. This track may also allow extraction from a set or group of alternative or equivalent tracks, such as a new extractor or constructor according to the previous embodiment. The created tracks reference each sub-picture track created in step 744, for example with a "scal" track reference type. If no composite or reconstruction track is provided (test 720 is "false", branch is "no"), then in step 730 the video portion of the media is encapsulated as a sub-picture track. Note that even if there are composite tracks or reconstruction tracks, sub-picture tracks may be grouped via a track group mechanism.

Finally, at step 750, a description for the spatial organization and relationships between subpicture tracks is generated. A 2D spatial relationship description track group box is added to each sub-picture track to describe the relative position and size of each sub-picture track within the original video source.

FIG. 8 illustrates an example of encapsulation of sub-picture tracks including several track groups to describe 2D spatial relationships. This example applies to both 2D and omnidirectional videos.

In this example, tracks #1 to #4 belong to a track group 81 of type “2dcc” where track_group_id is 10 and source_id 800 is 1. Tracks #5 to #8 belong to another track group 82 of type "2dcc" whose track_group_id is 20 and source_id 800 is 1. Further, there is a third track group 83 of type “2dcc” in which track_group_id is 30 and source_id 801 is 2. Additionally, there are several alternative groups 84-87. All tracks that belong to the same alternate group (i.e. have the same alternate_group identifier in the track header box 'tkhd' or all declare 'trgr' boxes with the same track_group_type and track_group_id) have alternate data Specify a group or collection of tracks containing. Alternative data includes alternative bitrates, codecs, languages, packet sizes, etc. These differentiated attributes can be displayed in the track selection box. Tracks within an alternate group only need to be played or streamed one at a time. In this example, tracks #1, #5, and #9 belong to the same alternate group 84 whose identifier is 100. For example, track #1 and track #5 are alternate tracks with different qualities, and track #9 is an alternate track for track #1 and track #5 from a codec perspective. Tracks #2, #6, and #10 belong to the same alternative group 85 whose identifier is 200. For example, track #2 and track #6 are alternative tracks with different resolutions, and track #10 is It is an alternative track to track #2 and track #6 in terms of frame rate, and so on.

Track groups 81 and 82 have the same source_id 800 and track group 83 has a different source_id 801, which means that the sub-picture tracks belonging to track groups 81 and 82 (with respect to other constraints, i.e. approximately one sub-picture track per alternative group) tracks) can be combined. Conversely, the sub-picture tracks of track group 83 cannot be combined with the sub-picture tracks of track groups 81 and 82 because they do not have the same source_id even if they belong to the same alternative group. The source_id parameter indicates to the player which sub-picture tracks can be part of the same spatial configuration. For a given spatial location, one sub-picture track can be considered visually equivalent to another sub-picture track at the same given spatial location. This is useful for (sub-picture) track selection when media content is provided in multiple tracks. Furthermore, depending on the selected sub-picture track, it can be dynamically adapted (quality/bitrate and resolution) to display the same spatial configuration. The configuration of FIG. 8 allows a writer or packager at 650 (FIG. 6) or 150 (FIG. 1) to create a reconstruction or composition track for arranging spatial portions of different tracks into a larger image for rendering. Can be built. According to embodiments of the invention, reconstructed or composite tracks can reference track groups using the proposed extractor with different copy modes. This is useful, for example, to indicate that the spatial part corresponding to track #1 or track #5 may be reconstructed by either of these two tracks. This is indicated by a writer or packager having an extractor that references alternative groups equal to 100 (84). Similarly, the data of the second spatial part can be extracted from one track of the alternative group 85. Again, the reconstructed track includes an extractor with copy_mode set to "sample" mode and references the track group.

Figures 9a and 9b show an example of explicit reconstruction from a set or group of tracks. In the illustrated example, the groups of tracks correspond to alternative sets of sub-picture tracks.

According to embodiments of the invention, a new type of extractor is used in the extractor or reconstruction track, such as 900 in Figure 9a or 950 in Figure 9b. In Figures 9a and 9b, a particular track reference type "esra" 901 or 951 is shown to indicate "explicit spatial reconstruction with substitution". A track reference of type "scal" may be used instead. The step "track selection" 902 or 952 corresponds to the step performed by the parser to select a track within the track group referenced by the track group according to 901 or 951. 9a is very similar to the configuration on FIG. 8, i.e. the alternative tracks are indicated by track groups of type "alte" (903 and 904), whereas FIG. 9b shows e.g. ) are considering alternative ways of writing alternative tracks. The subset is identified by a unique identifier that may also be used as a ref_index in the sample constructor (track 950 with ID #100) with a specific alternative. The subset identifier can be declared as a parameter of the "2dsr" box. Using a subset_identifier costs 4 bytes per subpicture track, and declaring an "alte" track costs 24 bytes per subpicture track.

FIG. 10 is a schematic block diagram of a computing device 1000 for implementing one or more embodiments of the invention. Computing device 1000 may be a device such as a microcomputer, a workstation, or a light portable device. The computer device 1000 is
- central processing unit (CPU) 1001, such as a microprocessor;
- in embodiments of the invention, similar to registers adapted to record variables and parameters necessary for reading and writing manifests and/or encoding video and/or reading and writing data under a given file format; a random access memory (RAM) 1002 for storing executable code for such a method, the memory capacity of which is expandable, for example, by an optional RAM connected to an expansion port;
- ROM (read-only memory) 1003 storing a computer program for implementing embodiments of the present invention;
- A network interface 1004, typically connected to a communication network through which the digital data to be processed is transmitted and received. Network interface 1004 may be a single network interface or may be comprised of a set of different network interfaces (eg, wired and wireless interfaces or different types of wired or wireless interfaces). Data is written to the network interface for transmission and read from the network interface for reception under the control of a software application running on CPU 1001;
- User interface (UI) 1005 for accepting input from the user and displaying information to the user;
-Hard disk (HD) 1006;
- I/O module 1007 for transmitting and receiving data to and from external devices such as video sources and displays
A communication bus connected to the

The executable code may be stored either in read-only memory 1003, on hard disk 1006, or on a removable digital medium, such as a disk. According to a variant, the executable code of the program is received by the communication network via the network interface 1004 in order to be stored in one of the storage means of the communication device 1000, such as a hard disk 1006, before being executed. can do.

The central processing unit 1001 is adapted to control and direct the execution of a program or instructions of a program or portions of software code according to embodiments of the invention, these instructions being stored in one of the aforementioned storage means. be done. After power-up, CPU 1001 may execute instructions from main RAM memory 1002 for software applications after those instructions are loaded from, for example, program ROM 1003 or hard disk (HD) 1006. Such a software application, when executed by CPU 1001, causes the steps of the flowcharts shown in the previous figures to be executed.

In this embodiment, the device is a programmable device that implements the invention using software. However, the invention may alternatively be implemented in hardware (eg, in the form of an application specific integrated circuit (ASIC)).

Although the invention has been described herein with reference to particular embodiments, it is not intended that the invention be limited to particular embodiments, and modifications that fall within the scope of this invention will be apparent to those skilled in the art. It will be.

For example, the present invention can be incorporated into devices such as cameras, smartphones, head-mounted displays, and tablets that function as remote controls for televisions and multimedia displays, for example to zoom in on a particular region of interest. You can also use the same device to customize your viewing of multimedia presentations by selecting specific areas of interest. These devices and methods also allow users to select and share their favorite video portions with other connected devices. Furthermore, if the surveillance camera is compatible with the data provision method according to the invention, it is also possible to use a smartphone or tablet to monitor what is happening in a particular area of the building under surveillance.

It is not intended that the scope of the invention be limited, but rather determined solely by the scope of the claims appended hereto. In particular, different features from different embodiments can be interchanged where appropriate.

Claims (11)

A method of encapsulating media data into a media file, the method comprising:
including in the media file a first track that includes a plurality of samples each including a set of one or more NAL units ;
including in the media file a second track that includes an extractor having a reference structure that references a sample included in the first track or one or more NAL units included within the sample;
including;
The extractor identifies and copies a data entity included in the sample or one or more NAL units included in the sample , including information that specifies a mode for copying from a plurality of predetermined modes. Contains mode attributes
Method. The plurality of modes are :
a first mode in which the data entity includes a set of all NAL units included in the sample;
a second mode in which the data entity includes one NAL unit of the set of NAL units;
a third mode in which the data entity comprises a payload of one NAL unit of the set of NAL units;
a fourth mode in which the data entity includes one NAL unit at a specified position within the set of NAL units;
a fifth mode in which the data entity includes a payload of one NAL unit at a specified position within the set of NAL units;
2. The method of claim 1 , comprising one or more of: 3. The method of claim 2 , wherein in the second or third mode, the one NAL unit is the first NAL unit in the sample. 3. The method of claim 2 , wherein in the second or third mode, the one NAL unit is the last NAL unit in the sample. A method of generating a media file, the method comprising:
encoding media content into media data;
encapsulating the media data into a plurality of tracks according to the method of claim 1;
generating at least one media file including the encapsulated media data;
method including. A method of transmitting a media file, the method comprising:
generating a media file by a server device according to the method of claim 5 ;
transmitting the generated media file to a client device;
method including. A method of parsing a media file to generate media data, the method comprising:
Obtaining a first track from the media file that includes an extractor having a reference structure that references a sample included in a second track or one or more NAL units included within the sample ;
obtaining the second track from the media file including a plurality of samples each including a set of one or more NAL units ;
retrieving data entities identified by a copy mode attribute included in the extractor from a sample of the second track;
including;
The copy mode attribute includes information that specifies, from a plurality of predetermined modes, a mode for identifying and copying a data entity included in the sample or one or more NAL units included in the sample.
Method. A computing device that encapsulates media data into a media file, the computing device comprising:
means for including in said media file a first track including a plurality of samples each including a set of one or more NAL units ;
means for including in the media file a second track that includes an extractor having a reference structure that references a sample contained in the first track or one or more NAL units contained within the sample;
Equipped with
The extractor identifies and copies a data entity included in the sample or one or more NAL units included in the sample , including information that specifies a mode for copying from a plurality of predetermined modes. Contains mode attributes
A computer device configured to: A computing device that parses a media file to generate media data, the computing device comprising:
means for obtaining a first track from the media file that includes an extractor having a reference structure that references a sample contained in a second track or one or more NAL units contained within the sample;
means for obtaining the second track from the media file, the second track including a plurality of samples each including a set of one or more NAL units ;
means for obtaining data entities identified by a copy mode attribute included in the extractor from a sample of the second track;
Equipped with
The copy mode attribute includes information that specifies, from a plurality of predetermined modes, a mode for identifying and copying a data entity included in the sample or one or more NAL units included in the sample.
A computer device configured to: A computer readable storage medium storing computer program instructions for implementing the method according to any one of claims 1 to 7 . 8. A computer program, which when executed causes the method of any one of claims 1 to 7 to be carried out.

Images