JP7704938B2 - Transmission method, reception method, transmission device, and reception device - Google Patents

Description

The present invention relates to a transmission method, a reception method, a transmission device, and a reception device.

As broadcasting and communication services become more advanced, the introduction of ultra-high definition video content such as 8K (7680 x 4320 pixels, hereinafter also referred to as 8K4K) and 4K (3840 x 2160 pixels, hereinafter also referred to as 4K2K) is being considered. A receiving device needs to decode the encoded data of the received ultra-high definition video in real time and display it, but video with a resolution such as 8K in particular imposes a large processing load when decoding, making it difficult to decode such video in real time with a single decoder. Therefore, a method is being considered that reduces the processing load per decoder and achieves real-time processing by parallelizing the decoding process using multiple decoders.

The encoded data is multiplexed based on a multiplexing method such as MPEG-2 TS (Transport Stream) or MMT (MPEG Media Transport) before being transmitted. For example, Non-Patent Document 1 discloses a technique for transmitting encoded media data packet by packet according to MMT.

Information technology - High efficiency coding and media delivery in heterogeneous environment - Part1: MPEG media transport (MMT), ISO/IEC DIS 23008-1

As broadcasting and communication services become more advanced, the introduction of ultra-high definition video content such as 8K and 4K (3840 x 2160 pixels) is being considered. In the MMT/TLV method, the reference clock on the sending side is synchronized with the 64-bit long format NTP defined in RFC 5905, and time stamps such as PTS (Presentation Time Stamp) and DTS (Decode Time Stamp) are added to the synchronous media based on the reference clock. Furthermore, the reference clock information on the sending side is sent to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information.

However, in such an MMT transmission method, when leap second adjustments are made to reference time information such as NTP, there is an issue that the multiple access units, which are the multiple second data units stored in the MPU, which is the first data unit received by the receiving device, cannot be decoded or presented at the intended time even in accordance with the DTS or PTS associated with the multiple access units.

The present invention provides a transmission method that allows a receiving device to reproduce a specified data unit at the intended time, even if leap second adjustments are made to the reference time information that serves as the basis for the reference clocks of the transmitting and receiving devices.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method for storing data constituting an encoded stream in a specified data unit and transmitting the data, which generates presentation time information indicating the presentation time of the specified data unit based on time information received from an external source, and transmits the specified data unit, the generated presentation time information, and identification information indicating whether the presentation time information was generated based on the time information before a leap second adjustment, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time from the time information, and the control information including a UTC-NPT reference descriptor indicating the relationship between UTC (Universal Time Coordinated) time and NPT (Normal Play Time) time.

A receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an encoded stream is stored, the method receiving the predetermined data unit, presentation time information indicating the presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before a leap second adjustment, and playing the received predetermined data unit based on the received presentation time information and the identification information, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time to the time information, and the control information including a UTC-NPT reference descriptor indicating the relationship between UTC (Universal Time Coordinated) time and NPT (Normal Play Time) time.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method for storing data constituting an encoded stream in a predetermined data unit and transmitting the data, which generates presentation time information indicating the presentation time of the predetermined data unit based on time information received from an external source, and transmits the predetermined data unit, the generated presentation time information, and identification information indicating whether the presentation time information was generated based on the time information before a leap second adjustment, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time from the time information, and the time information being NTP (Network Time Protocol).

A receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an encoded stream is stored, the method receiving the predetermined data unit, presentation time information indicating the presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before a leap second adjustment, and reproducing the received predetermined data unit based on the received presentation time information and the identification information, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time from the time information, and the time information being NTP (Network Time Protocol).

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method for storing data constituting an encoded stream in a specified data unit and transmitting the data, which generates presentation time information indicating the presentation time of the specified data unit based on time information received from an external source, and transmits the specified data unit, the generated presentation time information, and identification information indicating whether the presentation time information was generated based on the time information before a leap second adjustment, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time that is a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time to the time information.

A receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an encoded stream is stored, the method receiving the predetermined data unit, presentation time information indicating the presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before a leap second adjustment, and reproducing the received predetermined data unit based on the received presentation time information and the identification information, the identification information (i) being stored in a control message that is control information, and (ii) indicating whether the presentation time information was generated based on the time information from a time that is a predetermined period before the time immediately before the leap second adjustment to the time immediately before the leap second adjustment, the presentation time information being time information generated by adding a predetermined time to the time information.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method for storing data constituting an encoded stream in a specified data unit and transmitting the data, which generates presentation time information indicating the presentation time of the specified data unit based on reference time information received from an external source, and transmits the specified data unit, the generated presentation time information, and identification information indicating whether the presentation time information was generated based on the reference time information before leap second adjustment, and the identification information is stored in a control message, which is control information.

A receiving method according to one aspect of the present invention is a receiving method for receiving a predetermined data unit in which data constituting an encoded stream is stored, the method receiving the predetermined data unit, presentation time information indicating the presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before leap second adjustment, and reproducing the received predetermined data unit based on the received presentation time information and the identification information, and the identification information is stored in a control message, which is control information.

A transmitting device according to one aspect of the present invention is a transmitting device that stores data constituting an encoded stream in a specified data unit and transmits the data, and includes a generating unit that generates presentation time information indicating the presentation time of the specified data unit based on time information received from an external source, and a transmitting unit that transmits the specified data unit, the presentation time information generated by the generating unit, and identification information indicating whether the presentation time information was generated based on the time information before leap second adjustment, and the identification information is stored in a control message that is control information.

A transmission method according to one aspect of the present invention is a receiving device that receives a predetermined data unit in which data constituting an encoded stream is stored, the receiving device including a receiving unit that receives the predetermined data unit, presentation time information indicating the presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before leap second adjustment, and a reproduction unit that reproduces the predetermined data unit received by the receiving unit based on the presentation time information and the identification information received by the receiving unit, the identification information being stored in a control message that is control information.

In order to achieve the above object, a receiving method according to one aspect of the present invention is a receiving method by a receiving device that receives a first data unit in which data constituting an encoded stream is stored, the first data unit stores a plurality of second data units, and the receiving method includes receiving (i) the first data unit, (ii) first time information indicating a presentation time of the first data unit, (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decode time that requires correction, and that a second data unit other than the first second data unit in the presentation order among the plurality of second data units may be included in the first data unit, and (iv) second time information that is relative time information for calculating the presentation time or decode time of each of the plurality of second data units using the first time information, and determining whether the other second data unit is included in the first data unit indicated by the received identification information that the other second data unit may be included.

In addition, a receiving device according to one aspect of the present invention is a receiving device that receives a first data unit in which data constituting an encoded stream is stored, and the first data unit stores a plurality of second data units. The receiving device includes a receiving unit that receives (i) the first data unit, (ii) first time information indicating a presentation time of the first data unit, (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decoding time that requires correction, and that a second data unit other than the first second data unit in the presentation order among the plurality of second data units may be included in the first data unit, and (iv) second time information that is relative time information for calculating the presentation time or decoding time of each of the plurality of second data units using the first time information, and a determination unit that determines whether the other second data unit is included in the first data unit in which the identification information received by the receiving unit indicates that the other second data unit may be included.

In order to achieve the above object, a transmission method according to one aspect of the present invention is a transmission method for transmitting a first data unit in which data constituting an encoded stream is stored, the first data unit storing a plurality of second data units, and the transmission method generates first time information indicating a presentation time of the first data unit based on reference time information received from an external source, and transmits (i) the first data unit, (ii) the generated first time information, and (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decoding time that requires correction, and that a second data unit other than the first second data unit in the presentation order among the plurality of second data units may be included in the first data unit.

These general or specific aspects may be realized as a system, device, integrated circuit, computer program, or computer-readable recording medium such as a CD-ROM, or as any combination of a system, device, integrated circuit, computer program, and recording medium.

The present invention can reproduce multiple second data units stored in a first data unit at the intended time, even when leap second adjustments are made to the reference time information that serves as the basis for the reference clocks of the transmitting and receiving devices.

FIG. 1 is a diagram showing an example of dividing a picture into slice segments. FIG. 2 is a diagram showing an example of a PES packet sequence in which picture data is stored. FIG. 3 is a diagram showing an example of division of a picture according to the first embodiment. FIG. 4 is a diagram showing an example of dividing a picture according to a comparative example of the first embodiment. FIG. 5 is a diagram showing an example of data of an access unit according to the first embodiment. FIG. 6 is a block diagram of a transmitting device according to the first embodiment. FIG. 7 is a block diagram of a receiving device according to the first embodiment. FIG. 8 is a diagram illustrating an example of an MMT packet according to the first embodiment. FIG. 9 is a diagram showing another example of an MMT packet according to the first embodiment. FIG. 10 is a diagram illustrating an example of data input to each decoding unit according to the first embodiment. In FIG. FIG. 11 is a diagram showing an example of an MMT packet and header information according to the first embodiment. FIG. 12 is a diagram showing another example of data input to each decoding unit according to the first embodiment. In FIG. FIG. 13 is a diagram showing an example of division of a picture according to the first embodiment. FIG. 14 is a flowchart of a transmission method according to the first embodiment. FIG. 15 is a block diagram of a receiving device according to the first embodiment. FIG. 16 is a flowchart of a receiving method according to the first embodiment. FIG. 17 is a diagram showing an example of an MMT packet and header information according to the first embodiment. FIG. 18 is a diagram showing an example of an MMT packet and header information according to the first embodiment. FIG. 19 is a diagram showing the configuration of an MPU. FIG. 20 is a diagram showing the structure of the MF metadata. FIG. 21 is a diagram for explaining the data transmission order. FIG. 22 is a diagram showing an example of a method for performing decoding without using header information. FIG. 23 is a block diagram of a transmitting device according to the second embodiment. FIG. 24 is a flowchart of a transmission method according to the second embodiment. FIG. 25 is a block diagram of a receiving device according to the second embodiment. FIG. 26 is a flowchart of an operation for identifying the start position of an MPU and a NAL unit position. FIG. 27 is a flowchart of an operation of obtaining initialization information based on a transmission order type and decoding media data based on the initialization information. FIG. 28 is a flowchart of the operation of a receiving device when a low-delay presentation mode is provided. FIG. 29 is a diagram showing an example of a transmission order of MMT packets when auxiliary data is transmitted. FIG. 30 is a diagram for explaining an example in which a transmitting device generates auxiliary data based on the configuration of moof. FIG. 31 is a diagram for explaining reception of auxiliary data. FIG. 32 is a flowchart of a receiving operation using auxiliary data. Figure 33 shows the structure of an MPU made up of multiple movie fragments. Figure 34 is a diagram for explaining the transmission order of MMT packets when an MPU with the configuration of Figure 33 is transmitted. Figure 35 is the first diagram for explaining an example of operation of a receiving device when one MPU is composed of multiple movie fragments. Figure 36 is a second diagram for explaining an example of operation of a receiving device when one MPU is composed of multiple movie fragments. FIG. 37 is a flowchart showing the operation of the receiving method described with reference to FIGS. FIG. 38 is a diagram showing a case where non-VCL NAL units are individually aggregated as data units. FIG. 39 is a diagram showing a case where non-VCL NAL units are grouped together into a data unit. FIG. 40 is a flowchart of the operation of the receiving device when packet loss occurs. Figure 41 is a flowchart of the receiving operation when an MPU is divided into multiple movie fragments. FIG. 42 is a diagram showing an example of a prediction structure of pictures for each TemporalId when implementing temporal scalability. FIG. 43 is a diagram showing the relationship between the decoding time (DTS) and the display time (PTS) in each picture in FIG. FIG. 44 is a diagram showing an example of a prediction structure of a picture that requires picture delay processing and reorder processing. Figure 45 is a diagram showing an example in which an MPU in MP4 format is divided into multiple movie fragments and stored in an MMTP payload and an MMTP packet. FIG. 46 is a diagram for explaining the calculation method and problems of the PTS and DTS. FIG. 47 is a flowchart of the receiving operation when the DTS is calculated using information for DTS calculation. FIG. 48 is a diagram for explaining a method of storing a data unit in a payload in MMT. FIG. 49 shows an operational flow of a transmitting device according to the third embodiment. FIG. 50 shows an operational flow of the receiving device according to the third embodiment. FIG. 51 is a diagram showing an example of a specific configuration of a transmitting device according to the third embodiment. FIG. 52 is a diagram showing an example of a specific configuration of a receiving device according to the third embodiment. Figure 53 shows a method of storing non-timed media in an MPU and a method of transmitting it in an MMTP packet. FIG. 54 shows an example in which a file is divided and a plurality of divided data pieces obtained are packetized and transmitted. FIG. 55 shows another example in which a plurality of divided data pieces obtained by dividing a file are packetized and transmitted. FIG. 56 is a diagram showing the syntax of a loop for each file in the asset management table. FIG. 57 shows an operational flow for identifying a divided data number in a receiving device. FIG. 58 shows an operational flow for identifying the number of divided data in the receiving device. FIG. 59 is an operational flow for determining whether to operate a fragment counter in a transmitting device. FIG. 60 is a diagram for explaining a method for identifying the number of divided data and the divided data number (when using a fragment counter). Figure 61 shows the operational flow of a transmitting device when utilizing a fragment counter. Figure 62 shows the operational flow of a receiving device when utilizing a fragment counter. FIG. 63 shows a service configuration in which the same program is transmitted using multiple IP data flows. FIG. 64 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 65 is a diagram showing an example of a specific configuration of a receiving device. FIG. 66 shows an operational flow by the transmitting device. FIG. 67 shows the operational flow by the receiving device. FIG. 68 shows a receiving buffer model based on the receiving buffer model defined in ARIB STD-B60, particularly when only a broadcast transmission path is used. FIG. 69 is a diagram showing an example in which multiple data units are aggregated and stored in one payload. FIG. 70 shows an example in which multiple data units are aggregated and stored in one payload, where a video signal in a NAL size format is treated as one data unit. Figure 71 is a diagram showing the structure of the payload of an MMTP packet in which the data unit length is not indicated. FIG. 72 shows an example of the extend field assigned to each packet. FIG. 73 shows the operational flow of the receiving device. FIG. 74 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 75 is a diagram showing an example of a specific configuration of a receiving device. FIG. 76 shows an operational flow by the transmitting device. FIG. 77 shows the operational flow by the receiving device. FIG. 78 is a diagram showing a protocol stack of the MMT/TLV method defined in ARIB STD-B60. FIG. 79 is a diagram showing the structure of a TLV packet. FIG. 80 is a block diagram illustrating an example of a receiving device. FIG. 81 is a diagram illustrating a time stamp descriptor. FIG. 82 is a diagram for explaining leap second adjustment. Figure 83 is a diagram showing the relationship between NTP time, MPU timestamp, and MPU presentation timing. FIG. 84 is a diagram for explaining a method for correcting a timestamp on the transmitting side. FIG. 85 is a diagram for explaining a method for correcting a timestamp in a receiving device. Figure 86 shows the operational flow by the transmitting side (transmitting device) when correcting the MPU timestamp at the transmitting side (transmitting device). Figure 87 shows the operational flow by a receiving device when correcting an MPU timestamp on the transmitting side (transmitting device). Figure 88 shows the operational flow by the transmitting side (transmitting device) when correcting the MPU timestamp in the receiving device. Figure 89 shows the operational flow of a receiving device when correcting an MPU timestamp in the receiving device. FIG. 90 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 91 is a diagram showing an example of a specific configuration of a receiving device. FIG. 92 shows an operational flow by the transmitting device. FIG. 93 shows the operational flow by the receiving device. FIG. 94 is a diagram showing an example of an extension of the MPU extended time stamp descriptor. Figure 95 is a diagram to explain a case where a discontinuity occurs in the MPU sequence numbers due to an adjustment of the MPU sequence numbers. FIG. 96 is a diagram for explaining a case where the packet sequence numbers become discontinuous at the timing of switching from normal equipment to redundant equipment. Figure 97 shows the operational flow of a receiving device when a discontinuity occurs in the MPU sequence number or packet sequence number. FIG. 98 is a diagram for explaining a method for correcting a timestamp in a receiving device when leap seconds are inserted. FIG. 99 is a diagram for explaining a method for correcting a timestamp in a receiving device when leap seconds are deleted. FIG. 100 shows the operational flow of the receiving device. FIG. 101 is a diagram showing an example of a specific configuration of a transmission/reception system. FIG. 102 is a diagram showing a specific configuration of a receiving device. FIG. 103 shows the operational flow of the receiving device. FIG. 104 is a diagram for explaining a method for correcting a timestamp on the transmitting side (transmitting device) when leap seconds are inserted. FIG. 105 is a diagram for explaining a correction method for correcting a timestamp on the transmitting side (transmitting device) when leap seconds are deleted. FIG. 106 is a diagram showing the operation flow of the transmitting device described in FIG. FIG. 107 is a diagram showing the operation flow of the receiving device described in FIG. FIG. 108 is a diagram showing the operation flow of the transmitting device described in FIG. FIG. 109 is a diagram showing the operation flow of the receiving device described in FIG. FIG. 110 is a diagram showing an example of a specific configuration of a transmitting device. FIG. 111 is a diagram showing an example of a specific configuration of a receiving device. FIG. 112 shows the operational flow (transmission method) of the transmitting device. FIG. 113 shows the operation flow (receiving method) of the receiving device.

A transmission method according to one aspect of the present invention is a transmission method for transmitting a first data unit in which data constituting an encoded stream is stored, the first data unit storing a plurality of second data units, and the transmission method generates first time information indicating a presentation time of the first data unit based on reference time information received from an external source, and transmits (i) the first data unit, (ii) the generated first time information, and (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decoding time that requires correction, and that a second data unit other than the first second data unit in the presentation order among the plurality of second data units may be included in the first data unit.

This type of transmission method allows the receiving device to reproduce the encoded stream consisting of the first data unit at the intended time, even if leap second adjustments are made to the reference time information that serves as the basis for the reference clocks of the transmitting and receiving devices.

The transmission may further include transmitting second time information, which is relative time information for calculating the presentation time or decode time of each of the plurality of second data units using the first time information.

The first time information may also be time information that is subject to correction during leap second adjustment.

Furthermore, when the first time information satisfies a correction condition, the first time information is corrected, and in the transmission, information indicating whether the first time information was generated based on the reference time information before leap second adjustment in the generation and was not corrected in the correction is transmitted as the identification information, and when the identification information indicates that the first time information was generated based on the reference time information before leap second adjustment in the generation and was not corrected in the correction, it may indicate that the other second data unit is likely to be included in the first data unit.

The correction condition may also be that the first time information is generated based on the reference time information before leap second adjustment and is after a predetermined time.

The identification information may also include first identification information indicating whether the first time information was generated based on the reference time information before leap second adjustment, and second identification information indicating that the first time information has been corrected.

The first identification information may also be information indicating whether the first time information was generated based on the reference time information from a time a predetermined period before the time immediately before the leap second adjustment is performed to the time immediately before the leap second adjustment is performed.

The first time information may be absolute time information indicating the presentation time of the first second data unit in the presentation order among the plurality of second data units.

In addition, the transmission may include transmitting a predetermined packet that contains control information in which the first time information corresponding to the first data unit is stored, and a plurality of the first data units.

In addition, the transmission may include transmitting an MMTP (MPEG Media Transport Protocol) packet as the specified packet, in which a plurality of MPUs (Media Presentation Units) as the first data units are stored, the MMTP packet including, as control information, an MPU timestamp descriptor including the first time information and an MPU extended timestamp descriptor including the second time information, and the MPU may store a plurality of access units as the plurality of second data units.

The reference time information may also be NTP (Network Time Protocol).

In addition, a receiving method according to one aspect of the present invention is a receiving method by a receiving device that receives a first data unit in which data constituting an encoded stream is stored, the first data unit stores a plurality of second data units, and the receiving method includes receiving (i) the first data unit, (ii) first time information indicating a presentation time of the first data unit, and (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decoding time that requires correction, and that the first data unit may include other second data units other than the first second data unit in the presentation order among the plurality of second data units, and determining whether the other second data units are included in the first data unit indicated by the received identification information, which indicates that the other second data units may be included.

This receiving method can reproduce the encoded stream consisting of the first data unit at the intended time, even if leap second adjustments are made to the reference time information that serves as the basis for the reference clocks of the transmitting and receiving devices.

These comprehensive or specific aspects may be realized in a device, a system, an integrated circuit, a computer program, or a computer-readable recording medium such as a CD-ROM, or in any combination of a device, a system, an integrated circuit, a computer program, or a recording medium.

The following describes the embodiment in detail with reference to the drawings.

The embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present invention. Furthermore, among the components in the following embodiments, components that are not described in an independent claim that indicates a superordinate concept are described as optional components.

(Findings on which the present invention is based)
In recent years, the resolution of displays such as TVs, smartphones, and tablet terminals has been increasing. In particular, 8K4K (resolution 8K×4K) services are planned for broadcasting in Japan in 2020. Since it is difficult to decode ultra-high resolution moving images such as 8K4K in real time using a single decoder, a method of performing decoding processing in parallel using multiple decoders is being considered.

Since the encoded data is multiplexed and transmitted based on a multiplexing method such as MPEG-2 TS or MMT, the receiving device must separate the encoded video data from the multiplexed data prior to decoding. Hereinafter, the process of separating the encoded data from the multiplexed data is referred to as demultiplexing.

When parallelizing the decoding process, it is necessary to allocate the encoded data to be decoded to each decoder. When allocating the encoded data, the encoded data itself needs to be analyzed, and since the bit rate is particularly high for 8K content, the processing load associated with the analysis is large. As a result, the demultiplexing part becomes a bottleneck, making it impossible to play back in real time.

In video coding formats such as H.264 and H.265, which are standardized by MPEG and ITU, a transmitting device can divide a picture into multiple areas called slices or slice segments, and encode each divided area so that it can be decoded independently. Therefore, for example, in the case of H.265, a receiving device that receives a broadcast can separate data for each slice segment from the received data and output the data for each slice segment to a separate decoder, thereby achieving parallel decoding.

Figure 1 shows an example of dividing one picture into four slice segments in HEVC. For example, a receiving device has four decoders, and each decoder decodes one of the four slice segments.

In conventional broadcasting, a transmitting device stores one picture (an access unit in the MPEG system standard) in one PES packet and multiplexes the PES packet into a sequence of TS packets. For this reason, a receiving device had to separate the payload of the PES packet, analyze the data of the access unit stored in the payload, separate each slice segment, and output the data of each separated slice segment to a decoder.

However, the inventors have discovered that there is a problem in that analyzing access unit data and separating slice segments requires a large amount of processing, making it difficult to perform this processing in real time.

Figure 2 shows an example of how picture data divided into slice segments is stored in the payload of a PES packet.

As shown in FIG. 2, for example, data from multiple slice segments (slice segments 1 to 4) is stored in the payload of one PES packet. In addition, the PES packets are multiplexed into a sequence of TS packets.

(Embodiment 1)
In the following, an example will be described in which H.265 is used as the video encoding method, but this embodiment can also be applied to cases in which other encoding methods such as H.264 are used.

Figure 3 shows an example of dividing an access unit (picture) into division units in this embodiment. The access unit is divided into two equal parts horizontally and vertically, into a total of four tiles, by a function called tiles introduced by H.265. Also, there is a one-to-one correspondence between slice segments and tiles.

The reason for dividing the data into two equal parts horizontally and vertically will be explained below. First, when decoding, a line memory is generally required to store one horizontal line of data, but when it comes to ultra-high resolution such as 8K4K, the horizontal size becomes large, so the size of the line memory increases. When implementing a receiving device, it is desirable to be able to reduce the size of the line memory. In order to reduce the size of the line memory, vertical division is necessary. A data structure called a tile is required for vertical division. For these reasons, tiles are used.

On the other hand, images generally have high correlation in the horizontal direction, so coding efficiency improves when a wider range can be referenced horizontally. Therefore, from the viewpoint of coding efficiency, it is desirable to divide the access unit horizontally.

By dividing the access unit into two equal parts horizontally and vertically, these two characteristics can be achieved simultaneously, and both implementation and coding efficiency can be taken into consideration. If a single decoder is capable of decoding 4K2K video in real time, then an 8K4K image can be divided into four equal parts, and each slice segment can be divided into 4K2K, allowing the receiving device to decode the 8K4K image in real time.

Next, we will explain why tiles obtained by dividing an access unit in the horizontal and vertical directions correspond one-to-one to slice segments. In H.265, an access unit is composed of multiple units called NAL (Network Adaptation Layer) units.

The payload of a NAL unit stores any of the following: an access unit delimiter that indicates the start position of an access unit; an SPS (Sequence Parameter Set), which is initialization information used commonly by sequence unit during decoding; a PPS (Picture Parameter Set), which is initialization information used commonly within a picture during decoding; SEI (Supplemental Enhancement Information), which is not required for the decoding process itself but is required for processing and displaying the decoding results; and encoded data of a slice segment. The header of the NAL unit contains type information for identifying the data stored in the payload.

Here, the transmitting device transmits the encoded data in the MPEG-2 TS, MMT (MPEG Media Transport), MPEG DASH (Dynamic Adaptive
When multiplexing using a multiplexing format such as HTTP (Streaming over HTTP) or RTP (Real-time Transport Protocol), the basic unit can be set to the NAL unit. In order to store one slice segment in one NAL unit, it is desirable to divide the access unit into slice segment units when dividing the access unit into regions. For this reason, the transmitting device associates tiles with slice segments in a one-to-one relationship.

As shown in FIG. 4, the transmitting device can also set tiles 1 to 4 together as one slice segment. In this case, however, all tiles are stored in one NAL unit, making it difficult for the receiving device to separate the tiles in the multiplexing layer.

Note that there are two types of slice segments: independent slice segments that can be decoded independently, and reference slice segments that reference independent slice segments. Here, we will explain the case where independent slice segments are used.

Figure 5 is a diagram showing an example of data of an access unit that has been divided so that the boundaries between tiles and slice segments coincide as shown in Figure 3. The data of the access unit includes a NAL unit in which an access unit delimiter is stored, which is placed at the beginning, followed by NAL units of SPS, PPS, and SEI, and data of a slice segment in which data from tiles 1 to 4 is stored, which is placed after that. Note that the data of the access unit does not have to include some or all of the NAL units of SPS, PPS, and SEI.

Next, the configuration of the transmitting device 100 according to this embodiment will be described. FIG. 6 is a block diagram showing an example of the configuration of the transmitting device 100 according to this embodiment. This transmitting device 100 includes an encoding unit 101, a multiplexing unit 102, a modulation unit 103, and a transmitting unit 104.

The encoding unit 101 generates encoded data by encoding the input image, for example, according to H.265. In addition, the encoding unit 101 divides an access unit into four slice segments (tiles) and encodes each slice segment, for example, as shown in FIG. 3.

The multiplexing unit 102 multiplexes the encoded data generated by the encoding unit 101. The modulation unit 103 modulates the data obtained by the multiplexing. The transmission unit 104 transmits the modulated data as a broadcast signal.

Next, the configuration of the receiving device 200 according to this embodiment will be described. FIG. 7 is a block diagram showing an example of the configuration of the receiving device 200 according to this embodiment. This receiving device 200 includes a tuner 201, a demodulation unit 202, a demultiplexing unit 203, a plurality of decoding units 204A to 204D, and a display unit 205.

The tuner 201 receives a broadcast signal. The demodulation unit 202 demodulates the received broadcast signal. The demodulated data is input to the demultiplexing unit 203.

The demultiplexer 203 separates the demodulated data into division units, and outputs the data for each division unit to the decoders 204A to 204D. Here, a division unit is a divided area obtained by dividing an access unit, for example, a slice segment in H.265. Also, here, an 8K4K image is divided into four 4K2K images. Therefore, there are four decoders 204A to 204D.

The multiple decoding units 204A to 204D operate in synchronization with one another based on a predetermined reference clock. Each decoding unit decodes the coded data of the division unit according to the DTS (Decoding Time Stamp) of the access unit, and outputs the decoding result to the display unit 205.

The display unit 205 generates an 8K4K output image by integrating the multiple decoding results output from the multiple decoding units 204A to 204D. The display unit 205 displays the generated output image according to the PTS (Presentation Time Stamp) of the access unit obtained separately. When integrating the decoding results, the display unit 205 may perform filtering such as a deblocking filter in boundary areas between adjacent division units, such as tile boundaries, so that the boundaries are visually less noticeable.

In the above, the transmitting device 100 and the receiving device 200 that transmit or receive broadcasts have been described as examples, but content may be transmitted and received via a communication network. When the receiving device 200 receives content via a communication network, the receiving device 200 separates the multiplexed data from IP packets received via a network such as Ethernet.

In broadcasting, the transmission path delay from when the broadcast signal is transmitted until it reaches the receiving device 200 is constant. On the other hand, in a communication network such as the Internet, due to the effects of congestion, the transmission path delay from when the data transmitted from the server reaches the receiving device 200 is not constant. Therefore, the receiving device 200 often does not perform strictly synchronized playback based on a reference clock such as the PCR in the MPEG-2 TS of the broadcast. Therefore, the receiving device 200 may display the 8K4K output image on the display unit according to the PTS, without strictly synchronizing each decoding unit.

In addition, due to congestion in the communication network, etc., the decoding process for all division units may not be completed by the time indicated by the PTS of the access unit. In this case, the receiving device 200 skips displaying the access unit, or delays displaying until decoding of at least four division units is completed and generation of the 8K4K image is completed.

Content may be transmitted and received using a combination of broadcasting and communication. This method can also be applied when playing back multiplexed data stored on a recording medium such as a hard disk or memory.

Next, we will explain how to multiplex access units divided into slice segments when MMT is used as the multiplexing method.

Figure 8 shows an example of packetizing data of an HEVC access unit into MMT. Although SPS, PPS, SEI, etc. do not necessarily need to be included in the access unit, here we will show an example in which they are present.

NAL units that are located before the first slice segment in an access unit, such as the access unit delimiter, SPS, PPS, and SEI, are stored together in MMT packet #1. Subsequent slice segments are stored in separate MMT packets for each slice segment.

As shown in FIG. 9, a NAL unit that is placed before the first slice segment in an access unit may be stored in the same MMT packet as the first slice segment.

In addition, when a NAL unit such as End-of-Sequence or End-of-Bitstream, which indicates the end of a sequence or stream, is added after the final slice segment, it is stored in the same MMT packet as the final slice segment. However, since NAL units such as End-of-Sequence or End-of-Bitstream are inserted at the end point of the decoding process or the connection point of two streams, it may be desirable for the receiving device 200 to easily obtain these NAL units in the multiplexing layer. In this case, these NAL units may be stored in a different MMT packet from the slice segments. This allows the receiving device 200 to easily separate these NAL units in the multiplexing layer.

Note that TS, DASH, RTP, or the like may be used as the multiplexing method. In these methods, the transmitting device 100 stores different slice segments in different packets. This ensures that the receiving device 200 can separate the slice segments at the multiplexing layer.

For example, when TS is used, the encoded data is packetized into PES packets on a slice segment basis. When RTP is used, the encoded data is packetized into RTP packets on a slice segment basis. Even in these cases, the NAL unit and the slice segment that are placed before the slice segment may be packetized separately, as in MMT packet #1 shown in FIG. 8.

When TS is used, the transmitting device 100 indicates the unit of data stored in the PES packet by using a data alignment descriptor, for example. Also, since DASH is a method of downloading MP4 format data units called segments via HTTP, the transmitting device 100 does not packetize the encoded data when transmitting. For this reason, the transmitting device 100 may create subsamples in slice segment units and store information indicating the storage position of the subsamples in the MP4 header so that the receiving device 200 can detect slice segments in the multiplexing layer in the MP4.

The MMT packetization of slice segments is explained in detail below.

As shown in FIG. 8, by packetizing the encoded data, data that is commonly referenced when decoding all slice segments in an access unit, such as SPS and PPS, is stored in MMT packet #1. In this case, receiving device 200 concatenates the payload data of MMT packet #1 with the data of each slice segment, and outputs the obtained data to the decoding unit. In this way, receiving device 200 can easily generate input data for the decoding unit by concatenating the payloads of multiple MMT packets.

Figure 10 is a diagram showing an example in which input data to the decoders 204A to 204D is generated from the MMT packets shown in Figure 8. The demultiplexer 203 generates data necessary for the decoder 204A to decode slice segment 1 by linking the payload data of MMT packet #1 and MMT packet #2. The demultiplexer 203 similarly generates input data for the decoders 204B to 204D. That is, the demultiplexer 203 generates input data for the decoder 204B by linking the payload data of MMT packet #1 and MMT packet #3. The demultiplexer 203 generates input data for the decoder 204C by linking the payload data of MMT packet #1 and MMT packet #4. The demultiplexer 203 generates input data for the decoder 204D by linking the payload data of MMT packet #1 and MMT packet #5.

In addition, the demultiplexer 203 may remove NAL units that are not necessary for the decoding process, such as the access unit delimiter and SEI, from the payload data of MMT packet #1, and separate only the NAL units of the SPS and PPS that are necessary for the decoding process and add them to the slice segment data.

When the encoded data is packetized as shown in FIG. 9, the demultiplexer 203 outputs MMT packet #1, which includes the first data of the access unit in the multiplexing layer, to the first decoder 204A. The demultiplexer 203 also analyzes the MMT packet, which includes the first data of the access unit in the multiplexing layer, separates the SPS and PPS NAL units, and adds the separated SPS and PPS NAL units to each of the data of the second and subsequent slice segments to generate input data for each of the second and subsequent decoders.

Furthermore, it is desirable that the receiving device 200 can use information included in the header of the MMT packet to identify the type of data stored in the MMT payload and the index number of the slice segment in the access unit when the slice segment is stored in the payload. Here, the type of data is either pre-slice segment data (the NAL units arranged before the first slice segment in the access unit are collectively referred to as such) or slice segment data. When storing a unit obtained by fragmenting an MPU such as a slice segment in an MMT packet, a mode for storing MFU (Media Fragment Unit) is used. When using this mode, the transmitting device 100 can set, for example, the Data Unit, which is the basic unit of data in the MFU, to a sample (a data unit in MMT, equivalent to an access unit) or a subsample (a unit obtained by dividing a sample).

At this time, the header of the MMT packet includes a field called a fragmentation indicator and a field called a fragment counter.

The Fragmentation indicator indicates whether the data stored in the payload of the MMT packet is a fragmented Data unit, and if so, whether the fragment is the first or last fragment in the Data unit, or a fragment that is neither the first nor the last. In other words, the fragmentation indicator included in the header of a packet is identification information that indicates whether (1) the packet is included in the basic data unit, that is, the data unit, only the packet in question, (2) the data unit is divided and stored into multiple packets, and the packet in question is the first packet of the data unit, (3) the data unit is divided and stored into multiple packets, and the packet in question is a packet other than the first or last packet of the data unit, or (4) the data unit is divided and stored into multiple packets, and the packet in question is the last packet of the data unit.

The fragment counter is an index number that indicates which fragment in the data unit the data stored in the MMT packet corresponds to.

Therefore, by the transmitting device 100 setting the samples in the MMT to Data units and setting the data before the slice segment and each slice segment to fragment units of Data units, the receiving device 200 can identify the type of data stored in the payload using the information contained in the header of the MMT packet. In other words, the demultiplexing unit 203 can generate input data for each of the decoding units 204A to 204D by referring to the header of the MMT packet.

Figure 11 shows an example where a sample is set to a Data unit, and pre-slice segment data and slice segments are packetized as fragments of Data units.

The pre-slice segment data and the slice segment are divided into five fragments, fragment #1 to fragment #5. Each fragment is stored in an individual MMT packet. At this time, the values of the fragmentation indicator and fragment counter included in the header of the MMT packet are as shown in the figure.

For example, the Fragment indicator is a 2-bit binary value. The Fragment indicator of MMT packet #1, which is the head of the Data unit, the Fragment indicator of MMT packet #5, which is the last packet, and the Fragment indicators of MMT packets #2 to #4, which are packets in between, are each set to a different value. Specifically, the Fragment indicator of MMT packet #1, which is the head of the Data unit, is set to 01, the Fragment indicator of MMT packet #5, which is the last packet, is set to 11, and the Fragment indicators of MMT packets #2 to #4, which are packets in between, are set to 10. If a data unit contains only one MMT packet, the fragment indicator is set to 00.

Fragment counter is 4 in MMT packet #1, which is the total number of fragments (5) minus 1, and is decreased by 1 in each of the subsequent packets, until it is 0 in the final MMT packet #5.

Therefore, the receiving device 200 can identify the MMT packet that stores the pre-slice segment data by using either the fragment indicator or the fragment counter. The receiving device 200 can also identify the MMT packet that stores the Nth slice segment by referring to the fragment counter.

The header of the MMT packet also includes a sequence number within the MPU of the Movie Fragment to which the Data unit belongs, a sequence number for the MPU itself, and a sequence number within the Movie Fragment of the sample to which the Data unit belongs. By referring to these, the demultiplexer 203 can uniquely determine the sample to which the Data unit belongs.

Furthermore, since the demultiplexing unit 203 can determine the index number of the fragment within the data unit from the fragment counter, etc., it can uniquely identify the slice segment stored in the fragment even if packet loss occurs. For example, even if the demultiplexing unit 203 is unable to obtain fragment #4 shown in FIG. 11 due to packet loss, it can determine that the fragment received next after fragment #3 is fragment #5, and therefore can correctly output slice segment 4 stored in fragment #5 to decoding unit 204D instead of decoding unit 204C.

When a transmission path that guarantees no packet loss is used, the demultiplexer 203 can process the arriving packets periodically without referring to the header of the MMT packet to determine the type of data stored in the MMT packet or the index number of the slice segment. For example, when an access unit is transmitted by a total of five MMT packets including pre-slice data and four slice segments, the receiving device 200 can obtain the pre-slice data and the data of the four slice segments in sequence by processing the received MMT packets in sequence after determining the pre-slice data of the access unit from which decoding is to be started.

Below, we explain some variations on packetization.

Slice segments do not necessarily have to be divided both horizontally and vertically within the plane of an access unit; as shown in FIG. 1, they may be divided only horizontally or only vertically.

Also, if the access unit is divided only horizontally, tiles do not need to be used.

The number of divisions within an access unit is arbitrary and is not limited to four. However, the area size of slice segments and tiles must be equal to or larger than the lower limit of coding standards such as H.265.

The transmitting device 100 may store identification information indicating the division method within the surface of the access unit in an MMT message or a TS descriptor. For example, information indicating the number of divisions in the horizontal and vertical directions within the surface may be stored. Alternatively, a unique identification information may be assigned to the division method, such as dividing the access unit into two equal parts in the horizontal and vertical directions as shown in FIG. 3, or dividing the access unit into four equal parts in the horizontal direction as shown in FIG. 1. For example, if the access unit is divided as shown in FIG. 3, the identification information indicates mode 1, and if the access unit is divided as shown in FIG. 1, the identification information indicates mode 1.

In addition, information indicating constraints on coding conditions related to the intra-plane division method may be included in the multiplexing layer. For example, information indicating that one slice segment is composed of one tile may be used. Alternatively, information indicating that the reference block when performing motion compensation during decoding of a slice segment or tile is limited to a slice segment or tile at the same position in the screen, or limited to blocks within a predetermined range in an adjacent slice segment may be used.

The transmitting device 100 may also switch whether to divide an access unit into multiple slice segments depending on the resolution of the video. For example, the transmitting device 100 may not perform intra-plane division when the video to be processed has a resolution of 4K2K, but may divide the access unit into four when the video to be processed has a resolution of 8K4K. By predefining the division method for 8K4K video, the receiving device 200 can acquire the resolution of the video to be received, determine whether to perform intra-plane division and the division method, and switch the decoding operation.

Furthermore, the receiving device 200 can detect whether or not there is intra-plane division by referring to the header of the MMT packet. For example, if the access unit is not divided, if the MMT Data unit is set to a sample, the Data unit is not fragmented. Therefore, the receiving device 200 can determine that the access unit is not divided if the value of the Fragmentation counter included in the header of the MMT packet is always zero. Alternatively, the receiving device 200 may detect whether the value of the Fragmentation indicator is always 01. The receiving device 200 can also determine that the access unit is not divided if the value of the Fragmentation indicator is always 01.

The receiving device 200 can also handle cases where the number of divisions within a plane in an access unit does not match the number of decoding units. For example, if the receiving device 200 has two decoding units 204A and 204B that can decode 8K2K encoded data in real time, the demultiplexing unit 203 outputs two of the four slice segments that make up the 8K4K encoded data to the decoding unit 204A.

Figure 12 is a diagram showing an example of operation when MMT packetized data as shown in Figure 8 is input to two decoders 204A and 204B. Here, it is desirable for the receiving device 200 to be able to integrate and output the decoding results of the decoders 204A and 204B as is. Therefore, the demultiplexer 203 selects slice segments to output to each of the decoders 204A and 204B so that the decoding results of each of the decoders 204A and 204B are spatially continuous.

The demultiplexer 203 may select a decoder to use depending on the resolution or frame rate of the encoded data of the video. For example, if the receiving device 200 has four 4K2K decoders, and the resolution of the input image is 8K4K, the receiving device 200 performs the decoding process using all four decoders. Also, if the resolution of the input image is 4K2K, the receiving device 200 performs the decoding process using only one decoder. Alternatively, even if the plane is divided into four, if 8K4K can be decoded in real time by a single decoder, the demultiplexer 203 integrates all the divided units and outputs them to one decoder.

Furthermore, the receiving device 200 may determine the decoding unit to be used taking into consideration the frame rate. For example, if the receiving device 200 has two decoding units with an upper limit of 60 fps for the frame rate that can be decoded in real time when the resolution is 8K4K, there may be cases where 8K4K encoded data at 120 fps is input. In this case, if the plane is composed of four division units, slice segment 1 and slice segment 2 are input to the decoding unit 204A, and slice segment 3 and slice segment 4 are input to the decoding unit 204B, as in the example of FIG. 12. Since each of the decoding units 204A and 204B can decode up to 120 fps in real time if the resolution is 8K2K (half of 8K4K), the decoding process is performed by these two decoding units 204A and 204B.

Even if the resolution and frame rate are the same, the processing amount differs if the profile or level in the encoding method, or the encoding method itself, such as H.264 or H.265, is different. Therefore, the receiving device 200 may select the decoding unit to be used based on this information. Note that, when the receiving device 200 cannot decode all the encoded data received by broadcasting or communication, or when it cannot decode all the slice segments or tiles that constitute the area selected by the user, it may automatically determine the slice segments or tiles that can be decoded within the processing range of the decoding unit. Alternatively, the receiving device 200 may provide a user interface for the user to select the area to be decoded. At this time, the receiving device 200 may display a warning message indicating that all areas cannot be decoded, or may display information indicating the number of areas, slice segments, or tiles that can be decoded.

The above method can also be applied when MMT packets storing slice segments of the same encoded data are transmitted and received using multiple transmission paths, such as broadcasting and communication.

In addition, the transmitting device 100 may perform coding so that the areas of each slice segment overlap in order to make the boundaries between the division units less noticeable. In the example shown in FIG. 13, an 8K4K picture is divided into four slice segments 1 to 4. Each of slice segments 1 to 3 is, for example, 8K x 1.1K, and slice segment 4 is 8K x 1K. Adjacent slice segments also overlap each other. In this way, motion compensation during coding can be efficiently performed at the boundaries when divided into four, as shown by the dotted lines, improving the image quality of the boundary parts. In this way, image quality degradation at the boundary parts is reduced.

In this case, the display unit 205 cuts out an 8K x 1K area from the 8K x 1.1K area and integrates the resulting area. Note that the transmitting device 100 may separately transmit information indicating whether the slice segments are coded with overlap and the extent of the overlap, by including the information in the multiplexing layer or coded data.

Note that a similar technique can be applied when tiles are used.

The flow of operation of the transmitting device 100 will be described below. FIG. 14 is a flowchart showing an example of the operation of the transmitting device 100.

First, the encoding unit 101 divides a picture (access unit) into a plurality of slice segments (tiles), which are a plurality of regions (S101). Next, the encoding unit 101 generates encoded data corresponding to each of the plurality of slice segments by encoding each of the plurality of slice segments so that the slice segments can be decoded independently (S102). Note that the encoding unit 101 may encode the plurality of slice segments using a single encoding unit, or may process the slice segments in parallel using multiple encoding units.

Next, the multiplexing unit 102 multiplexes the multiple encoded data generated by the encoding unit 101 by storing the multiple encoded data in multiple MMT packets (S103). Specifically, as shown in Figs. 8 and 9, the multiplexing unit 102 stores the multiple encoded data in multiple MMT packets so that encoded data corresponding to different slice segments is not stored in one MMT packet. Also, as shown in Fig. 8, the multiplexing unit 102 stores control information commonly used for all decoding units in a picture in MMT packet #1, which is different from the multiple MMT packets #2 to #5 in which the multiple encoded data are stored. Here, the control information includes at least one of an access unit delimiter, an SPS, a PPS, and an SEI.

In addition, the multiplexing unit 102 may store the control information in the same MMT packet as one of the multiple MMT packets in which the multiple encoded data are stored. For example, as shown in FIG. 9, the multiplexing unit 102 may store the control information in the first MMT packet (MMT packet #1 in FIG. 9) of the multiple MMT packets in which the multiple encoded data are stored.

Finally, the transmitting device 100 transmits multiple MMT packets. Specifically, the modulation unit 103 modulates the data obtained by multiplexing, and the transmitting unit 104 transmits the modulated data (S104).

Fig. 15 is a block diagram showing an example of the configuration of the receiving device 200, and is a diagram showing in detail the configuration of the demultiplexing unit 203 and the subsequent stages shown in Fig. 7. As shown in Fig. 15, the receiving device 200 further includes a decoding command unit 206. The demultiplexing unit 203 also includes a type discrimination unit 211, a control information acquisition unit 212, a slice information acquisition unit 213, and a decoded data generation unit 214.

The flow of operation of the receiving device 200 will be described below. Figure 16 is a flowchart showing an example of operation of the receiving device 200. Here, the operation for one access unit is shown. When decoding processing for multiple access units is performed, the processing of this flowchart is repeated.

First, the receiving device 200 receives, for example, multiple packets (MMT packets) generated by the transmitting device 100 (S201).

Next, the type determination unit 211 analyzes the header of the received packet to obtain the type of encoded data stored in the received packet (S202).

Next, the type discrimination unit 211 determines whether the data stored in the received packet is pre-slice segment data or slice segment data based on the type of the acquired encoded data (S203).

If the data stored in the received packet is pre-slice segment data (Yes in S203), the control information acquisition unit 212 acquires the pre-slice segment data of the access unit to be processed from the payload of the received packet, and stores the pre-slice segment data in memory (S204).

On the other hand, if the data stored in the received packet is data of a slice segment (No in S203), the receiving device 200 uses the header information of the received packet to determine which of the multiple areas the data stored in the received packet is encoded data for. Specifically, the slice information acquisition unit 213 analyzes the header of the received packet to acquire the index number Idx of the slice segment stored in the received packet (S205). Specifically, the index number Idx is the index number within the Movie Fragment of the access unit (sample in MMT).

Note that the processing of step S205 may be performed together with step S202.

Next, the decoding data generation unit 214 determines a decoding unit that will decode the slice segment (S206). Specifically, the index number Idx and multiple decoding units are associated in advance, and the decoding data generation unit 214 determines the decoding unit that will decode the slice segment, which corresponds to the index number Idx acquired in step S205.

As described in the example of FIG. 12, the decoded data generation unit 214 may determine the decoding unit to decode the slice segment based on at least one of the resolution of the access unit (picture), the method of dividing the access unit into multiple slice segments (tiles), and the processing capabilities of the multiple decoding units included in the receiving device 200. For example, the decoded data generation unit 214 determines the method of dividing the access unit based on identification information in a descriptor such as an MMT message or a TS section.

Next, the decoded data generation unit 214 generates multiple input data (combined data) to be input to the multiple decoding units by combining control information, which is included in one of the multiple packets and is used commonly for all decoding units in the picture, with each of the multiple encoded data of the multiple slice segments. Specifically, the decoded data generation unit 214 acquires slice segment data from the payload of the received packet. The decoded data generation unit 214 generates input data to the decoding unit determined in step S206 by combining the pre-slice segment data stored in memory in step S204 with the acquired slice segment data (S207).

After step S204 or S207, if the data of the received packet is not the final data of the access unit (No in S208), the processing from step S201 onwards is performed again. In other words, the above processing is repeated until input data for the multiple decoding units 204A to 204D corresponding to all slice segments included in the access unit is generated.

Note that the timing at which a packet is received is not limited to the timing shown in FIG. 16, and multiple packets may be received in advance or sequentially and stored in a memory, etc.

On the other hand, if the data of the received packet is the final data of the access unit (Yes in S208), the decoding command unit 206 outputs the multiple input data generated in step S207 to the corresponding decoding units 204A to 204D (S209).

Next, the multiple decoding units 204A to 204D generate multiple decoded images by decoding the multiple input data in parallel according to the DTS of the access unit (S210).

Finally, the display unit 205 generates a display image by combining the multiple decoded images generated by the multiple decoding units 204A to 204D, and displays the display image according to the PTS of the access unit (S211).

The receiving device 200 obtains the DTS and PTS of the access unit by analyzing the header information of the MPU or the payload data of the MMT packet that stores the header information of the Movie Fragment. When TS is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the PES packet. When RTP is used as the multiplexing method, the receiving device 200 obtains the DTS and PTS of the access unit from the header of the RTP packet.

When integrating the decoding results of the multiple decoding units, the display unit 205 may perform filtering such as deblocking filtering at the boundary between adjacent division units. When displaying the decoding results of a single decoding unit, filtering is not necessary, so the display unit 205 may switch the processing depending on whether filtering is performed at the boundary between the decoding results of the multiple decoding units. Whether filtering is necessary may be specified in advance depending on the presence or absence of division. Alternatively, information indicating whether filtering is necessary may be stored separately in the multiplexing layer. Information necessary for filtering, such as filter coefficients, may be stored in the SPS, PPS, SEI, or slice segment. The decoding units 204A to 204D or the demultiplexing unit 203 acquire this information by analyzing the SEI, and output the acquired information to the display unit 205. The display unit 205 performs filtering using this information. When this information is stored in the slice segment, it is preferable that the decoding units 204A to 204D acquire this information.

In the above explanation, an example was given in which the types of data stored in a fragment are two types: pre-slice segment data and slice segments. However, the types of data may be three or more. In this case, a case distinction is made in step S203 according to the type.

In addition, when the data size of a slice segment is large, the transmitting device 100 may fragment the slice segment and store it in the MMT packet. In other words, the transmitting device 100 may fragment the pre-slice-segment data and the slice segment. In this case, if the access unit and the data unit are set to be equal as in the packetization example shown in FIG. 11, the following problem occurs.

For example, if slice segment 1 is divided into three fragments, slice segment 1 is divided into three packets with Fragment counter values of 1 to 3 and transmitted. Furthermore, from slice segment 2 onwards, the Fragment counter value becomes 4 or more, and it becomes impossible to associate the Fragment counter value with the data stored in the payload. Therefore, the receiving device 200 cannot identify the packet that stores the first data of the slice segment from the information in the header of the MMT packet.

In such a case, the receiving device 200 may analyze the data in the payload of the MMT packet to identify the start position of the slice segment. Here, there are two types of formats for storing NAL units in the multiplexing layer in H.264 or H.265: a format called a byte stream format in which a start code consisting of a specific bit string is added immediately before the NAL unit header, and a format called a NAL size format in which a field indicating the size of the NAL unit is added.

The byte stream format is used in MPEG-2 systems and RTP, etc. The NAL size format is used in MP4, as well as DASH and MMT, which use MP4, etc.

When a byte stream format is used, the receiving device 200 analyzes whether the first data of the packet matches the start code. If the first data of the packet matches the start code, the receiving device 200 can detect whether the data contained in the packet is slice segment data by obtaining the type of NAL unit from the following NAL unit header.

On the other hand, in the case of the NAL size format, the receiving device 200 cannot detect the start position of the NAL unit based on the bit string. Therefore, in order to obtain the start position of the NAL unit, the receiving device 200 needs to shift the pointer by reading data by the size of the NAL unit, starting from the first NAL unit of the access unit.

However, if the size of the subsample unit is indicated in the header of an MPU or Movie Fragment in MMT, and the subsample corresponds to pre-slice data or a slice segment, the receiving device 200 can identify the start position of each NAL unit based on the size information of the subsample. Therefore, the transmitting device 100 may include information indicating whether or not information of the subsample unit is present in an MPU or Movie Fragment in information that the receiving device 200 acquires when it starts receiving data, such as an MPT in MMT.

The MPU data is an extension of the MP4 format. In MP4, there are modes in which parameter sets such as SPS and PPS of H.264 or H.265 can be stored as sample data, and modes in which they cannot be stored. Information for identifying this mode is indicated as the entry name of SampleEntry. When a mode in which it can be stored is used and the parameter set is included in the sample, the receiving device 200 acquires the parameter set by the method described above.

On the other hand, when a mode that cannot store the parameter set is used, the parameter set is stored as Decoder Specific Information in the SampleEntry, or is stored using a stream for the parameter set. Here, since the stream for the parameter set is not generally used, it is preferable that the transmitting device 100 stores the parameter set in the Decoder Specific Information. In this case, the receiving device 200 analyzes the SampleEntry transmitted as metadata of the MPU or metadata of the Movie Fragment in the MMT packet to obtain the parameter set referenced by the access unit.

When the parameter set is stored as sample data, the receiving device 200 can obtain the parameter set required for decoding by referring only to the sample data without referring to the SampleEntry. In this case, the transmitting device 100 does not need to store the parameter set in the SampleEntry. In this way, the transmitting device 100 can use the same SampleEntry in different MPUs, thereby reducing the processing load on the transmitting device 100 when generating an MPU. Another advantage is that the receiving device 200 does not need to refer to the parameter set in the SampleEntry.

Alternatively, the transmitting device 100 may store one default parameter set in SampleEntry, and store the parameter set referenced by the access unit in the sample data. In conventional MP4, it was common to store parameter sets in SampleEntry, so there is a possibility that a receiving device may stop playback if a parameter set does not exist in SampleEntry. This problem can be solved by using the above method.

Alternatively, the transmitting device 100 may store a parameter set in the sample data only when a parameter set different from the default parameter set is used.

In addition, since it is possible to store parameter sets in SampleEntry in both modes, the transmitting device 100 may always store parameter sets in VisualSampleEntry, and the receiving device 200 may always obtain parameter sets from VisualSampleEntry.

In the MMT standard, MP4 header information such as Moov and Moof is transmitted as MPU metadata or movie fragment metadata, but the transmitting device 100 does not necessarily have to transmit MPU metadata and movie fragment metadata. Furthermore, the receiving device 200 can also determine whether SPS and PPS are stored in the sample data based on the service, asset type, or whether MPU meta is transmitted or not in the ARIB (Association of Radio Industries and Businesses) standard.

Figure 17 shows an example in which pre-slice segment data and each slice segment are set to different Data units.

In the example shown in FIG. 17, the data sizes of the data before the slice segment and slice segments 1 to 4 are Length #1 to Length #5, respectively. The field values of the Fragmentation indicator, Fragment counter, and Offset included in the header of the MMT packet are as shown in the figure.

Here, Offset is offset information indicating the bit length (offset) from the beginning of the encoded data of the sample (access unit or picture) to which the payload data belongs to to the first byte of the payload data (encoded data) included in the MMT packet. Note that the value of the Fragment counter is described as starting from a value obtained by subtracting 1 from the total number of fragments, but it may start from another value.

Figure 18 is a diagram showing an example of when a Data unit is fragmented. In the example shown in Figure 18, slice segment 1 is divided into three fragments, which are stored in MMT packets #2 to #4, respectively. In this case, if the data size of each fragment is Length #2_1 to Length #2_3, respectively, the values of each field are as shown in the figure.

In this way, when a data unit such as a slice segment is set to Data unit, the start of the access unit and the start of the slice segment can be determined based on the field values of the MMT packet header as follows:

The start of the payload in a packet with an Offset value of 0 is the start of the access unit.

The start of the payload of a packet whose Offset value is different from 0 and whose Fragmentation indicator value is 00 or 01 is the start of a slice segment.

In addition, if fragmentation of the data unit does not occur and no packet loss occurs, the receiving device 200 can identify the index number of the slice segment to be stored in the MMT packet based on the number of slice segments obtained after detecting the beginning of the access unit.

Furthermore, even if the data unit of the data before the slice segment is fragmented, the receiving device 200 can similarly detect the beginning of the access unit and the slice segment.

In addition, even if packet loss occurs or the SPS, PPS, and SEI included in the slice segment pre-data are set to different data units, the receiving device 200 can identify the MMT packet that stores the first data of the slice segment based on the analysis result of the MMT header, and then analyze the slice segment header to identify the start position of the slice segment or tile within the picture (access unit). The amount of processing involved in analyzing the slice header is small, and the processing load is not a problem.

In this way, each of the multiple coded data of the multiple slice segments is in one-to-one correspondence with a basic data unit (Data unit), which is a unit of data stored in one or more packets. In addition, each of the multiple coded data is stored in one or more MMT packets.

The header information of each MMT packet includes a Fragmentation indicator (identification information) and an Offset (offset information).

The receiving device 200 determines that the start of payload data included in a packet having header information including a Fragmentation indicator whose value is 00 or 01 is the start of the encoded data of each slice segment. Specifically, the receiving device 200 determines that the start of payload data included in a packet having header information including an Offset whose value is not 0 and a Fragmentation indicator whose value is 00 or 01 is the start of the encoded data of each slice segment.

In the example of FIG. 17, the beginning of a Data unit is either the beginning of an access unit or the beginning of a slice segment, and the value of the Fragmentation indicator is 00 or 01. Furthermore, the receiving device 200 can detect the beginning of an access unit or the beginning of a slice segment without referring to the Offset by referring to the type of the NAL unit and determining whether the beginning of a Data Unit is an access unit delimiter or a slice segment.

In this way, by transmitting device 100 packetizing data so that the beginning of the NAL unit always starts at the beginning of the payload of the MMT packet, receiving device 200 can detect the beginning of an access unit or slice segment by analyzing the fragmentation indicator and the NAL unit header, even in cases where the data before the slice segment is divided into multiple data units. The type of the NAL unit is present in the first byte of the NAL unit header. Therefore, when analyzing the header portion of the MMT packet, receiving device 200 can obtain the type of the NAL unit by analyzing an additional byte of data.

In the case of audio, the receiving device 200 only needs to be able to detect the beginning of an access unit and make a determination based on whether the value of the Fragmentation indicator is 00 or 01.

As described above, when storing encoded data that has been encoded so that it can be divided and decoded in a PES packet of MPEG-2 TS, the transmitting device 100 can use a data alignment descriptor. Below, an example of a method for storing encoded data in a PES packet is described in detail.

For example, in HEVC, the transmitting device 100 can use the data alignment descriptor to indicate whether the data stored in the PES packet is an access unit, a slice segment, or a tile. The alignment types in HEVC are specified as follows:

Alignment type=8 indicates a HEVC slice segment. Alignment type=9 indicates a HEVC slice segment or access unit. Alignment type=12 indicates a HEVC slice segment or tile.

Therefore, by using type 9, for example, the transmitting device 100 can indicate that the data of the PES packet is either a slice segment or pre-slice segment data. Since a type indicating a slice rather than a slice segment is also separately defined, the transmitting device 100 may use a type indicating a slice rather than a slice segment.

The DTS and PTS included in the header of a PES packet are set only in the PES packet that contains the first data of an access unit. Therefore, if the type is 9 and the PES packet contains a DTS or PTS field, the receiving device 200 can determine that the PES packet stores the entire access unit or the first division unit of the access unit.

The transmitting device 100 may also enable the receiving device 200 to distinguish the data contained in the packet using a field such as transport_priority, which indicates the priority of a TS packet that stores a PES packet that includes the first data of an access unit. The receiving device 200 may also determine the data contained in the packet by analyzing whether the payload of the PES packet is an access unit delimiter. The data_alignment_indicator in the PES packet header indicates whether data is stored in the PES packet according to these types. If this flag (data_alignment_indicator) is set to 1, it is guaranteed that the data stored in the PES packet complies with the type indicated in the data alignment descriptor.

In addition, the transmitting device 100 may use the data alignment descriptor only when PES packetization is performed in units that can be divided and decoded, such as slice segments. This allows the receiving device 200 to determine that the coded data has been PES packetized in units that can be divided and decoded if the data alignment descriptor is present, and to determine that the coded data has been PES packetized in access unit units if the data alignment descriptor is not present. Note that if the data_alignment_indicator is set to 1 and the data alignment descriptor is not present, it is specified in the MPEG-2 TS standard that the unit of PES packetization is the access unit.

If the PMT contains a data alignment descriptor, the receiving device 200 can determine that the PES packetization is performed in units that can be divided and decoded, and can generate input data for each decoding unit based on the packetized units. If the PMT does not contain a data alignment descriptor and the receiving device 200 determines that parallel decoding of the encoded data is necessary based on the program information or information in other descriptors, the receiving device 200 generates input data for each decoding unit by analyzing the slice header of the slice segment. If the encoded data can be decoded by a single decoding unit, the receiving device 200 decodes the data of the entire access unit by the decoding unit. If information indicating whether the encoded data is composed of units that can be divided and decoded, such as slice segments or tiles, is separately indicated by a PMT descriptor, the receiving device 200 may determine whether the encoded data can be decoded in parallel based on the analysis result of the descriptor.

In addition, the DTS and PTS included in the header of a PES packet are set only in the PES packet that contains the first data of an access unit, so when an access unit is divided and packetized into PES packets, the second and subsequent PES packets do not contain information indicating the DTS and PTS of the access unit. Therefore, when performing decoding processes in parallel, each of the decoding units 204A-204D and the display unit 205 uses the DTS and PTS stored in the header of the PES packet that contains the first data of the access unit.

(Embodiment 2)
In the second embodiment, a method for storing data of NAL size format in an MPU based on MP4 format in MMT will be described. Note that, as an example, a method for storing data in an MPU used in MMT will be described below, but such a method for storing data can also be applied to DASH, which is based on the same MP4 format.

[Method of storing in MPU]
In the MP4 format, a plurality of access units are collected and stored in one MP4 file. In the MPU used in MMT, data for each media is stored in one MP4 file, and the data can include any number of access units. Since the MPU is a unit that can be decoded by itself, for example, the MPU stores access units in units of GOPs.

Figure 19 shows the structure of an MPU. At the beginning of an MPU are ftyp, mmpu, and moov, which are collectively defined as MPU metadata. In moov, initialization information common to the files and an MMT hint track are stored.

In addition, moof stores initialization information and size for each sample and subsample, information that can identify the presentation time (PTS) and decoding time (DTS) (sample_duration, sample_size, sample_composition_time_offset), as well as data_offset, which indicates the position of the data.

In addition, each of the multiple access units is stored as a sample in mdat (mdat box). The data in moof and mdat excluding the samples is defined as movie fragment metadata (hereafter referred to as MF metadata), and the sample data in mdat is defined as media data.

Figure 20 is a diagram showing the structure of MF metadata. As shown in Figure 20, in more detail, MF metadata consists of type, length, and data of moof box (moof), and type and length of mdat box (mdat).

When storing access units in MP4 data, there are modes in which parameter sets such as H.264 and H.265 SPS and PPS can be stored as sample data, and modes in which they cannot be stored.

Here, in the above-mentioned non-storable mode, the parameter set is stored in the Decoder Specific Information of the SampleEntry in the moov. Also, in the above-mentioned storable mode, the parameter set is included in the sample.

The MPU metadata, MF metadata, and media data are each stored in the MMT payload, and the fragment type (FT) is stored in the header of the MMT payload as an identifier to identify these data. FT=0 indicates MPU metadata, FT=1 indicates MF metadata, and FT=2 indicates media data.

Note that FIG. 19 illustrates an example in which MPU metadata units and MF metadata units are stored as data units in the MMT payload, but units such as ftyp, mmpu, moov, and moof may be stored as data units in the MMT payload in data unit units. Similarly, FIG. 19 illustrates an example in which sample units are stored as data units in the MMT payload. However, data units may be configured in sample units or NAL unit units, and such data units may be stored in the MMT payload in data unit units. Such data units may be further fragmented and stored in the MMT payload.

[Conventional transmission methods and issues]
Conventionally, when multiple access units are encapsulated in the MP4 format, moov and moof are created when all samples to be stored in the MP4 are available.

When transmitting MP4 format in real time using broadcasting, for example, if the samples stored in one MP4 file are in GOP units, then moov and moof are created after the GOP unit time samples are accumulated, resulting in delays due to encapsulation. This type of encapsulation on the transmitting side always results in an end-to-end delay that is longer by the GOP unit time. This makes it difficult to provide services in real time, and leads to degradation of service for viewers, especially when live content is transmitted.

Figure 21 is a diagram for explaining the data transmission order. When MMT is applied to broadcasting, as shown in Figure 21 (a), if data is loaded onto MMT packets and transmitted in the order of the MPU configuration (MMT packets #1, #2, #3, #4, #5, #6 are transmitted in that order), a delay occurs in the transmission of the MMT packets due to encapsulation.

To prevent this delay due to encapsulation, a method has been proposed in which MPU header information such as MPU metadata and MF metadata is not sent (packets #1 and #2 are not sent, and packets #3-#6 are sent in that order), as shown in (b) of Figure 21. In addition, a method has been proposed in which media data is sent first without waiting for the creation of MPU header information, and the MPU header information is sent after the media data has been sent (sending packets #3-#6, #1, #2 in that order), as shown in (c) of Figure 20.

If the MPU header information is not transmitted, the receiving device decodes without using the MPU header information. Also, if the MPU header information is sent after the media data, the receiving device waits until it obtains the MPU header information before decoding.

However, in conventional MP4-compliant receiving devices, decoding without using MPU header information is not guaranteed. Also, if a receiving device performs decoding without using an MPU header through special processing, using a conventional transmission method would complicate the decoding process, making it highly likely that real-time decoding would be difficult. Also, if a receiving device waits to obtain MPU header information before decoding, it is necessary to buffer the media data until the receiving device obtains the header information, but a buffer model has not been specified and decoding has not been guaranteed.

Therefore, the transmitting device according to the second embodiment transmits the MPU metadata before the media data by storing only common information in the MPU metadata as shown in (d) of FIG. 20. Then, the transmitting device according to the second embodiment transmits the MF metadata, the generation of which is delayed, after the media data. This provides a transmitting method or receiving method that can guarantee the decoding of the media data.

Below, we will explain the reception method when using each of the transmission methods (a)-(d) in Figure 21.

In each transmission method shown in Figure 21, MPU data is first constructed in the following order: MPU metadata, MFU metadata, and media data.

After constructing the MPU data, if the transmitting device transmits data in the order of MPU metadata, MF metadata, and media data, as shown in (a) of Figure 21, the receiving device can perform decoding using either of the following methods (A-1) and (A-2).

(A-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data using the MPU header information.

(A-2) The receiving device decodes the media data without using the MPU header information.

Although both of these methods incur delays due to encapsulation on the sending side, they have the advantage that the receiving device does not need to buffer the media data to obtain the MPU header. If buffering is not performed, there is no need to install memory for buffering, and furthermore, no buffering delays occur. Furthermore, method (A-1) is applicable to conventional receiving devices as it uses MPU header information for decoding.

When the transmitting device transmits only media data, as shown in (b) of FIG. 21, the receiving device can perform decoding using the method (B-1) below.

(B-1) The receiving device decodes the media data without using the MPU header information.

Although not shown, if MPU metadata is transmitted prior to the transmission of the media data in FIG. 21(b), decoding can be performed using the method (B-2) below.

(B-2) The receiving device decodes the media data using the MPU metadata.

The advantages of both methods (B-1) and (B-2) above are that no delays due to encapsulation occur on the sending side, and there is no need to buffer media data to obtain the MPU header. However, both methods (B-1) and (B-2) do not use MPU header information for decoding, so special processing may be required for decoding.

When the transmitting device transmits data in the order of media data, MPU metadata, and MF metadata, as shown in (c) of FIG. 21, the receiving device can perform decoding using either of the following methods (C-1) and (C-2).

(C-1) After acquiring the MPU header information (MPU metadata and MF metadata), the receiving device decodes the media data.

(C-2) The receiving device decodes the media data without using the MPU header information.

When the above method (C-1) is used, it is necessary to buffer the media data in order to obtain the MPU header information. In contrast, when the above method (C-2) is used, there is no need to buffer the media data in order to obtain the MPU header information.

In addition, neither method (C-1) nor (C-2) causes delays due to encapsulation on the sending side. Also, method (C-2) does not use MPU header information, so special processing may be required.

When the transmitting device transmits data in the order of MPU metadata, media data, and MF metadata, as shown in (d) of FIG. 21, the receiving device can perform decoding using either of the following methods (D-1) and (D-2).

(D-1) After acquiring the MPU metadata, the receiving device further acquires the MF metadata and then decodes the media data.

(D-2) After acquiring the MPU metadata, the receiving device decodes the media data without using the MF metadata.

When the above method (D-1) is used, it is necessary to buffer the media data in order to obtain the MF metadata, but when the above method (D-2) is used, there is no need to buffer the media data in order to obtain the MF metadata.

The above method (D-2) does not use MF metadata for decryption, so special processing may be required.

As explained above, if decoding is possible using MPU metadata and MF metadata, there is an advantage that decoding can also be performed on conventional MP4 receiving devices.

In FIG. 21, the MPU data is structured in the order of MPU metadata, MFU metadata, and media data, and in moof, the position information (offset) for each sample and subsample is determined based on this structure. In addition, the MF metadata also includes data other than the media data in the mdat box (box size and type).

Therefore, when a receiving device identifies media data based on MF metadata, the receiving device reconstructs the data in the order in which the MPU data was constructed, regardless of the order in which the data was transmitted, and then decodes the data using the moov in the MPU metadata or the moof in the MF metadata.

In FIG. 21, the MPU data is composed of MPU metadata, MFU metadata, and media data in that order, but the MPU data may be composed in an order different from that shown in FIG. 21, and the position information (offset) may be determined.

For example, MPU data may be configured in the order of MPU metadata, media data, and MF metadata, and negative position information (offset) may be indicated in the MF metadata. In this case, regardless of the order in which the data is transmitted, the receiving device reconstructs the data in the order in which the MPU data was configured on the transmitting side, and then performs decoding using moov or moof.

In addition, the transmitting device may signal information indicating the order in which the MPU data is constructed, and the receiving device may reconstruct the data based on the signaled information.

As described above, the receiving device receives packetized MPU metadata, packetized media data (sample data), and packetized MF metadata in this order, as shown in (d) of FIG. 21. Here, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata.

Next, the receiving device reconstructs MPU data (MP4 format file) including the received MPU metadata, the received MF metadata, and the received sample data. Then, the receiving device decodes the sample data included in the reconstructed MPU data using the MPU metadata and MF metadata. The MF metadata is metadata including data that can be generated only after the sample data is generated on the transmitting side (e.g., length stored in mbox).

In more detail, the operation of the receiving device is performed by each of the components constituting the receiving device. For example, the receiving device includes a receiving unit that receives the data, a reconstruction unit that reconstructs the MPU data, and a decoding unit that decodes the MPU data. Each of the receiving unit, generating unit, and decoding unit is realized by a microcomputer, a processor, a dedicated circuit, etc.

[Method of decrypting without using header information]
Next, a method of decoding without using header information will be described. Here, a method of decoding without using header information in a receiving device will be described, regardless of whether or not the transmitting side sends header information. That is, this method is applicable to any of the transmission methods described with reference to FIG. 21. However, some of the decoding methods are applicable only to specific transmission methods.

Figure 22 is a diagram showing an example of a method for decoding without using header information. In Figure 22, only MMT payloads and MMT packets containing only media data are shown, and MMT payloads and MMT packets containing MPU metadata and MF metadata are not shown. In the following explanation of Figure 22, it is assumed that media data belonging to the same MPU are transmitted continuously. In addition, an example will be described in which samples are stored in the payload as media data, but in the following explanation of Figure 22, it is natural that NAL units or fragmented NAL units may be stored.

To decode media data, the receiving device must first obtain the initialization information required for decoding. Furthermore, if the media is video, the receiving device must obtain initialization information for each sample, identify the start position of the MPU, which is a random access unit, and obtain the start positions of the samples and NAL units. Furthermore, the receiving device must identify the decode time (DTS) and presentation time (PTS) of each sample.

Therefore, the receiving device can perform decoding without using header information, for example, by using the following method. Note that if the payload stores NAL unit units or fragmented NAL units, "sample" in the following explanation can be read as "NAL unit in sample."

<Random access (=identifying the first sample of an MPU)>
When header information is not transmitted, the receiving device can identify the first sample of an MPU using the following methods 1 and 2. Note that when header information is transmitted, method 3 can be used.

[Method 1] The receiving device acquires samples contained in MMT packets with 'RAP_flag = 1' in the MMT packet header.

[Method 2] The receiving device obtains samples with 'sample number = 0' in the MMT payload header.

[Method 3] When at least one of MPU metadata and MF metadata is transmitted before and/or after the media data, the receiving device acquires samples contained in the MMT payload in which the fragment type (FT) in the MMT payload header has been switched to media data.

In Methods 1 and 2, if a single payload contains multiple samples belonging to different MPUs, it is impossible to determine which NAL unit is a random access point (RAP_flag = 1 or sample number = 0). For this reason, a constraint is required such as not mixing samples from different MPUs in a single payload, or, if a single payload contains samples from different MPUs, setting RAP_flag to 1 if the last (or first) sample is a random access point.

In addition, in order for the receiving device to obtain the starting position of the NAL unit, it is necessary to shift the data read pointer by the size of the NAL unit, starting from the first NAL unit of the sample.

If the data is fragmented, the receiving device can identify the data unit by referring to the fragment_indicator and fragment_number.

Determining the DTS of a sample
There are two methods for determining the DTS of a sample: Method 1 and Method 2.

[Method 1] The receiving device determines the DTS of the first sample based on the prediction structure. However, this method requires analysis of the encoded data and may be difficult to decode in real time, so the following method 2 is preferable.

[Method 2] The receiving device transmits the DTS of the first sample separately and acquires the transmitted DTS of the first sample. Methods for transmitting the DTS of the first sample include, for example, transmitting the DTS of the first sample of an MPU using MMT-SI, or transmitting the DTS for each sample using the MMT packet header extension field. Note that the DTS may be an absolute value or a relative value to the PTS. In addition, the transmitting side may signal whether the DTS of the first sample is included.

Note that in both methods 1 and 2, the DTS of subsequent samples is calculated assuming a fixed frame rate.

As a method for storing the DTS for each sample in the packet header, other than using the extension area, there is a method of storing the DTS of the sample contained in the MMT packet in the 32-bit NTP timestamp field in the MMT packet header. If the DTS cannot be expressed by the number of bits in one packet header (32 bits), the DTS may be expressed using multiple packet headers. The DTS may also be expressed by combining the NTP timestamp field and extension area of the packet header. If no DTS information is included, a known value (e.g. ALL 0) is used.

Determining the PTS of a sample
The receiving device obtains the PTS of the first sample from the MPU timestamp descriptor for each asset included in the MPU. The receiving device calculates the PTS of subsequent samples from parameters indicating the display order of samples such as POC, assuming a fixed frame rate. In this way, transmission at a fixed frame rate is essential to calculate the DTS and PTS without using header information.

In addition, when MF metadata is being transmitted, the receiving device can calculate the absolute values of the DTS and PTS from the relative time information of the DTS and PTS from the first sample indicated in the MF metadata and the absolute value of the timestamp of the first sample of the MPU indicated in the MPU timestamp descriptor.

When analyzing the encoded data to calculate the DTS and PTS, the receiving device may use the SEI information included in the access unit.

<Initialization information (parameter set)>
[For video]
In the case of video, the parameter set is stored in the sample data. Also, if the MPU metadata and the MF metadata are not transmitted, it is guaranteed that the parameter set required for decoding can be obtained by referring to only the sample data.

Also, as shown in (a) and (d) of FIG. 21, when MPU metadata is transmitted before media data, it may be specified that no parameter set is stored in SampleEntry. In this case, the receiving device refers only to the parameter set in the sample, without referring to the parameter set in SampleEntry.

In addition, when MPU metadata is transmitted prior to media data, a parameter set common to MPUs or a default parameter set is stored in SampleEntry, and the receiving device may refer to the parameter set in SampleEntry and the parameter set in the sample. Storing a parameter set in SampleEntry makes it possible to perform decoding even on conventional receiving devices that cannot play back data unless a parameter set is present in SampleEntry.

[For audio]
In the case of audio, the LATM header is necessary for decoding, and in MP4, it is essential that the LATM header is included in the sample entry. However, if the header information is not transmitted, it is difficult for the receiving device to obtain the LATM header, so the LATM header is separately included in the control information such as SI. The LATM header may be included in a message, a table, or a descriptor. The LATM header may also be included in a sample.

Before starting decoding, the receiving device obtains the LATM header from the SI or the like, and starts decoding the audio. Alternatively, as shown in (a) and (d) of Figure 21, if the MPU metadata is transmitted before the media data, the receiving device can receive the LATM header before the media data. Therefore, if the MPU metadata is transmitted before the media data, decoding can be performed even using a conventional receiving device.

＜Other＞
The transmission order and the type of the transmission order may be notified as control information such as an MMT packet header, a payload header, or an MPT or other table, a message, a descriptor, etc. The type of the transmission order here is, for example, the four types of transmission order shown in (a) to (d) of FIG. 21, and an identifier for identifying each type may be stored in a location where it can be obtained before the start of decoding.

In addition, different transmission order types may be used for audio and video, or a common type may be used for audio and video. Specifically, for example, audio may be transmitted in the order of MPU metadata, MF metadata, and media data, as shown in (a) of FIG. 21, and video may be transmitted in the order of MPU metadata, media data, and MF metadata, as shown in (d) of FIG. 21.

By using the method described above, the receiving device can perform decoding without using header information. In addition, if the MPU metadata is transmitted before the media data (Figures 21(a) and 21(d)), decoding can be performed even with a conventional receiving device.

In particular, by transmitting the MF metadata after the media data ((d) in Figure 21), delays due to encapsulation are not incurred and decoding can be performed even by conventional receiving devices.

[Configuration and operation of transmitting device]
Next, the configuration and operation of a transmission device will be described. Fig. 23 is a block diagram of a transmission device according to the second embodiment, and Fig. 24 is a flowchart of a transmission method according to the second embodiment.

As shown in FIG. 23, the transmitting device 15 includes an encoding unit 16, a multiplexing unit 17, and a transmitting unit 18.

The encoding unit 16 generates encoded data by encoding the video or audio to be encoded, for example, according to H.265 (S10).

The multiplexing unit 17 multiplexes (packetizes) the encoded data generated by the encoding unit 16 (S11). Specifically, the multiplexing unit 17 packetizes each of the sample data, MPU metadata, and MF metadata that constitute an MP4 format file. The sample data is data in which a video signal or an audio signal is encoded, the MPU metadata is an example of the first metadata, and the MF metadata is an example of the second metadata. Both the first metadata and the second metadata are metadata used to decode the sample data, but the difference between them is that the second metadata includes data that can be generated only after the sample data is generated.

Here, data that can be generated only after sample data is generated is, for example, data other than sample data stored in mdat in MP4 format (data in the header of mdat, i.e., type and length shown in FIG. 20). Here, the second metadata only needs to include at least a part of this data, namely, length.

The transmitting unit 18 transmits the packetized MP4 format file (S12). The transmitting unit 18 transmits the MP4 format file, for example, by the method shown in FIG. 21(d). That is, the transmitting unit 18 transmits the packetized MPU metadata, the packetized sample data, and the packetized MF metadata in this order.

Note that each of the encoding unit 16, the multiplexing unit 17, and the transmission unit 18 is realized by a microcomputer, a processor, or a dedicated circuit, etc.

[Configuration of receiving device]
Next, the configuration and operation of a receiving device according to the second embodiment will be described.

As shown in FIG. 25, the receiving device 20 includes a packet filtering unit 21, a transmission order type discrimination unit 22, a random access unit 23, a control information acquisition unit 24, a data acquisition unit 25, a PTS and DTS calculation unit 26, an initialization information acquisition unit 27, a decoding command unit 28, a decoding unit 29, and a presentation unit 30.

[Operation 1 of the receiving device]
First, an operation of the receiving device 20 for identifying the MPU start position and the NAL unit position when the media is video will be described. Fig. 26 is a flowchart of such an operation of the receiving device 20. Note that, here, it is assumed that the transmission order type of the MPU data is stored in the SI information by the transmitting device 15 (multiplexing unit 17).

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type discrimination unit 22 analyzes the SI information obtained by packet filtering to obtain the transmission order type of the MPU data (S21).

Next, the transmission order type discrimination unit 22 determines (discriminates) whether or not the data after packet filtering contains MPU header information (at least one of MPU metadata and MF metadata) (S22). If the MPU header information is included (Yes in S22), the random access unit 23 identifies the first sample of the MPU by detecting that the fragment type of the MMT payload header has switched to media data (S23).

On the other hand, if the MPU header information is not included (No in S22), the random access unit 23 identifies the first sample of the MPU based on the RAP_flag in the MMT packet header or the sample number in the MMT payload header (S24).

The transmission order type discrimination unit 22 also determines whether the packet-filtered data contains MF metadata (S25). If it is determined that MF metadata is included (Yes in S25), the data acquisition unit 25 acquires the NAL units by reading the NAL units based on the sample, subsample offset, and size information included in the MF metadata (S26). On the other hand, if it is determined that MF metadata is not included (No in S25), the data acquisition unit 25 acquires the NAL units by reading data of the size of the NAL units in order from the first NAL unit of the sample (S27).

Note that even if it is determined in step S22 that MPU header information is included, the receiving device 20 may identify the first sample of the MPU using the process of step S24 instead of step S23. Also, when it is determined that MPU header information is included, the process of step S23 and the process of step S24 may be used in combination.

In addition, even if it is determined in step S25 that MF metadata is included, the receiving device 20 may obtain the NAL unit using the process of step S27 without using the process of step S26. In addition, when it is determined that MF metadata is included, the process of step S23 and the process of step S24 may be used in combination.

It is also possible that, when it is determined in step S25 that MF metadata is included, the MF data is transmitted after the media data. In this case, the receiving device 20 may buffer the media data and wait until the MF metadata is acquired before performing the process of step S26, or the receiving device 20 may determine whether or not to perform the process of step S27 without waiting for the MF metadata to be acquired.

For example, the receiving device 20 may determine whether to wait for acquisition of MF metadata based on whether it has a buffer with a buffer size capable of buffering media data. The receiving device 20 may also determine whether to wait for acquisition of MF metadata based on whether the end-to-end delay is small. The receiving device 20 may also perform the decoding process mainly using the process of step S26, and use the process of step S27 in the case of a processing mode when packet loss or the like occurs.

Note that if the transmission order type is predetermined, steps S22 and S26 may be omitted, and in this case, the receiving device 20 may determine the method for identifying the first sample of the MPU and the method for identifying the NAL unit, taking into account the buffer size and the end-to-end delay.

If the transmission order type is known in advance, the transmission order type determination unit 22 is not required in the receiving device 20.

Although not illustrated in FIG. 26 above, the decode command unit 28 outputs the data acquired by the data acquisition unit to the decode unit 29 based on the PTS and DTS calculated by the PTS, DTS calculation unit 26 and the initialization information acquired by the initialization information acquisition unit 27. The decode unit 29 decodes the data, and the presentation unit 30 presents the decoded data.

[Operation 2 of the receiving device]
Next, an operation of the receiving device 20 to obtain the initialization information based on the transmission order type and to decode the media data based on the initialization information will be described. FIG. 27 is a flowchart of such an operation.

First, the packet filtering unit 21 performs packet filtering on the received file. The transmission order type discrimination unit 22 analyzes the SI information obtained by packet filtering and obtains the transmission order type (S301).

Next, the transmission order type discrimination unit 22 determines whether or not MPU metadata has been transmitted (S302). If it is determined that MPU metadata has been transmitted (Yes in S302), the transmission order type discrimination unit 22 determines, based on the analysis result of step S301, whether or not MPU metadata has been transmitted before media data (S303). If MPU metadata has been transmitted before media data (Yes in S303), the initialization information acquisition unit 27 decodes the media data based on the common initialization information included in the MPU metadata and the initialization information of the sample data (S304).

On the other hand, if it is determined that the MPU metadata was transmitted after the media data (No in S303), the data acquisition unit 25 buffers the media data until the MPU metadata is acquired (S305), and performs the processing of step S304 after the MPU metadata is acquired.

Also, if it is determined in step S302 that the MPU metadata has not been transmitted (No in S302), the initialization information acquisition unit 27 decodes the media data based only on the initialization information of the sample data (S306).

Note that if the decryption of the media data is guaranteed only based on the initialization information of the sample data on the sending side, the processing based on the determinations of steps S302 and S303 is not performed, and the processing of step S306 is used.

The receiving device 20 may also determine whether or not to buffer the media data before step S305. In this case, if the receiving device 20 determines that the media data is to be buffered, it proceeds to the process of step S305, and if it determines that the media data is not to be buffered, it proceeds to the process of step S306. The determination of whether or not to buffer the media data may be made based on the buffer size and occupancy of the receiving device 20, or may be made taking into account the end-to-end delay, for example by selecting the buffer with the smaller end-to-end delay.

[Operation 3 of the receiving device]
Here, the details of the transmission method and the reception method in the case where the MF metadata is transmitted after the media data ((c) in FIG. 21 and (d) in FIG. 21) are described. The following describes the case of (d) in FIG. 21 as an example. Note that in the transmission, only the method of (d) in FIG. 21 is used, and no signaling of the transmission order type is performed.

As described above, when data is transmitted in the order of MPU metadata, media data, and MF metadata as shown in (d) of FIG.
(D-1) The receiving device 20 acquires the MPU metadata, and then acquires the MF metadata, and then decodes the media data.

(D-2) After acquiring the MPU metadata, the receiving device 20 decodes the media data without using the MF metadata.
There are two possible decoding methods:

Here, D-1 requires buffering of media data to obtain MF metadata, but since decoding can be performed using MPU header information, it can be decoded by a conventional MP4-compliant receiving device. Meanwhile, D-2 does not require buffering of media data to obtain MF metadata, but since decoding cannot be performed using MF metadata, special processing is required for decoding.

In addition, the method shown in FIG. 21(d) has the advantage that the MF metadata is sent after the media data, so there is no delay due to encapsulation and end-to-end delay can be reduced.

The receiving device 20 can select one of the two decoding methods described above depending on the capabilities of the receiving device 20 and the quality of service that the receiving device 20 provides.

The transmitting device 15 must ensure that the decoding operation in the receiving device 20 can be performed with reduced occurrence of buffer overflow and underflow. For example, the following parameters can be used as elements for defining the decoder model when decoding using the D-1 method.

Buffer size for reconfiguring the MPU (MPU buffer)
For example, buffer size=maximum rate×maximum MPU time×α, where the maximum rate is the upper limit rate of the profile and level of the encoded data+the overhead of the MPU header. The maximum MPU time is the maximum time length of a GOP when 1 MPU=1 GOP (video).

Here, audio may be in GOP units common to the video, or in a different unit. α is a margin to prevent overflow, and may be multiplied or added to the maximum rate x maximum MPU time. If multiplied, α ≧ 1, and if added, α ≧ 0.

Upper limit of the decoding delay time from when data is input to the MPU buffer until it is decoded (TSTD_delay in the STD of MPEG-TS)
For example, at the time of transmission, DTS is set so that the time when MPU data acquisition is completed at the receiver is less than or equal to DTS, taking into account the maximum MPU time and the upper limit of the decoding delay time.

The transmitting device 15 may also assign a DTS and a PTS according to the decoder model when decoding using the D-1 method. This allows the transmitting device 15 to guarantee decoding by a receiving device that performs decoding using the D-1 method, while also transmitting auxiliary information required when decoding is performed using the D-2 method.

For example, the transmitting device 15 can ensure the operation of a receiving device that decodes using the D-2 method by signaling the pre-buffering time in the decoder buffer when decoding using the D-2 method.

The pre-buffering time may be included in SI control information such as messages, tables, and descriptors, or may be included in the header of an MMT packet or MMT payload. The SEI in the encoded data may also be overwritten. The DTS and PTS for decoding using the D-1 method may be stored in the MPU timestamp descriptor and SampleEntry, and the DTS and PTS for decoding using the D-2 method, or the pre-buffering time, may be described in the SEI.

If the receiving device 20 only supports MP4-compliant decoding operations using the MPU header, it may select decoding method D-1, and if it supports both D-1 and D-2, it may select either one of them.

The transmitting device 15 assigns a DTS and a PTS to ensure the decoding operation of one of the streams (D-1 in this example), and may also transmit auxiliary information to assist the decoding operation of the other stream.

In addition, when the D-2 method is used, the end-to-end delay is more likely to be large due to delays caused by pre-buffering of MF metadata compared to when the D-1 method is used. Therefore, when it is desired to reduce the end-to-end delay, the receiving device 20 may select the D-2 method for decoding. For example, when it is desired to always reduce the end-to-end delay, the receiving device 20 may always use the D-2 method. In addition, the receiving device 20 may use the D-2 method only when operating in a low-delay presentation mode in which it is desired to present live content, channel selection, zapping operations, etc. with low delay.

Figure 28 is a flowchart of such a receiving method.

First, the receiving device 20 receives the MMT packet and acquires the MPU data (S401). Then, the receiving device 20 (transmission order type discrimination unit 22) determines whether to present the program in low-latency presentation mode (S402).

If the program is not presented in low-latency presentation mode (No in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using the header information (S405). In addition, the receiving device 20 (PTS, DTS calculation unit 26, decoding command unit 28, decoding unit 29, presentation unit 30) performs decoding and presentation processing based on the PTS and DTS assigned by the transmitting side (S406).

On the other hand, when the program is presented in low-latency presentation mode (Yes in S402), the receiving device 20 (random access unit 23 and initialization information acquisition unit 27) acquires random access and initialization information using a decoding method that does not use header information (S403). The receiving device 20 also performs decoding and presentation processing based on auxiliary information for decoding without using the PTS, DTS, and header information added by the transmitting side (S404). Note that in steps S403 and S404, processing may be performed using MPU metadata.

[Transmission and reception method using auxiliary data]
The above describes the transmission and reception operations in the cases where MF metadata is transmitted after the media data (cases (c) and (d) of Figure 21). Next, a method is described in which the transmitting device 15 transmits auxiliary data having some of the functions of the MF metadata, thereby allowing decoding to begin earlier and reducing end-to-end delay. Here, an example is described in which auxiliary data is further transmitted based on the transmission method shown in (d) of Figure 21, but the method using auxiliary data is also applicable to the transmission methods shown in (a) to (c) of Figure 21.

Figure 29 (a) shows an MMT packet transmitted using the method shown in Figure 21 (d). That is, data is transmitted in the following order: MPU metadata, media data, and MF metadata.

Here, sample #1, sample #2, sample #3, and sample #4 are samples included in the media data. Note that, although an example in which the media data is stored in the MMT packet in sample units is described here, the media data may be stored in the MMT packet in NAL unit units, or in units obtained by dividing the NAL units. Note that multiple NAL units may be aggregated and stored in the MMT packet.

As explained above in D-1, in the case of the method shown in (d) of FIG. 21, that is, when data is transmitted in the order of MPU metadata, media data, and MF metadata, there is a method in which after acquiring the MPU metadata, the MF metadata is acquired, and then the media data is decoded. This type of D-1 method requires buffering of the media data to acquire the MF metadata, but has the advantage that the D-1 method can be applied to conventional MP4-compliant receiving devices because the decoding is performed using the MPU header information. On the other hand, it has the disadvantage that the receiving device 20 must wait to start decoding until the MF metadata is acquired.

In contrast, in a method using auxiliary data, as shown in (b) of Figure 29, the auxiliary data is transmitted before the MF metadata.

MF metadata contains information indicating the DTS, PTS, offsets and sizes of all samples contained in a movie fragment. In contrast, auxiliary data contains information indicating the DTS, PTS, offsets and sizes of some of the samples contained in a movie fragment.

For example, MF metadata contains information for all samples (sample #1-sample #4), whereas auxiliary data contains information for only some samples (sample #1-sample #2).

In the case shown in FIG. 29(b), the auxiliary data is used to enable decoding of sample #1 and sample #2, so the end-to-end delay is smaller than in the D-1 transmission method. Note that the auxiliary data may include any combination of sample information, and the auxiliary data may be transmitted repeatedly.

For example, in (c) of FIG. 29, when transmitting auxiliary information at timing A, transmitting device 15 includes information on sample #1 in the auxiliary information, and when transmitting auxiliary information at timing B, transmitting device 15 includes information on sample #1 and sample #2 in the auxiliary information. When transmitting auxiliary information at timing C, transmitting device 15 includes information on sample #1, sample #2, and sample #3 in the auxiliary information.

The MF metadata includes information on sample #1, sample #2, sample #3, and sample #4 (information on all samples in the movie fragment).

Ancillary data does not necessarily need to be sent immediately after it is generated.

In addition, the header of an MMT packet or MMT payload specifies a type that indicates that auxiliary data is stored.

For example, when auxiliary data is stored in the MMT payload using MPU mode, the fragment_type field value (e.g., FT=3) is specified as a data type indicating that it is auxiliary data. The auxiliary data may be data based on the moof configuration, or may have another configuration.

When auxiliary data is stored in the MMT payload as a control signal (descriptor, table, message), a descriptor tag, table ID, message ID, etc. are specified to indicate that it is auxiliary data.

The PTS or DTS may also be stored in the header of an MMT packet or MMT payload.

[Example of generating auxiliary data]
An example in which a transmitting device generates auxiliary data based on the moof configuration will be described below. Fig. 30 is a diagram for explaining an example in which a transmitting device generates auxiliary data based on the moof configuration.

In a normal MP4, a moof is created for a movie fragment, as shown in Figure 20. The moof contains information indicating the DTS, PTS, offset, and size of the samples contained in the movie fragment.

Here, the transmitting device 15 constructs an MP4 (MP4 file) using only a portion of the sample data that constitutes the MPU, and generates auxiliary data.

For example, as shown in FIG. 30(a), the transmitting device 15 generates an MP4 using only sample #1 of samples #1-#4 that make up the MPU, and the moof+mdat header is treated as auxiliary data.

Next, as shown in (b) of FIG. 30, the transmitting device 15 generates an MP4 using samples #1 and #2 from among samples #1-#4 that make up the MPU, and the header of moof+mdat is treated as the next auxiliary data.

Next, as shown in (c) of FIG. 30, the transmitting device 15 generates an MP4 using samples #1, #2, and #3 of samples #1-#4 that make up the MPU, and the header of moof+mdat is treated as the next auxiliary data.

Next, as shown in (d) of FIG. 30, the transmitting device 15 generates all MP4s from samples #1-#4 that make up the MPU, and the moof+mdat header becomes the movie fragment metadata.

Note that, although the transmitting device 15 generates auxiliary data for each sample here, it may generate auxiliary data for every N samples. The value of N is any number, and for example, if auxiliary data is transmitted M times when transmitting one MPU, N may be set to total samples/M.

In addition, the information indicating the offset of a sample in moof may be the offset value after the sample entry area of the subsequent sample number is secured as a NULL area.

In addition, auxiliary data may be generated so as to fragment the MF metadata.

[Example of reception operation using auxiliary data]
The reception of auxiliary data generated as described in Fig. 30 will be described. Fig. 31 is a diagram for explaining the reception of auxiliary data. In Fig. 31(a), the number of samples constituting an MPU is 30, and auxiliary data is generated and transmitted every 10 samples.

In FIG. 30(a), auxiliary data #1 contains sample information for samples #1-#10, auxiliary data #2 contains sample information for samples #1-#20, and MF metadata contains sample information for samples #1-#30.

Note that samples #1-#10, samples #11-#20, and samples #21-#30 are stored in one MMT payload, but may be stored in sample units or NAL units, or may be stored in fragment or aggregate units.

The receiving device 20 receives packets of MPU meta, samples, MF meta, and auxiliary data.

The receiving device 20 concatenates the sample data in the order in which it is received (to the end), and after receiving the latest auxiliary data, updates the previous auxiliary data. In addition, the receiving device 20 can finally construct a complete MPU by replacing the auxiliary data with MF metadata.

When receiving auxiliary data #1, receiving device 20 concatenates the data as shown in the upper part of (b) of Figure 31 to construct an MP4. This allows receiving device 20 to parse samples #1-#10 using the MPU metadata and information in auxiliary data #1, and to perform decoding based on the PTS, DTS, offset, and size information contained in the auxiliary data.

Furthermore, when receiving auxiliary data #2, receiving device 20 concatenates the data as shown in the middle part of (b) in FIG. 31 to construct an MP4. This enables receiving device 20 to parse samples #1-#20 using the MPU metadata and information in auxiliary data #2, and to perform decoding based on the PTS, DTS, offset, and size information contained in the auxiliary data.

Furthermore, when the receiving device 20 receives the MF metadata, it concatenates the data as shown in the lower part of (b) of FIG. 31 to construct an MP4. This enables the receiving device 20 to parse samples #1-#30 using the MPU metadata and the MF metadata, and to perform decoding based on the PTS, DTS, offset, and size information contained in the MF metadata.

If there is no auxiliary data, the receiving device 20 can obtain sample information only after receiving the MF metadata, and therefore it is necessary to start decoding after receiving the MF metadata. However, by having the transmitting device 15 generate and transmit auxiliary data, the receiving device 20 can obtain sample information using the auxiliary data without waiting for the reception of the MF metadata, and therefore the decoding start time can be advanced. Furthermore, by having the transmitting device 15 generate auxiliary data based on the moof described using FIG. 30, the receiving device 20 can parse using a conventional MP4 parser as is.

In addition, the newly generated auxiliary data and MF metadata contain sample information that overlaps with previously transmitted auxiliary data. Therefore, even if previous auxiliary data cannot be obtained due to packet loss, it is possible to reconstruct the MP4 and obtain sample information (PTS, DTS, size, and offset) by using the newly obtained auxiliary data and MF metadata.

Note that auxiliary data does not necessarily have to include information about past sample data. For example, auxiliary data #1 may correspond to sample data #1-#10, and auxiliary data #2 may correspond to sample data #11-#20. For example, as shown in (c) of FIG. 31, the transmitting device 15 may send complete MF metadata as a data unit, and fragment units of the data unit as auxiliary data in sequence.

In addition, the transmitting device 15 may repeatedly transmit auxiliary data or repeatedly transmit MF metadata to deal with packet loss.

The MMT packets and MMT payloads in which auxiliary data is stored contain an MPU sequence number and an asset ID, as well as MPU metadata, MF metadata, and sample data.

The above-described reception operation using auxiliary data will be explained using the flowchart in FIG. 32. FIG. 32 is a flowchart of the reception operation using auxiliary data.

First, the receiving device 20 receives an MMT packet and analyzes the packet header and payload header (S501). Next, the receiving device 20 analyzes whether the fragment type is auxiliary data or MF metadata (S502), and if the fragment type is auxiliary data, overwrites and updates the previous auxiliary data (S503). At this time, if there is no previous auxiliary data for the same MPU, the receiving device 20 treats the received auxiliary data as new auxiliary data as it is. Then, the receiving device 20 acquires samples based on the MPU metadata, auxiliary data, and sample data, and performs decoding (S507).

On the other hand, if the fragment type is MF metadata, in step S505, the receiving device 20 overwrites the previous auxiliary data with MF metadata (S505). Then, the receiving device 20 obtains the sample in the form of a complete MPU based on the MPU metadata, MF metadata, and sample data, and performs decoding (S506).

Although not shown in FIG. 32, in step S502, if the fragment type is MPU metadata, the receiving device 20 stores the data in a buffer, and if the fragment type is sample data, the receiving device 20 stores the data concatenated at the end for each sample in a buffer.

If auxiliary data cannot be obtained due to packet loss, the receiving device 20 can either overwrite the sample with the latest auxiliary data or use previous auxiliary data to decode the sample.

The transmission cycle and the number of transmissions of the auxiliary data may be predetermined values. Information on the transmission cycle and the number of times (count, countdown) may be transmitted together with the data. For example, the transmission cycle, the number of times of transmission, and timestamps such as initial_cpb_removal_delay may be stored in the data unit header.

By sending auxiliary data containing information about the first sample of the MPU at least once before initial_cpb_removal_delay, it is possible to comply with the CPB buffer model. At this time, the MPU timestamp descriptor stores a value based on the picture timing SEI.

Note that the transmission method for receiving operations in which such auxiliary data is used is not limited to the MMT method, but can also be applied to cases such as streaming transmission of packets configured in ISOBMFF file format, such as MPEG-DASH.

[Transmission method when one MPU consists of multiple movie fragments]
In the above description from Fig. 19 onwards, one MPU is composed of one movie fragment, but here, a case where one MPU is composed of multiple movie fragments will be described. Fig. 33 shows the structure of an MPU composed of multiple movie fragments.

In Figure 33, samples (#1-#6) stored in one MPU are stored in two movie fragments. The first movie fragment is generated based on samples #1-#3, and a corresponding moof box is generated. The second movie fragment is generated based on samples #4-#6, and a corresponding moof box is generated.

The headers of the moof box and mdat box in the first movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #1. Meanwhile, the headers of the moof box and mdat box in the second movie fragment are stored in the MMT payload and MMT packet as movie fragment metadata #2. Note that in Figure 33, the MMT payload in which the movie fragment metadata is stored is hatched.

The number of samples constituting an MPU and the number of samples constituting a movie fragment are arbitrary. For example, the number of samples constituting an MPU may be the number of samples in a GOP unit, and two movie fragments may be constructed by using half the number of samples in a GOP unit as a movie fragment.

Note that, although an example is shown here in which one MPU contains two movie fragments (a moof box and an mdat box), one MPU may contain more than two movie fragments, or may contain three or more movie fragments. Also, the samples stored in the movie fragment do not have to be divided into equal samples, but may be divided into any number of samples.

In FIG. 33, the MPU metadata unit and the MF metadata unit are each stored as a data unit in the MMT payload. However, the transmitting device 15 may store units such as ftyp, mmpu, moov, and moof as data units in the MMT payload in data unit units, or may store data units in the MMT payload in fragmented units. The transmitting device 15 may also store data units in aggregated units in the MMT payload.

In addition, in FIG. 33, samples are stored in the MMT payload in sample units. However, transmitting device 15 may configure data units in NAL unit units or units that aggregate multiple NAL units instead of sample units, and store the data units in the MMT payload. Transmitting device 15 may also store data units in fragmented units in the MMT payload, or may store data units in aggregated units in the MMT payload.

In FIG. 33, the MPU is configured in the order of moof#1, mdat#1, moof#2, mdat#2, and an offset is added to moof#1 as if the corresponding mdat#1 were attached after it. However, an offset may also be added to mdat#1 as if it were attached before moof#1. In this case, however, movie fragment metadata cannot be generated in the form of moof+mdat, and the headers of moof and mdat are transmitted separately.

Next, we will explain the transmission order of MMT packets when an MPU with the configuration described in Figure 33 is transmitted. Figure 34 is a diagram for explaining the transmission order of MMT packets.

Figure 34 (a) shows the transmission order when transmitting MMT packets in the configuration order of the MPUs shown in Figure 33. Specifically, Figure 34 (a) shows an example in which MPU meta, MF meta #1, media data #1 (samples #1-#3), MF meta #2, and media data #2 (samples #4-#6) are transmitted in this order.

(b) in Figure 34 shows an example of transmitting MPU meta, media data #1 (samples #1-#3), MF meta #1, media data #2 (samples #4-#6), and MF meta #2 in that order.

(c) in Figure 34 shows an example of transmitting media data #1 (samples #1-#3), MPU meta, MF meta #1, media data #2 (samples #4-#6), and MF meta #2 in that order.

MF meta #1 is generated using samples #1-#3, and MF meta #2 is generated using samples #4-#6. Therefore, when the transmission method of FIG. 34(a) is used, a delay occurs in the transmission of sample data due to encapsulation.

In contrast, when the transmission methods in Figures 34(b) and 34(c) are used, samples can be transmitted without waiting for the generation of MF meta, so no delay due to encapsulation occurs and end-to-end delay can be reduced.

Also, in the transmission order of Figure 34 (a), one MPU is divided into multiple movie fragments, and the number of samples stored in the MF meta is smaller than in the case of Figure 19, so the amount of delay due to encapsulation can be made smaller than in the case of Figure 19.

In addition to the method shown here, for example, the transmitting device 15 may concatenate MF meta #1 and MF meta #2 and transmit them together at the end of the MPU. In this case, the MF meta of different movie fragments may be aggregated and stored in one MMT payload. Also, the MF meta of different MPUs may be aggregated and stored in the MMT payload.

[Reception method when one MPU consists of multiple movie fragments]
Here, an operation example of the receiving device 20 that receives and decodes the MMT packets transmitted in the transmission order described in (b) of Fig. 34 will be described. Figs. 35 and 36 are diagrams for explaining such an operation example.

The receiving device 20 receives MMT packets containing MPU meta, samples, and MF meta, each transmitted in the transmission order shown in FIG. 35. The sample data is concatenated in the order in which it is received.

At T1, which is the time when MF meta #1 is received, the receiving device 20 concatenates the data as shown in (1) of FIG. 36 to construct an MP4. This allows the receiving device 20 to obtain samples #1-#3 based on the MPU metadata and the information in MF meta #1, and to perform decoding based on the PTS, DTS, offset, and size information included in the MF meta.

Furthermore, at T2, which is the time when MF meta #2 is received, receiving device 20 concatenates the data as shown in (2) of FIG. 36 to construct an MP4. This allows receiving device 20 to obtain samples #4-#6 based on the MPU metadata and information in MF meta #2, and to perform decoding based on the PTS, DTS, offset, and size information in the MF meta. Receiving device 20 may also obtain samples #1-#6 based on information in MF meta #1 and MF meta #2 by concatenating the data as shown in (3) of FIG. 36 to construct an MP4.

By dividing one MPU into multiple movie fragments, the time it takes for the MPU to obtain the first MF meta is shortened, allowing the decoding start time to be accelerated. In addition, the buffer size for storing samples before decoding can be reduced.

The transmitting device 15 may set the division unit of the movie fragment so that the time from transmitting (or receiving) the first sample in the movie fragment to transmitting (or receiving) the MF meta corresponding to the movie fragment is shorter than the initial_cpb_removal_delay specified by the encoder. By setting it in this way, the receiving buffer can follow the cpb buffer, achieving low-latency decoding. In this case, absolute times based on initial_cpb_removal_delay can be used for the PTS and DTS.

The transmitting device 15 may also divide the movie fragments at equal intervals, or divide subsequent movie fragments at shorter intervals than the previous movie fragments. This allows the receiving device 20 to always receive MF meta containing information about a sample before decoding the sample, enabling continuous decoding.

The following two methods can be used to calculate the absolute time of the PTS and DTS.

(1) The absolute time of the PTS and DTS is determined based on the reception time (T1 or T2) of MF meta #1 or MF meta #2, and the relative time of the PTS and DTS included in the MF meta.

(2) The absolute time of the PTS and DTS is determined based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

In addition, (2-A) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on the initial_cpb_removal_delay specified by the encoder.

In addition, (2-B) the absolute time signaled by the transmitting device 15 may be an absolute time calculated based on a predicted value of the reception time of the MF meta.

In addition, MF meta #1 and MF meta #2 may be transmitted repeatedly. By repeatedly transmitting MF meta #1 and MF meta #2, the receiving device 20 can obtain the MF meta again even if it is unable to obtain the MF meta due to packet loss, etc.

The payload header of an MFU containing samples that constitute a movie fragment can store an identifier indicating the order of the movie fragment. On the other hand, an identifier indicating the order of the MF meta that constitutes the movie fragment is not included in the MMT payload. For this reason, the receiving device 20 identifies the order of the MF meta by packet_sequence_number. Alternatively, the transmitting device 15 may store and signal an identifier indicating which movie fragment the MF meta belongs to in control information (message, table, descriptor), MMT header, MMT payload header, or data unit header.

The transmitting device 15 may transmit the MPU meta, MF meta, and samples in a predetermined transmission order, and the receiving device 20 may perform the receiving process based on the predetermined transmission order. The transmitting device 15 may also signal the transmission order, and the receiving device 20 may select (determine) the receiving process based on the signaling information.

The above-described receiving method will be explained using FIG. 37. FIG. 37 is a flowchart showing the operation of the receiving method explained in FIG. 35 and FIG. 36.

First, the receiving device 20 determines (identifies) whether the data contained in the payload is MPU metadata, MF metadata, or sample data (MFU) based on the fragment type indicated in the MMT payload (S601, S602). If the data is sample data, the receiving device 20 buffers the sample and waits for the reception of MF metadata corresponding to the sample and the start of decoding (S603).

On the other hand, in step S602, if the data is MF metadata, the receiving device 20 obtains sample information (PTS, DTS, position information, and size) from the MF metadata, obtains a sample based on the obtained sample information, and decodes and presents the sample based on the PTS and DTS (S604).

Although not shown, if the data is MPU metadata, the MPU metadata contains initialization information necessary for decoding. Therefore, the receiving device 20 stores this information and uses it to decode the sample data in step S604.

When the receiving device 20 stores the received MPU data (MPU metadata, MF metadata, and sample data) in a storage device, it stores the data after rearranging it into the MPU configuration described in Figure 19 or Figure 33.

In addition, on the transmitting side, a packet sequence number is assigned to MMT packets with the same packet ID. At this time, the packet sequence number may be assigned after the MMT packets containing the MPU metadata, MF metadata, and sample data are rearranged in the transmission order, or the packet sequence number may be assigned in the order before rearrangement.

If packet sequence numbers are assigned in the order before rearrangement, the receiving device 20 can rearrange the data in the configuration order of the MPU based on the packet sequence numbers, making storage easier.

[Method for detecting the start of an access unit and the start of a slice segment]
A method for detecting the beginning of an access unit or the beginning of a slice segment based on information in the MMT packet header and the MMT payload header will be described.

Two examples are shown here: one in which non-VCL NAL units (such as access unit delimiters, VPS, SPS, PPS, and SEI) are collectively stored as data units in the MMT payload, and the other in which each non-VCL NAL unit is treated as a data unit, and the data units are aggregated and stored in a single MMT payload.

Figure 38 shows a case where non-VCL NAL units are aggregated as individual data units.

In the case of FIG. 38, the start of the access unit is an MMT packet whose fragment_type value is MFU, and is the start data of an MMT payload that includes a data unit whose aggregation_flag value is 1 and whose offset value is 0. In this case, the Fragmentation_indicator value is 0.

In the case of FIG. 38, the start of a slice segment is an MMT packet whose fragment_type value is MFU, and is the start data of an MMT payload whose aggregation_flag value is 0 and whose fragmentation_indicator value is 00 or 01.

Figure 39 shows a case where non-VCL NAL units are grouped together into a data unit. Note that the field values of the packet header are as shown in Figure 17 (or Figure 18).

In the case of Figure 39, the start of the access unit is the first data in the payload of a packet with an Offset value of 0.

In the case of FIG. 39, the start of a slice segment is the first data of the payload of a packet whose offset value is a value other than 0 and whose fragmentation indicator value is 00 or 01.

[Reception process when packet loss occurs]
Typically, when transmitting MP4 format data in an environment where packet loss occurs, the receiving device 20 restores packets using ALFEC (Application Layer FEC), packet retransmission control, or the like.

However, if AL-FEC cannot be used in streaming such as broadcasting and packet loss occurs, the packets cannot be restored.

After data is lost due to packet loss, the receiving device 20 needs to resume decoding of video and audio. To do this, the receiving device 20 needs to detect the beginning of an access unit or NAL unit and start decoding from the beginning of the access unit or NAL unit.

However, since there is no start code at the beginning of an MP4 format NAL unit, the receiving device 20 cannot detect the beginning of an access unit or NAL unit even if it analyzes the stream.

Figure 40 is a flowchart showing the operation of the receiving device 20 when packet loss occurs.

The receiving device 20 detects packet loss using the packet sequence number, packet counter, fragment counter, etc. in the header of the MMT packet or MMT payload (S701), and determines which packet has been lost based on the context (S702).

If the receiving device 20 determines that no packet loss has occurred (No in S702), it constructs an MP4 file and decodes the access units or NAL units (S703).

When it is determined that a packet loss has occurred (Yes in S702), the receiving device 20 generates a NAL unit equivalent to the NAL unit that has experienced packet loss using dummy data, and constructs an MP4 file (S704). When the receiving device 20 puts dummy data into the NAL unit, it indicates that the type of the NAL unit is dummy data.

In addition, the receiving device 20 can resume decoding by detecting the beginning of the next access unit or NAL unit and inputting the data from the beginning into the decoder based on the methods described in Figures 17, 18, 38, and 39 (S705).

In addition, if a packet loss occurs, the receiving device 20 may resume decoding from the beginning of the access unit and NAL unit based on the information detected based on the packet header, or may resume decoding from the beginning of the access unit and NAL unit based on the header information of the reconstructed MP4 file, which includes the NAL unit of dummy data.

When storing an MP4 file (MPU), the receiving device 20 may separately obtain and store (replace) packet data (such as NAL units) lost due to packet loss from broadcasting or communication.

At this time, when the receiving device 20 acquires the lost packet from the communication, it notifies the server of the information of the lost packet (packet ID, MPU sequence number, packet sequence number, IP data flow number, IP address, etc.) and acquires the packet. The receiving device 20 may not only acquire the lost packet, but may also acquire a group of packets before and after the lost packet at the same time.

[How to compose a movie fragment]
Here we will explain in detail how to configure movie fragments.

As described in FIG. 33, the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU are arbitrary. For example, the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU may be a fixed, predetermined number, or may be dynamically determined.

Here, by configuring the movie fragment on the transmitting side (transmitting device 15) to satisfy the following conditions, low-latency decoding can be guaranteed on the receiving device 20.

The conditions are as follows:

The transmitting device 15 generates and transmits MF meta in the form of movie fragments, which are units obtained by dividing sample data, so that the receiving device 20 can always receive MF meta including information about any sample (Sample(i)) before the decoding time (DTS(i)) of that sample.

Specifically, the transmitting device 15 constructs a movie fragment using samples (including the i-th sample) that have been encoded prior to DTS(i).

To ensure low-latency decoding, the following method, for example, is used to dynamically determine the number of samples that make up a movie fragment or the number of movie fragments that make up one MPU.

(1) When decoding starts, the decoding time DTS(0) of the first sample Sample(0) of the GOP is a time based on initial_cpb_removal_delay. The transmitting device constructs a first movie fragment using samples that have already been encoded at a time before DTS(0). The transmitting device 15 also generates MF metadata corresponding to the first movie fragment and transmits it at a time before DTS(0).

(2) The transmitting device 15 constructs the movie fragment so that the above conditions are satisfied for subsequent samples as well.

For example, if the first sample of a movie fragment is the kth sample, the MF meta of the movie fragment including the kth sample is transmitted by the decoding time DTS(k) of the kth sample. If the encoding completion time of the lth sample is before DTS(k) and the encoding completion time of the (l+1)th sample is after DTS(k), the transmitting device 15 constructs a movie fragment using the kth sample to the lth sample.

In addition, the transmitting device 15 may construct a movie fragment using samples from the kth sample to the lth sample.

(3) After completing the encoding of the last sample of the MPU, the transmitting device 15 constructs a movie fragment using the remaining samples, generates MF metadata corresponding to the movie fragment, and transmits it.

In addition, the transmitting device 15 may construct a movie fragment using only a portion of the samples that have been encoded, rather than constructing the movie fragment using all of the samples that have been encoded.

In the above, an example has been shown in which the number of samples constituting a movie fragment and the number of movie fragments constituting one MPU are dynamically determined based on the above conditions to ensure low-latency decoding. However, the method of determining the number of samples and the number of movie fragments is not limited to this method. For example, the number of movie fragments constituting one MPU may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions. Also, the number of movie fragments constituting one MPU and the time at which the movie fragments are divided (or the code amount of the movie fragments) may be fixed to a predetermined value, and the number of samples may be determined to satisfy the above conditions.

In addition, if the MPU is split into multiple movie fragments, information indicating whether the MPU is split into multiple movie fragments, attributes of the split movie fragments, or MF meta attributes for the split movie fragments may be transmitted.

Here, the attributes of a movie fragment are information that indicates whether the movie fragment is the first movie fragment of an MPU, the last movie fragment of an MPU, or some other movie fragment.

In addition, the attributes of MF meta are information that indicates whether the MF meta corresponds to the first movie fragment of the MPU, the last movie fragment of the MPU, or any other movie fragment.

The transmitting device 15 may also store and transmit the number of samples that make up a movie fragment and the number of movie fragments that make up one MPU as control information.

[Operation of the receiving device]
The operation of the receiving device 20 based on the movie fragment configured as above will be described.

The receiving device 20 determines the absolute time of each PTS and DTS based on the absolute time signaled from the transmitting side, such as the MPU timestamp descriptor, and the relative time of the PTS and DTS included in the MF meta.

Based on information on whether the MPU is divided into multiple movie fragments, the receiving device 20 performs the following processing based on the attributes of the divided movie fragments if the MPU is divided.

(1) When a movie fragment is the first movie fragment of an MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the absolute time of the PTS of the first sample included in the MPU timestamp descriptor and the relative times of the PTS and DTS included in the MF meta.

(2) If the movie fragment is not the first movie fragment of the MPU, the receiving device 20 generates the absolute times of the PTS and DTS using the relative times of the PTS and DTS included in the MF meta, without using the information in the MPU timestamp descriptor.

(3) When the movie fragment is the last movie fragment of an MPU, the receiving device 20 calculates the absolute times of the PTS and DTS of all samples and then resets the PTS and DTS calculation process (addition process of relative times). Note that the reset process may be performed on the first movie fragment of the MPU.

The receiving device 20 may determine whether the movie fragment is divided as follows. The receiving device 20 may also obtain attribute information of the movie fragment as follows.

For example, the receiving device 20 may determine whether the movie fragments have been split based on the movie_fragment_sequence_number field value, an identifier indicating the order of the movie fragments indicated in the MMTP (MMT Protocol) payload header.

Specifically, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the movie_fragment_sequence_number field value is 1, and the field value is 2 or greater.

In addition, the receiving device 20 may determine that an MPU is divided into multiple movie fragments if the number of movie fragments contained in one MPU is 1, the movie_fragment_sequence_number field value is 0, and the field value is a value other than 0.

The attribute information of the movie fragment may also be determined based on the movie_fragment_sequence_number.

In addition, without using movie_fragment_sequence_number, it is also possible to determine whether a movie fragment is divided and the attribute information of the movie fragment by counting the transmission of movie fragments and MF meta contained in one MPU.

By configuring the transmitting device 15 and the receiving device 20 as described above, the receiving device 20 can receive movie fragment metadata at intervals shorter than the MPU, making it possible to start decoding with low latency. In addition, it is possible to perform decoding with low latency using a decoding process based on the MP4 parsing method.

The receiving operation when the MPU is divided into multiple movie fragments as described above will be explained using a flowchart. Figure 41 is a flowchart of the receiving operation when the MPU is divided into multiple movie fragments. Note that this flowchart illustrates the operation of step S604 in Figure 37 in more detail.

First, the receiving device 20 acquires MF metadata if the data type is MF meta based on the data type indicated in the MMTP payload header (S801).

Next, the receiving device 20 determines whether the MPU is divided into multiple movie fragments (S802), and if the MPU is divided into multiple movie fragments (Yes in S802), determines whether the received MF metadata is the first metadata of the MPU (S803). If the received MF metadata is the first MF metadata of the MPU (Yes in S803), the receiving device 20 calculates the absolute times of the PTS and DTS from the absolute time of the PTS indicated in the MPU timestamp descriptor and the relative times of the PTS and DTS indicated in the MF metadata (S804), and determines whether the metadata is the last metadata of the MPU (S805).

On the other hand, if the received MF metadata is not the MF metadata at the beginning of an MPU (No in S803), the receiving device 20 does not use the information in the MPU timestamp descriptor, but calculates the absolute times of the PTS and DTS using the relative times of the PTS and DTS indicated in the MF metadata (S808), and proceeds to processing in step S805.

If it is determined in step S805 that this is the last MF metadata of the MPU (Yes in S805), the receiving device 20 calculates the absolute times of the PTS and DTS of all samples and then resets the PTS and DTS calculation process. If it is determined in step S805 that this is not the last MF metadata of the MPU (No in S805), the receiving device 20 ends the process.

Also, if it is determined in step S802 that the MPU is not divided into multiple movie fragments (No in S802), the receiving device 20 acquires sample data and determines the PTS and DTS based on the MF metadata transmitted after the MPU (S807).

Then, although not shown, the receiving device 20 finally performs decoding and presentation processing based on the determined PTS and DTS.

[Issues that arise when splitting movie fragments and their solutions]
So far, we have explained how to reduce the end-to-end delay by dividing movie fragments. From here on, we will explain new issues that arise when dividing movie fragments and how to solve them.

First, as background, we will explain the picture structure in the encoded data. Figure 42 shows an example of the prediction structure of pictures for each TemporalId when implementing temporal scalability.

In coding formats such as MPEG-4 AVC and HEVC (High Efficiency Video Coding), temporal scalability can be achieved by using B-pictures (bidirectional reference predictive pictures) that can be referenced from other pictures.

TemporalId shown in FIG. 42(a) is an identifier of the hierarchy of the coding structure, and the larger the value of TemporalId, the deeper the hierarchy. The square blocks indicate pictures, and Ix in the blocks indicates I picture (intra-screen predicted picture), Px indicates P picture (forward reference predicted picture), and Bx and bx indicate B picture (bidirectional reference predicted picture). The x in Ix/Px/Bx indicates the display order, which indicates the order in which the pictures are displayed. The arrows between pictures indicate the reference relationship, and for example, picture B4 indicates that a predicted image is generated using I0 and B8 as reference images. Here, it is prohibited for one picture to use another picture with a TemporalId greater than its own TemporalId as a reference image. The layers are defined to allow for temporal scalability; for example, decoding all pictures in Figure 42 results in 120 fps (frame per second) video, but decoding only layers with TemporalId from 0 to 3 results in 60 fps video.

Figure 43 shows the relationship between the decode time (DTS) and the display time (PTS) for each picture in Figure 42. For example, picture I0 shown in Figure 43 is displayed after the decoding of B4 is completed so that there is no gap in the decoding and display.

As shown in FIG. 43, when the prediction structure includes a B picture, the decoding order and the display order are different, and therefore picture delay processing and picture reordering processing are required after decoding the picture in the receiving device 20.

The above describes examples of picture prediction structures in temporal scalability, but even when temporal scalability is not used, picture delay and reorder processing may be required depending on the prediction structure. Figure 44 shows an example of a picture prediction structure that requires picture delay and reorder processing. Note that the numbers in Figure 44 indicate the decoding order.

As shown in FIG. 44, depending on the prediction structure, the first sample in decoding order may be different from the first sample in presentation order; in FIG. 44, the first sample in presentation order is the fourth sample in decoding order. Note that FIG. 44 shows an example of a prediction structure, and prediction structures are not limited to this structure. In other prediction structures, the first sample in decoding order may be different from the first sample in presentation order.

As with FIG. 33, FIG. 45 is a diagram showing an example in which an MPU in MP4 format is divided into multiple movie fragments and stored in an MMTP payload and MMTP packets. Note that the number of samples constituting an MPU and the number of samples constituting a movie fragment are arbitrary. For example, the number of samples constituting an MPU may be the number of samples in a GOP unit, and two movie fragments may be formed by using half the number of samples in a GOP unit as a movie fragment. One sample may be one movie fragment, or the samples constituting an MPU may not be divided.

Figure 45 shows an example in which one MPU contains two movie fragments (a moof box and an mdat box), but one MPU does not have to contain two movie fragments. One MPU may contain three or more movie fragments, or the number of samples contained in the MPU. Furthermore, the samples stored in a movie fragment do not have to be divided into an equal number of samples, but may be divided into any number of samples.

Movie fragment metadata (MF metadata) contains information on the PTS, DTS, offset, and size of the samples contained in the movie fragment, and when the receiving device 20 decodes a sample, it extracts the PTS and DTS from the MF meta that contains the information on the sample, and determines the decoding timing and presentation timing.

From here on, for the sake of detailed explanation, the absolute value of the decoding time of the i sample will be written as DTS(i), and the absolute value of the presentation time will be written as PTS(i).

The information for the i-th sample among the timestamp information stored in the moof in the MF meta is specifically the relative value of the decoded time of the i-th sample and the (i+1)-th sample, and the relative value of the decoded time of the i-th sample and the presentation time, hereafter referred to as DT(i) and CT(i).

Movie fragment metadata #1 contains DT(i) and CT(i) for samples #1-#3, and movie fragment metadata #2 contains DT(i) and CT(i) for samples #4-#6.

In addition, the absolute PTS value of the first access unit of the MPU is stored in the MPU timestamp descriptor, etc., and the receiving device 20 calculates the PTS and DTS based on the PTS_MPU of the first access unit of the MPU, and the CT and DT.

Figure 46 is a diagram to explain the calculation method and issues of PTS and DTS when an MPU is composed of samples #1-#10.

Figure 46(a) shows an example where an MPU is not divided into movie fragments, Figure 46(b) shows an example where an MPU is divided into two movie fragments of 5 sample units, and Figure 46(c) shows an example where an MPU is divided into 10 movie fragments of sample units.

As explained in Figure 45, when the PTS and DTS are calculated using the MPU timestamp descriptor and the timestamp information (CT and DT) in the MP4, the first sample in the presentation order in Figure 44 is the fourth in the decoding order. Therefore, the PTS stored in the MPU timestamp descriptor is the PTS (absolute value) of the fourth sample in the decoding order. Note that hereafter, this sample will be referred to as the A sample. Also, the first sample in the decoding order will be referred to as the B sample.

Since the only absolute time information related to the timestamp is the information in the MPU timestamp descriptor, the receiving device 20 cannot calculate the PTS (absolute time) and DTS (absolute time) of other samples until the A sample arrives. The receiving device 20 cannot calculate the PTS and DTS of the B sample either.

In the example of FIG. 46(a), sample A is included in the same movie fragment as sample B and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of sample B immediately after receiving the MF meta.

In the example of FIG. 46(b), sample A is included in the same movie fragment as sample B and is stored in one MF meta. Therefore, the receiving device 20 can determine the DTS of sample B immediately after receiving the MF meta.

In the example of (c) in FIG. 46, sample A is included in a different movie fragment from sample B. Therefore, the receiving device 20 cannot determine the DTS of sample B until it receives the MF meta including the CT and DT of the movie fragment that includes sample A.

Therefore, in the example of (c) in Figure 46, the receiving device 20 cannot start decoding immediately after the arrival of sample B.

In this way, if a movie fragment containing a B sample does not contain an A sample, the receiving device 20 cannot start decoding the B sample until it has received the MF meta for the movie fragment containing the A sample.

This issue occurs when the first sample in presentation order does not match the first sample in decoding order, and the movie fragment is split to the point where sample A and sample B are no longer stored in the same movie fragment. This issue also occurs regardless of whether the MF meta is postponed or postponed.

In this way, when the first sample in presentation order does not match the first sample in decoding order, if sample A and sample B are not stored in the same movie fragment, the DTS cannot be determined immediately after receiving sample B. Therefore, transmitting device 15 separately transmits the DTS (absolute value) of sample B, or information that allows the receiving side to calculate the DTS (absolute value) of sample B. Such information may be transmitted using control information, a packet header, etc.

The receiving device 20 uses this information to calculate the DTS (absolute value) of the B sample. Figure 47 is a flowchart of the receiving operation when the DTS is calculated using this information.

The receiving device 20 receives the movie fragment at the beginning of the MPU (S901), and determines whether the A sample and the B sample are stored in the same movie fragment (S902). If they are stored in the same movie fragment (Yes in S902), the receiving device 20 calculates the DTS using only the MF meta information, without using the DTS (absolute time) of the B sample, and starts decoding (S904). Note that in step S904, the receiving device 20 may determine the DTS using the DTS of the B sample.

On the other hand, if sample A and sample B are not stored in the same movie fragment in step S902 (No in S902), the receiving device 20 obtains the DTS (absolute time stamp) of sample B, determines the DTS, and starts decoding (S903).

In the above description, an example was described in which the absolute value of the decoded time and the absolute value of the presentation time of each sample were calculated using the MF meta (timestamp information stored in the moof in MP4 format) in the MMT standard. However, it goes without saying that the MF meta may be replaced with any control information that can be used to calculate the absolute value of the decoded time and the absolute value of the presentation time of each sample. Examples of such control information include control information in which the relative value CT(i) of the decoded time of the i-th sample and the (i+1)-th sample described above is replaced with the relative value of the presentation time of the i-th sample and the (i+1)-th sample, and control information that includes both the relative value CT(i) of the decoded time of the i-th sample and the (i+1)-th sample and the relative value of the presentation time of the i-th sample and the (i+1)-th sample.

(Embodiment 3)
[overview]
In the third embodiment, a content transmission method and data structure when content such as video, audio, subtitles, and data broadcasting is transmitted by broadcasting will be described. That is, a content transmission method and data structure specialized for playing back a broadcast stream will be described.

Note that in the third embodiment, an example will be described in which the MMT method (hereinafter also simply referred to as MMT) is used as the multiplexing method, but other multiplexing methods such as MPEG-DASH or RTP may also be used.

First, we will explain the details of how to store a data unit (DU) in the payload in MMT. Figure 48 is a diagram for explaining the method of storing a data unit in the payload in MMT.

In MMT, the transmitting device stores part of the data that constitutes the MPU in the MMTP payload as a data unit and transmits it with a header. The header includes an MMTP payload header and an MMTP packet header. The data unit may be in units of NAL units or in units of samples.

Figure 48 (a) shows an example in which a transmitting device aggregates multiple data units and stores them in one payload. In the example of Figure 48 (a), a data unit header (DUH: Data Unit Header) and a data unit length (DUL: Data Unit Length) are added to the beginning of each of the multiple data units, and multiple data units with the data unit header and data unit length are stored together in the payload.

Figure 48 (b) shows an example of storing one data unit in one payload. In the example of Figure 48 (b), a data unit header is added to the beginning of the data unit and stored in the payload. Figure 48 (c) shows an example of dividing one data unit, adding a data unit header to the divided data units, and storing them in the payload.

Data units come in a variety of types, including timed-MFU, which is media that includes information about synchronization such as video, audio, or subtitles; non-timed-MFU, which is media such as files that does not include information about synchronization; MPU metadata; and MF metadata. Data unit headers are determined according to the type of data unit. Note that MPU metadata and MF metadata do not have data unit headers.

In addition, although a transmitting device cannot in principle aggregate different types of data units, it may be specified to be able to aggregate different types of data units. For example, when the size of the MF metadata is small, such as when it is divided into movie fragments for each sample, the number of packets can be reduced by aggregating the MF metadata and media data, and further, the transmission capacity can be reduced.

If the data unit is an MFU, some information about the MPU, such as information for constructing the MPU (MP4), is stored as the header.

For example, the header of a timed-MFU includes movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter, while the header of a non-timed-MFU includes item_iD. The meaning of each field is specified in standards such as ISO/IEC23008-1 or ARIB STD-B60. The meaning of each field specified in such standards is explained below.

movie_fragment_sequence_number indicates the sequence number of the movie fragment to which the MFU belongs, and is also specified in ISO/IEC14496-12.

sample_number indicates the sample number to which the MFU belongs, and is also specified in ISO/IEC14496-12.

Offset indicates the offset of the MFU in the sample to which the MFU belongs, in bytes.

Priority indicates the relative importance of the MFU in the MPU to which it belongs, with an MFU with a higher priority number being more important than an MFU with a lower priority number.

The dependency_counter indicates the number of MFUs whose decoding process depends on the MFU (i.e., the number of MFUs whose decoding process cannot be performed unless this MFU is decoded). For example, when the MFU is HEVC and a B picture or P picture references an I picture, the B picture or P picture cannot be decoded unless the I picture is decoded.

Therefore, when the MFU is in sample units, the dependency_counter in the MFU of an I picture indicates the number of pictures that reference the I picture. When the MFU is in NAL unit units, the dependency_counter in the MFU belonging to the I picture indicates the number of NAL units that belong to the picture that references the I picture. Furthermore, in the case of a video signal that has been hierarchically coded in the temporal direction, since the MFU of the enhancement layer depends on the MFU of the base layer, the dependency_counter in the MFU of the base layer indicates the number of MFUs of the enhancement layer. This field can only be generated after the number of dependent MFUs has been determined.

item_id indicates an identifier that uniquely identifies the item.

[MP4 non-support mode]
As described in Fig. 19 and Fig. 21, the method of transmitting an MPU in MMT by a transmitting device includes a method of transmitting MPU metadata or MF metadata before or after media data, and a method of transmitting only media data. In addition, the receiving device may perform decoding using a receiving device or receiving method that conforms to MP4, or may perform decoding without using a header.

As an example of a data transmission method specialized for broadcast stream playback, there is a transmission method that does not support MP4 reconstruction on the receiving device.

An example of a transmission method that does not support MP4 reconstruction in a receiving device is a method that does not transmit metadata (MPU metadata and MF metadata), as shown in (b) of FIG. 21. In this case, the field value of the fragment type (information indicating the type of data unit) included in the MMTP packet is fixed to 2 (=MFU).

As explained above, if metadata is not transmitted, an MP4-compliant receiving device cannot decode the received data as MP4, but it is possible to decode it without using metadata (header).

Therefore, metadata is not necessarily essential information for broadcast stream decoding and playback. Similarly, the information in the data unit header in timed-MFU, described in FIG. 48, is information for reconstructing MP4 in the receiving device. Since there is no need to reconstruct MP4 for broadcast stream playback, the information in the data unit header in timed-MFU (hereinafter also referred to as timed-MFU header) is not necessarily information required for broadcast stream playback.

The receiving device can easily reconstruct the MP4 by using the metadata and the information for reconstructing the MP4 in the data unit header (hereinafter also referred to as MP4 configuration information). However, the receiving device cannot reconstruct the MP4 even if only either the metadata or the MP4 configuration information in the data unit header is transmitted. There is little benefit to transmitting only either the metadata or the information for reconstructing the MP4, and generating and transmitting unnecessary information leads to increased processing and reduced transmission efficiency.

The transmitting device therefore controls the data structure and transmission of the MP4 configuration information using the following method. The transmitting device determines whether to indicate the MP4 configuration information in the data unit header based on whether metadata is transmitted. Specifically, the transmitting device indicates the MP4 configuration information in the data unit header when metadata is transmitted, and does not indicate the MP4 configuration information in the data unit header when metadata is not transmitted.

As a method for not indicating MP4 configuration information in the data unit header, for example, the following method can be used.

1. The transmitting device reserves the MP4 configuration information and does not use it. This reduces the amount of processing on the sending side (the amount of processing on the transmitting device) that generates the MP4 configuration information.

2. The transmitting device deletes the MP4 configuration information and compresses the header. This reduces the amount of processing on the sending side that generates the MP4 configuration information, and also reduces transmission capacity.

When the transmitting device deletes the MP4 configuration information and compresses the header, the transmitting device may indicate a flag indicating that the MP4 configuration information has been deleted (compressed). The flag is indicated in the header (MMTP packet header, MMTP payload header, data unit header) or control information, etc.

In addition, information on whether metadata is transmitted may be determined in advance, or may be signaled separately in the header or control information and transmitted to the receiving device.

For example, the MFU header may store information on whether metadata corresponding to the MFU is being transmitted.

On the other hand, the receiving device can determine whether MP4 configuration information is indicated based on whether metadata is transmitted.

Here, if the data transmission order (e.g., MPU metadata, MF metadata, media data) is fixed, the receiving device may make the determination based on whether metadata is received before the media data.

If MP4 configuration information is indicated, the receiving device can use the MP4 configuration information to reconstruct the MP4. Alternatively, the receiving device can use the MP4 configuration information to detect the beginning of other access units or NAL units, etc.

Note that the MP4 configuration information may be all or part of the timed-MFU header.

Similarly, the transmitting device also determines whether metadata is transmitted in the non-timed-MFU header by setting the item
It may also be determined whether to show the id.

The transmitting device may indicate the MP4 configuration information in only one of the timed-MFU and non-timed-MFU. If the MP4 configuration information is indicated in only one of the two, the transmitting device determines whether to indicate the MP4 configuration information based on whether metadata is transmitted and whether it is a timed-MFU or non-timed-MFU. The receiving device can determine whether the MP4 configuration information is indicated based on whether metadata is transmitted and the timed/non-timed flag.

In the above description, the transmitting device determines whether to indicate the MP4 configuration information based on whether metadata (both MPU metadata and MF metadata) is transmitted. However, the transmitting device may not indicate the MP4 configuration information if part of the metadata (either MPU metadata or MF metadata) is not transmitted.

The transmitting device may also determine whether to indicate MP4 configuration information based on information other than metadata.

For example, modes such as MP4 support mode/MP4 non-support mode may be defined, and the transmitting device may indicate MP4 configuration information in the data unit header in the MP4 support mode, and may not indicate MP4 configuration information in the data unit header in the MP4 non-support mode. Also, the transmitting device may transmit metadata and indicate MP4 configuration information in the data unit header in the MP4 support mode, and may not transmit metadata and may not indicate MP4 configuration information in the data unit header in the MP4 non-support mode.

[Operation flow of the transmitting device]
Next, the operation flow of the transmitting device will be described with reference to FIG.

The transmitting device first determines whether to transmit metadata (S1001). If the transmitting device determines to transmit metadata (Yes in S1002), it proceeds to step S1003, generates MP4 configuration information, stores it in the header, and transmits it (S1003). In this case, the transmitting device also generates and transmits metadata.

On the other hand, if the transmitting device determines not to transmit metadata (No in S1002), it transmits the MP4 configuration information without generating it and without storing it in the header (S1004). In this case, the transmitting device does not generate or transmit metadata.

Whether or not to transmit metadata in step S1001 may be determined in advance, or may be determined based on whether metadata has been generated within the transmitting device, or whether metadata has been transmitted within the transmitting device.

[Operation flow of receiving device]
Next, the operation flow of the receiving device will be described with reference to FIG.

First, the receiving device determines whether metadata is being transmitted (S1101). Whether metadata is being transmitted can be determined by monitoring the fragment type in the MMTP packet payload. Also, whether metadata is being transmitted may be determined in advance.

When the receiving device determines that metadata has been transmitted (Yes in S1102), it reconstructs the MP4 and executes a decryption process using the MP4 configuration information (S1103). On the other hand, when the receiving device determines that metadata has not been transmitted (No in S1102), it does not reconstruct the MP4 and executes a decryption process without using the MP4 configuration information (S1104).

In addition, the receiving device can use the methods described above to detect random access points, the beginning of an access unit, the beginning of a NAL unit, and so on, without using MP4 configuration information, and can perform decoding processes, packet loss detection, and recovery from packet loss.

For example, the beginning of an access unit is the first data of an MMT payload whose aggregation_flag value is 1. In this case, the fragmentation_indicator value is 0.

The beginning of a slice segment is the first data of an MMT payload whose aggregation_flag value is 0 and whose fragmentation_indicator value is 00 or 01.

Based on the above information, the receiving device can detect the beginning of the access unit and the slice segment.

In addition, the receiving device may analyze the NAL unit header in a packet that includes the beginning of a data unit whose fragmentation_indicator value is 00 or 01, and detect that the type of the NAL unit is an AU delimiter and that the type of the NAL unit is a slice segment.

[Broadcast simple mode]
So far, we have described a method of transmitting data specialized for broadcast stream playback that does not support MP4 configuration information in the receiving device, but the method of transmitting data specialized for broadcast stream playback is not limited to this.

As a method for transmitting data specialized for broadcast stream playback, for example, the following method may be used.

- In a fixed broadcast reception environment, the transmitting device does not need to use AL-FEC. If AL-FEC is not used, FEC_type in the MMTP packet header is always fixed to 0.

- The transmitting device may always use AL-FEC in a mobile broadcast reception environment and in a communication UDP transmission mode. When AL-FEC is used, FEC_type in the MMTP packet header is always 0 or 1.

- The transmitting device does not need to perform bulk transmission of assets. If bulk transmission of assets is not performed, location_infolocation, which indicates the number of transmission locations of assets within the MPT, may be fixed to 1.

- The transmitting device does not need to perform hybrid transmission of assets, programs, and messages.

For example, a broadcast simple mode may be specified, and the transmitting device may be in MP4 non-support mode when in broadcast simple mode, or may use the data transmission method specialized for broadcast stream playback shown above. Whether or not the mode is broadcast simple mode may be determined in advance, or the transmitting device may store a flag indicating that the mode is broadcast simple mode as control information and transmit it to the receiving device.

In addition, the transmitting device may determine whether the mode is MP4 non-support mode (whether metadata is transmitted) as described in FIG. 49, and if the mode is MP4 non-support mode, the transmitting device may use the data transmission method specialized for broadcast stream playback shown above as broadcast simple mode.

When the receiving device is in broadcast simple mode, it is considered to be in MP4 non-support mode and can perform decoding processing without reconstructing the MP4.

In addition, when the receiving device is in broadcast simple mode, it can determine that the functions are specialized for broadcasting and perform reception processing specialized for broadcasting.

As a result, when in simple broadcast mode, not only can unnecessary processing be reduced for the transmitting device and receiving device by using only functions specialized for broadcasting, but transmission overhead can also be reduced by not compressing and transmitting unnecessary information.

Note that when the MP4 non-support mode is used, hint information supporting storage methods other than the MP4 format may be indicated.

Storage methods other than the MP4 format include, for example, directly storing MMT packets or IP packets, or converting MMT packets into MPEG-2 TS packets.

Note that in MP4 non-support modes, a format that does not conform to the MP4 structure may be used.

For example, in the case of an MP4 non-support mode, the data stored in the MFU may be in a format with a byte start code at the beginning of the NAL unit, rather than the MP4 format format with the size of the NAL unit at the beginning of the NAL unit.

In MMT, the asset type indicating the type of asset is described in 4CC registered in MP4REG (http://www.mp4ra.org), and when HEVC is used as the video signal, 'HEV1' or 'HVC1' is used. 'HVC1' is a format that may include a parameter set in the sample, while 'HEV1' is a format that does not include a parameter set in the sample, but includes a parameter set in the sample entry in the MPU metadata.

In the case of broadcast simple mode or MP4 non-support mode, if MPU metadata and MF metadata are not transmitted, it may be specified that a parameter set is always included in the sample. Also, it may be specified that the format is always 'HVC1' regardless of whether 'HEV1' or 'HVC1' is indicated in the asset type.

[Supplement 1: Transmitter]
As described above, when metadata is not transmitted, the MP4 configuration information is set to "reserved" and a transmitting device that is not in operation can be configured as shown in Fig. 51. Fig. 51 is a diagram showing an example of a specific configuration of a transmitting device.

The transmitting device 300 includes an encoding unit 301, an assignment unit 302, and a transmitting unit 303. Each of the encoding unit 301, the assignment unit 302, and the transmitting unit 303 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The encoding unit 301 encodes a video signal or an audio signal to generate sample data. Specifically, the sample data is a data unit.

The attachment unit 302 attaches header information including MP4 configuration information to sample data, which is data in which a video signal or an audio signal is encoded. The MP4 configuration information is information for reconstructing the sample data as an MP4 format file on the receiving side, and the content of the information varies depending on whether the presentation time of the sample data is specified or not.

As described above, the attachment unit 302 includes MP4 configuration information such as movie_fragment_sequence_number, sample_number, offset, priority, and dependency_counter in the header (header information) of timed-MFU, which is an example of sample data for which a presentation time has been determined (sample data including information regarding synchronization).

On the other hand, the attachment unit 302 includes MP4 configuration information such as item_id in the header (header information) of timed-MFU, which is an example of sample data for which the presentation time is not specified (sample data that does not include information regarding synchronization).

If the sending unit 303 does not send metadata corresponding to the sample data (for example, as in (b) of FIG. 21), the adding unit 302 adds header information that does not include MP4 configuration information to the sample data depending on whether the presentation time of the sample data has been determined.

Specifically, when the presentation time of the sample data is determined, the assigning unit 302 assigns header information that does not include the first MP4 configuration information to the sample data, and when the presentation time of the sample data is not determined, the assigning unit 302 assigns header information that includes the second MP4 configuration information to the sample data.

For example, as shown in step S1004 of FIG. 49, when the transmitting unit 303 does not transmit metadata corresponding to the sample data, the adding unit 302 sets the MP4 configuration information to reserved (fixed value), so that the MP4 configuration information is not actually generated and is not actually stored in the header (header information). Note that the metadata includes MPU metadata and movie fragment metadata.

The transmitting unit 303 transmits the sample data to which the header information has been added. More specifically, the transmitting unit 303 packetizes the sample data to which the header information has been added using the MMT method and transmits the packetized data.

As described above, in the transmission method and reception method specialized for playing back a broadcast stream, there is no need for the receiving device to reconstruct data units into MP4. If the receiving device does not need to reconstruct data into MP4, the processing load of the transmitting device is reduced by not generating unnecessary information such as MP4 configuration information.

On the other hand, the transmitting device must send the necessary information, but must maintain compatibility with the standard so that it does not have to send unnecessary additional information separately.

With a configuration like that of the transmitting device 300, by setting the area in which the MP4 configuration information is stored to a fixed value, it is possible to achieve the effect of not transmitting the MP4 configuration information, transmitting only the necessary information based on the standard, and eliminating the need to transmit unnecessary additional information. In other words, it is possible to reduce the configuration of the transmitting device and the amount of processing performed by the transmitting device. Furthermore, by not transmitting unnecessary data, it is possible to improve transmission efficiency.

[Supplement 2: Receiving device]
Moreover, a receiving device corresponding to the transmitting device 300 may be configured, for example, as shown in Fig. 52. Fig. 52 is a diagram showing another example of the configuration of the receiving device.

The receiving device 400 includes a receiving unit 401 and a decoding unit 402. The receiving unit 401 and the decoding unit 402 are realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The receiving unit 401 receives sample data, which is data in which a video signal or an audio signal has been encoded, and which is provided with header information including MP4 configuration information for reconstructing the sample data as a file in MP4 format.

If the receiving unit does not receive metadata corresponding to the sample data and the presentation time of the sample data is determined, the decoding unit 402 decodes the sample data without using the MP4 configuration information.

For example, as shown in step S1104 of FIG. 50, if the receiving unit 401 does not receive metadata corresponding to the sample data, the decoding unit 402 performs the decoding process without using the MP4 configuration information.

This reduces the configuration of the receiving device 400 and the amount of processing in the receiving device 400.

(Embodiment 4)
[overview]
In the fourth embodiment, a method of storing asynchronous (non-timed) media that does not include information related to synchronization such as files in an MPU and a method of transmitting the media in an MMTP packet will be described. Note that in the fourth embodiment, an MPU in MMT will be described as an example, but it can also be applied to DASH, which is based on the same MP4.

First, the details of how non-timed media (hereinafter also referred to as "asynchronous media data") is stored in the MPU will be explained using FIG. 53. FIG. 53 is a diagram showing how non-timed media is stored in the MPU and how it is transmitted in an MMTP packet.

An MPU that stores non-timed media consists of boxes such as ftype, mmpu, moov, and meta, and stores information about the files stored in the MPU. Multiple idat boxes can be stored in a meta box, and an idat box stores one file as an item.

Part of the ftyp, mmpu, moov, and meta boxes constitute one data unit as MPU metadata, and the item or idat box constitutes a data unit as an MFU.

After the data units are aggregated and fragmented, they are given a data unit header, an MMTP payload header, and an MMTP packet header and transmitted as an MMTP packet.

Note that FIG. 53 shows an example in which File #1 and File #2 are stored in one MPU. The MPU metadata is not divided, and the MFU is divided and stored in an MMTP packet, but this is not limited to the above, and data units may be aggregated or fragmented depending on the size of the data unit. Also, the MPU metadata does not need to be transmitted, in which case only the MFU is transmitted.

Header information such as the data unit header indicates the item ID (an identifier that uniquely identifies an item), and the MMTP payload header and MMTP packet header include a packet sequence number (a sequence number for each packet) and an MPU sequence number (a sequence number for the MPU, a number unique within an asset).

The data structure of the MMTP payload header and MMTP packet header other than the data unit header is the same as that of the timed media (hereinafter also referred to as "synchronous media data") described above, and includes aggregation_flag, fragmentation_indicator, fragment_counter, etc.

Next, a specific example of header information when a file (=Item=MFU) is divided and packetized will be explained using Figures 54 and 55.

Figures 54 and 55 are diagrams showing an example in which a file is divided into multiple pieces of divided data, each of which is packetized and transmitted. Specifically, Figures 54 and 55 show information (packet sequence number, fragment counter, fragmentation indicator, MPU sequence number, item ID) contained in the data unit header, MMTP payload header, or MMTP packet header, which is the header information for each divided MMTP packet. Note that Figure 54 shows an example in which File #1 is divided into M pieces (M <= 256), and Figure 55 shows an example in which File #2 is divided into N pieces (256 < N).

The split data number indicates the index of the split data from the beginning of the file, and this information is not transmitted. In other words, the split data number is not included in the header information. The split data number is also a number that is assigned to packets corresponding to each of the multiple split data obtained by splitting a file, and is a number that is assigned by incrementing by one from the first packet in ascending order.

The packet sequence number is the sequence number of packets with the same packet ID, and in Figures 54 and 55, the first divided data in the file is designated as A, and consecutive numbers are assigned to the last divided data in the file. The packet sequence number is assigned by incrementing the divided data at the beginning of the file by one in ascending order, and corresponds to the divided data number.

The fragment counter indicates the number of split data pieces that come after the current split data piece among the multiple split data pieces obtained by splitting one file. Furthermore, if the number of split data pieces, which is the number of split data pieces obtained by splitting one file, exceeds 256, the fragment counter indicates the remainder when the number of split data pieces is divided by 256. In the example of FIG. 54, since the number of split data pieces is 256 or less, the fragment counter field value is (M-split data number). On the other hand, in the example of FIG. 55, since the number of split data pieces exceeds 256, the fragment counter field value is (N-split data number) divided by 256 ((N-split data number) % 256).

The fragmentation indicator indicates the state of division of data stored in an MMTP packet, and is a value indicating whether the data is the first divided data in a divided data unit, the last divided data, any other divided data, or one or more undivided data units. Specifically, the fragmentation indicator is "01" for the first divided data, "11" for the last divided data, "10" for the remaining divided data, and "00" for an undivided data unit.

In this embodiment, when the number of divided data pieces exceeds 256, the remainder when the number of divided data pieces is divided by 256 is shown, but the number of divided data pieces is not limited to 256 and may be another number (a specified number).

When a file is divided as shown in Figures 54 and 55, and conventional header information is added to each of the multiple divided data obtained by dividing the file and transmitted, the receiving device does not have information that can be used to derive the number of divided data in the original file (divided data number) that the data stored in the received MMTP packet is, the number of divided data in the file, or the divided data number and number of divided data. For this reason, with the conventional transmission method, even if an MMTP packet is received, it is not possible to uniquely detect the divided data number or number of divided data of the data stored in the received MMTP packet.

For example, as shown in FIG. 54, if the number of divided data pieces is 256 or less, and it is known in advance that the number of divided data pieces is 256 or less, it is possible to identify the divided data number and the number of divided data pieces by referencing the fragment counter. However, if the number of divided data pieces is 256 or more, it is not possible to identify the divided data number and the number of divided data pieces.

When limiting the number of divided data pieces of a file to 256 or less, if the data size that can be transmitted in one packet is x [bytes], the maximum size of a transmittable file is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in this case the maximum size of a transmittable file is limited to 4k * 256 = 1M [bytes]. Therefore, if you want to transmit a file that is larger than 1 [Mbytes], you cannot limit the number of divided data pieces of the file to 256 or less.

In addition, for example, the first and last divided data of a file can be detected by referring to the fragmentation indicator. Therefore, it is possible to count the number of MMTP packets until an MMTP packet containing the last divided data of a file is received, or to calculate the divided data number and the number of divided data by combining the MMTP packet with the packet sequence number after receiving an MMTP packet containing the last divided data of a file. Therefore, the divided data number and the number of divided data may be signaled by combining the fragmentation indicator and the packet sequence number. However, if reception is started from an MMTP packet containing divided data in the middle of a file (i.e., divided data that is neither the first nor the last divided data of a file), the divided data number and the number of divided data of the divided data cannot be identified. The divided data number and the number of divided data of the divided data can be identified only after an MMTP packet containing the last divided data of the file is received.

To address the issue described in Figures 54 and 55, that is, to uniquely determine the split data number and number of split data of a file when a packet containing split data of the file is received midway, the following method is used.

First, let's explain the split data number.

For the split data number, the packet sequence number in the split data at the beginning of the file (item) is signaled.

The signaling method is to store it in the control information that manages the file. Specifically, in Figures 54 and 55, the packet sequence number A of the divided data at the beginning of the file is stored in the control information. The receiving device obtains the value of A from the control information and calculates the divided data number from the packet sequence number indicated in the packet header.

The split data number of a split data is calculated by subtracting the packet sequence number A of the first split data from the packet sequence number of the split data.

An example of control information for managing files is the asset management table defined in ARIB STD-B60. The asset management table indicates the file size and version information for each file, and is stored in a data transmission message for transmission. Figure 56 shows the syntax of a loop for each file in the asset management table.

If the area of the existing asset management table cannot be expanded, signaling may be performed using a 32-bit area in part of the item_info_byte field that indicates item information. A flag indicating whether the packet sequence number in the split data at the beginning of the file (item) is indicated in a part of the item_info_byte area may be indicated in the control information, for example in the reserved_future_use field.

When repeatedly transmitting files, such as in a data carousel, multiple packet sequence numbers may be indicated, or the packet sequence number of the first file to be transmitted immediately after may be indicated.

It does not have to be the packet sequence number of the split data at the beginning of the file, but any information that links the split data number of the file with the packet sequence number will suffice.

Next, we will explain the number of split data.

The order of loops for each file included in the asset management table may be defined as the transmission order of the files. This allows the first packet sequence numbers of two consecutive files in the transmission order to be known, and the number of divided data pieces of the previously transmitted file can be determined by subtracting the first packet sequence number of the previously transmitted file from the first packet sequence number of the later transmitted file. In other words, for example, if File #1 shown in FIG. 54 and File #2 shown in FIG. 55 are consecutive files in this order, the last packet sequence number of File #1 and the first packet sequence number of File #2 are assigned consecutive numbers.

The file division method may also be specified so that the number of divided data pieces of a file can be specified. For example, if the number of divided data pieces is N, the size of each of the 1st to (N-1)th divided data pieces can be set to L, and the size of the Nth divided data piece can be specified as a fraction (item_size-L*(N-1)), thereby allowing the number of divided data pieces to be calculated backwards from item_size shown in the asset management table. In this case, the integer value obtained by rounding up (item_size/L) becomes the number of divided data pieces. Note that the file division method is not limited to this.

The number of split data items may also be stored directly in the asset management table.

By using the above method, the receiving device receives the control information and calculates the number of divided data pieces based on the control information. It can also calculate a packet sequence number corresponding to the divided data number of the file based on the control information. Note that if the timing of receiving the packets of divided data is earlier than the timing of receiving the control information, the divided data number and the number of divided data pieces may be calculated at the timing of receiving the control information.

When the divided data number or the number of divided data is signaled using the above method, the divided data number or the number of divided data is not identified based on the fragment counter, and the fragment counter becomes unnecessary data. Therefore, in the transmission of asynchronous media, when the divided data number and information that can identify the divided data number and the number of divided data are signaled using the above method, the fragment counter may not be used, or the header may be compressed. This can reduce the amount of processing by the transmitting device and the receiving device, and can also improve transmission efficiency. In other words, when transmitting asynchronous media, the fragment counter may be reserved (disabled). Specifically, the value of the fragment counter may be set to a fixed value, for example, "0". Furthermore, when receiving asynchronous media, the fragment counter may be ignored.

When storing synchronous media such as video and audio, the order in which MMTP packets are sent at the sending device and the order in which the MMTP packets arrive at the receiving device match, and packets are not resent. In such a case, if there is no need to detect packet loss and reconstruct packets, the fragment counter may not be operated. In other words, in this case, the fragment counter may be reserved (disabled).

In addition, even without using a fragment counter, it is possible to detect random access points, the beginning of an access unit, the beginning of a NAL unit, etc., and it is possible to perform decoding processing, detection of packet loss, and recovery from packet loss.

Furthermore, when transmitting real-time content such as live broadcasts, lower latency transmission is required, and data must be packetized and sent in sequence starting from the data that has been fully encoded. However, when transmitting real-time content, with conventional fragment counters, the number of divided data pieces cannot be determined when the first divided data piece is sent, and so the first divided data piece is sent after all the encoding of the data unit has been completed and the number of divided data pieces has been determined, resulting in delays. Even in such cases, this delay can be reduced by using the above method without operating a fragment counter.

Figure 57 shows the operational flow for identifying the split data number on the receiving device.

The receiving device acquires control information that describes file information (S1201). The receiving device determines whether the control information indicates the packet sequence number of the beginning of the file (S1202), and if the control information indicates the packet sequence number of the beginning of the file (Yes in S1202), calculates a packet sequence number corresponding to the divided data number of the divided data of the file (S1203). Then, after acquiring an MMTP packet in which the divided data is stored, the receiving device identifies the divided data number of the file from the packet sequence number stored in the packet header of the acquired MMTP packet (S1204). On the other hand, if the control information does not indicate the packet sequence number of the beginning of the file (No in S1202), the receiving device acquires an MMTP packet including the last divided data of the file, and identifies the divided data number using the fragment indicator and packet sequence number stored in the packet header of the acquired MMTP packet (S1205).

Figure 58 shows the operational flow for determining the number of divided data pieces in the receiving device.

The receiving device acquires control information that describes file information (S1301). The receiving device determines whether the control information includes information that allows the number of divided data pieces of the file to be calculated (S1302), and if it determines that the control information includes information that allows the number of divided data pieces to be calculated (Yes in S1302), it calculates the number of divided data pieces based on the information included in the control information (S1303). On the other hand, if the receiving device determines that the number of divided data pieces cannot be calculated (No in S1302), it acquires an MMTP packet that includes the last divided data piece of the file, and then identifies the number of divided data pieces using the fragment indicator stored in the packet header of the acquired MMTP packet and the packet sequence number (S1304).

Figure 59 shows the operational flow for determining whether to operate a fragment counter in a transmitting device.

First, the transmitting device determines whether the media to be transmitted (hereinafter also referred to as "media data") is synchronous media or asynchronous media (S1401).

If the result of the determination in step S1401 is synchronous media (synchronous media in S1402), the transmitting device determines whether the MMTP packet order of transmission and reception is consistent in the environment in which the synchronous media is transmitted and whether packet reconstruction is unnecessary in the event of packet loss (S1403). If the transmitting device determines that it is unnecessary (Yes in S1403), it does not operate a fragment counter (S1404). On the other hand, if the transmitting device determines that it is unnecessary (No in S1403), it operates a fragment counter (S1405).

If the result of the determination in step S1401 is asynchronous media (asynchronous media in S1402), the transmitting device determines whether or not to operate a fragment counter based on whether the divided data number or the number of divided data is signaled using the method described above. Specifically, if the divided data number or the number of divided data is signaled (Yes in S1406), the transmitting device does not operate a fragment counter (S1404). On the other hand, if the divided data number or the number of divided data is not signaled (No in S1406), the transmitting device operates a fragment counter (S1405).

If the transmitting device does not operate a fragment counter, it may set the fragment counter value to reserved or perform header compression.

The transmitting device may decide whether to signal the above-mentioned fragment data number and number of fragment data based on whether a fragment counter is operated.

In addition, if the synchronous media does not operate a fragment counter, the transmitting device may signal the divided data number and the number of divided data in the asynchronous media using the method described above. Conversely, the operation of the synchronous media may be determined based on whether the asynchronous media operates a fragment counter. In this case, the synchronous media and the asynchronous media can have the same operation regarding whether or not to operate fragments.

Next, a method for identifying the number of split data and the split data number (when using a fragment counter) will be described. Figure 60 is a diagram for explaining a method for identifying the number of split data and the split data number (when using a fragment counter).

As explained using FIG. 54, if the number of split data pieces is 256 or less, and it is known in advance that the number of split data pieces is 256 or less, it is possible to identify the split data number and the number of split data pieces by referring to the fragment counter.

When limiting the number of divided data pieces in a file to 256 or less, if the data size that can be transmitted in one packet is x [bytes], the maximum size of a transmittable file is limited to x * 256 [bytes]. For example, in broadcasting, x = 4k [bytes] is assumed, in this case the maximum size of a transmittable file is limited to 4k * 256 = 1M [bytes].

If the file size exceeds the maximum size of a transmittable file, the file is split in advance so that the size of each split file is x * 256 [bytes] or less. Each of the multiple split files obtained by splitting the file is treated as a single file (item) and is further split into files of 256 or less. Each of the split data obtained by further splitting is stored in an MMTP packet and transmitted.

In addition, information indicating that the item is a split file, the number of split files, and the sequence numbers of the split files may be stored in the control information and transmitted to the receiving device. This information may also be stored in the asset management table, or may be indicated using part of the existing field item_info_byte.

When the item is one of multiple split files obtained by splitting one file, the receiving device can identify the other split files and reconstruct the original file. The receiving device can also uniquely identify the number of split data items and the split data number by using the number of split files, the split file index, and the fragment counter in the control information. The receiving device can also uniquely identify the number of split data items and the split data number without using a packet sequence number, etc.

Here, it is desirable that the item_id of each of the multiple split files obtained by splitting one file is the same. Note that when a different item_id is assigned, the item_id of the first split file may be indicated in order to uniquely reference the file from other control information, etc.

In addition, multiple split files may always belong to the same MPU. When multiple files are stored in an MPU, files of different types may not be stored, and files that are split from a single file may always be stored. The receiving device can detect file updates by checking the version information for each MPU without checking the version information for each item.

Figure 61 shows the operation flow of a transmitting device when using a fragment counter.

First, the transmitting device checks the size of the file to be transmitted (S1501). Next, the transmitting device determines whether the file size exceeds x*256 [bytes] (x is the data size that can be transmitted in one packet, for example, the MTU size) (S1502). If the file size exceeds x*256 [bytes] (Yes in S1502), the transmitting device divides the file so that the size of each divided file is less than x*256 [bytes] (S1503). Then, the transmitting device transmits the divided files as items, and transmits information about the divided files (for example, the fact that they are divided files, the sequence number of the divided files, etc.) in the control information (S1504). On the other hand, if the file size is less than x*256 [bytes] (No in S1502), the transmitting device transmits the file as an item as usual (S1505).

Figure 62 shows the operation flow of a receiving device when using a fragment counter.

First, the receiving device acquires and analyzes control information related to the transmission of files, such as an asset management table (S1601). Next, the receiving device determines whether the desired item is a split file (S1602). If the receiving device determines that the desired file is a split file (Yes in S1602), it acquires information for reconstructing the file, such as the split file and its index, from the control information (S1603). Then, the receiving device acquires the items that make up the split file and reconstructs the original file (S1604). On the other hand, if the receiving device determines that the desired file is not a split file (No in S1602), it acquires the file as usual (S1605).

In short, the transmitting device signals the packet sequence number of the divided data at the beginning of the file. The transmitting device also signals information that can identify the number of divided data. Alternatively, the transmitting device specifies a division rule that can identify the number of divided data. The transmitting device also performs reserved or header compression without operating a fragment counter.

When the packet sequence number of the data at the beginning of the file is signaled, the receiving device determines the split data number and the number of split data from the packet sequence number of the split data at the beginning of the file and the packet sequence number of the MMTP packet.

From another perspective, the transmitting device divides the file and transmits the data for each divided file. It signals information linking the divided files (sequence number, number of divisions, etc.).

The receiving device identifies the split data number and the number of split data pieces using the fragment counter and the sequence number of the split file.

This allows the split data number and split data to be uniquely identified. In addition, since the split data number of a split data item can be identified when an intermediate split data item is received, waiting time can be reduced and memory usage can be reduced.

In addition, by not using a fragment counter, the configuration of the transmitting/receiving device can reduce the amount of processing and improve transmission efficiency.

Figure 63 shows a service configuration in which the same program is transmitted over multiple IP data flows. Here, an example is shown in which part of the data (video and audio) of a program with service ID = 2 is transmitted over an IP data flow using the MMT method, and data with the same service ID but different from the part of the data is transmitted over an IP data flow using the advanced BS data transmission method (the file transmission protocol is different in this example, but may be the same protocol).

The transmitting device multiplexes the IP data to ensure that data consisting of multiple IP data flows is available at the receiving device by the time of decoding.

The receiving device can achieve guaranteed receiver operation by processing data consisting of multiple IP data flows based on the decoding time.

[Supplementary Note: Transmitting Device and Receiving Device]
As described above, a transmitting device that transmits data without operating a fragment counter can also be configured as shown in Figure 64. Also, a receiving device that receives data without operating a fragment counter can also be configured as shown in Figure 65. Figure 64 is a diagram showing an example of a specific configuration of a transmitting device. Figure 65 is a diagram showing an example of a specific configuration of a receiving device.

The transmitting device 500 includes a division unit 501, a configuration unit 502, and a transmitting unit 503. Each of the division unit 501, the configuration unit 502, and the transmitting unit 503 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The receiving device 600 includes a receiving unit 601, a determining unit 602, and a configuration unit 603. Each of the receiving unit 601, the determining unit 602, and the configuration unit 603 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

Detailed explanations of each component of the transmitting device 500 and the receiving device 600 are provided in the explanations of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 66. FIG. 66 shows the operation flow (transmission method) of the transmitting device.

First, the division unit 501 of the transmitting device 500 divides the data into multiple divided data (S1701).

Next, the configuration unit 502 of the transmitting device 500 configures multiple packets by adding header information to each of the multiple divided data and packetizing them (S1702).

Then, the transmitting unit 503 of the transmitting device 500 transmits the configured multiple packets (S1703). The transmitting unit 503 transmits the divided data information and the value of the invalidated fragment counter. Note that the divided data information is information for identifying the divided data number and the number of divided data. Also, the divided data number is a number indicating which divided data the relevant divided data is among the multiple divided data. The number of divided data is the number of the multiple divided data.

This reduces the amount of processing required by the transmitting device 500.

Next, the receiving method will be explained using FIG. 67. FIG. 67 shows the operation flow (receiving method) of the receiving device.

First, the receiving unit 601 of the receiving device 600 receives multiple packets (S1801).

Next, the determination unit 602 of the receiving device 600 determines whether or not split data information has been acquired from the multiple received packets (S1802).

Then, if the determination unit 602 determines that the divided data information has been acquired (Yes in S1802), the configuration unit 603 of the receiving device 600 configures data from the multiple received packets without using the fragment counter value included in the header information (S1803).

On the other hand, if the determination unit 602 determines that the divided data information has not been acquired (No in S1802), the construction unit 603 may construct data from the multiple received packets using the fragment counter value included in the header information (S1804).

This reduces the amount of processing required by the receiving device 600.

(Embodiment 5)
[overview]
In the fifth embodiment, a method of transmitting a transport packet (TLV packet) in a case where a NAL unit is stored in a multiplexing layer in a NAL size format will be described.

As described in the first embodiment, there are two types of formats for storing H.264 or H.265 NAL units in the multiplexing layer. One is a format called a byte stream format, which adds a start code consisting of a specific bit string immediately before the NAL unit header. The other is a format called a NAL size format, which adds a field indicating the size of the NAL unit. The byte stream format is used in MPEG-2 systems and RTP, and the NAL size format is used in MP4, or DASH and MMT that use MP4.

In the byte stream format, the start code consists of three bytes, and any additional bytes (bytes whose value is 0) can be added.

On the other hand, in the general MP4 NAL size format, the size information is indicated as one byte, two bytes, or four bytes. This size information is indicated in the lengthSizeMinusOne field in the HEVC sample entry. A value of "0" in this field indicates one byte, a value of "1" indicates two bytes, and a value of "3" indicates four bytes.

Here, in ARIB STD-B60 "Media transport method using MMT in digital broadcasting," standardized in July 2014, when storing a NAL unit in the multiplexing layer, if the output of the HEVC encoder is a byte stream, the byte start code is removed and the size of the NAL unit in bytes, expressed as 32 bits (unsigned integer), is added immediately before the NAL unit as length information. Note that MPU metadata including HEVC sample entries is not transmitted, and the size information is fixed at 32 bits (4 bytes).

In addition, ARIB STD-B60 "Media Transport Method Using MMT in Digital Broadcasting" specifies that in the reception buffer model that the transmitting device takes into account during transmission to ensure buffer operation in the receiving device, the pre-decoding buffer for video signals is the CPB.

However, there are the following issues. The CPB in the MPEG-2 system and the HRD in HEVC are specified on the assumption that the video signal is in byte stream format. For this reason, for example, if transmission packet rate control is performed on the assumption that the video signal is in byte stream format with a 3-byte start code, a receiving device that receives a transmission packet in a NAL size format with a 4-byte size field added may not be able to satisfy the receiving buffer model in ARIB STD-B60. In addition, the receiving buffer model in ARIB STD-B60 does not indicate a specific buffer size or extraction rate, making it difficult to guarantee the buffer operation in the receiving device.

Therefore, to solve the above problem, the receiver buffer model for ensuring buffer operation in the receiver is specified as follows.

Figure 68 shows a receive buffer model based on the receive buffer model specified in ARIB STD-B60, specifically when only broadcast transmission paths are used.

The receiving buffer model includes a TLV packet buffer (first buffer), an IP packet buffer (second buffer), an MMTP buffer (third buffer), and a pre-decoding buffer (fourth buffer). Note that de-jitter buffers and buffers for FEC are not required in broadcast transmission paths, so they are omitted.

The TLV packet buffer receives a TLV packet (transmission packet) from the broadcast transmission path, converts the IP packet, which is composed of a variable-length packet header (IP packet header, full header when the IP packet is compressed, compressed header when the IP packet is compressed) stored in the received TLV packet and a variable-length payload, into an IP packet (first packet) having a header-expanded fixed-length IP packet header, and outputs the IP packet obtained by the conversion at a constant bit rate.

The IP packet buffer converts the IP packets into MMTP packets (second packets) having a packet header and a variable-length payload, and outputs the MMTP packets obtained by the conversion at a constant bit rate. Note that the IP packet buffer may be merged into the MMTP buffer.

The MMTP buffer converts the output MMTP packets into NAL units, and outputs the NAL units obtained by the conversion at a constant bit rate.

The pre-decoding buffer sequentially accumulates the output NAL units, generates an access unit from the accumulated multiple NAL units, and outputs the generated access unit to the decoder at the decoding time corresponding to that access unit.

The reception buffer model shown in Figure 68 is characterized in that the MMTP buffer and pre-decoding buffer, which are buffers other than the upstream TLV packet buffer and IP packet buffer, follow the reception buffer model in MPEG-2 TS.

For example, the MMTP buffer for video (MMTP B1) is composed of a buffer equivalent to the transport buffer (TB) and multiplexing buffer (MB) in MPEG-2 TS. Also, the MMTP buffer for audio (MMTP Bn) is composed of a buffer equivalent to the transport buffer (TB) in MPEG-2 TS.

The buffer size of the transport buffer is a fixed value, similar to MPEG-2 TS. For example, it is set to n times the MTU size (n can be a decimal or an integer, and is 1 or greater).

The MMTP packet size is also specified so that the overhead rate of the MMTP packet header is smaller than the overhead rate of the PES packet header. This allows the transport buffer extraction rates RX1, RXn, and RXs in MPEG-2 TS to be applied as is to the extraction rates from the transport buffer.

The size of the multiplexing buffer and the extraction rate are MPEG-2
The MB size in TS and RBX1.

In addition to the above receive buffer model, the following constraints are imposed to solve the problem.

The HRD provision in HEVC is based on the byte stream format, and MMT is a NAL size format that adds a 4-byte size field to the beginning of the NAL unit. Therefore, during encoding, rate control is performed in the NAL size format to satisfy the HRD.

In other words, the transmitting device controls the rate of transmission packets based on the above receive buffer model and constraints.

The receiving device can perform reception processing using the above signals to perform decoding operations that do not cause underflow or overflow.

Even if the size area at the beginning of the NAL unit is not 4 bytes, rate control is performed to satisfy the HRD, taking into account the size area at the beginning of the NAL unit.

The extraction rate of the TLV packet buffer (the bit rate at which the TLV packet buffer outputs IP packets) is set taking into account the transmission rate after the IP header is expanded.

In other words, a TLV packet with a variable data size is input, the TLV header is removed, and the IP header is expanded (restored), and then the transmission rate of the output IP packet is taken into consideration. In other words, the amount of header increase or decrease is taken into consideration relative to the input transmission rate.

Specifically, the transmission rate of the output IP packets is not unique because the data size is variable, packets with compressed IP headers are mixed with packets without compressed IP headers, and the size of the IP headers differs depending on the packet type (such as IPv4 or IPv6). For this reason, the average packet length of the variable data size is determined, and the transmission rate of the IP packets output from the TLV packet is determined.

Here, in order to specify the maximum transmission speed after the IP header is expanded, the transmission rate is determined assuming that the IP header is always compressed.

In addition, when IPv4 and IPv6 packet types are mixed, or when packet types are specified without distinguishing between them, the transmission rate is determined assuming IPv6 packets, which have a large header size and a large growth rate after header expansion.

For example, when the average packet length of TLV packets input to the TLV packet buffer is S, and all IP packets stored in the TLV packets are IPv6 packets and are header-compressed, the maximum output transmission rate after removing the TLV header and expanding the IP header is given by
Input rate × {S/(S + IPv6 header compression amount)}
It becomes.

More specifically, the average packet length S of the TLV packet is defined as follows:
S = 0.75 x 1500 (1500 is the maximum MTU size assumed)
is set as the standard,
IPv6 header compression amount = TLV header length - IPv6 header length - UDP header length
= 3 - 40 - 8, the maximum output transmission rate after removing the TLV header and expanding the IP header is
Input rate × 1.0417 ≒ Input rate × 1.05
It becomes.

Figure 69 shows an example of aggregating multiple data units and storing them in a single payload.

In the MMT method, when aggregating data units, a data unit length and a data unit header are added before the data unit, as shown in Figure 69.

However, for example, when a video signal in the NAL size format is stored as one data unit, as shown in FIG. 70, there are two fields indicating the size for one data unit, and the information is duplicated. FIG. 70 shows an example of aggregating and storing multiple data units in one payload, in which a video signal in the NAL size format is treated as one data unit. Specifically, the size field at the beginning of the NAL size format (hereinafter referred to as the "size field") and the data unit length field located before the data unit header in the MMTP payload header are both fields indicating the size, and the information is duplicated. For example, if the length of the NAL unit is L bytes, the size field indicates L bytes, and the data unit length field indicates L bytes + "length of the size field" (bytes). Although the values indicated in the size field and the data unit length field do not match completely, they can be said to be duplicated because one value can be easily calculated from the other value.

In this way, when data containing data size information is stored as a data unit and multiple such data units are aggregated and stored in a single payload, there is an issue that the size information is duplicated, resulting in large overhead and poor transmission efficiency.

In view of this, in a transmitting device, when data including data size information is stored as a data unit and multiple such data units are aggregated and stored in a single payload, it is possible to store them as shown in Figures 71 and 72.

As shown in FIG. 71, it is possible to store a NAL unit including a size field as a data unit, and not indicate the data unit length that is conventionally included in the MMTP payload header. FIG. 71 shows the structure of the payload of an MMTP packet in which the data unit length is not indicated.

Also, as shown in FIG. 72, a flag indicating whether the data unit length is indicated and information indicating the length of the size area may be newly stored in the header. The location where the flag and information indicating the length of the size area are stored may be indicated on a data unit basis, such as in a data unit header, or may be indicated on a unit basis in which multiple data units are aggregated (on a packet basis). FIG. 72 shows an example shown in the extend area assigned on a packet basis. Note that the storage location of the newly indicated information is not limited to this, and may be an MMTP payload header, an MMTP packet header, or control information.

On the receiving side, if the flag indicating whether the data unit length is compressed indicates that the data unit length is compressed, the length information of the size area inside the data unit is obtained, and the size area is obtained based on the length information of the size area, and the data unit length can be calculated using the length information of the obtained size area and the size area.

By using the above method, the amount of data can be reduced on the sending side, improving transmission efficiency.

In addition, overhead may be reduced by reducing the size area rather than reducing the data unit length. When reducing the size area, information indicating whether the size area has been reduced or information indicating the length of the data unit length field may be stored.

Note that the MMTP payload header also contains length information.

When a NAL unit including a size field is stored as a data unit, the payload size field in the MMTP payload header may be reduced regardless of whether aggregation is performed or not.

In addition, even if data that does not include a size field is stored as a data unit, if it is aggregated and the data unit length is indicated, the payload size field in the MMTP payload header may be reduced.

When reducing the payload size area, a flag indicating whether reduction has been performed, length information of the reduced size field, or length information of the non-reduced size field may be indicated, as described above.

Figure 73 shows the operation flow of the receiving device.

As described above, the sending device stores the NAL unit including the size field as a data unit, and the data unit length included in the MMTP payload header is not indicated in the MMTP packet.

The following describes an example in which whether the data unit length is indicated is indicated by a flag or by length information in the size field in the MMTP packet.

The receiving device determines whether the data unit includes a size field and whether the data unit length has been reduced based on the information sent from the sending side (S1901).

If it is determined that the data unit length has been reduced (Yes in S1902), the length information of the size area inside the data unit is obtained, and then the size area inside the data unit is analyzed to calculate and obtain the data unit length (S1903).

On the other hand, if it is determined that the data unit length has not been reduced (No in S1902), the data unit length is calculated as usual from either the data unit length or the size area within the data unit (S1904).

Note that if the receiving device knows in advance the flag indicating whether the data unit length has been reduced or the length information of the size area, it is not necessary to transmit it. In this case, the receiving device performs the process shown in FIG. 73 based on predetermined information.

[Supplementary Note: Transmitting Device and Receiving Device]
As described above, a transmitting device that performs rate control so as to satisfy the regulations of the receiving buffer model during encoding can also be configured as shown in Fig. 74. Also, a receiving device that receives and decodes transmission packets transmitted from a transmitting device can also be configured as shown in Fig. 75. Fig. 74 is a diagram showing an example of a specific configuration of a transmitting device. Fig. 75 is a diagram showing an example of a specific configuration of a receiving device.

The transmitting device 700 includes a generating unit 701 and a transmitting unit 702. Each of the generating unit 701 and the transmitting unit 702 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The receiving device 800 includes a receiving unit 801, a first buffer 802, a second buffer 803, a third buffer 804, a fourth buffer 805, and a decoding unit 806. Each of the receiving unit 801, the first buffer 802, the second buffer 803, the third buffer 804, the fourth buffer 805, and the decoding unit (decoder) 806 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

Detailed explanations of each component of the transmitting device 700 and the receiving device 800 are provided in the explanations of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 76. FIG. 76 shows the operation flow (transmission method) of the transmitting device.

First, the generation unit 701 of the transmitting device 700 generates an encoded stream by performing rate control so as to satisfy the requirements of a predetermined receiving buffer model in order to guarantee the buffer operation of the receiving device (S2001).

Next, the transmitting unit 702 of the transmitting device 700 packetizes the generated encoded stream and transmits the transmission packets obtained by packetization (S2002).

Note that the receiving buffer model used in the transmitting device 700 is configured with the first to fourth buffers 802 to 805 of the receiving device 800, so a description of this will be omitted.

This allows the transmitting device 700 to guarantee the buffer operation of the receiving device 800 when transmitting data using a method such as MMT.

Next, the receiving method will be explained using Figure 77. Figure 77 shows the operation flow (receiving method) of the receiving device.

First, the receiving unit 801 of the receiving device 800 receives a transmission packet consisting of a fixed-length packet header and a variable-length payload (S2101).

Next, the first buffer 802 of the receiving device 800 converts the packet, which is composed of a variable-length packet header and a variable-length payload stored in the received transmission packet, into a first packet having a header-expanded fixed-length packet header, and outputs the first packet obtained by the conversion at a constant bit rate (S2102).

Next, the second buffer 803 of the receiving device 800 converts the first packet obtained by the conversion into a second packet consisting of a packet header and a variable-length payload, and outputs the second packet obtained by the conversion at a constant bit rate (S2103).

Next, the third buffer 804 of the receiving device 800 converts the output second packet into a NAL unit, and outputs the NAL unit obtained by the conversion at a constant bit rate (S2104).

Next, the fourth buffer 805 of the receiving device 800 sequentially accumulates the output NAL units, generates an access unit from the accumulated multiple NAL units, and outputs the generated access unit to the decoder at the decoding time corresponding to the access unit (S2105).

Then, the decoding unit 806 of the receiving device 800 decodes the access unit output by the fourth buffer (S2106).

This allows the receiving device 800 to perform decoding operations without underflow or overflow.

(Embodiment 6)
[overview]
In the sixth embodiment, a transmission method and a reception method will be described when leap second adjustment is performed on the time information that serves as the reference for the reference clock in the MMT/TLV transmission method.

Figure 78 shows the protocol stack for the MMT/TLV method defined in ARIB STD-B60.

In the MMT method, packets contain data such as video and audio in multiple MPUs (Media Packets).
The MMTP packet header is added to each first data unit such as a first presentation unit (MMTP unit) or a media fragment unit (MFU), and an MMTP packet is generated as a predetermined packet (MMTP packetization). Also, an MMTP packet header is added to control information such as a control message in MMTP, and an MMTP packet is generated as a predetermined packet. The MMTP packet header is provided with a field for storing a 32-bit short format NTP (Network Time Protocol: defined in IETF RFC 5905), and can be used for QoS control of a communication line, etc.

The reference clock of the sending side (transmitting device) is synchronized with the 64-bit long format NTP defined in RFC 5905, and time stamps such as PTS (Presentation Time Stamp) and DTS (Decode Time Stamp) are added to the synchronous media based on the synchronized reference clock. Furthermore, the reference clock information of the sending side is transmitted to the receiving side, and the receiving device generates a system clock in the receiving device based on the reference clock information received from the sending side.

Specifically, the PTS and DTS are stored in the MPU timestamp descriptor and the MPU extended timestamp descriptor, which are MMTP control information, and are stored in the MP table for each asset. They are then packetized as MMTP control messages and transmitted.

The MMTP packetized data is given a UDP header or IP header and encapsulated in an IP packet. At this time, a collection of packets with the same source IP address, destination IP address, source port number, destination port number, and protocol type in the IP header or UDP header is called an IP data flow. Note that IP packets of the same IP data flow have redundant headers, so some IP packets are header compressed.

In addition, as reference clock information, a 64-bit NTP timestamp is stored in the NTP packet and then in the IP packet. At this time, in the IP packet that stores the NTP packet, the source IP address, destination IP address, source port number, destination port number, and protocol type are fixed values, and the IP packet header is not compressed.

Figure 79 shows the structure of a TLV packet.

As shown in FIG. 79, a TLV packet can include transmission control information such as an IP packet, a compressed IP packet, an AMT (Address Map Table), and an NIT (Network Information Table) as data, and these data are identified using an 8-bit data type. In addition, in a TLV packet, a data length (in bytes) is indicated using a 16-bit field, followed by the data value. In addition, a TLV packet has 1 byte of header information before the data type, and the header information is stored in a total of 4 bytes of the header area. In addition, a TLV packet is mapped to a transmission slot in an advanced BS transmission method, and the mapping information is stored in TMCC (Transmission and Multiplexing Configuration Control) control information.

Figure 80 shows an example of a block diagram of a receiving device.

In the receiving device, the demodulation means first decodes the transmission path encoded data of the broadcast signal received by the tuner, performs error correction, etc., and extracts TLV packets. Then, the TLV/IP DEMUX means performs TLV DEMUX processing and IP DEMUX processing. The TLV DEMUX processing performs processing according to the data type of the TLV packet. For example, if the TLV packet contains a compressed IP packet, the compressed header of the compressed IP packet is restored. The IP DEMUX performs processing such as header analysis of IP packets and UDP packets to extract MMTP packets and NTP packets.

The NTP clock generation means regenerates the NTP clock from the extracted NTP packets. The MMTP DEMUX performs filtering of components such as video and audio and control information based on the packet ID stored in the extracted MMTP packet header. The control information acquisition means acquires the timestamp descriptor stored in the MP table, and the PTS/DTS calculation means calculates the PTS and DTS for each access unit. The timestamp descriptor includes both the MPU timestamp descriptor and the MPU extended timestamp descriptor.

The access unit playback means converts the video and audio filtered from the MMTP packets into data in units to be presented. Specifically, data in units to be presented is the NAL unit or access unit of the video signal, audio frames, subtitle presentation units, etc. The decoding and presentation means decodes and presents the access unit at the time when the PTS/DTS of the access unit match based on the reference time information of the NTP clock.

However, the configuration of the receiving device is not limited to this.

Next, we will explain the timestamp descriptor.

Figure 81 is a diagram to explain the timestamp descriptor.

The PTS and DTS are stored in the MPU timestamp descriptor as the first control information and the MPU extended timestamp descriptor as the second control information, which are control information for MMT, and are stored in the MP table for each asset, packetized as MMTP control messages, and then transmitted.

Figure 81 (a) shows the structure of an MPU timestamp descriptor defined in ARIB STD-B60. The MPU timestamp descriptor stores, for each of multiple MPUs, presentation time information (first time information) indicating the PTS (absolute value indicated by 64-bit NTP) of the first (initial) AU (hereinafter referred to as the "first AU") in presentation order among multiple access units (AUs) serving as second data units stored in the MPU. In other words, the presentation time information of the MPU assigned to the MPU is stored in the control information of the MMTP packet and transmitted.

Figure 81 (b) shows the configuration of an MPU extended time stamp descriptor. The MPU extended time stamp descriptor stores information for calculating the PTS and DTS of AUs included in each of multiple MPUs. The MPU extended time stamp descriptor includes relative information (second time information) from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor, and the PTS and DTS of each of the multiple AUs included in the MPU can be calculated based on both the MPU time stamp descriptor and the MPU extended time stamp descriptor. In other words, the PTS and DTS of AUs other than the first AU included in the MPU can be calculated based on the PTS of the first AU stored in the MPU time stamp descriptor and the relative information stored in the MPU extended time stamp descriptor.

In other words, the second time information is relative time information for calculating the PTS or DTS of each of the multiple AUs in conjunction with the first time information. In other words, the second time information is information that indicates the PTS or DTS of each of the multiple AUs in conjunction with the first time information.

NTP is a reference time information based on Coordinated Universal Time (UTC). UTC implements leap second adjustments (hereafter referred to as "leap second adjustments") to adjust for the difference with astronomical time, which is based on the Earth's rotational speed. Specifically, leap second adjustments are carried out at 9:00 a.m. Japan time, and involve the insertion or deletion of one second.

Figure 82 is a diagram to explain leap second adjustment.

Figure 82 (a) shows an example of leap second insertion in Japan time. As shown in Figure 82 (a), when a leap second is inserted, after 8:59:59 Japan time, it becomes 8:59:59 at the timing that would normally be 9:00:00, and the 8:59:59 range is repeated twice.

Figure 82 (b) shows an example of leap second deletion in Japan time. As shown in Figure 82 (b), with leap second deletion, after 8:59:58 Japan time, the time that would normally be 8:59:59 becomes 9:00:00, and one second around 8:59:59 is deleted.

In addition to a 64-bit timestamp, an NTP packet also contains a 2-bit leap_indicator. Leap_indicator is a flag that notifies in advance that a leap second adjustment will be performed; leap_indicator = 1 indicates a leap second will be inserted, and leap_indicator = 2 indicates a leap second will be deleted. Advance notification can be given from the beginning of the month in which the leap second adjustment will be performed, 24 hours in advance, or at any other time. Leap_indicator becomes 0 when the leap second adjustment is completed (9:00:00). For example, if advance notification is given 24 hours in advance, leap_indicator will be shown to be "1" or "2" from 9:00 Japan time on the day before the leap second adjustment is made to the time immediately before the leap second adjustment is made on the day of the leap second adjustment (i.e., the first time around 8:59:59 in the case of a leap second insertion, and the time around 8:59:58 in the case of a leap second deletion).

Next, we will explain the issues involved in adjusting for leap seconds.

Figure 83 is a diagram showing the relationship between NTP time, MPU timestamp, and MPU presentation timing. Note that NTP time is the time indicated by the NTP. Also, the MPU timestamp is a timestamp indicating the PTS of the first AU in the MPU. Also, the MPU presentation timing is the timing at which the receiving device should present the MPU according to the MPU timestamp. Specifically, (a) to (c) of Figure 83 are diagrams showing the relationship between NTP time, MPU timestamp, and MPU presentation time when no leap second adjustment occurs, when a leap second is inserted, and when a leap second is deleted, respectively.

Here, we will explain an example in which the NTP time (reference clock) on the sending side is synchronized with the NTP server, and the NTP time (system clock) on the receiving side is synchronized with the NTP time on the sending side. In this case, the receiving device reproduces the time based on the timestamp stored in the NTP packet transmitted from the sending side. In this case, since both the NTP time on the sending side and the NTP time on the receiving side are synchronized with the NTP server, an adjustment of ±1 second is made when adjusting for leap seconds. Also, the NTP time in Figure 83 is assumed to be the same for the NTP time on the sending side and the NTP time on the receiving side. Note that we will explain this assuming that there is no transmission delay.

The MPU timestamp in Figure 83 indicates the timestamp of the first AU in the presentation order among multiple AUs contained in each of multiple MPUs, and is generated (set) based on the NTP time indicated at the base of the arrow. Specifically, the presentation time information of the MPU is generated by adding a predetermined time (for example, 1.5 seconds in Figure 83) to the NTP time, which serves as the reference time information at the timing of generating the presentation time information of the MPU. The generated MPU timestamp is stored in the MPU timestamp descriptor.

The receiving device presents the MPU at the MPU presentation time based on the timestamp stored in the MPU timestamp descriptor.

Note that in FIG. 83, the playback time of one MPU is described as 1 second, but the playback time of one MPU may be other playback times, for example, 0.5 seconds or 0.1 seconds.

In the example of (a) in Figure 83, the receiving device can present MPUs #1-#5 in order based on the timestamps stored in the MPU timestamp descriptor.

However, in (b) of Figure 83, the insertion of a leap second causes the presentation times of MPU #2 and MPU #3 to overlap. For this reason, if the receiving device were to present an MPU based on the timestamp stored in the MPU timestamp descriptor, there would be two MPUs to present in the same time slot around 9:00:00, and the receiving device would be unable to determine which of the two MPUs should be presented. In addition, the MPU presentation time (8:59:59) indicated in the timestamp of MPU #1 would exist twice, and the receiving device would be unable to determine which of the two MPU presentation times should be presented.

In addition, in (c) of Figure 83, the receiving device cannot present MPU #3 because the MPU presentation time (8:59:59) indicated in the MPU timestamp of MPU #3 does not exist in the NTP time due to the deletion of leap seconds.

The above problems can be solved if the receiving device performs MPU decoding and presentation processing that is not based on a timestamp. However, it is difficult for a receiving device that performs processing based on a timestamp to perform different processing (processing that is not based on a timestamp) only when a leap second occurs.

Next, we will explain how to solve the issues that arise when adjusting leap seconds by correcting the timestamp on the sending side.

Figure 84 is a diagram for explaining a method for correcting a timestamp on the sending side. Specifically, (a) of Figure 84 shows an example of inserting a leap second, and (b) of Figure 84 shows an example of deleting a leap second.

First, let's explain the case of inserting a leap second.

As shown in (a) of Figure 84, when a leap second is inserted, the time up to just before the leap second is inserted (i.e., up to the first 8:59:59 in NTP time) is area A, and the time after the leap second is inserted (i.e., after the second 8:59:59 in NTP time) is area B. Note that areas A and B are temporal areas, and are time periods or periods. The MPU timestamp in (a) of Figure 84 is the same as the timestamp described in Figure 83, and is a timestamp generated (set) based on the NTP time at the timing when the MPU timestamp is added.

This section explains in detail how to correct the timestamp on the sending side when a leap second is inserted.

The sending side (transmitting device) performs the following processing:

1. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (a) of Figure 84) is included in area A, and the MPU timestamp (the value of the MPU timestamp before correction) indicates 9:00:00 or later, the MPU timestamp is corrected by subtracting 1 second, and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp was generated based on the NTP time included in area A, and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. Note that "9:00:00" here refers to the time that corresponds to the reference time for leap second adjustment to Japan time (in other words, the time calculated by adding 9 hours to UTC time). In addition, correction information, which is information indicating that the correction has been made, is sent to the receiving device separately.

2. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (a) of Figure 84) is in area B, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area B, the MPU timestamp is not corrected.

The receiving device presents the MPU based on the MPU timestamp and correction information indicating whether the MPU timestamp has been corrected (i.e., whether it contains information indicating that the MPU timestamp has been corrected).

When the receiving device determines that the MPU timestamp has not been corrected (i.e., that it does not contain information indicating that it has been corrected), it presents the MPU at the time when the timestamp stored in the MPU timestamp descriptor matches the NTP time of the receiving device (including both before and after correction). In other words, if the MPU timestamp is stored in the MPU timestamp descriptor of an MPU that is sent before the MPU whose MPU timestamp is corrected, the MPU is presented at the time when the MPU timestamp matches the NTP time before the leap second is inserted (i.e., before the first 8:59:59). Also, if the MPU timestamp stored in the MPU timestamp descriptor of a received MPU is sent after the MPU timestamp that is corrected, the MPU is presented at the time when the MPU timestamp matches the NTP time after the leap second is inserted (i.e., after the second 8:59:59).

In addition, if the MPU timestamp stored in the MPU timestamp descriptor of the received MPU has been corrected, the receiving device presents the MPU based on the NTP time after the leap second is inserted in the receiving device (i.e., after the second 8:59:59 time).

Information indicating that the MPU timestamp value has been corrected is stored and transmitted in a control message, descriptor, table, MPU metadata, MF metadata, MMTP packet header, etc.

Next, we will explain the case of deleting leap seconds.

As shown in (b) of Figure 84, when a leap second is deleted, the time immediately before the leap second is deleted (i.e., immediately before 9:00:00 in NTP time) is designated as region C, and the time after the leap second is deleted (i.e., after 9:00:00 in NTP time) is designated as region D. Note that regions C and D are temporal regions, and are time periods or periods. The MPU timestamp in (b) of Figure 84 is the same as the timestamp described in Figure 83, and is a timestamp generated (set) based on the NTP time at the timing when the MPU timestamp is added.

This section explains in detail how to correct the timestamp on the sending side when leap seconds are deleted.

The sending side (transmitting device) performs the following processing:

1. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (b) of Figure 84) is included in the C area and the MPU timestamp (the value of the MPU timestamp before correction) indicates 8:59:59 or later, the MPU timestamp is corrected by adding 1 second, and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp is generated based on the NTP time included in the C area and indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second. Note that "8:59:59" here is the time obtained by subtracting -1 second from the time that corresponds the reference time for leap second adjustment to Japan time (i.e., the time calculated by adding 9 hours to UTC time). In addition, correction information, which is information indicating that a correction has been made, is sent to the receiving device separately. Note that in this case, the correction information does not necessarily have to be sent.

2. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (b) of Figure 84) is in area D, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area D, the MPU timestamp is not corrected.

The receiving device presents the MPU based on the MPU timestamp. If there is correction information indicating whether the MPU timestamp has been corrected, the MPU may be presented based on the MPU timestamp and the correction information.

By performing the above process, even if a leap second adjustment is made to the NTP time, the receiving device can present a normal MPU using the MPU timestamp stored in the MPU timestamp descriptor.

In addition, the timing of the MPU timestamp assignment may be signaled to notify the receiving side of whether it is in the A region, the B region, the C region, or the D region. In other words, the receiving side may be notified of whether the MPU timestamp was generated based on the NTP time included in the A region, the B region, the C region, or the D region. In other words, identification information indicating whether the MPU timestamp (presented time) was generated based on the reference time information (NTP time) before the leap second adjustment may be transmitted. This identification information is assigned based on the leap_indicator included in the NTP packet, and is information indicating whether the MPU timestamp is set based on the time between 9:00 on the day before the leap second adjustment is performed in Japan time and the time immediately before the leap second adjustment is performed on the day of the leap second adjustment (that is, the first time of 8:59:59 in the case of leap second insertion, and the time of 8:59:58 in the case of leap second deletion). In other words, the identification information is information that indicates whether the MPU timestamp was generated based on the NTP time from a time a predetermined period (e.g., 24 hours) before the time immediately before the leap second adjustment to the time immediately before that time.

Next, we will explain how to solve the issues that arise when adjusting leap seconds by correcting the timestamp on the receiving device.

Figure 85 is a diagram for explaining a method for correcting a timestamp in a receiving device. Specifically, Figure 85(a) shows an example of leap second insertion, and Figure 85(b) shows an example of leap second deletion.

First, let's explain the case of inserting a leap second.

As shown in Figure 85(a), when a leap second is inserted, as in Figure 84(a), the time up to just before the leap second is inserted (i.e., up to the first 8:59:59 in NTP time) is region A, and the time after the leap second is inserted (i.e., after the second 8:59:59 in NTP time) is region B. Note that regions A and B are temporal regions, and are time periods or periods. The MPU timestamp in Figure 85(a) is the same as the timestamp described in Figure 83, and is a timestamp generated (set) based on the NTP time at the timing when the MPU timestamp is added.

This section provides a detailed explanation of how the receiving device corrects the timestamp when a leap second is inserted.

The sending side (transmitting device) performs the following processing:

- The generated MPU timestamp is not corrected, but is stored in the MPU timestamp descriptor and transmitted to the receiving device.

- Information indicating whether the timing at which the MPU timestamp was added is in area A or area B is transmitted to the receiving device as identification information. In other words, identification information indicating whether the MPU timestamp was generated based on the NTP time contained in area A or the NTP time contained in area B is transmitted to the receiving device.

The receiving device performs the following process:

In the receiving device, the MPU timestamp is corrected based on the MPU timestamp and identification information indicating whether the timing at which the MPU timestamp was added is in area A or area B.

Specifically, the following processing is performed:

1. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (a) of Figure 85) is included in area A, and the MPU timestamp (the value of the MPU timestamp before correction) indicates 9:00:00 or later, the MPU timestamp is corrected by subtracting 1 second from the original value by -1 second. In other words, if the MPU timestamp was generated based on the NTP time included in area A, and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. Note that "9:00:00" here refers to the time that corresponds to Japan time as the reference time for leap second adjustment (in other words, the time calculated by adding 9 hours to UTC time).

2. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (a) of Figure 85) is in area B, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area B, the MPU timestamp is not corrected.

If the MPU timestamp is not corrected, the MPU is presented at the time when the MPU timestamp stored in the MPU timestamp descriptor matches the NTP time of the receiving device (including before and after correction).

In other words, for an MPU timestamp received before the MPU timestamp to be corrected, the MPU is presented at a time that matches the NTP time before the leap second is inserted (i.e., before the first 8:59:59). Also, for an MPU timestamp received after the MPU timestamp to be corrected, the MPU is presented at a time that matches the NTP time after the leap second is inserted (after the second 8:59:59).

When correcting the MPU timestamp, the corrected MPU timestamp is presented by the receiving device based on the NTP time after the leap second is inserted (i.e., after the second 8:59:59 time).

The transmitting side transmits identification information indicating whether the timing at which the MPU timestamp was added is in area A or area B by storing it in a control message, descriptor, table, MPU metadata, MF metadata, MMTP packet header, etc.

Next, we will explain the case of deleting leap seconds.

As shown in (b) of Figure 85, when a leap second is deleted, similar to (a) of Figure 84, the time up to just before the leap second is deleted (i.e., up to just before 9:00:00 in NTP time) is designated as region C, and the time after the leap second is deleted (i.e., after 9:00:00 in NTP time) is designated as region D. Note that regions C and D are temporal regions, and are time periods or periods. The MPU timestamp in (b) of Figure 85 is the same as the timestamp described in Figure 83, and is a timestamp generated (set) based on the NTP time at the timing when the MPU timestamp is added.

This section provides a detailed explanation of how to correct timestamps on a receiving device when leap seconds are deleted.

The sending side (transmitting device) performs the following processing:

- The generated MPU timestamp is not corrected, but is stored in the MPU timestamp descriptor and transmitted to the receiving device.

- Information indicating whether the timing of the MPU timestamp was added in area C or area D is transmitted to the receiving device as identification information. In other words, identification information indicating whether the MPU timestamp was generated based on the NTP time contained in area C or the NTP time contained in area B is transmitted to the receiving device.

The receiving device performs the following process:

In the receiving device, the MPU timestamp is corrected based on the MPU timestamp and identification information indicating whether the timing at which the MPU timestamp was added is in area C or area D.

Specifically, the following processing is performed:

1. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (b) of Figure 85) is included in area C, and the MPU timestamp (the value of the MPU timestamp before correction) indicates 8:59:59 or later, the MPU timestamp value is corrected by adding 1 second. In other words, if the MPU timestamp was generated based on the NTP time included in area C, and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second. Note that "8:59:59" here is the time obtained by subtracting -1 second from the time that corresponds the reference time for leap second adjustment to Japan time (in other words, the time calculated by adding 9 hours to UTC time).

2. If the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in (b) of Figure 85) is in area D, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area D, the MPU timestamp is not corrected.

The receiving device presents the MPU based on the MPU timestamp and the corrected MPU timestamp.

By performing the above process, even if a leap second adjustment is made to the NTP time, the receiving device can present a normal MPU using the MPU timestamp stored in the MPU timestamp descriptor.

Even in this case, as in the case of correcting the MPU timestamp on the transmitting side described in FIG. 84, the receiving side may be notified by signaling whether the timing of adding the MPU timestamp is in area A, area B, area C, or area D. The details of the notification are the same, so they will not be explained here.

Additional information (identification information) such as information indicating whether the MPU timestamp has been corrected or not, and information indicating whether the timing at which the MPU timestamp was added is in area A, area B, area C, or area D, as described in Figures 84 and 85, may be enabled when the leap_indicator of the NTP packet indicates the deletion (leap_indicator = 2) or insertion (leap_indicator = 1) of a leap second. It may be enabled from any predetermined time (for example, 3 seconds before the leap second adjustment), or may be enabled dynamically.

The timing at which the validity period of the additional information ends and the signaling of the additional information ends may also be synchronized with the leap_indicator of the NTP packet, may become valid from a predetermined arbitrary time (for example, 3 seconds before the leap second adjustment), or may be dynamically enabled.

Note that the 32-bit timestamp stored in the MMTP packet and the timestamp information stored in the TMCC are also generated and assigned based on the NTP time, and therefore similar problems arise. For this reason, in the case of the 32-bit timestamp and the timestamp information stored in the TMCC, the timestamp can be corrected using a method similar to that of Figures 84 and 85, and processing based on the timestamp can be performed in the receiving device. For example, when storing the above additional information for the 32-bit timestamp stored in the MMTP packet, it may be indicated using the extension area of the MMTP packet header. In this case, it is indicated that it is additional information in the extension type of the multi-header type. Also, the time when the leap_indicator is set may indicate the additional information using some bits of the 32-bit or 64-bit timestamp.

In the example of FIG. 84, the corrected MPU timestamp is stored in the MPU timestamp descriptor, but both the pre-correction and post-correction MPU timestamps (i.e., the uncorrected MPU timestamp and the corrected MPU timestamp) may be transmitted to the receiving device. For example, the pre-correction MPU timestamp descriptor and the post-correction MPU timestamp descriptor may be stored in the same MPU timestamp descriptor, or may be stored in two MPU timestamp descriptors respectively. In this case, whether the MPU timestamp is the pre-correction MPU timestamp or the post-correction MPU timestamp may be identified based on the arrangement order of the two MPU timestamp descriptors or the order of description within the MPU timestamp descriptor. Also, the pre-correction MPU timestamp may always be stored in the MPU timestamp descriptor, and when a correction is made, the post-correction timestamp may be stored in the MPU extended timestamp descriptor.

In this embodiment, the NTP time has been described as being Japan time (9:00 a.m.), but it is not limited to Japan time. Leap second adjustments are based on UTC time and are corrected simultaneously around the world. Japan time is 9 hours ahead of UTC time and is expressed as a value of (+9) the time relative to UTC time. In this way, a time adjusted to a different time depending on the time difference depending on the location may be used.

As described above, by correcting the timestamp on the sending side or on the receiving device based on information indicating the timing at which the timestamp was added, normal reception processing using the timestamp becomes possible.

Note that a receiving device that continues decoding processing from well before the leap second adjustment time may be able to perform decoding processing and presentation processing without using a timestamp, but a receiving device that tunes in just before the leap second adjustment time may not be able to determine the timestamp and may not be able to present it until after the leap second adjustment is complete. Even in such cases, the correction method in this embodiment makes it possible to perform reception processing using a timestamp, making it possible to tune in even just before the leap second adjustment time.

Figure 86 shows the operation flow of the transmitting side (transmitting device) when correcting the MPU timestamp on the transmitting side (transmitting device) as described in Figure 84, and Figure 87 shows the operation flow of the receiving device.

First, the operational flow on the sending side (sending device) will be explained using Figure 86.

When a leap second is inserted or deleted, it is determined whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D (S2201). Note that cases in which a leap second is not inserted or deleted are not shown.

Here, areas A to D are defined as follows:

Area A: The time up until just before the leap second insertion (up to the first 8:59:59 range)
Area B: Time after leap second insertion (after the second 8:59:59)
C area: The time up to just before the leap second deletion (up to 9:00:00)
D area: Time after leap second deletion (after 9:00:00)

In step S2201, if it is determined that the area is A and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second and the corrected MPU timestamp is stored in the MPU timestamp descriptor (S2202).

Then, correction information indicating that the MPU timestamp has been corrected is signaled and transmitted to the receiving device (S2203).

Also, if it is determined in step S2201 that the area is C and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second, and the corrected MPU timestamp is stored in the MPU timestamp descriptor (S2205).

Also, if it is determined in step S2201 that the area is B or D, the MPU timestamp is stored in the MPU timestamp descriptor without correction (S2204).

Next, the operation flow of the receiving device will be explained using Figure 87.

Based on the information signaled from the transmitting side, it is determined whether the MPU timestamp has been corrected (S2301).

If it is determined that the MPU timestamp has been corrected (Yes in S2301), the MPU is presented based on the MPU timestamp at the NTP time after the leap second adjustment was performed in the receiving device (S2302).

If it is determined that the MPU timestamp has not been corrected (No in S2301), the MPU is presented based on the MPU timestamp at the NTP time.

When correcting an MPU timestamp, the MPU corresponding to that MPU timestamp is presented in the section where leap seconds are inserted.

Conversely, if the MPU timestamp is not corrected, the MPU corresponding to that MPU timestamp will not be presented in an interval where leap seconds are inserted, but will be presented in an interval where leap seconds are not inserted.

Figure 88 shows the operational flow on the transmitting side when correcting the MPU timestamp in the receiving device, as described in Figure 85, and Figure 89 shows the operational flow of the receiving device.

First, the operational flow on the sending side (sending device) will be explained using Figure 88.

Based on the leap_indicator in the NTP packet, it is determined whether a leap second adjustment (insertion or deletion) has been made (S2401).

If it is determined that a leap second adjustment is to be performed (Yes in S2401), the timing for adding the MPU timestamp is determined, and the identification information is signaled and transmitted to the receiving device (S2402).

On the other hand, if it is determined that leap second adjustment is not to be performed (No in S2041), processing ends with normal operation.

Next, the operation flow of the receiving device will be explained using Figure 89.

Based on the identification information signaled by the transmitting side (transmitting device), it is determined whether the timing of the MPU timestamp assignment is in area A, area B, area C, or area D (S2501). Areas A to D are the same as those defined above, so their explanation is omitted. Note that, as with Figure 87, the case where leap seconds are not inserted or deleted is not shown.

If it is determined in step S2501 that the area is A and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second (S2502).

If it is determined in step S2501 that the area is C and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second (S2504).

If it is determined in step S2501 that the area is B or D, the MPU timestamp is not corrected (S2503).

The receiving device further presents the MPU based on the corrected MPU timestamp in a process not shown.

When correcting the MPU timestamp, the MPU corresponding to the MPU timestamp is presented based on the MPU timestamp at the NTP time after the leap second adjustment is performed.

Conversely, if the MPU timestamp is not corrected, the MPU corresponding to that MPU timestamp will not be presented in an interval where leap seconds are inserted, but will be presented in an interval where leap seconds are not inserted.

In short, the transmitting side (transmitting device) determines, for each of multiple MPUs, the timing for assigning an MPU timestamp corresponding to that MPU. If the determination shows that the timing is a time up to just before the leap second insertion and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. Correction information indicating that the MPU timestamp has been corrected is signaled and transmitted to the receiving device. If the determination shows that the timing is a time up to just before the leap second deletion and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second.

The receiving device also presents an MPU based on the MPU timestamp at the NTP time after the leap second adjustment has been performed, if it indicates that the MPU timestamp has been corrected based on whether the MPU timestamp indicated by the correction information signaled by the sending side (transmitting device) has been corrected. If the MPU timestamp has not been corrected, it presents an MPU based on the MPU timestamp at the NTP time before the leap second adjustment.

The transmitting side (transmitting device) also determines the timing for assigning an MPU timestamp corresponding to each of multiple MPUs, and signals the result of the determination. The receiving device performs the following processing based on the information indicating the timing for assigning an MPU timestamp signaled by the transmitting side. Specifically, if the information indicating the timing indicates a time up to just before the leap second insertion and the MPU timestamp indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. If the information indicating the timing indicates a time up to just before the leap second deletion and the MPU timestamp indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second.

If the MPU timestamp has been corrected, the receiving device presents the MPU based on the MPU timestamp at the NTP time after the leap second adjustment has been performed. If the MPU timestamp has not been corrected, the receiving device presents the MPU based on the MPU timestamp at the NTP time before the leap second adjustment.

In this way, by determining the timing of adding the MPU timestamp during leap second adjustment, it becomes possible to correct the MPU timestamp, and the receiving device can determine which MPU to present, enabling appropriate reception processing using the MPU timestamp descriptor and MPU extended timestamp descriptor. In other words, even if a leap second adjustment is performed on the NTP time, the receiving device can present a normal MPU using the MPU timestamp stored in the MPU timestamp descriptor.

[Supplementary Note: Transmitting Device and Receiving Device]
As described above, a transmitting device that stores data constituting an encoded stream in an MPU and transmits the stored data may be configured as shown in Fig. 90. Also, a receiving device that receives an MPU in which data constituting an encoded stream is stored may be configured as shown in Fig. 91. Fig. 90 is a diagram showing an example of a specific configuration of a transmitting device. Fig. 91 is a diagram showing an example of a specific configuration of a receiving device.

The transmitting device 900 includes a generating unit 901 and a transmitting unit 902. Each of the generating unit 901 and the transmitting unit 902 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The receiving device 1000 includes a receiving unit 1001 and a playback unit 1002. Each of the receiving unit 1001 and the playback unit 1002 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

Detailed explanations of each component of the transmitting device 900 and the receiving device 1000 are provided in the explanations of the transmitting method and the receiving method, respectively.

First, the transmission method will be described using FIG. 92. FIG. 92 shows the operation flow (transmission method) of the transmitting device.

First, the generation unit 901 of the transmitting device 900 generates presentation time information (MPU timestamp) indicating the presentation time of the MPU as the first data unit based on the NTP time as reference time information received from outside (S2601).

Next, the transmission unit 902 of the transmitting device 900 transmits the MPU, the presentation time information generated by the generation unit 901, and identification information indicating whether the presentation time information (MPU timestamp) was generated based on the NTP time before the leap second adjustment (S2602).

As a result, the receiving device that receives the information transmitted from the transmitting device 900 can regenerate the MPU based on the identification information, even if a leap second adjustment has been performed, and therefore can regenerate the MPU at the intended time.

Next, the reception method will be explained using Figure 93. Figure 93 shows the operation flow (reception method) of the receiving device.

First, the receiving unit 1001 of the receiving device 1000 receives an MPU, presentation time information indicating the presentation time of the MPU, and identification information indicating whether the presentation time information (MPU timestamp) was generated based on the NTP time before the leap second adjustment (S2701).

Next, the playback unit 1002 of the receiving device 1000 plays the MPU received by the receiving unit 1001 based on the presentation time information (MPU timestamp) and identification information received by the receiving unit 1001 (S2702).

This allows the receiving device 1000 to play the MPU at the intended time even if a leap second adjustment has been performed.

In the sixth embodiment, playback refers to a process that includes at least one of decoding and presentation.

(Modification of the Sixth Embodiment)
Next, a specific example of a specific signaling method for the additional information (identification information) explained in Figures 84 and 85 will be described.

Here, we explain a method of signaling using a combination of the MPU timestamp descriptor shown in Figure 81(a) and the MPU extended timestamp descriptor shown in Figure 81(b).

Figure 94 is a diagram showing an example of an extension of the MPU extended time stamp descriptor. The underlined fields in Figure 94 are new fields added to the MPU extended time stamp descriptor shown in Figure 81 (b).

The NTP_leap_indicator in FIG. 94 indicates a flag indicating whether or not additional information (identification information) related to NTP leap second adjustment is indicated in the MPU extended timestamp descriptor as the second control information. If this flag is set, the mpu_presentation_time_type in the loop for each of multiple MPUs becomes valid.

In FIG. 94, mpu_presentation_time_type indicates whether the mpu_presentation_time of the same sequence number described in the MPU timestamp stamp descriptor has been corrected when the correction of the MPU timestamp as described in FIG. 84 is performed on the transmitting side (transmitting device). Also, in the case where the correction of the MPU timestamp as described in FIG. 85 is performed on the receiving device, mpu_presentation_time_type indicates whether the timing at which the mpu_presentation_time of the same sequence number described in the MPU timestamp stamp descriptor is assigned is area A, area B, area C, or area D.

In other words, the MPU extended timestamp descriptor stores identification information indicating whether the MPU timestamp was generated based on time information (NTP time) before leap second adjustment. In addition, the MPU extended timestamp descriptor stores identification information corresponding to the MPU in each loop of multiple MPUs.

In step S2602, the transmitting unit 902 of the transmitting device 900 transmits the MPU, an MPU timestamp descriptor as first control information in which the presentation time information (MPU timestamp) generated by the generating unit 901 is stored, and an MPU extended timestamp descriptor as second control information in which identification information indicating whether the presentation time information was generated based on time information before leap second adjustment (NTP time) is stored.

In step S2701, the receiving unit 1001 of the receiving device 1000 receives an MPU, an MPU timestamp descriptor, and an MPU extended timestamp descriptor. The playback unit 1002 of the receiving device 1000 then plays the MPU received by the receiving unit 1001 based on the MPU timestamp descriptor and the MPU extended timestamp descriptor received by the receiving unit 1001. In this way, the receiving device 1000 can analyze both the MPU timestamp descriptor and the MPU extended timestamp descriptor to perform decoding processing using the MPU timestamp during leap second adjustment. Therefore, the receiving device 1000 can play the MPU at the intended time even if a leap second adjustment is performed. In addition, since the identification information is stored in the MPU extended timestamp descriptor, the function of signaling whether or not the MPU timestamp was generated based on time information (NTP time) before the leap second adjustment can be expanded while the MPU timestamp descriptor is configured to correspond to the existing standard (MPEG). In this way, because the existing configuration can be used, even if the signaling function is expanded, design changes can be minimized, and manufacturing costs for the transmitting side (transmitting device) and receiving device can be reduced.

The timing at which the NTP_leap_indicator is set may be the same as the leap_indicator in the NTP packet, or may be any time while the leap_indicator is set. In other words, the NTP_leap_indicator is determined to a value according to the value of the leap_indicator in the NTP packet that corresponds to the reference NTP time at which the MPU timestamp was generated.

(Modification 2 of the sixth embodiment)
The issues relating to leap second adjustment described in this embodiment are not limited to cases where time based on NTP time is used, but arise similarly even when time based on UTC time is used.

If the time system is based on UTC and the presented time information (MPU timestamp) is generated and added based on that time system, by using a method similar to that described above, it is possible to correct and process the presented time information on the receiving device side using the presented time information and signaling information (additional information, identification information) even when adjusting for leap seconds.

For example, in ARIB STD-B60, when an event message is transmitted in order to notify an application running on a receiver of the time (time designation information) at which the broadcasting station designates the operation, an event message descriptor is stored in an MMTP packet, and the time at which the event message occurs is indicated in the event message descriptor. In this case, the data constituting the application is stored in an MPU as the first data unit and transmitted. There are multiple ways to specify the time at which the event message occurs, and this is indicated by time_mode. In time_mode, when the time at which the event message occurs is expressed in UTC, identification information is signaled indicating whether the UTC time is time information assigned based on time information before the leap second adjustment or time information assigned based on time information after the leap second adjustment. In other words, the identification information is information indicating whether the presented time information was generated based on time information before the leap second adjustment (NTP time), similar to the identification information described in this embodiment. It is desirable to indicate such identification information (signaling information) in the event message descriptor. For example, this can be signaled using the reserved_future_use field in the event message descriptor.

In other words, in step S2601, the generation unit 901 of the transmitting device 900 generates time designation information indicating the application operation time based on the NTP time as the reference time information received from the outside. Then, in step S2602, the transmitting unit 902 of the transmitting device 900 transmits the MPU and the event message descriptor as control information that stores the time designation information generated by the generation unit 901 indicating the application operation time (the time when the event message occurs) and identification information indicating whether the time information is time information before the leap second adjustment.

The transmitting side (transmitting device) may determine the time periods and programs that require leap second adjustment, and may prohibit the use of time_mode expressed in UTC for those time periods and programs that require leap second adjustment, and may specify the use of time_mode that is not related to UTC.

Alternatively, the sending side (sending device) may calculate the UTC time at which the event message is generated, determine whether the time at which the event message is generated is after a leap second adjustment, and if so, calculate the UTC time after the leap second adjustment and store it in the event message descriptor. Also, it may be possible to separately signal whether the presented time information has been leap second adjusted.

Similarly, the UTC (Universal Time Coordinated)-NPT (Normal Play Time) reference descriptor indicates the relationship between UTC and NPT using the UTC_Reference field and the NPT_Reference field. It signals whether the time information indicated in the UTC_Reference field or the NPT_Reference field is time information assigned based on the reference time information before the leap second adjustment (NTP time) or time information assigned based on the reference time information after the leap second adjustment. In other words, the UTC-NPT reference descriptor is control information that stores the presented time information generated by the generation unit 901 of the transmitting device 900 and identification information indicating whether the presented time information is time information before the leap second adjustment (NTP time). The identification information can be signaled using a reserved field in the UTC-NPT reference descriptor, etc.

The method of specifying the presentation time of subtitles or superimposed text described in ARIB STD-B60 or ARIB STD-B62 is indicated by the TMD (time control mode) in the additional identification information (Additional_Arib_Subtitle_Info). The additional identification information is stored in the asset loop of the MPT.

TMD (Time Control Mode) is MPU time stamp and NPT (Normal Play
In the case of a mode in which the time is based on the MPU Time, the presentation time of the subtitles or superimposed characters is indicated in the MPU extended time stamp descriptor or the UTC-NPT reference descriptor. The signaling information allows the receiving device to correct and process the time information using the time information and signaling information even during leap second adjustment.

When the time control mode is '0000' (UTC_TTML description), the time code in the ARIB-TTML subtitles indicates the presentation time corresponding to UTC. In this case, it signals whether the UTC time indicated in the time code is time information assigned based on the time information before the leap second adjustment, or time information assigned based on the time information after the leap second adjustment. It is desirable that the signaling information sent at this time be indicated in the ARIB-TTML subtitles.

When the time control mode is '0010' (UTC_TTML description), the time is based on the UTC time indicated in the reference_start_time field in the additional identification information. In this case, the additional identification information transmits the same signaling as when the time control mode is '0000' (UTC_TTML description). In this case, the transmitting side (transmitting device) transmits, as identification information, information indicating whether the UTC time indicated in the reference_start_time field is a time from a time a predetermined period (e.g., 24 hours) before the time immediately before the leap second adjustment (e.g., the time up to 9:00:00) (i.e., 9:00:00 on the day before the leap second adjustment) to the time immediately before the leap second adjustment is performed.

In addition, the transmitting side (transmitting device) may determine the time periods and programs that require leap second adjustments, prohibit the use of the time control mode expressed in UTC, and specify that a time control mode not related to UTC should be used.

The same applies to the MH-Expire descriptor.

(Seventh embodiment)
A typical broadcasting station has multiple transmission systems, one of which is used as normal equipment for the actual broadcasting service, and the other is used as redundant equipment for backup. This allows switching to the redundant equipment when an error occurs in the service provided to viewers due to a cause in the transmission equipment, etc. Here, in the MMT/TLV method, the following phenomenon occurs when switching to the redundant equipment.

1. Adjustment of MPU sequence number As a pre-processing for switching the redundant equipment, the MPU sequence number is adjusted. As shown in Fig. 95, when the MPU sequence number is adjusted, there may be a discontinuity in the MPU sequence number before and after the adjustment. Fig. 95 shows an example in which the MPU sequence number is duplicated.

The access unit and the MPU timestamp for the access unit are the same before and after the MPU sequence number is adjusted, but because the same MPU sequence number is assigned to different MPUs, the same MPU sequence number is assigned across two MPUs.

As a result, the MPU sequence number stored in the MMTP packet header and the MPU sequence number stored in the MPU timestamp descriptor and MPU extended timestamp descriptor will overlap in different MPUs. Similarly, the MPU sequence number may skip or go back.

2. Discontinuity of packet sequence numbers Because packet sequence numbers and packet counters are different between normal equipment and redundant equipment, when switching from normal equipment to redundant equipment, packet sequence numbers and packet counters may become discontinuous. As shown in Figure 96, packet sequence numbers are continuous in normal operation, but may skip, overlap, or return. Note that Figure 96 is a diagram for explaining a case where packet sequence numbers become discontinuous at the timing of switching from normal equipment to redundant equipment.

When the MPU sequence number or packet sequence number becomes discontinuous as in 1 or 2 above, there is a problem that a receiving device that operates by believing in the continuity of these sequence numbers may malfunction.

In addition, even if discontinuities occur in the MPU sequence number or packet sequence number, the access unit and the MPU timestamp for the access unit maintain continuity, and continuous data is sent to the receiving device.

This section explains the reception process when discontinuity occurs in the MPU sequence number or packet sequence number as described using Figures 95 and 96.

If discontinuities occur in the MPU sequence numbers or packet sequence numbers, a receiving device performing processing using MPU sequence numbers or packet sequence numbers as shown below may malfunction.

An example of a process that uses the MPU sequence number is a process that detects an MPU boundary by switching the MPU sequence number in the MMTP packet header.

In addition, as a process that uses the MPU sequence number, for example, the MPU timestamp descriptor or the MPU extended timestamp descriptor describes a timestamp for the MPU sequence number, and the timestamp is used for decoding and/or presentation.

As an example of a process that uses the packet sequence number, there is a process that uses the packet sequence number to detect packet loss, and determines that a packet loss has occurred if the packet sequence number is skipped, and that a packet replacement has occurred if the packet sequence number is returned.

To prevent malfunctions caused by processing that uses these MPU sequence numbers and packet sequence numbers, the receiving device determines whether the MPU sequence numbers or packet sequence numbers are discontinuous, and if it determines that they are discontinuous, performs processing that does not use the MPU sequence numbers or packet sequence numbers.

Here, the following methods can be used to determine whether the MPU sequence number or packet sequence number is discontinuous:

First, one of the above-mentioned determination methods is to signal additional information indicating switching to redundant equipment. This allows the receiving device to perform reception processing without using the MPU sequence number or packet sequence number based on the additional information.

Methods for signaling additional information include storing and transmitting the additional information in control messages, descriptors, tables, MPU metadata, MF metadata, MMTP packet headers, and MMTP packet extension headers.

Next, we explain how the receiving device can detect discontinuities using other information.

For example, there is a method of detecting a change in the MPU sequence number in the MMTP payload header as information indicating an MPU boundary, and a method of using the RAP_flag, which is a flag in the MMTP packet header that indicates whether or not it is a random access point. In addition, it is also possible to detect an MPU boundary by a change in a packet where the Sample_number becomes 0, or a change in fragment type. It is also possible to detect an MPU boundary by counting the number of access units.

Here, if there is a possibility that switching of redundant equipment will be performed, the MPU boundary is not determined using the switching of the MPU sequence number, but the MPU boundary is determined using another method. For example, a packet with RAP_flag set to 1 is determined to be the first packet in the MPU. Here, if the MPU sequence number of the packet after the MPU boundary is the same as the MPU sequence number of the previous packet, it is determined that an MPU sequence number duplication has occurred in the MMTP packet.

Alternatively, a packet with RAP_flag set to 1 is determined to be the first packet in the MPU. Here, if the MPU sequence number of the packet determined to be the first packet skips or goes back from the MPU sequence number of the previous packet, it is determined that a discontinuity in the MPU sequence numbers has occurred in the MMTP packet.

In addition, if a discontinuity in the MPU sequence number in the MMTP payload header always occurs before a discontinuity in the MPU sequence number or packet sequence number in the MPU timestamp descriptor or MPU extended timestamp descriptor, reception processing that does not use a timestamp or processing that does not use a packet sequence number may be performed based on the result of the determination.

In addition, if a discontinuity in the MPU sequence number in the MMTP payload header of a video packet always occurs before a discontinuity in the MPU sequence number in the MMTP payload header of an audio packet, or a discontinuity in the MPU sequence number or packet sequence number in the audio MPU timestamp descriptor or MPU extended timestamp descriptor, a receiving process that does not use an audio timestamp or a process that does not use a packet sequence number may be performed based on the detection result of the discontinuity in the MPU sequence number of the video packet. The result of this determination may be determined as the timing to start switching to the redundant equipment.

The receiving device may use a method of signaling additional information indicating switching to redundant equipment, and perform processing without using the MPU sequence number or packet sequence number only when additional information indicating switching to redundant equipment is indicated.

The receiving device may perform the process of detecting discontinuity of the RAP_flag and the MPU sequence number only when additional information indicating switching to redundant equipment is indicated using a method of signaling additional information indicating switching to redundant equipment.

Here, we will explain the judgment using the timestamp descriptor in the receiving device. Normally, a timestamp is written for the MPU sequence number and is often transmitted multiple times, but if the timestamp value is updated even though the MPU sequence number is the same, the receiving device will judge that the MPU sequence number has been duplicated.

In addition, if the timestamp value is updated by the presentation time of one MPU but the MPU sequence number is updated by two or more, and there is no correspondence between the timestamp value and the MPU sequence number, the receiving device determines that a discontinuity in the MPU sequence number has occurred.

Note that when updating the MPU timestamp or MPU extended timestamp, the version in the MP table may not be updated. Also, version information may be defined separately, and the version of the timestamp information and the version of information other than the timestamp information may be sent separately. This can reduce the frequency of version updates for information other than the timestamp information, and reduce the amount of reception processing.

Note that if a discontinuity occurs in the MPU sequence number, the relationship between the MPU timestamp and the timestamp information cannot be trusted. Therefore, the fact that the timestamp information is unreliable may be signaled in the descriptor. For example, a field similar to the NTP_leap_indicator and mpu_presentation_time_type described in FIG. 94 may be used to indicate that the MPU timestamp is assigned before or after the redundant system switchover, and that the timestamp is unreliable. It may also be indicated in the MPU extended timestamp descriptor, assuming that it is used in conjunction with the MPU timestamp descriptor.

By using the above method, the receiving device can determine that a discontinuity in the MPU sequence number or packet sequence number has occurred, making it possible to perform processing without using the MPU sequence number or packet sequence number, and preventing malfunctions during synchronous playback. The following describes the operational flow of the receiving device when a discontinuity in the MPU sequence number or packet sequence number as described above occurs. Figure 97 shows the operational flow of the receiving device when a discontinuity in the MPU sequence number or packet sequence number has occurred.

The receiving device determines the MPU boundary from the packet header or payload header in the MMTP packet. Specifically, the MPU boundary is determined by determining that a packet whose RAP_flag is 1 is the first packet in the MPU. Furthermore, the MPU sequence number is determined to determine whether it has increased by 1 from the MPU sequence number of the previous packet (S2801).

If RAP_flag is 1 and it is determined that the MPU sequence number has increased by 1 (Yes in S2801), it is determined that no discontinuity in the MPU sequence number has occurred, and reception processing is performed using the MPU sequence number (S2803).

On the other hand, if it is determined that RAP_flag is 0, or that RAP_flag is 1 but the MPU sequence number has not increased by 1 from the previous packet (No in S2801), it is determined that a discontinuity in the MPU sequence number has occurred, and reception processing is performed without using the MPU sequence number (S2802).

(Embodiment 8)
In the sixth embodiment, the correction of the MPU timestamp has been mainly described, but a method for correcting the PTS or DTS of each AU included in the MPU in a receiving device will be described. That is, a correction method for correcting the timestamps indicating the PTS or DTS of each of the other AUs excluding the first AU among the AUs included in the MPU in a receiving device will be described.

Figure 98 is a diagram to explain the method for correcting the timestamp in the receiving device when a leap second is inserted. Figure 98 shows the same case as Figure 85 (a).

Here, we will explain an example in which the NTP time on the sending side is synchronized with the NTP server, and the NTP time on the receiving side is synchronized with the NTP time on the sending side by regenerating it based on the timestamp stored in the NTP packet transmitted from the sending side. In this case, both the NTP time on the sending side and the NTP time on the receiving side are adjusted by +1 second when a leap second is inserted.

The NTP time in Figure 98 is assumed to be the same for the NTP time on the sending side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. In the following, only the method for correcting the PTS of the AU in the MPU is explained, but the DTS of the AU in the MPU can also be corrected using a similar method. In the following, it is assumed that one MPU contains 5 AUs, but the number of AUs contained in an MPU is not limited to this.

As with the transmission method described in (a) of Figure 85, the NTP time up until just before the leap second insertion (up to the first 8:59:59 range) is area A, and the NTP time after the leap second insertion (after the second 8:59:59) is area B.

The sending side (transmitting device) performs the following processing:

- As explained in embodiment 6 using FIG. 83, an MPU timestamp is generated, which is the presentation time information (first time information) of the MPU.

- The generated MPU timestamp is not corrected, but is stored in an MPU timestamp descriptor and transmitted to the receiving device. In other words, the first time information is time information that is not corrected on the transmitting side of the MPU when adjusting for leap seconds. In addition, identification information indicating whether the timing at which the MPU timestamp was added is in area A or area B is transmitted to the receiving device. Specifically, an MMTP packet containing multiple MPUs is transmitted. This MMTP packet contains, as control information, an MPU timestamp descriptor containing the MPU timestamp as the first time information, an MPU extended timestamp descriptor containing relative information as the second time information, and identification information.

In Figure 98, the timing at which the MPU timestamp was added is indicated at the base of the arrow.

The receiving device performs the following process:

As described in the sixth embodiment, the MPU extended time stamp descriptor stores the second time information for calculating the PTS or DTS for each AU as relative information from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor. At this time, the relative information stored in the MPU extended time stamp descriptor does not take into account the insertion of leap seconds.

The receiving device first receives an MMTP packet including an MPU, first time information, second time information, and identification information transmitted by the transmitting side (transmitting device), and calculates the PTS (DTS) for each of all AUs contained in the MPU based on both the first time information stored in the MPU timestamp descriptor and the second time information stored in the MPU extended timestamp descriptor, as shown in "PTS for each AU" in Figure 98.

Next, the receiving device corrects the PTS for each AU. The AUs to be corrected are those in which the timing of the assignment of the MPU timestamp of the MPU to which the AU belongs (the timing indicated by the origin of the arrow in Figure 98) is in area A, and the PTS for each AU before correction is 9:00:00 or later. The AUs to be corrected are the AUs indicated by underlines in Figure 98, namely AU #5 in MPU #1, and AU #1-AU #5 in MPU #2. If the receiving device determines that the AU corresponds to an AU whose PTS is to be corrected, it corrects the PTS of the AU by -1 second.

In other words, the correction determines whether or not the MPU timestamp of the MPU storing each of the multiple AUs is generated based on the reference time information before the leap second adjustment, and whether or not the calculated PTS or DTS of the AU satisfies the correction condition that it is after a specific time (9:00:00 in the case of leap second insertion).Then, the correction corrects the PTS or DTS of the AU that is determined to satisfy the above correction condition.

AUs whose MPU timestamp assignment timing of the MPU to which they belong is in the A region and whose PTS has not been corrected (AUs up to AU#4 in MPU#1 in Figure 98) present the AU at a time that matches the NTP time before the leap second is inserted (before the first 8:59:59: time). AUs whose PTS has been corrected or subsequent AUs (AUs from AU#5 onwards in MPU#1 in Figure 98) present the AU at a time that matches the NTP time after the leap second is inserted (after the second 8:59:59 time).

Note that the identification information indicating the timing of the assignment of the MPU timestamp must be presented at least to the MPU that contains the AU to be corrected. In other words, it is not necessarily necessary to present the identification information to MPUs that do not contain the AU to be corrected.

Figure 99 is a diagram to explain the method for correcting the timestamp in the receiving device when leap seconds are deleted. Figure 99 shows the same case as Figure 85 (b).

The sending side's NTP time is synchronized with the NTP server, and the receiving side's NTP time is regenerated based on the timestamp stored in the NTP packet transmitted from the sending side, so we will explain the case where it is synchronized with the sending side's NTP time. In this case, both the sending side's NTP time and the receiving side's NTP time are adjusted by -1 second when a leap second is deleted.

The NTP time in Figure 99 is assumed to be the same for the NTP time on the sending side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. In the following, only the method for correcting the PTS of the AU in the MPU is explained, but the DTS of all AUs in the MPU can also be corrected using a similar method. In the following, it is assumed that one MPU contains 5 AUs, but the number of AUs contained in an MPU is not limited to this.

As with the transmission method described in Figure 85, the NTP time up to just before the leap second deletion (up to 8:59:58) is region C, and the NTP time after the leap second deletion (after 9:00:00) is region D.

The sending side (transmitting device) performs the following processing:

- As explained in embodiment 6 using FIG. 83, an MPU timestamp is generated, which is the presentation time information (first time information) of the MPU.

- The generated MPU timestamp is not corrected, but is stored in an MPU timestamp descriptor and transmitted to the receiving device. In other words, the first time information is time information that is not corrected on the MPU transmitting side during leap second adjustment. In addition, identification information indicating whether the timing at which the MPU timestamp was added is in area C or area D is transmitted to the receiving device. Specifically, an MMTP packet containing multiple MPUs is transmitted. The MMTP packet has the same structure as the MMTP packet described in FIG. 98.

In Figure 99, the timing at which the MPU timestamp was added is indicated at the base of the arrow.

The receiving device performs the following process:

As described in the sixth embodiment, the MPU extended time stamp descriptor stores the second time information for calculating the PTS or DTS for each AU as relative information from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor. At this time, the relative information stored in the MPU extended time stamp descriptor does not take into account the insertion of leap seconds.

The receiving device first receives an MMTP packet including an MPU, first time information, second time information, and identification information transmitted by the transmitting side (transmitting device), and calculates the PTS (DTS) for each of all AUs contained in the MPU based on both the first time information stored in the MPU timestamp descriptor and the second time information stored in the MPU extended timestamp descriptor, as shown in "PTS for each AU" in Figure 99.

Next, the receiving device corrects the PTS for each AU. The AUs to be corrected are those for which the timing of the assignment of the MPU timestamp of the MPU to which the AU belongs (the timing indicated by the origin of the arrow in Figure 99) is in area C, and for which the PTS for each AU before correction is 8:59:59 or later. The AUs to be corrected are the AUs indicated by the underlines in Figure 99, namely AU#5 in MPU#2, and AU#1-AU#5 in MPU#3. If the receiving device determines that the AU corresponds to an AU for which PTS correction is to be performed, it corrects the PTS of the AU by +1 second.

In other words, the correction determines whether or not the MPU timestamp of the MPU storing each of the multiple AUs is generated based on the reference time information before the leap second adjustment, and whether or not the calculated PTS or DTS of the AU satisfies the correction condition that it is after a specific time (8:59:59 in the case of leap second deletion).Then, the correction corrects the PTS or DTS of the AU that is determined to satisfy the above correction condition.

The receiving device can perform synchronized playback of all AUs based on the NTP time after leap second deletion and the corrected PTS.

Note that the identification information indicating the timing of the assignment of the MPU timestamp must be presented at least to the MPU that contains the AU to be corrected. In other words, it is not necessarily necessary to present the identification information to MPUs that do not contain the AU to be corrected.

Figure 100 shows the operation flow of the receiving device described in Figures 98 and 99.

Note that the operational flow on the sending side is the same as that shown in Figure 88, so a detailed explanation is omitted.

First, the receiving device calculates timestamps (PTS and DTS) for each AU using the MPU timestamp descriptor and the MPU extended timestamp descriptor (S2901).

Next, based on the identification information signaled by the transmitting side, it is determined for each AU whether the timing of the MPU timestamp assignment for the MPU to which the AU belongs is area A, area B, area C, or area D (S2902). Here, areas A to D are the same as those defined above, so their explanation is omitted. Note that, as with Figure 89, the case where leap seconds are not inserted or deleted is not shown.

If it is determined in step S2902 that the MPU timestamp of the MPU to which the AU belongs was assigned based on the NTP time of area A, it is determined whether the AU timestamp calculated in step S2901 is 9:00:0: or later (S2903).

If it is determined that the AU timestamp is 9:00:00 or later (Yes in step S2903), the AU timestamp value is corrected by -1 second (S2904).

If it is determined in step S2902 that the MPU timestamp of the MPU to which the AU belongs was assigned based on the NTP time of the C area, it is determined whether the AU timestamp calculated in step S2901 is after 8:59:59 (S2905).

If it is determined that the AU timestamp is 8:59:59 or later (Yes in step S2905), the AU timestamp value is corrected by +1 second (S2906).

On the other hand, if it is determined in step S2902 that the MPU timestamp of the MPU to which the AU belongs was assigned based on the NTP time of area B or area D, or if it is determined in step S2903 that the AU timestamp value is not 90000 or later (No in step S2903), or if it is determined in step S2905 that the AU timestamp value is not 8:59:59 or later (No in step S2905), the AU timestamp is not corrected (step S2907) and the operation of the receiving device is terminated.

As described above, when first time information indicating the absolute value of the MPU timestamp and second time information indicating a relative value to the absolute value are transmitted, as represented by the MPU timestamp descriptor and the MPU extended timestamp descriptor, the transmitting side transmits, together with the absolute value of the timestamp, identification information indicating the NTP time domain to which the absolute value of the timestamp is assigned.

The receiving device calculates the timestamps (PTS and DTS) of all AUs based on the first time information indicating the absolute value of the MPU timestamp and the second time information indicating the relative value, and then corrects the timestamps by determining whether or not to correct the timestamps based on each timestamp and the identification information indicating the NTP time region to which the absolute value of the MPU timestamp that is the basis of the timestamps is assigned.

Additional information such as information indicating whether the timestamp for each AU was corrected in the receiving device and information indicating whether the timing of the MPU timestamp of the MPU to which each AU belongs was assigned in area A, area B, area C, or area D does not need to be indicated for all MPUs, but only needs to be indicated in the section that includes the MPU that contains the AU to be corrected.

For example, if the time difference between the timestamp (PTS or DTS) in any AU and the NTP time that is the basis for assigning the MPU timestamp of the MPU to which that AU belongs is N seconds, then when a leap second is inserted or deleted, the period in which the AU to be corrected exists is at least N seconds before the leap second adjustment. Therefore, area A when a leap second is inserted, or area C when a leap second is deleted, must be an area that indicates a period of at least N seconds before the leap second adjustment.

Conversely, the time indicating area A when a leap second is inserted and area C when a leap second is deleted can be defined as M seconds, and a constraint can be imposed so that the timestamp is always assigned such that N<M.

In addition, it is not necessary to signal the B region when a leap second is inserted, or the D region when a leap second is deleted. It is sufficient to at least indicate information indicating a transition from the A region to the B region when a leap second is inserted, and information indicating a transition from the C region to the D region when a leap second is deleted.

In short, the receiving device first calculates the timestamps of all access units based on the first time information (absolute time information) and the second time information (relative time information), and for each access unit, corrects the timestamp of that access unit based on the identification information indicating whether the NTP time on which the MPU timestamp of the MPU to which the access unit belongs is calculated is before the leap second adjustment. Specifically, if the identification information indicates the time immediately before the leap second insertion and the timestamp of the access unit indicates 9:00:00 or later, the timestamp is corrected by -1 second. Also, if the identification information indicates the time immediately before the leap second deletion and the timestamp of the access unit indicates 8:59:59 or later, the timestamp is corrected by +1 second. Also, in the case of an access unit whose timestamp has been corrected, the receiving device presents that access unit based on the reference time information after the leap second adjustment has been implemented.

In this way, by determining the timing of adding the MPU timestamp of the MPU to which the access unit belongs when making a leap second adjustment, it becomes possible to correct the timestamps of all access units, enabling appropriate reception processing using the MPU timestamp descriptor and MPU extended timestamp descriptor. In other words, even if a leap second adjustment is made to the reference time information that serves as the basis for the reference clocks of the transmitting and receiving devices, multiple access units stored in the MPU can be played back at the intended time.

[Additional Notes: Transmitting/receiving system and receiving device]
As described above, a transmission/reception system including a transmission device that stores data constituting an encoded stream in an MPU and transmits the stored data, and a reception device that receives the transmitted MPU, can be configured as shown in Fig. 101. The reception device can also be configured as shown in Fig. 102. Fig. 101 is a diagram showing an example of a specific configuration of the transmission/reception system. Fig. 102 is a diagram showing an example of a specific configuration of the reception device.

The transmission/reception system 1100 includes a transmitting device 1200 and a receiving device 1300.

The transmitting device 1200 includes a generating unit 1201 and a transmitting unit 1202. Each of the generating unit 1201 and the transmitting unit 1202 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The generation unit 1201 generates first time information (MPU timestamp) indicating the presentation time of the first data unit (MPU) based on reference time information (NTP time) received from the outside.

The transmitting unit 1202 transmits the first data unit (MPU), the first time information (MPU timestamp) generated by the generating unit 1201, second time information (relative information) indicating the presentation time (PTS) or decoding time (DTS) of each of the multiple second data units (AU) using the first time information (MPU timestamp), and identification information.

The receiving device 1300 includes a receiving unit 1301, a calculating unit 1302, and a correcting unit 1303. Each of the receiving unit 1301, the calculating unit 1302, and the calculating unit 1303 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

A detailed explanation of each component of the receiving device 1300 will be provided in the explanation of the receiving method.

The receiving method will be explained using FIG. 103. FIG. 103 shows the operation flow (receiving method) of the receiving device.

First, the receiving unit 1301 of the receiving device 1300 receives a first data unit (MPU), first time information (MPU timestamp), second time information (relative information), and identification information (S3001).

Next, the calculation unit 1302 of the receiving device 1300 uses the first time information (MPU timestamp) and the second time information (relative information) received by the receiving unit 1301 to produce a presentation time (PTS) or a decoding time (DTS) for each of the multiple second data units (AUs) received by the receiving unit 1302 (S3002).

Next, the correction unit 1303 of the receiving device 1300 corrects the presentation time (PTS) or decoding time (DTS) of each of the multiple second data units (AUs) calculated by the calculation unit 1302 based on the identification information received by the receiving unit 1301 (S3003).

This allows the receiving device 1300 to properly play back the multiple second data units stored in the first data unit even if a leap second adjustment has been performed.

In the eighth embodiment, playback refers to a process that includes at least one of the processes of decoding and presenting.

(Embodiment 9)
In the eighth embodiment, a method for correcting the PTS or DTS for each AU included in an MPU in a receiving device is described, but in the ninth embodiment, a method for correcting the PTS or DTS in a transmitting device is described. That is, a correction method for correcting the timestamps indicating the PTS or DTS of each of the other AUs excluding the first AU among the multiple AUs included in an MPU in a transmitting device is described.

Figure 104 is a diagram to explain the method of correcting the timestamp on the sending side (sending device) when a leap second is inserted. Figure 104 shows the same case as Figure 84 (a).

Here, we will explain an example in which the NTP time on the sending side is synchronized with the NTP server, and the NTP time on the receiving side is synchronized with the NTP time on the sending side by regenerating it based on the timestamp stored in the NTP packet transmitted from the sending side. In this case, both the NTP time on the sending side and the NTP time on the receiving side are adjusted by +1 second when a leap second is inserted.

The NTP time in Figure 104 is assumed to be the same for the NTP time on the sending side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. In the following, only the method for correcting the PTS of the AU in the MPU is explained, but the DTS of the AU in the MPU can also be corrected using a similar method. In the following, it is assumed that one MPU contains 5 AUs, but the number of AUs contained in an MPU is not limited to this.

The sending side (transmitting device) performs the following processing:

1. As described in embodiment 6 using FIG. 83, a generation process is performed to generate an MPU timestamp, which is the presentation time information (first time information) of the MPU.

2. If the timing of the MPU timestamp assignment (the timing indicated by the origin of the arrow in Figure 104) is included in area A and the MPU timestamp (the MPU timestamp value before correction) indicates 9:00:00 or later, a correction process is performed in which the MPU timestamp value is corrected by -1 second and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp was generated based on the NTP time included in area A and indicates 9:00:00 or later, the MPU timestamp is corrected by -1 second. Note that "9:00:00" here refers to the time that corresponds to Japan time as the reference time for leap second adjustment (in other words, the time calculated by adding 9 hours to UTC time).

In addition, if the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in Figure 104) is in area B, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area B, the MPU timestamp is not corrected.

The MPU extended time stamp descriptor stores the second time information for calculating the PTS or DTS for each AU as relative information from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor. At this time, the relative information stored in the MPU extended time stamp descriptor does not take into account the insertion of leap seconds.

Therefore, in the generation process of 1 above, if the MPU timestamp as the first time information satisfies the correction condition, the MPU timestamp is corrected. Therefore, the first time information is the time information that is subject to correction when adjusting for leap seconds. The correction condition is that the MPU timestamp is generated based on the NTP time before the leap second adjustment and is after 9:00:00.

3. A transmission process is performed to transmit an MMTP packet containing multiple MPUs to a receiving device. The MMTP packet includes, as control information, an MPU timestamp descriptor including an MPU timestamp as the first time information, an MPU extended timestamp descriptor including relative information as the second time information, and identification information.

The identification information here is information that allows a receiving device to identify an MPU that includes an AU that requires correction by inserting leap seconds to the PTS or DTS when the receiving device calculates the PTS or DTS for each AU in the MPU using the MPU timestamp and relative information in the case where the MPU timestamp does not meet the correction conditions for the MPU timestamp in the generation process described above in 1 and the MPU timestamp is not corrected on the transmitting side.

In other words, the identification information is information that indicates that an AU other than the first AU (hereinafter referred to as an "AU requiring correction") may be included in the MPU, which is an AU presented in a PTS that requires correction due to a leap second adjustment being made to the reference time information, or an AU presented in a DTS that requires correction. In other words, the identification information can also be said to be information that indicates whether or not the receiving device is to determine whether or not an AU requiring correction is included in the MPU.

The identification information is, for example, information indicating whether the MPU timestamp is information that was generated based on the NTP time before the leap second adjustment in the generation process described above in 1 and has not been corrected in the correction process described above in 2. In other words, the identification information includes first identification information indicating whether the MPU timestamp was generated based on the NTP time before the leap second adjustment, and second identification information indicating that the MPU timestamp has been corrected.

In this way, when the identification information indicates that the MPU timestamp is information that was generated based on the NTP time before the leap second adjustment in the generation process and has not been corrected in the correction process, it indicates that an AU that needs correction may be included in the MPU. Conversely, when the identification information indicates that the MPU timestamp is information that was not generated based on the NTP time before the leap second adjustment in the generation process or has been corrected in the correction process, it indicates that an AU that needs correction is not likely to be included in the MPU (i.e., the receiving device is not allowed to determine whether or not an AU that needs correction is included in the MPU).

The processing on the sending side (sending device) is explained in detail using Figure 104.

In the example shown in Figure 104, MPU #2 meets the correction condition in the correction process 2 above, and as underlined, the MPU timestamp of MPU #2 is corrected by -1 second on the transmitting side, while the MPU timestamps of the other MPUs are not corrected. The MPU timestamp calculated as above is shown in the MPU timestamp descriptor, while the MPU extended timestamp descriptor (not shown) shows the PTS (DTS) for each AU in the MPU in question using relative information from the MPU timestamp, as in the conventional case.

Also, MPU#1 in Figure 104 is an MPU that includes an AU that requires correction. Here, Figure 104 shows the PTS for each AU calculated based on the MPU timestamp descriptor and the MPU extended timestamp descriptor. Although the PTS for AU#5 in MPU#1 is calculated to be 9:00:00.1, it is actually an AU that should be presented at around 8:59:59 for the second time after the leap second is inserted, and a correction of -1 second is required. In other words, AU#5 in MPU#1 is an AU that requires correction.

As described above, in the transmitting device, an MPU that satisfies the conditions that the timing of the assignment of the MPU timestamp (the timing indicated by the origin of the arrow in FIG. 104) is in the A region and does not satisfy the correction condition (i.e., the MPU timestamp before correction is before 9:00:00 as a specified time, and at least one of the PTS (DTS) values of all AUs in the MPU calculated using the MPU timestamp before correction and relative information is after 9:00:00) can be determined to be an MPU that contains an AU that needs correction, and information indicating that the MPU contains an AU that needs correction is signaled from the transmitting device to the receiving device. In other words, the MPU can be said to be an MPU that contains both an AU that should be presented at 8:59:59 for the first time and an AU that should be presented at 8:59:59 for the second time when the MPU timestamp is not corrected.

The receiving device performs the following process:

1. Performs reception processing to receive MMTP packets sent by the sender (transmitter).

2. A calculation process is performed to calculate the PTS (DTS) of all AUs based on the MPU timestamp descriptor and MPU extended timestamp descriptor contained in the control information of the MMTP packet.

3. Based on the identification information signaled by the transmitting side (transmitting device) indicating whether an MPU may contain an AU that requires correction, a determination process is performed to determine whether an MPU that is indicated by the identification information to be likely to contain an AU that requires correction contains an AU that requires correction. Specifically, it is determined whether the PTS (DTS) of all AUs contained in the MPU is 9:00:00 or later, and an AU with a PTS (DTS) of 9:00:00 or later is determined to be an AU that requires correction, thereby determining whether the MPU contains an AU that requires correction. Then, a -1 second correction is performed on the PTS (DTS) of the AU that requires correction.

The processing of the receiving device is explained in detail using Figure 104.

In FIG. 104, the transmitting side signals that MPU#1 is an MPU that may contain an AU that requires correction, so AU#5, whose PTS calculated based on the timestamp descriptor in MPU#1 indicates 9:00:00 or later, becomes the target of correction, and a -1 second correction is performed. As a result, after correction, the PTS of AU#5 in MPU#1 becomes 8:59:59.1.

Note that if the transmitting side signals that an MPU is unlikely to contain an AU that requires correction, the receiving device does not perform the above-mentioned judgment process. Therefore, for the multiple AUs stored in the MPU, the PTS (DTS) of the multiple AUs calculated based on the MPU timestamp descriptor and the MPU extended timestamp descriptor are not corrected. By doing the above, the PTS (DTS) of all AUs can be calculated.

When the receiving device presents or decodes using a timestamp corrected by either the transmitting device or the receiving device based on information indicating whether the MPU timestamp was corrected on the transmitting side or whether the PTS (DTS) was corrected on the receiving device, the receiving device always presents or decodes based on the NTP time after the leap second insertion (after the second 8:59:59 time). On the other hand, when presenting or decoding using a timestamp before correction, the receiving device presents or decodes based on the NTP time before the leap second insertion (before the first 8:59:59 time). In the example of FIG. 104, the AUs up to AU#4 in MPU#1 are presented based on the NTP time before the leap second insertion (before the first 8:59:59 time), and the AUs from AU#5 onwards in MPU#1 are presented based on the NTP time after the leap second insertion (after the second 8:59:59 time).

In this embodiment, it has been explained that on the transmitting side, the MPU extended timestamp descriptor stores a relative value from the MPU timestamp without taking leap second insertion into account as in the past. However, it is also possible to take into account the correction due to leap second insertion and store a relative value after correcting the timestamp of AU #5 in MPU #1 by -1 second. At this time, it is separately signaled that the AU in question has been corrected and is the AU to be presented or decoded based on the NTP time after the leap second insertion (after the second 8:59:59 seconds).

The receiving device calculates the PTS or DTS of all AUs based on the MPU timestamp descriptor and the MPU extended timestamp descriptor. At this time, since the relative values have already been corrected, there is no need for the receiving device to correct the timestamps of the AUs, and all AUs can be presented or decoded based on information indicating whether the MPU timestamp has been corrected and information indicating whether the PTS or DTS of the AUs in the MPU has been corrected.

Figure 105 is a diagram to explain the method of correcting the timestamp on the sending side (sending device) when leap seconds are deleted. Figure 105 shows the same case as Figure 84 (b).

Here, we will explain an example in which the sending side's NTP time is synchronized with the NTP server, and the receiving side's NTP time is synchronized with the sending side's NTP time by regenerating it based on the timestamp stored in the NTP packet transmitted from the sending side. In this case, both the sending side's NTP time and the receiving side's NTP time are adjusted by -1 second when a leap second is inserted.

The NTP time in Figure 105 is assumed to be the same for the NTP time on the sending side and the NTP time on the receiving side. In the following, it is assumed that there is no transmission delay. In the following, only the method for correcting the PTS of the AU in the MPU is explained, but the DTS of the AU in the MPU can also be corrected using a similar method. In the explanation of Figure 104, it is assumed that one MPU contains 5 AUs, but the number of AUs contained in an MPU is not limited to this.

The sending side (transmitting device) performs the following processing:

1. As described in embodiment 6 using FIG. 83, a generation process is performed to generate an MPU timestamp, which is the presentation time information (first time information) of the MPU.

2. If the timing of the MPU timestamp assignment (the timing indicated by the origin of the arrow in Figure 105) is included in the C area and the MPU timestamp (the MPU timestamp value before correction) indicates 8:59:59 or later, a correction process is performed in which the MPU timestamp value is corrected by +1 second and the corrected MPU timestamp is stored in the MPU timestamp descriptor. In other words, if the MPU timestamp was generated based on the NTP time included in the C area and indicates 8:59:59 or later, the MPU timestamp is corrected by +1 second. Note that "8:59:59" here is the time obtained by subtracting -1 second from the time that corresponds the reference time for leap second adjustment to Japan time (in other words, the time calculated by adding 9 hours to UTC time).

In addition, if the timing at which the MPU timestamp is assigned (the timing indicated by the origin of the arrow in Figure 105) is in area D, the MPU timestamp is not corrected. In other words, if the MPU timestamp is generated based on the NTP time included in area D, the MPU timestamp is not corrected.

In addition, the MPU extended time stamp descriptor stores the second time information for calculating the PTS or DTS for each AU as relative information from the PTS of the first AU of the MPU stored in the MPU time stamp descriptor. At this time, the relative information stored in the MPU extended time stamp descriptor does not take into account the deletion of leap seconds.

Therefore, in the generation process of 1 above, if the MPU timestamp as the first time information satisfies the correction condition, the MPU timestamp is corrected. Therefore, the first time information is the time information that is subject to correction when adjusting for leap seconds. The correction condition is that the MPU timestamp is generated based on the NTP time before the leap second adjustment and is 8:59:59 or later.

3. A transmission process is performed to transmit an MMTP packet containing multiple MPUs to a receiving device. The MMTP packet includes, as control information, an MPU timestamp descriptor including an MPU timestamp as the first time information, an MPU extended timestamp descriptor including relative information as the second time information, and identification information.

The identification information here is information that allows a receiving device to identify an MPU that includes an AU that requires correction by removing leap seconds for the PTS or DTS when the receiving device calculates the PTS or DTS for each AU in the MPU using the MPU timestamp and relative information in a case where the MPU timestamp does not meet the correction conditions for the MPU timestamp in the generation process described above in 1 and the MPU timestamp is not corrected on the transmitting side.

The identification information here is the same as the identification information explained in leap second insertion, so we will not explain it here.

The processing on the sending side (sending device) is explained in detail using Figure 105.

In the example shown in Figure 105, MPU #3 meets the condition for correcting the MPU timestamp in the correction process above 2, and as underlined, the transmitting side corrects the MPU timestamp of MPU #3 by +1 second, while the MPU timestamps of the other MPUs are not corrected. The MPU timestamp calculated as above is shown in the MPU timestamp descriptor, while the MPU extended timestamp descriptor (not shown) shows the PTS (DTS) for each AU in the MPU as relative information from the MPU timestamp, as in the conventional case.

Also, MPU #2 in Figure 105 is an MPU that includes an AU that requires correction.

Here, 105 shows the PTS for each AU calculated based on the MPU timestamp descriptor and the MPU extended timestamp descriptor. Although the PTS for AU#5 in MPU#2 is calculated as 8:59:59.1, it is actually an AU that should be presented in the 9:00:00 range after the leap second has been deleted, and a correction of +1 second is required. In other words, AU#5 in MPU#2 is an AU that requires correction.

As described above, in the transmitting device, an MPU that satisfies the conditions that the timing of the assignment of the MPU timestamp (the timing indicated by the origin of the arrow in FIG. 105) is in the C region and does not satisfy the correction condition (i.e., the MPU timestamp before correction is before the specified time of 8:59:59, and at least one of the PTS (DTS) values of all AUs in the MPU calculated using the MPU timestamp before correction and relative information is after 8:59:59) can be determined to be an MPU that contains an AU that needs correction, and information indicating that the MPU contains an AU that needs correction is signaled from the transmitting device to the receiving device. In other words, the MPU in question can be said to be an MPU that contains both an AU that should be presented around 8:59:58 before the leap second deletion and an AU that should be presented around 9:00:00 after the leap second deletion when the MPU timestamp is not corrected.

The receiving device performs the following process:

1. Performs reception processing to receive MMTP packets sent by the sender (transmitter).

2. A calculation process is performed to calculate the PTS (DTS) of all AUs based on the MPU timestamp descriptor and MPU extended timestamp descriptor contained in the control information of the MMTP packet.

3. Based on the identification information signaled by the transmitting side (transmitting device) indicating whether an MPU may contain an AU that requires correction, a determination process is performed to determine whether an MPU that is indicated by the identification information to possibly contain an AU that requires correction contains an AU that requires correction. Specifically, it is determined whether the PTS (DTS) of all AUs contained in the MPU is 8:59:59 or later, and an AU with a PTS (DTS) of 8:59:59 or later is determined to be an AU that requires correction, thereby determining whether the MPU contains an AU that requires correction. Then, a +1 second correction is performed on the PTS (DTS) of the AU that requires correction.

In FIG. 105, the transmitting side signals that MPU#2 is an MPU that may contain an AU that requires correction, so AU#5, whose PTS calculated based on the timestamp descriptor in MPU#2 indicates 8:59:59 or later, becomes the target of correction, and a +1 second correction is performed. As a result, after correction, the PTS of AU#5 in MPU#2 becomes 9:00:00.1.

Note that if the transmitting side signals that an MPU is unlikely to contain an AU that requires correction, the receiving device does not perform the above-mentioned judgment process. Therefore, for the multiple AUs stored in the MPU, the PTS (DTS) of the multiple AUs calculated based on the MPU timestamp descriptor and the MPU extended timestamp descriptor are not corrected. By doing the above, the PTS (DTS) of all AUs can be calculated.

The receiving device can present or decode the AU using the corrected PTS (DTS).

In this embodiment, it has been explained that on the transmitting side, the MPU extended timestamp descriptor stores a relative value from the MPU timestamp without taking leap second deletion into account as in the past, but it is also possible to take into account the correction due to leap second deletion and store a relative value after correcting the timestamp of AU#5 in MPU#1 by +1 second.

The receiving device calculates the PTS or DTS of all AUs based on the MPU timestamp descriptor and the MPU extended timestamp descriptor. At this time, since the relative values have already been corrected, the receiving device does not need to correct the AU timestamps, and all AUs can be presented or decoded.

The transmitting side (transmitting device) may be able to select which correction method to use from among the correction methods for leap second insertion or leap second deletion described above. When the transmitting side (transmitting device) selects a correction method, it signals the receiving device which method it used for correction. Based on the signaled information, the receiving device can switch the correction in the receiving device or the presentation operation or decoding operation.

Figure 106 is a diagram showing the operation flow of the transmitting device described in Figure 104. Figure 107 is a diagram showing the operation flow of the receiving device described in Figure 104.

In FIG. 106, the transmitting device determines whether the timing of the MPU's timestamp assignment is in area A or area B (S3101). Here, area A and area B are the same as those defined above, so the explanation is omitted. Note that, as with FIG. 89, the case where leap seconds are not inserted is not shown.

If it is determined in step S3101 that it is area B, the MPU timestamp is not corrected (S3102) and the process ends.

On the other hand, if it is determined in step S3101 that the area is A, it is determined whether the MPU timestamp is after 9:00:00 seconds (S3103).

If it is determined that the MPU timestamp is 9:00:00 or later (Yes in step S3103), the MPU timestamp is corrected by -1 second, the receiving device is signaled that the MPU timestamp has been corrected (S3104), and the processing ends.

On the other hand, if it is determined that the MPU timestamp is not after 9:00:00 (No in step S3103), the timestamps (PTS and DTS) of all AUs contained in the MPU are calculated using the MPU timestamp descriptor and the MPU extended timestamp descriptor (S3105).

Next, it is determined whether the MPU includes an AU whose calculated timestamp is after 9:00:00 (S3106).

If it is determined that the calculated timestamp includes an AU that is after 9:00:00 (Yes in S3106), the receiving device signals that the timestamp of the AU in the MPU needs to be corrected (S3107).

On the other hand, if it is determined that the MPU does not include an AU whose calculated timestamp is after 9:00:00 (No in S3106), the processing is terminated.

In FIG. 107, the receiving device calculates timestamps for all AUs using the MPU timestamp descriptor and the MPU extended timestamp descriptor (S3201).

Next, based on the signaled identification information, it is determined whether or not the timestamp of the AU in the MPU needs to be corrected (S3202).

If it is determined that the timestamp of an AU in the MPU needs to be corrected (Yes in step S3202), the timestamp of the AU indicating a timestamp after 9:00:00 is corrected by -1 second (S3205).

On the other hand, if it is determined that the timestamp of the AU in the MPU does not need to be corrected (No in step S3202), it is determined whether the MPU timestamp has been corrected based on the signaled identification information (S3203).

If it is determined that the MPU timestamp has not been corrected (No in step S3203), for timestamps before the timestamp to be corrected, the AU is presented based on the NTP time before the leap second insertion, and for timestamps after the timestamp to be corrected, the AU is presented based on the NTP time after the leap second insertion (S3204).

After step S3205, or if it is determined in step S3203 that the timestamp has been corrected (Yes in step S3203), the AU is presented based on the NTP time after the leap second insertion has been performed (S3206).

Figure 108 shows the operation flow of the transmitting device described in Figure 105. Figure 109 shows the operation flow of the receiving device described in Figure 105.

In FIG. 108, the transmitting device determines whether the timing of the MPU's timestamp assignment is in region C or region D (S3301). Here, region C and region D are the same as those defined above, so the explanation is omitted. Note that, as with FIG. 89, the case where leap seconds are not deleted is not shown.

If it is determined in step S3301 that the area is D, the MPU timestamp is not corrected (S3302) and the process ends.

On the other hand, if it is determined in step S3301 that the area is C, it is determined whether the MPU timestamp is after 8:59:59 (S3303).

If it is determined that the MPU timestamp is 8:59:59 or later (Yes in step S3303), the MPU timestamp is corrected by +1 second (S3304) and the process ends.

On the other hand, if it is determined that the MPU timestamp is not 8:59:59 or later (No in step S3303), the timestamps (PTS and DTS) of all AUs contained in the MPU are calculated using the MPU timestamp descriptor and the MPU extended timestamp descriptor (S3305).

Next, it is determined whether the calculated timestamp includes an AU after 8:59:59 (S3306).

If the calculated timestamp is after 8:59:59 but is determined to include an AU (Yes in S3306), the receiving device signals that the timestamp of the AU in the MPU needs to be corrected (S3307).

On the other hand, if it is determined that the MPU does not include an AU whose calculated timestamp is after 8:59:59 (No in S3306), the process ends.

In FIG. 109, the receiving device calculates timestamps for all AUs using the MPU timestamp descriptor and the MPU extended timestamp descriptor (S3401).

Next, based on the signaled identification information, it is determined whether or not the timestamp of the AU in the MPU needs to be corrected (S3402).

If it is determined that the timestamp of an AU in the MPU needs to be corrected (Yes in step S3402), the timestamp of the AU whose timestamp indicates 8:59:59 or later is corrected by +1 second (S3403).

After step S3403, or if it is determined in step S3402 that correction of the timestamp of the AU in the MPU is not necessary (No in step S3402), the AU is presented based on the NTP time.

According to this, (i) an MPU, (ii) an MPU timestamp, and (iii) identification information indicating that a second data unit is presented at a presentation time that requires correction or is decoded at a decode time that requires correction, and that a second data unit other than the first second data unit in the presentation order among the multiple second data units may be included in the first data unit, are transmitted. Therefore, even if leap second adjustment is performed on the reference time information that is the basis for the reference clocks of the transmitting side and receiving device, the receiving device can play back the multiple second data units stored in the first data unit at the intended time.

[Supplementary Note: Transmitting Device and Receiving Device]
As described above, a transmitting device that stores data constituting an encoded stream in an MPU and transmits the stored data may be configured as shown in Fig. 110. Also, a receiving device that receives an MPU in which data constituting an encoded stream is stored may be configured as shown in Fig. 111. Fig. 110 is a diagram showing an example of a specific configuration of a transmitting device. Fig. 111 is a diagram showing an example of a specific configuration of a receiving device.

The transmitting device 1400 includes a generating unit 1401 and a transmitting unit 1402. Each of the generating unit 1401 and the transmitting unit 1402 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

The receiving device 1500 includes a receiving unit 1501 and a determining unit 1502. Each of the receiving unit 1501 and the determining unit 1502 is realized, for example, by a microcomputer, a processor, or a dedicated circuit.

Detailed explanations of each component of the transmitting device 1400 and the receiving device 1500 are provided in the explanations of the transmitting method and the receiving method, respectively.

First, the transmission method will be described with reference to FIG. 112. FIG. 112 shows the operation flow (transmission method) of the transmitting device.

First, the generating unit 1401 of the transmitting device 1400 generates first time information indicating the presentation time of the MPU as the first data unit based on reference time information received from the outside (S3501).

Next, the transmitting unit 1402 of the transmitting device 1400 transmits the first data unit (MPU), the first time information (MPU timestamp), and the identification information (S3502). Here, the first time information (MPU timestamp) is information generated by the generating unit 1401. The identification information is information indicating that the second data unit (access unit) is presented at a presentation time (PTS) that needs to be corrected, or is decoded at a decoding time (DTS) that needs to be corrected, and that a second data unit (access unit) other than the first second data unit (access unit) in the presentation order among the multiple second data units (access units) may be included in the first data unit (MPU).

As a result, the receiving device that receives the information transmitted from the transmitting device 1400 can regenerate the AU in the MPU based on the identification information, even if a leap second adjustment has been performed, and therefore can regenerate the AU in the MPU at the intended time.

Next, the receiving method will be explained using FIG. 113. FIG. 113 shows the operation flow (receiving method) of the receiving device.

First, the receiving unit 1501 of the receiving device 1500 receives a first data unit (MPU), first time information (MPU timestamp), and identification information (S3601). The first time information and identification information are the same as the first time information and identification information described for the transmitting device 1400.

Next, the judgment unit 1502 of the receiving device 1500 judges whether or not the first data unit (MPU) in which the identification information received by the receiving unit 1501 indicates that another second data unit (access unit) may be included (S3602).

This allows the receiving device 1500 to play the AU in the MPU at the intended time even if a leap second adjustment is performed.

Other Embodiments
Although the transmitting device, receiving device, transmitting method and receiving method according to the embodiment have been described above, the present invention is not limited to these embodiments.

Furthermore, each processing unit included in the transmitting device and receiving device according to the above-mentioned embodiment is typically realized as an LSI, which is an integrated circuit. These may be individually implemented as single chips, or may be integrated into a single chip to include some or all of them.

In addition, the integrated circuit is not limited to LSI, but may be realized by a dedicated circuit or a general-purpose processor. It is also possible to use an FPGA (Field Programmable Gate Array) that can be programmed after LSI manufacturing, or a reconfigurable processor that can reconfigure the connections and settings of circuit cells inside the LSI.

In each of the above embodiments, each component may be configured with dedicated hardware, or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

In other words, the transmitting device and the receiving device include a processing circuit and a storage device electrically connected to the processing circuit (accessible from the processing circuit). The processing circuit includes at least one of dedicated hardware and a program execution unit. In addition, when the processing circuit includes a program execution unit, the storage device stores a software program to be executed by the program execution unit. The processing circuit uses the storage device to execute the transmitting method or the receiving method according to the above embodiment.

Furthermore, the present invention may be the above software program, or a non-transitory computer-readable recording medium on which the above program is recorded. Needless to say, the above program can be distributed via a transmission medium such as the Internet.

Furthermore, all the numbers used above are examples to specifically explain the present invention, and the present invention is not limited to the exemplified numbers.

The division of functional blocks in the block diagram is one example, and multiple functional blocks may be realized as one functional block, one functional block may be divided into multiple blocks, or some functions may be transferred to other functional blocks. In addition, the functions of multiple functional blocks having similar functions may be processed in parallel or in a time-sharing manner by a single piece of hardware or software.

The order in which the steps included in the above-mentioned transmission method or reception method are executed is merely an example to specifically explain the present invention, and the steps may be executed in an order other than the above. Also, some of the steps may be executed simultaneously (in parallel) with other steps.

The transmitting device, receiving device, transmitting method, and receiving method according to one or more aspects of the present invention have been described above based on the embodiments, but the present invention is not limited to these embodiments. As long as they do not deviate from the spirit of the present invention, various modifications conceived by those skilled in the art to the present embodiments, and forms constructed by combining components of different embodiments, may also be included within the scope of one or more aspects of the present invention.

The present invention can be applied to devices or equipment that transport media such as video data and audio data.

15, 100, 300, 500, 700, 900, 1200, 1400 Transmitting device 16, 101, 301 Encoding unit 17, 102 Multiplexing unit 18, 104 Transmitting unit 20, 200, 400, 600, 800, 1000, 1300, 1500 Receiving device 21 Packet filtering unit 22 Transmission order type discrimination unit 23 Random access unit 24, 212 Control information acquisition unit 25 Data acquisition unit 26 Calculation unit 27 Initialization information acquisition unit 28, 206 Decoding command unit 29, 204A, 204B, 204C, 204D, 402 Decoding unit 30 Presentation unit 201 Tuner 202 Demodulation unit 203 Demultiplexing unit 205 Display unit 211 Type discrimination unit 213 Slice information acquisition unit 214 Decoded data generation unit 302 Addition unit 303, 503, 702, 902, 1202, 1402 Transmission unit 401, 601, 801, 1001, 1301, 1501 Reception unit 501 Division unit 502, 603 Composition unit 602 Determination unit 701, 901, 1201, 1401 Generation unit 802 First buffer 803 Second buffer 804 Third buffer 805 Fourth buffer 806 Decoding unit 1002 Reproduction unit 1302 Calculation unit 1303 Correction unit 1502 Determination unit

Claims (4)

A transmission method for storing data constituting an encoded stream in a predetermined data unit and transmitting the data, comprising the steps of:
generating presentation time information indicating a presentation time of the predetermined data unit based on time information received from an external device;
Transmitting the predetermined data unit, the generated presentation time information, and identification information indicating whether the presentation time information was generated based on the time information before leap second adjustment;
The identification information (i) is stored in a control message that is control information, and (ii) indicates whether the presented time information was generated based on the time information from a time that is a predetermined period before the time immediately before a leap second adjustment is performed to the time immediately before the leap second adjustment is performed,
the presentation time information is time information generated by adding a predetermined time to the time information,
The control information includes a UTC-NPT reference descriptor indicating a relationship between a Universal Time Coordinated (UTC) time and a Normal Play Time (NPT) time. A method for receiving a predetermined data unit in which data constituting an encoded stream is stored, comprising the steps of:
receiving the predetermined data unit, presentation time information indicating a presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before leap second adjustment;
Reproducing the received predetermined data unit based on the received presentation time information and the received identification information;
The identification information (i) is stored in a control message that is control information, and (ii) indicates whether the presented time information was generated based on the time information from a time that is a predetermined period before the time immediately before a leap second adjustment is performed to the time immediately before the leap second adjustment is performed,
the presentation time information is time information generated by adding a predetermined time to the time information,
The control information includes a UTC-NPT reference descriptor indicating a relationship between a Universal Time Coordinated (UTC) time and a Normal Play Time (NPT) time. A transmitting device that transmits data constituting an encoded stream by storing the data in a predetermined data unit, comprising:
a generating unit that generates presentation time information indicating a presentation time of the predetermined data unit based on time information received from an external device;
a transmission unit that transmits the predetermined data unit, the presentation time information generated by the generation unit, and identification information indicating whether the presentation time information was generated based on the time information before leap second adjustment,
The identification information (i) is stored in a control message that is control information, and (ii) indicates whether the presented time information was generated based on the time information from a time that is a predetermined period before the time immediately before a leap second adjustment is performed to the time immediately before the leap second adjustment is performed,
the presentation time information is time information generated by adding a predetermined time to the time information,
The control information includes a UTC-NPT reference descriptor indicating a relationship between a Universal Time Coordinated (UTC) time and a Normal Play Time (NPT) time. A receiving device for receiving a predetermined data unit in which data constituting an encoded stream is stored, comprising:
a receiving unit that receives the predetermined data unit, presentation time information indicating a presentation time of the predetermined data unit, and identification information indicating whether the presentation time information was generated based on time information before leap second adjustment;
a reproduction unit that reproduces the predetermined data unit received by the receiving unit based on the presentation time information and the identification information received by the receiving unit,
The identification information (i) is stored in a control message that is control information, and (ii) indicates whether the presented time information was generated based on the time information from a time that is a predetermined period before the time immediately before a leap second adjustment is performed to the time immediately before the leap second adjustment is performed,
the presentation time information is time information generated by adding a predetermined time to the time information,
The control information includes a UTC-NPT reference descriptor indicating a relationship between UTC (Universal Time Coordinated) time and NPT (Normal Play Time) time.

Images