JP7685346B2 - Transmitting device and receiving device - Google Patents

Description

The present invention relates to a transmitting device and a receiving device.

Conventionally, ISO/IEC 23008-1 (hereafter, MMT (MPEG Media Transport)) and ISO/IEC 23009-1 (hereafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)) are known as standards for transmitting content such as 2D video and audio. In these standards, a timestamp descriptor indicating the absolute time at which the content is presented is defined as one piece of control information (MMT-SI (Signaling Information)).

Meanwhile, mechanisms for transmitting content such as 360° video and 3D objects have also been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which the user moves their head while in a seated position, and 6DoF, which involves viewpoint movement within the range in which the user can move freely. In such mechanisms, the positional relationship between the 360° video and the 3D objects is indicated by a scene description.

3GPP TR 26.928 V16.1.0 December 2020

In light of the above background, the inventors focused on usage scenarios in which content of different formats is combined and presented, and discovered that because the above-mentioned scene description is independent of MMT-SI, content of different formats cannot be properly synchronized.

Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a transmitting device and a receiving device that enable appropriate synchronization of content of different formats.

The transmission device disclosed includes a first processing unit that processes first control information associated with a first content that does not have freedom of viewpoint, a second processing unit that processes second control information associated with a second content that has freedom of viewpoint, and a transmission unit that transmits the first content, the first control information, the second content, and the second control information, where the first control information includes first absolute time information indicating the presentation time of the first content, and at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content.

The receiving device disclosed herein includes a receiving unit that receives a first content that does not have freedom of viewpoint, first control information associated with the first content, a second content that has freedom of viewpoint, and second control information associated with the second content, a first processing unit that processes the first control information, and a second processing unit that processes the second control information, where the first control information includes first absolute time information indicating the presentation time of the first content, and at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content.

The present invention provides a transmitting device and a receiving device that enable appropriate synchronization of content of different formats.

FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. FIG. 2 is a block diagram showing a transmitting device 100 according to the embodiment. FIG. 3 is a block diagram showing a receiving device 200 according to the embodiment. FIG. 4 is a diagram for explaining the second content according to the embodiment. FIG. 5 is a diagram for explaining the first method according to the embodiment. FIG. 6 is a diagram for explaining the first method according to the embodiment. FIG. 7 is a diagram for explaining the second method according to the embodiment. FIG. 8 is a diagram for explaining the second method according to the embodiment.

Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic, and the ratios of the dimensions may differ from the actual ones.

Therefore, specific dimensions should be determined with reference to the following explanation. Of course, there are also parts in which the dimensional relationships and ratios differ between the drawings.

[Disclosure Summary]
A transmitting device according to the disclosure summary includes a first processing unit that processes first control information associated with a first content that does not have viewpoint freedom, a second processing unit that processes second control information associated with a second content that has viewpoint freedom, and a transmitting unit that transmits the first content, the first control information, the second content, and the second control information, wherein the first control information includes first absolute time information indicating the presentation time of the first content, and at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content.

The receiving device according to the disclosure outline includes a receiving unit that receives a first content that does not have freedom of viewpoint, first control information associated with the first content, a second content that has freedom of viewpoint, and second control information associated with the second content, a first processing unit that processes the first control information, and a second processing unit that processes the second control information, where the first control information includes first absolute time information indicating the presentation time of the first content, and at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content.

In the summary of the disclosure, when focusing on a new usage scene in which a first content and a second content are presented in combination, even if the first control information and the second control information are independent, at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content, under the premise that the first control information includes first absolute time information indicating the presentation time of the first content. With such a configuration, the presentation times of the first content and the second content can be appropriately identified, and the first content and the second content can be appropriately synchronized.

[Embodiment]
(Transmission System)
A transmission system according to an embodiment will be described below. Fig. 1 is a diagram showing a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100 and a receiving device 200.

In an embodiment, the transmitting device 100 transmits a first content that does not have freedom of viewpoint and a second content that has freedom of viewpoint to the receiving device 200. Furthermore, the transmitting device 100 transmits first control information associated with the first content and second control information associated with the second content to the receiving device 200.

The first content may include at least one of 2D video and audio. The first content and the first control information may be transmitted in a first method. The first method may be a method conforming to ISO/IEC 23008-1 (hereinafter, MMT (MPEG Media Transport)). In the following, a case will be illustrated in which the first method is MMTP (MMT Protocol) conforming to MMT. The first control information may be referred to as MMT-SI (Signaling Information).

The second content may include a 360° video and a 3D object. The second content and the second control information may be transmitted in a second manner. The second content may conform to 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which a user moves their head while in a sitting position, or 6DoF, which involves viewpoint movement within the range in which a user can move freely. Since the second content has viewpoint freedom, it may include two or more 360° videos and two or more 3D objects at the same time (frame). The second control information may be referred to as a scene description.

Here, the second control information may be transmitted using the first method described above. That is, the second control information may be transmitted using the same first method (e.g., MMTP) as the first control information.

The transmission from the transmitting device 100 to the receiving device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communication network.

The transmission system may be, but is not limited to, a digital wireless transmission system. The digital wireless transmission system may be a system used for 4K or 8K satellite broadcasting. The receiving device 200 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display.

(Transmitting device)
A transmitting device according to an embodiment will be described below. Fig. 2 is a block diagram showing a transmitting device 100 according to an embodiment.

As shown in FIG. 2, the transmitting device 100 has a first processing unit 101, a second processing unit 102, and a transmitting unit 103.

The first processing unit 101 processes first control information (MMT-SI) associated with the first content that does not have viewpoint freedom. The first processing unit 101 may process the first content. That is, the first processing unit 101 may process 2D video or audio. For example, the first processing unit 101 may generate an MMTP packet by packetizing the 2D video, audio, and MMT-SI.

For example, MMTP packets are stored in IP (Internet Protocol) packets. IP packets may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).

Here, the first content is processed in units (hereinafter, MPUs; Media Processing Units) that are separated by a certain time width. An MPU includes one or more access units. An access unit may also be treated as an MFU (Media Fragment Unit). An MFU related to 2D video may be referred to as a NAL (Network Abstraction Layer) unit, and an MFU related to audio may be referred to as an MHAS (MPEG-H 3D Audio Stream) packet.

MMT-SI includes a PA (Package Access) message, and the PA message includes an MPT (MMT Package Table) that indicates a list of first contents. Furthermore, MMT-SI includes first absolute time information that indicates the presentation time of the first content. The first absolute time information may mean the presentation time of the MPU, i.e., the time of the access unit that is first presented in the MPU. The first absolute time information may be referred to as an MPU timestamp descriptor.

The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or may be a time provided by GPS (Global Positioning System). The reference time may be a time provided by an NTP (Network Time Protocol) server or a PTP (Precision Time Protocol) server.

The second processing unit 102 processes second control information (scene description) associated with the second content having freedom of viewpoint. The second processing unit 102 may process the second content. That is, the second processing unit 102 may process 360° video or 3D objects.

For example, the second processing unit 102 may generate MMTP packets by packetizing the scene description. The second processing unit 102 may apply a projective transformation such as ERP (Equirectangular projection) or cube map to the 360° video to convert the 360° video into a 2D video. The second processing unit 102 may generate metadata indicating the type of projective transformation applied to the 360° video. The second processing unit 102 may encode the 3D object in a mesh format. For the mesh format encoding, ISO/IEC 14496-16 "Animation framework extension (AFX)" may be used. The second processing unit 102 may encode the 3D object in a point cloud format. For the point cloud format encoding, ISO/IEC 23090-5 "Video-based Point Cloud Compression" may be used.

Here, the second content is compiled into one file in units separated by a certain time width. The certain time width may be 500 ms. For example, if the frame rate is 60 fps (frames per second), one file contains 30 frames.

The scene description is generated for each file and includes information for each frame that identifies the 360° video and 3D objects. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, an information element (scale_object) indicating the size of the 3D object in the frame, etc.

The transmitter 103 transmits the first content, the first control information, the second content, and the second control information (hereinafter, the transmission signal).

For ease of explanation, FIG. 2 illustrates a case in which the first processing unit 101 and the second processing unit 102 are separate processing units, but the embodiment is not limited to this. The first processing unit 101 and the second processing unit 102 may be configured as a single processing unit.

(Receiving device)
A receiving device according to an embodiment will be described below. Fig. 3 is a block diagram showing a receiving device 200 according to an embodiment.

As shown in FIG. 3, the receiving device 200 has a receiving unit 201, a first processing unit 202, and a second processing unit 203.

The receiving unit 201 receives the first content, the first control information, the second content, and the second control information (hereinafter, the received signal). The received signal may be synonymous with the transmission signal described above.

The first processing unit 202 processes first control information (MMT-SI) associated with the first content that does not have viewpoint freedom. The first processing unit 202 may process the first content. For example, the first processing unit 202 processes the first content based on the MMT-SI and outputs the first content (2D video and audio). Furthermore, the first processing unit 202 identifies the presentation time of the first content based on the first absolute time information (MPU timestamp descriptor) included in the MMT-SI.

The second processing unit 203 processes second control information (scene description) associated with the second content having freedom of viewpoint. The second processing unit 203 may process the second content. For example, the second processing unit 203 processes the second content based on the scene description and outputs the second content (360° video and 3D objects). Here, the second processing unit 203 outputs the 360° video and 3D objects based on viewpoint information (viewpoint position and line of sight direction). The viewpoint information may be input from a user I/F (Interface) or may be input from a sensor (acceleration sensor, GPS sensor, etc.) included in the receiving device 200.

For ease of explanation, FIG. 3 illustrates a case in which the first processing unit 202 and the second processing unit 203 are separate processing units, but the embodiment is not limited to this. The first processing unit 202 and the second processing unit 203 may be configured as a single processing unit.

(Second Content)
The second content according to the embodiment will be described below. Here, the second content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.

For example, as shown in FIG. 4, at t=0, a 360° video may be displayed without a 3D object being displayed. The 360° video may be considered to be a background video of the 3D object. At t=1, a 3D object may be displayed in a form superimposed on the 360° video. Furthermore, at t=1, the position and rotation of the 3D object superimposed on the 360° video may be changed.

The above-mentioned scene description includes information elements that indicate the position, rotation, and size of the 3D object for each of t=0, t=1, and t=2, and allows the 3D object to be appropriately superimposed on the 360° video.

(Usage scenario)
In the above-mentioned background, a usage scene according to the embodiment will be described. As described above, the scene description associated with the second content is control information independent of the MMT-SI associated with the first content. That is, although the second content is consistent, the first content and the second content cannot be appropriately synchronized. In the embodiment, focusing on such a usage scene, the following mechanism is introduced.

Note that in the following, synchronization means that the presentation times of the first content (e.g., MPU) and the second content (file) are properly aligned. Therefore, synchronization may include the presentation times of the 2D video and the 3D object being aligned, and may also include the presentation times of the audio and the 3D object being aligned. Similarly, synchronization may include the presentation times of the 2D video and the 360° video being aligned, and may also include the presentation times of the audio and the 360° video being aligned.

(First method)
In the first method, the MMT-SI includes second absolute time information indicating a presentation time of the second content. The second absolute time information may be a presentation time of a first frame included in the scene description.

For example, the second absolute time information may be generated using UTC as the reference time. The reference time may be the TAI or a time provided by a GPS. The reference time may be a time provided by an NTP server or a time provided by a PTP server. Furthermore, the second absolute time information may be generated based on the same reference time as the first absolute time information (i.e., the MPU timestamp descriptor).

Here, the fixed time span (MPU time span) of the first content presented by one MMT-SI may be the same as the fixed time span (file time span) of the second content presented by one scene description. In such a case, the MPU timestamp descriptor included in the MMT-SI may be reused as the second absolute time information. In other words, the MMT-SI does not need to include second absolute time information in addition to the MPU timestamp descriptor, and it is sufficient to include one MPU timestamp descriptor.

Specifically, as shown in Figure 5, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in Figure 5), so that the 2D video and audio can be synchronized.

On the other hand, the presentation time of the first frame included in the scene description is identified by referencing the MPU timestamp descriptor included in MMT-SI. The presentation times of the second and subsequent frames included in the scene description can be identified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is identified by adding 1/30 x n to the time identified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

In the first method, as shown in FIG. 6, the scene description does not need to include the second absolute time information, but only needs to include information elements for each frame indicating the object name, frame number, and the position, rotation, and size of the 3D object.

Here, the scene description may be treated as an MFU, similar to NAL units and MHAS units, and may be transmitted in the form of MMTP packets through packetization. Alternatively, the scene description may be transmitted using a protocol such as HTTP (Hyper Text Transfer Protocol).

In the first method, the receiving device 200 uses the MMT-SI as an entry point to check whether or not a scene description (second content) is present, and if a scene description is present, the receiving device 200 uses the MPU timestamp descriptor to identify the presentation time of the second content.

(Second method)
In a second method, the scene description includes second absolute time information indicating a presentation time of the second content, The second absolute time information may be a presentation time of a first frame included in the scene description.

For example, the second absolute time information may be generated using UTC as the reference time, as in the first method. The reference time may be the TAI or a time provided by a GPS. The reference time may be a time provided by an NTP server or a time provided by a PTP server. Furthermore, the second absolute time information may be generated based on the same reference time as the first absolute time information (i.e., the MPU timestamp descriptor).

Specifically, as shown in Figure 7, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in Figure 5), so that the 2D video and audio can be synchronized.

On the other hand, the presentation time of the first frame included in the scene description is identified by referencing the second absolute time information (simply timestamp in Figure 7) included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be identified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is identified by adding 1/30 x n to the time identified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

In the second method, as shown in FIG. 8, the scene description must include second absolute time information in addition to the information elements shown in FIG. 6. FIG. 8 illustrates a case where the reference time is UTC, and "absolute_time_UTC":"0xda192c2f.813953de(2015-12-14 20:53:19.504)" is added as the second absolute time information.

In FIG. 8, "absolute_time_UTC" includes a time expressed in 64 bits and a time expressed in the form of a date and time. However, "absolute_time_UTC" may include a time expressed in 64 bits without including the latter, or may include a time expressed in the form of a date and time without including the former. "absolute_time_UTC" may include a time expressed in another format.

Here, the scene description may be treated as an MFU, similar to NAL units and MHAS units, and may be transmitted in the form of MMTP packets through packetization. Alternatively, the scene description may be transmitted using a protocol such as HTTP.

In the second method, the receiving device 200 uses the MMT-SI as an entry point to check whether or not there is a second content (scene description) to be presented together with the first content, and if a scene description is present, identifies the presentation time of the second content based on the MPU timestamp descriptor included in the MMT-SI. The receiving device 200 may also identify the presentation time of the second content based on second absolute time information included in the scene description.

(Action and Effects)
In the embodiment, when focusing on a new usage scene in which a first content and a second content are presented in combination, even if the first control information and the second control information are independent, on the premise that the first control information includes first absolute time information indicating the presentation time of the first content, at least one of the first control information and the second control information includes second absolute time information indicating the presentation time of the second content. With such a configuration, it is possible to appropriately identify the presentation times of the first content and the second content, and it is possible to appropriately synchronize the first content and the second content.

Specifically, according to the first method, the MPU timestamp descriptor included in the MMT-SI is used as the second absolute time information, and the receiving device 200 can synchronize the 2D video and audio, and can also synchronize the first content (2D video and audio) and the second content (360° video and 3D object) by referring to the MPU timestamp descriptor included in the MMT-SI. Therefore, the first content and the second content can be synchronized while suppressing an increase in the processing load on the receiving device 200.

On the other hand, according to the second method, the second absolute time information is included in the scene description, so the receiving device 200 can present the second content on the premise that it is synchronized with the first content, without relying on the MPU timestamp descriptor included in the MMT-SI. For example, it is also possible to switch between a state in which the second content is presented without being combined with the first content and a state in which the second content is presented in combination with the first content.

In an embodiment, the scene description may be transmitted in the form of an MMTP packet. According to such a configuration, the receiving device 200 can identify the presence of the scene description by analyzing the MPT included in the MMT-SI. Furthermore, the receiving device 200 can identify the presentation time of the second content by either the first method or the second method. The receiving device 200 can easily present the first content and the second content in a state in which the first content and the second content are synchronized.

[Other embodiments]
Although the present invention has been described by the above disclosure, the description and drawings forming a part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art.

Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents specified in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.

In the above disclosure, an MPU timestamp descriptor is exemplified as the first absolute time information included in MMT-SI. However, the above disclosure is not limited to this. The first absolute time information included in MMT-SI may be an MPU extended timestamp descriptor.

Although not specifically mentioned in the above disclosure, the receiving device 200 may request a portion of the second content from the transmitting device 100 as necessary. This configuration can save bandwidth associated with the transmission of the second content and suppress an increase in the processing load of the receiving device 200.

In the above disclosure, MMTP is exemplified as the transmission method of the first content. However, the above disclosure is not limited to this. The transmission method of the first content may be a method conforming to ISO/IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the first control information may be MPD (Media Presentation Description). That is, in the above disclosure, MMT-SI may be read as MPD.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the transmitting device 100 and the receiving device 200. The program may also be recorded on a computer-readable medium. Using the computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

Alternatively, a chip may be provided that is configured with a memory that stores programs for executing the processes performed by the transmitting device 100 and the receiving device 200, and a processor that executes the programs stored in the memory.

10...transmission system, 100...transmitting device, 101...first processing unit, 102...second processing unit, 103...transmitting unit, 200...receiving device, 201...receiving unit, 202...first processing unit, 203...second processing unit

Claims (6)

a first processing unit that processes first control information associated with a first content that does not have freedom of viewpoint;
a second processing unit that processes second control information associated with a second content having freedom of viewpoint;
a transmitter that transmits the first content, the first control information, the second content, and the second control information,
the first control information includes first absolute time information indicating a presentation time of the first content,
A transmitting device, wherein at least one of the first control information and the second control information includes second absolute time information indicating a presentation time of the second content. The transmitting device according to claim 1, wherein the first control information includes the second absolute time information. The transmitting device according to claim 1, wherein the second control information includes the second absolute time information. The transmission device according to any one of claims 1 to 3, wherein the second absolute time information is generated based on the same reference time as the first absolute time information. The transmitting device according to any one of claims 1 to 4, wherein the transmitting unit transmits the second control information in the same format as the first control information. a receiving unit that receives a first content having no freedom of viewpoint, first control information associated with the first content, a second content having freedom of viewpoint, and second control information associated with the second content;
A first processing unit that processes the first control information;
A second processing unit that processes the second control information,
the first control information includes first absolute time information indicating a presentation time of the first content,
At least one of the first control information and the second control information includes second absolute time information indicating a presentation time of the second content,
The first processing unit specifies a presentation time of the first content based on the first absolute time information;
A receiving device , wherein a processing unit out of the first processing unit and the second processing unit that processes the second absolute time information specifies a presentation time of the second content based on the second absolute time information .

Images