JP7685347B2 - Transmitting device and receiving device - Google Patents

Description

The present invention relates to a transmitting device and a receiving device.

Conventionally, ISO/IEC 23008-1 (hereafter, MMT (MPEG Media Transport)) and ISO/IEC 23009-1 (hereafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)) are known as standards for transmitting content such as 2D video and audio. In these standards, a timestamp descriptor indicating the absolute time at which the content is presented is defined as one piece of control information (MMT-SI (Signaling Information)).

Meanwhile, mechanisms for transmitting content such as 360° video and 3D objects have also been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which the user moves their head while in a seated position, and 6DoF, which involves viewpoint movement within the range in which the user can move freely. In such mechanisms, the positional relationship between the 360° video and the 3D objects is indicated by a scene description.

3GPP TR 26.928 V16.1.0 December 2020

In light of the above background, the inventors focused on usage scenarios in which content of different formats is combined and presented, and discovered that because the above-mentioned scene description is independent of MMT-SI, content of different formats cannot be properly synchronized.

Therefore, the present invention has been made to solve the above-mentioned problems, and aims to provide a transmitting device and a receiving device that enable appropriate synchronization of content of different formats.

The transmission device disclosed herein includes a first processing unit that processes first control information associated with a first content having freedom of viewpoint, and a transmission unit that transmits a second content that does not have freedom of viewpoint in addition to the first content and the first control information, and the first control information includes reference information for identifying the second content and first absolute time information that indicates the presentation time of the first content.

The receiving device disclosed herein includes a receiving unit that receives a first content having freedom of viewpoint, first control information associated with the first content, and a second content having no freedom of viewpoint, and a first processing unit that processes the first control information, and the first control information includes reference information for identifying the second content and first absolute time information indicating the presentation time of the first content.

The present invention provides a transmitting device and a receiving device that enable appropriate synchronization of content of different formats.

FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. FIG. 2 is a block diagram showing a transmitting device 100 according to the embodiment. FIG. 3 is a block diagram showing a receiving device 200 according to the embodiment. FIG. 4 is a diagram for explaining the first content according to the embodiment. FIG. 5 is a diagram for explaining a control method according to the embodiment. FIG. 6 is a diagram for explaining a control method according to the embodiment. FIG. 7 is a diagram for explaining a control method according to the first modified example. FIG. 8 is a diagram for explaining a control method according to the first modified example. FIG. 9 is a diagram for explaining conversion information according to the second modification. FIG. 10 is a diagram for explaining conversion information according to the second modification. FIG. 11 is a diagram for explaining conversion information according to the second modification.

Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are denoted by the same or similar reference numerals. However, it should be noted that the drawings are schematic, and the ratios of the dimensions may differ from the actual ones.

Therefore, specific dimensions should be determined with reference to the following explanation. Of course, there are also parts in which the dimensional relationships and ratios differ between the drawings.

[Disclosure Summary]
A transmitting device according to the disclosure summary includes a first processing unit that processes first control information associated with a first content having viewpoint freedom, and a transmitting unit that transmits, in addition to the first content and the first control information, a second content that does not have viewpoint freedom, and the first control information includes reference information for identifying the second content and first absolute time information indicating the presentation time of the first content.

The receiving device according to the disclosure includes a receiving unit that receives a first content having freedom of viewpoint, first control information associated with the first content, and a second content having no freedom of viewpoint, and a first processing unit that processes the first control information, and the first control information includes reference information for identifying the second content and first absolute time information indicating the presentation time of the first content.

In the summary of the disclosure, when focusing on a new usage scenario in which a first content and a second content are presented in combination, the first control information includes reference information for identifying the second content and first absolute time information indicating the presentation time of the first content. With this configuration, the first content and the second content can be appropriately synchronized based on the first control information.

[Embodiment]
(Transmission System)
A transmission system according to an embodiment will be described below. Fig. 1 is a diagram showing a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100 and a receiving device 200.

In an embodiment, the transmitting device 100 transmits a first content having freedom of viewpoint and a second content having no freedom of viewpoint to the receiving device 200. Furthermore, the transmitting device 100 transmits first control information associated with the first content and second control information associated with the second content to the receiving device 200.

The first content may include a 360° video and a 3D object. The first content and the first control information may be transmitted in a first manner. The first content may conform to 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range in which a user moves their head while in a sitting position, or 6DoF, which involves viewpoint movement within the range in which a user can move freely. Since the first content has viewpoint freedom, it may include two or more 360° videos and two or more 3D objects at the same time (frame). The first control information may be referred to as a scene description.

The second content may include at least one of 2D video and audio. The second content and the second control information may be transmitted in a second method. The second method may be a method conforming to ISO/IEC 23008-1 (hereinafter, MMT (MPEG Media Transport)). In the following, a case will be exemplified in which the second method is MMTP (MMT Protocol) conforming to MMT. The second control information may be referred to as MMT-SI (Signaling Information).

The transmission from the transmitting device 100 to the receiving device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communication network.

The transmission system may be, but is not limited to, a digital wireless transmission system. The digital wireless transmission system may be a system used for 4K or 8K satellite broadcasting. The receiving device 200 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display.

(Transmitting device)
A transmitting device according to an embodiment will be described below. Fig. 2 is a block diagram showing a transmitting device 100 according to an embodiment.

As shown in FIG. 2, the transmitting device 100 has a first processing unit 101, a second processing unit 102, and a transmitting unit 103.

The first processing unit 101 processes first control information (scene description) associated with the first content having freedom of viewpoint. The first processing unit 101 may process the first content. That is, the first processing unit 101 may process a 360° video image or a 3D object.

For example, the first processing unit 101 may generate MMTP packets by packetizing the scene description. The second processing unit 102 may apply a projective transformation such as ERP (Equirectangular projection) or cube map to the 360° video to convert the 360° video into a 2D video. The second processing unit 102 may generate metadata indicating the type of projective transformation applied to the 360° video. The first processing unit 101 may encode the 3D object in a mesh format. For the mesh format encoding, ISO/IEC 14496-16 "Animation framework extension (AFX)" may be used. The first processing unit 101 may encode the 3D object in a point cloud format. For the point cloud format encoding, ISO/IEC 23090-5 "Video-based Point Cloud Compression" may be used.

Here, the first content is compiled into one file in units separated by a certain time width. The certain time width may be 500 ms. For example, if the frame rate is 60 fps (frames per second), one file contains 30 frames.

The scene description is generated for each file and includes information for each frame that identifies the 360° video and 3D objects. For example, the scene description includes an information element (object_name) indicating the name of the 3D object in the frame, an information element (frame_number) indicating the frame number, an information element (translation_object) indicating the position of the 3D object in the frame, an information element (rotation_object) indicating the rotation of the 3D object in the frame, an information element (scale_object) indicating the size of the 3D object in the frame, etc.

The second processing unit 102 processes second control information (MMT-SI) associated with the second content that does not have viewpoint freedom. The second processing unit 102 may process the second content. That is, the second processing unit 102 may process 2D video or may process audio. For example, the second processing unit 102 may generate an MMTP packet by packetizing the 2D video, audio, and MMT-SI.

For example, MMTP packets are stored in IP (Internet Protocol) packets. IP packets may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).

Here, the second content is processed in units (hereinafter, MPU; Media Processing Unit) that are separated by a certain time width. The MPU includes one or more access units. The access unit may be treated as an MFU (Media Fragment Unit). The MFU related to 2D video may be referred to as a NAL (Network Abstraction Layer) unit, and the MFU related to audio may be referred to as an MHAS (MPEG-H 3D Audio Stream) packet.

The MMT-SI includes a PA (Package Access) message, and the PA message includes an MPT (MMT Package Table) that indicates a list of second contents. Furthermore, the MMT-SI includes second absolute time information that indicates the presentation time of the second content. The second absolute time information may mean the presentation time of the MPU, i.e., the time of the access unit that is first presented in the MPU. The second absolute time information may be referred to as an MPU timestamp descriptor.

The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or may be a time provided by GPS (Global Positioning System). The reference time may be a time provided by an NTP (Network Time Protocol) server or a PTP (Precision Time Protocol) server.

The transmitter 103 transmits the first content, the first control information, the second content, and the second control information (hereinafter, the transmission signal).

For ease of explanation, FIG. 2 illustrates a case in which the first processing unit 101 and the second processing unit 102 are separate processing units, but the embodiment is not limited to this. The first processing unit 101 and the second processing unit 102 may be configured as a single processing unit.

(Receiving device)
A receiving device according to an embodiment will be described below. Fig. 3 is a block diagram showing a receiving device 200 according to an embodiment.

As shown in FIG. 3, the receiving device 200 has a receiving unit 201, a first processing unit 202, and a second processing unit 203.

The receiving unit 201 receives the first content, the first control information, the second content, and the second control information (hereinafter, the received signal). The received signal may be synonymous with the transmission signal described above.

The first processing unit 202 processes first control information (scene description) associated with the first content having freedom of viewpoint. The first processing unit 202 may process the first content. For example, the first processing unit 202 processes the first content based on the scene description and outputs the first content (360° video and 3D objects). Here, the first processing unit 202 outputs the 360° video and 3D objects based on viewpoint information (viewpoint position and line of sight direction). The viewpoint information may be input from a user I/F (Interface) or may be input from a sensor (acceleration sensor, GPS sensor, etc.) included in the receiving device 200.

The second processing unit 203 processes second control information (MMT-SI) associated with the second content that does not have viewpoint freedom. The second processing unit 203 may process the second content. For example, the second processing unit 203 processes the second content based on the MMT-SI and outputs the second content (2D video and audio). Furthermore, the second processing unit 203 identifies the presentation time of the second content based on the second absolute time information (MPU timestamp descriptor) included in the MMT-SI.

For ease of explanation, FIG. 3 illustrates a case in which the second processing unit 203 and the first processing unit 202 are separate processing units, but the embodiment is not limited to this. The second processing unit 203 and the first processing unit 202 may be configured as a single processing unit.

(First Content)
The first content according to the embodiment will be described below. Here, the first content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.

For example, as shown in FIG. 4, at t=0, a 360° video may be displayed without a 3D object being displayed. The 360° video may be considered to be a background video of the 3D object. At t=1, a 3D object may be displayed in a form superimposed on the 360° video. Furthermore, at t=1, the position and rotation of the 3D object superimposed on the 360° video may be changed.

The above-mentioned scene description includes information elements that indicate the position, rotation, and size of the 3D object for each of t=0, t=1, and t=2, and allows the 3D object to be appropriately superimposed on the 360° video.

(Usage scenario)
In the above-mentioned background, a usage scene according to the embodiment will be described. As described above, the scene description associated with the first content is control information independent of the MMT-SI associated with the second content. That is, although the first content is consistent, the second content and the first content cannot be appropriately synchronized. In the embodiment, focusing on such a usage scene, the following mechanism is introduced.

Note that in the following, synchronization means that the presentation times of the first content (file) and the second content (e.g., MPU) are properly aligned. Therefore, synchronization may include the alignment of the presentation times of the 2D video and the 3D object, and may include the alignment of the presentation times of the audio and the 3D object. Similarly, synchronization may include the alignment of the presentation times of the 2D video and the 360° video, and may include the alignment of the presentation times of the audio and the 360° video.

(Control Method)
In the control method according to the embodiment, the scene description includes first absolute time information indicating a presentation time of the first content. The first absolute time information may be a presentation time of a first frame included in the scene description.

For example, the first absolute time information may be generated using UTC as the reference time. The reference time may be the TAI or a time provided by a GPS. The reference time may be a time provided by an NTP server or a time provided by a PTP server. Furthermore, the first absolute time information may be generated based on the same reference time as the second absolute time information (i.e., the MPU timestamp descriptor).

Furthermore, the scene description includes reference information for identifying the second content. The reference information may be information for identifying an MPU that constitutes the second content. In other words, the reference information is information for treating the second content (MPU) as an object included in the scene description.

Specifically, as shown in FIG. 5, the presentation time of the first frame included in the scene description is specified by the first absolute time information included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the first content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1/30×n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

On the other hand, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in FIG. 5), so that the 2D video and audio can be synchronized. Here, since the above-mentioned reference information is included in the scene description, the receiving device 200 can check whether or not there is a second content to be presented together with the first content, based on the reference information included in the scene description.

In the control method according to the embodiment, as shown in FIG. 6, the scene description includes, for each frame, information elements indicating the name of the object, the frame number, and the position, rotation, and size of the 3D object. Furthermore, the scene description includes first absolute time information and reference information.

Specifically, the scene description includes "absolute_time_UTC": "0xda192c2f.813953de (2015-12-14 20:53:19.504)" as the first absolute time information. In FIG. 6, "absolute_time_UTC" includes a time expressed in 64 bits and a time expressed in date and time. However, "absolute_time_UTC" may include a time expressed in 64 bits without including the latter, or may include a time expressed in date and time without including the former. "absolute_time_UTC" may include a time expressed in another format.

In the control example according to the embodiment, since MMT-SI is used, the scene description only needs to include information capable of identifying MMT-SI as reference information, and the information capable of identifying MMT-SI may be information for identifying a resource of the second content. For example, the scene description includes, as reference information, "object_name":"2d_video&audio", "uri":"mmt://example.co.jp://51216". "2d_video&audio" means that 2D video and audio are included as objects, and "mmt://example.co.jp://51216" is information for identifying a resource of the second content (port number "51216" of "example.co.jp").

The scene description may be transmitted using a protocol such as HTTP (Hyper Text Transfer Protocol).

In the control method according to the embodiment, the receiving device 200 uses the scene description as an entry point to check whether or not there is a second content (MMT-SI) to be presented together with the first content, and if MMT-SI is present, identifies the presentation time of the second content based on the first absolute time information included in the scene description. The receiving device 200 may also identify the presentation time of the second content (2D video and 3D audio) based on the MPU timestamp descriptor included in the MMT-SI.

(Action and Effects)
In the embodiment, in a case where a new usage scene in which a first content and a second content are presented in combination is focused on, the first control information includes reference information for identifying the second content and first absolute time information indicating a presentation time of the first content. With this configuration, the first content and the second content can be appropriately synchronized based on the first control information.

In the embodiment, MMT-SI (MPU timestamp descriptor) is used, so that the first content and the second content can be appropriately synchronized while synchronizing 2D video and audio, while suppressing modifications to the existing MMTP.

[Modification 1]
A first modification of the embodiment will be described below, focusing on differences from the above-described embodiment.

Specifically, in the embodiment, synchronization between 2D video and audio is achieved based on the second absolute time information (MPU timestamp descriptor) included in MMT-SI. In Modification Example 1, synchronization between 2D video and audio is also achieved based on information elements included in the scene description (first absolute time information and reference information). Therefore, in Modification Example 1, at least the MPU timestamp descriptor included in MMT-SI may be omitted. In Modification Example 1, the MMT-SI itself may be omitted. Below, a case in which the MMT-SI itself is omitted will be described.

(Control Method)
In the control method according to the first modification, the scene description includes first absolute time information indicating the presentation time of the first content and reference information for identifying the second content, similarly to the embodiment.

Specifically, as shown in FIG. 7, the presentation time of the first frame included in the scene description is identified by the first absolute time information included in the scene description, as in the embodiment.

On the other hand, since there is no MPU timestamp descriptor, the presentation time of the 2D video and audio is specified by the first absolute time information included in the scene description. The reference information included in the scene description includes information for identifying the MPU related to the 2D video and information for identifying the MPU related to the audio. The receiving device 200 can check whether or not there is a second content to be presented together with the first content, based on the reference information included in the scene description.

In the control method according to the first modification, as shown in FIG. 8, the scene description includes information elements similar to those shown in FIG. 6.

However, in the control example according to the first modification, since MMT-SI is omitted, the scene description includes, as reference information, information capable of identifying MMT-SI, information for directly identifying the MPU that constitutes the second content. For example, the scene description includes "uri":"mmt://example.co.jp://51216?asset_ID_value=100&mpu_sequence_number=10000" as reference information related to 2D video. "mmt://example.co.jp://51216" is information for identifying the resource of the second content (port number "51216" of "example.co.jp"). "asset_ID_value=100&mpu_sequence_number=10000" is an information element for identifying an MPU whose "asset_ID" is 100 and whose sequence number is 10000. The scene description includes "uri":"mmt://example.co.jp://51216?asset_ID_value=102&mpu_sequence_number=5000" as reference information related to audio. "mmt://example.co.jp://51216" is information for identifying the resource of the second content (port number "51216" of "example.co.jp"). "asset_ID_value=102&mpu_sequence_number=5000" is an information element for identifying an MPU whose "asset_ID" is 102 and whose sequence number is 5000.

In the control method according to the first modification, the receiving device 200 uses the scene description as an entry point to check whether or not there is a second content to be presented together with the first content, and if there is a second content, determines the presentation time of the second content (2D video and 3D audio) based on the first absolute time information included in the scene description without using MMT-SI.

(Action and Effects)
In the first modification, in a case where a new usage scene in which a first content and a second content are presented in combination is focused on, the first control information includes reference information for identifying the second content and first absolute time information indicating the presentation time of the first content. With this configuration, the first content and the second content can be appropriately synchronized based on the first control information.

In the first modification, it is possible to appropriately synchronize the first content and the second content while synchronizing the 2D video and audio, while saving the bandwidth associated with the transmission of MMT-SI.

[Modification 2]
The second modification of the embodiment will be described below, focusing on differences from the above-described embodiment.

In the embodiment, a case has been described in which the reference time of the first absolute time information is the same as the reference time of the second absolute time information (MPU timestamp descriptor). In contrast, in modification example 2, a case has been described in which the reference time of the first absolute time information is different from the reference time of the second absolute time information.

For example, the reference time of the first absolute time information (hereinafter, the first reference time) may be a time other than UTC (TAI, time provided by GPS). The reference time of the second absolute time information (MPU timestamp descriptor) (hereinafter, the second reference time) may be UTC. In such a case, it is preferable that at least one of the first control information (scene description) and the second control information (MMT-SI) includes conversion information between the first reference time and the second reference time.

First, we will explain the case where the scene description includes conversion information. The conversion information may include both first absolute time information expressed in the first reference time and first absolute time information expressed in the second reference time.

For example, if the first reference time is TAI and the second reference time is UTC, as shown in FIG. 9, the scene description may include both the first absolute time information expressed in TAI (absolute_time_TAI in FIG. 9) and the first absolute time information expressed in UTC (absolute_time_UTC in FIG. 9). If the first reference time is a 27 MHz clock and the second reference time is UTC, as shown in FIG. 10, the scene description may include both the first absolute time information expressed in the 27 MHz clock (27 MHz in FIG. 10) and the first absolute time information expressed in UTC (absolute_time_UTC in FIG. 10).

The 27 MHz clock is a system clock used in existing broadcasting systems (e.g., MPEG-2 Transport Stream) and existing content systems (e.g., MPEG-2 Program Stream). MPEG-2 Program Stream may be used for downloading MP4 files and the like, and may also be used for DVD videos.

Here, the conversion information includes the first absolute time information represented by the second reference time, but modification example 2 is not limited to this. The conversion information only needs to include information for converting the first absolute time information represented by the first reference time into the first absolute time information represented by the second reference time.

Secondly, we will explain the case where MMT-SI includes conversion information. The conversion information may include both the second absolute time information represented by the first reference time and the second absolute time information represented by the second reference time.

For example, if the first reference time is TAI and the second reference time is UTC, as shown in FIG. 11, the MMT-SI (MPU timestamp descriptor) may include both the second absolute time information expressed in UTC (absolute_time_UTC in FIG. 11) and the second absolute time information expressed in TAI (absolute_time_TAI in FIG. 11).

Here, the conversion information includes the second absolute time information represented by the second reference time, but modification example 2 is not limited to this. The conversion information only needs to include information for converting the second absolute time information represented by the first reference time into the second absolute time information represented by the second reference time.

[Other embodiments]
Although the present invention has been described by the above disclosure, the description and drawings forming a part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples and operating techniques will become apparent to those skilled in the art.

Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents specified in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.

In the above disclosure, an MPU timestamp descriptor is exemplified as the second absolute time information included in MMT-SI. However, the above disclosure is not limited to this. The second absolute time information included in MMT-SI may be an MPU extended timestamp descriptor.

Although not specifically mentioned in the above disclosure, the receiving device 200 may request a portion of the first content from the transmitting device 100 as necessary. This configuration can save bandwidth associated with the transmission of the first content and suppress an increase in the processing load of the receiving device 200.

In the above disclosure, MMTP is exemplified as the transmission method of the second content. However, the above disclosure is not limited to this. The transmission method of the second content may be a method conforming to ISO/IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the second control information may be MPD (Media Presentation Description). That is, in the above disclosure, MMT-SI may be read as MPD.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute each process performed by the transmitting device 100 and the receiving device 200. The program may also be recorded on a computer-readable medium. Using the computer-readable medium, it is possible to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, and may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

Alternatively, a chip may be provided that is configured with a memory that stores programs for executing the processes performed by the transmitting device 100 and the receiving device 200, and a processor that executes the programs stored in the memory.

10...transmission system, 100...transmitting device, 101...first processing unit, 102...second processing unit, 103...transmitting unit, 200...receiving device, 201...receiving unit, 202...first processing unit, 203...second processing unit

Claims (6)

a first processing unit that processes first control information associated with a first content having a freedom of viewpoint;
a transmitter that transmits, in addition to the first content and the first control information, a second content that does not have freedom of viewpoint;
A transmitting device, wherein the first control information includes reference information for identifying the second content to be presented together with the first content, and first absolute time information indicating the presentation time of the first content. a second processing unit that processes second control information associated with the second content;
The transmitting device according to claim 1 , wherein the second control information includes second absolute time information indicating a presentation time of the second content. The transmitting device according to claim 1, wherein the first control information includes the first absolute time information as information indicating the presentation time of the second content. The transmitting device according to claim 2, wherein the first control information includes conversion information between the reference time of the first absolute time information and the reference time of the second absolute time information. The transmitting device according to claim 2, wherein the second control information includes conversion information between the reference time of the first absolute time information and the reference time of the second absolute time information. A receiving unit that receives a first content having a degree of freedom of a viewpoint, first control information associated with the first content, and a second content having no degree of freedom of a viewpoint;
a first processing unit that processes the first control information,
the first control information includes reference information for identifying the second content to be presented together with the first content , and first absolute time information indicating a presentation time of the first content;
A receiving device, wherein the first processing unit identifies the presentation time of the first content based on the first absolute time information, and identifies the second content to be presented together with the first content based on the reference information .

Images