JP7825645B2 - Media processing device, transmitting device and receiving device - Google Patents

Description

The present disclosure relates to a media processing device, a transmitting device, and a receiving device.

Mechanisms for transmitting content such as 360° video and 3D objects have been proposed (for example, Non-Patent Document 1). Known mechanisms include 3DoF+ (Degree of Freedom), which involves movement of the viewpoint within the range of head movement of a seated user, and 6DoF, which involves movement of the viewpoint within the range of free movement of the user. In such mechanisms, the positional relationship between the 360° video and the 3D objects is indicated by a scene description.

3GPP TR 26.928 V16.1.0 December 2020

One aspect of the disclosure is a media processing device that includes a renderer that generates specific content that includes at least the content based on the configuration of the content having viewpoint freedom and specific viewpoint information, and an output unit that outputs the specific content generated by the renderer to the user terminal.

One aspect of the disclosure is a transmitting device that includes a transmitting unit that transmits a content configuration having viewpoint freedom, and the transmitting unit transmits specific viewpoint information used to generate specific content that includes at least the content.

One aspect of the disclosure is a receiving device that includes a receiving unit that receives a content configuration having viewpoint freedom, and the receiving unit receives specific viewpoint information used to generate specific content that includes at least the content.

FIG. 1 is a diagram showing a transmission system 10 according to an embodiment. FIG. 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the present invention. FIG. 3 is a diagram illustrating the second content according to the embodiment. FIG. 4 is a diagram showing a method for viewing a specific content according to the embodiment. FIG. 5 is a diagram for explaining the first operation example. FIG. 6 is a diagram for explaining the second operation example. FIG. 7 is a diagram for explaining the second operation example. FIG. 8 is a diagram for explaining the third operation example. FIG. 9 is a diagram for explaining the third operation example. FIG. 10 is a diagram for explaining the third operation example. FIG. 11 is a diagram for explaining the third operation example. FIG. 12 is a diagram for explaining the fourth operational example. FIG. 13 is a diagram for explaining the fourth operational example. FIG. 14 is a diagram for explaining the fourth operational example. FIG. 15 is a diagram for explaining the fifth operational example. FIG. 16 is a diagram for explaining the fifth operational example. FIG. 17 is a diagram for explaining the fifth operational example. FIG. 18 is a diagram for explaining the fifth operational example. FIG. 19 is a diagram for explaining the first method according to the first modification. FIG. 20 is a diagram for explaining the second method according to the first modification.

Next, an embodiment of the present invention will be described. In the following description of the drawings, the same or similar parts are designated by the same or similar reference numerals. However, it should be noted that the drawings are schematic, and the ratios of the dimensions may differ from those of the actual product.

Therefore, specific dimensions should be determined with reference to the explanation below. It goes without saying that the dimensional relationships and ratios between the drawings may differ.

[Disclosure Summary]
The media processing device according to the disclosure summary includes a renderer that generates specific content that includes at least the content based on a content structure having viewpoint freedom and specific viewpoint information, and an output unit that outputs the specific content generated by the renderer to the user terminal.

In the summary of the disclosure, the media processing device generates specific content based on the content configuration and specific viewpoint information. With this configuration, the media processing device can generate specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal that provides feedback on viewpoint information and a second user terminal that does not provide feedback on viewpoint information are mixed.

It should be noted that the specific content generated by the media processing device is generated based on viewpoint information, and the images included in the specific content can be treated as 2D images on the user terminal side. The viewpoint information used to generate the specific content may include specific viewpoint information or viewpoint information received from the user terminal.

[Embodiment]
(Transmission System)
A transmission system according to an embodiment will be described below. Fig. 1 shows a transmission system 10 according to an embodiment. As shown in Fig. 1, the digital wireless transmission system includes a transmitting device 100, a media processing device 200, and a user terminal 300.

In an embodiment, the transmitting device 100 transmits a first content that does not have viewpoint freedom and a second content that has viewpoint freedom to the media processing device 200. Furthermore, the transmitting device 100 transmits first control information associated with the first content and second control information associated with the second content to the media processing device 200.

The first content may include at least one of 2D video and audio. The first content and first control information may be transmitted using a first method. The first method may be a method compliant with ISO/IEC 23008-1 (hereinafter referred to as MMT (MPEG Media Transport)). The following describes an example in which the first method is MMTP (MMT Protocol) compliant with MMT. The first control information may be referred to as MMT-SI (Signaling Information).

The second content may include 360° video and 3D objects. The second content and second control information may be transmitted using a second method, or may be transmitted using a protocol such as HTTP (Hyper Text Transfer Protocol). The second content may conform to 3DoF+ (Degree of Freedom), which involves viewpoint movement within the range of head movement of a user in a seated position, or 6DoF, which involves viewpoint movement within the range of free movement of a user. Because the second content has viewpoint freedom, it may include two or more 360° video images and two or more 3D objects at the same time (frame). The second control information may be referred to as a scene description.

Here, the second control information may be transmitted using the first method described above. That is, the second control information may be transmitted using the same first method (e.g., MMTP) as the first control information. Alternatively, the second control information may be transmitted using a protocol such as HTTP.

Transmission from the transmitting device 100 to the media processing device 200 is not particularly limited, but may be transmission using satellite broadcasting, transmission using the Internet network, or transmission using a mobile communication network.

Although not particularly limited, the transmission system may be a digital wireless transmission system. The digital wireless transmission system may be a system used for 4K and 8K satellite broadcasting.

The media processing device 200 generates specific content including at least the second content described above based on the viewpoint information received from the user terminal 300, and transmits the generated specific content to the user terminal 300. Although not particularly limited, the transmission of the specific content may be via the Internet network or a mobile communication network.

The user terminal 300 may be a user terminal such as a smartphone, a tablet terminal, or a head-mounted display. As shown in FIG. 1, two or more user terminals 300 may be provided as the user terminal 300. In other words, two or more user terminals 300 may request the media processing device 200 to generate specific content. Each user terminal 300 may transmit different viewpoint information to the media processing device 200.

(Media processing device and user terminal)
A media processing device and a user terminal according to an embodiment of the present invention will now be described. Figure 2 is a block diagram showing a media processing device 200 and a user terminal 300 according to an embodiment of the present invention.

First, the media processing device 200 has a reception unit 210, a renderer 220, and an encoding processing unit 230.

The reception unit 210 receives viewpoint information. In an embodiment, the reception unit 210 constitutes a receiving unit that receives viewpoint information from the user terminal 300. The viewpoint information includes an information element indicating the viewpoint position of the user of the user terminal 300 and an information element indicating the line of sight of the user of the user terminal 300.

The renderer 220 generates specific content that includes at least the second content based on the viewpoint information. Because the specific content is generated based on the viewpoint information, it may include one 360° video and one 3D object at the same time (frame). Below, we will explain an example of a case where the specific content includes the first content in addition to the second content.

As shown in Figure 2, the renderer 220 generates first content including 2D video and audio as part of the specific content based on the first control information (MMT-SI). Viewpoint information is not required in generating the first content.

Specifically, the renderer 220 acquires the 2D video, audio, and MMT-SI in the form of an MMTP packet in which the 2D video, audio, and MMT-SI are packetized.

For example, MMTP packets are stored in IP (Internet Protocol) packets, which may be transmitted using UDP (User Datagram Protocol) or TCP (Transmission Control Protocol).

Here, the first content is processed in units (hereinafter referred to as MPUs; Media Processing Units) that are separated by a fixed time width. An MPU includes one or more access units. An access unit may also be treated as an MFU (Media Fragment Unit). An MFU related to 2D video may be referred to as a NAL (Network Abstraction Layer) unit, and an MFU related to audio may be referred to as an MHAS (MPEG-H 3D Audio Stream) packet.

MMT-SI includes a PA (Package Access) message, which includes an MPT (MMT Package Table) that lists the first content. Furthermore, MMT-SI includes an MPU timestamp descriptor that indicates the presentation time of the first content. The MPU timestamp descriptor may indicate the presentation time of the MPU, i.e., the time of the first access unit presented in the MPU.

The MPU timestamp descriptor may be generated using UTC (Coordinated Universal Time) as the reference time. The reference time may be TAI (International Atomic Time) or the time provided by GPS (Global Positioning System). The reference time may be the time provided by an NTP (Network Time Protocol) server or the time provided by a PTP (Precision Time Protocol) server.

Second, the renderer 220 generates second content including 360° video and 3D objects as part of the specific content based on the second control information (scene description). Viewpoint information is used in generating the second content.

Specifically, the renderer 220 may acquire the scene description in the form of an MMTP packet in which the scene description is packetized. There are no particular limitations on the method for acquiring the 360° video and 3D objects.

360° video may be converted into 2D video using a projective transformation such as ERP (Equirectangular Projection) or cube mapping. Metadata indicating the type of projective transformation applied to the 360° video may be added. 3D objects may be encoded in mesh format. ISO/IEC 14496-16 "Animation framework extension (AFX)" may be used for mesh format encoding. 3D objects may be encoded in point cloud format. ISO/IEC 23090-5 "Video-based Point Cloud Compression" may be used for point cloud format encoding.

Here, the second content is compiled into a single file in units separated by a fixed time interval. The fixed time interval may be 500 ms. For example, if the frame rate is 60 fps (frames per second), one file contains 30 frames.

A scene description is generated for each file and includes information for each frame that identifies the 360° video and 3D objects. For example, the scene description includes an information element (object_name) that indicates the name of the 3D object in the frame, an information element (frame_number) that indicates the frame number, an information element (translation_object) that indicates the position of the 3D object in the frame, an information element (rotation_object) that indicates the rotation of the 3D object in the frame, and an information element (scale_object) that indicates the size of the 3D object in the frame.

Third, the renderer 220 outputs specific content including the first content and the second content to the encoding processing unit 230. The renderer 220 may output the presentation time of the specific content to the encoding processing unit 230 together with the specific content.

Here, the presentation time of the specific content may be corrected based on the delay time between the media processing device 200 and the user terminal 300. Specifically, the renderer 220 may calculate the presentation time (T' = T + ΔT) of the specific content to be provided from the media processing device 200 to the user terminal 300 based on the presentation time (T) and delay time (ΔT) of the specific content provided from the transmitting device 100 to the media processing device 200. The delay time (ΔT) may be a value that is predetermined in the media processing device 200, or may be a value that differs for each user terminal 300.

Fourth, the renderer 220 may constitute a transmission unit that transmits viewpoint information used to generate the specific content to the user terminal 300. The viewpoint information used to generate the specific content may be transmitted from the encoding processing unit 230 to the user terminal 300.

For example, the transmission method for viewpoint information and specific content may be MMTP or HTTP. If MMTP is used as the transmission method for specific content, viewpoint information may be stored as metadata in OMAF (Omnidirectional Media Format) defined in ISO/IEC 23090-2.

The encoding processing unit 230 encodes the specific content generated by the renderer 220. In an embodiment, the encoding processing unit 230 may be an example of a transmission unit that transmits the specific content to the user terminal 300.

Furthermore, the encoding processing unit 230 may encode the presentation time of the specific content. The encoding processing unit 230 may transmit an information element indicating the presentation time to the user terminal 300 together with the specific content.

Here, any compression encoding method can be used as the compression encoding method used by the encoding processing unit 230. For example, the compression encoding method may be HEVC (High Efficiency Video Coding) or VVC (Versatile Video Coding).

As described above, the second content included in the specific content is generated based on viewpoint information, so the video included in the specific content can be treated as a 2D video that does not have viewpoint freedom.

For example, the transmission control method used to start and end viewing of specific content may include RTSP (Real Time Streaming Protocol). The transmission method may be MMTP or HTTP. If MMTP is used as the transmission method, the specific content may be stored in OMAF, as specified in ISO/IEC 23090-2.

As shown in FIG. 2, the user terminal 300 has a detection unit 310, a decoding processing unit 320, and a renderer 330.

The detection unit 310 detects the user's viewpoint position and gaze direction. The detection unit 310 may include an acceleration sensor or a GPS (Global Positioning System) sensor. The detection unit 310 may include a user I/F (e.g., a touch sensor, keyboard, mouse, controller, etc.) that is manually input by the user. The detection unit 310 may transmit viewpoint information (viewpoint position and gaze direction) to the media processing device 200. The detection unit 310 may output the viewpoint information (viewport) to the renderer 330.

The decoding processing unit 320 decodes the specific content received from the media processing device 200. The decoding processing unit 320 may also decode the presentation time received from the media processing device 200. The decoding processing unit 320 may output the specific content to the renderer 330, or may output the presentation time to the renderer 330.

The renderer 330 outputs the specific content decoded by the decoding processing unit 320. The renderer 330 may output the specific content based on the presentation time decoded by the decoding processing unit 320. For example, the renderer 330 may output the video content included in the specific content to a display and the audio content included in the specific content to a speaker.

Here, the renderer 330 may generate specific content with modified viewpoint position and line of sight direction based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.

(Second content)
The second content according to the embodiment will be described below. Here, the second content at t=0, t=1, and t=2 will be described. The time intervals between t=0, t=1, and t=2 are not particularly limited.

For example, as shown in Figure 3, at t=0, the 360° video may be displayed without the 3D object being displayed. The 360° video may be considered a background video for the 3D object. At t=1, the 3D object may be displayed superimposed on the 360° video. Furthermore, at t=1, the position and rotation of the 3D object superimposed on the 360° video may be changed.

The above-mentioned scene description includes information elements that indicate the position, rotation, and size of the 3D object for each of t=0, t=1, and t=2, allowing the 3D object to be appropriately superimposed on the 360° video.

(How to watch)
A viewing method according to the embodiment will be described below, in which viewing of a specific content including a first content and a second content will be exemplified.

As shown in FIG. 4, in step S11, the user terminal 300 sends an RTSP SETUP to the media processing device. The RTSP SETUP is a message indicating that viewing of specific content will begin.

Here, the RTSP SETUP includes the IP address of the user terminal 300, the listening port number, content identification information (content ID), etc. The RTSP SETUP may also include capability information of the user terminal 300 for viewing specific content. The capability information may include the frame rate, display resolution, etc. The display resolution may include the field of view (FoV). The capability information may also include information elements indicating the encoding method and compression method supported by the user terminal 300.

Here, a case is illustrated in which the capability information of the user terminal 300 is notified directly to the media processing device 200, but the embodiment is not limited to this. The capability information of the user terminal 300 may be notified to the transmitting device 100, and then notified from the transmitting device 100 to the media processing device 200.

In step S12, the media processing device 200 transmits a response to the RTSP SETUP. Here, an ACK is transmitted as a response indicating that the RTSP SETUP has been accepted.

In step S21, the user terminal 300 transmits the initial viewpoint information to the media processing device 200. The initial viewpoint information may be transmitted in the format of MMT-SI.

In step S22, the media processing device 200 generates initial specific content based on the initial viewpoint information (rendering process). For example, the media processing device 200 generates second content to be included in the initial specific content based on the initial viewpoint information and the scene description.

Here, the media processing device 200 may generate the initial specific content using a range wider than the display resolution of the user terminal 300 as the viewport. For example, the range wider than the display resolution may be a range of the display resolution + 20% in the horizontal direction and the display resolution + 20% in the vertical direction.

The media processing device 200 applies a compression encoding method to the initial specific content. Although not particularly limited, the compression encoding method may be HEVC or VVC.

In step S23, the media processing device 200 transmits the initial specific content corresponding to the initial viewpoint information to the user terminal 300. The media processing device 200 transmits the presentation time of the initial specific content to the user terminal 300. As described above, the presentation time (T') provided to the user terminal 300 may be determined based on the delay time (ΔT).

In addition, if a different value is used for the delay time (ΔT) for each user terminal 300, it can be identified on the media processing device 200 side by including the transmission time of the RTSP SETUP in the above-mentioned RTSP SETUP.

The user terminal 300 outputs the specific content based on the presentation time (T'). The user terminal 300 may generate specific content with the viewpoint position and line of sight corrected based on the difference between the viewpoint information received from the media processing device 200 and the viewpoint information input from the detection unit 310.

In step S31, the user terminal 300 transmits viewpoint information to the media processing device 200. The viewpoint information may be transmitted in the format of MMT-SI. Here, the user terminal 300 may transmit the viewpoint information at a predetermined interval (e.g., every 500 ms), or may transmit the viewpoint information in response to a change in at least one of the viewpoint position and the line of sight direction.

In step S32, the media processing device 200 generates specific content based on the viewpoint information received in step S31 (rendering process).

In step S33, the media processing device 200 transmits to the user terminal 300 specific content corresponding to the viewpoint information received in step S31.

The processing of steps S31 to S33 is the same as the processing of steps S21 to S23, except that the viewpoint information received in step S31 is used instead of the initial viewpoint information. Therefore, details of the processing of steps S31 to S33 will be omitted. The processing of steps S31 to S33 may be repeated at a predetermined interval, or may be repeated each time the user's viewpoint position or line of sight direction changes.

In step S41, the user terminal 300 sends an RTSP TEARDOWN to the media processing device. The RTSP TEARDOWN is a message indicating that viewing of specific content has ended.

In step S42, the media processing device 200 transmits a response to the RTSP TEARDOWN. Here, an ACK is transmitted as a response indicating that the RTSP TEARDOWN has been accepted.

Figure 4 illustrates an example in which steps S11 and S12 are performed based on RTSP, but the embodiment is not limited to this. Steps S11 and S12 may also be performed based on MMTP or HTTP.

Similarly, although the example illustrates a case in which steps S41 and S42 are performed based on RTSP, the embodiment is not limited to this. Steps S41 and S42 may also be performed based on MMTP or HTTP.

Figure 4 illustrates an example in which steps S31 to S33 are performed based on MMTP, but the embodiment is not limited to this. Steps S31 to S33 may also be performed based on other methods (e.g., HTTP).

Similarly, although the example illustrates a case in which steps S41 to S43 are performed based on MMTP, the embodiment is not limited to this. Steps S41 to S43 may also be performed based on other methods (e.g., HTTP).

(Example 1)
The above-described embodiment may include the following Operation Example 1. In Operation Example 1, media processing device 200 transmits to user terminal 300 viewpoint information used in generating specific content, in association with a sequence number added to the specific content.

Specifically, the media processing device 200 (renderer 220) generates second content including 360° video and 3D objects as part of specific content based on the second control information (scene description), as in the above-described embodiment. In generating the second content, viewpoint information received from the user terminal 300 is used.

In operation example 1, the renderer 220 associates the viewpoint information used in generating the specific content (here, the second content) with a sequence number. The renderer 220 transmits the viewpoint information used in generating the specific content to the user terminal 300, in association with the sequence number added to the specific content. The viewpoint information may be stored in a VP (View Port) message. The VP message may have the format of MMT-SI specified in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc. The VP message may be transmitted for each frame. The VP message may be transmitted to the user terminal 300 as a message related to MMTP (MMT-SI).

Although not particularly limited, a VP message may have the data structure shown in Figure 5. As shown in Figure 5, a VP message may include message_id, version, length, fov, viewpoint_pos_x, viewpoint_pos_y, viewpoint_pos_z, viewpoint_yaw, viewpoint_pitch, viewpoint_roll, viewport_width, viewport_height, mpu_sequence_number_flag, mpu_sequence_number, etc.

message_id is identification information that indicates the VP message. message_id may be 0x0204.

Version is information indicating the version of the MMTP protocol. Version may be 0x00.

Length is information indicating the length of the VP message.

fov is information that indicates the field of view.

viewpoint_pos_x is information indicating the x coordinate of the viewpoint position. viewpoint_pos_x is an example of viewpoint information used in generating specific content.

viewpoint_pos_y is information indicating the y coordinate of the viewpoint position. viewpoint_pos_y is an example of viewpoint information used in generating specific content.

viewpoint_pos_z is information indicating the z coordinate of the viewpoint position. viewpoint_pos_z is an example of viewpoint information used in generating specific content.

viewpoint_yaw is information indicating the yaw of the viewpoint position. viewpoint_yaw is an example of viewpoint information used in generating specific content.

viewpoint_pitch is information indicating the pitch of the viewpoint position. viewpoint_pitch is an example of viewpoint information used in generating specific content.

viewpoint_roll is information indicating the role of the viewpoint position. viewpoint_roll is an example of viewpoint information used in generating specific content.

viewport_width is information that indicates the width of the display area (specific content).

viewport_height is information that indicates the height of the display area (specific content).

mpu_sequence_number_flag is information indicating whether the mpu_sequence_number field exists. For example, if mpu_sequence_number_flag is 1, the mpu_sequence_number field exists, and if mpu_sequence_number_flag is 0, the mpu_sequence_number field may not exist.

mpu_sequence_number is the MPU sequence number of the video corresponding to the specific content indicated by the VP message. mpu_sequence_number is an example of a sequence number added to specific content.

Here, viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z are examples of information elements that indicate the user's viewpoint position in the three-dimensional space configured by the scene description. viewpoint_yaw, viewpoint_pitch, and viewpoint_roll are examples of information elements that indicate the user's line of sight in the three-dimensional space configured by the scene description. viewport_width and viewport_height are examples of information elements that indicate the number of pixels of the image included in specific content.

Under these assumptions, the media processing device 200 and the user terminal 300 may perform the operations described below.

First, the media processing device 200 (renderer 220) may determine viewpoint_pos_x, viewpoint_pos_y, and viewpoint_pos_z based on the viewpoint position used to generate the specific content. The renderer 220 may determine viewpoint_yaw, viewpoint_pitch, and viewpoint_roll based on the line of sight direction used to generate the specific content. The renderer 220 may determine viewport_width based on the number of horizontal pixels of the specific content, and viewport_height based on the number of vertical pixels of the specific content.

The media processing device 200 (encoding processing unit 230) may perform compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding processing unit 230 may transmit the VP message shown in FIG. 5 to the user terminal 300. In other words, the encoding processing unit 230 may transmit the viewpoint information used in generating the specific content to the user terminal 300, in association with the sequence number added to the specific content.

Second, the user terminal 300 (decoding processing unit 320) may be configured as a receiving unit that receives viewpoint information used in generating specific content from the media processing device 200, in association with a sequence number added to the specific content. That is, the decoding processing unit 320 may receive the VP message (MMT-SI) shown in Figure 5 from the media processing device 200.

The user terminal 300 (renderer 330) may associate the video constituting the decoded specific content with the viewpoint information included in the VP message based on the sequence number (mpu_sequence_number). Based on the difference between the viewpoint information detected by the detection unit 310 and the viewpoint information included in the VP message, the renderer 330 may identify a video in the range identified by the viewpoint information detected by the detection unit 310 from the display area defined by the information included in the VP message (viewport_width and viewport_height). The renderer 330 may display the identified video.

(Example 2)
The above-described embodiment may include the following Operation Example 2. In Operation Example 2, as shown in FIG. 6 , a case is assumed in which a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed. The first user terminal 400 may be a terminal such as a head-mounted display. The first user terminal may have the same functions as the above-described user terminal 300. The second user terminal 500 may be a terminal such as a volumetric display.

Specifically, in operation example 2, as shown in FIG. 6, the media processing device 200 (renderer 220) may constitute a receiving unit that receives content configuration and recommended viewport information from the transmitting device 100. The content configuration may be considered to include 2D video, audio, 360° video, and 3D objects. The content configuration may be considered to include MMT-SI and a scene description. The recommended viewport information may be considered to be an example of specific viewpoint information. The recommended viewport information may be information that defines (recommends) the position, direction, and angle of view at which to view video in a three-dimensional space formed by the specific content (a three-dimensional space constructed by the scene description). The recommended viewport information may include an information element that indicates at least one of an information element indicating a viewpoint position in the three-dimensional space formed by the specific content and an information element indicating a line-of-sight direction in the three-dimensional space formed by the specific content. The recommended viewport information may be considered to be viewpoint information primarily used by the second user terminal 500.

In the following, unless explicitly stated otherwise, the viewpoint information used in generating specific content may include specific viewpoint information (recommended viewport information) received from the transmitting device 100, or may include viewpoint information received from the user terminal 300.

Although not particularly limited, the recommended viewport information may be included in the scene description in the manner shown in Figure 7. As shown in Figure 7, the recommended viewport information may include camera_orientation, frame_number, translation, and yfov.

Camera_orientation is information that indicates the direction in which the image is viewed in the three-dimensional space constructed by the scene description. Camera_orientation can be considered synonymous with line of sight direction.

frame_number is information that indicates the frame number of the video to which camera_orientation, translation, and yfov are applied. Camera_orientation can be considered an example of specific viewpoint information.

Translation is information that indicates the position from which the image is viewed in the three-dimensional space constructed by the scene description. Translation may be considered synonymous with viewpoint position. Translation may also be considered an example of specific viewpoint information. For example, Figure 7 illustrates a case where the viewpoint position is [0,0,-50] when the frame number is 0, and moves to [0,0,-75] when the frame number is 2505.

yfov is information that indicates the field of view from which the image is viewed in the three-dimensional space constructed by the scene description.

Although not limited to this, camera_orientation and translation may be assigned by the content creator. Alternatively, if the content is captured by a camera, camera_orientation and translation may be assigned automatically by the camera's GPS and sensors.

Here, translation is an example of an information element that indicates the viewpoint position in a three-dimensional space formed by specific content. camera_orientation is an example of an information element that indicates the line of sight direction in a three-dimensional space formed by specific content.

Under these assumptions, the media processing device 200 and the user terminal 300 may perform the operations described below.

First, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information). The media processing device 200 (encoding processing unit 230) may perform compression encoding of the specific content generated by the renderer 220 and transmit the specific content. Here, the encoding processing unit 230 may transmit the specific content generated based on the specific viewpoint information to the first user terminal 400, or may transmit the specific content generated based on the specific viewpoint information to the second user terminal 500.

Second, the media processing device 200 (reception unit 210) may receive viewpoint information from the first user terminal 400 when the user terminal is a first user terminal 400 that feeds back viewpoint information. The media processing device (renderer 220) may generate specific content based on the viewpoint information received from the first user terminal 400. In such a case, the media processing device 200 (reception unit 210) may receive a reset signal from the first user terminal 400. In response to the reset signal, the media processing device (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information).

In such a case, the first user terminal 400 may have the same configuration as the user terminal 300. However, the first user terminal 400 (detection unit 310) may have the function of detecting a reset signal. The detection unit 310 may detect a user operation that inputs a reset signal. The detection unit 310 may transmit the reset signal to the media processing device 200.

Third, the media processing device 200 (renderer 220) may generate specific content based on specific viewpoint information (recommended viewport information) when the user terminal is a second user terminal 500 that does not feedback viewpoint information.

In such a case, the second user terminal 500 may not have a detection unit 310 that detects viewpoint information. The second user terminal 500 may not have a renderer 330. The second user terminal 500 may have the same configuration as the user terminal 300, except that it does not have the detection unit 310 or the renderer 330.

(Example 3)
The above-described embodiment may include the following Operation Example 3. In Operation Example 3, media processing device 200 (for example, selection unit 260, described below) may be configured as a receiving unit that receives, from transmitting device 100, quality information about each of two or more streams whose quality varies depending on the orientation of the 3D object based on viewpoint information, as quality information about the stream related to the 3D object included in specific content.

Specifically, in operation example 3, as shown in FIG. 8, the media processing device 200 has a selection unit 260 in addition to the configuration shown in FIG. 2. The selection unit 260 receives a scene description and a 3D object from the transmitting device 100. The selection unit 260 inputs a stream (3D object) selected from two or more streams to the renderer 220. The selection unit 260 may request the transmitting device 100 to transmit the selected stream. Note that, in a case where the media processing device 200 transmits specific content to multiple user terminals 300, the selection unit 260 may request the transmitting device 100 to transmit the stream required by each of the multiple user terminals 300, or may request the transmitting device 100 to transmit all streams.

Here, the selection unit 260 receives, from the transmitting device 100, quality information regarding each of two or more streams whose quality differs depending on the orientation of the 3D object based on the viewpoint information, as quality information of the stream regarding the 3D object.

The quality information may be information indicating the relative quality of each face that constitutes a bounding box for a 3D object. The bounding box may be represented by a three-dimensional rectangle onto which the 3D object is projected. For example, as shown in Figure 9, the bounding box may be defined by vertices A through H. In such a case, the faces of the bounding box include face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; face #3 represented by vertices A, B, C, and D; face #4 represented by vertices E, F, G, and H; face #5 represented by vertices A, D, H, and E; and face #6 represented by vertices D, C, G, and H.

In such a case, when we imagine viewing a 3D object in the 3D space constructed by the scene description, we assume that three sides are primarily observed. In other words, we assume that the remaining three sides are not observed very often.

In operation example 3, two or more streams related to 3D objects contained in specific content are prepared, with the quality varying depending on the orientation of the 3D object based on viewpoint information.

While not particularly limited, quality information may be included in the scene description in the manner shown in Figure 10. Figure 10 illustrates six streams with different 3D object orientations. The quality information may be expressed in the format "quality" [#1,#2,#3,#4,#5,#6]. Note that within [ ], #1 to #6 represent the quality indexes of surfaces #1 to #6. The quality index may take values ranging from 1 to 9. A higher quality index value may indicate higher quality. For example, in the stream identified by "id"="1", #1, #2, and #3 have high quality ("8"), while #4, #5, and #6 have low quality ("3"). In the stream identified by "id"="2", #1, #2, and #3 have low quality ("3"), while #4, #5, and #6 have high quality ("8").

Under these conditions, the media processing device 200 may perform the following operations. The following mainly describes the selection of a stream (3D object) from among two or more streams.

In mode 1, as shown in the upper part of Figure 11, the media processing device 200 (selection unit 260) may identify the vertex closest to the user's viewpoint (e.g., vertex B), and then identify the three faces that have the closest vertex (e.g., face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; and face #3 represented by vertices A, B, C, and D). The selection unit 260 may select the stream with the largest sum of the quality indexes of the three identified faces (in the example shown in Figure 10, the stream identified by "id"="1").

In addition, in mode 1, since the vertex closest to the user's viewpoint position is identified based on viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from two or more streams based on viewpoint information and quality information.

Mode 2 may be applied when a 3D object is scaled down or scaled up. For example, as shown in the middle of FIG. 11 , the media processing device 200 (selection unit 260) may identify the vertex closest to the user's viewpoint (e.g., vertex B) and then identify the three faces that have the closest vertex (e.g., face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; and face #3 represented by vertices A, B, C, and D). When a 3D object is scaled down, pixels of the 3D object are thinned out. Therefore, the selection unit 260 may select the stream with the smallest sum of the quality indexes of the three identified faces (in the example shown in FIG. 10 , the stream identified by "id"="2"). On the other hand, when a 3D object is scaled up, pixels of the 3D object are interpolated. Therefore, the selection unit 260 may select the stream with the largest sum of the quality indexes of the three identified faces (in the example shown in FIG. 10 , the stream identified by "id"="1").

In addition, in mode 2, since the vertex closest to the user's viewpoint position is identified based on viewpoint information, it can be considered that the selection unit 260 selects the stream to be transmitted to the user terminal 300 from two or more streams based on viewpoint information and quality information.

Mode 3 may be applied when two 3D objects (3D object #1 and 3D object #2) overlap in the user's line of sight. Here, stream selection for 3D object #1 will be described. For example, as shown in the lower part of Figure 11, the media processing device 200 (selection unit 260) may identify the vertex closest to the user's viewpoint (e.g., vertex B) and then identify three faces that share the closest vertex (e.g., face #1 represented by vertices A, B, F, and E; face #2 represented by vertices B, C, G, and F; and face #3 represented by vertices A, B, C, and D). Here, 3D object #2 overlaps the line connecting the vertex closest to the user's viewpoint (e.g., vertex B) and the user's viewpoint, and the identified three faces are obscured by 3D object #2. Therefore, the selection unit 260 may select the stream with the smallest sum of the quality indexes of the identified three faces (in the example shown in Figure 10, the stream identified by "id" = "2").

Note that in mode 3, the vertex closest to the user's viewpoint position is identified based on viewpoint information, and therefore the selection unit 260 may be considered to select from two or more streams the stream to be transmitted to the user terminal 300 based on viewpoint information and quality information. Furthermore, in mode 3, the overlap of two 3D objects is identified based on viewpoint information and 3D object placement information, and therefore the selection unit 260 may be considered to select from two or more streams the stream to be transmitted to the user terminal 300 based on viewpoint information, quality information, and placement information. 3D object placement information (e.g., "rotation_object," "scale_object," and "translation_object" shown in FIG. 10) may be included in the scene description. The 3D object placement information may be considered to be "link_area" shown in FIG. 10. That is, the selection unit 260 may receive 3D object placement information in three-dimensional space from the transmission device 100.

Note that in the quality information shown in Figure 10, the sum of the quality indexes of the six aspects is equal for each stream. However, the embodiment is not limited to this. The sum of the quality indexes of the six aspects may differ between two or more streams.

(Example 4)
The above-described embodiment may include the following Operation Example 4. Here, Operation Example 4 includes the following operations in addition to Operation Example 3. In Operation Example 4, media processing device 200 (e.g., selection unit 270, described below) may constitute a receiving unit that receives importance information for each of two or more objects included in specific content from transmitting device 100.

Specifically, in operation example 4, as shown in FIG. 12, the media processing device 200 has a selection unit 270 in addition to the configuration shown in FIG. 2. The selection unit 270 receives a scene description, a 3D object, and a 360° video from the transmission device 100. The selection unit 270 inputs a stream (3D object) selected from two or more streams to the renderer 220.

Here, the selection unit 270 receives importance information regarding each of two or more objects from the transmitting device 100. The objects may include 3D objects and 360° video.

For example, the importance information may be information indicating the relative importance between two or more objects. For example, consider a case in which object A (background), object B (person), and object C (dog) exist in a three-dimensional space constructed by a scene description, as shown in Figure 13. Object A (background) is an example of a 360° image, and object B (person) and object C (dog) are examples of 3D objects. In such a case, the importance information may be information indicating the relative importance between object A (background), object B (person), and object C (dog).

Although not particularly limited, importance information may be included in the scene description in the manner shown in Figure 14. In Figure 14, importance information may be represented by weight. Weight may take on values ranging from 1 to 9. A larger weight value may indicate higher importance. Figure 14 illustrates a case in which object A (background) identified by "object_id"="0" has the highest weight ("9"), object B (person) identified by "object_id"="1" has the lowest weight ("3"), and object C (dog) identified by "object_id"="2" has a weight ("8") higher than the weight of object B (person) but lower than the weight of object A (background).

Under these conditions, the media processing device 200 may perform the following operations. The following mainly describes the selection of a stream (3D object) from among two or more streams.

First, the media processing device 200 (selection unit 270) selects the stream with the highest quality for the 3D object with the highest importance. The stream selection method may be the same as in operation example 3. For example, because the importance of object C (dog) is greater than the importance of object B (person), the selection unit 270 selects the stream for object C (dog) with the largest sum of the quality indexes of the three faces with the vertex closest to the user's viewpoint.

Second, the media processing device 200 (selection unit 270) selects the lowest-quality stream for 3D objects other than the 3D object with the highest importance. Next, the selection unit 270 replaces the lowest-quality stream with a higher-quality stream for 3D objects in descending order of importance, within the scope in which specific conditions are satisfied. The specific conditions may include a first condition that the bandwidth of the line from the transmitting device 100 to the media processing device 200 is equal to or less than a threshold, and a second condition that the processing load of the media processing device 200 is equal to or less than a threshold. The specific conditions may be defined by a combination of the first and second conditions. For example, because the importance of object B (person) is lower than the importance of object C (dog), the selection unit 270 selects a high-quality stream for object B (person) within the scope in which the specific conditions are satisfied.

As described above, the media processing device 200 (selection unit 270) may be considered to select a stream to be transmitted to the user terminal 300 from among two or more streams based on viewpoint information, quality information, and importance information. The media processing device 200 (selection unit 270) may be considered to select a stream to be transmitted to the user terminal 300 from among two or more streams based on viewpoint information, quality information, placement information, and importance information.

In operation example 4, a case where one stream exists for 360° video is illustrated. However, the embodiment is not limited to this. Two or more streams of different qualities may exist for 360° video as well.

Operation example 4 illustrates a case where, for a 3D object, there are two or more streams whose quality varies depending on the orientation of the 3D object based on viewpoint information. However, the embodiment is not limited to this. For a 3D object, there may be two or more streams whose quality varies regardless of the orientation of the 3D object.

(Example 5)
The above-described embodiment may include the following Operation Example 3. In Operation Example 3, the media processing device 200 (e.g., the renderer 220) may constitute a receiving unit that receives, from the transmitting device 100, information elements that define a range of movement of the user's viewpoint position in a three-dimensional space configured by specific content (a three-dimensional space constructed by a scene description).

First, in operation example 5, the information element may include an information element (hereinafter referred to as the first information element) that restricts movement of the user's viewpoint inside a 3D object included in specific content. For example, as shown in FIG. 15, in cases where a 3D object is placed in a three-dimensional space constructed by a scene description, movement of the viewpoint inside the 3D object may be restricted. However, in cases where the 3D object is a building or where another scene exists inside the 3D object, movement of the viewpoint inside the 3D object may be permitted.

Second, in operation example 5, the information element may include an information element (hereinafter referred to as the second information element) that restricts movement of the user's viewpoint outside the three-dimensional space. For example, as shown in FIG. 16, the three-dimensional space may be defined by a combination of a rectangular parallelepiped and a spheroid. The number of rectangular parallelepipeds that define the three-dimensional space may be two or more, and the number of spheroids that define the three-dimensional space may be two or more. However, there may be cases in which movement of the user's viewpoint outside the three-dimensional space is permitted.

Although not particularly limited, the first information element may be included in the scene description in the manner shown in Figure 17. In Figure 17, the first information element may be represented by viewing_inside_object_flag. The viewing_inside_object_flag may be set for each 3D object. For example, when viewing_inside_object_flag is "0", movement of the viewpoint position into the 3D object may be restricted, and when viewing_inside_object_flag is "1", movement of the viewpoint position into the 3D object may be permitted.

Although not particularly limited, the second information element may be included in the scene description in the manner shown in FIG. 18. In FIG. 18, the second information element may include information elements (cuboid_center_x, cuboid_center_y, cuboid_center_z, cuboid_size_x, cuboid_size_y, cuboid_size_z) that define a rectangular parallelepiped that defines a three-dimensional space. cuboid_center_x, cuboid_center_y, and cuboid_center_z are information elements that indicate the center position of the rectangular parallelepiped, and cuboid_size_x, cuboid_size_y, and cuboid_size_z are information elements that indicate the size of the rectangular parallelepiped. The second information element may also include information elements (spheroid_center_x, spheroid_center_y, spheroid_center_z, spheroid_size_x, spheroid_size_y, spheroid_size_z) that define a spheroid that defines the three-dimensional space. spheroid_center_x, spheroid_center_y, and spheroid_center_z are information elements indicating the center position of a spheroid, and spheroid_size_x, spheroid_size_y, and spheroid_size_z are information elements indicating the size of the spheroid. Note that cuboid_enable may be an information element indicating whether a three-dimensional space is defined by a rectangular parallelepiped, and spheroid_enable may be an information element indicating whether a three-dimensional space is defined by a spheroid. Figure 18 illustrates an example in which a three-dimensional space is defined by two rectangular parallelepipeds and two spheroids.

Under these assumptions, the media processing device 200 (renderer 220) may perform the following operations.

First, when the user's viewpoint position moves outside the movement range, the renderer 220 may generate specific content by using the intersection of the trajectory of the user's viewpoint position and the boundary of the movement range as the viewpoint position. In other words, the renderer 220 may fix the viewpoint position at the position (boundary position) at the time when the viewpoint position attempts to move outside the movement range.

Second, when the user's viewpoint moves outside the movement range, the renderer 220 may notify the user that movement of the viewpoint is restricted. For example, the renderer 220 may display a message such as "You cannot move beyond this point."

(Action and effect)
In the embodiment, media processing device 200 generates specific content based on viewpoint information and then transmits the specific content to user terminal 300. With this configuration, there is no need for user terminal 300 to generate specific content that includes second content with viewpoint flexibility; user terminal 300 can present the specific content simply by providing viewpoint information to media processing device 200. Therefore, although a delay occurs between media processing device 200 and user terminal 300, the processing load on user terminal 300 can be reduced.

In operation example 1, the media processing device 200 transmits the viewpoint information used to generate the specific content to the user terminal 300 in association with the sequence number added to the specific content. With this configuration, the user terminal 300 can grasp the viewpoint information and sequence number used to generate the specific content, and therefore can appropriately display the specific content even in cases where the viewpoint information used to generate the specific content by the media processing device 200 differs from the viewpoint information used to display the specific content on the user terminal 300.

In operation example 2, the media processing device 200 receives the content configuration and specific viewpoint information (recommended viewport information) from the transmitting device 100. With this configuration, the media processing device 200 can generate specific content based on the specific viewpoint information, and can appropriately display the specific content even in a case where a first user terminal 400 that feeds back viewpoint information and a second user terminal 500 that does not feed back viewpoint information are mixed.

In operation example 2, the media processing device 200 generates specific content based on specific viewpoint information in response to a reset signal, even when the user terminal is the first user terminal 400. With this configuration, even in cases where the viewpoint position and line of sight direction become unknown on the first user terminal 400 in the three-dimensional space constructed by the scene description (a case where the user becomes lost in three-dimensional space), the reset signal can return the user to specific content based on specific viewpoint information.

In operation example 3, the media processing device 200 receives, from the transmitting device 100, quality information for each of two or more streams whose quality varies depending on the orientation of the 3D object based on viewpoint information, as quality information for streams related to a 3D object included in specific content. This configuration, based on the new knowledge that the quality of each face constituting the bounding box of a 3D object does not need to be uniform, makes it possible to appropriately display the 3D object while suppressing transmission traffic.

In operation example 4, the media processing device 200 receives importance information for each of two or more objects included in specific content from the transmitting device 100. With this configuration, by introducing a mechanism for setting the importance of each object, such as 360° video and 3D objects, it is possible to appropriately display each object included in the specific content while suppressing transmission traffic.

In operation example 5, the media processing device 200 receives from the transmitting device 100 information elements that define the range of movement of the user's viewpoint position in the three-dimensional space formed by the specific content. With this configuration, specific content including content with viewpoint flexibility can be appropriately displayed without causing a breakdown in the specific content displayed on the user terminal 300.

[Change example 1]
Modification 1 of the embodiment will be described below, focusing mainly on the differences from the embodiment.

In Modification Example 1, we explain a method for synchronizing a first content with a second content when a specific content includes both a first content and a second content.

In the following, synchronization means that the presentation times of the first content (e.g., MPU) and the second content (file) are properly synchronized. Therefore, synchronization may include the synchronization of the presentation times of the 2D video and the 3D object, and may also include the synchronization of the presentation times of the audio and the 3D object. Similarly, synchronization may include the synchronization of the presentation times of the 2D video and the 360° video, and may also include the synchronization of the presentation times of the audio and the 360° video.

In the first method, we will explain a case where the media processing device 200 synchronizes the first content and the second content based on the first control information (MMT-SI). Using the MMT-SI as an entry point, the media processing device 200 checks whether or not a scene description (second content) is present, and if a scene description is present, it uses the MPU timestamp descriptor to identify the presentation time of specific content including the first content and the second content.

Specifically, as shown in Figure 19, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in Figure 19), allowing synchronization of 2D video and audio.

On the other hand, the presentation time of the first frame included in the scene description is identified by referencing the MPU timestamp descriptor included in MMT-SI. The presentation times of the second and subsequent frames included in the scene description can be identified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is identified by adding 1/30 x n to the time identified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

In the first method, an example was given of a case in which the scene description does not include the presentation time of the first frame included in the scene description, but the scene description may also include the presentation time of the first frame included in the scene description.

In the second method, we will explain a case in which the media processing device 200 synchronizes the first content and the second content based on the second control information (scene description). Using the scene description as an entry point, the media processing device 200 checks whether or not MMT-SI (first content) is present, and if MMT-SI is present, determines the presentation time of specific content, including the first content and the second content, based on the presentation time included in the scene description.

In such a case, the scene description includes absolute time information indicating the presentation time of the second content. The absolute time information may be the presentation time of the first frame included in the scene description.

For example, absolute time information may be generated using UTC as the reference time. The reference time may be TAI or the time provided by GPS. The reference time may be the time provided by an NTP server or the time provided by a PTP server. Furthermore, absolute time information may be generated based on the same reference time as the MPU timestamp descriptor.

Furthermore, the scene description includes reference information for identifying the first content. The reference information may be information for identifying the MPU that constitutes the first content. In other words, the reference information is information for treating the first content (MPU) as an object included in the scene description.

Specifically, as shown in Figure 20, the presentation time of the first frame included in the scene description is specified by the absolute time information included in the scene description. The presentation times of the second and subsequent frames included in the scene description can be specified by the frame number included in the scene description and the frame rate of the second content. For example, if the frame rate is 30 fps, the presentation time of the nth frame is specified by adding 1/30 x n to the time specified by the MPU timestamp descriptor. However, the frame number of the first frame included in the scene description is "0".

On the other hand, 2D video and audio are presented based on the MPU timestamp descriptor (simply "timestamp" in Figure 20), allowing synchronization of the 2D video and audio. Here, because the above-mentioned reference information is included in the scene description, the media processing device 200 can check whether there is a first content to be presented together with the second content, based on the reference information included in the scene description.

In the second method, synchronization between 2D video and audio is achieved based on the MPU timestamp descriptor included in the MMT-SI, but in modification example 1, synchronization between 2D video and audio may also be achieved based on information elements included in the scene description (absolute time information and reference information). In such a case, at least the MPU timestamp descriptor included in the MMT-SI may be omitted. Furthermore, the MMT-SI itself may be omitted.

Note that if the reference time of the MPU timestamp descriptor included in the MMT-SI (hereinafter referred to as the first reference time) differs from the reference time of the absolute time information included in the scene description (the second reference time), at least one of the first control information (MMT-SI) and the second control information (scene description) may include conversion information between the first reference time and the second reference time. For example, the MMT-SI may include an MPU timestamp descriptor expressed in the second reference time (e.g., a reference time other than UTC) in addition to an MPU timestamp descriptor expressed in the first reference time (e.g., UTC). The scene description may include absolute time information expressed in the first reference time (e.g., UTC) in addition to absolute time information expressed in the second reference time (e.g., a reference time other than UTC).

In addition, the MPU timestamp descriptor included in the MMT-SI may be referred to as first absolute time information, and the absolute time information included in the scene description may be referred to as second absolute time information.

[Other embodiments]
Although the present invention has been described by the above disclosure, the descriptions and drawings that form part of this disclosure should not be understood as limiting the present invention. From this disclosure, various alternative embodiments, examples, and operating techniques will become apparent to those skilled in the art.

The above disclosure provides an example of a case in which the specific content includes both the first content and the second content, but the above disclosure is not limited to this. The specific content may include at least the second content.

Although not specifically mentioned in the above disclosure, terms related to MMT may be interpreted based on the contents specified in ISO/IEC 23008-1, ARIB STD-B60, ARIB TR-B39, etc.

In the above disclosure, an MPU timestamp descriptor is exemplified as the first absolute time information included in MMT-SI. However, the above disclosure is not limited to this. The first absolute time information included in MMT-SI may also be an MPU extended timestamp descriptor.

Although not specifically mentioned in the above disclosure, the media processing device 200 may request a portion of the second content from the transmitting device 100 as needed. This configuration can save bandwidth associated with the transmission of the second content and suppress an increase in the processing load on the media processing device 200.

In the above disclosure, MMTP is exemplified as the transmission method for the first content. However, the above disclosure is not limited to this. The transmission method for the first content may be a method that complies with ISO/IEC 23009-1 (hereinafter, MPEG-DASH (Dynamic Adaptive Stream over HTTP)). In such a case, the first control information may be MPD (Media Presentation Description). In other words, in the above disclosure, MMT-SI may be read as MPD.

Although not specifically mentioned in the above disclosure, "acquire" may also be read as "receive."

Although not particularly limited, operation example 2 may be expressed as follows: The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits specific viewpoint information used to generate specific content that includes at least the content. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives the specific viewpoint information used to generate specific content that includes at least the content. In such a case, the receiving device may be a media processing device 200 or a user terminal 300.

Although not particularly limited, operation example 3 may be expressed as follows: The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits quality information of a stream related to a three-dimensional object included in specific content that includes at least the content, where the quality information includes quality information for each of two or more streams whose quality varies depending on the orientation of the three-dimensional object. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives quality information of a stream related to a three-dimensional object included in specific content that includes at least the content, where the quality information includes quality information for each of two or more streams whose quality varies depending on the orientation of the three-dimensional object. In such a case, the receiving device may be a media processing device 200 or a user terminal 300.

Although not particularly limited, operation example 4 may be expressed as follows: The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits importance information regarding each of two or more objects included in specific content that at least includes the content. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives importance information regarding each of two or more objects included in specific content that at least includes the content. In such a case, the receiving device may be a media processing device 200 or a user terminal 300.

Although not particularly limited, operation example 4 may be expressed as follows: The transmitting device 100 includes a transmitting unit that transmits a content configuration with viewpoint freedom, and the transmitting unit transmits information elements that define a range of movement of a user's viewpoint position in a three-dimensional space configured by specific content that at least includes the content. The receiving device includes a receiving unit that receives a content configuration with viewpoint freedom, and the receiving unit receives the information elements that define a range of movement of a user's viewpoint position in a three-dimensional space configured by specific content that at least includes the content. In such a case, the receiving device may be a media processing device 200 or a user terminal 300.

Although not specifically mentioned in the above disclosure, a program may be provided that causes a computer to execute the processes performed by the transmitting device 100, the media processing device 200, and the user terminal 300. The program may also be recorded on a computer-readable medium. The computer-readable medium can be used to install the program on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transitory recording medium. The non-transitory recording medium is not particularly limited, but may be, for example, a CD-ROM, DVD-ROM, or other recording medium.

Alternatively, a chip may be provided that is configured from a memory that stores programs for executing each process performed by the transmitting device 100, the media processing device 200, and the user terminal 300, and a processor that executes the programs stored in the memory.

The above disclosure may have the following problems and effects.

Specifically, user terminals that display content may include a first user terminal that feeds back viewpoint information, such as a head-mounted display, and a second user terminal that does not feed back viewpoint information, such as a volumetric display.

After careful consideration, the inventors discovered that when considering a media device capable of generating specific content, including content with freedom of viewpoint, a mechanism is needed that can appropriately display specific content even in a situation where a first user terminal and a second user terminal are mixed.

The above disclosure makes it possible to provide a media processing device, a transmitting device, and a receiving device that enable the appropriate display of specific content, including content with viewpoint freedom.

This application claims priority from Japanese Patent Application No. 2022-019710 (filed February 10, 2022), the entire contents of which are incorporated herein by reference.

10...Transmission system, 100...Transmitting device, 200...Media processing device, 210...Receiving unit, 220...Renderer, 230...Encoding processing unit, 260...Selecting unit, 270...Selecting unit, 300...User terminal, 310...Detecting unit, 320...Decoding processing unit, 330...Renderer, 400...First user terminal, 500...Second user terminal

Claims (5)

a renderer that generates specific content including at least the content based on a configuration of the content having freedom of viewpoint and specific viewpoint information;
an output unit that outputs the specific content generated by the renderer to a user terminal;
an acquisition unit that acquires capability information of the user terminal,
The renderer generates the specific content according to capability information of the user terminal ;
a receiving unit configured to receive viewpoint information from the first user terminal when the user terminal is a first user terminal that feeds back viewpoint information;
The renderer generates the specific content based on the viewpoint information;
The output unit outputs the viewpoint information used to generate the specific content, as well as the specific content generated based on the viewpoint information, to the first user terminal . The media processing device of claim 1, wherein the specific viewpoint information includes at least one of an information element indicating a viewpoint position in a three-dimensional space formed by the specific content and an information element indicating a line-of-sight direction in the three-dimensional space. a receiving unit that receives a reset signal from the first user terminal;
The media processing device of claim 1 , wherein the renderer generates the specific content based on the specific viewpoint information in response to the reset signal. 4. The media processing device according to claim 1 , wherein the renderer generates the specific content based on the specific viewpoint information when the user terminal is a second user terminal that does not feed back viewpoint information. 5. The media processing device according to claim 1, wherein the capability information of the user terminal includes information elements indicating one or more parameters selected from a frame rate of the user terminal, a display resolution of the user terminal, an encoding method of the user terminal , or a compression method of the user terminal.