JP7740333B2 - Information processing device and information processing method - Google Patents

Description

This technology relates to an information processing device and information processing method that can be applied to the distribution of VR (Virtual Reality) images, etc.

In recent years, panoramic images that are captured by panoramic cameras and allow viewing in all directions have begun to be distributed as VR images. Furthermore, technology for distributing 6DoF (Degree of Freedom) images (also referred to as 6DoF content) that allow viewers (users) to view in all directions (freely select the viewing direction) and move freely in three-dimensional space (freely select the viewpoint position) has been developed.
Such 6DoF content dynamically reproduces a three-dimensional space using one or more three-dimensional objects according to the viewer's viewpoint position, line of sight direction, and viewing angle (viewing range) at each time.
In such video distribution, it is necessary to dynamically adjust (render) the video data presented to the viewer according to the viewer's field of view. For example, a technique disclosed in Patent Literature 1 can be cited as an example of such a technique.

Non-Patent Document 1 describes a process for estimating a saliency map for a spherical image.
In this estimation process, planar images from various camera directions are extracted from a spherical image, and a saliency map for each planar image is estimated using a saliency map estimation model for planar images. The saliency maps for each planar image are integrated, and a horizon bias is applied to the horizontal line direction at the center of the image, to estimate a saliency map for the spherical image.

Special Publication No. 2007-520925

Takao Yamanaka, "Technology for Estimating Prominent Locations in Images: Saliency Map Estimation Using Deep Learning," [online], September 15, 2020, FY2020 New Technology Briefing, Internet <URL: https://shingi.jst.go.jp/var/rev0/0001/1222/02_sophia_yamanaka.pdf>

The distribution of virtual images (virtual videos) such as VR videos is expected to become more widespread, and there is a demand for technology that enables the distribution of high-quality virtual videos.

In light of the above circumstances, the purpose of this technology is to provide an information processing device and information processing method that are capable of delivering high-quality virtual images.

In order to achieve the above object, an information processing device according to an embodiment of the present technology includes a rendering unit, an estimation unit, and a generation unit.
The rendering unit generates two-dimensional video data according to the user's field of view by performing a rendering process on three-dimensional space data that constitutes a virtual space based on field of view information relating to the user's field of view.
The estimation unit estimates a recognition position, recognized by the user, of a recognition target object recognized by the user in an out-of-field area of the virtual space that is not included in the user's field of view.
The generation unit generates a saliency map representing saliency in the out-of-field area based on the estimated recognition position of the recognition target object in the out-of-field area.

In this information processing device, the recognition position of the object to be recognized in the out-of-field area is estimated. A saliency map for the out-of-field area is generated based on the estimated recognition position. This makes it possible to generate a highly accurate saliency map for the out-of-field area, and to use the saliency map to deliver high-quality virtual video.

The estimation unit may set an object that has been the subject of rendering up to the current time as the object to be recognized.

The two-dimensional video data may be composed of a plurality of frame images that are consecutive in time series. In this case, the estimation unit may estimate the recognition position of the recognition target object that is not included in the frame image at the current time based on a position in the virtual space that corresponds to the position of the recognition target object in the most recent past frame image that includes the recognition target object.

The estimation unit may estimate as the recognition position a position in the virtual space corresponding to the position of the object to be recognized in the most recent frame image.

The estimation unit may estimate, as the recognition position, a position shifted along the movement direction of the object to be recognized from a position in the virtual space corresponding to the position of the object to be recognized in the most recent frame image.

When the estimation unit determines that the user has recognized a sound emitted by a recognition target object that is not included in the frame image at the current time, the estimation unit may estimate the position where the sound originated in the virtual space as the recognition position.

The three-dimensional space data may include three-dimensional space description data that defines the configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space. In this case, the three-dimensional space description data may include role information that represents the role of the recognition target object and fixed position information that represents a fixed position associated with the role. Furthermore, for a recognition target object that has specified role information set and is not included in the frame image at the current time, if a recognition target object that has the same role information has been rendered up to the current time, the estimation unit may estimate the fixed position associated with the role as the recognition position.

The estimation unit may estimate the fixed position associated with the role as the recognition position if the object to be recognized, to which the same role information has been set, has been rendered in the fixed position associated with the role up to the current time.

The three-dimensional space data may include three-dimensional space description data that defines the configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space. In this case, the three-dimensional space description data may include role information that represents the role of the recognition target object and fixed position information that represents a fixed position associated with the role. Furthermore, when the estimation unit determines that the user has recognized a sound emitted by a recognition target object that has predetermined role information set and is not included in the frame image at the current time, the estimation unit may estimate the fixed position associated with the role as the recognition position.

The estimation unit may estimate the fixed position associated with the role as the recognized position when it determines that the user recognized a sound made by the object to be recognized, to which the same role information has been set up until the current time, while the object is in the fixed position associated with the role.

The estimation unit may estimate the recognition position based on the position of the object to be recognized within the two-dimensional video data in which the object to be recognized is rendered.

The generation unit may generate the saliency map in which the saliency based on bottom-up attention in the out-of-field area is zero.

The generation unit may generate the saliency map representing saliency based on top-down attention in the out-of-field area based on the recognition position of the object to be recognized in the out-of-field area.

The generation unit may generate the saliency map for the out-of-field area and a saliency map representing the saliency of the two-dimensional video data.

The information processing device may further include a prediction unit that generates future field of view information as predicted field of view information based on the saliency map. In this case, the rendering unit may generate the two-dimensional video data based on the predicted field of view information.

The field of view information may include at least one of the position of the viewpoint, the direction of gaze, the rotation angle of the gaze, the position of the user's head, or the rotation angle of the user's head.

The field of view information may include the user's head rotation angle. In this case, the prediction unit may predict the user's future head rotation angle based on the saliency map.

The two-dimensional video data may be composed of a plurality of frame images that are consecutive in time series. In this case, the rendering unit may generate frame images based on the predicted field of view information and output them as predicted frame images.

An information processing method according to one embodiment of the present technology is an information processing method executed by a computer system, and includes generating two-dimensional video data corresponding to a user's field of view by performing a rendering process on three-dimensional spatial data that constitutes a virtual space based on field of view information related to the user's field of view.
A recognition position of a recognition target object recognized by the user in an out-of-field area of the virtual space that is not included in the user's field of view is estimated.
A saliency map representing saliency in the out-of-field area is generated based on the estimated recognition position of the object to be recognized in the out-of-field area.

FIG. 1 is a schematic diagram illustrating an example of the basic configuration of a server-side rendering system. FIG. 10 is a schematic diagram illustrating an example of a virtual video that can be viewed by a user. FIG. 10 is a schematic diagram for explaining a rendering process. FIG. 1 is a schematic diagram illustrating an example of the configuration of a server-side rendering system. FIG. 2 is a schematic diagram for explaining a recognition target object and a recognition position. 10 is a flowchart illustrating an example of generating a rendering video. FIG. 7 is a diagram for explaining the flowchart shown in FIG. 6, and is a schematic diagram showing the timing of obtaining and generating each piece of information. FIG. 1 is a schematic diagram illustrating an example of generating a saliency map based on bottom-up attention. FIG. 10 is a schematic diagram for explaining a problem with the panoramic saliency map of the comparative example. FIG. 10 is a schematic diagram showing an example of information described in a scene description file used as scene description information in the second embodiment. FIG. 11 is a schematic diagram showing an example of information described in a scene description file used as scene description information in the third embodiment. FIG. 11 is a schematic diagram showing an example of information described in a scene description file used as scene description information in the third embodiment. FIG. 11 is a schematic diagram for explaining estimation of a recognition position of a recognition target object in Example 3. FIG. 11 is a schematic diagram for explaining estimation of a recognition position of a recognition target object in Example 3. 10 is a flowchart illustrating an example of estimating the recognition position of a recognition target object. 10 is a flowchart illustrating an example of generating a panoramic saliency map. FIG. 10 is a schematic diagram illustrating an example of a panoramic saliency map. FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer (information processing device) that can realize the server device and the client device.

Below, an embodiment of the present technology is described with reference to the drawings.

[Server-side rendering system]
As an embodiment of the present technology, a server-side rendering system is configured. First, an example of the basic configuration and basic operation of the server-side rendering system will be described with reference to FIGS.
FIG. 1 is a schematic diagram showing an example of the basic configuration of a server-side rendering system.
FIG. 2 is a schematic diagram for explaining an example of a virtual video that can be viewed by a user.
FIG. 3 is a schematic diagram for explaining the rendering process.
The server-side rendering system can also be called a server-rendering type media distribution system.

As shown in FIG. 1, the server-side rendering system 1 includes an HMD (Head Mounted Display) 2 , a client device 3 , and a server device 4 .
The HMD 2 is a device used to display a virtual image to the user 5. The HMD 2 is worn on the head of the user 5 when in use.
For example, when VR video is distributed as virtual video, an immersive HMD 2 configured to cover the field of view of the user 5 is used.
When an AR (Augmented Reality) image is distributed as the virtual image, AR glasses or the like are used as the HMD 2 .
A device other than the HMD 2 may be used as a device for providing a virtual image to the user 5. For example, the virtual image may be displayed on a display provided on a television, a smartphone, a tablet terminal, a PC (Personal Computer), or the like.

As shown in FIG. 2, in this embodiment, 6DoF images are provided as VR images to a user 5 wearing an immersive HMD 2.
Within the virtual space S, which is a three-dimensional space, the user 5 can view images in a 360° range around them, including front, back, left, right, and up and down. For example, within the virtual space S, the user 5 can freely change his or her field of view (field of view) 7 by freely moving the position of the viewpoint and the direction of line of sight. In response to this change in the field of view 7 of the user 5, the image 8 displayed to the user 5 is switched. By performing actions such as turning the face, tilting the face, and looking back, the user 5 can view the surroundings within the virtual space S with a sensation similar to that of the real world.
In this way, the server-side rendering system 1 according to this embodiment makes it possible to deliver photorealistic free viewpoint video, and to provide a viewing experience from any viewpoint position.

In this embodiment, the HMD 2 acquires visual field information.
The visual field information is information relating to the visual field 7 of the user 5. Specifically, the visual field information includes any information that can identify the visual field 7 of the user 5 within the virtual space S.
For example, the visual field information may include the position of the viewpoint, the direction of the line of sight, the rotation angle of the line of sight, etc. Furthermore, the visual field information may include the position of the head of the user 5, the rotation angle of the head of the user 5, etc.
The rotation angle of the line of sight can be defined by, for example, a rotation angle about an axis extending in the line of sight direction, and the rotation angle of the head of the user 5 can be defined by a roll angle, a pitch angle, and a yaw angle when three mutually perpendicular axes set on the head are defined as a roll axis, a pitch axis, and a yaw axis.
For example, the axis extending in the direction of the face is defined as the roll axis. When the face of the user 5 is viewed from the front, the axis extending in the left-right direction is defined as the pitch axis, and the axis extending in the up-down direction is defined as the yaw axis. The roll angle, pitch angle, and yaw angle relative to these roll axis, pitch axis, and yaw axis are calculated as the rotation angle of the head. Note that the direction of the roll axis can also be used as the line of sight direction.
Any other information may be used that can identify the field of view of the user 5. As the field of view information, one of the above-exemplified pieces of information may be used, or a combination of a plurality of pieces of information may be used.

The method for acquiring the visual field information is not limited. For example, the visual field information can be acquired based on a detection result (sensing result) by a sensor device (including a camera) provided in the HMD 2.
For example, the HMD 2 is provided with a camera or distance measurement sensor that detects the area around the user 5, an inward-facing camera that can capture images of the left and right eyes of the user 5, etc. The HMD 2 is also provided with an IMU (Inertial Measurement Unit) sensor and a GPS.
For example, the position information of the HMD 2 acquired by GPS can be used as the viewpoint position of the user 5 or the head position of the user 5. Of course, the positions of the left and right eyes of the user 5 may be calculated in more detail.
It is also possible to detect the direction of the line of sight from captured images of the left and right eyes of the user 5 .
It is also possible to detect the rotation angle of the line of sight and the rotation angle of the head of the user 5 from the detection results of the IMU.

Furthermore, the self-position of the user 5 (HMD 2) may be estimated based on the detection results of a sensor device provided in the HMD 2. For example, the self-position estimation can calculate position information of the HMD 2 and posture information such as the direction in which the HMD 2 is facing. From the position information and posture information, it is possible to acquire field of view information.
The algorithm for estimating the self-position of the HMD 2 is not limited, and any algorithm such as SLAM (Simultaneous Localization and Mapping) may be used.
In addition, head tracking for detecting the movement of the head of the user 5 and eye tracking for detecting the movement of the gaze of the user 5 to the left and right may be performed.

Any other device or algorithm may be used to acquire the visual field information. For example, when a smartphone or the like is used as a device that displays a virtual image to the user 5, an image of the face (head) of the user 5 may be captured, and the visual field information may be acquired based on the captured image.
Alternatively, a device equipped with a camera, an IMU, etc. may be worn around the head or eyes of the user 5.
To generate the visual field information, any machine learning algorithm using, for example, a deep neural network (DNN) may be used. For example, by using an AI (artificial intelligence) that performs deep learning, it is possible to improve the accuracy of generating the visual field information.
It should be noted that the application of machine learning algorithms may be performed on any process within the present disclosure.

The HMD 2 and the client device 3 are connected to each other so that they can communicate with each other. The communication method for connecting the two devices to each other so that they can communicate with each other is not limited, and any communication technology may be used. For example, wireless network communication such as Wi-Fi, short-range wireless communication such as Bluetooth (registered trademark), etc. may be used.
The HMD 2 transmits the visual field information to the client device 3 .
The HMD 2 and the client device 3 may be integrated. That is, the HMD 2 may be equipped with the functions of the client device 3.

The client device 3 and the server device 4 each have hardware necessary for configuring a computer, such as a CPU, a ROM, a RAM, and an HDD (see FIG. 18 ). The CPU loads a program according to the present technology, which is pre-recorded in the ROM or the like, into the RAM and executes the program, thereby executing the information processing method according to the present technology.
For example, the client device 3 and the server device 4 can be realized by any computer such as a PC (Personal Computer). Of course, hardware such as an FPGA or an ASIC may also be used.
Of course, the client device 3 and the server device 4 are not limited to having the same configuration.

The client device 3 and the server device 4 are connected to each other via a network 9 so as to be able to communicate with each other.
The network 9 is constructed using, for example, the Internet or a wide area communication network. Alternatively, any wide area network (WAN) or local area network (LAN) may be used, and the protocol for constructing the network 9 is not limited.

The client device 3 receives the visual field information transmitted from the HMD 2. The client device 3 also transmits the visual field information to the server device 4 via the network 9.

The server device 4 receives the visual field information transmitted from the client device 3. Based on the visual field information, the server device 4 performs a rendering process on the three-dimensional space data that constitutes the virtual space S, thereby generating two-dimensional video data (rendered video) corresponding to the visual field 7 of the user 5.
The server device 4 corresponds to an embodiment of an information processing device according to the present technology. The server device 4 executes an embodiment of an information processing method according to the present technology.

As shown in FIG. 3, the three-dimensional space data includes scene description information and three-dimensional object data.
The scene description information corresponds to three-dimensional space description data that defines the configuration of the virtual space S (three-dimensional space). The scene description information includes various metadata for reproducing each scene of the 6DoF content.
The three-dimensional object data is data that defines a three-dimensional object in the virtual space S (three-dimensional space), that is, data of each object that constitutes each scene of the 6DoF content.
For example, data of three-dimensional objects such as people and animals, or data of three-dimensional objects such as buildings and trees, or data of three-dimensional objects such as the sky and sea that make up the background, etc. Multiple types of objects may be collectively configured as a single three-dimensional object, and the data for that object may be stored.
The three-dimensional object data may be composed of mesh data that can be expressed as shape data of a polyhedron and texture data that is data to be applied to the surfaces of the mesh data, or may be composed of a set of multiple points (point cloud).

As shown in FIG. 3, the server device 4 reproduces a virtual space S that constitutes each scene by arranging three-dimensional objects in the three-dimensional space based on the scene description information.
3, an XYZ coordinate system is set in the virtual space S, and a three-dimensional object is placed at a position defined by coordinate values. The coordinate values correspond to position information in the virtual space S and can also be called world coordinates. There are no limitations on the method for setting the XYZ coordinate system in the virtual space S, and any setting method may be adopted.
Using the reproduced virtual space S as a reference, the image as seen by the user 5 is extracted (rendering process), and a rendered image, which is a two-dimensional image viewed by the user 5, is generated.
The server device 4 encodes the generated rendering image and transmits it to the client device 3 via the network 9 .
The rendered image according to the user's field of view 7 can also be said to be an image of a viewport (display area) according to the user's field of view 7.

The client device 3 decodes the encoded rendering video transmitted from the server device 4. The client device 3 also transmits the decoded rendering video to the HMD 2.
2, the rendered image is reproduced by the HMD 2 and displayed to the user 5. Hereinafter, the image 8 displayed to the user 5 by the HMD 2 may be referred to as the rendered image 8.

[Advantages of a server-side rendering system]
Another example of a 6DoF video delivery system, such as the one shown in FIG. 2, is a client-side rendering system.
In the client-side rendering system, the client device 3 performs rendering processing on the three-dimensional spatial data based on the field of view information to generate two-dimensional video data (rendered video 8). The client-side rendering system can also be called a client-rendering type media distribution system.
In a client-side rendering system, it is necessary to distribute three-dimensional space data (three-dimensional space description data and three-dimensional object data) from the server device 4 to the client device 3 .
The three-dimensional object data is composed of mesh data or point cloud data. Therefore, the amount of data to be distributed from the server device 4 to the client device 3 becomes enormous. Furthermore, the client device 3 is required to have a fairly high processing capacity in order to execute the rendering process.

In contrast, in the server-side rendering system 1 according to this embodiment, the rendered image 8 after rendering is delivered to the client device 3. This makes it possible to sufficiently reduce the amount of data delivered. In other words, with a small amount of data delivered, it becomes possible to allow the user 5 to experience a 6DoF image of a large space composed of a huge amount of three-dimensional object data.
In addition, it becomes possible to offload the processing load on the client device 3 to the server device 4, making it possible for the user 5 to experience 6DoF video even when a client device 3 with low processing capabilities is used.

[Response delay issue]
In the server-side rendering system 1, the field of view information of the user 5 and the rendered image 8 are transmitted and received via a network 9. Therefore, there is a possibility that a response delay may occur in the display of the rendered image 8 in response to a movement of the viewpoint, etc.
For example, the user 5 changes the field of view 7 by moving his/her head. Field of view information is acquired by the HMD 2 and transmitted to the client device 3. The client device 3 transmits the received field of view information to the server device 4 via the network 9.
The server device 4 performs rendering processing on the three-dimensional spatial data based on the received visual field information of the user 5, and generates a rendered image 8. The generated rendered image 8 is encoded and transmitted to the client device 3 via the network 9.
The client device 3 decodes the received rendered image 8 and transmits it to the HMD 2. The HMD 2 displays the received rendered image 8 to the user 5.
The server-side rendering system 1 is configured to execute such a processing flow in real time in response to changes in the field of view of the user 5. In this case, there is a possibility that a delay occurs between when the user 5 changes the field of view and when that change is reflected in the image on the HMD 2, which is called a response delay.
This response delay can also be expressed as motion-to-photon latency (T_m2p). It is desirable to keep the delay time of this response delay below 20 msec, which is the limit of human perception.

This technology is extremely effective in solving the above-mentioned problem of response delay. An embodiment of a server-side rendering system 1 to which this technology is applied will be described in detail below.
In the following embodiment, a case where head motion information is used as visual field information of the user 5 will be exemplified.
The Head Motion information includes Position information (X, Y, Z) that represents the positional movement of the head of the user 5, and Orientation information (yaw, pitch, roll) that represents the rotational movement of the head of the user 5.
The position information (X, Y, Z) corresponds to position information in the virtual space S, and is defined by coordinate values (world coordinates) of an XYZ coordinate system set in the virtual space S.
The orientation information (yaw, pitch, roll) is defined by the roll angle, pitch angle, and yaw angle relative to a roll axis, pitch axis, and yaw axis that are set on the head of the user 5 and are perpendicular to each other.
Of course, application of the present technology is not limited to cases where head motion information (X, Y, Z, yaw, pitch, roll) is used as visual field information of the user 5. The present technology can also be applied to cases where other information is used as visual field information.

In the following embodiment, the server-side rendering system 1 acquires the field of view information of the user 5 in real time, and displays the rendered image to the user 5 .
In the following description, the time when the visual field information of the user 5 is acquired by the server-side rendering system 1 is referred to as the "current time." In other words, the time when the visual field information of the user 5 is acquired by the HMD 2 is referred to as the "current time."
As described above, there is a possibility that a response delay (T_m2p time) may occur between the time when the visual field information acquired at the "current time" is transmitted to the server device 4, the time when the rendering image 8 is generated, and the time when it is displayed by the HMD 2.
By applying this technology, it is possible to sufficiently mitigate the problem of response delays from the "current time," enabling the delivery of high-quality virtual video.

FIG. 4 is a schematic diagram showing an example of the configuration of a server-side rendering system 1 according to an embodiment of the present technology.
The server-side rendering system 1 shown in FIG. 4 includes an HMD 2, a client device 3, and a server device 4.
The HMD 2 is capable of acquiring in real time visual field information (head motion information) of the user 5. As described above, the time when the head motion information is acquired by the HMD 2 is the current time.
The HMD 2 acquires head motion information at a predetermined frame rate and transmits it to the client device 3. Therefore, "head motion information at the current time" is repeatedly transmitted to the client device 3 at the predetermined frame rate.
Similarly, the client device 3 also repeatedly transmits "head motion information at the current time" to the server device 4 at a predetermined frame rate.

The frame rate for acquiring head motion information (number of times head motion information is acquired per second) is set to be synchronized with the frame rate of the rendered video 8, for example.
For example, the rendered video 8 is composed of a plurality of frame images that are successive in time series. Each frame image is generated at a predetermined frame rate. The frame rate for acquiring head motion information is set so as to be synchronized with the frame rate of the rendered video 8. Of course, this is not a limitation.
As described above, AR glasses or a display may be used as a device for displaying a virtual image to the user 5 .

The server device 4 includes a data input unit 11, a head motion information recording unit 12, a prediction unit 13, a rendering unit 14, an encoding unit 15, and a communication unit 16. The server device 4 also includes a saliency map generation unit 17, a saliency map recording unit 18, and a recognition position estimation unit 19.
These functional blocks are realized by, for example, a CPU executing a program according to the present technology, and the information processing method according to the present embodiment is executed. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

The data input unit 11 reads out three-dimensional space data (scene description information and three-dimensional object data) and outputs it to the rendering unit 14 .
The three-dimensional spatial data is stored, for example, in a storage unit 68 (see FIG. 18 ) in the server device 4. Alternatively, the three-dimensional spatial data may be managed by a content server or the like that is communicably connected to the server device 4. In this case, the data input unit 11 acquires the three-dimensional spatial data by accessing the content server.

The communication unit 16 is a module for performing network communication, short-range wireless communication, etc. with other devices. For example, a wireless LAN module such as WiFi, or a communication module such as Bluetooth (registered trademark) is provided.
In this embodiment, the communication unit 16 realizes communication with the client device 3 via the network 9 .

The Head Motion information recording unit 12 records the visual field information (Head Motion information) received from the client device 3 via the communication unit 16 in the storage unit 68 (see FIG. 18 ). For example, a buffer or the like for recording the visual field information (Head Motion information) may be configured.
The "head motion information at the current time" transmitted at a predetermined frame rate is accumulated and held in the storage unit 68.

The prediction unit 13 generates future visual field information as predicted visual field information based on a panoramic saliency map (panoramic saliency map). In this embodiment, future head motion information of the user 5 is predicted and generated as predicted head motion information.
The predicted head motion information includes future position information (X, Y, Z) and future orientation information (yaw, pitch, roll). That is, in this embodiment, the head position and head rotation angle are predicted based on the omnidirectional saliency map.

The rendering unit 14 executes the rendering process exemplified in Fig. 3. That is, by executing the rendering process on the three-dimensional space data based on the visual field information relating to the visual field of the user 5, a rendered image 8 corresponding to the visual field 7 of the user 5 is generated.
In this embodiment, the rendering unit 14 generates frame images that constitute the rendered video 8 based on the predicted field of view information (predicted head motion information) generated by the prediction unit 13. Hereinafter, the frame images generated based on the predicted head motion information will be referred to as predicted frame images 20.
The rendering unit 14 is configured by, for example, a reproduction unit that reproduces the virtual space S, a renderer, a parameter setting unit that sets rendering parameters, etc. The rendering parameters include a resolution map that indicates the resolution for each region.
Any other configuration may be adopted as the rendering unit 14 .

The encoding unit 15 performs encoding (compression coding) on the rendering video 8 (predicted frame image 20) to generate distribution data. The distribution data is transmitted to the client device 3 via the communication unit 16.
For example, the encoding process is performed in real time for each region of the rendered video 8 (predicted frame image 20) based on a QP map (quantization parameter).
More specifically, in this embodiment, the encoding unit 15 can suppress image quality degradation due to compression of points of interest and important areas within the predicted frame image 20 by switching the quantization precision (QP: Quantization Parameter) for each area within the predicted frame image 20.
In this way, it is possible to suppress an increase in the load of distribution data and processing while maintaining sufficient video quality for areas that are important to the user 5. Note that the QP value here is a value that indicates the quantization step when using lossy compression efficiency, and a high QP value reduces the amount of coding and increases compression efficiency, resulting in greater image quality degradation due to compression, while a low QP value increases the amount of coding and decreases compression efficiency, making it possible to suppress image quality degradation due to compression.
Any other compression encoding technology may be used.
The encoding unit 15 is configured by, for example, an encoder, a parameter setting unit that sets encoding parameters, etc. The encoding parameters include the above-mentioned QP map, etc.
For example, the QP map is generated based on the resolution map set by the parameter setting unit of the rendering unit 14. Alternatively, the encoding unit 15 may have any other configuration.

The saliency map generator 17 generates a panoramic saliency map.
The saliency map generation unit 17 generates not only a saliency map for the field of view that represents the saliency of the rendering image (two-dimensional image data) 8 viewed by the user 5, but also a saliency map for the out-of-field area that is not included in the field of view 7 of the user 5.
The visual field saliency map is information that quantitatively represents how likely each pixel in the rendered image 8 is to attract attention, based on the mechanism of human visual attention.
The saliency map for the out-of-field area can be generated as a map in which the saliency of each pixel is calculated when the out-of-field area of the user 5 is represented as a 2D image.
The generation of the omnidirectional saliency map will be described in detail later. The saliency map is also called a saliency map. The omnidirectional saliency map can also be called a celestial sphere.

The saliency map recording unit 18 records the panoramic saliency map generated by the saliency map generation unit 17 in the memory unit 68 (see Figure 18). For example, a buffer or the like may be configured for recording the panoramic saliency map.

The recognition position estimation unit 19 estimates the recognition position of the recognition target object recognized by the user 5 in the out-of-field area of the virtual space S that is not included in the field of view 7 of the user 5 .
The object to be recognized is an object that is assumed to be recognized by the user 5 among the objects in the 6DoF content that the user 5 is viewing.
For example, an object that the user 5 has viewed up to the current time is set as a recognition target object. In other words, an object that has been a rendering target up to the current time is set as a recognition target object.
Alternatively, an object that has not been viewed up to the current time but whose sound is recognized by the user may be set as a recognition target object. For example, if the volume of the sound made by the object is greater than a predetermined reference value (threshold value), the object that made the sound may be recognized by the user 5 and set as a recognition target object. In this case, the type or content of the sound may be used as a condition for whether or not to set the object as a recognition target object.

The recognition position is defined by XYZ coordinate values (world coordinates) in the virtual space S shown in Fig. 3. The position where the user 5 would currently recognize the object to be recognized as being located in his or her mind is estimated as the recognition position.
Therefore, the recognition position does not necessarily coincide with the position where the object to be recognized actually exists in the 6DoF content.
The recognized position can also be said to be a grasped position that the user 5 grasps.

FIG. 5 is a schematic diagram for explaining a recognition target object and a recognition position.
5, for ease of understanding, the panoramic image SP of the virtual space S is expressed as an image that is long in the horizontal direction. In practice, the panoramic image SP is often saved as an equirectangular image.
In the following description, the predicted user's field of view 7 will be simply referred to as the user's field of view 7. The predicted frame image 20 will be described as the frame image 20.

5, three people P1 to P3 and a blinking lighting device L are placed in a virtual space S. Other objects such as trees, grass, roads, and buildings are also placed.
5A, the field of view 7 of the user 5 is first directed toward the person P1 on the right. Therefore, a frame image 20 including the person P1 is rendered. At this point, the person P1 is set as the object to be recognized. Then, the position (world coordinates) in the virtual space S where the person P1 is located is estimated as the recognition position.
In this way, it is possible to estimate the recognition position based on the position of the object to be recognized within the two-dimensional video data in which the object to be recognized is rendered.

At the timing shown in Figure 5A, the persons P2 and P3 on the left and the blinking lighting device L object on the right are located in the out-of-field area 21 that is not included in the field of view 7 of the user 5. Therefore, the persons P2 and P3 and the lighting device L are not rendered. As a result, although the persons P2 and P3 and the lighting device L are located in the virtual space S, they are determined to be not recognized by the user 5 and are not set as objects to be recognized.

5B, it is assumed that the field of view 7 of the user 5 is turned to the left. The people P2 and P3 are included in the field of view 7 of the user 5 and are rendered. As a result, the people P2 and P3 are set as objects to be recognized. Then, the positions (world coordinates) in the virtual space S where the people P2 and P3 are located are estimated as recognition positions.
Although the person P1 is outside the field of view 7 of the user 5 and is positioned in the out-of-field area 21, the setting as an object to be recognized is maintained because the person P1 is an object that has been rendered in the past.
The lighting device L is not rendered and is therefore not set as a recognition target object.

5C, it is assumed that the person P1 moves in the out-of-field area 21 toward the lighting device L. In this case, the position of the person P1 moves in the virtual space S.
Here, it is assumed that the user 5 is not aware of the movement of the person P1 in the out-of-field area 21. In this case, the recognized position of the person P1 in the user 5's mind does not match the actual position of the person P1.
The recognition position estimation unit 19 can estimate with high accuracy the recognition position in the brain of the user 5 when the recognition target object moves in the out-of-field area 21 as shown in FIG. 5C.

In this embodiment, the rendering unit 14 functions as an embodiment of a rendering unit according to the present technology.
The recognition position estimation unit 19 functions as an embodiment of an estimation unit according to the present technology.
The saliency map generation unit 17 functions as an embodiment of a generation unit according to the present technology.
The prediction unit 13 functions as an embodiment of a prediction unit according to the present technology.

The client device 3 includes a communication unit 23 , a decoding unit 24 , and a rendering unit 25 .
These functional blocks are realized by, for example, a CPU executing a program according to the present technology, and the information processing method according to the present embodiment is executed. Note that dedicated hardware such as an IC (integrated circuit) may be used as appropriate to realize each functional block.

The communication unit 23 is a module for performing network communication, short-range wireless communication, etc. with other devices. For example, a wireless LAN module such as WiFi, or a communication module such as Bluetooth (registered trademark) is provided.
The decoding unit 24 executes a decoding process on the distribution data, thereby decoding the encoded rendering video 8 (predictive frame image 20).
The rendering unit 25 performs rendering processing so that the decoded rendered video 8 (predicted frame image 20 ) can be displayed on the HMD 2 .

[Head Motion Information Prediction Accuracy]
For example, the server device 4 receives the "head motion information at the current time" and generates predicted head motion information for the future corresponding to the response delay (T_m2p time). Then, a predicted frame image 20 is generated based on the predicted head motion information and is displayed to the user 5 by the HMD 2.
If predicted head motion information can be generated with extremely high accuracy, it will be possible to display a rendered image 8 according to the field of view 7 of the user 5 at a time in the future corresponding to the response delay (T_m2p time) from the "current time," and the problem of response delay can be sufficiently suppressed.

The present inventors have conducted extensive research into head motion prediction in order to improve the accuracy of predicted head motion information.
First, the prediction error of head motion prediction tends to increase as the frequency of the head movement signal (sensing result) increases.
Due to the characteristics of the human body, rapid changes in rotational movement (high-frequency movement) are possible, but high-frequency movement with sudden changes in position such as forward/backward, up/down, and left/right tends to be difficult.
Therefore, of these two types of motion, the prediction error for positional movements (X, Y, Z) is low and has very little impact on viewing. On the other hand, the prediction error for rotational movements (yaw, pitch, roll) tends to be large and can easily affect viewing. In other words, improving the prediction accuracy for rotational movements (yaw, pitch, roll) is extremely important.

The inventors have focused on the panoramic saliency map to improve the accuracy of head motion prediction, especially for rotational movements (yaw, pitch, roll).
By generating a high-accuracy panoramic saliency map and using it for head motion prediction, it becomes possible to perform highly accurate prediction of rotational movements (yaw, pitch, roll).

[Generation of 2D video data (rendered video)]
An example of the operation of the server device 4 for generating a rendering image using the panoramic saliency map will be described.
FIG. 6 is a flowchart showing an example of generating a rendered image.
FIG. 7 is a diagram for explaining the flowchart shown in FIG. 6, and is a schematic diagram showing the timing of obtaining head motion information, generating predicted head motion information, generating a predicted frame image 20, and generating a panoramic saliency map.
In this embodiment, for ease of explanation, it is assumed that visual field information is acquired from the client device 3 at a predetermined frame rate, and that predicted head motion information, predicted frame images 20, and a panoramic saliency map are each generated at the same frame rate. Of course, the present invention is not limited to such processing.
7, the numbered boxes indicate the frames of each process, and in FIG. 7, the first frame at which the process starts to the 25th frame are schematically shown.
In addition, for each frame, a square indicates that the data shown on the left was acquired/generated. The number inside the square indicates which frame the data corresponds to.

First, it is set how far into the future from the "current time" the predicted head motion information is to be generated.
In this embodiment, the communication unit 16 measures the network delay with the client device 3 and identifies the predicted time of the target (step 101). That is, the response delay (T_m2p time) is measured, and the T_m2p time is identified as the predicted time.
In this embodiment, head motion information for a frame that is a predetermined number of frames in the future than the frame corresponding to the "current time" is predicted and generated as predicted head motion information.
The predetermined number of frames is set to the number of frames equivalent to the predicted time T_m2p.
For example, in this embodiment, head motion information for five frames ahead is predicted. For example, if "head motion information at the current time" is acquired in the tenth frame, head motion information for the fifteenth frame, which is five frames ahead, is predicted and generated as predicted head motion information. Of course, the specific number of frames is not limited and may be set arbitrarily.

The communication unit 16 acquires head motion information from the client device 3 (step 102). As shown in Figure 7, head motion information is acquired at a predetermined frame rate starting from the first frame. The head motion information acquired for each frame is used as is as the data corresponding to that frame.

The prediction unit 13 determines whether or not the amount of head motion information necessary for predicting head motion information has been accumulated (step 103).
In this embodiment, it is assumed that 10 frames of head motion information are required to predict head motion information, but the number of frames is not limited to a specific number and may be set arbitrarily.
For example, for frames 1 to 9, the amount of head motion information required for predicting head motion information has not been accumulated, so the answer in step 103 is No and the process returns to step 102. Therefore, generation of the rendered video 8 (predicted frame image 20) is not executed until the 10th frame.
When the head motion information for the tenth frame is acquired, it is determined that the amount of head motion information required for predicting the head motion information has been accumulated, and the answer to step 103 is Yes, and the process proceeds to step 104 .

In step 104, the prediction unit 13 determines whether or not a panoramic saliency map corresponding to the "head motion information at the current time" acquired in step 102 has already been generated.
In this embodiment, predicted visual field information (predicted head motion information) is generated using as input historical information of visual field information (head motion information) up to the current time and a panoramic saliency map corresponding to the current time.
The panoramic saliency map corresponding to the current time is a panoramic saliency map generated in the past. Specifically, the panoramic saliency map includes a saliency map for the visual field that indicates the saliency of the predicted frame image 20 generated based on predicted visual field information (predicted head motion information) predicted in the past, and a saliency map for an out-of-visual field area that is not included in the visual field 7 (predicted visual field) of the user 5 based on predicted visual field information (predicted head motion information) predicted in the past.

In the example shown in FIG. 7, the omnidirectional saliency map corresponding to the "head motion information at the current time" refers to the omnidirectional saliency map corresponding to the frame in which the "head motion information at the current time" is acquired.
That is, when the number inside the square shape indicating the Head Motion information and the number inside the square shape indicating the omnidirectional saliency map are equal, a pair of corresponding "Head Motion information at the current time" and omnidirectional saliency map is formed.

For example, when Head Motion information for the 10th frame is acquired, the frame corresponding to the current time is frame 10. In step 104, it is determined whether a panoramic saliency map corresponding to the 10th frame (a panoramic saliency map represented by a square with the number 10 written inside) has been generated.
7, up to the tenth frame, predicted head motion information has not been generated, and predicted frame image 20 has not been generated either. Therefore, since a panoramic saliency map has not been generated, step 104 becomes No, and the process proceeds to step 105.

In step 105, the prediction unit 13 generates predicted visual field information (predicted head motion information) based on the history information of visual field information (head motion information) up to the current time.
In this way, when a panoramic saliency map for a frame corresponding to the current time has not been generated, predicted head motion information may be generated based only on the history information of head motion information up to the current time.
In this embodiment, at frame 10, predicted head motion information for the future, that is, five frames ahead, is generated based on the history information of head motion information from frame 1 to frame 10. Therefore, as shown in Fig. 7, at the 10th frame, predicted head motion information corresponding to frame 15, five frames ahead, is generated (predicted head motion information represented by a square with the number 15 written inside).
There are no particular limitations on the specific algorithm for generating predicted head motion information based on the history information of head motion information up to the current time, and any algorithm may be used. For example, any machine learning algorithm may be used.

The rendering unit 14 performs the rendering process illustrated in FIG. 3 based on the predicted head motion information, and generates a rendered video 8 (predicted frame image 20) (step 106). In this embodiment, a predicted frame image 20 corresponding to 15 frames is generated based on predicted head motion information for the next five frames.

The recognition position estimation unit 19 estimates the recognition position of the object to be recognized in the out-of-field area (step 107). In this embodiment, the recognition position of the object to be recognized that the user 5 recognizes is estimated in the out-of-field area that is not included in the user 5's field of view 7 (predicted field of view) based on the predicted field of view information (predicted head motion information).

The saliency map generator 17 generates a panoramic saliency map corresponding to the 15 frames based on the predicted frame image 20 and the estimated recognition positions (step 108).
The generated panoramic saliency map is recorded and held by the saliency map recording unit 18. As illustrated in Fig. 7, for the 10th frame, a panoramic saliency map corresponding to the 15th frame is recorded.

7, in this embodiment, for the frame corresponding to the "current time", a frame image five frames ahead is generated as a predicted frame image 20. Also, for the frame corresponding to the "current time", a panoramic saliency map for five frames ahead is generated.
In the present disclosure, a frame image generated at the "current time," i.e., a frame image generated in a frame corresponding to the "current time," is referred to as the "frame image at the current time." Therefore, in this embodiment, the future predicted frame image 20 generated in a frame corresponding to the "current time" corresponds to the "frame image at the current time."
On the other hand, the "frame image corresponding to the current time (predicted frame image)" corresponds to a frame image (predicted frame image) generated five frames in the past.

The encoding unit 15 encodes the predicted frame image 20. The communication unit 16 transmits the encoded predicted frame image 20 to the client device 3 (step 109).
The predicted frame image 20 generated for the tenth frame is transmitted to the HMD 2 via the client device 3 as the first frame of the 6DoF video content, and is displayed to the user 5. This starts the distribution of a virtual video with the effects of response delays sufficiently suppressed.
The rendering unit 14 determines whether or not processing has been completed for all frame images (step 110). Here, it is assumed that processing has been performed up to frame 25, as shown in FIG.
Therefore, step 110 is negative and the process returns to step 102.

From frame 11 to frame 14 shown in FIG. 7, step 104 is determined as No, and the processing flow proceeds from step 105 to step 106 is executed.
In frame 15, the omnidirectional saliency map corresponding to frame 15, which was generated in the past frame 10, exists as the omnidirectional saliency map corresponding to the acquired "Head Motion information at the current time." Therefore, step 104 becomes Yes, and the process proceeds to step 111.

In step 111, future head motion information is predicted using the historical information of visual field information (head motion information) up to the current time and the omnidirectional saliency map corresponding to the current time as input, and predicted head motion information is generated.
There are no particular limitations on the specific algorithm for generating predicted head motion information using the history information of head motion information and the omnidirectional saliency map as input, and any algorithm may be used. For example, any machine learning algorithm may be used.
From then on, up to frame 25, step 104 returns Yes and the panoramic saliency map is used to generate highly accurate predicted Head Motion information.
If the processing for all frame images is completed, the answer in step 109 is Yes, and the video generation and distribution processing ends.

[Considerations on generating omnidirectional saliency maps]
The inventors have given extensive consideration to the generation of a panoramic saliency map.
Human visual attention can be divided into two types: exogenous attention (bottom-up attention) triggered by visual stimuli before object recognition, and endogenous attention (top-down attention) triggered by interest in or concern for an object after object recognition. The keyword "saliency" is used in both bottom-up and top-down attention.

An example of generating a saliency map based on bottom-up attention is to extract features such as brightness, color, direction, direction of motion, and depth from the input video (2D image) that attract exogenous attention (bottom-up attention) through visual stimuli before a person recognizes an object. Each feature map is calculated so that areas where the value of each feature is significantly different from the surrounding area are assigned a high level of saliency, and then the final saliency map is generated by integrating them.

FIG. 8 is a schematic diagram illustrating an example of bottom-up attention-based saliency map generation that can be performed by the present system.
In the example shown in FIG. 8, a predicted frame image 20 is input as an input frame.
A feature extraction process is performed on the predicted frame image 20, and the features of luminance, color, and direction that attract bottom-up attention are extracted. Note that the predicted frame image 20 of the previous frame, etc., may be used for feature extraction.
For each feature amount of brightness, color, and direction, a feature image is generated in which the feature amount is converted into brightness, and a Gaussian pyramid of the feature image is generated.
Furthermore, a depth map and a motion vector map image are acquired as parameters (rendering information) related to the rendering process from a renderer constituting the rendering unit 14. The depth map is data including distance information (depth information) to an object to be rendered. The motion vector map image is data including motion information of the object to be rendered.
The depth map image is used as a depth feature image to generate a Gaussian pyramid, and the motion vector map image is used as a motion direction feature image to generate a Gaussian pyramid.
Center-surround subtraction is performed on each Gaussian pyramid of features, generating feature maps for brightness, color, orientation, motion direction, and depth. These feature maps are then combined to generate a saliency map 22 based on bottom-up attention.
There are no particular limitations on the specific algorithms for the feature extraction process, Gaussian pyramid generation process, center-surround difference process, and feature map integration process for each feature. For example, each process can be realized using well-known techniques.

The depth map image acquired from the renderer is not a depth value estimated by performing 2D image analysis or the like on the predicted frame image 20, but an accurate value obtained in the rendering process. Therefore, by receiving this depth map image directly from the renderer and using it as "depth" feature information in generating the saliency map 22, it becomes possible to generate a more accurate saliency map 22 with higher precision.
The motion vector map image obtained from the renderer is not a value estimated by performing 2D image analysis or the like on the predicted frame image 20, but an accurate value obtained in the rendering process. Therefore, by receiving this depth map image directly from the renderer and using it as feature information on the "motion direction" in generating the saliency map 22, it becomes possible to generate a more accurate saliency map with higher precision.
It should be noted that other feature quantities such as "brightness" and "color" can also be calculated in the rendering process and used as rendering information.
The algorithm for generating a saliency map based on bottom-up attention is not limited, and any other algorithm may be used, for example, any machine learning algorithm.
For example, feature information such as "depth" and "direction of movement" may be acquired by performing 2D image analysis on the predicted frame image 20, and this information may be used.

Top-down attention is directed after object recognition based on its meaning, so salience is given to the object.
For example, an object that generally attracts people's attention, such as a human face, is detected from an image and given a saliency rating. In addition, saliency is given to the display area of an object based on the type of object (idol, baseball player, vehicle, etc.), the importance of the object in the 6DoF content, the user's preference for the object (degree of interest or preference, etc.), etc.
The algorithm for generating the saliency map based on top-down attention is not limited, and any algorithm may be used, for example, any machine learning algorithm.

It is also possible to generate a saliency map that includes both saliency based on bottom-up attention and saliency based on top-down attention all at once. For example, by capturing actual human gaze data and using it to create and train training data, it is possible to build a machine learning model that can generate a saliency map that includes both bottom-up and top-down attention.

[Comparative Example of a Method for Generating a Panoramic Saliency Map]
Here, a method for generating a panoramic saliency map will be described as a comparative example.
A 2D image is rendered for each viewport while changing the viewport at regular intervals across the entire virtual space S (hereinafter, the rendered 2D image will be referred to as a viewport image). Then, for each viewport image, a saliency map based on bottom-up attention or top-down attention as described above is generated and integrated.
The panoramic saliency map of this comparative example serves as information representing the saliency of all viewport images for the user 5. In other words, it serves as information representing the saliency of all areas in the virtual area S when they are within the field of view of the user 5.

Therefore, when the panoramic saliency map of this comparative example is used, there is a high possibility that the following problems will occur.
9 is a schematic diagram for explaining the problem with the panoramic saliency map 30 of the comparative example. In FIG. 9, the saliency based on the top-down attention generated by the persons P1 to P3 and the saliency based on the bottom-up attention generated by the lighting device L are schematically illustrated as white regions.

(1) The saliency resulting from bottom-up attention is not seen outside the visual field of the user 5, and therefore there is no visual stimulation and it is essentially equal to zero. The omnidirectional saliency map 30 of the comparative example is created on the assumption that all areas in the virtual space S are within the visual field. Therefore, if the omnidirectional saliency map 30 of the comparative example is used as is to predict visual field information, saliency that is outside the visual field and cannot be recognized by the user 5 will have a negative impact on the prediction of visual field information.
For example, at the timing shown in Fig. 5A , the blinking lighting device L on the right side is not recognized by the user 5. In the omnidirectional saliency map 30 of the comparative example, as shown in Fig. 9A , saliency based on bottom-up attention is assigned to the pixel region of the blinking lighting device L. As a result, saliency that does not exist in the brain of the user 5 is generated, which adversely affects the prediction of visual field information.

(2) Even in the case of saliency resulting from top-down attention, the saliency of an object that is outside the field of view of the user 5 and has not yet been captured (visually recognized) in the viewport is essentially zero because the user 5 is not even aware of its existence. When the omnidirectional saliency map 30 of the comparative example is used, saliency that is outside the field of view and cannot be recognized by the user 5 adversely affects the prediction of visual field information.
For example, at the timing shown in Fig. 5A, the people P2 and P3 on the left side are not recognized by the user 5. In the omnidirectional saliency map 30 of the comparative example, as shown in Fig. 9A, the people P2 and P3 are assigned saliency based on top-down attention. As a result, saliency that does not exist in the brain of the user 5 is generated, which adversely affects the prediction of visual field information.

(3) Furthermore, in the case of salience resulting from top-down attention, an object previously viewed may be moving in the out-of-field area 21. In this case, the user 5 will not notice the object moving. Therefore, the user will likely recognize that the object is in a position that corresponds to the situation he or she perceived the last time he or she viewed the object.
For example, if an object was stationary, we would likely recognize it as being in the position where we last saw it, and if the object was moving when we last saw it, we would likely recognize it as being some distance along the direction of movement from where we last saw it.
The omnidirectional saliency map 30 of the comparative example is based on the assumption that moving objects and objects after they have moved are also within the field of view, and therefore saliency based on top-down attention is assigned to moving objects and objects after they have moved.
For example, if an object was stationary when last viewed, i.e., if the user is not aware that the object has moved since then, salience will occur in a location completely different from the location of recognition (grasping location) in the brain.
If the object was moving when last seen, there is no problem if the perceived location (perceived location) in the user's mind matches the actual location of the object. However, it is not necessarily likely that the location the user expects to be and the actual location of the object will match. Therefore, there is a high possibility that saliency will occur in a location different from the perceived location (perceived location) in the user's mind.
For example, at the timing shown in Fig. 5C, the user 5 does not recognize the movement of the person P1. In the omnidirectional saliency map 30 of the comparative example, as shown in Fig. 9B, the person P1 near the lighting device L1 is given saliency based on top-down attention. As a result, saliency that does not exist in the brain of the user 5 is generated, which adversely affects the prediction of visual field information.
As described above, the omnidirectional saliency map 30 of the comparative example does not take into consideration saliency outside the viewport (outside the field of view). Therefore, if the omnidirectional saliency map 30 of the comparative example is used as is for predicting Head Orientation, which is the next direction to which the viewport will be directed from the current viewport, the prediction accuracy may actually decrease or may be useless, thereby failing to achieve the objective of improving prediction accuracy.
To achieve high prediction accuracy, it is important to accurately reflect the actual attention state of the user 5 with respect to the inside and outside of the visual field.

[Estimation of the recognition position of the object to be recognized]
In this embodiment, the recognition target object is set and the recognition target position of the recognition target object is estimated by the recognition position estimation unit 19. These processes are executed for each frame.
Hereinafter, several examples of estimating the recognition position of a recognition target object will be described.
Example 1
The object that is the subject of rendering by the rendering unit 14 is set as the object to be recognized.
For example, at the timing shown in Fig. 5A, a person P1 is set as the object to be recognized, and at the timing shown in Fig. 5B, persons P1 and P2 are set as the objects to be recognized.

In each frame, each time a "current time frame image" (a future predicted frame image 20 generated at the current time) is generated, the recognition position of each set object to be recognized is estimated.
For a recognition target object included in the "frame image at the current time", the recognition position is estimated based on the position in the virtual space S corresponding to the position of the recognition target object in the frame image.
For example, at the timing shown in Figure 5A, person P1 is included in frame image (predicted frame image) 20, so the recognized position of person P1 is estimated based on the position in virtual space S corresponding to the position of person P1 in frame image 20.

The position in the virtual space S corresponding to the position of the person P1 in the frame image 20 becomes the position of the person P1 placed in the virtual space S when rendering the frame image 20. The position of the person P1 is estimated as the recognized position recognized by the user 5.
If a state in which the person P1 is moving has been rendered, that is, if the user 5 recognizes that the person P1 is moving, a position in the virtual space S corresponding to the position of the person P1 in the frame image 20, that is, a position shifted along the direction of movement from the position of the person P1 placed in the virtual space S, may be estimated as the recognized position recognized by the user 5. The amount of shift may be set appropriately to an amount that is thought to allow the user 5 to predict the destination.
Here, it is assumed that the position in the virtual space S corresponding to the position of the person P1 in the frame image 20 is directly estimated as the recognized position.

For example, at the timing shown in Figure 5B, the position in virtual space S corresponding to the positions of people P2 and P3 in frame image 20 is estimated as the recognized position recognized by user 5 for people P2 and P3.

For a recognition target object that is not included in the "frame image at the current time," the recognition position is estimated based on the position in the virtual space S that corresponds to the position of the recognition target object in the most recent past frame image 20 that includes the recognition target object. In this embodiment, the position in the virtual space S that corresponds to the position of the recognition target object in the most recent frame image 20 is directly estimated as the recognition position.
The most recent past frame image can also be said to be the last frame image rendered.

For example, at the timing shown in Fig. 5B, person P1 is not included in frame image 20. For person P1 not included in frame image 20, the position in virtual space S corresponding to the position of person P1 in frame image 20 at the timing shown in Fig. 5A is directly estimated as the recognized position.
At the timing shown in Fig. 5C, in the out-of-field area 21, person P1 is moving toward the lighting device L. In this embodiment, the position in the virtual space S corresponding to the position of person P1 in the most recent frame image 20, i.e., the frame image 20 at the timing shown in Fig. 5A, is maintained as the recognized position.
This makes it possible to prevent the occurrence of saliency that does not exist in the brain of the user 5, and makes it possible to predict visual field information with high accuracy.

For example, each time a recognition target object is rendered, the position in virtual space S corresponding to the position of the recognition target object in the rendered frame image 20 is updated and stored as the recognition position. As a result, for recognition target objects that are not included in the frame image 20 at the current time, the position in virtual space S corresponding to the position of the recognition target object in the most recent frame image 20 is held as the recognition position. In this way, the recognition position may be updated for each frame.

Example 2
Even if the user 5 moves an object out of his or her field of vision, the user 5 may be able to determine the object's location by listening to sounds such as footsteps or voices emitted from the object. This embodiment 2 is designed to address such a situation. Fig. 10 is a schematic diagram showing an example of information described in a scene description file used as scene description information in this embodiment.
In this embodiment, when 6DoF content is generated, audio data such as footsteps and voices emitted from the objects is generated in association with each object information described in the scene description file.
Based on this information, the renderer will render the associated audio data so that it plays from the position of each object.

In the example shown in FIG. 10, the following information is stored as object information:
Name...name of object Position...position of object Url...address of three-dimensional object data Audio...name of audio data of sound emitted from object In the example shown in Figure 10, the following information is stored as audio data information linked to object information.
Name...Name of the audio data Url...Address of the audio data

In the example shown in Figure 10, in a hide-and-seek scene, audio data information for "Voice and Footsteps 1" and "Voice and Footsteps 2" is linked to video object information for "Hiding Person 1" and "Hiding Person 2." Examples of audio data for "Voice and Footsteps 1" and "Voice and Footsteps 2" include responses such as "Not yet" or "That's enough" to the Oni's call "Are you ready?", as well as the footsteps of someone running away from the Oni.

The recognition position estimation unit 19 receives from the renderer information on the presence or absence of audio data associated with each object, current volume information, etc. Then, it determines whether the user 5 is currently hearing and recognizing the sound emitted from the object based on whether the volume level exceeds a reference value (threshold) that is a volume level used as a reference for determining whether the user 5 can hear and recognize the sound. Note that the reference volume level may be set arbitrarily.
If there is associated audio data and the volume level exceeds a reference level, the current position of the object in the virtual space S, i.e., the position where the sound is generated in the virtual space S, is estimated as the recognized position recognized by the user 5.
That is, in this embodiment, when it is determined that the user 5 has recognized the sound emitted by a recognition target object that is not included in the “frame image at the current time”, the position where the sound originated within the virtual space S is estimated as the recognition position.

For example, in a scene configured based on the scene description file shown in Fig. 10, the recognized positions of people hiding are estimated based on the voices of the people playing hide-and-seek. For example, suppose a voice saying "It's not yet" is uttered from "Hiding Person 1," who is not included in the frame image. If the volume of the voice exceeds a reference value, it is determined that the voice was heard by user 5, and the current position of "Hiding Person 1" in virtual space S is estimated as the recognized position.
This makes it possible to prevent the occurrence of saliency that does not exist in the brain of the user 5, and makes it possible to predict visual field information with high accuracy.
When the sound emitted from the object to be recognized is interrupted and can no longer be heard, the position where the sound was last heard, that is, the position where the sound was most recently heard in the past, is maintained as the recognition position.

Example 3
If the user 5 has previously viewed a scene similar to the scene currently being viewed, the user may grasp the position of an object by associating it with his or her memory of that scene. This embodiment 3 is designed to take such a case into consideration, and the recognized position in the brain of the user 5 is estimated based on past viewing information of a scene similar to the current scene.
11 and 12 are schematic diagrams showing an example of information described in a scene description file used as scene description information in this embodiment.
In this embodiment, when generating 6DoF content, the object information described in the scene description file stores the role information of each object in the current scene and its fixed position information (world coordinates) when playing that role.
That is, the scene description file stores role information that indicates the role of the object to be recognized, and a fixed position (world coordinates) related to the role.

In the examples shown in FIGS. 11 and 12, the following information is stored as object information:
Name...Name of object Position...Position of object Url...Address of 3D object data Role...Role information FixedPos...Fixed position information related to role

11 and 12, in a baseball scene, role information and regular position information for players "Ada A-san" and "Bgawa B-suke" when on offense and when on defense are stored. Fig. 11 is a scene description file when on offense, and Fig. 12 is a scene description file when on defense.
Every time the offense and defense switch positions, the attacking scene description file and the defensive scene description file are updated. Also, if the role of an object changes between offense and defense, the attacking and defensive scene description files are updated accordingly.
For example, when the scene description file is updated from Fig. 11 to Fig. 12 in response to a change of offense and defense, player "A-da A-san" changes from the role of "next batter" to the role of "first baseman," and player "B-gawa B-suke" changes from the role of "batter" to the role of "pitcher."
The FixedPos of the "batter,""nextbatter,""pitcher," and "first baseman" are assumed to be positions in world coordinates relating to the following positions.
"Batter"...batter's box "Next batter"...next batter "Pitcher"...pitcher's mound "First baseman"...position of first base These FixedPos indicate the general position for that role, and are different from Position, which indicates the actual position of the object.
Based on this information, the recognition position estimation unit 19 determines whether the user 5 has previously seen a scene similar to the current scene, and estimates the recognition position.

13 and 14 are schematic diagrams for explaining estimation of the recognition position of the recognition target object in the third embodiment.
13 and 14, a baseball stadium is configured as the virtual space S, and a player "A-ta A-fu" 32 and a player "B-gawa B-suke" 33 are also placed within the virtual space S.
The direction of the line of sight of the eyes in the figure corresponds to the field of view (predicted field of view) 7 of the user 5, and a frame image (predicted frame image) 20 of the area of the field of view 7 is generated.

13A, first, the user 5 watches a scene in which the player "A-da A-husband" 32 plays first base. That is, the user 5 understands that the player "A-da A-husband" 32 is at first base in the defensive scene. Of course, the player "A-da A-husband" 32 is set as a recognition target object.
After that, as the offense and defense switch, player "A-da A-san" 32 becomes the "next batter" and moves to the next batter's circle. As shown in Fig. 13B, user 5 follows player "A-da A-san" 32 with his eyes as he moves to the next batter's circle.
14A, the user 5 turns his field of view 7 toward the player "Bgawa Bsuke" 33, who is now the "batter," and watches the batting. At this time, the player "Ada Ao" 32, who is the object to be recognized, is outside the field of view and is no longer included in the frame image 20.
After that, the offense and defense switch, and player "A-da A-san" 32 moves to the first base position as the "first baseman." Meanwhile, as shown in FIG. 14B , user 5 turns his field of view 7 toward the spectator seats and watches the spectators in the cheering section. Player "A-da A-san" 32 moves to first base in an area outside the field of view of user 5, and user 5 does not watch this movement.
User 5 can understand that, due to the change of offense and defense, player "A-da A-san" 32 is now playing defense. Furthermore, based on the past viewing experience of FIG. 13A , user 5 knows that player "A-da A-san" 32 is at first base when playing defense. Therefore, it is conceivable that user 5 associates player "A-da A-san" 32 with the position of first base in his mind, just as when viewing FIG. 13A , and updates his mental position.
In accordance with this update in the brain, the recognized position estimation unit 19 estimates the recognized position of the player "A-da A-husband" 32 to the "position of first base," which is a fixed position associated with the role of "first baseman." That is, based on the fact that the scene in which the role of the player "A-da A-husband" 32 is "first baseman" as in the viewing in Fig. 13A is being viewed (the player "A-da A-husband" 32 has been rendered in the frame image 20), as well as the fact that the role of the player "A-da A-husband" 32 has again become "first baseman" due to the scene update from the viewing in Fig. 14A to the viewing in Fig. 14D, the recognized position estimation unit 19 estimates the recognized position of the player "A-da A-husband" 32 in the brain of the user 5 to the "position of first base."
In this way, in this third embodiment, role information of the object to be recognized in the scene at the time of past viewing (rendering) is retained, and if the object to be recognized is currently out of the field of view, the role of the object is updated in a scene update, and if there is viewing experience at the time of that role, the recognition position is estimated to be the fixed position of that role.

In the view shown in FIG. 14B, the player "A-ta A-san" 32 corresponds to an object to be recognized that has predetermined role information ("first baseman") that is not included in the "frame image at the current time".
Then, if the player "A-ta A-fu" 32 with the same role information ("first baseman") has been rendered up to the current time, the fixed position associated with the role ("first base position") is estimated as the recognized position.
This makes it possible to prevent the occurrence of saliency that does not exist in the user's brain, and makes it possible to predict visual field information with high accuracy.

The scene in which the role of "A-ta A-huu" player 32 is "first baseman" may be viewed in another past baseball game. That is, even if a scene in which the role of "A-ta A-huu" player 32 is "first baseman" is viewed not only in the game currently being watched but also in another game watched in the past, the "position of first base" may be estimated as the recognized position.
That is, as long as a recognition target object to which the same role information is set has been rendered up to the current time, it is possible to estimate a fixed position related to the role as the recognition position.

In the past, when a state in which the player "A-ta A-fu" 32 was in his regular position "first base position" was rendered, the "first base position" may be estimated as the recognized position.
That is, if a recognition target object (player "A-ta A-fu" 32) with the same role information ("first baseman") set has been rendered in a fixed position related to the role ("first base position") up to the current time, the fixed position related to the role ("first base position") may be estimated as the recognition position.
This makes it possible to estimate the recognition position when the user is sure that the player "A-ta A-fu" 32 is at the "first base position" during defense. On the other hand, since the recognition target object with role information set is often at the "regular position" in most cases, it is highly likely that the object will be rendered as being at the "regular position."

It is also possible to integrate the processes of (Example 1) to (Example 3) to estimate the recognition position of the recognition target object. For example, the processes are executed with priority in the order of (Example 1), (Example 2), and (Example 3).
First, the information with the highest priority is determined to be visual information from the eyes, and (Example 1) is executed. That is, the position where the user 5 is thought to have recognized the recognition target object with his/her eyes is estimated as the recognition position.
When the user 5 is visually recognizing the recognition target object, the position confirmed with the eyes becomes the recognition position of the recognition target object. The recognition position estimation unit 19 determines whether the user 5 is visually recognizing the recognition target object based on whether the recognition target object has been rendered in a frame image (predicted frame image) 20, and estimates the position of the recognition target object in the virtual space S at the time of rendering as the recognition position.
In this case, since no other information is required to estimate the recognition position, the sound information used in (Example 2) and the role information and fixed position information used in (Example 3) are not acquired. When the object to be recognized moves out of the field of view 7 (i.e., is no longer rendered), the sound information used in (Example 2) and the role information and fixed position information used in (Example 3) are acquired. When this information is not available, the last viewing position (the position at the time of last rendering) is maintained as the recognition position.

The next highest priority information is based on sound information emitted from the object to be recognized, and the second embodiment is executed. That is, the recognition position is estimated based on the presence or absence of audio data linked to the object to be recognized and the current generation status information of the audio data (generation position, volume information, etc.).
When the object to be recognized is out of the field of view and there is no visual information, the position where the object is thought to have been recognized by sound is estimated as the recognized position.

As the information with the next highest priority, the role information and the fixed position information are acquired, and (Example 3) is executed. That is, the recognition position is estimated based on the information on whether the user 5 has had a past viewing experience in a scene similar to the current scene and the fixed position information of the object to be recognized in that scene.
When neither visual information nor sound information is acquired, a position that is thought to be associated with a position based on past viewing experiences of a scene similar to the current one is estimated as the recognized position.
If there is no visual information, sound information, role information, or fixed position information, the user 5 is unaware of the existence of the object, and therefore pays zero attention to the object (there is no setting of an object to be recognized, and there is no recognition position).

15 is a flowchart showing an example of estimating the recognition position of a recognition target object. The process shown in FIG. 15 can be said to be an example of a process that integrates the processes of (Example 1) to (Example 3).
Rendering information and scene information for all objects to be recognized in the scene are obtained (step 201).
The rendering information includes any information related to the rendering of the recognition target object. Here, the rendering information includes rendering history information of the recognition target object up to the current time, position information of the recognition target object within the frame image 20, etc.
The scene information includes scene description information relating to the object to be recognized, such as history information of the scene description information up to the current time.
In step 301, an object that is rendered for the first time in the "frame image at the current time" is also set as an object to be recognized, and rendering information and frame information are obtained.

It is determined whether there are any unprocessed objects to be recognized (step 202).
If there are any unprocessed objects to be recognized (Yes in step 202), one of the unprocessed objects to be recognized is selected, and the processing from step 203 onwards is executed.

It is determined whether the selected object to be recognized is included in the "frame image at the current time" (i.e., whether it has been rendered) (step 203). If the object to be recognized is included in the "frame image at the current time" (Yes in step 203), the current role information of the object to be recognized is added to the role list (step 204).
The role list is a list into which role information is input when a recognition target object with role information set thereto has been viewed (i.e., has been rendered) up to the current time. If role information has not been set for the recognition target object, it is not added to the role list. The role list can also be called a role viewed list.
The recognition position is estimated to be the current position of the object to be recognized (step 205). Here, the current position of the object to be recognized corresponds to the position in the virtual space S that corresponds to the position of the object to be recognized in the "frame image at the current time". In step 205, the initial recognition position of an object that is rendered for the first time in the "frame image at the current time" is estimated.
The recognition position of the object to be recognized, whose recognition position has been estimated up to the current time, is updated. Of course, there may be cases where the result is the same as the recognition position estimated in the past.
The estimation of the recognition position of this object to be recognized is now complete, and the process returns to step 202.

If the object to be recognized is not included in the "frame image at the current time" (No in step 203), it is determined whether audio data is associated with the object to be recognized and whether the volume at the time of current rendering exceeds the reference value (step 206).
If step 206 is positive (Yes in step 206), the current role (role information) of the object to be recognized is added to the role list (step 204).
In this way, in this embodiment, even if it is determined that the user 5 has recognized a sound emitted by a recognition target object to which the same role information has been set up until the current time, the role is added to the role list in the same manner as when a recognition target object to which the same role information has been set is rendered.
The recognition position is estimated to be the current position of the recognition target object (step 205). Here, the current position of the recognition target object corresponds to the generation position of the sound emitted from the recognition target object in the virtual space S. The recognition position of the recognition target object whose estimated position has been estimated up to the current time is updated.
The estimation of the recognition position of this object to be recognized is now complete, and the process returns to step 202.

If the result of step 206 is negative (No in step 206), it is determined whether the current role of the object to be recognized has changed since the last time the recognition position was updated (step 207).
If the current role has not changed since the last time the recognized position was updated (No in step 207), the recognized position is not updated (i.e., the recognized position remains unchanged), and the process returns to step 202.

If the current role has changed since the last time the recognition position was updated (Yes in step 207), it is determined whether user 5 has previously viewed a scene in which the object to be recognized plays the current role (step 208).
The determination in step 208 is made by referring to whether the current role information of the object to be recognized is entered in the role list. If the current role information of the object to be recognized is entered in the role list, the result in step 208 is affirmative. If the current role information of the object to be recognized is not entered in the role list, the result in step 208 is negative.

If the user 5 has not previously viewed a scene in which the object to be recognized plays the current role (No in step 208), the recognition position is not updated (i.e., the recognition position remains unchanged), and the process returns to step 202.
If the user 5 has previously viewed a scene in which the object to be recognized plays the current role (Yes in step 208), the recognition position is estimated to be the position of the current object to be recognized (step 209).
Here, the current position of the object to be recognized corresponds to a fixed position related to the role. The recognition position of the object to be recognized, whose estimated position has been estimated in the past frame up to the current time, is updated.
The estimation of the recognition position of this object to be recognized is now complete, and the process returns to step 202.

If there are no unprocessed recognition target objects (No in step 202), the process of estimating the recognition positions of all recognition target objects ends (step 210).
As shown in FIG. 15, by using visual information, current sound information, and past audio information, it is possible to estimate the recognition position of the recognition target object with high accuracy.

In the estimation example shown in FIG. 15 , for a recognition target object to which predetermined role information that is not included in the “frame image at the current time” is set, if it is determined that the user 5 has recognized a sound emitted by a recognition target object to which the same role information is set by the current time, a fixed position related to the role is estimated as the recognition position.
Alternatively, if it is determined that user 5 has recognized a sound made by an object to be recognized that has the same role information set up to the current time and is in a fixed position related to the role, the fixed position related to the role may be estimated as the recognized position.

[Generating a panoramic saliency map]
The generation of the panoramic saliency map by the saliency map generation unit 17 will be described.
FIG. 16 is a flowchart illustrating an example of generating a panoramic saliency map.
First, in step 301, a saliency map for the visual field is generated based on the “frame image at the current time.” In this embodiment, a saliency map for the visual field that includes both saliency based on bottom-up attention and saliency based on top-down attention is generated.
For example, saliency based on bottom-up attention and saliency based on top-down attention are detected separately, and then these are added together to generate a saliency map for the field of view corresponding to the final viewport image.
Also in the same step 301, a panoramic saliency map of the sky (in this embodiment, an equirectangular image in which all pixels have a value of zero) is prepared, and the saliency map for the field of view is pasted at the location corresponding to the viewport.
This makes it possible to generate a panoramic saliency map so that saliency based on bottom-up attention is not generated in the out-of-field area. That is, it is possible to generate a panoramic saliency map in which saliency based on bottom-up attention is zero in the out-of-field area. As a result, it is possible to prevent unnecessary saliency from being generated, and it is possible to solve the above-mentioned problem point (1).
Note that any other method may be used to avoid generating saliency based on bottom-up attention in the out-of-field region, such as masking out-of-field regions after generating a full-sphere saliency map (including both saliency based on bottom-up attention and saliency based on top-down attention).
On the other hand, according to the method of this embodiment, in which a saliency map for the entire field of view is pasted into the field of view area of the sky omnidirectional saliency map, it is possible to reduce the processing load and shorten the processing time.

The recognition positions of all objects to be recognized in the out-of-view area estimated by the recognition position estimation unit 19 are acquired (step 302).
It is determined whether or not there is an object to be recognized in an unprocessed out-of-field area (hereinafter referred to as an out-of-field object) (step 303).
If there are any unprocessed out-of-field objects (Yes in step 303), one of the unprocessed out-of-field objects is selected, and the process of step 304 is executed.

In step 304, the position of the out-of-field object in the omnidirectional saliency map (position on the 2D map) is calculated based on the recognized position of the out-of-field object in the virtual space S. The top-down based saliency generated by the out-of-field object is placed at the calculated position.
An out-of-view object is an object that has been rendered in the past, and therefore has previously had its saliency detected based on top-down attention in step 301. For example, the saliency value of each pixel along the shape of the object is detected.
In this embodiment, the saliency based on the top-down attention for the object to be recognized detected in step 301 is retained. Then, in step 304, the retained saliency (shape and value) based on the top-down attention is reused and arranged on the omnidirectional saliency map.
The top-down attention-based saliency placement of this out-of-view object based on its perceived location is now complete, and we return to step 303.

In addition, in step 304, a method may be adopted in which, after generating a saliency map (top-down attention) for the entire sky, the position at which saliency occurs in the out-of-field area is adjusted to match the estimated recognition position.
On the other hand, by reusing the saliency based on the top-down attention detected in step 101 as in this embodiment, rendering processing can be performed only in the viewport, which reduces the processing load and processing time.

If there are no unprocessed out-of-field objects (No in step 302), the process of generating a panoramic saliency map that generates saliency based on top-down attention of out-of-field objects based on their recognized position is terminated.

Salience based on top-down attention is generated at the recognition position recognized in the brain of the user 5. That is, a saliency map representing saliency based on top-down attention in the out-of-field area is generated based on the recognition position of the object to be recognized in the out-of-field area.
This makes it possible to prevent the generation of unnecessary saliency from locations different from the recognition location in the brain, thereby solving the above-mentioned problems (2) and (3).

Fig. 17 is a schematic diagram showing an example of a panoramic saliency map generated by this embodiment. Fig. 17A is an example of a panoramic saliency map 35 generated at the timing shown in Fig. 5A. Fig. 17B is an example of a panoramic saliency map 35 generated at the timing shown in Fig. 5C.
17A , in the panoramic saliency map 35 generated at the timing shown in FIG. 5A , saliency based on the top-down attention is generated only for the person P1 who is recognized by the user 5. No saliency based on the bottom-up attention is generated for the lighting device L that is not recognized by the user 5. Furthermore, no saliency based on the top-down attention is generated for the persons P1 and P2 who are not recognized by the user 5.
17B , for user 5 who is not aware of the movement of person P1, salience based on the top-down attention of person P1 is generated at the position of person P1 before the movement. Furthermore, salience based on the bottom-up attention of lighting device L, which is not recognized by user 5, is not generated.
In this way, the panoramic saliency map 35 generated by this embodiment avoids the occurrence of saliency that does not exist in the brain of the user 5, resulting in a highly accurate panoramic saliency map.

As described above, the server-side rendering system 1 according to this embodiment estimates the recognition position of the recognition target object in the out-of-field area 21. Based on the estimated recognition position, a panoramic saliency map 35 including a saliency map in the out-of-field area 21 is generated.
This makes it possible to reflect the attention of the user 5 to areas outside the field of view, depending on the viewing situation at that time, in the omnidirectional saliency map 35. As a result, it becomes possible to generate a highly accurate omnidirectional saliency map 35.
Since the accurate and precise omnidirectional saliency map 35 is generated, it is possible to generate predicted head motion information (particularly orientation information) with extremely high accuracy, and it is possible to sufficiently suppress the problem of response delay (T_m2p time). In other words, it is possible to realize the delivery of high-quality virtual video using the omnidirectional saliency map 35.
The high-precision panoramic saliency map 35 generated in this embodiment can also be used for other purposes.

<Other embodiments>
The present technology is not limited to the above-described embodiments, and various other embodiments can be realized.

In the above, an example is given in which a 6DoF video is distributed as a virtual image. However, the present technology is not limited to this and can also be applied to cases in which a 3DoF video, a 2D video, or the like is distributed. Furthermore, instead of a VR video, an AR video or the like may be distributed as a virtual image.
The present technology can also be applied to stereo images (for example, right-eye images and left-eye images) for viewing 3D images.
The present technology can be applied to content that displays any virtual space where an out-of-field area may occur. Furthermore, the saliency map of the out-of-field area is not limited to the saliency map of the entire virtual space, but may also be generated for a portion of the virtual space that becomes an out-of-field area.

FIG. 18 is a block diagram showing an example of the hardware configuration of a computer (information processing device) 60 that can realize the server device 4 and the client device 3.
The computer 60 includes a CPU 61, a ROM (Read Only Memory) 62, a RAM 63, an input/output interface 65, and a bus 64 that interconnects these components. The input/output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, a drive unit 70, and the like.
The display unit 66 is a display device using, for example, a liquid crystal display, an electroluminescence display, etc. The input unit 67 is, for example, a keyboard, a pointing device, a touch panel, or other operating device. When the input unit 67 includes a touch panel, the touch panel can be integrated with the display unit 66.
The storage unit 68 is a non-volatile storage device such as a HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable recording medium 71 such as an optical recording medium or magnetic recording tape.
The communication unit 69 is a modem, a router, or other communication equipment that can be connected to a LAN, WAN, etc. and that communicates with other devices. The communication unit 69 may communicate either wired or wirelessly. The communication unit 69 is often used separately from the computer 60.
Information processing by the computer 60 having the above-described hardware configuration is realized by cooperation between software stored in the storage unit 68 or the ROM 62, etc., and the hardware resources of the computer 60. Specifically, the information processing method according to the present technology is realized by loading a program constituting the software stored in the ROM 62, etc., into the RAM 63 and executing it.
The program is installed in the computer 60 via, for example, a recording medium 61. Alternatively, the program may be installed in the computer 60 via a global network or the like. Alternatively, any other computer-readable non-transitory storage medium may be used.

An information processing device according to the present technology may be constructed by a plurality of computers connected to each other so as to be able to communicate via a network or the like working together to execute an information processing method and program according to the present technology.
That is, the information processing method and program according to the present technology can be executed not only in a computer system configured by a single computer, but also in a computer system in which multiple computers operate in conjunction with each other.
In this disclosure, a system refers to a collection of multiple components (devices, modules (components), etc.), regardless of whether all of the components are housed in the same housing. Therefore, multiple devices housed in separate housings and connected via a network, and a single device housed in a single housing with multiple modules, are both systems.
The information processing method and program execution according to the present technology by a computer system include both cases where, for example, obtaining visual field information, performing rendering processing, setting a recognition target object, estimating a recognition position, generating a panoramic saliency map, etc. are performed by a single computer, and cases where each process is performed by a different computer. Furthermore, the execution of each process by a specific computer includes having another computer execute part or all of the process and obtaining the results.
In other words, the information processing method and program according to the present technology can also be applied to a cloud computing configuration in which a single function is shared and processed jointly by multiple devices via a network.

The configurations and processing flows of the server-side rendering system, HMD, server device, client device, etc. described with reference to the drawings are merely one embodiment and may be modified as desired without departing from the spirit of this technology. In other words, any other configurations, algorithms, etc. may be adopted to implement this technology.

In this disclosure, to facilitate understanding of the explanation, words such as "approximately,""almost," and "roughly" are used as appropriate. However, there is no clear difference between using and not using words such as "approximately,""almost," and "roughly."
That is, in the present disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center,""central,""uniform,""equal,""same,""orthogonal,""parallel,""symmetrical,""extended,""axialdirection,""cylindrical,""cylindrical,""ring-shaped," and "annular," are concepts that include "substantially center,""substantiallycentral,""substantiallyuniform,""substantiallyequal,""substantially the same,""substantiallyorthogonal,""substantiallyparallel,""substantiallysymmetrical,""substantiallyextended,""substantially axial direction,""substantiallycylindrical,""substantiallycylindrical,""substantiallyring-shaped,""substantiallyannular," and the like.
For example, this also includes states that fall within a predetermined range (e.g., a range of ±10%) based on criteria such as "perfectly centered,""perfectlycentral,""perfectlyuniform,""perfectlyequal,""perfectly the same,""perfectlyperpendicular,""perfectlyparallel,""perfectlysymmetrical,""perfectlyextended,""perfectlyaxial,""perfectlycylindrical,""perfectlycylindrical,""perfectlyring-shaped," and "perfectly annular."
Therefore, even if the words "roughly,""almost,""approximately," etc. are not added, it may include concepts that can be expressed by adding "roughly,""almost,""approximately," etc. Conversely, a state expressed by adding "roughly,""almost,""approximately," etc. does not necessarily exclude a complete state.

In the present disclosure, expressions using "than", such as "greater than A" and "smaller than A", are expressions that comprehensively include both concepts that include the case where something is equivalent to A and concepts that do not include the case where something is equivalent to A. For example, "greater than A" is not limited to cases that do not include equivalent to A, but also includes "A or greater". Furthermore, "smaller than A" is not limited to "less than A" but also includes "A or less".
When implementing the present technology, specific settings and the like may be appropriately adopted from the concepts included in "greater than A" and "smaller than A" so as to achieve the effects described above.

It is also possible to combine at least two of the features of the present technology described above. In other words, the various features described in each embodiment may be combined in any way, regardless of the embodiment. Furthermore, the various effects described above are merely examples and are not limiting, and other effects may also be achieved.

The present technology can also be configured as follows.
(1)
a rendering unit that generates two-dimensional video data according to the user's field of view by performing a rendering process on three-dimensional space data that constitutes a virtual space based on field of view information regarding the user's field of view;
an estimation unit that estimates a recognition position of a recognition target object recognized by the user in an out-of-field area of the virtual space that is not included in the user's field of view;
and a generation unit that generates a saliency map that represents saliency in the out-of-field area based on the estimated recognition position of the recognition target object in the out-of-field area.
(2) The information processing device according to (1),
The information processing device, wherein the estimation unit sets an object that has been a rendering target up to a current time as the recognition target object.
(3) The information processing device according to (1) or (2),
the two-dimensional video data is composed of a plurality of frame images that are successive in time series,
The information processing device wherein the estimation unit estimates the recognition position of the recognition target object that is not included in the frame image at the current time based on a position in the virtual space that corresponds to a position of the recognition target object in a most recent past frame image that includes the recognition target object.
(4) The information processing device according to (3),
The information processing device, wherein the estimation unit estimates, as the recognition position, a position in the virtual space corresponding to a position of the object to be recognized in the most recent frame image.
(5) The information processing device according to (3) or (4),
The information processing device, wherein the estimation unit estimates, as the recognition position, a position shifted along a movement direction of the recognition target object from a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image.
(6) The information processing device according to any one of (3) to (5),
When it is determined that the user has recognized a sound emitted by the object to be recognized that is not included in the frame image at the current time, the estimation unit estimates the position where the sound is generated in the virtual space as the recognition position.
(7) The information processing device according to any one of (3) to (6),
the three-dimensional space data includes three-dimensional space description data that defines a configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space;
the three-dimensional space description data includes role information representing a role of the object to be recognized, and fixed position information representing a fixed position associated with the role;
When the recognition target object is set with predetermined role information that is not included in a frame image at a current time, and the recognition target object is set with the same role information, the estimation unit estimates the fixed position related to the role as the recognition position when the recognition target object is set with the same role information that is not included in a frame image at a current time and has been rendered before the current time.
(8) The information processing device according to (7),
The information processing device, wherein the estimation unit estimates the fixed position related to the role as the recognition position when a state in which the recognition target object, to which the same role information is set, has been rendered up to a current time and is in the fixed position related to the role.
(9) The information processing device according to any one of (3) to (8),
the three-dimensional space data includes three-dimensional space description data that defines a configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space;
the three-dimensional space description data includes role information representing a role of the object to be recognized, and fixed position information representing a fixed position associated with the role;
When it is determined that the user has recognized a sound emitted by the recognition target object, to which predetermined role information that is not included in a frame image at a current time has been set, the estimation unit estimates the fixed position related to the role as the recognition position by the current time.
(10) The information processing device according to (9),
When it is determined that the user has recognized a sound made by the object to be recognized, to which the same role information has been set up until the current time, while the object is in the fixed position related to the role, the estimation unit estimates the fixed position related to the role as the recognition position.
(11) The information processing device according to any one of (1) to (10),
The information processing device, wherein the estimation unit estimates the recognition position based on a position of the recognition target object in the two-dimensional video data in which the recognition target object is rendered.
(12) The information processing device according to any one of (1) to (11),
The information processing device wherein the generation unit generates the saliency map in which saliency based on bottom-up attention in the area outside the visual field is zero.
(13) The information processing device according to any one of (1) to (12),
The information processing device wherein the generation unit generates the saliency map representing saliency based on top-down attention in the out-of-field region based on the recognition position of the recognition target object in the out-of-field region.
(14) The information processing device according to any one of (1) to (13),
The information processing device is configured to generate the saliency map in the out-of-field region and a saliency map representing saliency of the two-dimensional video data.
(15) The information processing device according to any one of (1) to (14), further comprising:
a prediction unit that generates future visual field information as predicted visual field information based on the saliency map;
The rendering unit generates the two-dimensional video data based on the predicted field of view information.
(16) The information processing device according to (15),
The information processing device, wherein the field of view information includes at least one of a viewpoint position, a line of sight direction, a line of sight rotation angle, a head position of the user, or a head rotation angle of the user.
(17) The information processing device according to (16),
the field of view information includes a rotation angle of the user's head;
The information processing device, wherein the prediction unit predicts a future head rotation angle of the user based on the saliency map.
(18) The information processing device according to any one of (15) to (17),
the two-dimensional video data is composed of a plurality of frame images that are successive in time series,
The rendering unit generates a frame image based on the predicted view information and outputs the generated frame image as a predicted frame image.
(19)
generating two-dimensional video data according to the user's field of view by performing a rendering process on three-dimensional space data constituting a virtual space based on field of view information relating to the user's field of view;
Estimating a recognition position of a recognition target object recognized by the user in an out-of-field area of the virtual space that is not included in the user's field of view;
An information processing method executed by a computer system, which generates a saliency map representing saliency in the out-of-field area based on the estimated recognition position of the recognition target object in the out-of-field area.

S...Virtual space 1...Server-side rendering system 2...HMD
DESCRIPTION OF SYMBOLS 3: Client device 4: Server device 5: User 7: User's field of view 8: Rendered image 13: Prediction unit 14: Rendering unit 17: Saliency map generation unit 18: Saliency map recording unit 19: Recognition position estimation unit 20: Predicted frame image (frame image)
35... Full-dome saliency map 60... Computer

Claims (19)

a rendering unit that generates two-dimensional video data according to the user's field of view by performing a rendering process on three-dimensional space data that constitutes a virtual space based on field of view information regarding the user's field of view;
an estimation unit that estimates a recognition position of a recognition target object that is assumed to be recognized by the user as existing in the virtual space among objects existing in the virtual space, in an out-of-field area of the virtual space that is not included in the user's field of view; and
and a generation unit that generates a saliency map that represents saliency in the out-of-field area based on the estimated recognition position of the recognition target object in the out-of-field area. 2. The information processing device according to claim 1,
The information processing device, wherein the estimation unit sets an object that has been a rendering target up to a current time as the recognition target object. 2. The information processing device according to claim 1,
the two-dimensional video data is composed of a plurality of frame images that are successive in time series,
The information processing device wherein the estimation unit estimates the recognition position of the recognition target object that is not included in the frame image at the current time based on a position in the virtual space that corresponds to a position of the recognition target object in a most recent past frame image that includes the recognition target object. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation unit estimates, as the recognition position, a position in the virtual space corresponding to a position of the object to be recognized in the most recent frame image. 4. The information processing device according to claim 3,
The information processing device, wherein the estimation unit estimates, as the recognition position, a position shifted along a movement direction of the recognition target object from a position in the virtual space corresponding to the position of the recognition target object in the most recent frame image. 4. The information processing device according to claim 3,
When it is determined that the user has recognized a sound emitted by the recognition target object that is not included in the frame image at the current time, the estimation unit estimates the position where the sound is generated in the virtual space as the recognition position. 4. The information processing device according to claim 3,
the three-dimensional space data includes three-dimensional space description data that defines a configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space;
the three-dimensional space description data includes role information representing a role of the object to be recognized, and fixed position information representing a fixed position associated with the role;
When the recognition target object, to which predetermined role information not included in the frame image at the current time is set, has been rendered up to the current time, the estimation unit estimates the fixed position related to the role as the recognition position. 8. The information processing device according to claim 7,
The information processing device, wherein the estimation unit estimates the fixed position related to the role as the recognition position when a state in which the recognition target object, to which the same role information is set, has been rendered up to a current time and is in the fixed position related to the role. 4. The information processing device according to claim 3,
the three-dimensional space data includes three-dimensional space description data that defines a configuration of the virtual space and three-dimensional object data that defines three-dimensional objects in the virtual space;
the three-dimensional space description data includes role information representing a role of the object to be recognized, and fixed position information representing a fixed position associated with the role;
When it is determined that the user has recognized a sound emitted by the recognition target object, to which predetermined role information that is not included in the frame image at the current time has been set, the estimation unit estimates the fixed position related to the role as the recognition position by the current time. 10. The information processing device according to claim 9,
When it is determined that the user has recognized a sound made by the object to be recognized, to which the same role information has been set up until the current time, while the object is in the fixed position related to the role, the estimation unit estimates the fixed position related to the role as the recognition position. 2. The information processing device according to claim 1,
The information processing device, wherein the estimation unit estimates the recognition position based on a position of the recognition target object in the two-dimensional video data in which the recognition target object is rendered. 2. The information processing device according to claim 1,
The information processing device wherein the generation unit generates the saliency map in which saliency based on bottom-up attention in the area outside the visual field is zero. 2. The information processing device according to claim 1,
The information processing device wherein the generation unit generates the saliency map representing saliency based on top-down attention in the out-of-field region based on the recognition position of the recognition target object in the out-of-field region. 2. The information processing device according to claim 1,
The information processing device is configured to generate the saliency map for the out-of-field region and a saliency map representing saliency of the two-dimensional video data. The information processing device according to claim 1, further comprising:
a prediction unit that generates future visual field information as predicted visual field information based on the saliency map;
The rendering unit generates the two-dimensional video data based on the predicted field of view information. 16. The information processing device according to claim 15,
The information processing device, wherein the field of view information includes at least one of a viewpoint position, a line of sight direction, a line of sight rotation angle, a head position of the user, or a head rotation angle of the user. 17. The information processing device according to claim 16,
the field of view information includes a rotation angle of the user's head;
The information processing device, wherein the prediction unit predicts a future head rotation angle of the user based on the saliency map. 16. The information processing device according to claim 15,
the two-dimensional video data is composed of a plurality of frame images that are successive in time series,
The rendering unit generates a frame image based on the predicted view information and outputs the generated frame image as a predicted frame image. generating two-dimensional video data according to the user's field of view by performing a rendering process on three-dimensional space data constituting a virtual space based on field of view information relating to the user's field of view;
Estimating a recognition position of a recognition target object that is assumed to be recognized by the user as existing in the virtual space among objects existing in the virtual space, in an out-of-field area of the virtual space that is not included in the user's field of view, and that is assumed to be recognized by the user at the current time ;
An information processing method executed by a computer system, which generates a saliency map representing saliency in the out-of-field area based on the estimated recognition position of the recognition target object in the out-of-field area.