JP7557442B2 - 3DCG rendering device, system, method and program - Google Patents

Description

The present invention relates to a 3DCG rendering device, system, method, and program that renders 3DCG from a user's viewpoint and outputs a rendered image, and in particular to a 3DCG rendering device, system, method, and program that can output an image with high-quality motion parallax expression even when the user's viewpoint changes rapidly.

There are two methods of expressing stereoscopic images using 3DCG: one that uses a head-mounted display such as a head-mounted display (HMD), and one that uses a stationary 3D display such as a light field display.

Either method requires accurately estimating the user's head posture (position of both eyes), rendering the 3DCG from viewpoints corresponding to the user's left and right eyes, and presenting the resulting two-view images to the user's eyes. This makes it possible to present the user with a three-dimensional image as if the 3DCG were right in front of their eyes.

If an image rendered from a different viewpoint from the actual one is presented at this time, a phenomenon known as cybersickness (VR sickness) occurs, significantly impairing the quality of the experience. In addition, the user's head posture is constantly changing between when the system renders the 3DCG and when it is displayed on the screen. In particular, when the user consciously moves their head, the display delay (motion-to-photon latency) can lead to large errors in head posture, and the quality of the experience can be significantly impaired due to the mismatch between the reality and the head posture of the image (geometric inconsistency in the 3DCG display).

Non-Patent Document 1 discloses a technology that reduces errors and improves image quality by predicting the position of the user's eyes based on the delay time until the image appears on the display.

On the other hand, recent improvements in mobile network technology have made it possible to transmit high-quality video with low latency, and 3DCG rendering systems have emerged in which 3DCG rendering is performed on a cloud server and the HMD receives and plays the video.

In such a system, in addition to the display processing delay mentioned above, there is also a delay due to the time required for network transmission, which can lead to a large error between the head pose assumed during rendering and the actual head pose at the moment the image is displayed, which can significantly degrade the quality of experience.

Patent document 1 discloses a technology in which, in addition to predicting the user's head pose on the server side, the HMD estimates the latest head pose after receiving the video and corrects the display position of the video to match the latest head pose.

Patent document 2 discloses a technology in which the latest head pose is estimated after a rendered image is generated in an HMD, the head pose is re-predicted, and the display position of the image is corrected to match the re-prediction result.

Patent document 3 discloses a method for image display correction that uses affine transformation (translation, rotation, and scaling) of the entire image or each object region.

Patent Publication No. 2021-56783 JP 2017-76119 A Patent Publication No. 2019-532395

S. Lee, B. Kang and D. Nam, "Position Prediction for Eye-tracking based 3D Display", Digital Holography and 3-D Imaging, 2019

With any of the above prior art technologies, it is difficult to resolve the geometric inconsistencies in the model's representation of motion parallax caused by prediction errors in head pose, and there is a particular issue of impaired perceived quality when displaying 3DCG images that are displayed at a depth close to the user.

In Non-Patent Document 1, prediction errors in the position of the user's eyes directly lead to geometric inconsistencies in the model display. Because prediction of the position of the eyes is performed using linear prediction, the inconsistency increases if the user's head suddenly starts moving or stops.

In Patent Documents 1 and 2, the image display position is corrected using the latest head posture information, but the correction is limited to translation. Therefore, although accurate correction is possible if the user's head posture (position of both eyes) is only a rotational movement without translational movement, changes in appearance caused by translational movement cannot be corrected, resulting in geometric inconsistencies.

Changes in appearance caused by translational motion (motion parallax representation) occur according to the relative positions of multiple objects relative to one another. For example, there are differences in the speed at which the display position of each object moves as the user's head translates, changes in the occlusion state when objects overlap, and changes in the appearance of a single object (e.g., when the head moves to the right, a new face on the right becomes visible, and when the head moves to the left, a new face on the left becomes visible).

In Patent Document 3, the representation of motion parallax associated with the relative positions of multiple objects is relatively accurately corrected, but it is unable to correct changes in the appearance of a single object. This poses the problem of geometric inconsistencies occurring, particularly when 3DCG is displayed at a depth close to the user.

The object of the present invention is to provide a 3DCG rendering device, system, method, and program that solves the above technical problems and transmits a rendered video and a depth image together to a display device, and performs three-dimensional image correction using the latest head pose and depth image when correcting the image on the display device, thereby realizing accurate motion parallax expression even when there is a large prediction error in the head pose.

To achieve the above object, the present invention is a 3DCG rendering system that is configured by connecting a local terminal that displays a rendered image from a user's viewpoint and a rendering server that renders 3DCG from the user's viewpoint and transmits it to the local terminal via a network, and is characterized by having the following configuration.

(1) The local terminal is equipped with a viewpoint estimation means for estimating the user's viewpoint at each time as a reference viewpoint at a reference time and estimating it as a latest viewpoint at a latest time that is a delay time after the reference time, the rendering server is equipped with a rendering means for generating a rendering image of the reference viewpoint and outputting it together with its depth image, and the local terminal is further equipped with an image correction means for correcting the rendering image of the reference viewpoint at the latest time to a rendering image of the latest viewpoint based on the depth image, the reference viewpoint, and the latest viewpoint.

(2) The rendering server further includes a viewpoint prediction means for calculating the user's viewpoint at a predicted time when a predicted time has elapsed from the reference time based on the predicted time of the delay time τ and a reference viewpoint as a predicted viewpoint, the rendering means outputs a rendering image of the predicted viewpoint and its depth image, and the image correction means corrects the rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image, the predicted viewpoint, and the latest viewpoint.

(3) When correcting the rendering image of the predicted viewpoint to the rendering image of the latest viewpoint, the surrounding rendering images are used instead of or in addition to the rendering image of the predicted viewpoint.

(4) The image correction means calculates an optimal value for the predicted time based on the difference between the latest time and the predicted time, and feeds this back to the viewpoint prediction means, and the viewpoint prediction means calculates the predicted viewpoint using the time when the optimal value has elapsed from the reference time as the predicted time.

(5) The transmission bit rate from the rendering server to the local terminal is made variable depending on the difference between the latest viewpoint and the predicted viewpoint.

(1) The rendering image of the reference viewpoint is corrected to the rendering image of the latest viewpoint based on the depth image and the latest viewpoint, so that the image correction can reflect three-dimensional changes more than when correction is made without using a depth image. Therefore, even if the user's viewpoint changes significantly during rendering, highly accurate image correction is possible, and it becomes possible to present the user with an image with high-quality motion parallax expression.

(2) The predicted viewpoint is calculated based on the reference viewpoint, and the rendering image of the predicted viewpoint is corrected to the rendering image of the latest viewpoint based on the depth image and the latest viewpoint, so the amount of correction can be reduced compared to when correcting the rendering image of the reference viewpoint. This enables highly accurate image correction, further improving the quality of the rendering image.

(3) By using a peripheral rendering image instead of or in addition to a rendering image of a predicted viewpoint, it is possible to reduce the amount of communication between the rendering means and the image correction means while mitigating the problem of disocclusion.

(4) The optimal value of the prediction time is calculated based on the difference between the latest viewpoint and the predicted viewpoint, and is fed back to the viewpoint prediction means, so that the error between the latest viewpoint and the predicted viewpoint can be reduced. As a result, the amount of correction required when correcting the rendered image of the predicted viewpoint can be reduced, enabling highly accurate image correction and further improving the quality of the rendered image.

(5) The transmission bit rate of the rendering image and the depth image is variable depending on the difference between the predicted viewpoint and the latest viewpoint, which makes it possible to reduce the transmission delay and improve the quality of experience at the same time.

1 is a functional block diagram of a 3DCG rendering system according to a first embodiment of the present invention. FIG. 4 is a functional block diagram of an image correction unit. This is a diagram comparing the appearance of rendering results. FIG. 11 is a functional block diagram of a 3DCG rendering device according to a second embodiment of the present invention. This is a diagram (part 1) showing a rendering method using predicted peripheral viewpoint. This is a diagram (part 2) showing a rendering method using predicted peripheral viewpoint. This is a diagram (part 3) showing a rendering method using predicted peripheral viewpoint. FIG. 11 is a functional block diagram of a 3DCG rendering system according to a third embodiment of the present invention. FIG. 11 is a functional block diagram of a 3DCG rendering device according to a fourth embodiment of the present invention.

The following describes in detail an embodiment of the present invention with reference to the drawings. Figure 1 is a block diagram showing the configuration of the main parts of a 3DCG rendering system according to a first embodiment of the present invention, which is configured by connecting a local terminal (in this embodiment, a display device) 1, which is used for viewing 3DCG by a user, and a rendering server 2, which is used for rendering the 3DCG, via a network NW.

The display device 1 comprises a communication unit 10, a viewpoint estimation unit 11, an image correction unit 12, and a display unit 13. Instead of the display unit 13, an interface for video output may be provided so that a general-purpose display device can be connected. The rendering server 2 comprises a communication unit 20, a viewpoint prediction unit 21, and a rendering unit 22.

In the display device 1, the viewpoint estimation unit 11 estimates viewpoint information P in the global coordinate system of the user viewing the 3D model. In this embodiment, viewpoint information Pt of each of the left and right eyes (hereinafter, sometimes referred to as reference viewpoint Pt) is estimated at a reference time t, and then viewpoint information Pt+τ (hereinafter, sometimes referred to as latest viewpoint Pt+τ) is estimated at a latest time t+τ that is a predetermined delay time τ after the reference time t.

Here, the reference time t means the time at which the viewpoint information is estimated, and the delay time τ is the time that elapses until the image correction unit 12, which will be described in detail later, corrects the rendered image generated based on the reference viewpoint Pt to a rendered image of the latest viewpoint Pt+τ. The reference viewpoint Pt is transmitted to the rendering server 2 via the communication interface 10 and the network. The latest viewpoint Pt+τ is provided to the image correction unit 12.

The viewpoint estimation unit 11 estimates the user's viewpoint information P using any viewpoint estimation technique depending on the format of the display unit 13 on which it is displayed. For example, if the display is an HMD, it is possible to use a tracker mounted on the HMD (which generally estimates the head pose with six degrees of freedom using sensors such as a camera, acceleration sensor, and gyroscope), and if it is a light field display, it is possible to use an eye tracker mounted on the display (which generally estimates the three-dimensional position of both eyes using a high-speed camera and depth sensor).

In this embodiment, the format of the viewpoint information P is not particularly limited. In general, 4x4 model view projection (MVP) matrix information is required for geometric transformation to display each vertex of a 3D model on a display surface. Since the MVP matrix can be calculated, for example, from the three-dimensional position and viewpoint direction of both eyes in a global coordinate system (six degrees of freedom per eye), or the three-dimensional position and direction of the head in a global coordinate system (six degrees of freedom in total), it is desirable to handle the viewpoint information in these formats. Note that the MVP matrix itself may be handled as viewpoint information.

In the rendering server 2, the communication unit 20 acquires the reference viewpoint Pt from the viewpoint estimation unit 11 via the network. The viewpoint prediction unit 21 calculates viewpoint information P't+τ' (hereinafter sometimes referred to as predicted viewpoint P't+τ') at the user's predicted time t+τ' based on the predicted time τ' of the delay time τ.

The predicted time τ' of such a delay time τ is generally called the Motion-to-Photon Latency, and is the sum of various processing times, such as the 3DCG rendering process, display process, and data communication process between each process. It is desirable for the predicted time τ' to match the delay time τ, but the delay time τ fluctuates due to various factors, making it difficult to predict accurately at a given time. Therefore, the predicted time τ' may be a fixed value, or may be dynamically adjustable by the user.

The viewpoint prediction unit 21 calculates the predicted viewpoint P't+τ' of the user at the predicted time t+τ' using any viewpoint prediction technique. For example, by using a general time series filter such as a Kalman filter, future viewpoint information can be estimated from time series data of past viewpoint information. Non-Patent Document 1 discloses a method of applying two types of smoothing filters (moving average method and Savitzky-Golay method) to the time series data of viewpoint information to remove jitter noise, and then estimating the predicted viewpoint with a time series filter.

The rendering unit 22 generates a rendered image of the predicted viewpoint P't+τ' and its depth image by rendering 3DCG using the predicted viewpoint P't+τ' provided by the viewpoint prediction unit 21. In this embodiment, the format of the 3D model used is not particularly limited, but it is assumed that a general 3D model including vertex coordinates/normals/color/texture coordinates, etc. will be used.

Here, the rendered image refers to an RGB color image that is the general result of 3DCG rendering, and can be generated by calculating the model view projection matrix M't+τ' from the predicted viewpoint P't+τ' and rendering the 3DCG.

When performing binocular stereoscopic vision, two different RGB color images corresponding to the left and right eyes are generated. In this case, two different model-view projection matrices ML't+τ' and MR't+τ' corresponding to the left and right eyes are calculated and rendered separately.

When rendering 3DCG from the input viewpoint information, the rendering unit 22 obtains depth images D (D ^L , D ^R ) of the left and right eyes from a memory area generally called a Z buffer, and provides them together with the rendering image to the image correction unit 12 via a network.

In the display device 1, the image correction unit 12 acquires the predicted viewpoint P't+τ', the rendered image and its depth image D (D ^L , D ^R ) from the rendering unit 22 of the rendering server 2, and acquires the latest viewpoint Pt+τ at the latest time t+τ from the viewpoint estimation unit 11. Then, image correction is performed to approximately convert the rendered image rendered at the predicted viewpoint P't+τ' to an image rendered at the latest viewpoint Pt+τ.

When performing binocular stereoscopic vision, the image correction unit 12 performs correction processing on the rendering images I ^L and I ^R of each eye to generate corrected rendering images I ^L'' , I ^R'' , and outputs these to the display unit 13 as rendering images for output.

Figure 2 is a functional block diagram showing the configuration of the image correction unit 12, which mainly consists of a corrected depth image generation unit 121 and a corrected image generation unit 122. Here, the operation of the image correction unit 12 will be explained using binocular stereoscopic vision as an example.

The corrected depth image generation unit 121 generates corrected depth images D ^{L ", D R" corresponding to the latest viewpoint Pt+τ based on the predicted viewpoint P't+τ', the latest viewpoint Pt+τ, and the depth images D L, D R. The corrected image generation unit 122 generates corrected images (corrected rendering images) I L" , I R" corresponding} to the latest viewpoint Pt+τ based on the predicted viewpoint P't+τ', the latest viewpoint Pt+τ, the rendering images I ^L , I ^R , and the corrected depth images D ^{L "} , D ^{R" .}

The corrected depth image generation unit 121 generates corrected depth images D ^L ", D ^R" by performing viewpoint conversion on the pixel coordinates of the depth images D ^L , D ^R. In this embodiment, based on the predicted viewpoint P't+τ' and the latest viewpoint Pt+τ, a rotation matrix R _dL and a translation vector t dL are calculated to convert the three-dimensional coordinate X _d of the depth image D ^L in the global coordinate system into the three-dimensional coordinate X _dL" of the corrected depth image _D ^L" .

Furthermore, each pixel coordinate (u _d , v _d ) of the depth image D and its depth value d (u _d , v _d ) are converted (back projected) into three-dimensional coordinates (X _d ^L" , Y _d ^L" , Z _d ^L" ) by applying the following equation (1). The corresponding two-dimensional pixel coordinates (u _d ^L" , v _d ^L" ) of the corrected depth image D ^L" are calculated by applying the following equation (2) to these three-dimensional coordinates. As shown in the following equation (3), the depth value d ^L" (u _d ^L" , v _d ^L" ) becomes the three-dimensional coordinate Z _d ^L" .

Here, K is the internal parameter matrix of the camera, and a value suitable for the display device 1 to be used is set in advance. Alternatively, an appropriate parameter matrix may be obtained from the display unit 13 via a specified API, or may be automatically calculated by applying a specified HMD calibration method when the system is started for the first time.

In this embodiment, the depth image ^DL can be converted into a corrected depth image DL ^" by performing each of the above processes on all pixel coordinates of the depth image ^DL . In addition, a corrected depth image D ^R" for the right eye can be generated by repeating each of the above procedures for the right eye.

Instead of the depth image D, a reduced depth image Ds obtained by reducing the size (reducing the resolution) of the depth image D may be used to generate reduced corrected depth images DsL ^" , DsR ^" , and the reduced corrected depth images DsL ^" , DsR ^" may be enlarged to generate D ^L" , D ^R. This reduces the number of pixel conversions, which is expected to reduce the processing load.

Furthermore, since the generated corrected depth image D ^L generally contains noise and defects, noise removal and smoothing may be performed by applying a median filter or a bilateral filter.

In addition, if multiple depth images D are provided, the above processes are performed on all depth images D, and the corrected depth images obtained for each depth image D are integrated. This results in a corrected depth image in which defects in the disocclusion regions have been corrected.

The corrected image generation unit 122 receives the rendered image I (I ^L , I ^R ) of the predicted viewpoint P't+τ' as input, and generates rendered images (corrected images) I ^L'' , I ^R'' of the latest viewpoint Pt+τ by performing viewpoint transformation of the pixel coordinates. Specifically, based on the predicted viewpoint P't+τ' and the reference viewpoint Pt+τ, it calculates a rotation matrix R _iL and a translation vector t iL that convert the three-dimensional coordinates X _iL of the rendered image I ^L in the global coordinate system into three-dimensional coordinates X _iL'' ^of the corrected image I ^L'' , and calculates pixel coordinates (u _{iL , v iL ) of the rendered image I L corresponding to the pixel coordinates (u iL'' ,} ^v _{iL '' )} of the corrected image I L'' using the following equation (4).

Here, the color values of the corrected image I ^L'' can be obtained by referencing the color values of the pixel coordinates (u _iL , v _iL ) of the rendered image I ^L. By repeating each of the above processes for all pixel coordinates of the corrected image I ^L'' , a corrected image I ^L'' for the left eye can be generated. Furthermore, by switching the rotation matrix and translation vector for the right eye and repeating the same processes, a corrected image I ^R'' for the right eye can also be generated.

The technology for generating an image from a new viewpoint from an image captured or rendered from a specific viewpoint is generally known as Novel View Synthesis (NVS), and there are many techniques for this. Therefore, the image correction unit 12 of this embodiment is not limited to the configuration shown in FIG. 2, and any NVS technique can be applied.

In this embodiment, the rendered image and its depth image transmitted from the rendering server 2 to the display device 1 are encoded and compressed by any video encoding method such as H.264. At this time, the image correction unit 12 calculates the difference between the predicted viewpoint P't+τ' and the latest viewpoint Pt+τ, and if the difference is equal to or greater than a predetermined threshold, it may provide bit rate control information to the rendering server 2, and the communication unit 20 may reduce the bit rate (image quality/resolution) of the rendered image.

In this way, when the difference between the predicted viewpoint P't+τ' and the latest viewpoint Pt+τ is large and the accuracy of viewpoint prediction decreases, the accuracy of image correction also decreases. In this embodiment, however, the bit rate of the rendering image is lowered according to the difference to reduce the amount of transmission delay, thereby increasing the accuracy of viewpoint estimation and thus improving the accuracy of image correction. This makes it possible to effectively improve the accuracy of image correction even when the accuracy of viewpoint prediction is poor, thereby improving the quality of experience.

Alternatively, the image correction unit 12 may provide bit rate control information to the rendering server 2 when the difference between the predicted viewpoint P't+τ' and the latest viewpoint Pt+τ is equal to or greater than a predetermined threshold, and the communication unit 20 may increase the bit rate (image quality/resolution) of the depth image. If the difference is large, the accuracy of viewpoint prediction decreases, and the accuracy of image correction also decreases. In this embodiment, however, the bit rate of the depth image increases according to the difference, improving accuracy, and therefore improving the accuracy of image correction can be expected. This makes it possible to effectively improve the accuracy of image correction even when the accuracy of viewpoint prediction is poor, thereby improving the quality of experience.

Here, when calculating the difference between the predicted viewpoint P't+τ' and the latest viewpoint Pt+τ, the image correction unit 12 may lower the bit rate of the rendering image or increase the bit rate of the depth image if the norm of the translation vector t _i L during viewpoint transformation (three-dimensional coordinate transformation) is equal to or greater than a predetermined threshold. This allows the delay to be effectively reduced or the accuracy of the depth image to be improved only when the amount of stereoscopic image correction is large, thereby improving the accuracy of image correction.

The communication path between the communication units 10 and 20 is not limited to general networks such as the Internet, WAN, or LAN, but may be a p2p network such as Bluetooth, or a serial communication such as USB.

The display unit 13 displays the corrected image output by the image correction unit 12 on a display. In this embodiment, various display devices such as a monocular/binocular HMD, a smartphone, a tablet, a light field display, etc. can be used as the display unit 13, and the type of display is not particularly limited. Meanwhile, depending on the number of corrected images used by the display unit 13, the rendering unit 22 and the image correction unit 12 need to generate an appropriate number of rendering images (number of viewpoints).

For example, in the case of a monocular HMD, smartphone, or tablet, the number of rendering images and corrected images is one. In contrast, in the case of a binocular HMD or a light field display with viewpoint tracking, two rendering images and corrected images are required. Furthermore, in the case of a light field display (multi-eye display or super multi-eye display) that requires input of images from three or more viewpoints, the rendering unit 22 and the image corrector 12 generate the number of rendering images and corrected images required for the display (number of viewpoints).

Figure 3 shows a comparison of how the rendering image appears when viewed at the actual binocular position, as the correct image, according to this embodiment and other methods.

The rendering image at the predicted viewpoint P't+τ' deviates greatly from the correct image due to the influence of the delay time τ. Furthermore, in the conventional method of correcting the rendering image at the predicted viewpoint P't+τ' with the latest viewpoint Pt+τ, a moderate deviation occurs because the correction cannot reflect three-dimensional changes. In contrast, if the rendering image at the predicted viewpoint P't+τ' is corrected with the latest viewpoint Pt+τ and the depth image as in this embodiment, the correction can reflect three-dimensional changes, thereby reducing the deviation.

Figure 4 is a functional block diagram showing the configuration of a 3DCG rendering device 3 according to a second embodiment of the present invention. The same reference numerals as those described above represent the same or equivalent parts, and therefore their explanation will be omitted.

In the above first embodiment, the configuration for implementing the present invention is distributed between the display device 1 and the rendering server 2, and the functions of the present invention are implemented by a system in which the display device 1 and the rendering server 2 operate cooperatively via a network.

In contrast, in this embodiment, each function is realized by a single 3DCG rendering device 3. Such a 3DCG rendering device 3 can be configured by implementing an application (program) that realizes each function on a general-purpose computer or server. Alternatively, it can be configured as a dedicated machine or a single-function machine in which part of the application is implemented as hardware or software.

In this embodiment, there is no need to consider the network delay considered in the first embodiment, so the viewpoint prediction unit 21 can remove the network delay equivalent from the predicted time τ' of the delay time τ. As a result, the difference between the delay time τ and the predicted time τ' becomes smaller, so the accuracy of the viewpoint after correction by the image correction unit 12 can be improved.

In addition, in each of the above embodiments, the delay time τ used by the viewpoint prediction unit 21 to calculate the predicted viewpoint P't+τ' has been described as being fixed, but the present invention is not limited to this, and the delay time τ may be adaptively controlled based on the difference between the delay time τ and the predicted time τ'.

That is, the image correction unit 12 may calculate an optimal value of the predicted time τ' that minimizes the difference between the latest time t+τ and the predicted time t+τ', and feed this back to the viewpoint prediction unit 21, which may then use this optimal value as the predicted time τ' to calculate the predicted viewpoint P't+τ'. This reduces the error of the predicted viewpoint P't+τ' relative to the latest viewpoint Pt+τ, reduces the amount of correction by the image correction unit 12, and enables highly accurate image correction, thereby further improving the quality of the rendered image.

Here, in the first and second embodiments, it has been described that when performing binocular stereoscopic vision, the rendering unit 22 obtains two depth images D ^L , D ^R corresponding to two rendering images I ^L , I ^R for both eyes (left and right), and provides the rendering images I ^L , I ^R and the depth images D ^L , D ^R to the image correction unit 12, but the present invention is not limited to this.

For example, one common depth image D ^LR may be acquired by calculating an MVP matrix MC't+τ' corresponding to a viewpoint at the center position of both eyes (central viewpoint) from the predicted viewpoint P't+τ' and performing rendering. This allows the amount of communication between the rendering unit 22 and the image correction unit 12 to be reduced compared to the case of transmitting two depth images D ^L and D ^R , so that the transmission delay can be reduced particularly in the first embodiment. The image correction unit 12 corrects the two rendering images I ^L and I ^R to two rendering images I ^L'' , I ^R'' corresponding to the latest viewpoint Pt+τ in one common depth image D ^LR .

Alternatively, as shown in Figure 5, the viewpoint prediction unit 21 may predict, in addition to the predicted viewpoints P't+τ' (P ^L' , P ^R' ) of the left and right eyes, surrounding viewpoints (predicted peripheral viewpoints) P ^{L'2 ,} P ^R'2 , for example viewpoints outside the predicted viewpoints P L', P ^R' , and the rendering unit 22 may generate peripheral rendering images I ^L2 , I ^R2 corresponding to the predicted peripheral viewpoints P ^L'2 , P ^R'2 in addition to the rendering images I ^L , I ^R corresponding to the predicted viewpoints P ^L' , ^P R'.

The rendering images I ^L , I ^R and the peripheral rendering images I ^L2 , I ^R2 are provided to the image correction unit 22 together with their respective depth images. However, since the depth images of the peripheral rendering images I ^L2 , I ^R2 can be generated based on the rendering images I ^L , I ^R of the predicted viewpoints P ^L' , P ^R' , which are different viewpoints, it is also possible to provide only the depth images of the rendering images I ^L , I ^R. Note that the predicted peripheral viewpoints P ^L'2 , P ^R'2 are not limited to the outside of the predicted viewpoints P ^L' , P ^R' , and the upper or lower side of the predicted viewpoints P ^L' , P ^R' may be the predicted peripheral viewpoints.

As shown in FIG. 6, the viewpoint prediction unit 21 may adjust the placement of the predicted peripheral viewpoints depending on the direction of the user's head movement. Here, FIG. 6(a) shows an example where the direction of the user's head movement is vertical, and FIG. 6(b) shows an example where the direction of the user's head movement is horizontal. Note that the direction of the user's head movement is not limited to the vertical and horizontal directions in FIG. 6(a) and (b), and can be any direction, such as a diagonal direction.

For example, the viewpoint prediction unit 21 calculates the rotation axis of the head movement passing through the center of the 3DCG model by the cross product of the three points of the center of the displayed 3DCG model, the reference viewpoint Pt, and the predicted viewpoint P't+τ'. Then, the predicted peripheral viewpoints P ^L'2 , P ^R'2 can be calculated by rotating one of the predicted viewpoints P ^L' , P ^R ' in the forward direction and the other in the reverse direction by a rotation angle θ around the rotation axis of the head movement. Here, the rotation angle θ may be specified as a fixed value, for example, 5 degrees, or may be adjusted according to the speed of the user's head movement, such as increasing the rotation angle θ as the speed increases. By adjusting the arrangement of the predicted peripheral viewpoints in consideration of the direction of the user's head movement, it becomes possible to generate a corrected image with higher accuracy.

The image correction unit 22 repeats each of the above processes using the rendering images I ^L , I ^R , the peripheral rendering images I ^L2 , I ^R2 and the depth image to perform stereoscopic image correction, thereby eliminating the problem known as disocclusion, in which new faces of a 3D model that are not included in the rendering image are displayed missing due to motion parallax.

Alternatively, as shown in Fig. 7, the viewpoint prediction unit 21 may predict only the predicted peripheral viewpoints P ^L'2 , P ^R'2 , and the rendering unit 22 may generate only the peripheral rendering images I ^L2 , I ^R2 based on the MVP matrices M ^L2 , M ^R2 corresponding to the predicted peripheral viewpoints P ^{L'2 ,} P ^R'2 . At this time, the depth images of the peripheral rendering images I ^L2 , I ^R2 generated by rendering with the same MVP matrix as the peripheral rendering images I L2 , I ^R2 are acquired. The predicted peripheral viewpoint ^P'2t +τ' (P ^L'2 , P ^R'2 ) and its depth image are provided to the image correction unit 12 instead of the predicted viewpoint P't+τ' and its depth image.

The image correction unit 22 generates a corrected image corresponding to the latest viewpoint Pt+τ using the predicted peripheral viewpoint ^P'2t +τ' and its depth image as the predicted viewpoint P't+τ' and its depth image, respectively, and outputs this as a rendering image.

In addition, when the surrounding rendering image and its depth image are input from the rendering unit 22, when calculating the pixel coordinates u _iL , v iL of the rendering image I ^L corresponding to the pixel coordinates u _iL '', v _{iL ''} of the corrected image I ^L'' , it may be possible to check whether the converted depth value d ^IL matches the depth value of the corresponding depth image.

As a result, if there is no match (the difference is greater than a predetermined threshold), it is determined that the pixel is not reflected in the rendering image ^IL (is obscured), and a similar coordinate transformation is performed using another image ( ^IR or a surrounding rendering image) instead.If the depth values match (the difference is less than a predetermined threshold), the color value of the pixel coordinates of the rendering image is referenced.

Generally, it is possible that occlusion occurs and a point in the correct correspondence is hidden in the converted image, but this can be detected by the above check. In addition, a similar process is also performed on the corrected image I ^R'' for the right eye. As a result, when candidates for pixel coordinates of multiple rendering images corresponding to pixel coordinates u _iL'' and v _iL'' of the corrected image I ^L'' for the left eye are obtained, it becomes possible to refer to color values from the correct corresponding points, which not only reduces loss but also enables the generation of a highly accurate corrected image.

In addition, in the above equation (4), the color values of the corrected image I ^L'' are obtained by referencing the color values of the pixel coordinates (u _iL , v _iL ) in the rendered image I ^L , but if a rendered image I ^R for the right eye or peripheral rendered images I ^L2 and I ^R2 exist, the color values of those pixel coordinates can also be referenced. This makes it possible to generate a corrected image I ^L'' with fewer lossy parts.

In this way, by using the peripheral rendering images I ^L2 , I ^R2 instead of the rendering images I ^L , I ^R , it is possible to reduce the amount of communication between the rendering unit 22 and the image correction unit 12 while mitigating the problem of disocclusion.

Up to this point, we have explained the case where the rendering unit 22 generates fixed rendering images corresponding to the left and right eyes and provides them to the image correction unit 12, assuming binocular stereoscopic vision. However, it is also possible to adjust the number of viewpoints to be rendered by receiving input of the number of viewpoints Np that require rendering from the viewpoint estimation unit 11. This makes it possible to reduce unnecessary rendering and save computer resources.

Furthermore, in each of the above embodiments, the rendering server 2 is provided with a viewpoint prediction unit 21, which predicts viewpoint information P't+τ' at time t+τ' when the rendered image is predicted to be provided to the user, and the rendered image is generated based on the predicted viewpoint P't+τ'.

However, the present invention is not limited to this, and the viewpoint prediction unit 21 may be omitted, as in the 3DCG rendering system according to the third embodiment shown in FIG. 8, and the 3DCG rendering device according to the fourth embodiment shown in FIG. 9, and the rendering unit 22 may generate a rendering image I and its depth image D based on the reference viewpoint Pt.

With this configuration, the larger the delay time τ, the greater the amount of correction that the image correction unit 12 makes when correcting the rendering image based on the depth image D, so there is a possibility that the quality of the corrected image will deteriorate compared to the embodiments in which a viewpoint prediction unit 21 is provided.

However, when applied to a 3DCG rendering device 3 that does not need to consider network delay as in the fourth embodiment, or when applied to a 3DCG rendering system that needs to consider network delay as in the third embodiment, the delay time τ will be small if the system is used in an environment that promises low delay. In addition, the time that the viewpoint prediction unit 21 spends on viewpoint prediction can be shortened, so that a sufficiently high-quality rendering image can be provided even if the viewpoint prediction unit 21 is omitted.

The present invention makes it possible to generate high-quality rendering images in a short time and provide them in real time even via communication infrastructure, making it possible to provide a wide variety of entertainment to many people regardless of geographic or economic disparities. As a result, it will be possible to contribute to Goal 9 "Build resilient infrastructure and promote inclusive and sustainable industrialization" and Goal 11 "Make cities inclusive, safe, resilient and sustainable" of the United Nations-led Sustainable Development Goals (SDGs).

1...display device, 2...rendering server, 10...communication unit, 11...viewpoint estimation unit, 12...image correction unit, 13...display unit, 20...communication unit, 21...viewpoint prediction unit, 22...rendering unit, 121...corrected depth image generation unit, 122...corrected image generation unit

Claims (15)

A 3DCG rendering device that renders 3DCG from a user's viewpoint and outputs a rendered image,
A viewpoint estimation means for estimating a viewpoint of a user at each time as a reference viewpoint at a reference time t and as a latest viewpoint at a latest time t+τ which is a delay time τ from the reference time;
A viewpoint prediction means for calculating a user's viewpoint at a predicted time t+τ' when the predicted time τ' has elapsed from the reference time t based on a predicted time τ' of the delay time τ and a reference viewpoint as a predicted viewpoint;
a rendering means for generating a rendering image of the predicted viewpoint and outputting the rendering image together with a depth image;
and an image correction means for correcting, at a latest time, the rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image, the predicted viewpoint, and the latest viewpoint;
A 3DCG rendering device that outputs a rendering image from the latest viewpoint. The viewpoint prediction means calculates predicted viewpoints of the left and right eyes of the user at a predicted time,
the rendering means outputs a rendering image of each predicted viewpoint and a depth image corresponding to an intermediate viewpoint between each predicted viewpoint;
2. The 3DCG rendering device according to claim 1 , wherein the image correction means corrects each rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image of the intermediate viewpoint, the predicted viewpoint and the latest viewpoint. The viewpoint prediction means calculates a predicted viewpoint of the user at a predicted time and predicted peripheral viewpoints which are peripheral viewpoints of the user,
the rendering means outputs at least a peripheral rendering image and a depth image thereof out of a rendering image of a predicted viewpoint and peripheral rendering images of the predicted peripheral viewpoints;
2. The 3DCG rendering device according to claim 1 , wherein the image correction means corrects at least a peripheral rendering image out of the rendering image of the predicted viewpoint and the peripheral rendering images of the predicted peripheral viewpoints to a rendering image of a latest viewpoint. The viewpoint prediction means calculates a predicted viewpoint of the left and right eyes of the user at a predicted time,
The 3DCG rendering device according to claim 3, characterized in that the predicted peripheral viewpoints are calculated by rotating one of the predicted viewpoints of the left and right eyes in the forward direction and the other in the reverse direction by a predetermined angle around a rotation axis of head movement that passes through the center of the 3DCG model. 5. The 3DCG rendering device according to claim 4 , wherein the predetermined angle is set in accordance with a speed of head movement about the rotation axis, the faster the speed is, the larger the angle is set. The 3DCG rendering device according to any one of claims 1 to 5, characterized in that the image correction means calculates an optimal value of the predicted time τ' based on the difference between the latest time and the predicted time and feeds it back to the viewpoint prediction means, and the viewpoint prediction means calculates the predicted viewpoint using the time at which the optimal value has elapsed from a reference time t as the predicted time. In a 3DCG rendering system configured by connecting a local terminal that displays a rendered image from a user's viewpoint and a rendering server that renders 3DCG from the user's viewpoint and transmits it to the local terminal via a network,
The local terminal,
A viewpoint estimation means is provided for estimating a viewpoint of a user at each time as a reference viewpoint at a reference time t and as a latest viewpoint at a latest time t+τ which is a delay time τ from the reference time,
The rendering server,
A viewpoint prediction means for calculating a user's viewpoint at a predicted time t+τ' when the predicted time τ' has elapsed from the reference time t based on a predicted time τ' of the delay time τ and a reference viewpoint as a predicted viewpoint;
A rendering means for generating a rendering image of the predicted viewpoint and outputting it together with a depth image thereof,
The local terminal further comprises:
an image correction means for correcting, at a latest time, the rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image, the predicted viewpoint and the latest viewpoint;
A 3DCG rendering system that outputs a rendering image of the latest viewpoint. The viewpoint prediction means calculates predicted viewpoints of the left and right eyes of the user at a predicted time,
the rendering means outputs a rendering image of each predicted viewpoint and a depth image corresponding to an intermediate viewpoint between each predicted viewpoint;
8. The 3DCG rendering system according to claim 7 , wherein the image correction means corrects each rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image of the intermediate viewpoint, the predicted viewpoint and the latest viewpoint. The viewpoint prediction means calculates a predicted viewpoint of the user at a predicted time and predicted peripheral viewpoints which are peripheral viewpoints of the user,
the rendering means outputs at least a peripheral rendering image and a depth image thereof out of a rendering image of a predicted viewpoint and peripheral rendering images of the predicted peripheral viewpoints;
8. The 3DCG rendering system according to claim 7 , wherein the image correction means corrects at least a peripheral rendering image out of the rendering image of the predicted viewpoint and the peripheral rendering images of the predicted peripheral viewpoints to a rendering image of a latest viewpoint. The viewpoint prediction means calculates predicted viewpoints of the left and right eyes of the user at a predicted time,
The 3DCG rendering system according to claim 9, characterized in that the predicted peripheral viewpoints are calculated by rotating one of the predicted viewpoints of the left and right eyes in the forward direction and the other in the reverse direction by a predetermined angle around a rotation axis of head movement that passes through the center of the 3DCG model. 11. The 3DCG rendering system according to claim 10 , wherein the predetermined angle is set in accordance with a speed of head movement about the rotation axis, the faster the speed is, the larger the angle is set. The 3DCG rendering system according to any one of claims 7 to 11, characterized in that the image correction means calculates an optimal value of the predicted time τ' based on the difference between the latest time and the predicted time and feeds it back to the viewpoint prediction means, and the viewpoint prediction means calculates the predicted viewpoint using the time at which the optimal value has elapsed from the reference time t as the predicted time. 13. The 3DCG rendering system according to claim 7 , wherein a transmission bit rate from said rendering server to said local terminal is variable according to a difference between said latest viewpoint and said predicted viewpoint. A 3DCG rendering method in which a computer renders 3DCG from a user's viewpoint and outputs a rendered image, comprising:
The user's viewpoint at each time is estimated as a reference viewpoint at a reference time t, and as a latest viewpoint at the latest time t+τ, which is a delay time τ from the reference time, and
Calculate a user's viewpoint at a predicted time t+τ' when the predicted time τ' has elapsed from the reference time t based on a predicted time τ' of the delay time τ and a reference viewpoint, as a predicted viewpoint;
A rendering image of the predicted viewpoint is generated and output together with the depth image;
At a latest time, the rendering image of the predicted viewpoint is corrected to a rendering image of a latest viewpoint based on the depth image, the predicted viewpoint, and the latest viewpoint;
A 3DCG rendering method comprising: outputting a rendering image from the latest viewpoint. In a 3DCG rendering program that renders 3DCG from the user's viewpoint and outputs the generated rendered image,
A procedure for estimating the user's viewpoint at each time as a reference viewpoint at a reference time t and as a latest viewpoint at a latest time t+τ, which is a delay time τ from the reference time;
A step of calculating a user's viewpoint at a predicted time t+τ' when the predicted time τ' has elapsed from the reference time t based on a predicted time τ' of the delay time τ and a reference viewpoint as a predicted viewpoint;
generating a rendering image of the predicted viewpoint and outputting it together with a depth image;
and at a latest time, correcting the rendering image of the predicted viewpoint to a rendering image of the latest viewpoint based on the depth image, the predicted viewpoint, and the latest viewpoint;
A 3DCG rendering program that causes a computer to execute the steps of: outputting a rendering image of the latest viewpoint.

Images