JP7572066B2 - Networked VR system - Google Patents

Description

The present invention relates to a network-based VR system in which a remote server generates three-dimensional virtual reality (hereinafter "VR") information, and a local client renders and displays this three-dimensional information on a display, and the system eliminates display inconsistencies caused by the round trip time (hereinafter "RTT") between the remote server and the local client.

The inventor has developed a particle simulation that reproduces on a computer the movement of swarm robots using microtubules and nanostructures using NA, but the number of particles that must be processed simultaneously can reach hundreds of thousands. An operator wearing a head-mounted display remotely controls virtual objects (e.g. particles made of many atoms) that exist only in VR using hand movements (hand gestures). The process of generating three-dimensional information for the remotely controlled virtual object is executed by a remote server. Because the amount of data and number of calculations required to generate three-dimensional information for VR are enormous, it is desirable for the remote server to be high-performance, and it is connected to multiple local clients.

Using a high-performance remote server has both advantages and disadvantages. The RTT is generated, which consists of the transmission time required to transmit information from a local client to the remote server, the calculation time required to generate 3D information on the remote server, and the reception time required to transmit the generated 3D information to the local client.
If for some reason communication becomes extremely slow, or if a job occurs that exceeds the performance of the remote server, causing a significant delay in the generation of three-dimensional information, the local client can simply suspend work for a while and then resume it after observing the situation.
However, the problem is that RTT is unavoidable during remote operation.

Because RTT occurs, it is impossible to render VR 3D information that perfectly corresponds to the current movement of the controlled object (e.g. the operator's hand), and VR 3D information that is out of date by the RTT will be rendered. This means that the actual movement of the controlled object and the rendering of the virtual object in VR do not match, frustrating the operator and hindering their thinking.

Based on this awareness, the present inventor proposed in Non-Patent Document 1 that the future position of the operating object is predicted by the RTT and 3D information of the virtual target object is obtained based on the predicted position. For prediction, regression equations and machine learning can be used, and experiments were conducted for each method.
The results showed that when the RTT was short, there was not much difference between the results, but as the RTT became longer, regression prediction became more prone to error than machine learning.
Therefore, we adopted a method to predict the movement of the operated object using machine learning, but a problem still remains: a gap occurs between the predicted position after dt milliseconds and the actual position of the object after dt milliseconds has passed.

To solve this problem, Non-Patent Document 1 attempts to learn in real time based on gaps within a specified time period, update the machine learning parameters, and immediately use the updated parameters to predict the movement of the manipulated object, i.e., perform real-time inference. If the manipulated object in the predicted position and the generated 3D information are displayed simultaneously in VR, there is almost no sense of discomfort. Of course, there will be a gap between the current position of the manipulated object and the predicted position, but the local client updates the parameters of the predictive model that uses machine learning in real time based on this gap. In other words, the local client performs real-time inference and real-time learning in parallel.

Gutmann, G., Konagaya, A.: Real-time inferencing and training of artificial neural network for adaptive latency negation in distributed virtual environments. In: 2020 International Congress on Human-Computer Interaction, Optimization and Robotic Applications (HORA), pp. 1-8. IEEE (2020)DOI: 10.1109/HORA49412.2020.9152833

However, Non-Patent Document 1 describes the issues, a comparison of experimental results under multiple conditions, future prospects, etc., but does not clearly state specific means for transitioning to practical use. In addition, it does not particularly mention how to deal with the case where the gap between the predicted position and the actual position of the manipulation object is abnormal.
Therefore, in order to create a system that can withstand practical use, the present invention aims to provide a method for improving the accuracy of the predicted position of the operating object and to resolve cases where a gap between the predicted position and the actual position must be judged to be abnormal.

In order to solve the above-mentioned problems, the invention according to claim 1 comprises:
A network-type VR system including a local client that renders 3D information of VR on a head-mounted display or a 2D display worn by an operator, and a remote server that generates 3D information of a virtual object in VR,
The local client includes a means for acquiring a position of an operation object at a current timestamp T from the rendered three-dimensional information;
A means for inferring in real time a predicted position of the object dt milliseconds after a current timestamp T using an AI prediction model utilizing machine learning;
means for transmitting the predicted position based on the inference to the remote server and receiving 3D information of a virtual reality (VR) generated by the remote server using the predicted position;
means for displaying the generated VR three-dimensional information on the head-mounted display or two-dimensional display ;
The present invention is characterized in that real-time learning is performed using the difference between the predicted position of the object dt milliseconds after T and the actual position of the object when dt milliseconds have elapsed since T, and the parameters of the AI prediction model are updated based on the results of this learning .
Here, dt is the RTT.
In other words, the subject of this invention is to predict the future position of the operation object by the RTT and render the 3D information of the VR based on this predicted position. This reduces the mismatch between the actual operation by the operator and the 3D information of the VR, allowing for comfortable operation.
In the following embodiment, the manipulation object is the operator's hands, but it may be any object or robot.

By updating the parameters of the AI prediction model based on the results of real-time learning , it is possible to expect reasonable position predictions without accumulating gaps between predicted and actual positions.

It is desirable to further provide a real-time monitoring means for reinitializing parameters of the AI prediction model when the difference between the predicted position dt milliseconds after T and the actual position of the object when dt milliseconds have elapsed from T continuously indicates an abnormal value.
This will prevent users from continuing with meaningless operations and will provide an opportunity to review any problems with the AI prediction model.

VR 3D information can be displayed in real time on a 3D or 2D display, providing a display that feels natural to the operator, allowing work to proceed comfortably and efficiently without disrupting thinking.

FIG. 1 is a diagram for explaining the concept of RTT. FIG. 11 is a diagram comparing the actual position and the predicted position of the manipulation object. FIG. 13 is a diagram illustrating a cooperative process between a local client and a remote server. 1 is a diagram showing an overall configuration of a system according to a first embodiment and a block structure of each component; FIG. 2 is a flow diagram of an overall process according to the first embodiment; 11 is a flowchart of a process for eliminating a predicted position of a manipulation object that is not suitable for learning data according to the first embodiment. FIG. FIG. 13 is a diagram illustrating a case in which the parameters of the AI prediction model according to the first embodiment should be reinitialized. FIG. 4 is a diagram illustrating a change in RTT according to the first embodiment. FIG. 11 is a diagram for explaining a case where a local client and a remote server according to the first embodiment execute asynchronously.

The system according to the embodiment of the present invention (hereinafter referred to as "the system") will be described in the following order.
[1] Overview of this system
[1-1] Meaning of RTT
[1-2] Movement of the control object
[1-3] Cooperation between local client and remote server [2] Configuration of this system
[2-1] Overall system configuration
[2-2] Functional block configuration [3] Operation of this system
[3-1] Overall processing flow
[3-2] Monitoring the suitability of learning data
[3-3] Changes in RTT

[1] Overview of this system First, we will discuss RTT, which is an important concept for this system, and related matters.

[1-1] Meaning of RTT
The RTT is the total time from transmitting data from a local client to a remote server, executing the 3D information generation process on the remote server, and receiving the processing results by the local client. This will be explained using the example of FIG.
T0, T1, ... on the left side of Figure 1 represent timestamps (time of day (milliseconds) expressed as an integer) at equal intervals. S0, S1, ... are transmission timestamps, and R0, R1, ... are reception timestamps. In the example of Figure 1, S0 is the transmission timestamp at time T0, and R0 is the reception timestamp at time T2. In Figure 1, information sent at time T0 (= S0) is processed by a remote server, and the result is received at time T2 (= R0). The RTT at this time is R0-S0 (= T2-T0). Information sent at time T1 (= S1) is processed by a remote server, and the result is received at time T4 (= R2). The RTT at this time is R2-S1 (= T4-T1).

Here, the RTT with a transmission timestamp of T1 takes 1.5 times longer than the RTT with a transmission timestamp of T0. In this way, the RTT varies depending on the communication environment, etc. This change in RTT will be described later.
The combination of sending and receiving times is stored in short-term memory for the past two seconds or so, because this information may also be referenced in real-time inference using an AI prediction model.

[1-2] Movement of the object This system aims to eliminate the delay in 3D information display caused by RTT by performing real-time inference and real-time learning in parallel. We will explain why eliminating the delay is a problem that needs to be solved with reference to Figure 2.
X(T) is the actual position of the object at the current time (timestamp T). Even if this position information is sent to a remote server, when the result is returned, the object is at the position X(T+RTT). If the remote server generates 3D information assuming that the object is at the position X(T), this 3D information will have a gap from the current object position X(T+RTT), and if this is displayed on a display, the operator will feel uncomfortable. Therefore, the object position after T and RTT is predicted, and this predicted position f(X(T),RTT) is sent to the remote server instead of the current X(T). However, using the predicted position to eliminate the delay in display causes another problem. That is, the predicted position f(X(T),RTT) and the actual position after T and RTT, X(T+RTT), usually do not completely match.

It is important in this system to reduce this discrepancy as much as possible.
However, if the object to be operated is a human finger, the problem becomes more complicated because the finger has a high degree of freedom of movement and the speed of movement is not constant. For example, the speed of moving the palm sideways is different from the speed of bending the fingers.
Therefore, it is important to accurately predict the position of the controlled object. Prediction methods include regression prediction using a quadratic polynomial and prediction using machine learning. Regression prediction is comparable to machine learning when the RTT is small, but as the RTT increases, the likelihood of prediction errors, that is, predictions that are far from the actual position, increases. For this reason, we decided to use machine learning for predictions in this system.

In machine learning, a large amount of data may be collected and learned offline from the collected data, but in this system, data obtained by user operations is learned in real time and the parameters of the AI prediction model are updated sequentially. This allows the user's operations to be immediately reflected in the inference, making it possible to make predictions with a small gap with the actual location. In the case of Figure 2, if the difference between the predicted location after RTT has elapsed from T and the actual location is immediately learned, the gap between the predicted location after another RTT has elapsed from time T+RTT and the actual location can be reduced.
Many machine learning methods have been proposed, such as ADAM (Adaptive Moment Estimation), so you can choose the appropriate method.

[1-3] Coordination between local client and remote server In this system, the remote server is responsible for processing large amounts of data. This enormous amount of calculation is shared by the high-performance and expensive remote server, and the operator performing design work, etc. uses the functions of the remote server from a PC (local client) connected to the head-mounted display he or she wears.
FIG. 3 is a diagram illustrating the cooperative operation between a local client and a remote server.
The local client returns to the local client the result generated by the remote server from the information sent by the local client when the local client was at S0 (step F101, hereinafter simply referred to as "F101"). The RTT at this time is R0-S0 (=RTT0). The local client predicts the position of the object after RTT0 from the current time S1 (F102) and sends it to the remote server (F103). The remote server generates three-dimensional information using the predicted value of time S1+RTT0 (F104) and sends it to the local client (F106). In parallel with the generation of three-dimensional information by the remote server, the local client renders the three-dimensional information received at time R0 (F105). The local client updates the RTT to R1-S1 (=RTT1) at the received time R1, and predicts the position of the object at the time obtained by adding RTT1 to the current time S2 (F107). The predicted value is transmitted to the remote server (F108), and the remote server generates 3D information using the predicted value at time S2+RTT1 (F109), and transmits the information to the local client (F110). In this manner, the rendering on the local client side and the generation of 3D information on the remote server side (hereinafter, sometimes referred to as "simulation") are alternately executed, which is the basic cooperative operation.

The position of the object is predicted at a time that is RTT ahead of the current time, but in the example in Figure 3, the RTT value is the difference between the most recent sending time and receiving time. Therefore, the RTT does not necessarily have a constant value. For this reason, it is not realistic for the local client and remote server to execute alternately as in Figure 3, and in reality, the execution is asynchronous as shown in Figure 9, which will be described later.

Above, we have explained (1-1) the concept of RTT, (1-2) the meaning of transmitting the future predicted position of the operating object to a remote server, and (1-3) cooperation between the local client and the remote server, which are important for this system.
Next, the configuration of this system will be described.

[2] Configuration of the System [2-1] Overall Configuration of the System As shown in FIG. 4, the system has a local client 1 (hereinafter referred to as "client 1") connected to a remote server 2 (hereinafter referred to as "server 2") via a communication network N.
The client 1 predicts the position dt milliseconds later based on the current position of the operation object received from the head mounted display 3 , and renders the three-dimensional information generated by the server 2 to display it on the head mounted display 3 .

In this way, the client 1 and the server 2 are connected via the communication network N and share roles, which is why the RTT problem occurs.
However, the RTT that is an issue with this system is limited to the RTT that normally occurs between server 2, and it does not assume that it will deal with long-lasting delays caused by problems with the communications company, or delays caused by processing congestion due to the occurrence of jobs that exceed the capacity of server 2. When such abnormal delays occur, the operator (hereinafter referred to as the "user") can simply suspend his or her work and resume it after the problem is resolved.

4, the client 1 includes a current position input unit 11, a real-time inference unit 12, a real-time learning unit 13, a real-time monitoring unit 14, a server transmission/reception unit 15, and a rendering processing unit 16. In addition, the client 1 includes input/output units such as a keyboard, a mouse, a display, and a printer, and a memory unit for storing computer programs and processing results.

The current position input unit 11 receives the position of the user's hand captured by a stereoscopic camera 31 attached to the head mounted display 3 using, for example, "Leap Motion" by Leap Motion Inc. The hand position may be represented by the coordinates of the tips of the ten fingers, or the coordinates of the center of the palm may be added.

If the time when the actual hand position is received is T, the real-time inference unit 12 inputs the current actual position and the value of dt as variables of the real-time AI prediction model obtained by machine learning, and predicts the hand position at T + dt. In this embodiment, the most recent RTT is used as this dt. The value of the RTT changes for each collaborative operation between the client 1 and the server 2, but in the following explanation, the RTT subscript will be omitted and it will simply be expressed as dt. The hand in the predicted position may also be called a virtual hand.

The real-time learning unit 13 uses the difference between the virtual hand position dt milliseconds from the present and the actual hand position captured by the stereoscopic camera 31 at dt milliseconds after T to update the parameters of the real-time AI prediction model through machine learning, and passes them to the real-time inference unit 12.

When the real-time learning unit 13 learns according to the difference between the virtual hand position and the actual hand position, the real-time monitoring unit 14 monitors whether the difference is appropriate, and if it exceeds a predetermined threshold, it excludes it from the learning target and reinitializes the real-time AI prediction model.

The server transceiver 15 transmits the position of the virtual hand dt milliseconds from the present to the server 2, and receives three-dimensional information generated based on the predicted position from the server 2. This three-dimensional information refers to the state of a virtual object (which does not exist in the real world, but exists only in the VR space and is displayed on the glasses-type display 32) that is being operated by the user's hand.

The rendering processing unit 16 combines the simulation results received from the server 2 with the virtual hand located at the position predicted by the real-time inference unit 12, and displays them on the glasses-type display 32 of the head-mounted display 3.

The server 2 includes a client transmission/reception unit 21 that transmits and receives information to and from the client 1, and a three-dimensional information generation unit 22 that performs simulations. It also includes input/output units such as a keyboard, mouse, display, and printer, and a storage unit that stores computer programs and processing results.

The head mounted display 3 includes a stereoscopic camera 31 that captures the movements of the user's hands, and a glasses-type display 32 that projects virtual objects in VR onto the user's eyes.
The configuration of this system has been explained above. Next, the operation of this system will be explained.

[3] Operation of the Present System [3-1] Overall Processing Flow Next, the overall processing of the present system will be described with reference to FIG.
When a user wears the head mounted display 3, a virtual object in the VR space is displayed on the glasses-type display 32. When the user operates this with hand gestures as if it were actually in front of the user's eyes, the virtual object appears to change in accordance with the movement of the user's hand.
The client 1 receives the current hand positions of the user captured by the stereoscopic camera 31 attached to the head mounted display 3 (step F1, hereinafter simply referred to as "F1").

The time information at this time is time stamp T. The current position acquired in F1 is sent to the real-time learning process (F2). The position of the virtual hand dt milliseconds before the current position is also sent to the real-time learning process in F11, which will be described later. In this way, data on the actual position and the predicted position with the same time stamp are accumulated in the memory unit (F3). Since the data accumulated in F3 usually has a gap between the predicted position and the actual measured value, an AI prediction model using this gap is learned in real time (F4). Real-time learning means that the parameters of the AI prediction model are updated each time data is input, and does not mean learning accumulated data after the fact.
The parameter information of the AI prediction model is updated by the learning in F4 (F5). The parameters updated in F5 are immediately output to the inference process (F6), and the hand position dt milliseconds later is predicted based on the current position information obtained from the camera 31 (F7). This becomes the virtual hand position with a timestamp of T + dt milliseconds added, and is transmitted to the server 2 (F8). The timestamp at the time of transmission is Si.

The server 2 executes a simulation process using the position of the virtual hand at T+dt milliseconds (F9), and the client 1 receives the execution result at timestamp Ri (F10). Since it took dt milliseconds to receive the result from the server 2, the current time is T+dt, and this predicted position is considered to be the same time as the current measured value input from the camera 31 at the current time (dt has passed since T). Therefore, this predicted position becomes an input value for the real-time learning process (F11). In this way, the inference process of predicting the position of the virtual hand and the learning process of updating the parameters of the AI prediction model by machine learning are executed in parallel.
The simulation results generated by the server 2 are rendered by the client 1 (F12) and displayed on the glasses-type terminal 32 (F13). Since software for displaying the simulation results in a VR space is commercially available, this software can also be used.

[3-2] Monitoring the suitability of learning data In step F3 of Fig. 5, data for real-time learning is accumulated, but if bad data that is not suitable for learning is included, it must be monitored and removed in real time. This will be explained with reference to Fig. 6. This real-time monitoring process is called from the processing of step F3 of Fig. 5.

The actual hand's current position (X(T+dt)) dt after time T and the predicted position (f(X(T), dt)) of the virtual hand having time stamp T+dt are output from step F3 (step F201). At this time, the counter variable #R is set to zero (F202).
If the gap between the two is within a predetermined normal range (Yes in F203), the counter variable #R is set to zero (F204). Then, the following items are monitored for RTTs executed within the most recent S seconds (F205). That is,
・Average moving speed of the object Vave
・Average object movement distance Xave
・Gap E (T+dt) between current position and predicted position
After monitoring the maximum, average and variance of RTT, the process transitions to F203 again.
The maximum, average, etc. RTT values are used to analyze and improve real-time monitoring algorithms.

If the gap is not within the normal range (No in F203), 1 is added to the counter variable #R (F206). If the gap between the current position and the predicted position is not improved even when #R exceeds a predetermined threshold L (Yes in F207), it is determined that the real-time inference has failed, and the real-time AI prediction model is reinitialized (F208). If the gap between the current position and the predicted position is improved (No in F207), the average moving speed and average moving distance are referenced (F209).
If at least one of them exceeds the physical limits of humans (Yes in F209), it is determined that a malfunction of the stereoscopic camera 31 has occurred, and the current position at time T+dt is regarded as an input error and ignored (F210). If it cannot be determined in F209 that there is an input error (No in F209), it is determined that an extreme network delay has occurred, and the object position after dt is sent from the client 1 to the server 2, and the RTT until the simulation result is returned is measured (F211).
When the state is F210 or F211, the state of affairs is observed while continuing real-time learning (F212), each item of F205 is monitored, and the state transitions to F203.

Real-time monitoring will now be described using a concrete example.
In the example of Figure 7, P0 represents the actual hand position, P1 represents the actual position dt after P0, VP1 represents the predicted position of the virtual hand dt after P0, and g1 represents the gap between P1 and VP1. Similarly, P2 represents the actual position dt after P1, VP2 represents the predicted position of the virtual hand dt after P1, and g2 represents the gap between P2 and VP2. The same applies to P3, P4, etc.
Here, if the gaps g1, g2, g3, and g4 are not within the normal range, #R in Fig. 6 will have a value of 4. If the threshold value L is set to 3, #R>L, and if no improvement in the gaps is obtained, the AI prediction model will be reinitialized.

In [3-1] and [3-2] above, we have explained the operation of this system. The first thing to note is that in order to eliminate the gap between the user's hand position and the virtual hand position as much as possible, real-time learning and real-time inference are performed simultaneously in parallel, and the parameters updated by learning are immediately used for inference. However, in addition to the gap in hand position, this system has an uncertain factor in that the RTT is not constant, so this point will be explained with reference to Figure 8.

[3-3] Changes in RTT In the example of Figure 8, RTT1 from when a packet is sent to server 2 at timestamp s1 (=0) to when the processing result of server 2 is received at r1 (=50) is 50. Similarly, RTT2 from when a packet is sent at s2 and received at r2 is 60. Here, a delay occurred in the RTT from 50 to 60. Furthermore, RTT3 became 70, and a delay occurred from the most recent 60 to 70. After that, RTT4, RTT5, ... continued to have a value of 70.
An RTT of 50 to 70 milliseconds is within the expected range, and since the RTT has stabilized since changing to 70, there is no problem. When predicting the position of the virtual hand, change dt to 50 -> 60 -> 70. However, if the RTT is frequently changing even within the range of 50 to 70 milliseconds, it is preferable to find out the cause. For that reason, it is a good idea to record the RTT for the most recent 2 seconds or so, as shown in Figure 1.

Note that the RTT values used in FIG. 8 are merely examples and do not limit the scope of the present invention. In general, the time required for rendering is 60 Hz, i.e., approximately 16.7 milliseconds or less, for a two-dimensional display, and 90 Hz to 120 Hz, i.e., approximately 11.1 milliseconds to 8.3 milliseconds or less, for a VR headset. In addition, it is desirable for the simulation step execution time on the server to be shorter than the time required for rendering, and the shorter it is, the higher the accuracy of the simulation will be. Furthermore, the transfer time between a client and a server in a normal communication environment is several milliseconds or less in a LAN environment, and several milliseconds to several tens of milliseconds in a WAN environment or WiFi environment. Furthermore, when using an overseas server via an undersea cable, the RTT value may take approximately 200 milliseconds, but this is within the range that can be fully handled by the present invention.

If the RTT changes, the rendering process on the client 1 side and the simulation process on the server 2 side cannot be synchronized as shown in Figure 3, but will be asynchronous as shown in Figure 9. In simulation sim1, 3D information is generated using the position of the virtual hand most recently received at time S1 + RTT0. In simulations sim2 and sim3, new position information for the next virtual hand is not transmitted, so server 2 predicts the position of the virtual hand after time s1 + RTT0 + n * steps (n = 1, 2, ...) and generates 3D information. In simulation sim4, 3D information is generated using the position of the virtual hand most recently received at time S2 + RTT1. In other words, simulations sim2, sim3, etc. generate interpolation data to prevent unnatural drawing when rendering 3D information and displaying it.

Although the embodiment of the present invention has been described above, the present invention is not limited to the above embodiment. Various modifications can be made without departing from the scope of the claims.
For example, the display is not limited to a glasses-type display, but may be a two-dimensional display.
Also, although the remote server executes only the generation of three-dimensional information, it is also possible to execute real-time inference and real-time learning on the remote server and have the client execute only the rendering.

The present invention is expected to be used widely in many fields, including not only scientific and engineering applications such as designing nanostructures using DNA, but also entertainment fields such as games.

1: Local client 11: Current position input unit 12: Real-time inference unit 13: Real-time learning unit 14: Real-time monitoring unit 15: Server transmission/reception unit 16: Rendering processing unit 2: Remote server 21: Client transmission/reception unit 22: Three-dimensional information generation unit 3: Head-mounted display 31: Stereo camera 32: Glasses-type display N: Communication network

Claims (2)

A network-type VR system including a local client that renders 3D information of VR on a head-mounted display or a 2D display worn by an operator, and a remote server that generates 3D information of a virtual object in VR,
The local client includes a means for acquiring a position of an operation object at a current timestamp T from the rendered three-dimensional information;
A means for inferring in real time a predicted position of the object dt milliseconds after a current timestamp T using an AI prediction model utilizing machine learning;
means for transmitting the predicted position based on the inference to the remote server and receiving 3D information of a virtual reality (VR) generated by the remote server using the predicted position;
means for displaying the generated VR three-dimensional information on the head-mounted display or two-dimensional display ;
A network-type VR system characterized by performing real-time learning using the difference between the predicted position of the operation object dt milliseconds after T and the actual position of the operation object when dt milliseconds have passed since T, and updating the parameters of the AI prediction model based on the results of this learning . The network-type VR system according to claim 1, further comprising a real-time monitoring means for reinitializing parameters of the AI prediction model when the difference between the predicted position dt milliseconds after T and the actual position of the operating object when dt milliseconds have passed since T continuously shows an abnormal value .

Images