JP6452808B2 - Low latency simulation apparatus and method using direction prediction, and computer program therefor - Google Patents

Description

  Each embodiment relates to a simulation apparatus and method, and a computer program therefor, and more particularly, a simulation that reduces latency by correcting a time difference between a signal measured by a sensor and a displayed image. Regarding technology.

  In recent years, as the processing capability of mobile devices such as smart phones has dramatically improved, cases of performing computer simulations on mobile devices are also increasing. The simulated object is converted into an image through a rendering process and used in an application such as a game, or forms part of a user interface associated with the operating system of a mobile device. Displayed on the mobile device.

  As one type of simulation on a mobile device, there is one that simulates a model (model), which is a virtual object, reflecting the movement of an object, and renders the simulation result as an image. The movement of the object that is the basis of the simulation can be measured through one or more sensors. By simulating the model based on the sensor data, the movement of the object can be replicated. You can move the model.

  However, since the measurement period of the sensor and the period in which the simulated model is rendered as an image do not match (asynchronous), there is a problem in that latency occurs. That is, there is a time delay in the rendered and displayed image compared to the sensor data of the mobile device. These time delays cause the user to feel unnatural when viewing the image or to feel uncomfortable when interacting with the image.

  According to one aspect of the present invention, there is provided a simulation apparatus and method that corrects a time delay between a signal measured by a sensor and a displayed image to reduce latency, and a computer program therefor. be able to.

  A simulation apparatus according to an embodiment includes a sensor unit configured to acquire movement information of an object using one or more sensors; and calculates direction information of the object using the movement information. A direction calculation unit configured as described above; and a simulation unit configured to simulate a physical object based on the direction information received from the direction calculation unit and render as an image. The direction calculation unit may include a time delay correction unit configured to correct the direction information using a time difference between an acquisition time point and a rendering time point of the motion information.

  The sensor unit may be configured to periodically acquire the motion information, and the simulation unit may be configured to periodically render the image. In this case, the time delay correction unit may be configured to calculate a time delay from each acquisition time point of the motion information to a subsequent rendering time point and correct the direction information using the time delay. The time delay correction unit is configured to correct the direction information by calculating an angular velocity of the object from the motion information and adding a product of the angular velocity and the time delay to the direction information. Also good.

  The direction calculation unit may be configured to correct the direction information of the object by further using a time difference between a rendering time point and a time point when the image is displayed on the display unit. The simulation unit may be configured to periodically render the image, and the display unit may be configured to periodically display the image. At this time, the time delay correction unit may be configured to correct the direction information by further using a time delay from each rendering viewpoint to a subsequent image display time point.

  The simulation method according to an embodiment includes a step in which a simulation apparatus obtains motion information of an object using one or more sensors; the simulation apparatus obtains direction information of the object using the motion information. A step of calculating; the simulation device correcting the direction information using a time difference between the acquisition time point and the rendering time point of the motion information; and the simulation device using the corrected direction information. And simulating a physical object and rendering it as an image.

  The step of obtaining the motion information and the step of rendering the image may be performed periodically. At this time, the step of correcting the direction information includes calculating a time delay from each acquisition time of the motion information to a subsequent rendering time; and correcting the direction information using the time delay. May include. Further, the step of correcting the direction information using the time delay includes calculating an angular velocity of the object from the motion information; and adding a product of the angular velocity and the time delay to the direction information. May be included.

  The simulation method may further include displaying the image on a display unit. At this time, the step of correcting the direction information may be performed by further using a time difference between a rendering time point and a time point when the image is displayed. In addition, the step of rendering the image and the step of displaying the image on a display unit are periodically performed, and the step of correcting the direction information further includes a time delay from each rendering viewpoint to a subsequent image display time point. It may be performed using.

  A computer program according to an embodiment may be recorded on a medium so as to perform the simulation method in combination with hardware.

  A simulation apparatus and method according to an aspect of the present invention provides a time delay between a sensor measurement period and an image rendering period and / or a time delay from the image rendering until the image is actually displayed on the screen. The simulation is performed using the direction information predicted in consideration of the above. As a result, the latency between the movement of the object and the displayed image can be reduced, and a natural simulation result can be obtained.

1 is a schematic block diagram of a simulation apparatus according to an embodiment. It is a conceptual diagram for demonstrating the time difference between the acquisition time of motion information, a rendering time, and an image display time in the simulation method which concerns on one Example.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  The simulation apparatus according to the embodiment is configured to simulate a model according to the motion of the target object and render the simulated model as an image. In this specification, a subject means a device or a person who intends to measure movement. For example, the subject is a wearable device that can be worn by the user, such as a mobile device such as a smart phone that the user has, or a head-mounted device (HMD). Good. Alternatively, the user's movement may be measured through one or more sensors that are directly targeted and attached to the user or at a remote location.

  In this specification, the model means data corresponding to a virtual physical object to be simulated by reflecting the motion of the object. For example, the model may be a mesh defined including one or more vertices. The model may be simulated according to the movement of the object by measuring the movement of the object and calculating a physical equation so that the interaction according to the movement is reflected in the position and speed of each part of the model. For example, if the user tilts the mobile device, the image displayed on the mobile device can be tilted as well, or if the user moves in front of the sensor, a character similar to the user is displayed. For example, it may be possible to make it move in the same way.

  FIG. 1 is a schematic block diagram of a simulation apparatus according to an embodiment.

  Referring to FIG. 1, the simulation apparatus may include a sensor unit 10, a direction calculation unit 20, and a simulation unit 30. In one embodiment, the simulation apparatus may further include a display unit 40. The simulation apparatus according to the embodiment may be completely hardware, or may be entirely software, or may be partially hardware and partially have a software aspect. For example, the simulation apparatus may refer to hardware having data processing capability and operation software for driving the hardware. In this specification, terms such as “unit”, “system”, and “apparatus” are intended to refer to a combination of hardware and software driven by the hardware. For example, the hardware may be a data processing device that includes a CPU or other processor. Further, software driven by hardware may refer to a process being executed, an object, an executable file, an execution thread, a program, and the like.

  Each unit constituting the simulation apparatus according to the embodiment is not necessarily intended to indicate a separate component physically separated. In FIG. 1, the sensor unit 10, the direction calculation unit 20, the simulation unit 30, and the display unit 40 are illustrated as separate blocks that are separated from each other. However, depending on the embodiment, the sensor unit 10 and the direction calculation unit 20 are illustrated. Some or all of the simulation unit 30 and the display unit 40 may be integrated in the same device. In addition, the direction calculation unit 20 and the simulation unit 30 are merely functionally divided devices based on operations performed in the computing device in which they are implemented, and are not necessarily separated from each other. Does not mean.

  For example, the simulation apparatus according to the embodiment is embodied in a form integrated in a mobile device such as a smartphone, measures the movement of the mobile device with the mobile device as a target, and determines a predetermined model according to the measured movement. And the simulated image may be displayed on the screen of the mobile device. However, this is exemplary, and in other embodiments, one or more of the sensor unit 10, the direction calculation unit 20, the simulation unit 30, and the display unit 40 are physically separated from other units. It may be embodied in a separate device. For example, the sensor unit 10, the direction calculation unit 20, the simulation unit 30, and the display unit 40 may be components that are communicably connected to each other under a distributed computing environment.

  The sensor unit 10 is a part for acquiring movement information of the target object. The motion information is information that is the basis for calculating the direction information of the target object, and the sensor unit 10 is configured to obtain one or more types of sensor data that can calculate the direction information. May include sensors. For example, in one embodiment, the sensor unit 10 includes an accelerometer 110, a magnetometer 120, and a gyroscope 130, but is not limited thereto. The form of the motion information obtained by the sensor unit 10 may be different depending on the type of sensor. For example, motion information may be obtained in the form of a gravity vector in the case of the accelerometer 110, a direction relative to magnetic north in the case of the magnetometer 120, and an angular velocity in the case of the gyroscope 130.

The direction calculation unit 20 is a part for receiving the motion information measured by the sensor unit 10 and calculating the direction information of the target object therefrom. In one embodiment, the direction calculation unit 20 may include a signal merging unit 210 and a time delay correction unit 220. The signal merging unit 210 merges motion information measured by each sensor of the sensor unit 10 and calculates direction information therefrom. In one embodiment, the signal merging unit 210 first calculates the first direction information by merging the motion information of the accelerometer 110 and the magnetometer 120, and uses the motion information of the gyroscope 130 to generate the second direction. Calculate information. In this case, since the first direction information depends on the magnetic direction and acceleration, the direction deviation (drift) is small, but it includes noise, and the second direction information has little noise but the direction deviation. Is included. The signal merging unit 210 may calculate the merged direction information by merging the first and second direction information as shown in Equation 1 below.
[Equation 1]
θ _fusion = αθ _g + (1−α) θ _am
In Equation 1, θ _am represents first direction information, θ _g represents second direction information, and θ _fusion represents merged direction information. In Equation 1, α and (1-α) are weight values for the first and second direction information, respectively, and the value of α is the merged direction information θ _fusion of the actual object. It may be determined appropriately to match the direction. By calculating the direction information as described above, it is possible to obtain the direction information in which both the direction shift and the noise are reduced.

  The time delay correction unit 220 corrects the direction information of the target object calculated by the signal merging unit 210 in consideration of the time difference between the acquisition time point of the motion information used to calculate the direction information and the rendering time point. It is a part of. In one embodiment, the time delay correction unit 220 may correct the direction information by further using a time difference between the rendering time point and the time point when the image is actually displayed on the display unit 40. For this purpose, the time delay correction unit 220 may acquire information about these operation cycles from the simulation unit 30 and / or the display unit 40, respectively. The specific operation of the time delay correction unit 220 will be described later with reference to FIG.

  The simulation unit 30 is a part for receiving the direction information of the target object from the direction calculation unit 20 and performing a simulation on the model based on the information. The direction information received by the simulation unit 30 indicates the direction information that is primarily calculated by the signal merging unit 210 and corrected by the time delay correction unit 220. The simulation unit 30 reflects the direction information of the target object and updates the position and speed of each part of the model for each time step so that the model is simulated according to the movement of the target object. Further, the simulation unit 30 renders the simulated model as an image to be displayed on the screen.

  The simulation may be performed so that the model replicates the movement of the object, or the model changes according to the movement of the object or interacts with the object. It is not limited. In addition, since the process of performing a computer simulation operation using a given condition with respect to a two-dimensional or three-dimensional model is well known to ordinary engineers in the technical field to which the present invention belongs, In order to clarify, detailed description is omitted in this specification.

  The display unit 40 is a part for displaying the image rendered by the simulation unit 30 so that the user can see it. For example, the display unit 40 may include a liquid crystal display (Liquid Crystal Display), an active matrix organic EL (Active-Matrix Organic Light-Emitting Diode; AMOLED), or the like, but is not limited thereto. When the simulation apparatus according to the present embodiment is integrated and implemented in a mobile device such as a smartphone, the display unit 40 may be implemented in the form of a touch screen that is a display unit and an input unit of the mobile device.

  FIG. 2 is a conceptual diagram for explaining a time difference between a motion information acquisition time point, a rendering time point, and an image display time point in a simulation method according to an embodiment.

Referring to FIGS. 1 and 2, measurement of motion information by the sensor unit 10 is periodically performed, and each measurement time point returns every first period (T ₁ ). In addition, rendering by the simulation unit 30 is also performed periodically, and each rendering time point returns every second period (T ₂ ). The first period (T ₁ ) and the second period (T ₂ ) may be the same or different from each other, but each measurement time point and each rendering time point do not coincide with each other due to a difference in the period or the start time point (asynchronous). ). As a result, a time delay (D _t1 ) occurs between each measurement time point and each adjacent rendering time point. When the first period (T ₁ ) and the second period (T ₂ ) are different from each other, the time delay (D _t1 ) is different for each time point.

The time delay correction unit 220 of the direction calculation unit 20 corrects the direction information of the object using the time delay (D _t1 ) from the acquisition time of the motion information used for calculating the direction information to the subsequent rendering time. To do. That is, the direction information corrected by the time delay correction unit 220 corresponds to the prediction of the direction information at the time of rendering. The time delay correction unit 220 transmits the predicted direction information to the simulation unit 30 so that simulation and rendering are performed based on the information. Specifically, taking the time points 501 and 502 in FIG. 2 as an example, the time delay correction unit 220 extends from the measurement time point 501 of the motion information used to calculate the current direction information to the next adjacent rendering time 502. The time delay 503 is calculated. Next, the time delay correction unit 220 calculates the angular velocity of the target object using the motion information obtained at the measurement time point 501, and multiplies the calculated angular velocity by the time delay 503 to add to the direction information.

In other words, the direction information before being corrected by the time delay correction unit 220 is θ _current , the angular velocity of the object according to the recently measured motion information is ω, and the time between the latest motion information measurement time and the subsequent adjacent rendering time When the time delay is D _t1 , the direction information θ _new corrected by the time delay correction unit 220 may be calculated as in the following equation 2.
[Equation 2]
θ _new = θ _current + ω × D _t1
In one embodiment, the time delay correction unit 220 may correct the direction information by further considering the time delay between the rendering time point and the time point when the image is actually displayed on the display unit 40. That is, the direction information corrected in this way corresponds to information obtained by predicting in advance the direction of the object at the time when the image is actually displayed.

Referring to FIG. 2, the image display by the display unit 40 is periodically performed, and the display time point of each image returns every third period (T ₃ ). At this time, a time delay (D _t2 ) occurs between each rendering time point and each image display time point on the display unit 40, and the time delay correction unit 220 causes the time delay between the sensor measurement time point and the rendering time point described above. In addition to (D _t1 ), the direction information may be corrected by further considering the time delay (D _t2 ). In this case, the direction information θnew corrected by the time delay correction unit 220 may be calculated as the following Expression 3.
[Equation 3]
θ _new = θ _current + ω × (D _t1 + D _t2 )
That is, the simulation apparatus and method according to the embodiment described above predicts the direction information of the object at the time of rendering, or considers the time delay between the sensor measurement period and the image rendering period. Further, in consideration of the time delay between the rendering cycle and the image display cycle, the direction information of the object at the time of image display is predicted. By performing simulation and rendering using the predicted direction information in this way, the latency generated due to the measurement period of the sensor not matching the simulation or display period is reduced, and according to the movement of the object. There is an advantage that a natural simulation result can be obtained.

  The operation by the simulation apparatus according to the embodiment described above may be at least partially embodied as a computer program and recorded on a computer-readable recording medium. The computer-readable recording medium includes any kind of recording device in which data that can be read by the computer is stored. Examples of computer-readable recording media include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc., and carrier wave (for example, transmission through the Internet). Also embodied in the form of. Further, the computer-readable recording medium may be distributed to computer systems connected via a network, and a computer-readable code may be stored and executed in a distributed manner. The functional program, code, and code segment for embodying the present embodiment can be easily understood by ordinary engineers in the technical field to which the present embodiment belongs.

  The present invention as seen above has been described with reference to the embodiments shown in the drawings. However, this is merely an example, and any person with ordinary knowledge in the field will understand this. It will be appreciated that various modifications and variations of the embodiments are possible. However, such variations should be considered to be within the technical protection scope of the present invention. Therefore, it can be said that the true technical protection scope of the present invention must be determined by the technical idea of the appended claims.

  Each embodiment relates to a simulation apparatus and method, and a computer program therefor.

Claims (7)

A sensor unit that obtains movement information of an object using one or more sensors;
A direction calculation unit that calculates direction information of the object using the motion information; and a simulation unit that simulates a physical object based on the direction information received from the direction calculation unit and renders the image as an image,
The direction calculation section, viewed contains a time delay correcting unit for correcting the direction information by utilizing the time difference between the time stamp and the rendering time of the motion information,
The sensor unit periodically acquires the movement information,
The simulation unit periodically renders the image,
The time delay correction unit calculates a time delay from each acquisition time point of the motion information to a subsequent rendering time point, corrects the direction information using the time delay,
The simulation apparatus , wherein the time delay correction unit corrects the direction information by calculating an angular velocity of the object from the motion information and adding a product of the angular velocity and the time delay to the direction information . The direction calculation unit further corrects the direction information of the object by further using a time difference between a rendering time point and a time point when the image is displayed on the display unit.
The simulation apparatus according to claim 1. The display unit periodically displays the image,
The time delay correction unit corrects the direction information by further using a time delay from each rendering time point to a subsequent image display time point;
The simulation apparatus according to claim 2 . A simulation device acquiring movement information of an object using one or more sensors;
The simulation device calculating direction information of the object using the motion information;
The simulation device corrects the direction information using a time difference between the acquisition time and the rendering time of the motion information; and the simulation device uses the corrected direction information to make a physical object. to simulate the, only including the step of rendering as an image,
Obtaining the motion information and rendering the image are performed periodically;
The step of correcting the direction information includes:
Calculating a time delay from each acquisition time of the motion information to a subsequent rendering time; and
Modifying the direction information using the time delay
Including
The step of correcting the direction information using the time delay includes:
Calculating an angular velocity of the object from the movement information; and
Adding the product of the angular velocity and the time delay to the direction information
including,
Simulation method. Further comprising displaying the image on a display unit;
The step of modifying the direction information is further performed using a time difference between a rendering time point and a time point when the image is displayed.
The simulation method according to claim 4 . The step of displaying the image on the display unit is periodically performed.
The step of modifying the direction information is further performed using a time delay from each rendering time point to a subsequent image display time point.
The simulation method according to claim 5 . Combined with hardware,
Obtaining movement information of an object using one or more sensors;
Calculating the direction information of the object using the movement information;
Correcting the direction information using a time difference between the acquisition time and the rendering time of the motion information; and simulating a physical object using the corrected direction information and rendering as an image. Stored in the medium to carry out ,
Obtaining the motion information and rendering the image are performed periodically;
The step of correcting the direction information includes:
Calculating a time delay from each acquisition time of the motion information to a subsequent rendering time; and
Modifying the direction information using the time delay
Including
The step of correcting the direction information using the time delay includes:
Calculating an angular velocity of the object from the movement information; and
Adding the product of the angular velocity and the time delay to the direction information
Including computer programs.

Images