JP7553408B2 - Drawing device and program - Google Patents

Description

The present invention relates to a drawing device and a program for drawing avatars that can be used for remote communication, etc.

In a system that estimates a user's movements and applies them to an avatar to achieve remote communication over a network, if transmission delays could be reduced, smoother communication would be possible. Examples of related art that address this issue are disclosed in Patent Documents 1 and 2, which disclose the following techniques.

Patent Document 1 discloses a method for estimating the future movements of two users in virtual reality (VR) intended for gaming applications, and controlling the fast-forwarding and delay of the video to synchronize the timing at which the avatars' gazes meet when they look at each other. Patent Document 2 discloses a method for suppressing transmission delays in applications in which users participate in streaming videos as avatars, by transmitting reference posture identification data that can identify information about the posture of the viewing user with a small amount of data, rather than posture feature data that requires a large amount of data.

JP 2020-009295 A Patent No. 6731532

However, with the conventional technologies described above, there are issues with smoothly realizing remote communication, etc., given the existence of delays due to transmission, etc.

In Patent Document 1, the gazes of the avatars meet, but the user whose video is fast-forwarded does not yet see the other user's avatar, and conversely, the user whose video is delayed must wait for their movements to be reflected, which results in the problem of being unable to communicate smoothly. In addition, the amount of delay in transmission and reception fluctuates depending on the network conditions, making it difficult to predict the timing for the gazes to meet.

Patent document 2 attempts to suppress transmission delays by reducing the amount of transmission, but there is an unavoidable delay in the time it takes to recognize posture and the time it takes to transmit, and there is a problem in that it is not possible to avoid or suppress the effects of the delays themselves.

In view of the problems with the conventional technology described above, the present invention aims to provide a drawing device and program that can draw avatars while effectively suppressing the effects of delays caused by transmission, etc.

To achieve the above object, the present invention is a drawing device having a first feature including: an estimation unit that receives, in chronological order, control parameters for avatar drawing from a user for whom an avatar is to be drawn as recognition information that recognizes the state of the user, and estimates the amount of delay between the time of the most recent recognition information among the received recognition information and the current time; a prediction unit that predicts recognition information corresponding to the current time by locating the most recent recognition information in the history of the received recognition information by the amount of delay on the future side; and a drawing unit that uses the predicted recognition information to draw the user's avatar as if it were at the current time.

The second feature of the present invention is that the prediction unit predicts the recognition information corresponding to the current time using a prediction model selected from a plurality of types of prediction models prepared in advance, and applies each of the plurality of types of prediction models to the history of the received recognition information to evaluate the prediction accuracy of each prediction model, and selects the prediction model with the best prediction accuracy. The present invention is also characterized by being a program that causes a computer to function as the drawing device.

According to the first feature, the amount of delay is estimated, and the recognition information at the current time is predicted from the history of received recognition information as being located in the future by the width of this amount of delay, and an avatar is drawn based on this predicted recognition information, so that the effect of delay can be effectively reduced by the prediction when drawing an avatar. According to the second feature, prediction accuracy is evaluated for multiple types of prediction models, and the one with the best prediction accuracy is used, making it possible to make highly accurate predictions.

FIG. 1 is a configuration diagram of a communication system according to an embodiment. 1 is a flowchart of an operation of a communication system according to an embodiment. 2 is a functional block diagram of a recognition device and a drawing device in a communication system according to an embodiment. 10 is a flowchart of an operation of a recognition device according to an embodiment. 10 is a flowchart of an operation of a drawing device according to an embodiment. FIG. 13 is a diagram illustrating a schematic example of estimating a delay amount. 11 is a diagram showing a schematic example in which a plurality of prediction models are prepared in advance in a prediction unit according to a delay value to be predicted and a model type to be used for prediction. FIG. FIG. 13 is a diagram showing a schematic example of evaluating prediction accuracy by actually applying prediction to each prediction model in order to select an optimal one from among a plurality of types of prediction models in a prediction unit. FIG. 13 is a schematic diagram showing a case where the distribution of delay values is a normal distribution, as an example of setting values that can actually occur from the distribution of delay values in order to effectively prepare a small number of multiple delay value candidates in a prediction unit. FIG. 8 is a diagram showing a modification of FIG. 7 . This is an example in which a graph is simultaneously plotted for three candidate delay values, a time series of actual values, and a forecast time series obtained by applying forecasting to each candidate delay value. FIG. 1 is a diagram illustrating a hardware configuration of a typical computer.

FIG. 1 is a configuration diagram of a communication system 100 according to one embodiment. The communication system 100 includes a recognition device 10 as a first terminal used by a first user U1, and a drawing device 20 as a second terminal used by a second user U2. The recognition device 10 and the drawing device 20 can communicate with each other via a network NW configured as the Internet and/or a LAN (local area network), etc. Both the recognition device 10 and the drawing device 20 can be configured as mobile terminals such as smartphone terminals, tablet terminals, and laptop computer terminals, or as computer devices of any other configuration.

Figure 2 is a flowchart of the operation of the communication system 100 according to one embodiment. As illustrated in Figure 1, the steps in Figure 2 as the overall operation of the communication system 100 are as follows:

The recognition device 10 has an imaging unit 11 as a camera that provides an imaging function, and in step S1, it takes an image of the first user U1, recognizes the facial expression, etc., and transmits this recognition information to the drawing device 20 in real time via the network NW. The drawing device 20 has a presentation unit 24 as a display that provides at least a display function, and in step S2, it receives the recognition information transmitted from the recognition device 10, predicts and draws the avatar A1 of the first user U1 for the current time based on the history of the recognition information up to the current time in real time (the history that has been transmitted from the recognition device 10 and has been received) and the performance of the prediction application evaluated from this history, and displays this avatar A1 on the presentation unit 24, thereby providing an avatar display for the second user U2.

In step S3, the current time i is updated to the next time i+1, and the process returns to step S1, where similar steps are repeated for each real-time instant. That is, for each real-time instant i=1, 2, ..., the steps in FIG. 2 are repeated to obtain the recognition information p(i) of the first user U1 in step S1, and to draw and display the avatar A1(i) of the first user in step S2. In this way, the second user U2 can grasp in real time the appearance (whole body pose and/or facial expression, etc.) of the remotely located first user U1 by looking at the avatar A1(i) that moves in real time as it is drawn in real time, and can thus remotely communicate with the first user U1 in real time via the avatar A1(i).

Here, the recognition information p(i) and the like transmitted from the recognition device 10 via the network NW are received by the drawing device 20 with a delay due to communication. (Furthermore, each process of the recognition device 10 and the drawing device 20 requires a certain time to complete even if it is a short time, and a processing delay occurs.) The flow of Fig. 2 can be executed by synchronizing the processing timing in the recognition device 10 and the drawing device 20 at a common time i = 1, 2, ..., but when drawing the avatar A1(i) in step S2 at the current time i, the drawing device 20 is not able to receive the recognition information p(i) at the current time i due to the presence of a delay due to communication, etc., and is in a state where it has only completed reception of the recognition information p(1), p(2), ..., p(iK-1), p(iK) at K (K>0) past times i = 1, 2, ..., iK-1, iK. (Here, time i=1 is the initial time when the operation in FIG. 2 is started in communication system 100.) Therefore, in drawing device 20, by predicting recognition information p(i) at current time i (recognition information p(i) such as facial expression of first user U1 at current time i) from the reception history of recognition information up to K past times, and drawing avatar A1(i) according to this predicted recognition information p(i), it is possible to realize smooth real-time communication using avatar drawing under the assumption that there is a delay. Details of this will be described later. (Note that the recognition information p(i) predicted by the drawing device 20 is usually different from the recognition information p(i) of the actual measurement value actually recognized by the recognition device 10, which is received by the drawing device 20 at a time after the current time i. Since it is clear from the context of the explanation whether p(i) etc. are actual measurements or predicted values, they will be expressed as the same "p(i)" etc. for the sake of simplicity of symbolic notation. Also, when a distinction is necessary, they will be distinguished by notations such as p(i) _{[actual measurement]} and p(i) _[prediction] .)

In FIG. 2, step S2 follows step S1 for the sake of convenience in describing the flowchart, but step S2 does not have to be executed until step S1 is completed. That is, at each real-time time i=1, 2, ..., the recognition device 10 executes step S1, and in parallel with this, the rendering device 20 executes step S20. That is, steps S1 and S2 may be executed in parallel at each real-time time i=1, 2, ..., whose timing is managed by step S3.

In the configurations of Figs. 1 and 2, the first user U1 is the one whose avatar A1 is drawn, and the second user U2 is the one who views the drawn avatar A1. However, by adding a configuration in which the roles are swapped, it is possible to perform two-way remote communication between the first user U1 and the second user U2 via their avatars. (That is, the recognition device 10 as the first terminal used by the first user U1 is additionally provided with a function equivalent to the drawing device 20 for drawing the avatar A2 (not shown in Fig. 1) of the second user U2, and the drawing device 20 as the second terminal used by the second user U2 is additionally provided with a function equivalent to the recognition device 10 for recognizing the facial expressions and other aspects of the second user U2, thereby enabling two-way communication.) Similarly, the number of users participating in remote communication is not limited to the first user U1 and the second user U2, and it is possible to perform remote communication by simultaneously drawing the avatars of any number of users.

However, for the sake of clarity and simplicity, the following explanation will be limited to the case where the number of people and roles involved in remote communication using avatars are as shown in Figures 1 and 2 (two people and one-way), but it is possible to carry out remote communication using avatars in both directions with any number of people in the same way as described above.

Figure 3 is a functional block diagram of the recognition device 10 and drawing device 20 in the communication system 100 according to one embodiment. The recognition device 10 includes an imaging unit 11, a recognition unit 12, and an optional recording unit 13. The drawing device 20 includes an estimation unit 21, a prediction unit 22, a drawing unit 23, a presentation unit 24, and a memory unit 25.

Figure 4 is a flowchart of the operation of the recognition device 10 according to one embodiment, and Figure 5 is a flowchart of the drawing device 20 according to one embodiment. Steps S11 to S13 in Figure 4 correspond to step S1 in Figure 2, and steps S21 to S24 in Figure 5 correspond to step S2 in Figure 2. Similarly, steps S14 and S25 in Figures 4 and 5 correspond to step S3 in Figure 2, and timing management of each common real-time time i=1, 2, ... is performed in the recognition device 10 and the drawing device 20, respectively.

Below, the details of each functional unit of the recognition device 10 and the drawing device 20 in FIG. 3 will be described while explaining each step in FIG. 4 and FIG. 5. Note that, as a preprocessing before the recognition device 10 starts the flow in FIG. 4 and the drawing device 20 starts the flow in FIG. 5, the recognition device 10 and the drawing device 20 perform time synchronization using an existing method such as NTP (Network Time Protocol). The recognition device 10 and the drawing device 20 perform processing at each real-time point using the synchronized clock (a timekeeping function provided by a computer operating system, etc.). As in the explanation of FIG. 2, in the explanation of FIG. 4 and FIG. 5, each common real-time point is i=1, 2, ..., and the time at the beginning of the flow (initial time) is i=1. Note that in real time, each time i=1, 2, ... specified by an integer is arranged at intervals of, for example, 1/30 seconds as a discrete processing timing such as 30 fps (frames per second).

<Step S11...imaging unit 11 (and audio recording unit 13)>
4 is started (initial time i=1), or at each real-time time i thereafter (i≧2), in step S11, the imaging unit 11 captures an image of the first user U1 to obtain a frame image F(i) linked to the current time i, and outputs this image F(i) to the recognition unit 12 before proceeding to step S12. The imaging unit 11 is configured as a camera in terms of hardware, and is capable of capturing an image of the first user U1.

As an additional embodiment, in step S1, the recording unit 13 may further record the speech of the first user U1, and obtain the speech information v(i) at the time i (speech information v(i) in the real time period between the time i, which is a discrete processing timing, and the subsequent time i+1 (or the previous time i-1)). The recording unit 13 is configured as a microphone in terms of hardware, and by recording in the environment where the first user U1 is present, speech information including the speech of the first user U1 can be obtained. When obtaining the speech information v(i), an existing method may be applied for removing environmental sounds other than the speech of the first user U1 as noise from the recorded data.

<Step S12...Recognition Unit 12>
In step S12, the recognition unit 12 recognizes the first user U1 captured in the image F(i) obtained in step S11, and calculates recognition information p(i) as an avatar control parameter required to render the first user U1 as an avatar A1, and then the process proceeds to step S13. As is clear from the notation as a variable, this recognition information p(i) is acquired as being linked to the current time i (corresponding to the capture time of the image F(i)), which is the processing timing.

The recognition information p(i) calculated by the recognition unit 12 can be of any type according to a predetermined model used for the avatar drawn by the drawing device 20, but can generally be calculated as follows in the form of an N-dimensional vector listing the values of multiple N parameters at current time i: _{p k} (i) is the k-th (1≦k≦N) parameter.
p(i)=(p ₁ (i),p ₂ (i),…,p _N (i))

For example, when drawing a facial expression of an avatar, the recognition information p(i) may be calculated as a parameter of blendshape, which is an existing method. In blendshape, the facial expression of a user to be drawn as an avatar is expressed by weighting a plurality of mesh models (each mesh model expresses some facial expression state), so the weight of the kth (1≦k≦N) mesh model may be calculated as the vector element p _k (i). For example, when drawing a body pose of an avatar, the recognition information p(i) may be calculated as a parameter expressing the coordinates of each joint of the user using any predetermined human body skeleton joint model. In this case, the coordinates (two-dimensional image coordinates (u,v) or three-dimensional coordinates (x,y,z)) of the kth (1≦k≦N) joint may be calculated as the vector element p _k (i). When drawing both a facial expression and a whole-body pose of an avatar, a combination of the weights of the blendshapes and the joint coordinates may be used as the recognition information p(i).

Regarding the method for specifically calculating the recognition information p(i) as the avatar control parameter from the image F(i) in the recognition unit 12, an existing method according to the type of avatar control parameter may be used. For example, facial feature points may be extracted from the image F(i) using a pre-trained deep learning network or the like, and the distribution of these feature points may be converted into blend shape weights, or a method may be used in which the blend shape weights are output immediately from the image F(i) (without going through feature point extraction) using a deep learning network or the like. The joint coordinates may also be output from the image F(i) using a pre-trained deep learning network or the like.

<Step S13...Transmission Processing>
In step S13, the recognition information p(i) obtained in step S12 is transmitted from the recognition device 10 to the drawing device 20, and the process proceeds to step S14. If recording is performed by the recording unit 13 in step S11, this voice information v(i) is also transmitted from the recognition device 10 to the drawing device 20 in step S13. The transmitted recognition information p(i) is stored in the storage unit 25 of the drawing device 20 and is referenced and used. The transmitted voice information v(i) is used as voice data that is played back in synchronization with the avatar movement when the avatar is drawn in real time by the drawing device 20.

When transmitting both the recognition information p(i) and the voice information v(i), they may be transmitted via the communication interface as soon as they are acquired by the recognition device 10. For example, if the voice information v(i) is acquired first and the recognition information p(i) is acquired later within a real-time period corresponding to the current time i (the period between the current time i, which is a discrete processing timing, and the next time i+1), the voice information v(i) may be immediately packetized as a voice packet as soon as it is acquired and transmitted, and the recognition information p(i) obtained later may also be packetized in a predetermined format and transmitted. The recognition information p(i) and the voice information v(i) may also be transmitted from the recognition device 10 to the drawing device 20 after being packetized via the communication interface with a timestamp t (a timestamp t that is recorded at intervals finer than time i=1, 2, ... (for example, t=0.01, 0.02, 0.03, ... as a 100-fold subdivision interval)) in addition to the time i, which is the processing timing.

<Step S14...Time update>
In step S14, the current time i is updated to the next time i+1, and the process returns to step S11, whereby the above steps are similarly repeated for the next real-time time i+1.

Above, the steps in FIG. 4 have been described as the processing contents of the recognition device 10 according to one embodiment. Next, the steps in FIG. 5 will be described as the processing contents of the drawing device 20 according to one embodiment.

<Step S21...Reception processing>
When the flow of Fig. 5 is started (initial time i = 1), or at each real-time time i (i ≧ 2) thereafter, in step S21, the drawing device 20 receives the recognition information p(j) (and the voice information v(j)) transmitted from the recognition device 10 in step S13 of Fig. 4, and then proceeds to step S22. As described with reference to Fig. 2 regarding the overall operation of the communication system 100, the latest recognition information p(j) that can be received at the drawing device 20 at the current time i is K pieces of past recognition information p(iK) (K > 0) as viewed from the current time i due to the presence of delays, etc. Depending on the occurrence of delays, etc., one or more pieces of recognition information p(j) at time j may be received at the current time i, or none may be received at all.

As in the case of transmission by the recognition device 10, when reception is performed by the drawing device 20, the recognition information p(j) (and the voice information v(j)) received at the communication interface may be linked to a timestamp t (which is generally more precise than the discrete processing timings i, j) that is the reception time of the packet.

The received recognition information p(j) is accumulated and stored in the storage unit 25, which is constituted by a storage medium as hardware. That is, the storage unit 25 stores the recognition information p(1), p(2), ..., p(j) that has been continuously received up to the current time i as history, and provides this information as a reference for processing by the estimation unit 21 and the prediction unit 22, which will be described later. Note that some of the recognition information p(1), p(2), ..., p(j) may be partially lost due to packet loss during transmission, etc. (or may be delayed more than other packets and not be received at the current time i, and therefore may not exist).

<Step S22...Estimation Unit 21>
In step S22, the estimation unit 21 refers to the storage unit 25, estimates the amount of delay, and outputs this amount of delay to the prediction unit 22, and then the process proceeds to step S23.

This delay amount is a delay in the recognition information, etc., actually obtained at the current time i, which is taken into consideration in order to draw and display avatar A1(i) at the current time i in the drawing unit 23 and the presentation unit 24 described later, and can be estimated in each embodiment. In the first embodiment, if the latest recognition information (recognition information in the most recent past) stored in the storage unit 25 is recognition information p(i-K), the delay amount may be estimated as K (K times the interval between discrete processing timings specified as time i=1, 2, ...).

In the second embodiment, the timestamps of the packets sent and received may be used to determine the amount of delay as the difference between the sending timestamp t _[sent] and the receiving timestamp t _[received] in the most recently received recognition information p(iK) (corresponding to the transmission time of the data packet of the recognition information).

In the third embodiment, as shown in FIG. 6, in addition to the transmission time t2, the recognition time t1 of the recognition unit 12, the prediction time t3 of the prediction unit 22, and the drawing presentation time t4 of the drawing unit 23 and the presentation unit 24 may also be estimated and combined with the transmission time t2 to obtain the delay amount (=t1+t2+t3+t4) in real time (real time distinct from discrete processing timing). For the transmission time t2, the value of the first or second embodiment may be used, and for the processing times (time required to complete processing) t1, t3, and t4 of each functional unit, fixed values obtained in advance through a preliminary experiment may be used, or actual values (average values, etc.) within a specified recent period may be recorded and used. For the transmission time t2 according to the first or second embodiment, only the most recent value may be used, or actual values (average values, etc.) within a specified recent period may be recorded and used.

In the fourth embodiment, which is a modification of the third embodiment, on the premise that recording is performed by the recording unit 13 in the recognition device 10, in addition to the recognition information p(j), audio information v(j) is also transmitted and received to draw an avatar with synchronized audio. If the audio information v(j) can be received earlier by the time required for the recognition process to obtain the recognition information p(j), the amount of delay of the recognition information can be calculated by subtracting the amount of delay of the audio information from the amount of delay of the recognition information. Since the recognition information and the audio information are synchronized and the amount of delay is reduced, the effect of improving the accuracy of the prediction unit 22 is obtained.

In the example of FIG. 6, an example of the fourth embodiment is shown in the lower part enclosed with parentheses, and while the delay amount in the third embodiment is "t1+t2+t3+t4", in the fourth embodiment, the delay amount can be estimated as "t1+t2+t3+t4-t5" obtained by subtracting the delay t5 of the voice information. The delay t5 of the voice information can be calculated as the time difference t5 between the time stamps of the transmission and reception in the packet of the voice information v(j), just like the difference between the transmission time stamp t _{[transmission]} and the reception time stamp t[ _reception] of the recognition information in the second embodiment. The time difference t5 may be the latest single value, or the actual value (average value, etc.) within a specified past period may be recorded and used.

In the fourth embodiment, the definition of real time when the drawing device 20 draws an avatar is shifted backward from the accurate real time managed by NTP or the like by the transmission time t5 of the voice information (i.e., the time when the voice information arrives is considered to be real time for the purpose of drawing the avatar), and the predicted time width is reduced by t5, thereby making it possible to reduce the amount of delay while ensuring the comfort of communication using the avatar. (The prediction by the prediction unit 22 described later is generally expected to have higher prediction accuracy as the delay (= the time width to which the prediction is applied) is smaller, so the fourth embodiment can achieve this effect.) Similarly, in addition to the transmission time t5 of the voice information, the time t6 (actual value or pre-investigation value) required for a predetermined process (processing to decode compressed voice information into a reproducible voice waveform by pulse code modulation or the like) until the voice information is actually played back on the presentation unit 24 may be taken into consideration, and the delay may be estimated as "t1+t2+t3+t4-t5-t6".

<Step S23...Prediction Unit 22>
In step S23, the prediction unit 22 predicts the current recognition information p(i) from the transition of past recognition information (p(1), p(2), ..., p(iK)), outputs the predicted recognition information p(i) to the drawing unit 23, and then proceeds to step S24.

Here, to explain the processing of the prediction unit 22, the terms are defined as follows. The most recent (latest) past time i-K at which a change in the recognition information is obtained at the current time i is defined as the "current time t" that serves as the basis for prediction, and the current time t+n (≒i) that is the subject of prediction is defined as the "future time t+n" based on the "current time t".

Here, the prediction unit 22 uses the delay amount output by the estimation unit 21 as the value of n (n>0) for determining future time t+n from the current time t. In the first embodiment of the delay amount, the "current time t" matches the "past time i-K at which the most recent recognition information p(i-K) exists," and therefore n=K. In the second to fourth embodiments of the delay amount, the value of n is expected to be close to the value of K (the time width of K intervals between discrete processing timings i=1, 2, ...), but does not necessarily match K.

As shown in a table format in Fig. 7, the prediction unit 22 constructs in advance a plurality of prediction models M ab (a=1,2,..., b=1,2,...) designated by distinguishing between candidates a for delay value n (here, a=1,2,... as delay value IDs, not actual delay values) and model types b used for prediction (b=1,2,... as model IDs), selects an optimal model from among the plurality of prediction _{models M ab ,} and predicts prediction information for future time t+n from the current time t by using the selected optimal model M _ab . (Note that, as a single case, an embodiment in which the model type b used for prediction is one, rather than two or more, is also possible.)

In the example of Fig. 7, three delay value candidates a (candidate a for determining delay value T _a ) are preset according to the range of delay values: a=1 (delay value T ₁ =0.5 seconds), a=2 (delay value T ₂ =1 second), and a=3 (delay value T ₃ =2 seconds), and three model types b used for prediction are preset: b=1 (linear interpolation), b=2 (CNN (convolutional neural network)), and b=3 (RNN (recurrent neural network)), resulting in a total of 3 x 3 = 9 prediction models M _ab being prepared in advance. When using CNN or RNN, the network parameters are learned in advance using training data.

In the prediction unit 22, in regard to which of the multiple prediction models M _ab to use, the candidate a for the delay value may be determined as the one that corresponds to the delay amount n obtained from the estimation unit 21 (if there is no match, the closest one). For example, in the case of the prediction model M _ab in Fig. 7, if the delay amount n is n = 1.2 seconds, a = 2 (delay is 1 second) may be determined.

In addition, in the prediction unit 22, in regard to which of the multiple prediction models _Mab to use, for model type b, the performance scores score(b ₎ are evaluated by comparing the respective values _p (tL ₎ , p(t-L+1 ₎ , ..., p(t-1 ₎ , p(t ₎ predicted for the future recognition information T _a ahead at past times tLT a, t-L+1-T a, t-L+2-T a, ..., tT a for L+1 fixed periods T a to L+T a for each model type b, with the actual values, and the one with the best score score(b) is adopted. For example, if the predicted delay width in prediction model M _ab is T _a (in the example of Figure 7, a = 1, 2, 3, T ₁ = 0.5 seconds, T ₂ = 1 second, T ₃ = 2 seconds), the prediction error error(b) of model b at the delay width can be calculated as follows, for example, by the sum of squares of the Euclidean distance, and then the score(b) can be calculated using a predetermined reduction function f as score(b) = f(error(b)) so that the smaller the prediction error, the higher the score.

The range k= _Ta ,...,L+ _Ta for calculating the above sum is the range closest to the current time and the range in which the actual measurement value (among which the actual value that can be used for the evaluation of the prediction performance) for the delay width _Ta can actually be received (as described above, since p(t) is the closest to the current time among the actual measurement values that can be received, the value predicted for the future time _tTa + _Ta at the past time _tTa can be evaluated using this actual measurement value), and the number of k that falls within the range and takes the sum is L+1 as a normalization term. Note that in the above sum, the delay width _Ta is an integer similar to the discrete processing timing i=1,2,... given by an integer in FIG. 5 and the like. The delay width _Ta of each model M _ab may be defined as satisfying this constraint, or if this constraint is not satisfied, the value of how many discrete processing timings i=1,2,... the delay width _Ta in real time corresponds to may be a value obtained by rounding down the integer using the floor function floor( _Ta ).

In addition, when calculating the above sum, each predicted value “p(t-k+T _a ) _{[prediction by model b]} ” can be calculated by using the value predicted for the future, T _a ahead, at time tk (when recognition information up to time tk has already been received), which is the time when the latest data available for prediction is obtained.

In Fig. 8, as a schematic example for evaluating prediction accuracy, when predicting recognition information (only one element of a multidimensional vector is shown on the vertical axis, and time is shown on the horizontal axis) using a certain model b, a graph of actual values (part before the current time t) is shown with a solid line, and a graph of predicted values (part in the future than the current time t) is shown with a dashed line. In Fig. 8, n1, n2, and n3 are shown as three examples of delay width T _a .

Here, it is desirable from the viewpoint of prediction accuracy to comprehensively set the delay width T _a that determines the model M _ab at small intervals over a large number of candidates. On the other hand, however, if there are many candidates a, the number of evaluations of the performance score score(b) will also increase, and it is expected that the amount of calculation in the prediction unit 22 will increase.

As shown in the schematic diagram of the normal distribution in Figure 9, the delay value candidate a can be determined in advance by experiments, etc., and the delay value (or range) that can actually occur within that distribution can be set discretely as candidate a. In the example of Figure 9, three delay value candidates a are set: the average value n2 of the normal distribution, and n1 and n3 that are within a certain range from this average value n2 (0<n1<n2<n3).

In the above embodiment, in a certain prediction model b, from the viewpoint of ensuring prediction accuracy, learning or the like is performed individually for each candidate a of the delay value, and a different model M _ab is prepared for each candidate a of the delay value. In another embodiment, when the prediction model b is, for example, an RNN (recurrent neural network), a method of predicting all recognition information for a plurality of candidates a of the delay value together and a method of predicting each individually may be adaptively selected. The selection can be made by utilizing past prediction errors. For example, when it is determined that the error of the past prediction results is small, the recognition information is predicted together to reduce the processing load, and when it is determined that the error of the past prediction results is large, the recognition information is predicted individually to improve the prediction accuracy.

Fig. 10 is a diagram showing a modified example of the example of Fig. 7 as an example of this other embodiment. For the RNN of prediction model b=3, in addition to models _M13 , _M23 , and _M33 that individually predict the recognition information at each delay width of a=1, 2, and 3, a model _M03 that comprehensively predicts the recognition information at these three delay widths is prepared in advance. (That is, this model _M03 is a model that collectively predicts and outputs three pieces of recognition information in the future by three delay widths of a=1, 2, and 3.) The decision of which prediction model to use from among the prediction models shown in Fig. 10 may be made according to a predetermined rule according to the evaluated prediction error. For example, for a candidate delay value a (delay width T _a ), the prediction errors of the linear interpolation model M _a1 , the CNN model M _a2 , and the comprehensive RNN model M ₀₃ may be evaluated, and if the comprehensive RNN model M ₀₃ is evaluated as having the highest prediction accuracy, the prediction error of the individual RNN model M _a3 may be evaluated and the model with the best accuracy may be determined to be used, or the errors of the first four models M _a1 , M _a2 , M _a3 , and M ₀₃ may all be evaluated and the best model may be determined. The evaluation of the comprehensive RNN model M ₀₃ is not performed for each discrete processing timing i=1, 2, ... shown in step S25 of FIG. 5, but is performed for three times a=1, 2, 3 at once, so that it may be intermittently evaluated at intervals between these three times. In the example of FIG. 10, a model that predicts multiple times at once is added for the RNN, but a batch prediction model may be added for other learning models such as CNN in the same manner.

<Step S24...Rendering Unit 23 and Presentation Unit 24>
In step S24, the drawing unit 23 draws the avatar A1 of the first user U1 using the recognition information p(t+n) predicted for the current time i by the prediction unit 22, and the drawn avatar A1 is displayed on the presentation unit 24, and then the process proceeds to step S25. If audio information v(iK) is obtained from the recording unit 13, the presentation unit 24 also plays back the audio information v(iK) in synchronization with the display of the avatar A1.

The drawing in the drawing unit 23 may be performed by reflecting the facial expression, pose, etc. of the recognition information p(t+n) on the model of the avatar previously set by the first user U1. The avatar model may be a live-action model of the first user U1 himself or any other character, and the drawing manner on the image plane may be two-dimensional drawing or three-dimensional drawing (drawing in which a three-dimensional model is projected onto a two-dimensional image plane). When drawing an avatar in a three-dimensional virtual space, the three-dimensional virtual space may be determined by a virtual viewpoint specified by the second user using the drawing device 20, and the avatar may be drawn on the image plane. For the drawing, an existing method in augmented reality or three-dimensional computer graphics may be used, and the avatar may be displayed on the presentation unit 24 in a form in which the avatar is superimposed on a background image, etc., separately captured by the drawing device 20. If the display constituting the presentation unit 24 is an optical see-through type head-mounted display, etc., only the avatar may be drawn and superimposed on the background of the real world.

<Step S25...Time update>
In step S25, the current time i is updated to the next time i+1, and the process returns to step S21, whereby the above steps are similarly repeated for the next real-time time i+1.

As described above, according to the embodiment of the present invention, unreceived recognition information at the current time is predicted based on the delay in the recognition information transmitted from the recognition device 10, and the unreceived recognition information is approximated to the actual recognition information of the first user U1 at the recognition device 10 at the current time, which leads to effectively concealing transmission delays and processing delays, thereby realizing smooth communication. In addition, in presenting an avatar, the movement of the other party is predicted based on an adaptive selection of multiple prediction methods, and an avatar at the time of reception is presented, thereby suppressing transmission delays.

Figure 11 shows example graphs EX1, EX2, and EX3 in which a sample time series (actual value time series) of one parameter with recognition information is plotted simultaneously with a predicted value time series obtained by applying prediction by RNN to this actual value time series, for three candidate values of delay (i.e., predicted time) n1 = 30 ms, n2 = 100 ms, and n3 = 250 ms. The time axis is shared, and the actual value time series is shown with a thick line and the predicted value time series with a thin line, and it can be seen that the two match well. As such, since highly accurate prediction of recognition information is possible using a pre-trained RNN or the like, in an embodiment of the present invention, delays can be absorbed by highly accurate predictions, enabling smooth communication.

Various supplementary, additional, and alternative examples are explained below.

(1) According to an embodiment of the present invention, it is possible to realize realistic remote communication. This makes it possible to hold remote conferences and the like without necessarily requiring actual travel to remote locations, and by saving the energy resources required for user travel, it is possible to reduce carbon dioxide emissions, which makes it possible to contribute to Goal 13 of the United Nations-led Sustainable Development Goals (SDGs) "Take urgent action to combat climate change and its impacts."

(2) As the candidate a for the delay value K of the multiple prediction models M _ab prepared in advance in the prediction unit 22, only one predicted value at a discrete time (for example, 1 second later, 2 seconds later, 3 seconds later, etc.) can be obtained as illustrated in FIG. 7. Regarding the predicted value of this discrete delay width, only one discrete time closest to the specified delay time may be selected, or a value close to the specified delay time may be interpolated. For example, if the specified delay time is 1.2 seconds, the discrete predicted value at the closest time 1 second later may be used, or the weighted sum of the discrete predicted value at 1 second later and the discrete predicted value at 2 seconds later in a ratio of 0.8:0.2 may be used as the output of the prediction unit 22 as the two closest predicted values.

(3) As an additional process for making the avatar move smoothly and naturally even when the prediction method b of model M _ab is switched, the predicted value before and after the switch may be a value obtained by cross-fading the predicted values of the prediction models before and after the switch. For example, if the predicted value of the b=1 prediction model is used until time t=TA-1, and this is switched after time t=TA and the b=2 prediction model is determined to be the best, then for each time t=TA, TA+1, ..., TA+N during the N+1 subsequent fixed period, the value of the b=1 prediction model before the switch is p1(t) and the value of the b=2 prediction model after the switch is p2(t), the following cross-faded weighted sum value may be used to gradually switch from the value of the b=1 prediction model to the value of the new b=2 prediction model.
{(t-TA+1)*p2(t)+(TA+Nt)*p1(t)}／(N+1)

(4) FIG. 12 is a diagram showing an example of the hardware configuration of a general computer device 70. Each of the recognition device 10 and the drawing device 20 constituting the communication system 100 can be realized as one or more computer devices 70 having such a configuration. When each of the recognition device 10 and the drawing device 20 is realized by two or more computer devices 70, information required for processing may be transmitted and received via a network. The computer device 70 includes a CPU (Central Processing Unit) 71 that executes predetermined instructions, a GPU (Graphics Processing Unit) 72 as a dedicated processor that executes some or all of the execution instructions of the CPU 71 in place of the CPU 71 or in cooperation with the CPU 71, a RAM 73 as a main storage device that provides a work area for the CPU 71 (and the GPU 72), a ROM 74 as an auxiliary storage device, a communication interface 75, a display 76, an input interface 77 that accepts user input via a mouse, keyboard, touch panel, etc., a camera 78, a microphone 79, and a speaker 80, and a bus BS for transmitting and receiving data between them.

Each functional unit of the communication system 100 can be realized by a CPU 71 and/or a GPU 72 that reads from a ROM 74 a predetermined program corresponding to the function of each unit and executes it. Both the CPU 71 and the GPU 72 are a type of computing device (processor). Here, when display-related processing is performed, the display 76 also operates in conjunction with the GPU 72, and when communication-related processing related to data transmission and reception is performed, the communication interface 75 also operates in conjunction with the GPU 72. The avatar display function of the presentation unit 24 may be realized by the display 76, and the audio playback function may be realized by the speaker 80. The recording function of the recording unit 13 may be realized by the microphone 79. The imaging unit 11 may be realized as a camera.

100... communication system, 10... recognition device, 20... drawing device
11: imaging unit, 12: recognition unit, 13: recording unit, 21: estimation unit, 22: prediction unit, 23: drawing unit, 24: presentation unit, 25: storage unit

Claims (8)

receiving, in chronological order, control parameters for drawing the avatar from a user for whom an avatar is to be drawn, as recognition information that recognizes the state of the user;
an estimation unit that estimates a delay amount between a time of the most recent past recognition information among the already received recognition information and a current time;
a prediction unit for predicting recognition information corresponding to the current time based on a history of the received recognition information;
a drawing unit that draws an avatar of the user at the current time using the predicted recognition information ;
The prediction unit predicts recognition information corresponding to the current time by using a prediction model selected from a plurality of types of prediction models prepared in advance;
The drawing device, wherein the prediction unit selects a prediction model for a candidate delay amount corresponding to the delay amount from among a plurality of prediction models for each candidate delay amount prepared in advance . receiving, in chronological order, control parameters for drawing the avatar from a user for whom an avatar is to be drawn, as recognition information that recognizes the state of the user;
an estimation unit that estimates a delay amount between a time of the most recent past recognition information among the already received recognition information and a current time;
a prediction unit for predicting recognition information corresponding to the current time based on a history of the received recognition information;
a drawing unit that draws an avatar of the user at the current time using the predicted recognition information ;
The prediction unit predicts recognition information corresponding to the current time by using a prediction model selected from a plurality of types of prediction models prepared in advance;
A drawing device characterized in that the prediction unit selects, for a predetermined type of prediction model, from among a model that predicts each of a plurality of candidate delay amounts individually and a model that predicts a plurality of candidate delay amounts collectively . 3. The drawing device according to claim 2 , wherein the predetermined type of prediction model is a recurrent neural network model. The drawing device according to any one of claims 1 to 3, characterized in that the prediction unit applies each of the multiple types of prediction models to the history of the recognition information that has already been received, evaluates the prediction accuracy of each prediction model, and selects the prediction model with the best prediction accuracy. receiving, in chronological order, control parameters for drawing the avatar from a user for whom an avatar is to be drawn, as recognition information that recognizes the state of the user;
an estimation unit that estimates a delay amount between a time of the most recent past recognition information among the already received recognition information and a current time;
a prediction unit for predicting recognition information corresponding to the current time based on a history of the received recognition information;
a drawing unit that draws an avatar of the user at the current time using the predicted recognition information ;
On the side of the user for whom the avatar is to be drawn, frame images in a time series obtained by imaging the user are analyzed to acquire the recognition information, and timestamps of the frame images are associated with the corresponding recognition information and received;
The estimation unit estimates the delay amount as a difference between the timestamp and a current time,
At the user for whom the avatar is to be rendered, recordings of the user's utterances are acquired in chronological order, with the timestamps being linked to the recordings;
Receive the recording;
The drawing device, wherein the estimation unit estimates the amount of delay by further subtracting a delay including at least a transmission delay until the recording is received from the difference. 6. The drawing device according to claim 1 , wherein the recognition information is obtained as representing a facial expression and/or a body pose of the user. The prediction unit predicts recognition information corresponding to the current time by using a prediction model selected from a plurality of types of prediction models prepared in advance;
7. The drawing device according to claim 1, wherein, when the type of prediction model selected is switched at a certain time, during a certain period of time after the switch, recognition information corresponding to the current time is predicted as a weighted sum of recognition information corresponding to the current time using a first type prediction model before the switch and recognition information corresponding to the current time using a second type prediction model after the switch . 8. A program for causing a computer to function as the drawing device according to claim 1.

Images